Before we explain the mechanisms of the buffer overflow exploits, it is impotant to understand the basics of process organization in memory.
Each process is divided into three regions:
Both PUSH and POP instructions are in fact a two-step process: first, the data is put on (or remove from) the top of the stack, and next the stack pointer register is updated. The stack grows downward when the stack pointer register is decreased for the PUSH instruction, pointing to lower memory address as the amount of data on the stack increases. Moreover, the stack pointer register is increased for the POP operation, pointing to higher memory address as the amount of data on the stack decreases. The stack grows upward when stack pointer is increased for the PUSH operation and is decreased for the POP operation. On the Intel processors, the stack grows down. Therefore, we will follow this model in our examples.
We will illustrate how the data is organised in the stack using the following example (Listing 1):
void myfunction (char *str, int len) { char buffer[20]; #ifdef BUFFER_OVERFLOW strcpy(buffer,str); #endif } void main() { myfunction(0,8); }
Listing 1: example1.c
We will compile example1.c
with the –S
switch to generate assembly code to examine how the parameters to a function (in this case myfunction
) and its local variables are organised on the stack.
$ gcc –S example1.s example1.c
The file example1.s
contains the following instructions (comments were added after compilation):
myfunction: pushl %ebp # save base frame pointer [3] movl %esp, %ebp subl $40, %esp # reserve space for local variables [4] leave ret # return from call [5] main: . . . . . . pushl $8 # save argument len [1] pushl $0 # save argument str [1] call myfunction # pass control to function and put [2] . . . # return address on the stack . . .
Listing 2: example1.
As illustrated, the C call to myfunction(str,len)
in main()
(Listing 1) translates to more than just one line of the assembler code as shown in Listing 2. Based on the above example, we will describe the process of the function call and its interaction with the stack.
We can present the process of passing control to a function in the following steps:
When the function returns from the call [5], the saved (in [2]) return address (ret) will be used to pass control to the instruction just after the call.
In this example, the exploit occurs when the return address is overwritten by some data. This occurs when there is no boundary checking when moving data to the buffer (calling strcpy
in Listing 1). Usually, the moved data is random and the result will be the segmentation violation.
Nearly 100 percent of the time the segmentation violation is the result of the buffer overflow because the buffer contains the random data not executable code. The operating system traps this situation and stops the process execution, so there will be no damage to the other parts of the system except the affected process.
However, what happens if the buffer contains an executable program, which will start executing without our knowledge? This situation is not desirable. It is almost impossible that the random data will execute for long time without a crash. Therefore, at this stage, we have to do a little bit hacking to get the expected results.
>> Examples
This article was originally published on LinuxPlanet.
Property of TechnologyAdvice. © 2025 TechnologyAdvice. All Rights Reserved
Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.