¶ … proportion of attacks based on buffer overflows is increasing each year-in recent years, buffer overflow attacks have become the most widely used type of security attack . Buffer flow attacks are believed to have begun as early as the 1960's, but became commonly known in 1988 with the infamous Internet Worm attack that infected tens of thousands of hosts. The most popular form of buffer flow exploitation is to attack buffers on the stack, referred to as stack smashing attack (Baratloo, Singh and Tsai). As a counter measure, hardware vendors have added the ability to turn off stack execution. For example, Sun has added the ability to turn off stack execution on more recent versions of Solaris. Although effective, turning off stack execution isn't always feasible for many operating environments for a variety of reasons. Even so, this feature is helpful to many implementations and is better than other alternatives. Figure 1: Number of Reported CERT Security Advisories and the Number Attributable to Buffer Overflow

Source: Proceedings 7th Network and Distributed System Security Symposium

Buffer overflows allow attackers access to the process stack. A buffer overflow attack places new programmatic code on a process stack and causes the process to execute this code when returning from a function (Roamer, 2000). Thus, the code is executed with the same rights as the running process, allowing the attacker to control the program. Once this happens, the intruder can extract maximum damage by attacking setuid 0 programs and any daemons running as root.

To understand how the process stack...

The stack starts at a high memory address and works its way down to a low memory address. Things are either pushed onto the stack or popped off the stack. When something is pushed onto the stack, the value that is being pushed is copied into the memory location pointed to by the stack pointer, and the stack pointer is decremented to reflect the next spot on the stack. When a function is called, local arguments are pushed onto the stack, then the return address (code segment), then the old base pointer (so it's known where on the stack you were before this function was called), and then local variables to that function.
All attackers need to do is find a program that will let them insert data into it that doesn't check the length of the data (Bijjam). If the data written is outside of the process address space, the function will get a segmentation violation when it returns and tries to read the next instruction. The buffer overflow then allows the attack to change the return address of a function so that they can change the flow of execution of the program so that the program that spawns a shell is executed. From this shell, the attacker issues commands of their choice.

Making the stack non-executable is a commonly proposed method for stopping buffer overflow attacks. However this method has several drawbacks and may be inappropriate in certain situations (Bijjam). For example, patching and recompiling the kernel is not…