Make:

For the uninitiated, a buffer overflow is what happens when user supplied data exceeds the storage space that was allocated to hold it. When this happens, the excess data overwrites adjacent areas of the program's memory, namely the "stack." The stack is where a program stores function variable data and return pointer information which is needed by the program to know where to return to when one of its functions has finished execution.

An inadvertant buffer overflow can cause a program to crash, or an unwanted glitch to occur. If you've ever applied a security update to your system, however, you've probably heard mention that a particular buffer overflow was found such that "under specific circumstances, exploitation of the buffer overflow could lead to the execution or arbitrary code or system compromise." I made that one up, but the stock warning is usually something like that.

Usually, a buffer overflow bug is exploited by filling the stack with executable code and overwriting the return address pointer, altering it to point at your code instead of the the real calling function. This is a stack-based buffer overflow exploit. In the case of the iPhone and iPod touch, however, this method can't be used.

The devices have a mechanism which prevents execution of code in the stack. Worse yet, the return pointer is stored in a register and not on the stack.

Word on the street, though, is that the devices do not prevent execution of code in the heap. The heap is where runtime-allocated variable data is stored and this is where a heap-based buffer overflow exploit might be of use. I suspect that the brilliant person who figures out how to get full access to the iPod touch or the 1.1.1 firmware iPhone will do it with a heap-based overflow.

With a heap-based overflow, the desired executable code payload is located in the heap. Ie. it's in runtime allocated memory, say as part of a big TIFF image that's loaded by Safari. The buffer overflow exploit essentially works the same, but instead of overwriting the stack pointer with an address in the stack, you overwrite it with a pointer to the heap, directing program execution back to the payload.

Wait a minute... what about the stack not being used for the return pointer? This is the big complication. Because the return pointer is stored in the link register, it's not as simple as overwriting the return pointer in the stack. Instead, you'd need to stumble across a saved link register value or maybe a function pointer in the stack and overwrite it with the location of the tiff payload data in the heap. Yikes.

The 1.0.2 iPhone software can be debugged, though, and we know what libtiff's internals are, so maybe this isn't totally impossible. It's way over my head, but there are a lot of taller programmers out there.

The upside is that, unlike the stack, the heap is not very limited in space. You could make an absolutely huge payload in the TIFF, fill it with lots of empty "no op" instructions and the desired payload at the end. Assuming you could stumble across a suitable function pointer to modify, and you could make a crude guess of where in the heap that TIFF was, you'd have a pretty good chance of hitting the payload code. You win if you hit any location in the no op portion of the payload, and the bigger the payload is, the easier this gets.

More reading:
Buffer overflow discussion on iPhone Dev Wiki - Link
TIFF Exploit discussion on iPod Touch Dev Wiki - Link
w00w00 on Heap Overflows: great technical article on several heap-based overflow approaches - Link
Buffer overflow @ Wikipedia - Link