One of the things I notice is that at the end of a sub routine there are typicially two types of returns, both with the mnemonic RET, but with different op-codes.
There are, actually, 4 different opcodes for RET. 0xC3 is for NEAR return, 0xCB is for FAR return (retf), 0xC2 is the same as RET near, but add an immediate value to (R|E)SP. And there is 0xCA, which is the same as a far return, but adds the immediate value as 0xC2 does.
The two op-codes I've seen are C2 and C3. C3 takes no parameter. C2 takes a two byte "word" parameter. What I can't figure out is what you are supposed to pass to that parameter.
Certain languages, for example, PASCAL, passes arguments through the stack and expect the
called function to do the cleanup. Others, like C, do the cleanup in the
caller. Let's say we have, in PASCAL:
function F( x : Integer ) : Integer;
begin
F := 2 * x;
end;
The pascal compiler will, probably, create something like:
F:
mov eax,[esp+4]
add eax,eax
ret 4 ; dispose of the argument on the stack.
But a C compiler with a function like this:
int f( int x ) { return x + x; }
Probably will create:
_f:
mov eax,[esp+4]
add eax,eax
ret ; DON'T dispose of the argument on the stack
; leting the caller to do this
When f() is called, the C compiler does:
; Example: y = f(2); -- supose y is ECX.
push 2 ; pushes the 2 argument to the stack
call _f
add esp,4 ; clears the stack
mov ecx,eax
I also notice that just prior to the RET, there is usually a LEAVE command.
LEAVE is an old instruction and should not be used anymore. Essentially it does a pair of instructions at once: `mov esp,ebp/pop ebp`, but this isn't necessary anymore... Not even the usage of EBP as a substitute for using ESP, because, since 386 ESP
can be used in an effective address (that address between [ and ]). Until 286 only BX,BP,SI and DI could be used. 386 changed this.
Using NASM you can use structures to make sure the arguments are in their proper positions without having to calculate the relative stack position yourself... Let's take a look at f(), above... Before you call f() the 2 is pushed to the stack, so ESP points to the position where this 2 is... The CALL instruction pushes contents of EIP, so RET will know where to return to, then CALL jumps to f. Your stack will be like this after the CALL:
ESP+4 -> 2
ESP -> EIP
Notice that we can use a structure like this:
struc fstk
resd 1 ; offset 0, where ESP points to... the return address.
.x: resd 1 ; offset 4, where out argument is.
endstruc
_f:
mov eax,[esp + fstk.x] ; fstk.x is the offset 4
add eax,eax
ret
No need to use `EBP` instead of `ESP`.