x64 calling convention questions

[nullptr]

hi guys,

i'm confused about calling functions in x64. I heard, that if the function is called it needs always to make a shadow space for rcx, rdx, r8, r9 on the stack. Even if the proc needs less then 4 args. Cleaning after call is also needed.

First question is do i understood this correct and second - wired thing to me is:

   mov   rcx, rax
   mov   rdx, offset msg
   mov   r8, lengthof msg
   mov   r9, offset BytesWritten
   mov   qword [rsp + 8 * 4], 0
   call   WriteConsole

mov qword [rsp + 8 * 4], 0

i think it should be:

sub    rsp, [4 * 8]
push    0

how this affects on the frames of other functions, that are called by WriteConsole if the stack pointer remains unchanged after
mov qword [rsp + 8 * 4], 0

thanks in advance

[encryptor256]

Hi!

All your questions can be answered by reading this:

• x64 Software Conventions: http://msdn.microsoft.com/en-us/library/7kcdt6fy.aspx

I heard, that if the function is called it needs always to make a shadow space for rcx, rdx, r8, r9 on the stack

Answer can be found on "x64 Software Conventions", link above.

Cleaning after call is also needed.

No, somebody has lied to you about this!
Answer can be found on "x64 Software Conventions", link above.

Some answers also could be found here:

• Annotated x64 Disassembly: http://msdn.microsoft.com/en-us/library/windows/hardware/ff538083(v=vs.85).aspx

[nullptr]

No, somebody has lied to you about this!

look at this linki:
http://blogs.msdn.com/b/oldnewthing/archive/2004/01/14/58579.aspx
the fifth dot says:

"The callee does not clean the stack. It is the caller's job to clean the stack."

it is a bit old reference. So what is right?
Thanks for reply.

[encryptor256]

Cleaning after call is also needed.

Well, it seems, your right then, im just mixing up.
I was thinking that you are talking about cdecl,
clearing stack after function call.

CDECL:
push dword [num]
push format
call printf ; Clear stack needed
add esp,dword 8

64BIT:
mov rdx,qword [num]
mov rcx,format
call printf; Clear stack not needed

P.S.
I was thinking that you are the one who is mixing something up, but it seems it's me!

[nullptr]

I'm not familiar with this new x64 fastcall convention. So i'm not sure whether this what i write below is right, but if you call in 64 bits you should also make a "spill" space on the stack for the regs.
first dot in previous link:

"The first four parameters to a function are passed in rcx, rdx, r8 and r9. Any further parameters are pushed on the stack. Furthermore, space for the register parameters is reserved on the stack, in case the called function wants to spill them; this is important if the function is variadic."

as far as i know there is only one calling convention in x64 - fastcall. but i'm noob and as i said i'm not familiar with 64 bits. so if there is someone how knows better i'll gladly hear him. and call in x64 should probably look like this.

mov rdx,qword [num]
mov rcx,format
sub rsp, [4 * 8]  ;spill space
call printf
add rsp, [4 * 8]  ;cleaning stack

[encryptor256]

If take a look at your previous link, then your current code is not very pleasant.

mov rdx,qword [num]
mov rcx,format
sub rsp, [4 * 8]  ;spill space
call printf
add rsp, [4 * 8]  ;cleaning stack

What is this "[4 * 8]"?

He clearly shows how to do that,
let's say this is main:
let's say this is avoidmain:

Code: [Select]

avoidmain:
        sub    rsp, 0x28
        mov     dword ptr [rsp+0x20], 5     ; output parameter 5
        mov     r9d, 4                      ; output parameter 4
        mov     r8d, 3                      ; output parameter 3
        mov     edx, 2                      ; output parameter 2
        mov     ecx, 1                      ; output parameter 1
        call    SomeFunction                ; Go Speed Racer!
When SomeFunction returns, the stack is not cleaned, so it still looks like it did above. To issue the second call, then, we just shove the new values into the space we already reserved:

        mov     dword ptr [rsp+0x20], 10    ; output parameter 5
        mov     r9d, 9                      ; output parameter 4
        mov     r8d, 8                      ; output parameter 3
        mov     edx, 7                      ; output parameter 2
        mov     ecx, 6                      ; output parameter 1
        call    SomeFunction                ; Go Speed Racer!
CallThatFunction is now finished and can clean its stack and return.
        add     rsp, 0x28
        ret

Catchy name for rsp register, rsp => Respirator!

You allocate space only once, at functions prolog,
deallocate at epilog.
You have to plan your local variables, stack,...

• Prolog and Epilog: http://msdn.microsoft.com/en-us/library/tawsa7cb.aspx

And that's it!

P.S.
This code works always:

Code: [Select]

         jmp $


[nullptr]

Thanks for help. I need read more about it.

[Rob Neff]

i'm confused about calling functions in x64. I heard, that if the function is called it needs always to make a shadow space for rcx, rdx, r8, r9 on the stack. Even if the proc needs less then 4 args. Cleaning after call is also needed.

For x64 Windows programming you must always reserve space on the stack prior to the call for the 4 register parameters whether you use all, or any, of that space or not.  The only exception to this rule is a leaf function - a function that does not make any additional calls.  The stack location for function parameters, parameter order, and the stack cleanup required by the caller is just like good ol' CDECL.

As Encryptor256 mentioned there is an optimization available that enables you to create your stack frame ( in the prologue ) the size of the largest call ( the one with the most parameters ) you will make within your own function and use that frame for all calls executed within your function.  Then only cleanup and the end of your function ( epilogue ).