64 bit Linux stack question

[Gunner]

I am in the process of converting one of my 32 bit MASM programs which uses the Win32 API, to Linux using libcURL, cstdlib, and GTK+3 and of course NASM (at about 1500 lines of 14,000 converted so have a ways to go)

Red Zone - We can only use this if our function is a leaf function (does not call another function), what happens to the red zone if we use it in a non leaf function?  I am guessing since functions don't need an epilogue to save rsp, our red zone gets over written.

Alignment - at the start of every function we need to be 16 byte aligned; fine, call pushes 8 bits so we just need to sub rsp, 8 at the beginning of the function and add rsp, 8 at the end.  Of course if we push one register, we don't need to sub rsp.

Here is the thing, why does printf (or any other vararg function) segfault if we don't align the stack and zero rax if we don't use xmm regs?

This will segfault:

Code: [Select]

SomeFunction:
    mov     rsi, somestring
    mov     rdi, fmtstr
    call    printf
    retThis won't:

Code: [Select]

SomeFunction:
    mov     rsi, somestring
    mov     rdi, fmtstr
    mov     rax, 0
    call    printf
    retThis won't segfault either:

Code: [Select]

SomeFunctionAligned:
    sub     rsp, 8
    mov     rsi, somestring
    mov     rdi, fmtstr
    call    printf
    add     rsp, 8
    retnor will this:

Code: [Select]

SomeFunctionAligned:
    sub     rsp, 8
    mov     rsi, somestring
    mov     rdi, fmtstr
    mov     rax, 0
    call    printf
    add     rsp, 8
    ret

[encryptor256]

x86-64

This is rule on linux:

Stack aligned on 16 bytes boundary. 128 bytes red zone below stack.

I understand it this ways - The last thing you allocate in procedure is this redzone and don't touch it and make it sure that stack is aligned.

RBP or stack frame is used for local variable \ local stack space addressing, after that, comes red zone.

I think this is how it should be on linux:

Code: [Select]

SomeFunctionAligned:
  push rbp
mov rbp,rsp
sub rsp,0x80

mov rsi,txt_format
mov rdi,number
call printf

add rsp,0x80
pop rbp
ret

I think this is how it should be on windows:

Code: [Select]

SomeFunctionAligned:
push rbp
mov rbp,rsp
sub rsp,0x20

mov rcx,txt_format
mov rdx,number
call printf

add rsp,0x20
pop rbp
ret

I think stack frames and red zones are used for exception handling or debugging.

If you call external function like "printf" then you definitely need to allocate stack and red zone.

Did I answered at least of one of your questions? - NO?! If I were you I would compile C or C++ program on linux, then study and consult the assembly source code behind it! - Easy, peasy!

Bye,
Encryptor256.

[Rob Neff]

Hi, Gunner,

Red Zone - We can only use this if our function is a leaf function (does not call another function), what happens to the red zone if we use it in a non leaf function?  I am guessing since functions don't need an epilogue to save rsp, our red zone gets over written.

I talked briefly regarding these questions in a previous thread of yours.

The red zone exists beneath RSP.  It is an area on the stack that a function may use to temporarily store data and is guaranteed not to be mucked with by the operating system.  Since a leaf function makes no additional calls the 128 bytes below RSP will not be modified - not even by the operating system during an interrupt.  That's why you can safely use that area for local variables.  If your function is not a leaf function then your stack will obviously be modified by the called function thus you will have no idea what the called function did to that area.

Alignment - at the start of every function we need to be 16 byte aligned; fine, call pushes 8 bits so we just need to sub rsp, 8 at the beginning of the function and add rsp, 8 at the end.  Of course if we push one register, we don't need to sub rsp.

Pretty much.  That's why NASMX always uses RBP as the register pushed.  That way if the defined function has either local variables or stack parameters we can reference them as we previously did with the C calling convention ( eg: mov rax, [rbp+16] ).

Here is the thing, why does printf (or any other vararg function) segfault if we don't align the stack and zero rax if we don't use xmm regs?

I've not traced through x64 printf to determine the rax requirement, so please take the following with a grain of salt, but my suspicions would leave me to believe that, without it, varargs won't know when to stop using xmm regs and when to reference the stack for any remaining doubles.  It may be that simple or may be something more complicated.  Unfortunately, I can't say for sure.  However, I will say that the convention of number of xmm regs used contained in rax is also used on the BSDs in addition to Linux so apparently there's an x64 usage "standard" here.  Hopefully, we'll get Peter, Cyril, or someone more knowledgable to clear this up for both of us.