Author Topic: Nasm64 & C combination - epilogue,prologue, stack alignment - OSX64  (Read 11176 times)

Offline rouki

  • Jr. Member
  • *
  • Posts: 5
The following program crashes (Segmentation fault):

Code: [Select]
//something.c



int somefunc3();
void somefunc2();

void* globalptr;

void somefunc1(void* regs)
{
    globalptr = regs;   
    somefunc2();
}

int foo()
{

    return somefunc3();
}


int main(void)
{
    show_all_registers();
    foo();
    show_all_registers();
}

Code: [Select]
//something1.asm

extern _somefunc1

global _somefunc2
global _somefunc3

section .text


%macro RESTORE_REGISTERS 0
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
    pop rcx
%endmacro

%macro SAVE_REGISTERS 0
    push rcx
    push rcx
    push rcx
    push rcx
    push rcx
    push rcx
    push rcx
    push rcx       
    push rcx
    push rcx
    push rcx
    push rcx   
    push rcx
    push rcx
    push rcx
    push rcx
%endmacro

_somefunc3:
    push rbp
    mov rbp, rsp
    SAVE_REGISTERS
    mov rdi, rsp

    sub rsp,8
    call    _somefunc1
    add rsp,8

    pop rbp
    ret

_somefunc2:
    push rbp
    mov rbp, rsp
    RESTORE_REGISTERS
    pop rbp
ret

few notes:

Please don't try to understand what this program does because you won't find anything that makes sense. This is just a user mode app which I created in order to understand something.

show_all_registers is just a function that prints out to the screen all of the 64 bit registers.

Here's what happens before it crashes:

64 Bit registers:
RAX=10767ad00, RCX=1, RDX=10767ab70, RBX=0, RSP=7fff58585bd0, RBP=7fff58585bd0, RSI=20000000200,     RDI=7
Segmentation fault: 11
Using GDB it seems that the crash occurs on somefunc2 (When restoring the registers)

I think that it has something to do with stack alignment or the epilogues & epilogues I wrote for the ASM functions. Still kinda newbie so it is most likely something silly.

thanks

Offline encryptor256

  • Full Member
  • **
  • Posts: 250
  • Country: lv
  • Win64 .
    • On Youtube: encryptor256
Re: Nasm64 & C combination - epilogue,prologue, stack alignment - OSX64
« Reply #1 on: October 12, 2014, 08:08:28 AM »
The following program crashes

Congrats!

Please don't try to understand

All right!

Here's what happens before it crashes:

64 Bit registers:
RAX=10767ad00, RCX=1, RDX=10767ab70, RBX=0, RSP=7fff58585bd0, RBP=7fff58585bd0, RSI=20000000200,     RDI=7
Segmentation fault: 11
Using GDB it seems that the crash occurs on somefunc2 (When restoring the registers)

Interesting!

I think that it has something to do with

Thinking is good, but not enough, but it will get you somewhere! Sometimes thinking is the cause of the problem.

Still kinda newbie so it is most likely something silly.

Certainly!

thanks

Thanks for the source code that throws exceptions, wish you luck there!
Encryptor256's Investigation \ Research Department.

Offline Frank Kotler

  • NASM Developer
  • Hero Member
  • *****
  • Posts: 2667
  • Country: us
Re: Nasm64 & C combination - epilogue,prologue, stack alignment - OSX64
« Reply #2 on: October 12, 2014, 09:11:26 AM »
You are correct, Rouki. I don't understand a word of it. My first thought is "this'll never work". I am unable to test 64-bit code at the moment, and I don't understand it very well anyway...

However... it looks to me as if your epilogues are "incomplete".
Code: [Select]
the_func:
    push rbp
    mov rbp, rsp

    ; trash my stack
    ; but please don't trash rbp!

    mov rsp, rbp ; <-
    pop rbp
    ret
That might just "save your asm"... maybe.

Quote
kinda newbie

"Everybody knows" that call and ret use the stack. Nobody is born knowing it. If you're new enough that you don't... you'll never learn any younger. :) That could explain your problem.

It looks as if "show_all_registers" is crashing before it finishes, but perhaps that's all it does. If GDB says it crashes in somefunc2, it probably does, but I would bet not until it gets to the ret, not while "restoring regs" (which it doesn't).

Try "completing" your epilogues and see if that helps any...

Best,
Frank