newbie amd64 linux opengl calls

[gens]

hello,
just a quick newbie question

im doing some drawing in opengl for fun
i found alot of tutorials on 32bit asm, but not many on 64bit so i took a 32bit triangle example(glut+gl) and tried modifing it as i got a 64bit sistem
after alot of head bashing around the difference in registers, stack form(still not clear, but dont matter) and so on yestrday i found out that the problem was in the calling convention and i finaly got a window to draw in

now the problem i got is... hard for me as a newbie to explain.. basicly it dosent draw anything

as i got it, it dosen't send the GL(display) part to GLUT (does it have to?)
i figured that by that in gdb dissasembly id dosent show that part of the program
does it have to be on the stack then pushed ?

anyway, the modified version
http://pastebin.com/pnJNC8Ge

and for reference the 32bit original
http://pastebin.com/rsmApSca

i tried changing floats to dwords as the registers RDI, RSI, RDX, RCX, R8, and R9 are used for integer
but i think the problem is i just dont understand asm so good yet

[gens]

PS(cant edit previous post)

another peculiarity i found was the window is not squared when i put the sides to be the same

[gens]

just realised that i just dissasambled "main" part, so i guess the "push display" calls the display part?
shud be the same for amd64?
problem in the stack ?

[gens]

ah, its for a callback function

callback works, tested

problem is now in the glcolor3f or glvertex3f now
as the f is for floating point and the amd64 calling convencion states FPU's go in the xmm? registers i tried to put the numbers there, dont work

new version:
http://pastebin.com/ZCXnybxJ

[Frank Kotler]

Someday real soon I've gotta upgrade! I know nothing about 64-bit code... but isn't there something in there about putting the number of floating-point parameters in al? I wonder if that would help?

Best,
Frank

[gens]

thx for the reply
good thought as there is something with vector registers, but i think it is declared as 3 hence the 3f (tried anyway, dont work:))

dissasembling a square example(shudn't matter much that its square)

Code: [Select]

  400c6b:       f3 0f 10 0d 51 02 00    movss  0x251(%rip),%xmm1        # 400ec4 <_IO_stdin_used+0x3c>
  400c72:       00
  400c73:       f3 0f 10 05 49 02 00    movss  0x249(%rip),%xmm0        # 400ec4 <_IO_stdin_used+0x3c>
  400c7a:       00
  400c7b:       e8 f0 fd ff ff          callq  400a70 <glVertex2f@plt>
and mine

Code: [Select]

  400b6c:       f3 0f 10 04 25 ba 12    movss  0x6012ba,%xmm0
  400b73:       60 00
  400b75:       f3 0f 10 0c 25 c6 12    movss  0x6012c6,%xmm1
  400b7c:       60 00
  400b7e:       f3 0f 10 14 25 c6 12    movss  0x6012c6,%xmm2
  400b85:       60 00
  400b87:       e8 14 fe ff ff          callq  4009a0 <glVertex3f@plt>

what i know im doing wrong is that i didnt make a stack frame, but as i figured thats for kernel security or something like that and shudnt make it not work
il try something again tomorow

the case with 64vs32bit sofar that i gathered is:
 system calls are different(http://www.acsu.buffalo.edu/~charngda/linux_syscalls_64bit.html)
 calling convention is from the registers, not the stack (for speed, and they are divided by classes)
 stack needs to be 16*N bit wide
 push and pop are 8bit, not 4
 registers are usualy taken as 32bit, but padded to qwords (still fuzzy on that; and was something about 4+1bit, think it was about pointers)
http://www.x86-64.org/documentation/abi.pdf is the best resource i found, but its giving me headackes

a couple generic nasm questions in the meantime if i might ask:
 why everybody put % infront of registers, does it matter ?
 instruction pointer can be manipulated to point anywhere in the program, like loop around and such ? (thats the part thats different in the parts above)
 and one that i cant find anywhere an answer; where is the stack and how big is it ?

[Frank Kotler]

Well... "%" in front of registers is AT&T syntax (what your disassembly shows) - you shouldn't use it in Nasm.

The instruction pointer, rip, should be able to point anywhere in your program (in .text section) and should loop around as usual. (AFAIK)

It's up to the OS "where" the stack is, and it should grow as necessary, so it should be "as big as it needs to be"... until you run out of memory. You "shouldn't" need to worry about that.

One minor nit-pick...

push and pop are 8bit, not 4

I'm sure you mean 8/4 bytes, not bits.

Wish I could help more...

Best,
Frank

[Bryant Keller]

I'm probably the worst person to be giving advice on this topic since I haven't updated to 64-bit systems and I tend to avoid game/graphics programming... however, from what I've read on the subject, you need to make sure that there is enough space on the stack for all 4 registers (even if you don't use them) for compatibility with "unprototyped functions" and vararg C/C++ functions. My modified version of your code simply provides this space... I don't have a 64-bit system to try this on, so let me know if it works.

Code: [Select]

; ----------------------------------------------------------------------------
; triangle.asm
;
; A very simple *Linux* opengl application using the glut library.  it
; draws a nicely colored triangle in a top-level application window.
; Version of numit_or: numit_or@cantv.net
;
; Compile with:
;    nasm -felf triangle.asm -o triangle.o
;    gcc triangle.o -o triangle -L/usr/X11R6/lib -lGL -lglut -lglu
;
; you can find nasm in:
;   http://nasm.sourceforge.net/
;
; ----------------------------------------------------------------------------
;
; libGL is in
;    libMesaGL1 pack (Mandrake)
;   mesag3 - mesag-dev - mesa-common-dev packs (Debian)
; libglut and libGLU are in:
;    libMesaglut pack (Mandrake)
;   freeglut3 - freeglut3-dev - libglut3 - libglut3-dev (Debian)
;
; ----------------------------------------------------------------------------
   global   main
   extern   glClear
   extern   glBegin
   extern   glEnd
   extern   glColor3f
   extern   glVertex3f
   extern   glFlush
   extern   glutInit
   extern   glutInitDisplayMode
   extern   glutInitWindowPosition
   extern   glutInitWindowSize
   extern   glutCreateWindow
   extern   glutDisplayFunc
   extern   glutMainLoop

GL_COLOR_BUFFER_BIT   equ    16384
GL_POLYGON      equ   9

section   .data
title   db   'A Simple Triangle', 0
zero   dd   0.0
one   dd   1.0
half   dd   0.5
neghalf dd   -0.5

section .text

display:
   sub rsp, 32h

      mov     rdi, GL_COLOR_BUFFER_BIT
      call    glClear         ; glClear(GL_COLOR_BUFFER_BIT)

      mov     rdi, GL_POLYGON
      call    glBegin         ; glBegin(GL_POLYGON)

      movss   xmm0, [zero]
      movss   xmm1, [zero]
      movss   xmm2, [one]
      call    glColor3f      ; glColor3f(1, 0, 0)

      movss   xmm0, [zero]
      movss   xmm1, [neghalf]
      movss   xmm2, [neghalf]
      call    glVertex3f      ; glVertex(-.5, -.5, 0)

      movss   xmm0, [zero]
      movss   xmm1, [one]
      movss   xmm2, [zero]
      call    glColor3f      ; glColor3f(0, 1, 0)

      movss   xmm0, [zero]
      movss   xmm1, [neghalf]
      movss   xmm2, [half]
      call    glVertex3f      ; glVertex(.5, -.5, 0)

      movss   xmm0, [one]
      movss   xmm1, [zero]
      movss   xmm2, [zero]
      call    glColor3f      ; glColor3f(0, 0, 1)

      movss   xmm0, [zero]
      movss   xmm1, [half]
      movss   xmm2, [zero]
      call    glVertex3f      ; glVertex(0, .5, 0)

      call    glEnd         ; glEnd()

      call    glFlush         ; glFlush()

   add rsp, 32h
   ret

main:
   sub   rsp, 32h

      mov   rsi, rcx
      call  glutInit

      mov   rsi, 0
      call  glutInitDisplayMode

      mov   rsi, 100
      mov   rdi, 100
      call  glutInitWindowPosition

      mov   rdi, 400
      mov   rsi, 300
      call  glutInitWindowSize

      mov   rdi, title
      call  glutCreateWindow

      mov   rdi, display
      call  glutDisplayFunc

      call  glutMainLoop

   add rsp, 32h
   leave  ;;;< is this even needed anymore?
   ret

[gens]

Frank, thx for the answers. And ye its bytes , am new to asm so its all still bit chaotic, needs to settle in.

Bryant, nop its not it, but i think reserving space is necessary because of the kernel(security), probably to avoid memory leaks.
In short, i have no clue , something to figure out another day.

But! On another note, and the reason for my nice typing... I got it working!
Problem was in the order the numbers were moved to the registers. Why? I don't know, its something to figure out.
Disassembling showed me that the first time, but it didn't seem all that important. Have tried it now just for the hell of it.
Oh, and i see gcc uses xorps xmms?, xmms? to put all zeros to a registry, bit of gugling showed bitwise operations should be a lot faster.

anyway, tada!

Code: [Select]

; ----------------------------------------------------------------------------
; triangle.asm
;
; A very simple *Linux* opengl application using the glut library.  it
; draws a nicely colored triangle in a top-level application window.
; Version of numit_or: numit_or@cantv.net
;
; Compile with:
;    nasm -felf triangle.asm -o triangle.o
;    gcc triangle.o -o triangle -L/usr/X11R6/lib -lGL -lglut -lglu
;
; you can find nasm in:
;   http://nasm.sourceforge.net/
;
; ----------------------------------------------------------------------------
;
; libGL is in
;    libMesaGL1 pack (Mandrake)
;   mesag3 - mesag-dev - mesa-common-dev packs (Debian)
; libglut and libGLU are in:
;    libMesaglut pack (Mandrake)
;   freeglut3 - freeglut3-dev - libglut3 - libglut3-dev (Debian)
;
; ----------------------------------------------------------------------------
   global   main
   extern   glClear
   extern   glBegin
   extern   glEnd
   extern   glColor3f
   extern   glVertex3f
   extern   glFlush
   extern   glutInit
   extern   glutInitDisplayMode
   extern   glutInitWindowPosition
   extern   glutInitWindowSize
   extern   glutCreateWindow
   extern   glutDisplayFunc
   extern   glutMainLoop

GL_COLOR_BUFFER_BIT   equ    16384
GL_POLYGON      equ   9

section   .data
title   db   'A Simple Triangle', 0
zero   dd   0.0
one   dd   1.0
half   dd   0.5
neghalf dd   -0.5

section .text
display:
#   push   rbp
#   mov   rsp, rbp
   mov   rdi, GL_COLOR_BUFFER_BIT
   call   glClear         ; glClear(GL_COLOR_BUFFER_BIT)
   mov   rdi, GL_POLYGON
   call   glBegin         ; glBegin(GL_POLYGON)
   movss   xmm2, [zero]
   movss   xmm1, [zero]
   movss   xmm0, [one]
   call   glColor3f      ; glColor3f(1, 0, 0)
   movss   xmm2, [zero]
   movss   xmm1, [neghalf]
   movss   xmm0, [neghalf]
   call   glVertex3f      ; glVertex(-.5, -.5, 0)
   movss   xmm2, [zero]
   movss   xmm1, [one]
   movss   xmm0, [zero]
   call   glColor3f      ; glColor3f(0, 1, 0)
   movss   xmm2, [zero]
   movss   xmm1, [neghalf]
   movss   xmm0, [half]
   call   glVertex3f      ; glVertex(.5, -.5, 0)
   movss   xmm2, [one]
   movss   xmm1, [zero]
   movss   xmm0, [zero]
   call   glColor3f      ; glColor3f(0, 0, 1)
   movss   xmm2, [zero]
   movss   xmm1, [half]
   movss   xmm0, [zero]
   call   glVertex3f      ; glVertex(0, .5, 0)
   call   glEnd         ; glEnd()
   call   glFlush         ; glFlush()
#   pop   rbp
   ret

main:
   mov   rsi, rcx
   call   glutInit
   mov   rsi, 0
   call   glutInitDisplayMode
   mov   rsi, 100
   mov   rdi, 100
   call   glutInitWindowPosition
   mov   rdi, 300
   mov   rsi, 300
   call   glutInitWindowSize
   mov   rdi, title
   call   glutCreateWindow
   mov   rdi, display
   call   glutDisplayFunc
   call   glutMainLoop
   leave
   ret
Some later day i'l clean it up, optimise it some more, make it be friendly to the rest of the system, make it do tricks like a Sierpinski triangle or idk.
And ofc mention the original author when i find where i got it from, as out of the couple of opengl/glut examples this one made some sense to me.
Was fun, thx for the help, you pointed me in the right direction.