What's an ideal assembly language calling convention?

[Dr1v3n]

Assume I am not trying to make my code work with any other code or the output of any compiler...

What would be some options I should consider for a calling convention internal to my own code?

I've learned cdecl, fastcall, stdcall, etc... But these are all common conventions output by compilers. I want to learn a way I may do this as just an assembly programmer.

First, is there even one convention I should follow? Why or why not? I'm thinking for example, if I write a procedure which is to take 2 strings and a length, I could simply write it to expect the 2 string pointers to be in rsi, rdi, and the length in rcx for example... But maybe if I have a proc that takes 2 numbers in and does some math, then returns a result, that proc could expect the numbers to be in rcx and rdx, and then the return in rax and so on and so forth...

Generally, I think I would prefer registers so as to reduce the amount of memory reads/writes and stack manipulation, since most general purpose registers are assumed to be clobbered anyway...

Does anyone have any guidance on this? Is there a good blog, forum post, article, etc...?

Thanks

[vitsoft]

It depends on the programming style: If you are used to hold most of temporary information in GPR between calls of procedures, it is convenient when those procedures do not clobber registers. If you keep temporary information in memory-variables, you don't have to care if the call of registry-convention procedure destroys GPR, but prior to CALL you need to load the input registers from memory anyway, so the gain of performance is lower.

Choice of calling convention is a trade-off between performance and programmer's convenience. It also depends on the complexity of procedures and the number of input arguments. With register-calling convention we may run out of available input registers soon, especially in 32bit mode.
StdCall seems more universal and convenient, see https://euroassembler.eu/easource/#CallingConvention, and it is compatible with 32bit Windows API.

Unlike C-functions we don't have to restrict the procedure output to just one value in assembly language, the result(s) can be returned in many registers and|or flags.

First, is there even one convention I should follow? Why or why not? I'm thinking for example, if I write a procedure which is to take 2 strings and a length, I could simply write it to expect the 2 string pointers to be in rsi, rdi, and the length in rcx for example...

Yes, such procedure could be this simple:

Compare PROC ; Compare strings at RSI, RDI with the length in RCX. Return ZF when equal. Clobbers RSI,RDI,RCX,Rflags.
     REP CMPS
     RET
    ENDPROC

However, you will need to load pointers from memory to RSI,RDI,RCX before each CALL Compare (of course, no sane programmer would create a procedure from just simple REP CMPS).
I prefer standard-calling convention even in 64bit mode, where arguments are pushed on stack before CALL and in the end they are discarded by RET IMM. Inside the procedure they are accessible relative to RSP or RBP, see StdCall Compare as an example.

[Elawig57]

I am new here and I am also want to know What is a calling convention assembly language? MyAARPMedicare

[fredericopissarra]

I am new here and I am also want to know What is a calling convention assembly language?

A calling convention is a set of rules on how to call a function: Which registers are used, how the stack is used, how data is returned, which registers cannot be changed when a function returns,... High level programming languages, when translating your code to machine language, use some "convention" to do this things, so how the registers, stack etc are used in a predictable way.

It is useful to adopt one convention (any) if you want to remember easily how your functions should be called. For example: For 32 bits programs writen in C there is a convenient convention called 'cdecl', where all arguments to a function are pushed to the stack, from right to left, and the return value is returned in a register (EAX no Intel processors -- or EDX:EAX pair if more then 32 bits). This way to acess the data passed to the function all you have to do is read them from the stack. In 64 bits, usually 'cdecl' uses registers to pass the first 4 or 6 arguments (Microsoft or SysV systems, respectively) - there is, also, rules to use floating point...

In assembly you can use your own convention, unless interfacing with a C function (for example). Another example. Lets say I have a C function to test if an int is odd:

Code: [Select]

_Bool isOdd( int x ) { return (x & 1) != 0; }The compiler probably will create something like this (32 bits code):

Code: [Select]

; Entry: [ESP+4] = x
; Output: EAX = boolean
_isOdd:
  mov eax,dword [esp+4]
  and eax,1
  retSince the function is 'called' the ESP points to memory with the return address and the pushed value is in ESP+4. And the return value is in EAX. But notice this is different for 64 bits, on Windows, for example:

Code: [Select]

; Entry: ECX = x
; Output: EAX
isOdd:
  mov eax,ecx
  and eax,1
  retLinux and SysV systems, in x86-64 mode, EDI is used, instead of ECX.

In your code, in asm, you could use a convention where boolean values could be returned in ZF (zero flag) instead on EAX. The ROM-BIOS, in real mode, uses something like this, returning an error condition in CF (carry flag).

As it was said before, in assembly, the convention used should be "convenient" (pun intended) accordingly to your needs. There are no strict rules there, you can make your own at some level.

But if you intend to interface your code with a high level language created code or with syscalls or a library function, you must obey a "standard" calling convention. Microsoft has its own, Linux and Unixes has their own, microcontrollers have their own. They are "standard" for the processor and the environment (and the compiler in use)...

Notice I cited just ONE calling convention here: cdecl. There are others: fastcall, stdcall, pascal, ... GCC for instance hava an attribute called regparm which allow the use of 3 registers to pass arguments, even in 32 bits functions...

Here's a quick, and not complete, article on https://en.wikipedia.org/wiki/X86_calling_conventions, in Wikipedia, for reference.