41
Programming with NASM / Re: Problem with calculating root
« Last post by fredericopissarra on January 30, 2025, 07:44:59 PM »A note of caution... If you intend to calculate the square root of an unsigned long long int, the double precision has only 53 bits of precision (while unsigned long long int has 64)... So the first example is insufficient: You need a bigger floating point precision that is not available to SSE... You must use fp87 instead:
Code: [Select]
; Input: RDI = x
; Output RAX = sqrt(x)
squareroot:
fild qword [rsp-8],rdi ; using red zone here.
fsqrt
fistp qword [rsp-8]
mov rax,[rsp-8]
ret