Author Topic: What type of rounding is used when an integer is divided?  (Read 11993 times)

Offline hhh3h

  • Jr. Member
  • *
  • Posts: 41
What type of rounding is used when an integer is divided?
« on: May 09, 2012, 06:34:19 PM »
I read that floating point division uses the "round half to even" method.

Quote from: wiki (Round Half to Even)
+23.5 becomes +24, +22.5 becomes +22, -22.5 becomes -22, and -23.5 becomes -24

I was wondering what method integer division uses?  Someone said it was truncated, which is equivalent to always rounding "down" (toward negative infinity).  Is that correct?

Thank you.

EDIT: If this matters, I suppose this question is directed towards the x86-64 platform.
« Last Edit: May 09, 2012, 06:38:52 PM by hhh3h »

Offline Frank Kotler

  • NASM Developer
  • Hero Member
  • *****
  • Posts: 2667
  • Country: us
Re: What type of rounding is used when an integer is divided?
« Reply #1 on: May 10, 2012, 02:16:02 AM »
Yeah, integer division truncates, but that's towards zero, not towards negative infinity. The remainder is in ah, dx, edx, or rdx depending on the size.

The FPU is configurable via the "control word". "Round to nearest even" is probably a common default (useful for statistics, supposedly), but I wouldn't count on it!

Best,
Frank


Offline hhh3h

  • Jr. Member
  • *
  • Posts: 41
Re: What type of rounding is used when an integer is divided?
« Reply #2 on: May 11, 2012, 02:15:02 AM »
Thanks, I appreciate your reply and the correction on truncate vs down / towards negative infinity

Why would I not want to count on the configurable control word?  Are you saying it might not be accurate for statistics purposes, or do you mean that it might not always be defaulted to that setting?

Offline Frank Kotler

  • NASM Developer
  • Hero Member
  • *****
  • Posts: 2667
  • Country: us
Re: What type of rounding is used when an integer is divided?
« Reply #3 on: May 11, 2012, 04:27:45 AM »
Merely that it might not always default to the same value. I was joking about the signal to noise ratio on a newsgroup ( news:alt.lang.asm ) being in danger of causing an underflow exception, and we fooled around with an example. Turned out it behaved differently in Dos vs Linux - apparently someone changed the exception handling. (I forget details) So you "might" discover rounding mode changed, too. If it's "mission critical", you might want to set the control word to a known value (and put it back).

But this has nothing to do with integer division, of course...

Best,
Frank


Offline hhh3h

  • Jr. Member
  • *
  • Posts: 41
Re: What type of rounding is used when an integer is divided?
« Reply #4 on: May 11, 2012, 04:30:05 AM »
Ahh ok good to know :)