Author Topic: Best way to prevent integer overflow?  (Read 14111 times)

Offline hhh3h

  • Jr. Member
  • *
  • Posts: 41
Best way to prevent integer overflow?
« on: May 21, 2012, 09:57:24 PM »
I am interested in the best way to deal with integer overflows.

I read that the carry flag (JC) is set when an integer overflow occurs.  Is it better "trap" errors by allowing the overflow to occur and then checking that flag, or is it better to prevent the error from occurring in the first place by analyzing the operands prior to doing the operation?

Or perhaps there is a third option that I have not considered.  Please let me know strategy what you prefer, keeping in mind both safety and performance.

Thank you

Offline Frank Kotler

  • NASM Developer
  • Hero Member
  • *****
  • Posts: 2667
  • Country: us
Re: Best way to prevent integer overflow?
« Reply #1 on: May 21, 2012, 10:34:12 PM »
I'm not sure how you'd analyse the operands to determine if an overflow would occur, except by doing the operation and checking the flags. The carry flag would indicate an unsigned overflow, the overflow flag would indicate signed overflow. (the CPU doesn't know if you intend to treat integers as signed or unsigned - it's up to you to use appropriate instructions)

Strictly speaking, it isn't an error for an overflow to occur, since we're implicitly doing "modulo arithmetic", but it probably won't provide the result you expect. The question is, what do you propose to do about it? You can abort the calculation entirely, or "saturate" the result. The "best way" probably depends on what you intend to do... or intend to prevent...

Best,
Frank


Offline hhh3h

  • Jr. Member
  • *
  • Posts: 41
Re: Best way to prevent integer overflow?
« Reply #2 on: May 22, 2012, 07:46:09 PM »
Thank you Frank.  That is great info for a newbie like me.

What I meant about analyzing the operands is something like what they are talking about in this StackOverflow question: http://stackoverflow.com/questions/3944505/detecting-signed-overflow-in-c-c

I think I like your way of checking the flag(s) better.

Offline Keith Kanios

  • Full Member
  • **
  • Posts: 383
  • Country: us
    • Personal Homepage
Re: Best way to prevent integer overflow?
« Reply #3 on: May 22, 2012, 11:40:10 PM »
I think I like your way of checking the flag(s) better.

You are comparing a CPU architecture to a programming language.

x86 implements carry and overflow flags in a specific manner, with specific instructions to further test/utilize them.

C, to be portable, has abstracted such specifics away into what is essentially a common mathematical model.

In short, you are comparing "what" (C) with "how" (x86).