Integer overflow. The result of an integer operation caused the most significant bit of the result to carry.


Short explanation:

If you use fixed precision datatypes (smallint, integer, bigint, decimal and numeric), it is possible that the result of calculation doesn't fit the datatype. Try casting the values in complex expressions as double precision and see whether the error goes away. If it works and you don't care about being too precise, you can leave it at that. Otherwise you need to check every operation and calculate the result.

Details:

Here's an example: if you multiply 9.12 with 8.11 (both numeric(18,2)) you would get 73.9632. If Firebird would store that into numeric(18,2) datatype, we would lose 0.0032. Doesn't look much, but when you have complex calculations, you can easily loose thousands (dollars or euros). Therefore, the result is stored in numeric(18,4).

Problems are rarely seen with such low precision as 2. Let's use some bigger precision. For example, numeric(18,6) times numeric(18,6) yields numeric(18,12) result, meaning that maximal value it can store is 9223372.036854775807. If (for example) you wish to keep only 6 digits of precision, you could use something like:

cast(value1 as numeric(18,3)) * cast(value2 as numeric(18,3))

which would yield numeric(18,6) result, but it is quite possible that you would get more accurate result by casting to double:

cast(cast(value1 as double precision) * cast(value2 as double precision) as numeric(18,6))


Also, if you have mixed multiplications and divisions it helps to change the order of operations, so that the overflow doesn't happen.

Furl  del.icio.us  co.mments  digg  YahooMyWeb 

Do you find this FAQ incorrect or incomplete? Please e-mail us what needs to be changed. To ensure quality, each change is checked by our editors (and often tested on live Firebird databases), before it enters the main FAQ database. If you desire so, the changes will be credited to your name. To learn more, visit our add content page.

If you are a commercial tool maker and your tool features a great way to handle the issue written about in this FAQ, please check out our advertisement page.



All contents are copyright © 2007-2008 FirebirdFAQ.org unless otherwise stated in the text.


Links   Firebird   News   FlameRobin   Tool reviews  
Add content   Advertise   About  
 
Newbies
SQL
Installation and setup
Backup and restore
Performance
Security
Connectivity and API
HOWTOs
Errors and error codes
Miscellaneous