programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
• Campbell Ritchie
• Jeanne Boyarsky
• Ron McLeod
• Paul Clapham
• Liutauras Vilda
Sheriffs:
• paul wheaton
• Rob Spoor
• Devaka Cooray
Saloon Keepers:
• Stephan van Hulst
• Tim Holloway
• Carey Brown
• Frits Walraven
• Tim Moores
Bartenders:
• Mikalai Zaikin

# Strange rounding

Ranch Hand
Posts: 142
• Number of slices to send:
Optional 'thank-you' note:
Hi,
We have a legacy Math utility class containing the following methods:

I realise it should be using BigDecimal everywhere but for the moment I can't change the code. What we are currently seeing is that in production on Websphere, the call

is returning 100.36999999, rather than 100.37. This does not happen in windows or in Tomcat. I have created a simple script and just called that exact line above (outside the application) and it works fine!

Following through the code, the result of the primitive subtraction is 100.36999999999989, which is seen in windows as well. When this passed to the roundDouble method, the factor is 10^8 = 100000000. This is multiplied by the input so its 100.36999999999989 * 100000000 = 10036999999. This is then rounded to the nearest int which is 10037000000. Finally we divide by the factor so its 10037000000 / 100000000 = 100.37.

I am getting 100.36999999 as the final result...so either the Math.round is truncating 10036999999, rather than rounding it or final divide 10037000000 / 100000000 is going wrong and returning 100.36999999.

Can anyone throw any ideas out why there would be a difference, especially between a running application calling a method and a simple script just calling the same method in the same jar on the same environment but outside the application, or any ideas on what could be going wrong?

Thanks

[ UD: added linebreak to preserve layout ]

Rancher
Posts: 43081
77
• Number of slices to send:
Optional 'thank-you' note:
Not sure what's going on, but you may want to look into using java.lang.StrictMath instead of java.lang.Math. Its API is identical, but it produces identical results across platforms (which Math is not required to do).

Tom Johnson
Ranch Hand
Posts: 142
• Number of slices to send:
Optional 'thank-you' note:
Thanks for the pointer....although I see that Math.pow simply delegates to StrictMath.pow while Math.round and StrictMath.round are the implemented exactly the same:

where floor is Math class simply delegates to StrictMath.floor.

So shouldn't calls to Math/StrictMath pow and round always return the same result, cross platform as well?

Ulf Dittmer
Rancher
Posts: 43081
77
• Number of slices to send:
Optional 'thank-you' note:
I've never checked what StrictMath does exactly. From what you say it sounds as if both should do the same thing in this case. Which means that any platform differences must originate elsewhere in the calculation. I'd do a step-by-step comparison of the intermediate values on both platforms to determine where that happens.

Tom Johnson
Ranch Hand
Posts: 142
• Number of slices to send:
Optional 'thank-you' note:
Yep I tried that....the code that seems to be going wrong is in a utility jar that we use, which I cannot change. So to do what you suggest, I copied the methods from the utility jar into my class and called them instead....should be the same thing. Unfortunately not! I couldn't get the error to happen calling the copied code, even though its identical! I can only think the utility jar is compiled "differently" than my copied version....or does this even even make sense?

Marshal
Posts: 79508
379
• Number of slices to send:
Optional 'thank-you' note:
If I remember correctly, the simple Math class methods (eg max, min, abs) use their own code,something like thisand the "complicated" methods simply call StrictMath, like thisAt least I think most of them do that.

 Don't get me started about those stupid light bulbs.