double data type and output

[Howard Watson]

Pervasive SQL 2000i sp4, jdbc.v2 type4 driver

The output below is from a java servlet. The same query in the Pervasive SQL Data Manager does not result in any calculated results with 13 digits to the right of the decimal point. None of the output is formatted numbers just whatever is stored in the variable.

Any ideas where this is coming from. I'd like to understand or resolve this before I apply formatting.

Query: SELECT unitsAlloc,priceUnit,(priceUnit * unitsAlloc) AS dollarsUsed FROM table

priceUnit and unitsAlloc are Pervasive SQL double data types

String sUnitsAlloc = String.valueOf(rs.getDouble("unitsAlloc"));
String sPriceUnit = String.valueOf(rs.getDouble("priceUnit"));
String sDollarsUsed = String.valueOf(rs.getDouble("dollarsUsed"));

//also tried this to calculate dollarsUsed
double dUnitsAlloc = rs.getDouble("unitsAlloc");
double dPriceUnit = rs.getDouble("priceUnit");
double dDollarsUsed = dUnitsAlloc * dPriceUnit;
String sDollarsUsed = String.valueOf(dDollarsUsed);

//output
while(rs.next()) {
out.println("<td>" + sUnitsAlloc + "</td>");
out.println("<td>" + sPriceUnit + "</td>");
out.println("<td>" + sDollarsUsed + "</td>");
}

Some sample output:

Units Used Unit Price Dollars Used
1.0 150.0 150.0
1.0 150.0 150.0
1.0 150.0 150.0
1.0 150.0 150.0
1.0 150.0 150.0
1.0 150.0 150.0
13.0 62.83 816.79
22.0 62.83 1382.26
20.0 62.83 1256.6
22.0 62.83 1382.26
21.0 62.83 1319.43
21.0 62.83 1319.43
21.0 62.83 1319.43
10.0 84.07 840.6999999999999
20.0 84.07 1681.3999999999999
23.0 84.07 1933.61
20.0 84.07 1681.3999999999999
20.0 84.07 1681.3999999999999
22.0 84.07 1849.54
20.0 84.07 1681.3999999999999
22.0 84.07 1849.54
19.0 84.07 1597.33
0.0 2.84 0.0
0.0 2.84 0.0
30.25 2.84 85.91
116.0 2.84 329.44
0.0 2.84 0.0
105.0 2.84 298.2
105.0 2.84 298.2
77.0 2.84 218.67999999999998
113.0 2.84 320.91999999999996
1.0 150.0 150.0
1.0 150.0 150.0
1.0 150.0 150.0
18.0 58.71 1056.78
22.0 58.71 1291.6200000000001
20.0 58.71 1174.2
31.0 73.13 2267.0299999999997
31.0 73.13 2267.0299999999997
30.0 73.13 2193.8999999999996
14.0 35.02 490.28000000000003

[Jeanne Boyarsky]

Howard,
If not sure I understand what you are expecting as output. Are you saying that the Data Manager gives you zeros? Or is this about the precision of the doubles? Or something else entirely?

[Howard Watson]

Hi Jeanne,

Did a little research while writing this. Remember why I liked BCD.
I can let this go. Thanks.

I expected:
22.0 * 58.71 = 1291.62 and not 1291.6200000000001
113.0 * 2.84 = 320.92 and not 320.91999999999996
30.0 * 73.13 = 2193.90 and not 2193.8999999999996...

Maybe it is something I don't remember about floating point numeric processing. I don't see a pattern in the results that explains why
18.0 * 58.71 = 1056.78 and
22.0 * 58.71 = 1291.6200000000001

The Data Manager (a query tool) returns 22.0 * 58.71 = 1291.62, but maybe it is rounding internally.

[Jeanne Boyarsky]

Howard,
One of our sherriffs wrote an article on this last year.

[Adeel Ansari]

wiered. i haven't got into this kinda problem.
Jeanne i love you for this.

thanks
[ November 21, 2004: Message edited by: adeel ansari ]