[precision error] compute percentage out of two large int64

(Les Way) #1

Hi all,

I have 2 large int64 values, A and B. One of them, namely B, is close to math.MaxInt64.

I want to know what percentage is A compared to B. Namely I want to compute (A / B) * 100. This percentage must be a float64 value.

My fear is that result := float64(A) / float64(B) * 100 could potentially make a rounding error, when converting A and B to float64, then again when dividing them.

How legitimate is this fear ? What is your experience, how much of a precision loss should I be expecting ?

(Norbert Melzer) #2

You said float, you will have rounding errors. No way around that.

And if B is larger than A, then using integer division will always result in 0. This is a huge error rate and can be considered a very huge rounding error.

So, by doing float conversion early, you probably reduce the rounding error…

(Sean Killian) #3

I agree with NobbZ but want to add that you might be able to mitigate some of the rounding issues by using the math/big package’s big.NewRat or (*big.Rat).SetFrac64 and then calling (*big.Rat).Float64 which I suspect will be measurably slower than converting a and b to float64s up front but might get you more accurate results? Not sure though. When in doubt, test and measure it!

(Christophe Meessen) #4

It depends if you accept some precision loss or if you want the exact result with as many decimal digits as possible.

If you accept loss in precision, you can pre-divide A and B by 4096. This correspond to 12 bits which correspond to the sign and exponent bits of the float64 number. This division ensure that the rounding is the same on A and B. The division by 4096 is canceled out with A/B.

You wont get the exact result, but the precision on the final result may be good enough for your need.