[precision error] compute percentage out of two large int64


(Les Way) #1

Hi all,

I have 2 large int64 values, A and B. One of them, namely B, is close to math.MaxInt64.

I want to know what percentage is A compared to B. Namely I want to compute (A / B) * 100. This percentage must be a float64 value.

My fear is that result := float64(A) / float64(B) * 100 could potentially make a rounding error, when converting A and B to float64, then again when dividing them.

How legitimate is this fear ? What is your experience, how much of a precision loss should I be expecting ?


(Norbert Melzer) #2

You said float, you will have rounding errors. No way around that.

And if B is larger than A, then using integer division will always result in 0. This is a huge error rate and can be considered a very huge rounding error.

So, by doing float conversion early, you probably reduce the rounding error…


(Sean Killian) #3

I agree with NobbZ but want to add that you might be able to mitigate some of the rounding issues by using the math/big package’s big.NewRat or (*big.Rat).SetFrac64 and then calling (*big.Rat).Float64 which I suspect will be measurably slower than converting a and b to float64s up front but might get you more accurate results? Not sure though. When in doubt, test and measure it!