Follow-up to Weird FP Ruby Item of the Day

Ok, ok, joke's on me. ;-) That will learn me to fire off blog posts too quickly. To clarify: Chad and I understand floating point arithmetic and its accuracy problems. What we weren't thinking clearly on is Float#to_s.

Specifically, what was freaking us out was not the loss of precision on an FP conversion, but the fact that, when asked its value, it was rounding up. To make the point more clearly, we should have included all the output in the original ruby session:

  > s = "40.87"
  > f = s.to_f
  > ft = f * 100
  > fti = ft.to_i

I would have liked for this to be true:

  > ft * 100

but instead, what was true was:

  > ft * 100

Our confusion stemmed from the fact that we weren't thinking about the fact that irb uses #to_s to display all its values. And, to demonstrate:

  > 4086.9999999999995.to_s

Which might be appropriate, but is still annoying. ;-) Especially given:

  > 4086.999999999995.to_s

(note: one less unit of precision)

which is what I was originally hoping to see.

To make it all weirder for us, "40.87" was the very first value we tried running through this little pain mill, and the only one with any resulting errors. C'est la vie, I guess.

Update: And, in JavaScript:

  > f
  > f * 100
  > (f * 100).toString()

Reported correctly. ;-)

Get In Touch