Follow-up to Weird FP Ruby Item of the Day

Ok, ok, joke's on me. ;-) That will learn me to fire off blog posts too quickly. To clarify: Chad and I understand floating point arithmetic and its accuracy problems. What we weren't thinking clearly on is Float#to_s.

Specifically, what was freaking us out was not the loss of precision on an FP conversion, but the fact that, when asked its value, it was rounding up. To make the point more clearly, we should have included all the output in the original ruby session:

  > s = "40.87"
  "40.87" 
  > f = s.to_f
  40.87
  > ft = f * 100
  4087.0
  > fti = ft.to_i
  4086

I would have liked for this to be true:

  > ft * 100
  4086.9

but instead, what was true was:

  > ft * 100
  4087.0

Our confusion stemmed from the fact that we weren't thinking about the fact that irb uses #to_s to display all its values. And, to demonstrate:

  > 4086.9999999999995.to_s
  4087.0

Which might be appropriate, but is still annoying. ;-) Especially given:

  > 4086.999999999995.to_s
  "4086.99999999999"

(note: one less unit of precision)

which is what I was originally hoping to see.

To make it all weirder for us, "40.87" was the very first value we tried running through this little pain mill, and the only one with any resulting errors. C'est la vie, I guess.

Update: And, in JavaScript:

  > f
  40.87
  > f * 100
  4086.9999999999995
  > (f * 100).toString()
  4086.9999999999995

Reported correctly. ;-)

Get In Touch