Forum Discussion

msaunders's avatar
msaunders
Contributor
11 years ago
Solved

Check point failing with matching values

Hi all,



I have a checkpoint that is checking that the value is properly stored in an infragistics grid. it seems to fail at certain times through the test. I am stepping the value up by .1 at a time and then checking from 0.1 to 10.  below are the first 3 fail points from 0-1.















Actual value

0.3

Expected value

0.3















Actual value

0.8

Expected value

0.8















Actual value

0.9

Expected value

0.9





I think it is a floating point math failure as when i change it to Math.round( i )

the failure points change, as you can see its rounding right but not what i wanted. its not consistent either, why does 1.5 work and 3.5 but not .5 and 2.5?















Actual value

0.5

Expected value

1















Actual value

2.5

Expected value

3















Actual value

4.5

Expected value

5





So next i tried toPrecision  that was working until i got to 1 but then a precision of 1 ends up not matching. does anyone have a good idea of how to get the values to match?  I notice when i use toPrecision it changes it to a string also. I am pretty new to JS so i am not sure what is out there to help here. thanks in advance!