Solved: RoundDown incorrect in certain instances

bkornfuehrer · ‎10-21-2020

I found some very interesting RoundDown behavior and looking for some perspective here.

The initial number which sparked this was 17240.17. When entered into RoundDown(17240.17, 2) the result is 17240.16

This happens for .19, .26, and .35 as well. To add one final wrinkle this seems to happen only on whole numbers from 16384.17 and 20971.17

I appreciate any feedback on this. In the meantime I am using RoundDown(17240.17, 3) which seems to be working correctly

Thanks!

v-bofeng-msft · ‎10-22-2020

Hi @bkornfuehrer :

The point is Floating-point arithmetic is approximate, so it can sometimes give unexpected results with many documented examples. For example,you might expect the formula 55 / 100 * 100 to return exactly 55 and RoundUp((55 / 100 * 100) - 55,0) to return exactly zero. However, the latter formula returns 1.Because (55 / 100 * 100) - 55 returns 7.1054 x 10^–15, which is very small but not zero.

I think this link will help you a lot:

Number and Currency

Best Regards,

Bof

View solution in original post

v-bofeng-msft · ‎10-22-2020

Hi @bkornfuehrer :

The point is Floating-point arithmetic is approximate, so it can sometimes give unexpected results with many documented examples. For example,you might expect the formula 55 / 100 * 100 to return exactly 55 and RoundUp((55 / 100 * 100) - 55,0) to return exactly zero. However, the latter formula returns 1.Because (55 / 100 * 100) - 55 returns 7.1054 x 10^–15, which is very small but not zero.

I think this link will help you a lot:

Number and Currency

Best Regards,

Bof