H
hfarner
Guest
OK, in a datatable i have column A and B as double.
I create column C as an expression A*B type Double
I create column D as an expression Convert(A*B*100,System.Int64)/100
So, if A = 3 and B=3.331 then C=9.993 and D=9.99
However, if A=3 and B=3.333 then C= 9.999 but D=10
This seems like incorrect behavior. I tried a lot of things, but the second expression always comes out wrong. every other number seems to work correctly... what am I missing.... did i wake up in bizarro land? I need a way to get a result to 2 decimal places. the expression should be working.....
So, I took it further. If you just do an expression Convert(A*B*100,System.Int64) the return for the first example, 3*3.331 is 999 but the result for 3*3.333 is 100 not 999
what is this black magic that has befallen me..... all i wanted to do was round to 2 decimal places and i ended up wandering in the .net desert.
Continue reading...
I create column C as an expression A*B type Double
I create column D as an expression Convert(A*B*100,System.Int64)/100
So, if A = 3 and B=3.331 then C=9.993 and D=9.99
However, if A=3 and B=3.333 then C= 9.999 but D=10
This seems like incorrect behavior. I tried a lot of things, but the second expression always comes out wrong. every other number seems to work correctly... what am I missing.... did i wake up in bizarro land? I need a way to get a result to 2 decimal places. the expression should be working.....
So, I took it further. If you just do an expression Convert(A*B*100,System.Int64) the return for the first example, 3*3.331 is 999 but the result for 3*3.333 is 100 not 999
what is this black magic that has befallen me..... all i wanted to do was round to 2 decimal places and i ended up wandering in the .net desert.
Continue reading...