F
fea4you
Guest
Hi, colleague,
I use Visual Basic (Visual Studio 2017) and encountered one problem with simple arithmetic calculation.
Function Cal_N60(ByVal Zmi, ByVal Ni) As Double
Dim Cri1 As Double = Calcu_Cr(Cri, Zmi)
Cal_N60 = Ce * Cb * Cs * Cd * Cri1 * Ni
End Function
Ce, Cb, Cs, and Cd are all declared as public double.
When Zmi passed as 40, Ni as 24, the Cri1 calculated is 0.9987220891752353...
Ce=1.3, Cb=1, Cs=1.0, and Cd=1.
VB calculate Ce * Cb * Cs * Cd * Cri1=1.3*1*1.3*1*0.9987220891752353=1.67094...
The correct number should be 1.68784...
I changed the order of variables and found VB calculated 1.3*0.9987220891752353 wrong.
I then set Cri1=0.998722. VB calculates 1.3*0.998722=1.2983386. CORRECT!
I then convert the called double to string, then string to value with 6 digits, it calculated wrong again.
Anyone encountered similar problem before? What's the problem of this? Is there a way to fix it?
Thank you very much,
Geo
Continue reading...
I use Visual Basic (Visual Studio 2017) and encountered one problem with simple arithmetic calculation.
Function Cal_N60(ByVal Zmi, ByVal Ni) As Double
Dim Cri1 As Double = Calcu_Cr(Cri, Zmi)
Cal_N60 = Ce * Cb * Cs * Cd * Cri1 * Ni
End Function
Ce, Cb, Cs, and Cd are all declared as public double.
When Zmi passed as 40, Ni as 24, the Cri1 calculated is 0.9987220891752353...
Ce=1.3, Cb=1, Cs=1.0, and Cd=1.
VB calculate Ce * Cb * Cs * Cd * Cri1=1.3*1*1.3*1*0.9987220891752353=1.67094...
The correct number should be 1.68784...
I changed the order of variables and found VB calculated 1.3*0.9987220891752353 wrong.
I then set Cri1=0.998722. VB calculates 1.3*0.998722=1.2983386. CORRECT!
I then convert the called double to string, then string to value with 6 digits, it calculated wrong again.
Anyone encountered similar problem before? What's the problem of this? Is there a way to fix it?Thank you very much,
Geo
Continue reading...