Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.databases.ms-access > #10353 > unrolled thread

Weird Int Stuff

Back to article view | Back to comp.databases.ms-access

Contents

  Weird Int Stuff Neil <xxx@gmail.com> - 2015-02-14 21:11 -0600
    Re: Weird Int Stuff Neil <xxx@gmail.com> - 2015-02-14 21:22 -0600
      Re: Weird Int Stuff "Phil" <phil@stantonfamily.co.uk> - 2015-02-15 09:41 +0000
        Re: Weird Int Stuff Patrick Finucane <patrickfinucanetx@gmail.com> - 2015-02-15 10:41 -0800
          Re: Weird Int Stuff Neil <xxx@gmail.com> - 2015-02-15 20:45 -0600
            Re: Weird Int Stuff "Albert D. Kallal" <PleaseNOSpamkallal@msn.com> - 2015-02-15 22:04 -0700

#10353 — Weird Int Stuff

Can anyone explain to me why this is:

? int(760.025 * 100 + .5)
  76002

? int(76002.5 + .5)
  76003

Same thing happens if you use fix. But clng gives 76003 in both cases.

[toc] | [next] | [standalone]

#10354

On 2/14/2015 9:11 PM, Neil wrote:
> Can anyone explain to me why this is:
>
> ? int(760.025 * 100 + .5)
>   76002
>
> ? int(76002.5 + .5)
>   76003
>
> Same thing happens if you use fix. But clng gives 76003 in both cases.


And, to add to the weirdness, if I create a function and pass it the 
760.025 value as a variant, it works correctly:

Function Temp(var As Variant) As Variant

     Temp = Int(var * 100 + 0.5)

End Function

? temp(760.025)
  76003

(But if I hard-code 760.025 into the function, instead of using the var 
parameter, it then returns 76002 instead of 76003.)

[toc] | [prev] | [next] | [standalone]

#10355

On 15/02/2015 03:22:01, Neil wrote:
> On 2/14/2015 9:11 PM, Neil wrote:
>> Can anyone explain to me why this is:
>>
>> ? int(760.025 * 100 + .5)
>>   76002
>>
>> ? int(76002.5 + .5)
>>   76003
>>
>> Same thing happens if you use fix. But clng gives 76003 in both cases.
> 
> 
> And, to add to the weirdness, if I create a function and pass it the
> 760.025 value as a variant, it works correctly:
> 
> Function Temp(var As Variant) As Variant
> 
> Temp = Int(var * 100 + 0.5)
> 
> End Function
> 
> ? temp(760.025)
> 76003
> 
> (But if I hard-code 760.025 into the function, instead of using the var
> parameter, it then returns 76002 instead of 76003.)
> 

Take a look at https://docs.python.org/2/tutorial/floatingpoint.html.

Basically, it is saying that all data is stored as a binary number, and there
are limitations to the accuracy of that number depending on the number of 0's
and 1's you can store. So (I am guessing), maybe after the multiplcation, the
equivalent of binary number is
76002.4999999999999999999999999999999999999999999999999, and in the second
case, 76002.500000000000000000000000000000000000000000000000000001

Phil

---
This email has been checked for viruses by Avast antivirus software.
http://www.avast.com

[toc] | [prev] | [next] | [standalone]

#10356

On Sunday, February 15, 2015 at 4:41:52 AM UTC-5, Phil wrote:
> On 15/02/2015 03:22:01, Neil wrote:
> > On 2/14/2015 9:11 PM, Neil wrote:
> >> Can anyone explain to me why this is:
> >>
> >> ? int(760.025 * 100 + .5)
> >>   76002
> >>
> >> ? int(76002.5 + .5)
> >>   76003
> >>
> >> Same thing happens if you use fix. But clng gives 76003 in both cases.
> > 
> > 
> > And, to add to the weirdness, if I create a function and pass it the
> > 760.025 value as a variant, it works correctly:
> > 
> > Function Temp(var As Variant) As Variant
> > 
> > Temp = Int(var * 100 + 0.5)
> > 
> > End Function
> > 
> > ? temp(760.025)
> > 76003
> > 
> > (But if I hard-code 760.025 into the function, instead of using the var
> > parameter, it then returns 76002 instead of 76003.)
> > 
> 
> Take a look at https://docs.python.org/2/tutorial/floatingpoint.html.
> 
> Basically, it is saying that all data is stored as a binary number, and there
> are limitations to the accuracy of that number depending on the number of 0's
> and 1's you can store. So (I am guessing), maybe after the multiplcation, the
> equivalent of binary number is
> 76002.4999999999999999999999999999999999999999999999999, and in the second
> case, 76002.500000000000000000000000000000000000000000000000000001
> 
> Phil
> 
> ---
> This email has been checked for viruses by Avast antivirus software.
> http://www.avast.com

I think you are right Phil since if you remove the Int(), the results are as expected.

[toc] | [prev] | [next] | [standalone]

#10357

On 2/15/2015 12:41 PM, Patrick Finucane wrote:
> On Sunday, February 15, 2015 at 4:41:52 AM UTC-5, Phil wrote:
>> On 15/02/2015 03:22:01, Neil wrote:
>>> On 2/14/2015 9:11 PM, Neil wrote:
>>>> Can anyone explain to me why this is:
>>>>
>>>> ? int(760.025 * 100 + .5)
>>>>    76002
>>>>
>>>> ? int(76002.5 + .5)
>>>>    76003
>>>>
>>>> Same thing happens if you use fix. But clng gives 76003 in both cases.
>>>
>>>
>>> And, to add to the weirdness, if I create a function and pass it the
>>> 760.025 value as a variant, it works correctly:
>>>
>>> Function Temp(var As Variant) As Variant
>>>
>>> Temp = Int(var * 100 + 0.5)
>>>
>>> End Function
>>>
>>> ? temp(760.025)
>>> 76003
>>>
>>> (But if I hard-code 760.025 into the function, instead of using the var
>>> parameter, it then returns 76002 instead of 76003.)
>>>
>>
>> Take a look at https://docs.python.org/2/tutorial/floatingpoint.html.
>>
>> Basically, it is saying that all data is stored as a binary number, and there
>> are limitations to the accuracy of that number depending on the number of 0's
>> and 1's you can store. So (I am guessing), maybe after the multiplcation, the
>> equivalent of binary number is
>> 76002.4999999999999999999999999999999999999999999999999, and in the second
>> case, 76002.500000000000000000000000000000000000000000000000000001
>>
>> Phil
>>
>> ---
>> This email has been checked for viruses by Avast antivirus software.
>> http://www.avast.com
>
> I think you are right Phil since if you remove the Int(), the results are as expected.
>

The strange thing is, there's no way to rely on doing arithmetic in code 
(unless you store in variables).

In addition to the original that I posted:

? int(760.025 * 100 + .5)
76002

? int(76002.5 + .5)
76003

there's also this:

? int(cdbl(760.025*100)+.5)
76003

So wrapping the floating point multiplication results in CDbl causes the 
answer to be correct.

However, this is not always the case.

?int(0.0055 * 100 + 1.45))
1

?int(0.55 + 1.45))
2

However, when CDbl is applied in this case, it doesn't fix it, as it did 
in the other one:

?int(cdbl(.0055 * 100) + 1.45)
1

So this is wacky how it's so inconsistent. If it always lost 0.0000001 
(or whatever it loses) that would be one thing. But the inconsistency is 
bizarre.

The really interesting thing here is that if you store the floating 
point value in a variable, it works correctly. But only if the variable 
is a Variant. As noted above:

Function Temp(var As Variant) As Variant

     Temp = Int(var * 100 + 0.5)

End Function

? temp(760.025)
  76003

But if I change the var parameter to a Double:

Function Temp(var As Double) As Variant

     Temp = Int(var * 100 + 0.5)

End Function

? temp(760.025)
  76002

So it works correctly if the floating point number used in the 
multiplication is stored as a Variant. But it works incorrectly if it's 
stored as a Double.

So why would a Variant variable be superior to a Double variable in this 
case? You'd think that either way Access would be performing a floating 
point calculation, and you'd get the same results either way.

[toc] | [prev] | [next] | [standalone]

#10358

Neil"  wrote in message news:mbrlj0$mg6$1@dont-email.me...

>So it works correctly if the floating point number used in the 
>multiplication is stored as a Variant. But it works incorrectly if it's 
>stored as a Double.

You don't know what kind of data type you getting.

The additional problem using double (or cdbl()) is that is a floating point 
number. As pointed out by others, such numbers cannot be represented 
perfectly, but are ONLY an approximation.

One of the "first" classes in computing science taught the above. This means 
for business applications you have to use integer data type values, else you 
get rounding errors.

Also, what is the default data type of   760.025 ????? (turns out it is 
double)


We can force the data types of expressions by going back to the VERY old 
days of BASIC, and they still (amazingly) work.

Data Types             Suffix
Single-precision     !
Integer                     %
Double-precision     #
Long-integer             &
String                         $
Currency                    @

Debug.Print Int(CDbl(760.025 * 100) + 0.5)  --> 76003

The issue above is the "double". If we go:

Debug.Print Int(760.025! * 100 + 0.5)        --->  76003      (note the use 
of the ! to case to single data type)
Debug.Print Int(760.025 * 100 + 0.5)        ---->  76002

What the above shows, is 760.025 is seen as a double. And that has enough 
precision to produce correct results.

but try this:

   Debug.Print (760.025 * 100) - 76002.5     -->  -2.27373675443232E-12

Note how the number does have "junk" 12 digits over!!

The simple matter is you MUST use integer data types for ANY financial 
calculations. And there are several integer types.

The 3 we can use are packed decimal, long, and currency data types (all 3 
are integer data types).

Try this code - and it EVEN works this way in Excel:

Public Sub TestAdd()
   Dim MyNumber      As Single
   Dim I             As Integer
  For I = 1 To 10
      MyNumber = MyNumber + 1.01
      Debug.Print MyNumber
   Next I

End Sub

Here is the output of the above:
1.01
2.02
3.03
4.04
5.05
6.06
7.070001
8.080001
9.090001
10.1

You can see that after just 7 additions rounding is occurring!

Note how after JUST 7 simple little additions Access is now spitting out 
wrong numbers and has rounding errors!

New computer rule and lesson:

Computers do NOT store floating numbers to exact and thus such results are 
subject to rounding.

In "most" cases using double precision will push the rounding so far off to 
the right and you not notice!

However, for anything involving accounting, or "money", then use the 
currency data type . As noted, they are integers (scaled) and don't suffer 
from rounding errors.

A short "boot camp" and survival article of mine for Access developers and 
rounding can be found here:

Simple rounding errors can bite you in Access!
http://www.kallal.ca/Articles/rounding/Rounding.html

Regards,

Albert D. Kallal    (Access MVP)
Edmonton, Alberta Canada
PleaseNoSpam_kallal@msn.com 

[toc] | [prev] | [standalone]

Back to top | Article view | comp.databases.ms-access

csiph-web