Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.databases.ms-access > #10355

Re: Weird Int Stuff

Show all headers | View raw

On 15/02/2015 03:22:01, Neil wrote:
> On 2/14/2015 9:11 PM, Neil wrote:
>> Can anyone explain to me why this is:
>>
>> ? int(760.025 * 100 + .5)
>>   76002
>>
>> ? int(76002.5 + .5)
>>   76003
>>
>> Same thing happens if you use fix. But clng gives 76003 in both cases.
> 
> 
> And, to add to the weirdness, if I create a function and pass it the
> 760.025 value as a variant, it works correctly:
> 
> Function Temp(var As Variant) As Variant
> 
> Temp = Int(var * 100 + 0.5)
> 
> End Function
> 
> ? temp(760.025)
> 76003
> 
> (But if I hard-code 760.025 into the function, instead of using the var
> parameter, it then returns 76002 instead of 76003.)
> 

Take a look at https://docs.python.org/2/tutorial/floatingpoint.html.

Basically, it is saying that all data is stored as a binary number, and there
are limitations to the accuracy of that number depending on the number of 0's
and 1's you can store. So (I am guessing), maybe after the multiplcation, the
equivalent of binary number is
76002.4999999999999999999999999999999999999999999999999, and in the second
case, 76002.500000000000000000000000000000000000000000000000000001

Phil

---
This email has been checked for viruses by Avast antivirus software.
http://www.avast.com

Back to comp.databases.ms-access | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Weird Int Stuff Neil <xxx@gmail.com> - 2015-02-14 21:11 -0600
  Re: Weird Int Stuff Neil <xxx@gmail.com> - 2015-02-14 21:22 -0600
    Re: Weird Int Stuff "Phil" <phil@stantonfamily.co.uk> - 2015-02-15 09:41 +0000
      Re: Weird Int Stuff Patrick Finucane <patrickfinucanetx@gmail.com> - 2015-02-15 10:41 -0800
        Re: Weird Int Stuff Neil <xxx@gmail.com> - 2015-02-15 20:45 -0600
          Re: Weird Int Stuff "Albert D. Kallal" <PleaseNOSpamkallal@msn.com> - 2015-02-15 22:04 -0700

csiph-web