Groups > comp.graphics.apps.gnuplot > #3238

Re: incorrect arithmetik?

Newsgroups	comp.graphics.apps.gnuplot
Date	2016-02-17 23:30 -0800
References	<n72l7f$865$1@newsserver.rrzn.uni-hannover.de> <n72ugm$5ls$1@solani.org> <n73e83$oa9$1@dont-email.me>
Message-ID	<ceb65942-6451-4c90-94c8-6f1e056851b9@googlegroups.com> (permalink)
Subject	Re: incorrect arithmetik?
From	Gene Wagenbreth <gwagenbreth@gmail.com>

Show all headers | View raw

On Tuesday, January 12, 2016 at 9:50:31 AM UTC-8, Jörg Buchholz wrote:
> On 12.01.2016 14:19, Karl Ratzsch wrote:
> > Am 12.01.2016 um 11:40 schrieb Jörg Buchholz:
> >> I can not found the solution to the following problem:
> >>
> >> b=7.2-6.9
> >> c=8.2-7.9
> >>
> >> show variables
> >
> >> b = 0.3
> >> c = 0.299999999999999
> >>
> >> Why is c not 0.3?
> >
> > Floating point arithmetics. Most decimal fractions cannot be represented
> > exactly by a binary float variable.
> >
> > 8.2 is 1000.00110011....
> > 7.2 is 111.11100110011....
> >
> > If the two errors add up in one direction, the result of your
> > subtraction will be closer to a value that is in fact off by 1 in the
> > least significant digit.
> 
> Thanks for the explanation, I think I understand it.
> 
> Jörg

You can add some arithmetic to do rounding to fix this.

Back to comp.graphics.apps.gnuplot | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

incorrect arithmetik? Jörg Buchholz <bookwood4new@freenet.de> - 2016-01-12 11:40 +0100
  Re: incorrect arithmetik? Karl Ratzsch <mail.kfr@gmx.net> - 2016-01-12 14:19 +0100
    Re: incorrect arithmetik? Jörg Buchholz <bookwood4news@freenet.de> - 2016-01-12 18:50 +0100
      Re: incorrect arithmetik? Gene Wagenbreth <gwagenbreth@gmail.com> - 2016-02-17 23:30 -0800
  Re: incorrect arithmetik? Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2016-01-12 17:06 +0100

csiph-web