Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.graphics.apps.gnuplot > #2396 > unrolled thread

Integer overflow when using binary and skip

Back to article view | Back to comp.graphics.apps.gnuplot

Contents

  Integer overflow when using binary and skip a.daitche@gmail.com - 2014-04-07 03:43 -0700
    Re: Integer overflow when using binary and skip Karl <mail.kfr@gmx.net> - 2014-04-07 14:44 +0200
    Re: Integer overflow when using binary and skip Axel Berger <Axel.Berger@Gmx.De> - 2014-04-07 17:18 +0200
      Re: Integer overflow when using binary and skip Axel Berger <Axel.Berger@Gmx.De> - 2014-04-07 17:21 +0200
        Re: Integer overflow when using binary and skip a.daitche@gmail.com - 2014-04-09 03:48 -0700
          Re: Integer overflow when using binary and skip a.daitche@gmail.com - 2014-04-09 03:53 -0700
            Re: Integer overflow when using binary and skip Axel Berger <Axel.Berger@Gmx.De> - 2014-04-09 20:27 +0200
    Re: Integer overflow when using binary and skip Ethan A Merritt <merritt@u.washington.edu> - 2014-04-09 16:37 -0700
    Re: Integer overflow when using binary and skip Ethan A Merritt <sfeam@users.sourceforge.net> - 2014-04-14 12:49 -0700

#2396 — Integer overflow when using binary and skip

Hi,

i am plotting data from a large binary file using something like
splot "file.bin" binary record=100000 skip=2000*100000*3*4 u 1:2:3 w d

The file contains the positions of 100000 particles at 10000 time-points. The above example should plot the particle positions at the 2000th time-point, but throws the following error message
"Number of bytes to skip must be positive integer"

The reason is that 2000*100000*3*4 is larger then the largest 32bit integer and an overflow would occur. To prevent this gnuplot converts the number to a float, which, unfortunately, is not accepted as an argument for skip.

Ideally i would tell gnuplot to use 64bit integer, however this doesn't seem to be possible. Do you have any ideas for a workaround? 

With best regards
Anton

[toc] | [next] | [standalone]

#2397

Am 07.04.2014 12:43, schrieb a.daitche@gmail.com:
> Hi,
> 
> i am plotting data from a large binary file using something like
> splot "file.bin" binary record=100000 skip=2000*100000*3*4 u 1:2:3 w d
> 
> The file contains the positions of 100000 particles at 10000 time-points. The above example should plot the particle positions at the 2000th time-point, but throws the following error message
> "Number of bytes to skip must be positive integer"
> 
> The reason is that 2000*100000*3*4 is larger then the largest 32bit integer and an overflow would occur. To prevent this gnuplot converts the number to a float, which, unfortunately, is not accepted as an argument for skip.
> 
> Ideally i would tell gnuplot to use 64bit integer, however this doesn't seem to be possible. Do you have any ideas for a workaround? 
> 
> With best regards
> Anton
> 

Halve the skip length, and invalidate every other datapoint by some
other means?

	flag=0
	plot ... using 1:2:((flag=!flag)?$3:NaN)


 K

[toc] | [prev] | [next] | [standalone]

#2398

a.daitche@gmail.com wrote:
> Do you have any ideas for a workaround?

100 000 * 10 000 * 4 fits, so make it three skips.

Axel

[toc] | [prev] | [next] | [standalone]

#2399

Axel Berger wrote:
> 100 000 * 10 000 * 4 fits, so make it three skips.

Forgot about the integer being signed, so only multiply by 2 and make it
six skips.

Axel

[toc] | [prev] | [next] | [standalone]

#2400

Am Montag, 7. April 2014 17:21:22 UTC+2 schrieb Axel Berger:
> Axel Berger wrote:
> 
> > 100 000 * 10 000 * 4 fits, so make it three skips.
> 
> 
> 
> Forgot about the integer being signed, so only multiply by 2 and make it
> 
> six skips.

I don't quite understand what you mean by "make three skips". Just putting the skips after each other 
splot "file.bin" binary record=100000 skip=1000*100000*3*4:1000*100000*3*4 u 1:2:3 w d
gives me the error                                                                            "More parameters specified than data records specified"

Anton

[toc] | [prev] | [next] | [standalone]

#2401

Am Mittwoch, 9. April 2014 12:48:35 UTC+2 schrieb a.da...@gmail.com:
> Am Montag, 7. April 2014 17:21:22 UTC+2 schrieb Axel Berger:
> 
> > Axel Berger wrote:
> 
> > 
> 
> > > 100 000 * 10 000 * 4 fits, so make it three skips.
> 
> > 
> 
> > 
> 
> > 
> 
> > Forgot about the integer being signed, so only multiply by 2 and make it
> 
> > 
> 
> > six skips.
> 
> 
> 
> I don't quite understand what you mean by "make three skips". Just putting the skips after each other 
> 
> splot "file.bin" binary record=100000 skip=1000*100000*3*4:1000*100000*3*4 u 1:2:3 w d
> 
> gives me the error                                                                            "More parameters specified than data records specified"
> 

Inserted the wrong command. I ment 
splot "idt_r.dat" binary record=100000 skip=1000*100000*3*4 skip=1000*100000*3*4 u 1:2:3 w d
with error 
"Duplicated or contradicting arguments in datafile options"

Anton

[toc] | [prev] | [next] | [standalone]

#2405

a.daitche@gmail.com wrote:
> "Duplicated or contradicting arguments in datafile options"

Sorry, in that case it was a stupid idea - hadn't actually tried it out.

Axel

[toc] | [prev] | [next] | [standalone]

#2406

a.daitche@gmail.com wrote:

> Hi,
> 
> i am plotting data from a large binary file using something like
> splot "file.bin" binary record=100000 skip=2000*100000*3*4 u 1:2:3 w d
> 
> The file contains the positions of 100000 particles at 10000 time-points.
> The above example should plot the particle positions at the 2000th
> time-point, but throws the following error message "Number of bytes to
> skip must be positive integer"
> 
> The reason is that 2000*100000*3*4 is larger then the largest 32bit
> integer and an overflow would occur. To prevent this gnuplot converts the
> number to a float, which, unfortunately, is not accepted as an argument
> for skip.
> 
> Ideally i would tell gnuplot to use 64bit integer, however this doesn't
> seem to be possible. Do you have any ideas for a workaround?

I would worry about a more fundamental problem.
Gnuplot is going to pass that integer as an argument to fseek(),
but fseek itself does not guarantee to handle anything larger than
a 32 bit integer:

fseek (3)
SYNOPSIS
       #include <stdio.h>
       int fseek(FILE *stream, long offset, int whence);

Now it's true that any platform supporting files larger than 2GB probably
also supports some variant of fseek that accepts a 64-bit offset, but given
the number of platforms on which gnuplot tries to compile and run I
wouldn't bet on there being a single universally compatible solution.

I suppose we could test at configuration time for the presence of
fseeko(), and use that instead if possible.  On the other hand I 
wouldn't be surprised if platforms with fseeko() are actually less common
than systems where the "long" argument to fseek is 64 bits already.

So a feature request to handle large (i.e. > 2GB) files sounds reasonable
to me, but I think it should include a preliminary analysis of other
places the problem might arise.  I see several other places in the gnuplot
code where fseek() is called. Is this really the only command that cares?
What about internal seeks in binary matrix files?  

	Ethan (gnuplot development team)

> 
> With best regards
> Anton

[toc] | [prev] | [next] | [standalone]

#2409

a.daitche@gmail.com wrote:

> Hi,
> 
> i am plotting data from a large binary file using something like
> splot "file.bin" binary record=100000 skip=2000*100000*3*4 u 1:2:3 w d
> 
> The file contains the positions of 100000 particles at 10000 time-points.
> The above example should plot the particle positions at the 2000th
> time-point, but throws the following error message "Number of bytes to
> skip must be positive integer"
> 
> The reason is that 2000*100000*3*4 is larger then the largest 32bit
> integer and an overflow would occur. To prevent this gnuplot converts the
> number to a float, which, unfortunately, is not accepted as an argument
> for skip.
> 
> Ideally i would tell gnuplot to use 64bit integer, however this doesn't
> seem to be possible. Do you have any ideas for a workaround?
> 
> With best regards
> Anton

Large file support has been added to the development branch and will be
in the release candidate for version 5, which we hope will be ready in
the next month or so.

A subset of this change has been back-ported to version 4.6 so that
the specific problem reported here is handled on 64-bit systems
(but not 32bit systems).  Unfortunately this fix just missed the bugfix
release 4.6.5, so it will not be in an official release version for
another 6 months or so.

As always, both the current version (4.6) and the development version
(5.0) can be built from source code available on SourceForge.

	Ethan

[toc] | [prev] | [standalone]

Back to top | Article view | comp.graphics.apps.gnuplot

csiph-web