Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.graphics.apps.gnuplot > #2406

Re: Integer overflow when using binary and skip

Show all headers | View raw

a.daitche@gmail.com wrote:

> Hi,
> 
> i am plotting data from a large binary file using something like
> splot "file.bin" binary record=100000 skip=2000*100000*3*4 u 1:2:3 w d
> 
> The file contains the positions of 100000 particles at 10000 time-points.
> The above example should plot the particle positions at the 2000th
> time-point, but throws the following error message "Number of bytes to
> skip must be positive integer"
> 
> The reason is that 2000*100000*3*4 is larger then the largest 32bit
> integer and an overflow would occur. To prevent this gnuplot converts the
> number to a float, which, unfortunately, is not accepted as an argument
> for skip.
> 
> Ideally i would tell gnuplot to use 64bit integer, however this doesn't
> seem to be possible. Do you have any ideas for a workaround?

I would worry about a more fundamental problem.
Gnuplot is going to pass that integer as an argument to fseek(),
but fseek itself does not guarantee to handle anything larger than
a 32 bit integer:

fseek (3)
SYNOPSIS
       #include <stdio.h>
       int fseek(FILE *stream, long offset, int whence);

Now it's true that any platform supporting files larger than 2GB probably
also supports some variant of fseek that accepts a 64-bit offset, but given
the number of platforms on which gnuplot tries to compile and run I
wouldn't bet on there being a single universally compatible solution.

I suppose we could test at configuration time for the presence of
fseeko(), and use that instead if possible.  On the other hand I 
wouldn't be surprised if platforms with fseeko() are actually less common
than systems where the "long" argument to fseek is 64 bits already.

So a feature request to handle large (i.e. > 2GB) files sounds reasonable
to me, but I think it should include a preliminary analysis of other
places the problem might arise.  I see several other places in the gnuplot
code where fseek() is called. Is this really the only command that cares?
What about internal seeks in binary matrix files?  

	Ethan (gnuplot development team)

> 
> With best regards
> Anton

Back to comp.graphics.apps.gnuplot | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Integer overflow when using binary and skip a.daitche@gmail.com - 2014-04-07 03:43 -0700
  Re: Integer overflow when using binary and skip Karl <mail.kfr@gmx.net> - 2014-04-07 14:44 +0200
  Re: Integer overflow when using binary and skip Axel Berger <Axel.Berger@Gmx.De> - 2014-04-07 17:18 +0200
    Re: Integer overflow when using binary and skip Axel Berger <Axel.Berger@Gmx.De> - 2014-04-07 17:21 +0200
      Re: Integer overflow when using binary and skip a.daitche@gmail.com - 2014-04-09 03:48 -0700
        Re: Integer overflow when using binary and skip a.daitche@gmail.com - 2014-04-09 03:53 -0700
          Re: Integer overflow when using binary and skip Axel Berger <Axel.Berger@Gmx.De> - 2014-04-09 20:27 +0200
  Re: Integer overflow when using binary and skip Ethan A Merritt <merritt@u.washington.edu> - 2014-04-09 16:37 -0700
  Re: Integer overflow when using binary and skip Ethan A Merritt <sfeam@users.sourceforge.net> - 2014-04-14 12:49 -0700

csiph-web