Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #57996 > unrolled thread

fread a file of unknown size

Back to article view | Back to comp.lang.c

Contents

  fread a file of unknown size ruben safir <ruben@mrbrklyn.com> - 2015-03-08 14:47 -0400
    Re: fread a file of unknown size Richard Heathfield <rjh@cpax.org.uk> - 2015-03-08 18:59 +0000
      Re: fread a file of unknown size James Kuyper <jameskuyper@verizon.net> - 2015-03-08 18:21 -0400
        Re: fread a file of unknown size Keith Thompson <kst-u@mib.org> - 2015-03-08 18:43 -0700
      Re: fread a file of unknown size fmassei@gmail.com - 2015-03-08 15:41 -0700
        Re: fread a file of unknown size raltbos@xs4all.nl (Richard Bos) - 2015-03-10 16:04 +0000
      Re: fread a file of unknown size Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2015-03-08 18:44 -0400
    Re: fread a file of unknown size jacob navia <jacob@spamsink.net> - 2015-03-08 20:32 +0100
    Re: fread a file of unknown size Keith Thompson <kst-u@mib.org> - 2015-03-08 12:48 -0700
    Re: fread a file of unknown size glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-03-08 23:39 +0000
      Re: fread a file of unknown size James Kuyper <jameskuyper@verizon.net> - 2015-03-08 20:40 -0400
        Re: fread a file of unknown size glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-03-09 01:24 +0000
          Re: fread a file of unknown size Keith Thompson <kst-u@mib.org> - 2015-03-09 07:53 -0700
            Re: fread a file of unknown size glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-03-09 17:36 +0000
              Re: fread a file of unknown size Keith Thompson <kst-u@mib.org> - 2015-03-09 10:44 -0700
                Re: fread a file of unknown size glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-03-09 20:07 +0000

#57996 — fread a file of unknown size

if you use fread to read a file of unknown size with a set buffer size
of say 500 bytes, when you get to the end of the file you are likely to
read a stub of 500 bytes of info and the eof marker.

And fread will not red that block and set and oef that can be read with
feof.  That is how I understand thinga.  So we can either use fseek to
discover the file size and fread accordingly or use fstat?

is this the proper approach?

Ruben

[toc] | [next] | [standalone]

#57997

On 08/03/15 18:47, ruben safir wrote:
> if you use fread to read a file of unknown size with a set buffer size
> of say 500 bytes, when you get to the end of the file you are likely to
> read a stub of 500 bytes of info and the eof marker.
>
> And fread will not red that block

fread ought to return the number of bytes read. Instead, it returns the 
number of blocks. I expect there was a good reason once.

Because of this, if I need to know how many bytes I've just read (and 
who doesn't?), I generally specify a blocksize of 1 and a count of (in 
your case) 500, rather than the other way around. That tells me exactly 
how many bytes were read on the last call.

Note that eof is not a marker. It is a state that the file position 
indicator can have. If the file position indicator is in the end-of-file 
state, feof() will yield true (non-0) for that stream.


> and set and oef that can be read with
> feof.

No, you can't read anything with feof, which only yields information 
about whether the stream is in the end-of-file state.


> That is how I understand thinga.  So we can either use fseek to
> discover the file size and fread accordingly or use fstat?
>
> is this the proper approach?

If the only reason you're reading the file is to find out how big it is, 
don't. Instead, use a library call (fstat (POSIX) or _filelength 
(Borland) or _OurLatestGuessAtAGoodFunctionName (Microsoft) or whatever 
it might be). Then wrap the call in your own function to abstract your 
code away from the dependency.

If you're reading the file for some other reason, do it the way I said 
above - read n 1-byte records, rather than 1 n-byte record, and keep 
careful track.

-- 
Richard Heathfield
Email: rjh at cpax dot org dot uk
"Usenet is a strange place" - dmr 29 July 1999
Sig line 4 vacant - apply within

[toc] | [prev] | [next] | [standalone]

#58003

On 03/08/2015 02:59 PM, Richard Heathfield wrote:
> On 08/03/15 18:47, ruben safir wrote:
>> if you use fread to read a file of unknown size with a set buffer size
>> of say 500 bytes, when you get to the end of the file you are likely to
>> read a stub of 500 bytes of info and the eof marker.
>>
>> And fread will not red that block
> 
> fread ought to return the number of bytes read. Instead, it returns the 
> number of blocks. I expect there was a good reason once.

I gather that there are systems, or at least particular devices on some
systems, where the number of bytes read will always be a multiple of a
block size characteristic of that device.

Even on systems where that is not true, if the file I'm reading has a
fixed record size, a valid read from an undamaged file will always read
in an integer number of records, so it still makes sense.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#58010

James Kuyper <jameskuyper@verizon.net> writes:
> On 03/08/2015 02:59 PM, Richard Heathfield wrote:
>> On 08/03/15 18:47, ruben safir wrote:
>>> if you use fread to read a file of unknown size with a set buffer size
>>> of say 500 bytes, when you get to the end of the file you are likely to
>>> read a stub of 500 bytes of info and the eof marker.
>>>
>>> And fread will not red that block
>> 
>> fread ought to return the number of bytes read. Instead, it returns the 
>> number of blocks. I expect there was a good reason once.
>
> I gather that there are systems, or at least particular devices on some
> systems, where the number of bytes read will always be a multiple of a
> block size characteristic of that device.

But fread() is defined as a sequence of calls to fgetc() (though of
course it doesn't literally have to call fgetc()).  N1570 7.21.8.1p2:

    The fread function reads, into the array pointed to by ptr,
    up to nmemb elements whose size is specified by size, from
    the stream pointed to by stream. For each object, size calls
    are made to the fgetc function and the results stored, in the
    order read, in an array of unsigned char exactly overlaying the
    object. The file position indicator for the stream (if defined)
    is advanced by the number of characters successfully read. If an
    error occurs, the resulting value of the file position indicator
    for the stream is indeterminate. If a partial element is read,
    its value is indeterminate.

> Even on systems where that is not true, if the file I'm reading has a
> fixed record size, a valid read from an undamaged file will always read
> in an integer number of records, so it still makes sense.

fread() attempts to read a specified number of elements, where you can
specify the size in bytes of each element.

The question is, what do you want to do if reading an element (record)
is only partly successful.  Say you're reading 1024-byte records from a
file that's 10752 bytes long (that's exactly 10.5 records).  If you care
how many bytes of that last partial record were read, use a size of 1.
If you don't care (i.e., if a partial read is simply a failure), use a
size equal to the record size.

I discussed this issue here: http://stackoverflow.com/a/8589688/827263

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#58004

On Sunday, March 8, 2015 at 7:59:16 PM UTC+1, Richard Heathfield wrote:
> fread ought to return the number of bytes read. Instead, it returns the 
> number of blocks. I expect there was a good reason once.
> 

Maybe there's one even today: if you are interested in bytes you just put
1 in the second parameter, but if you are reading, let's say, fixed-length
structures, maybe the number of items read would be more useful.

.. dunno.

I also believe that returning the number of bytes would have made much more
sense.

Ciao!

[toc] | [prev] | [next] | [standalone]

#58057

fmassei@gmail.com wrote:

> On Sunday, March 8, 2015 at 7:59:16 PM UTC+1, Richard Heathfield wrote:
> > fread ought to return the number of bytes read. Instead, it returns the 
> > number of blocks. I expect there was a good reason once.
> 
> Maybe there's one even today: if you are interested in bytes you just put
> 1 in the second parameter, but if you are reading, let's say, fixed-length
> structures, maybe the number of items read would be more useful.
> 
> .. dunno.
> 
> I also believe that returning the number of bytes would have made much more
> sense.

Let's put it this way.

If you're interested in the number of bytes read, getting that from
fread() is easy: use a 1 for the block size.

If you're interested in the number of intact records read, getting that
from fread() is just as easy: use the size of the record for the block
size.

If you want a hypothetical freadbytes() where you don't have to specify
that you want to know the number of bytes, that's easy to implement in
terms of fread(). Going the other way isn't hard, but it requires an
extra calculation that fread() now doesn't.

There was no good reason to restrict fread() to report bytes only, and
some good reason to allow it to report record size.

Richard

[toc] | [prev] | [next] | [standalone]

#58005

On Sunday March 8 2015 14:59, in comp.lang.c, "Richard Heathfield"
<rjh@cpax.org.uk> wrote:

> On 08/03/15 18:47, ruben safir wrote:
>> if you use fread to read a file of unknown size with a set buffer size
>> of say 500 bytes, when you get to the end of the file you are likely to
>> read a stub of 500 bytes of info and the eof marker.
>>
>> And fread will not red that block
> 
> fread ought to return the number of bytes read. Instead, it returns the
> number of blocks. I expect there was a good reason once.

I imagine that the use-case looks something like

  struct some_intricate_storage_structure savedData[10];
  size_t n_blocks_read;
  int block;

  n_blocks_read = fread(savedData,sizeof savedData[0],10,fd);

  if (n_blocks_read)
    for (block = 0; block < n_blocks_read; ++block)
    { /* process savedData[block] */ }


[snip]
-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request

[toc] | [prev] | [next] | [standalone]

#57998

Le 08/03/2015 19:47, ruben safir a écrit :
> if you use fread to read a file of unknown size with a set buffer size
> of say 500 bytes, when you get to the end of the file you are likely to
> read a stub of 500 bytes of info and the eof marker.
>
> And fread will not red that block and set and oef that can be read with
> feof.  That is how I understand thinga.  So we can either use fseek to
> discover the file size and fread accordingly or use fstat?
>
> is this the proper approach?
>
> Ruben
>


You use

fread(ptr,1, 500,stream);

and fread will return the number of bytes read.

Then

size_t br;

while ((br=fread(ptr,1,500,stream)) == 500)
   ;

/* br contains now the number of bytes read in the last  block */

[toc] | [prev] | [next] | [standalone]

#58000

ruben safir <ruben@mrbrklyn.com> writes:
> if you use fread to read a file of unknown size with a set buffer size
> of say 500 bytes, when you get to the end of the file you are likely to
> read a stub of 500 bytes of info and the eof marker.
>
> And fread will not red that block and set and oef that can be read with
> feof.  That is how I understand thinga.  So we can either use fseek to
> discover the file size and fread accordingly or use fstat?
>
> is this the proper approach?

You've already gotten some good answers.  One more thing: Don't use the
feof() function to decide when to stop reading.  Use the result returned
by whatever function you're using to read from the file (fread() in your
case).

feof() returns the value of the end-of-file flag for the file.  That
flag is set *after* you attempt to read past the end of the file -- and
it's not set if there was an error (that sets the error flag, which you
can query with ferror()).  So if there's an error reading from the file,
just calling feof() won't detect it, and you can have an infinite loop.

You can use feof() and/or ferror() *after* reaching the end of the file
to find out why there was no more input.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#58006

ruben safir <ruben@mrbrklyn.com> wrote:
> if you use fread to read a file of unknown size with a set buffer size
> of say 500 bytes, when you get to the end of the file you are likely to
> read a stub of 500 bytes of info and the eof marker.
 
> And fread will not red that block and set and oef that can be read with
> feof.  That is how I understand thinga.  So we can either use fseek to
> discover the file size and fread accordingly or use fstat?
 
> is this the proper approach?

Best is to fread() until it returns zero. 

In most cases, that makes a nice while (or for) loop. 

As I understand it, it is Pascal where you are supposed to test for EOF
before reading, and that "bad things" (tm) happen if you keep reading.
C doesn't work that way.

Normally, once you reach EOF fread() will keep returning zero.
It might be that for some tape systems, after reading zero bytes that
it will continue on to the next file.

It has been some years since I thought about how tail -f works, but note
that a disk file can grow while you are reading it. 

Theoretically, you use feof() after you read zero bytes to find out if
it was an actual EOF or an error. In most cases, I don't care which one,
and besides, systems are reliable enough that most kinds of errors
that might have occured years ago, don't occur. 

-- glen

[toc] | [prev] | [next] | [standalone]

#58007

On 03/08/2015 07:39 PM, glen herrmannsfeldt wrote:
> ruben safir <ruben@mrbrklyn.com> wrote:
>> if you use fread to read a file of unknown size with a set buffer size
>> of say 500 bytes, when you get to the end of the file you are likely to
>> read a stub of 500 bytes of info and the eof marker.
>  
>> And fread will not red that block and set and oef that can be read with
>> feof.  That is how I understand thinga.  So we can either use fseek to
>> discover the file size and fread accordingly or use fstat?
>  
>> is this the proper approach?
> 
> Best is to fread() until it returns zero.

Just saying "use fread() until it returns zero" doesn't answer the key
question of how fread() should be used. As Richard has already pointed
out, you can avoid certain problems by using 1 as the element size, and
the size of your buffer as the number of elements to be read.

If you do it the other way around, reading 1 element with a size equal
to the buffer, there's a problem when you reach the end of the file if
the file size is not an exact multiple of the buffer size. "If a partial
element is read, its value is indeterminate." (7.2.81.p2). Among the
possibilities allowed by this clause is that the buffer provided might
be unchanged by call to fread(), or zeroed-out, or filled with garbage.
If you want to do something with that last part of the file, it's
imperative that that the value not be indeterminate.

Therefore, if is possible for the file to not be an exact multiple of
some fixed block/record size, you'd better use an element size of 1, so
that a partial element is not a possibility.

> It has been some years since I thought about how tail -f works, but note
> that a disk file can grow while you are reading it. 

Whether that is possible (I've used systems where it wasn't), and how to
deal with it, if necessary (it often isn't), are outside the scope of
the C standard. "How to deal with it" varies significantly between
different operating systems.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#58008

James Kuyper <jameskuyper@verizon.net> wrote:

(snip, I wrote)

>> Best is to fread() until it returns zero.
 
> Just saying "use fread() until it returns zero" doesn't answer the key
> question of how fread() should be used. As Richard has already pointed
> out, you can avoid certain problems by using 1 as the element size, and
> the size of your buffer as the number of elements to be read.

I think I most often use 1, but if the program requires whole records
(such as a whole struct) then putting in that size and processing the
number returned might be the best choice. 
 
> If you do it the other way around, reading 1 element with a size equal
> to the buffer, there's a problem when you reach the end of the file if
> the file size is not an exact multiple of the buffer size. "If a partial
> element is read, its value is indeterminate." (7.2.81.p2). Among the
> possibilities allowed by this clause is that the buffer provided might
> be unchanged by call to fread(), or zeroed-out, or filled with garbage.
> If you want to do something with that last part of the file, it's
> imperative that that the value not be indeterminate.

Yes.

> Therefore, if is possible for the file to not be an exact multiple of
> some fixed block/record size, you'd better use an element size of 1, so
> that a partial element is not a possibility.
 
>> It has been some years since I thought about how tail -f works, but note
>> that a disk file can grow while you are reading it. 
 
> Whether that is possible (I've used systems where it wasn't), and how to
> deal with it, if necessary (it often isn't), are outside the scope of
> the C standard. "How to deal with it" varies significantly between
> different operating systems.

Yes, but does the C standard allow for a file to reach EOF, and then,
on the next read, not be at EOF?

I did once (OS/2 1.0 days) port gnu tail to OS/2, and it did work.
I didn't try very hard to figure out how it worked, just that it did.

That was in the days when unix systems were too expensive for home use,
but I could run OS/2 (someone gave me a copy of 1.0) on an AT clone,
with grep and diff (two most useful for development) and some other
slightly less useful programs.

-- glen

[toc] | [prev] | [next] | [standalone]

#58017

glen herrmannsfeldt <gah@ugcs.caltech.edu> writes:
[...]
> Yes, but does the C standard allow for a file to reach EOF, and then,
> on the next read, not be at EOF?

I don't think so, but fseek() clears the end-of-file indicator.

> I did once (OS/2 1.0 days) port gnu tail to OS/2, and it did work.
> I didn't try very hard to figure out how it worked, just that it did.

I think tail (specifically `tail -f`) uses fseek(file, 0, SEEK_CUR).

[...]

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#58024

Keith Thompson <kst-u@mib.org> wrote:

(snip, I wrote)
>> Yes, but does the C standard allow for a file to reach EOF, and then,
>> on the next read, not be at EOF?
 
> I don't think so, but fseek() clears the end-of-file indicator.
 
>> I did once (OS/2 1.0 days) port gnu tail to OS/2, and it did work.
>> I didn't try very hard to figure out how it worked, just that it did.
 
> I think tail (specifically `tail -f`) uses fseek(file, 0, SEEK_CUR).

Okay, I was going to say that once fread() returned zero it would
keep doing that, but then I remembered tail.  But then it was about 25
years ago that I was looking at tail.  I did remember wondering if
it would work the same way.

The biggest problem I remember with gnu utilities is that they would
use 0 as an argument to a function expecting (char *). When I complained
to the gnu people, they had some comment about systems having different
sized (int) and (char*) not being supported.

-- glen

[toc] | [prev] | [next] | [standalone]

#58025

glen herrmannsfeldt <gah@ugcs.caltech.edu> writes:
> Keith Thompson <kst-u@mib.org> wrote:
> (snip, I wrote)
>>> Yes, but does the C standard allow for a file to reach EOF, and then,
>>> on the next read, not be at EOF?
>  
>> I don't think so, but fseek() clears the end-of-file indicator.
>  
>>> I did once (OS/2 1.0 days) port gnu tail to OS/2, and it did work.
>>> I didn't try very hard to figure out how it worked, just that it did.
>  
>> I think tail (specifically `tail -f`) uses fseek(file, 0, SEEK_CUR).
>
> Okay, I was going to say that once fread() returned zero it would
> keep doing that, but then I remembered tail.  But then it was about 25
> years ago that I was looking at tail.  I did remember wondering if
> it would work the same way.
>
> The biggest problem I remember with gnu utilities is that they would
> use 0 as an argument to a function expecting (char *). When I complained
> to the gnu people, they had some comment about systems having different
> sized (int) and (char*) not being supported.

I don't suppose you have an example?

If there's a visible prototype, passing 0 to a non-variadic function
expecting a char* is not a problem.  It will be implicitly converted to
char*, and converting a constant 0 to a pointer type yields a null
pointer.  (Personally I'd use NULL or (char*)NULL just for the sake of
clarity.)

Passing 0 to a function expecting a char* either when there's no visible
prototype, or when it corresponds to the ", ..." in a variadic function,
does have undefined behavior.  It can break if char* and int are of
different sizes, if a null pointer is not represented as all-bits-zero,
if int and char* are passed as arguments using different mechanisms, or
for any other conceivable reason.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#58031

Keith Thompson <kst-u@mib.org> wrote:

(snip, I wrote)

>> The biggest problem I remember with gnu utilities is that they would
>> use 0 as an argument to a function expecting (char *). When I complained
>> to the gnu people, they had some comment about systems having different
>> sized (int) and (char*) not being supported.
 
> I don't suppose you have an example?
 
> If there's a visible prototype, passing 0 to a non-variadic function
> expecting a char* is not a problem.  It will be implicitly converted to
> char*, and converting a constant 0 to a pointer type yields a null
> pointer.  (Personally I'd use NULL or (char*)NULL just for the sake of
> clarity.)

That was in 1990.  I was running MSC 6.0 (the first with OS/2 support)
and large mode, so pointers were segment selector/offset in 32 bits,
and int at 16 bits. 
 
> Passing 0 to a function expecting a char* either when there's no visible
> prototype, or when it corresponds to the ", ..." in a variadic function,
> does have undefined behavior.  It can break if char* and int are of
> different sizes, if a null pointer is not represented as all-bits-zero,
> if int and char* are passed as arguments using different mechanisms, or
> for any other conceivable reason.

Microsoft had ANSI C support around 1989, but I don't know how fast
people went back and added the prototypes.  I do know that it wouldn't
run until I put in (char*)0 for the 0 in the right places.

-- glen

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web