Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #62085 > unrolled thread

Mysterious slow IO on fread() to large buffer on large Dell system

Back to article view | Back to comp.lang.c

Contents

  Mysterious slow IO on fread() to large buffer on large Dell system mathog <dmathog@gmail.com> - 2015-05-15 18:45 -0700
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Ian Collins <ian-news@hotmail.com> - 2015-05-16 14:09 +1200
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Siri Cruz <chine.bleu@yahoo.com> - 2015-05-15 19:20 -0700
      Re: Mysterious slow IO on fread() to large buffer on large Dell system mathog <dmathog@gmail.com> - 2015-05-15 22:29 -0700
        Re: Mysterious slow IO on fread() to large buffer on large Dell system Siri Cruz <chine.bleu@yahoo.com> - 2015-05-16 08:47 -0700
      Re: Mysterious slow IO on fread() to large buffer on large Dell system Robert Wessel <robertwessel2@yahoo.com> - 2015-05-16 02:11 -0500
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-05-15 19:23 -0700
    Re: Mysterious slow IO on fread() to large buffer on large Dell system glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-05-16 05:05 +0000
      Re: Mysterious slow IO on fread() to large buffer on large Dell system Melzzzzz <mel@zzzzz.com> - 2015-05-16 07:20 +0200
      Re: Mysterious slow IO on fread() to large buffer on large Dell system mathog <dmathog@gmail.com> - 2015-05-15 22:27 -0700
        Re: Mysterious slow IO on fread() to large buffer on large Dell system Melzzzzz <mel@zzzzz.com> - 2015-05-16 07:59 +0200
        Re: Mysterious slow IO on fread() to large buffer on large Dell system Ian Collins <ian-news@hotmail.com> - 2015-05-16 18:31 +1200
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2015-05-16 10:57 -0400
      Re: Mysterious slow IO on fread() to large buffer on large Dell system Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2015-05-16 10:51 -0600
      Re: Mysterious slow IO on fread() to large buffer on large Dell system Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-05-16 11:41 -0700
        Re: Mysterious slow IO on fread() to large buffer on large Dell system Melzzzzz <mel@zzzzz.com> - 2015-05-16 21:01 +0200
          Re: Mysterious slow IO on fread() to large buffer on large Dell system glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-05-16 19:37 +0000
            Re: Mysterious slow IO on fread() to large buffer on large Dell system Noob <root@127.0.0.1> - 2015-05-16 22:14 +0200
      Re: Mysterious slow IO on fread() to large buffer on large Dell system gamo <gamo@telecable.es> - 2015-05-17 12:36 +0200
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Paul <nospam@needed.com> - 2015-05-16 11:29 -0400
    Re: Mysterious slow IO on fread() to large buffer on large Dell system mathog <dmathog@gmail.com> - 2015-05-16 10:50 -0700
      Re: Mysterious slow IO on fread() to large buffer on large Dell system Johann Klammer <klammerj@NOSPAM.a1.net> - 2015-05-17 01:43 +0200
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Stephen Sprunk <stephen@sprunk.org> - 2015-05-16 16:40 -0500
    Re: Mysterious slow IO on fread() to large buffer on large Dell system mathog <dmathog@gmail.com> - 2015-05-26 09:58 -0700

Page 1 of 2  [1] 2  Next page →

#62085 — Mysterious slow IO on fread() to large buffer on large Dell system

Hi all,

On a big Dell system ( PowerEdge T620, >500Gb RAM, 48 CPUs  Xeon(R) CPU 
E5-2695 v2 @ 2.40GHz, Centos 6.6 ) the following snippet of code is used 
to open and read in a 13.75 Gb file:

       /* error checking removed for brevity */
       size_t len_file, rlen;
       int status;

       fin=fopen("filename", "rb");
       status = fseek(fin, 0L, SEEK_END);
       len_file = ftell(fin);
       fseek(fin, 0L, SEEK_SET);
       buffer=malloc(len_file); // this takes only a few seconds
       rlen = fread(buffer, 1, len_file, fin); // this takes forever

This takes 15 minutes(!!!), which works out to a "whopping" 15.3Mb/s. 
Nothing unusual in the way it is run, it is just:

   program_name filename

Conversely, this is very fast:

time dd if=filename bs=8192 of=/dev/null
1679369+1 records in
1679369+1 records out
13757396191 bytes (14 GB) copied, 3.2568 s, 4.2 GB/s

real    0m3.258s
user    0m0.113s
sys     0m3.149s

The only clue I see is that "migration" processes keep popping up while 
the C program runs, as if the OS is for some reason pushing the process 
from CPU to CPU, and each time it does that it also has to somehow move 
that large array.  There is nothing else running on this box except 
background processes.

If the len_file value is hardcoded in and the fseek()'s removed it is
just as slow.

The program was compiled with:

   gcc -Wall -std=c99 -pedantic -o program_name program_name.c

Another mode of this program just copies record ranges from the input
to the output - without hanging onto them in a big buffer.  With the 
range set to all records it is reasonably fast given the large number of 
calls to fread, taking ~100s to copy the entire file, and that includes 
writing it back out again.  All it does is read binary records one at a 
time in a loop around

   rlen = fread(buffer,1,reclen,fin);

where reclen was 7 in the test and there were 1965335400 records
(and so just as many calls to fread()).

Anybody have an idea what is going on here?  It seems to be
an issue with the large buffer interacting strangely with the OS more 
than the fread() per se.

Thanks,

David Mathog

[toc] | [next] | [standalone]

#62086

mathog wrote:
> Hi all,
>
> On a big Dell system ( PowerEdge T620, >500Gb RAM, 48 CPUs  Xeon(R) CPU
> E5-2695 v2 @ 2.40GHz, Centos 6.6 ) the following snippet of code is used
> to open and read in a 13.75 Gb file:
>
>         /* error checking removed for brevity */
>         size_t len_file, rlen;
>         int status;
>
>         fin=fopen("filename", "rb");
>         status = fseek(fin, 0L, SEEK_END);
>         len_file = ftell(fin);
>         fseek(fin, 0L, SEEK_SET);
>         buffer=malloc(len_file); // this takes only a few seconds
>         rlen = fread(buffer, 1, len_file, fin); // this takes forever
>
> This takes 15 minutes(!!!), which works out to a "whopping" 15.3Mb/s.
> Nothing unusual in the way it is run, it is just:

Why don't you simply memory map the file?

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#62087

In article <mj67fq$jq1$1@dont-email.me>, mathog <dmathog@gmail.com> wrote:

> Hi all,
> 
> On a big Dell system ( PowerEdge T620, >500Gb RAM, 48 CPUs  Xeon(R) CPU 
> E5-2695 v2 @ 2.40GHz, Centos 6.6 ) the following snippet of code is used 
> to open and read in a 13.75 Gb file:
> 
>        /* error checking removed for brevity */
>        size_t len_file, rlen;
>        int status;
> 
>        fin=fopen("filename", "rb");
>        status = fseek(fin, 0L, SEEK_END);
>        len_file = ftell(fin);
>        fseek(fin, 0L, SEEK_SET);
>        buffer=malloc(len_file); // this takes only a few seconds
>        rlen = fread(buffer, 1, len_file, fin); // this takes forever

Doctor! Doctor! It hurts when I do this!

Then stop doing that.


Do you realise you're blowing out your virtual memory for what amounts to a disk 
to disk copy? stdio is for reading in slices of files that will likely fit in 
the working set. If you want to map in a file, map in the file. Linux has a 
kernel call to do that.

-- 
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted.
'I desire mercy, not sacrifice.'
When is a Kenyan not a Kenyan? When he's a Canadian.
That's People's Commissioner Siri Cruz now. Punch!

[toc] | [prev] | [next] | [standalone]

#62092

Siri Cruz wrote:

>
> Do you realise you're blowing out your virtual memory for what amounts to a disk
> to disk copy? stdio is for reading in slices of files that will likely fit in
> the working set. If you want to map in a file, map in the file. Linux has a
> kernel call to do that.

It also accepts input from stdin, and  fread() works either way.

Regards,

David Mathog


---
This email has been checked for viruses by Avast antivirus software.
http://www.avast.com

[toc] | [prev] | [next] | [standalone]

#62114

In article <mj6khr$p3l$2@dont-email.me>, mathog <dmathog@gmail.com> wrote:

> Siri Cruz wrote:
> 
> >
> > Do you realise you're blowing out your virtual memory for what amounts to a 
> > disk
> > to disk copy? stdio is for reading in slices of files that will likely fit 
> > in
> > the working set. If you want to map in a file, map in the file. Linux has a
> > kernel call to do that.
> 
> It also accepts input from stdin, and  fread() works either way.

The kernel should only increase the working set after the process is in 
incipient thrash. The kernel will then guess a new working set until incipient 
thrash again. The process will eventually get all the real memory it wants and 
more, but this kind of real memory allocation is contrary to goals of virtual 
memory. 

malloc doesn't need to touch allocated pages, so malloc itself can only be 
expected to allocate VM not RM. Some mallocs will actually just map in a new 
segment when the allocation is this large.

Some systems let you lock pages in RM (wired down pages), which will allocate 
RM. However unices today should have a map in which is intended to add large 
files like this to the VM, and then let it page in as needed. If mapin rejects 
because the file is not on a pageable device, you can copy, with buffer sized 
slices, to a temporary which is then mapped in.

-- 
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted.
'I desire mercy, not sacrifice.'
When is a Kenyan not a Kenyan? When he's a Canadian.
That's People's Commissioner Siri Cruz now. Punch!

[toc] | [prev] | [next] | [standalone]

#62102

On Fri, 15 May 2015 19:20:07 -0700, Siri Cruz <chine.bleu@yahoo.com>
wrote:

>In article <mj67fq$jq1$1@dont-email.me>, mathog <dmathog@gmail.com> wrote:
>
>> Hi all,
>> 
>> On a big Dell system ( PowerEdge T620, >500Gb RAM, 48 CPUs  Xeon(R) CPU 
>> E5-2695 v2 @ 2.40GHz, Centos 6.6 ) the following snippet of code is used 
>> to open and read in a 13.75 Gb file:
>> 
>>        /* error checking removed for brevity */
>>        size_t len_file, rlen;
>>        int status;
>> 
>>        fin=fopen("filename", "rb");
>>        status = fseek(fin, 0L, SEEK_END);
>>        len_file = ftell(fin);
>>        fseek(fin, 0L, SEEK_SET);
>>        buffer=malloc(len_file); // this takes only a few seconds
>>        rlen = fread(buffer, 1, len_file, fin); // this takes forever
>
>Doctor! Doctor! It hurts when I do this!
>
>Then stop doing that.
>
>
>Do you realise you're blowing out your virtual memory for what amounts to a disk 
>to disk copy? stdio is for reading in slices of files that will likely fit in 
>the working set. If you want to map in a file, map in the file. Linux has a 
>kernel call to do that.


He does have 500GB of RAM.  13.75GB should certainly fit in his
working set.

That being said, I agree this is likely some issue with virtual
memory.  Perhaps the processes is being limited to less than 14GB of
physical RAM (despite there being much more on the system).  Since
this is a server, it may well be set up with those sorts of limits.
One way to do that in Linux is with "control groups".  There are
doubtless other methods.

While mmaping a file like this might be a better solution, expecting
to load 13.75GB of data into memory in a process on a 500GB machine is
certainly not unreasonable.

In any event, this is quite OT here.

[toc] | [prev] | [next] | [standalone]

#62088

On Saturday, May 16, 2015 at 2:46:45 AM UTC+1, mathog wrote:
> 
> Anybody have an idea what is going on here?  It seems to be
> an issue with the large buffer interacting strangely with the OS more 
> than the fread() per se.
> 
You need to do a bit of experimental programming.

Open the file, and read it character by character using fgetc(). If
it still takes about fifteen minutes, that means that the problem
is in the file. If it zips through, then your problem is in the buffer.

If the problem is in the file, try creating a new file of similar size.
If that one reads OK, then it's something to do with the way your file
is saved on disk (fragmentation, a dicky disk sector, something like 
that). So you can solve the problem by regenerating the file. If you
still have the problem, then the disk is slow for access to huge files,
your only real option is a different type of drive.

If the problem is in the buffer, then it could be that your buffer 
is being paged out to disk. Whilst you've got 500GB RAM, somewhere
the OS has been told only to give the process 4GB or so. 
Alternatively, there could be come catastrophic cache interaction. 
If the latter, it's hard to solve, you'll need to understand how the
cache works. There'll probably be some low-level way round it. 

[toc] | [prev] | [next] | [standalone]

#62089

mathog <dmathog@gmail.com> wrote:
 
> On a big Dell system ( PowerEdge T620, >500Gb RAM, 48 CPUs  Xeon(R) CPU 
> E5-2695 v2 @ 2.40GHz, Centos 6.6 ) the following snippet of code is used 
> to open and read in a 13.75 Gb file:
 
(snip)

>       fin=fopen("filename", "rb");
>       status = fseek(fin, 0L, SEEK_END);
>       len_file = ftell(fin);
>       fseek(fin, 0L, SEEK_SET);
>       buffer=malloc(len_file); // this takes only a few seconds
>       rlen = fread(buffer, 1, len_file, fin); // this takes forever
 
> This takes 15 minutes(!!!), which works out to a "whopping" 15.3Mb/s. 
> Nothing unusual in the way it is run, it is just:
 
>   program_name filename
 
> Conversely, this is very fast:
 
> time dd if=filename bs=8192 of=/dev/null
> 1679369+1 records in
> 1679369+1 records out
> 13757396191 bytes (14 GB) copied, 3.2568 s, 4.2 GB/s
 
You have to be careful with the timing.  Did you do the first one first,
and then the second one?

On a system with many gigabytes of disk cache in memory? 

I suspect that the second was reading from disk cache, not actual disk.

I am not sure there is a good way to flush the disk cache, other than
reboot. If you do that, then time them the other way around, you will
have a better idea.

-- glen

[toc] | [prev] | [next] | [standalone]

#62090

On Sat, 16 May 2015 05:05:23 +0000 (UTC)
glen herrmannsfeldt <gah@ugcs.caltech.edu> wrote:

> mathog <dmathog@gmail.com> wrote:
>  
> > On a big Dell system ( PowerEdge T620, >500Gb RAM, 48 CPUs  Xeon(R)
> > CPU E5-2695 v2 @ 2.40GHz, Centos 6.6 ) the following snippet of
> > code is used to open and read in a 13.75 Gb file:
>  
> (snip)
> 
> >       fin=fopen("filename", "rb");
> >       status = fseek(fin, 0L, SEEK_END);
> >       len_file = ftell(fin);
> >       fseek(fin, 0L, SEEK_SET);
> >       buffer=malloc(len_file); // this takes only a few seconds
> >       rlen = fread(buffer, 1, len_file, fin); // this takes forever
>  
> > This takes 15 minutes(!!!), which works out to a "whopping"
> > 15.3Mb/s. Nothing unusual in the way it is run, it is just:
>  
> >   program_name filename
>  
> > Conversely, this is very fast:
>  
> > time dd if=filename bs=8192 of=/dev/null
> > 1679369+1 records in
> > 1679369+1 records out
> > 13757396191 bytes (14 GB) copied, 3.2568 s, 4.2 GB/s
>  
> You have to be careful with the timing.  Did you do the first one
> first, and then the second one?
> 
> On a system with many gigabytes of disk cache in memory? 
> 
> I suspect that the second was reading from disk cache, not actual
> disk.
> 
> I am not sure there is a good way to flush the disk cache, other than
> reboot. If you do that, then time them the other way around, you will
> have a better idea.
> 
> -- glen
On Linux it is:
# echo 3 > /proc/sys/vm/drop_caches

[toc] | [prev] | [next] | [standalone]

#62091

glen herrmannsfeldt wrote:
> You have to be careful with the timing.  Did you do the first one first,
> and then the second one?
>
> On a system with many gigabytes of disk cache in memory?
>
> I suspect that the second was reading from disk cache, not actual disk.

I did them both twice with the same results.  So not a read from disk 
vs. read from cache issue.  It also isn't just one particular file that 
does this, it is any file, including just a pile of zero bytes made with dd.

Thanks

---
This email has been checked for viruses by Avast antivirus software.
http://www.avast.com

[toc] | [prev] | [next] | [standalone]

#62093

On Fri, 15 May 2015 22:27:19 -0700
mathog <dmathog@gmail.com> wrote:

> glen herrmannsfeldt wrote:
> > You have to be careful with the timing.  Did you do the first one
> > first, and then the second one?
> >
> > On a system with many gigabytes of disk cache in memory?
> >
> > I suspect that the second was reading from disk cache, not actual
> > disk.
> 
> I did them both twice with the same results.  So not a read from disk 
> vs. read from cache issue.  It also isn't just one particular file
> that does this, it is any file, including just a pile of zero bytes
> made with dd.
> 
> Thanks
> 
Have you tried to read file chunk by chunk? I mean not to read whole
content at once, rather to read eg 4kb by 4kb and discard. Perhaps what
takes time is OS allocating large number of pages?

[toc] | [prev] | [next] | [standalone]

#62097

mathog wrote:
> glen herrmannsfeldt wrote:
>> You have to be careful with the timing.  Did you do the first one first,
>> and then the second one?
>>
>> On a system with many gigabytes of disk cache in memory?
>>
>> I suspect that the second was reading from disk cache, not actual disk.
>
> I did them both twice with the same results.  So not a read from disk
> vs. read from cache issue.  It also isn't just one particular file that
> does this, it is any file, including just a pile of zero bytes made with dd.

What happens if you map the file then read to the end?

What platform are you on?

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#62111

On Friday May 15 2015 21:45, in comp.lang.c, "mathog" <dmathog@gmail.com>
wrote:

> Hi all,
> 
> On a big Dell system ( PowerEdge T620, >500Gb RAM, 48 CPUs  Xeon(R) CPU
> E5-2695 v2 @ 2.40GHz, Centos 6.6 ) the following snippet of code is used
> to open and read in a 13.75 Gb file:
> 
>        /* error checking removed for brevity */
>        size_t len_file, rlen;
>        int status;
> 
>        fin=fopen("filename", "rb");
>        status = fseek(fin, 0L, SEEK_END);
>        len_file = ftell(fin);
>        fseek(fin, 0L, SEEK_SET);
>        buffer=malloc(len_file); // this takes only a few seconds
>        rlen = fread(buffer, 1, len_file, fin); // this takes forever
> 
> This takes 15 minutes(!!!), 

If I read that fread() call correctly, you are (atomically) reading the file
one byte at a time, for as many bytes as will fit in your buffer. That's
gotta be expensive.

How about reading the file once, in <<buffer_sized>> chunks? That should
remove the excessive memory-to-memory copy that happens with the other read.

Replace your fread() with
        rlen = fread(buffer, len_file, 1, fin);
and see if that improves your throughput.


-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request

[toc] | [prev] | [next] | [standalone]

#62119

Lew Pitcher <lew.pitcher@digitalfreehold.ca> writes:

> On Friday May 15 2015 21:45, in comp.lang.c, "mathog" <dmathog@gmail.com>
> wrote:
>
>> Hi all,
>> 
>> On a big Dell system ( PowerEdge T620, >500Gb RAM, 48 CPUs  Xeon(R) CPU
>> E5-2695 v2 @ 2.40GHz, Centos 6.6 ) the following snippet of code is used
>> to open and read in a 13.75 Gb file:
>> 
>>        /* error checking removed for brevity */
>>        size_t len_file, rlen;
>>        int status;
>> 
>>        fin=fopen("filename", "rb");
>>        status = fseek(fin, 0L, SEEK_END);
>>        len_file = ftell(fin);
>>        fseek(fin, 0L, SEEK_SET);
>>        buffer=malloc(len_file); // this takes only a few seconds
>>        rlen = fread(buffer, 1, len_file, fin); // this takes forever
>> 
>> This takes 15 minutes(!!!), 
>
> If I read that fread() call correctly, you are (atomically) reading the file
> one byte at a time, for as many bytes as will fit in your buffer. That's
> gotta be expensive.

While that's the effect of the call, I certainly hope that isn't the
underlying implementation....

> How about reading the file once, in <<buffer_sized>> chunks? That should
> remove the excessive memory-to-memory copy that happens with the other read.
>
> Replace your fread() with
>         rlen = fread(buffer, len_file, 1, fin);
> and see if that improves your throughput.

Though that would be an interesting experiment.

Another one would be to write a five-line program that would use read()
to read the file into a buffer and see how long it takes -- that would
determine whether the problem is in the standard IO library or somewhere
at a lower level.

[toc] | [prev] | [next] | [standalone]

#62122

On Saturday, May 16, 2015 at 3:57:16 PM UTC+1, Lew Pitcher wrote:
> 
> If I read that fread() call correctly, you are (atomically) reading the file
> one byte at a time, for as many bytes as will fit in your buffer. That's
> gotta be expensive.
> 
He's got 13GB or so to read.
The operation is taking 15 minutes. But the computer runs at 2.4Ghz.
So if I'm doing my calculations correctly, each byte is taking 166
operations to read. That seems way too high, even if there's a function 
call for each one.

[toc] | [prev] | [next] | [standalone]

#62123

On Sat, 16 May 2015 11:41:09 -0700 (PDT)
Malcolm McLean <malcolm.mclean5@btinternet.com> wrote:

> On Saturday, May 16, 2015 at 3:57:16 PM UTC+1, Lew Pitcher wrote:
> > 
> > If I read that fread() call correctly, you are (atomically) reading
> > the file one byte at a time, for as many bytes as will fit in your
> > buffer. That's gotta be expensive.
> > 
> He's got 13GB or so to read.
> The operation is taking 15 minutes. But the computer runs at 2.4Ghz.
> So if I'm doing my calculations correctly, each byte is taking 166
> operations to read. That seems way too high, even if there's a
> function call for each one.

I have experienced extremely slow memory initialization on first pass
when disk cache occupies most of ram. Say it's about 20-200 times slower
then on second initialization pass. Say it could take around minute to
initialize chunk of 800mb on my machine.
This issue seems fixed in latest kernel 4.0.3

[toc] | [prev] | [next] | [standalone]

#62125

Melzzzzz <mel@zzzzz.com> wrote:
> On Sat, 16 May 2015 11:41:09 -0700 (PDT)
> Malcolm McLean <malcolm.mclean5@btinternet.com> wrote:
 
(snip)

>> He's got 13GB or so to read.
>> The operation is taking 15 minutes. But the computer runs at 2.4Ghz.
>> So if I'm doing my calculations correctly, each byte is taking 166
>> operations to read. That seems way too high, even if there's a
>> function call for each one.
 
> I have experienced extremely slow memory initialization on first pass
> when disk cache occupies most of ram. Say it's about 20-200 times slower
> then on second initialization pass. Say it could take around minute to
> initialize chunk of 800mb on my machine.
> This issue seems fixed in latest kernel 4.0.3

So, another test would be to fill the malloc()ed space with some value,
then start timing read the data, and stop timing. That separates memory
allocation, especially copy on write, from file reading.

Yes, many systems don't actually make the malloc() memory available
until you actually use it.

-- glen

[toc] | [prev] | [next] | [standalone]

#62127

On 16/05/2015 21:37, glen herrmannsfeldt wrote:

> Yes, many systems don't actually make the malloc() memory available
> until you actually use it.

[OT] Related topic:

https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

[toc] | [prev] | [next] | [standalone]

#62146

El 16/05/15 a las 16:57, Lew Pitcher escribió:
> If I read that fread() call correctly, you are (atomically) reading the file
> one byte at a time, for as many bytes as will fit in your buffer. That's
> gotta be expensive.
>
> How about reading the file once, in <<buffer_sized>> chunks? That should
> remove the excessive memory-to-memory copy that happens with the other read.
>
> Replace your fread() with
>          rlen = fread(buffer, len_file, 1, fin);
> and see if that improves your throughput.

Your solution seems to be the better.
If I were the OP I'll try this as fast as I could.

I wrote only to tell that I have done similar experiments
with other language, and found that in my PC a lenght of
1024*1024 bytes to read at a time, is nearly as good as
read the entire file "in a stream." But not less than
that, for files that weight gigabytes.

Regards.

-- 
http://www.telecable.es/personales/gamo/
The generation of random numbers is too important to be left to chance

[toc] | [prev] | [next] | [standalone]

#62112

mathog wrote:
> Hi all,
> 
> On a big Dell system ( PowerEdge T620, >500Gb RAM, 48 CPUs  Xeon(R) CPU 
> E5-2695 v2 @ 2.40GHz, Centos 6.6 ) the following snippet of code is used 
> to open and read in a 13.75 Gb file:
> 
>       /* error checking removed for brevity */
>       size_t len_file, rlen;
>       int status;
> 
>       fin=fopen("filename", "rb");
>       status = fseek(fin, 0L, SEEK_END);
>       len_file = ftell(fin);
>       fseek(fin, 0L, SEEK_SET);
>       buffer=malloc(len_file); // this takes only a few seconds
>       rlen = fread(buffer, 1, len_file, fin); // this takes forever
> 
> This takes 15 minutes(!!!), which works out to a "whopping" 15.3Mb/s. 
> Nothing unusual in the way it is run, it is just:
> 
>   program_name filename
> 
> Conversely, this is very fast:
> 
> time dd if=filename bs=8192 of=/dev/null
> 1679369+1 records in
> 1679369+1 records out
> 13757396191 bytes (14 GB) copied, 3.2568 s, 4.2 GB/s
> 
> real    0m3.258s
> user    0m0.113s
> sys     0m3.149s
> 
> The only clue I see is that "migration" processes keep popping up while 
> the C program runs, as if the OS is for some reason pushing the process 
> from CPU to CPU, and each time it does that it also has to somehow move 
> that large array.  There is nothing else running on this box except 
> background processes.
> 
> If the len_file value is hardcoded in and the fseek()'s removed it is
> just as slow.
> 
> The program was compiled with:
> 
>   gcc -Wall -std=c99 -pedantic -o program_name program_name.c
> 
> Another mode of this program just copies record ranges from the input
> to the output - without hanging onto them in a big buffer.  With the 
> range set to all records it is reasonably fast given the large number of 
> calls to fread, taking ~100s to copy the entire file, and that includes 
> writing it back out again.  All it does is read binary records one at a 
> time in a loop around
> 
>   rlen = fread(buffer,1,reclen,fin);
> 
> where reclen was 7 in the test and there were 1965335400 records
> (and so just as many calls to fread()).
> 
> Anybody have an idea what is going on here?  It seems to be
> an issue with the large buffer interacting strangely with the OS more 
> than the fread() per se.
> 
> Thanks,
> 
> David Mathog

Not that this is necessarily the right answer, but
I happen to like it as a cheap way to improve some
free software I use. (Add a couple lines to the code
and see what happens.)

         #define FILEBUFF 65536
         setvbuf(in, (char *) NULL, _IOFBF, FILEBUFF);

This creates a pre-buffer, so an attempt at reading
1 character at a time, doesn't become a time-wasting
disaster. You're paying for a lot of context switches,
if reading one character at a time in any case.

My application code in the file I copied that out of,
was actually working on 512 byte blocks.

    while ( (inbyte=fread(&inbuf, 1, 512, in)) > 0 ) {
        /* Work on blocks of data, 512 bytes at a time */
    }

By doing the I/O that way, sometimes I can get 300MB/sec
using ramdisks for storage (purely for benchmarking
the modified code).

If you attempt to set the file buffer larger than that,
it doesn't always help. You can use system performance
monitoring tools, to see what size is actually
being used. My particular setup, that value
happened to be 64K. If set to 262144, mine was still
using 64K. Any value bigger than 4K is generally
useful and will help a bit. The file system type
obviously plays a part. And the hard drive has limits
on max buffer size for any one command. I fully
expected I could do more than 64K, but it didn't
turn out that way. YMMV.

    Paul

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.c

csiph-web