Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #62086

Re: Mysterious slow IO on fread() to large buffer on large Dell system

Show all headers | View raw

mathog wrote:
> Hi all,
>
> On a big Dell system ( PowerEdge T620, >500Gb RAM, 48 CPUs  Xeon(R) CPU
> E5-2695 v2 @ 2.40GHz, Centos 6.6 ) the following snippet of code is used
> to open and read in a 13.75 Gb file:
>
>         /* error checking removed for brevity */
>         size_t len_file, rlen;
>         int status;
>
>         fin=fopen("filename", "rb");
>         status = fseek(fin, 0L, SEEK_END);
>         len_file = ftell(fin);
>         fseek(fin, 0L, SEEK_SET);
>         buffer=malloc(len_file); // this takes only a few seconds
>         rlen = fread(buffer, 1, len_file, fin); // this takes forever
>
> This takes 15 minutes(!!!), which works out to a "whopping" 15.3Mb/s.
> Nothing unusual in the way it is run, it is just:

Why don't you simply memory map the file?

-- 
Ian Collins

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Mysterious slow IO on fread() to large buffer on large Dell system mathog <dmathog@gmail.com> - 2015-05-15 18:45 -0700
  Re: Mysterious slow IO on fread() to large buffer on large Dell system Ian Collins <ian-news@hotmail.com> - 2015-05-16 14:09 +1200
  Re: Mysterious slow IO on fread() to large buffer on large Dell system Siri Cruz <chine.bleu@yahoo.com> - 2015-05-15 19:20 -0700
    Re: Mysterious slow IO on fread() to large buffer on large Dell system mathog <dmathog@gmail.com> - 2015-05-15 22:29 -0700
      Re: Mysterious slow IO on fread() to large buffer on large Dell system Siri Cruz <chine.bleu@yahoo.com> - 2015-05-16 08:47 -0700
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Robert Wessel <robertwessel2@yahoo.com> - 2015-05-16 02:11 -0500
  Re: Mysterious slow IO on fread() to large buffer on large Dell system Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-05-15 19:23 -0700
  Re: Mysterious slow IO on fread() to large buffer on large Dell system glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-05-16 05:05 +0000
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Melzzzzz <mel@zzzzz.com> - 2015-05-16 07:20 +0200
    Re: Mysterious slow IO on fread() to large buffer on large Dell system mathog <dmathog@gmail.com> - 2015-05-15 22:27 -0700
      Re: Mysterious slow IO on fread() to large buffer on large Dell system Melzzzzz <mel@zzzzz.com> - 2015-05-16 07:59 +0200
      Re: Mysterious slow IO on fread() to large buffer on large Dell system Ian Collins <ian-news@hotmail.com> - 2015-05-16 18:31 +1200
  Re: Mysterious slow IO on fread() to large buffer on large Dell system Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2015-05-16 10:57 -0400
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2015-05-16 10:51 -0600
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-05-16 11:41 -0700
      Re: Mysterious slow IO on fread() to large buffer on large Dell system Melzzzzz <mel@zzzzz.com> - 2015-05-16 21:01 +0200
        Re: Mysterious slow IO on fread() to large buffer on large Dell system glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2015-05-16 19:37 +0000
          Re: Mysterious slow IO on fread() to large buffer on large Dell system Noob <root@127.0.0.1> - 2015-05-16 22:14 +0200
    Re: Mysterious slow IO on fread() to large buffer on large Dell system gamo <gamo@telecable.es> - 2015-05-17 12:36 +0200
  Re: Mysterious slow IO on fread() to large buffer on large Dell system Paul <nospam@needed.com> - 2015-05-16 11:29 -0400
  Re: Mysterious slow IO on fread() to large buffer on large Dell system mathog <dmathog@gmail.com> - 2015-05-16 10:50 -0700
    Re: Mysterious slow IO on fread() to large buffer on large Dell system Johann Klammer <klammerj@NOSPAM.a1.net> - 2015-05-17 01:43 +0200
  Re: Mysterious slow IO on fread() to large buffer on large Dell system Stephen Sprunk <stephen@sprunk.org> - 2015-05-16 16:40 -0500
  Re: Mysterious slow IO on fread() to large buffer on large Dell system mathog <dmathog@gmail.com> - 2015-05-26 09:58 -0700

csiph-web