Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #10311 > unrolled thread

Best way to write the vector content to a file.

Back to article view | Back to comp.lang.c++

Contents

  Best way to write the vector content to a file. sravanreddy001 <sravanreddy001@gmail.com> - 2011-09-19 20:05 -0700
    Re: Best way to write the vector content to a file. Ian Collins <ian-news@hotmail.com> - 2011-09-20 15:12 +1200
    Re: Best way to write the vector content to a file. sravanreddy001 <sravanreddy001@gmail.com> - 2011-09-19 20:28 -0700
      Re: Best way to write the vector content to a file. Ian Collins <ian-news@hotmail.com> - 2011-09-20 15:37 +1200
        Re: Best way to write the vector content to a file. sravanreddy001 <sravanreddy001@gmail.com> - 2011-09-19 20:50 -0700
          Re: Best way to write the vector content to a file. Ian Collins <ian-news@hotmail.com> - 2011-09-20 16:03 +1200
        Re: Best way to write the vector content to a file. sravanreddy001 <sravanreddy001@gmail.com> - 2011-09-19 20:47 -0700
          Re: Best way to write the vector content to a file. Ian Collins <ian-news@hotmail.com> - 2011-09-20 16:09 +1200
      Re: Best way to write the vector content to a file. sravanreddy001 <sravanreddy001@gmail.com> - 2011-09-19 20:29 -0700
    Re: Best way to write the vector content to a file. Juha Nieminen <nospam@thanks.invalid> - 2011-09-20 06:44 +0000
    Re: Best way to write the vector content to a file. Goran <goran.pusic@gmail.com> - 2011-09-20 00:37 -0700
      Re: Best way to write the vector content to a file. Ian Collins <ian-news@hotmail.com> - 2011-09-20 19:45 +1200
        Re: Best way to write the vector content to a file. Goran <goran.pusic@gmail.com> - 2011-09-20 02:01 -0700
    Re: Best way to write the vector content to a file. Rui Maciel <rui.maciel@gmail.com> - 2011-09-20 10:56 +0100
    Re: Best way to write the vector content to a file. Fulvio Esposito <esposito.fulvio@gmail.com> - 2011-09-20 04:03 -0700
    Re: Best way to write the vector content to a file. sravanreddy001 <sravanreddy001@gmail.com> - 2011-09-20 05:00 -0700
      Re: Best way to write the vector content to a file. Goran <goran.pusic@gmail.com> - 2011-09-20 06:55 -0700
      Re: Best way to write the vector content to a file. Paavo Helde <myfirstname@osa.pri.ee> - 2011-09-20 13:39 -0500
    Re: Best way to write the vector content to a file. sravanreddy001 <sravanreddy001@gmail.com> - 2011-09-20 17:50 -0700

#10311 — Best way to write the vector content to a file.

Hi,

Can anyone give the best way (faster) to write the vector contents to a file?

Is there a better way to do it than below method?

        for(unsigned i=0;i<terms.size();i++){
		sprintf(temp,"%d %s\n",i,terms[i].c_str());
 		ids_n_terms = ids_n_terms + temp;
	}

Is the string addition a very expensive operation? (will a new memory allocation takes place)

there are around 5,00,000 (strings) in the vector, and these have to stored in 
<id term> format.

Thanks,
Sravan.

[toc] | [next] | [standalone]

#10312

On 09/20/11 03:05 PM, sravanreddy001 wrote:
> Hi,
>
> Can anyone give the best way (faster) to write the vector contents to a file?
>
> Is there a better way to do it than below method?
>
>          for(unsigned i=0;i<terms.size();i++){
> 		sprintf(temp,"%d %s\n",i,terms[i].c_str());
>   		ids_n_terms = ids_n_terms + temp;
> 	}

Where do you do the write?

> Is the string addition a very expensive operation? (will a new memory allocation takes place)
>
> there are around 5,00,000 (strings) in the vector, and these have to stored in
> <id term>  format.

You will probably be I/O limited, so optimising your code won't make any 
significant difference.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10313

I'm actually writing this to a file.
But, file operation is just one simple write.
(the string that is constructed in the loop is written.. and the file size is around 5 MB.)

but.. string+ = string + some_thing;

if this is going to allocate a new memory location, then.. writing it multiple times to file.. will any of this approach help?

One my i3 2nd gen computer.. it took around 10 mins. (using one single core)
not multithreaded.

[toc] | [prev] | [next] | [standalone]

#10314

On 09/20/11 03:28 PM, sravanreddy001 wrote:

Context is nice!

> I'm actually writing this to a file.
> But, file operation is just one simple write.
> (the string that is constructed in the loop is written.. and the file size is around 5 MB.)
>
> but.. string+ = string + some_thing;
>
> if this is going to allocate a new memory location, then.. writing it multiple times to file.. will any of this approach help?
>
> One my i3 2nd gen computer.. it took around 10 mins. (using one single core)
> not multithreaded.

You would probably be better off just using an ofstream and writing a 
line at a time.  I have one application that reads and writes in excess 
of 1GB with fstreams.  Reading takes a couple of minutes, writing about 5.

Your huge string will do a lot of progressively bigger memory moves, 
which are a bigger issue than the allocations.  Kind of a 
macro-deoptimisation!

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10316

Also, I'm reading all the file content in one GO.

seeking until the end of file.. read from start to end.
this is efficient than reading line by line right?

[toc] | [prev] | [next] | [standalone]

#10318

On 09/20/11 03:50 PM, sravanreddy001 wrote:
> Also, I'm reading all the file content in one GO.

Please keep the context from the post you are relying to, otherwise your 
posts don't make sense.

> seeking until the end of file.. read from start to end.
> this is efficient than reading line by line right?

You are worrying about the wrong thing, just create an ifstream and read 
the entries in one at a time.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10317

do you think, writing so many times... 5 lakhs times(lines) is more expencive... more disk writes.. thats why i came to that approach..

so.. will wiriting to file every 20-30 lines at once.. will it help me?
it will atleast reduce the disk writes by 20-30 times..

[toc] | [prev] | [next] | [standalone]

#10319

On 09/20/11 03:47 PM, sravanreddy001 wrote:
> do you think, writing so many times... 5 lakhs times(lines) is more expencive... more disk writes.. thats why i came to that approach..
>
> so.. will wiriting to file every 20-30 lines at once.. will it help me?
> it will atleast reduce the disk writes by 20-30 times..

Leave the buffering to the stream and the OS, it's their job.  Unless 
you flush the stream each line and force synchronous writes on the file, 
you will not see millions of writes.  On my system, that much data would 
be coalesced into one disk write!

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10315

Is there a way to automatically push the vector contents to file in an efficient manner?

[toc] | [prev] | [next] | [standalone]

#10323

sravanreddy001 <sravanreddy001@gmail.com> wrote:
>        for(unsigned i=0;i<terms.size();i++){
>                sprintf(temp,"%d %s\n",i,terms[i].c_str());
>                ids_n_terms = ids_n_terms + temp;
>        }

  I don't understand why you can't write to the file directly. Creating
the contents of the file into a dynamic string first like that is
significantly more inefficient.

[toc] | [prev] | [next] | [standalone]

#10325

On Sep 20, 5:05 am, sravanreddy001 <sravanreddy...@gmail.com> wrote:
> Hi,
>
> Can anyone give the best way (faster) to write the vector contents to a file?
>
> Is there a better way to do it than below method?
>
>         for(unsigned i=0;i<terms.size();i++){
>                 sprintf(temp,"%d %s\n",i,terms[i].c_str());
>                 ids_n_terms = ids_n_terms + temp;
>         }
>
> Is the string addition a very expensive operation? (will a new memory allocation takes place)

Well, yes. The above loop is bound to get very expensive, because you
want to append. That means, from time to time (possibly at every
append!), "allocate more space, copy old string there, append new
content, free old space".

You should instead just use ofstream, and play with it's rdbuf-
>pubsetbuf to find a sweet spot for speed (depends on the system, but
reasonable buffer sizes like 1K-8K (1024 to 8192) are a good guess). I
wouldn't even be surprised if you find that the default works just as
well. E.g.

ofstream f(whatever);
char buffer[try sizes here]; // speed to be gained here.
f.rdbuf->pubsetbuf(buffer, sizeof(buffer));

your loop here, but instead of sprintf and +, do

   f << i << " " << terms[i] << endl;

That does conversion to text for you.

Goran.

[toc] | [prev] | [next] | [standalone]

#10326

On 09/20/11 07:37 PM, Goran wrote:
> On Sep 20, 5:05 am, sravanreddy001<sravanreddy...@gmail.com>  wrote:
>> Hi,
>>
>> Can anyone give the best way (faster) to write the vector contents to a file?
>>
>> Is there a better way to do it than below method?
>>
>>          for(unsigned i=0;i<terms.size();i++){
>>                  sprintf(temp,"%d %s\n",i,terms[i].c_str());
>>                  ids_n_terms = ids_n_terms + temp;
>>          }
>>
>> Is the string addition a very expensive operation? (will a new memory allocation takes place)
>
> Well, yes. The above loop is bound to get very expensive, because you
> want to append. That means, from time to time (possibly at every
> append!), "allocate more space, copy old string there, append new
> content, free old space".
>
> You should instead just use ofstream, and play with it's rdbuf-
>> pubsetbuf to find a sweet spot for speed (depends on the system, but
> reasonable buffer sizes like 1K-8K (1024 to 8192) are a good guess). I
> wouldn't even be surprised if you find that the default works just as
> well. E.g.
>
> ofstream f(whatever);
> char buffer[try sizes here]; // speed to be gained here.
> f.rdbuf->pubsetbuf(buffer, sizeof(buffer));
>
> your loop here, but instead of sprintf and +, do
>
>     f<<  i<<  " "<<  terms[i]<<  endl;

Flushing after each line rather defeats the effort of fiddling with the 
ofstream's buffer!

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10330

On Sep 20, 9:45 am, Ian Collins <ian-n...@hotmail.com> wrote:
> On 09/20/11 07:37 PM, Goran wrote:
>
>
>
>
>
>
>
>
>
> > On Sep 20, 5:05 am, sravanreddy001<sravanreddy...@gmail.com>  wrote:
> >> Hi,
>
> >> Can anyone give the best way (faster) to write the vector contents to a file?
>
> >> Is there a better way to do it than below method?
>
> >>          for(unsigned i=0;i<terms.size();i++){
> >>                  sprintf(temp,"%d %s\n",i,terms[i].c_str());
> >>                  ids_n_terms = ids_n_terms + temp;
> >>          }
>
> >> Is the string addition a very expensive operation? (will a new memory allocation takes place)
>
> > Well, yes. The above loop is bound to get very expensive, because you
> > want to append. That means, from time to time (possibly at every
> > append!), "allocate more space, copy old string there, append new
> > content, free old space".
>
> > You should instead just use ofstream, and play with it's rdbuf-
> >> pubsetbuf to find a sweet spot for speed (depends on the system, but
> > reasonable buffer sizes like 1K-8K (1024 to 8192) are a good guess). I
> > wouldn't even be surprised if you find that the default works just as
> > well. E.g.
>
> > ofstream f(whatever);
> > char buffer[try sizes here]; // speed to be gained here.
> > f.rdbuf->pubsetbuf(buffer, sizeof(buffer));
>
> > your loop here, but instead of sprintf and +, do
>
> >     f<<  i<<  " "<<  terms[i]<<  endl;
>
> Flushing after each line rather defeats the effort of fiddling with the
> ofstream's buffer!

Whoops! (blushes)

Goran.

[toc] | [prev] | [next] | [standalone]

#10331

sravanreddy001 wrote:

> Hi,
> 
> Can anyone give the best way (faster) to write the vector contents to a
> file?
> 
> Is there a better way to do it than below method?
> 
>         for(unsigned i=0;i<terms.size();i++){
> sprintf(temp,"%d %s\n",i,terms[i].c_str());
>  ids_n_terms = ids_n_terms + temp;
> }
> 
> Is the string addition a very expensive operation? (will a new memory
> allocation takes place)

Before this loop, have you allocated any memory to your temp c-string?



Rui Maciel

[toc] | [prev] | [next] | [standalone]

#10332

On Tuesday, September 20, 2011 5:05:44 AM UTC+2, sravanreddy001 wrote:
> Hi,
> 
> Can anyone give the best way (faster) to write the vector contents to a file?
> 
> Is there a better way to do it than below method?
> 
>         for(unsigned i=0;i<terms.size();i++){
> 		sprintf(temp,"%d %s\n",i,terms[i].c_str());
>  		ids_n_terms = ids_n_terms + temp;
> 	}
> 
> Is the string addition a very expensive operation? (will a new memory allocation takes place)
> 
> there are around 5,00,000 (strings) in the vector, and these have to stored in 
> <id term> format.
> 
> Thanks,
> Sravan.

if you use standard file stream classes, you always get buffered I/O. For non-trivial implementation, buffer size is choosen as what is best for OS you are running on! Moreover, if the file is on a network filesystem, the single call can be broken in pieces to be sent over the network. 

It always depends on your context. There's no general solution aside from let standard library and OS do what they know how to do best!

Cheers,
Fulvio Esposito

[toc] | [prev] | [next] | [standalone]

#10333

Wow...

<Special>Fulvio Esposito, Ian Collins, Goran, Juha Nieminen:</Special>
Thanks a lot for the valuable inputs. This is really content that u have provided.

I've another question. I'm reading all the file in a single go, instead of line by line.

This is efficient then other scenario's right?
each file is around 5kb - 10kb.

@Did any one worked on creating forward indexes. (Info retrieval.)

[toc] | [prev] | [next] | [standalone]

#10336

On Sep 20, 2:00 pm, sravanreddy001 <sravanreddy...@gmail.com> wrote:
> Wow...
>
> <Special>Fulvio Esposito, Ian Collins, Goran, Juha Nieminen:</Special>
> Thanks a lot for the valuable inputs. This is really content that u have provided.
>
> I've another question. I'm reading all the file in a single go, instead of line by line.
>
> This is efficient then other scenario's right?
> each file is around 5kb - 10kb.

You absolutely must measure to know.

That said, depending on what you're doing with resulting string, even
this is probably not as efficient as simply reading off the stream and
"converting" into vector. It's not efficient because you need to read
all data, then you copy^^^ it into a string, then you parse that
string. If you simply read into your structure, you avoid the copy^^^.
Reading itself, due to all sorts of buffering that are happening
behind your back, won't be faster either way.

Goran.

[toc] | [prev] | [next] | [standalone]

#10344

sravanreddy001 <sravanreddy001@gmail.com> wrote in
news:27708310.2161.1316520026653.JavaMail.geo-discussion-forums@yqog21: 

> Wow...
> 
> <Special>Fulvio Esposito, Ian Collins, Goran, Juha Nieminen:</Special>
> Thanks a lot for the valuable inputs. This is really content that u
> have provided. 
> 
> I've another question. I'm reading all the file in a single go,
> instead of line by line. 
> 
> This is efficient then other scenario's right?
> each file is around 5kb - 10kb.

In principle reading in one go is more efficient with such small files as 
less work is done, and the whole file content will probably fit in the 
cpu caches. However, the only way to find out is to measure. Profile your 
code to find bottlenecks, then deal with them. Do not attempt to guess 
that the file reading is bottleneck (may well be and may not, the only 
way to find out is to measure). For example, if the conversion of the 
text format data takes 10 times more time than file reading, then it does 
not really matter how do you read the file.

Note that if the file reading is a bottleneck, installing a faster disk 
or more RAM (for file caching) may fix the problem.

Nevertheless, AFAIK the fastest way to get the contents of file into your 
process is to use memory mapping. However, this is platform-specific and 
not standardized, and I'm not sure it makes any real difference with such 
small files. No need to take this route without a good reason.

hth
Paavo

[toc] | [prev] | [next] | [standalone]

#10354

Hi all,
thanks a lot for the valuable inputs.

I followed your(all of you) guidelines..

now.. its taking just 2 seconds.. against initial 10 minutes.

Thanks a lot. :)

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c++

csiph-web