Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.ruby > #4365

Re: Optimize write of large file

Show all headers | View raw

On 5/12/2011 07:58, Yoann M. wrote:
> Hello,
> I have data to process and to write into files progressively. The data
> files are in the end very large, but I append to them small strings. I
> suppose buffering the strings before apending to the file would be
> faster. I don't need the files to be written before the end of the whole
> process (i.e. I don't use their content).
> 
> I've searched for info about how File buffer its data but it seems we
> can not configure anything about this, did I miss something ?
> My first idea was to buffer everything myself, appending lines to a
> string, or an array of strings and write when I reach a big enough
> amount of data. But if File uses a buffer anyway, it would be a waste of
> time I suppose ?
> Do you have any advice to optimize the writing of large files ?

As mentioned, the file writes are already being buffered by lower
layers; however, if you are closing and reopening the files throughout
your processing, the buffers aren't helping you much.  Try to ensure
that you open each file only once and keep those file references around
to use until you know you're permanently done writing to each one.
Unless you have a large number of files to open, you shouldn't have to
worry about resource constraints on the number of concurrently open files.

-Jeremy

Back to comp.lang.ruby | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Optimize write of large file "Yoann M." <yoann6@gmail.com> - 2011-05-12 07:58 -0500
  Re: Optimize write of large file Markus Schirp <mbj@seonic.net> - 2011-05-12 08:17 -0500
  Re: Optimize write of large file Jeremy Bopp <jeremy@bopp.net> - 2011-05-12 09:19 -0500
  Re: Optimize write of large file "Yoann M." <yoann6@gmail.com> - 2011-05-12 10:07 -0500
    Re: Optimize write of large file Robert Klemme <shortcutter@googlemail.com> - 2011-05-12 10:25 -0500
    Re: Optimize write of large file Markus Schirp <mbj@seonic.net> - 2011-05-12 10:51 -0500
  Re: Optimize write of large file "Yoann M." <yoann6@gmail.com> - 2011-05-13 02:21 -0500

csiph-web