Optimize write of large file

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!feeds.phibee-telecom.net!talisker.lacave.net!lacave.net!not-for-mail
From	"Yoann M." <yoann6@gmail.com>
Newsgroups	comp.lang.ruby
Subject	Optimize write of large file
Date	Thu, 12 May 2011 07:58:36 -0500
Organization	Service de news de lacave.net
Lines	19
Message-ID	<f93da777afad73da30f77526cdcab9ee@ruby-forum.com> (permalink)
NNTP-Posting-Host	bristol.highgroove.com
Content-Type	text/plain; charset=UTF-8
Content-Transfer-Encoding	7bit
X-Trace	talisker.lacave.net 1305205469 57786 65.111.164.187 (12 May 2011 13:04:29 GMT)
X-Complaints-To	abuse@lacave.net
NNTP-Posting-Date	Thu, 12 May 2011 13:04:29 +0000 (UTC)
X-Received-From	This message has been automatically forwarded from the ruby-talk mailing list by a gateway at comp.lang.ruby. If it is SPAM, it did not originate at comp.lang.ruby. Please report the original sender, and not us. Thanks! For more details about this gateway, please visit: http://blog.grayproductions.net/categories/the_gateway
X-Mail-Count	383059
X-Ml-Name	ruby-talk
X-Rubymirror	Yes
X-Ruby-Talk	<f93da777afad73da30f77526cdcab9ee@ruby-forum.com>
Xref	x330-a1.tempe.blueboxinc.net comp.lang.ruby:4352

Show key headers only | View raw

Hello,
I have data to process and to write into files progressively. The data
files are in the end very large, but I append to them small strings. I
suppose buffering the strings before apending to the file would be
faster. I don't need the files to be written before the end of the whole
process (i.e. I don't use their content).

I've searched for info about how File buffer its data but it seems we
can not configure anything about this, did I miss something ?
My first idea was to buffer everything myself, appending lines to a
string, or an array of strings and write when I reach a big enough
amount of data. But if File uses a buffer anyway, it would be a waste of
time I suppose ?
Do you have any advice to optimize the writing of large files ?
Thanks !

-- 
Posted via http://www.ruby-forum.com/.

Back to comp.lang.ruby | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Optimize write of large file "Yoann M." <yoann6@gmail.com> - 2011-05-12 07:58 -0500
  Re: Optimize write of large file Markus Schirp <mbj@seonic.net> - 2011-05-12 08:17 -0500
  Re: Optimize write of large file Jeremy Bopp <jeremy@bopp.net> - 2011-05-12 09:19 -0500
  Re: Optimize write of large file "Yoann M." <yoann6@gmail.com> - 2011-05-12 10:07 -0500
    Re: Optimize write of large file Robert Klemme <shortcutter@googlemail.com> - 2011-05-12 10:25 -0500
    Re: Optimize write of large file Markus Schirp <mbj@seonic.net> - 2011-05-12 10:51 -0500
  Re: Optimize write of large file "Yoann M." <yoann6@gmail.com> - 2011-05-13 02:21 -0500

csiph-web