Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.ruby > #4381

Re: Optimize write of large file

Show all headers | View raw

IMHO the primary speed bottleneck is the disk drive itself and the "possible"
File-System fragmentation.

RAM just let the operating system do the writes "as optimal as
possible". The effect of drastically more ram wont be more than 1-5%.

When you use a ramdisk this will differ much ;)

But when you are worried about file persistence you should not do this
*g*

I do not knew any details about your use case, there are other
possiblities:
* writing direkt to the block device, bypassing file systems
* mirror ramdisk writes to other machines for persistence
* ?

On Fri, May 13, 2011 at 12:07:30AM +0900, Yoann M. wrote:
> You're right, doing the buffer myself does not make it faster. For
> writing 10 millions lines, with an array of strings, one string, and no
> homemade-buffer (code is attached) :
> Buffer array : 11.141s
> Buffer string : 9.748s
> No buffer : 10.344s
> 
> Don't you think using more RAM before writing on disk could make the
> process faster ? I thought so, then I'd like to say to File how much RAM
> it can uses to speed things up, because I can use a lot of RAM.
> 
> Regards
> 
> Attachments:
> http://www.ruby-forum.com/attachment/6191/test_write.rb
> 
> 
> -- 
> Posted via http://www.ruby-forum.com/.
> 

-- 
Markus Schirp
Phone:  049 201 / 647 59 63
Mobile: 049 178 / 529 91 42
Web:    www.seonic.net
Email:  info@seonic.net
Seonic IT-Systems GbR
Anton Shatalov & Markus Schirp
Walterhohmannstraße 1
D-45141 Essen

Back to comp.lang.ruby | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Optimize write of large file "Yoann M." <yoann6@gmail.com> - 2011-05-12 07:58 -0500
  Re: Optimize write of large file Markus Schirp <mbj@seonic.net> - 2011-05-12 08:17 -0500
  Re: Optimize write of large file Jeremy Bopp <jeremy@bopp.net> - 2011-05-12 09:19 -0500
  Re: Optimize write of large file "Yoann M." <yoann6@gmail.com> - 2011-05-12 10:07 -0500
    Re: Optimize write of large file Robert Klemme <shortcutter@googlemail.com> - 2011-05-12 10:25 -0500
    Re: Optimize write of large file Markus Schirp <mbj@seonic.net> - 2011-05-12 10:51 -0500
  Re: Optimize write of large file "Yoann M." <yoann6@gmail.com> - 2011-05-13 02:21 -0500

csiph-web