Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #63909

Re: Python Fast I/o

Show key headers only | View raw

On Wed, Jan 15, 2014 at 1:24 AM, Ayushi Dalmia
<ayushidalmia2604@gmail.com> wrote:
> On Tuesday, January 14, 2014 7:33:08 PM UTC+5:30, Chris Angelico wrote:
>> On Wed, Jan 15, 2014 at 12:50 AM, Ayushi Dalmia
>>
>> <ayushidalmia2604@gmail.com> wrote:
>>
>> > I need to write into a file for a project which will be evaluated on the basis of time. What is the fastest way to write 200 Mb of data, accumulated as a list into a file.
>>
>> >
>>
>> > Presently I am using this:
>>
>> >
>>
>> > with open('index.txt','w') as f:
>>
>> >       f.write("".join(data))
>>
>> >       f.close()
>>
>>
>>
>> with open('index.txt','w') as f:
>>
>>     for hunk in data:
>>
>>         f.write(hunk)
>>
>>
>>
>> You don't need to f.close() - that's what the 'with' block guarantees.
>>
>> Iterating over data and writing each block separately means you don't
>>
>> have to first build up a 200MB string. After that, your performance is
>>
>> going to be mainly tied to the speed of your disk, not anything that
>>
>> Python can affect.
>>
>>
>>
>> ChrisA

Your quoted text is becoming double spaced, because of bugs in the
Google Groups client. Please either edit this before posting, or
switch to a better newsreader, or use the mailing list:

https://mail.python.org/mailman/listinfo/python-list

Thanks!

> Which is more fast?
> Creating a 200 Mb string and then dumping into a file or dividing the 200 Mb string into chunks and then writing those chunks. Won't writing the chunks call more i/o operation?
>

When you're writing two hundred megabytes, the number of I/O
operations is dominated by that. You have to write that many sectors,
and nothing can change that. Joining your list of strings before
writing incurs the cost of building up a single 200MB string, but even
that is likely to be insignificant in the scheme of things (if you
have the memory available, it won't take very long compared to the
time it takes to write to the disk). Python will buffer its writes, so
you don't have to worry about the details. It's going to do the right
thing for you; you can concentrate on making your code look right.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Python Fast I/o Ayushi Dalmia <ayushidalmia2604@gmail.com> - 2014-01-14 05:50 -0800
  Re: Python Fast I/o Chris Angelico <rosuav@gmail.com> - 2014-01-15 01:03 +1100
    Re: Python Fast I/o Ayushi Dalmia <ayushidalmia2604@gmail.com> - 2014-01-14 06:24 -0800
      Re: Python Fast I/o Chris Angelico <rosuav@gmail.com> - 2014-01-15 01:32 +1100
      Re: Python Fast I/o Roy Smith <roy@panix.com> - 2014-01-14 09:40 -0500
  Re: Python Fast I/o Tim Chase <python.list@tim.thechases.com> - 2014-01-14 08:18 -0600

csiph-web