Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #45550

RE: How to write fast into a file in python?

Show key headers only | View raw

BTW, I've downloaded from the following places:

http://stromberg.dnsalias.org/svn/bufsock/trunk/bufsock.py
http://stromberg.dnsalias.org/~dstromberg/backshift/documentation/html/python2x3-pysrc.html

Are those the latest versions?

----------------------------------------
> From: carlosnepomuceno@outlook.com
> To: python-list@python.org
> Subject: RE: How to write fast into a file in python?
> Date: Sun, 19 May 2013 08:31:08 +0300
> CC: lokeshkoppaka@gmail.com
>
> Thanks Dan! I've never used CPython or PyPy. Will try them later.
>
> I think the main difference between your create_file_numbers_file_like()
> and the fastwrite5.py I sent earlier is that I've used cStringIO
> instead of StringIO. It took 12s less using cStringIO.
>
> My numbers are much greater, but I've used Python 2.7.5 instead:
>
> C:\src\Python>python create_file_numbers.py
> time taken to write a file of size 52428800  is  39.1199457743 seconds
>
> time taken to write a file of size 52428800  is  14.8704800436 seconds
>
> time taken to write a file of size 52428800  is  23.0011990985 seconds
>
>
> I've downloaded bufsock.py and python2x3.py. The later one was hard to remove the source code from the web page.
>
> Can I use them on my projects? I'm not used to the UCI license[1]. What's the difference to the GPL?
>
>
>
>
> [1] http://stromberg.dnsalias.org/~dstromberg/UCI-license.html
>
> ________________________________
>> Date: Sat, 18 May 2013 12:38:30 -0700
>> Subject: Re: How to write fast into a file in python?
>> From: drsalists@gmail.com
>> To: lokeshkoppaka@gmail.com
>> CC: python-list@python.org
>>
>>
>> With CPython 2.7.3:
>> ./t
>> time taken to write a file of size 52428800 is 15.86 seconds
>>
>> time taken to write a file of size 52428800 is 7.91 seconds
>>
>> time taken to write a file of size 52428800 is 9.64 seconds
>>
>>
>> With pypy-1.9:
>> ./t
>> time taken to write a file of size 52428800 is 3.708232 seconds
>>
>> time taken to write a file of size 52428800 is 4.868304 seconds
>>
>> time taken to write a file of size 52428800 is 1.93612 seconds
>>
>
>> Here's the code:
>> #!/usr/local/pypy-1.9/bin/pypy
>> #!/usr/bin/python
>>
>> import sys
>> import time
>> import StringIO
>>
>> sys.path.insert(0, '/usr/local/lib')
>> import bufsock
>>
>> def create_file_numbers_old(filename, size):
>> start = time.clock()
>>
>> value = 0
>> with open(filename, "w") as f:
>> while f.tell() < size:
>> f.write("{0}\n".format(value))
>> value += 1
>>
>> end = time.clock()
>>
>> print "time taken to write a file of size", size, " is ", (end
>> -start), "seconds \n"
>>
>> def create_file_numbers_bufsock(filename, intended_size):
>> start = time.clock()
>>
>> value = 0
>> with open(filename, "w") as f:
>> bs = bufsock.bufsock(f)
>> actual_size = 0
>> while actual_size < intended_size:
>> string = "{0}\n".format(value)
>> actual_size += len(string) + 1
>> bs.write(string)
>> value += 1
>> bs.flush()
>>
>> end = time.clock()
>>
>> print "time taken to write a file of size", intended_size, " is ",
>> (end -start), "seconds \n"
>>
>>
>> def create_file_numbers_file_like(filename, intended_size):
>> start = time.clock()
>>
>> value = 0
>> with open(filename, "w") as f:
>> file_like = StringIO.StringIO()
>> actual_size = 0
>> while actual_size < intended_size:
>> string = "{0}\n".format(value)
>> actual_size += len(string) + 1
>> file_like.write(string)
>> value += 1
>> file_like.seek(0)
>> f.write(file_like.read())
>>
>> end = time.clock()
>>
>> print "time taken to write a file of size", intended_size, " is ",
>> (end -start), "seconds \n"
>>
>> create_file_numbers_old('output.txt', 50 * 2**20)
>> create_file_numbers_bufsock('output2.txt', 50 * 2**20)
>> create_file_numbers_file_like('output3.txt', 50 * 2**20)
>>
>>
>>
>>
>> On Thu, May 16, 2013 at 9:35 PM,
>> <lokeshkoppaka@gmail.com<mailto:lokeshkoppaka@gmail.com>> wrote:
>> On Friday, May 17, 2013 8:50:26 AM UTC+5:30,
>> lokesh...@gmail.com<mailto:lokesh...@gmail.com> wrote:
>>> I need to write numbers into a file upto 50mb and it should be fast
>>>
>>> can any one help me how to do that?
>>>
>>> i had written the following code..
>>>
>>>
>> -----------------------------------------------------------------------------------------------------------
>>>
>>> def create_file_numbers_old(filename, size):
>>>
>>> start = time.clock()
>>>
>>>
>>>
>>> value = 0
>>>
>>> with open(filename, "w") as f:
>>>
>>> while f.tell()< size:
>>>
>>> f.write("{0}\n".format(value))
>>>
>>> value += 1
>>>
>>>
>>>
>>> end = time.clock()
>>>
>>>
>>>
>>> print "time taken to write a file of size", size, " is ", (end
>> -start), "seconds \n"
>>>
>>>
>> ------------------------------------------------------------------------------------------------------------------
>>>
>>> it takes about 20sec i need 5 to 10 times less than that.
>> size = 50mb
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>>
>>
>> -- http://mail.python.org/mailman/listinfo/python-list
> --
> http://mail.python.org/mailman/listinfo/python-list 		 	   		  

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

How to write fast into a file in python? lokeshkoppaka@gmail.com - 2013-05-16 20:20 -0700
  Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-17 03:51 +0000
  Re: How to write fast into a file in python? lokeshkoppaka@gmail.com - 2013-05-16 21:35 -0700
    Re: How to write fast into a file in python? Dave Angel <davea@davea.name> - 2013-05-17 07:58 -0400
    RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 18:20 +0300
      Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-17 16:42 +0000
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 20:25 +0300
      Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-17 17:47 +0000
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 21:18 +0300
          Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-18 04:01 +0000
            Re: How to write fast into a file in python? Chris Angelico <rosuav@gmail.com> - 2013-05-18 15:28 +1000
            Re: How to write fast into a file in python? 88888 Dihedral <dihedral88888@googlemail.com> - 2013-05-18 04:09 -0700
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 21:33 +0300
        RE: How to write fast into a file in python? Fábio Santos <fabiosantosart@gmail.com> - 2013-05-18 08:49 +0100
        Re: How to write fast into a file in python? Chris Angelico <rosuav@gmail.com> - 2013-05-19 00:29 +1000
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-18 20:00 +0300
          Re: How to write fast into a file in python? Tim Roberts <timr@probo.com> - 2013-05-19 19:04 -0700
        Re: How to write fast into a file in python? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-05-18 15:14 -0400
          Re: How to write fast into a file in python? Roy Smith <roy@panix.com> - 2013-05-18 15:37 -0400
          Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-18 22:23 +0000
        Re: How to write fast into a file in python? Fábio Santos <fabiosantosart@gmail.com> - 2013-05-18 22:19 +0100
        Re: How to write fast into a file in python? Dave Angel <davea@davea.name> - 2013-05-18 22:41 -0400
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 06:53 +0300
        Re: How to write fast into a file in python? MRAB <python@mrabarnett.plus.com> - 2013-05-19 16:44 +0100
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-20 13:34 +0300
    Re: How to write fast into a file in python? Dan Stromberg <drsalists@gmail.com> - 2013-05-18 12:38 -0700
    RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 08:31 +0300
    RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 08:42 +0300
    Re: How to write fast into a file in python? Chris Angelico <rosuav@gmail.com> - 2013-05-19 19:21 +1000
    RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 12:41 +0300

csiph-web