Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #45527

Re: How to write fast into a file in python?

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

With CPython 2.7.3:
./t
time taken to write a file of size 52428800  is  15.86 seconds

time taken to write a file of size 52428800  is  7.91 seconds

time taken to write a file of size 52428800  is  9.64 seconds


With pypy-1.9:
./t
time taken to write a file of size 52428800  is  3.708232 seconds

time taken to write a file of size 52428800  is  4.868304 seconds

time taken to write a file of size 52428800  is  1.93612 seconds


Here's the code:
#!/usr/local/pypy-1.9/bin/pypy
#!/usr/bin/python

import sys
import time
import StringIO

sys.path.insert(0, '/usr/local/lib')
import bufsock

def create_file_numbers_old(filename, size):
    start = time.clock()

    value = 0
    with open(filename, "w") as f:
        while f.tell() < size:
            f.write("{0}\n".format(value))
            value += 1

    end = time.clock()

    print "time taken to write a file of size", size, " is ", (end -start),
"seconds \n"

def create_file_numbers_bufsock(filename, intended_size):
    start = time.clock()

    value = 0
    with open(filename, "w") as f:
        bs = bufsock.bufsock(f)
        actual_size = 0
        while actual_size < intended_size:
            string = "{0}\n".format(value)
            actual_size += len(string) + 1
            bs.write(string)
            value += 1
        bs.flush()

    end = time.clock()

    print "time taken to write a file of size", intended_size, " is ", (end
-start), "seconds \n"


def create_file_numbers_file_like(filename, intended_size):
    start = time.clock()

    value = 0
    with open(filename, "w") as f:
        file_like = StringIO.StringIO()
        actual_size = 0
        while actual_size < intended_size:
            string = "{0}\n".format(value)
            actual_size += len(string) + 1
            file_like.write(string)
            value += 1
        file_like.seek(0)
        f.write(file_like.read())

    end = time.clock()

    print "time taken to write a file of size", intended_size, " is ", (end
-start), "seconds \n"

create_file_numbers_old('output.txt', 50 * 2**20)
create_file_numbers_bufsock('output2.txt', 50 * 2**20)
create_file_numbers_file_like('output3.txt', 50 * 2**20)




On Thu, May 16, 2013 at 9:35 PM, <lokeshkoppaka@gmail.com> wrote:

> On Friday, May 17, 2013 8:50:26 AM UTC+5:30, lokesh...@gmail.com wrote:
> > I need to write numbers into a file upto 50mb and it should be fast
> >
> > can any one help me how to do that?
> >
> > i had written the following code..
> >
> >
> -----------------------------------------------------------------------------------------------------------
> >
> > def create_file_numbers_old(filename, size):
> >
> > start = time.clock()
> >
> >
> >
> > value = 0
> >
> > with open(filename, "w") as f:
> >
> > while f.tell()< size:
> >
> > f.write("{0}\n".format(value))
> >
> > value += 1
> >
> >
> >
> > end = time.clock()
> >
> >
> >
> > print "time taken to write a file of size", size, " is ", (end -start),
> "seconds \n"
> >
> >
> ------------------------------------------------------------------------------------------------------------------
> >
> > it takes about 20sec i need 5 to 10 times less than that.
> size = 50mb
> --
> http://mail.python.org/mailman/listinfo/python-list
>

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

How to write fast into a file in python? lokeshkoppaka@gmail.com - 2013-05-16 20:20 -0700
  Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-17 03:51 +0000
  Re: How to write fast into a file in python? lokeshkoppaka@gmail.com - 2013-05-16 21:35 -0700
    Re: How to write fast into a file in python? Dave Angel <davea@davea.name> - 2013-05-17 07:58 -0400
    RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 18:20 +0300
      Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-17 16:42 +0000
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 20:25 +0300
      Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-17 17:47 +0000
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 21:18 +0300
          Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-18 04:01 +0000
            Re: How to write fast into a file in python? Chris Angelico <rosuav@gmail.com> - 2013-05-18 15:28 +1000
            Re: How to write fast into a file in python? 88888 Dihedral <dihedral88888@googlemail.com> - 2013-05-18 04:09 -0700
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 21:33 +0300
        RE: How to write fast into a file in python? Fábio Santos <fabiosantosart@gmail.com> - 2013-05-18 08:49 +0100
        Re: How to write fast into a file in python? Chris Angelico <rosuav@gmail.com> - 2013-05-19 00:29 +1000
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-18 20:00 +0300
          Re: How to write fast into a file in python? Tim Roberts <timr@probo.com> - 2013-05-19 19:04 -0700
        Re: How to write fast into a file in python? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-05-18 15:14 -0400
          Re: How to write fast into a file in python? Roy Smith <roy@panix.com> - 2013-05-18 15:37 -0400
          Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-18 22:23 +0000
        Re: How to write fast into a file in python? Fábio Santos <fabiosantosart@gmail.com> - 2013-05-18 22:19 +0100
        Re: How to write fast into a file in python? Dave Angel <davea@davea.name> - 2013-05-18 22:41 -0400
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 06:53 +0300
        Re: How to write fast into a file in python? MRAB <python@mrabarnett.plus.com> - 2013-05-19 16:44 +0100
        RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-20 13:34 +0300
    Re: How to write fast into a file in python? Dan Stromberg <drsalists@gmail.com> - 2013-05-18 12:38 -0700
    RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 08:31 +0300
    RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 08:42 +0300
    Re: How to write fast into a file in python? Chris Angelico <rosuav@gmail.com> - 2013-05-19 19:21 +1000
    RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 12:41 +0300

csiph-web