Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #45465 > unrolled thread

How to write fast into a file in python?

Started bylokeshkoppaka@gmail.com
First post2013-05-16 20:20 -0700
Last post2013-05-19 12:41 +0300
Articles 10 on this page of 30 — 12 participants

Back to article view | Back to comp.lang.python


Contents

  How to write fast into a file in python? lokeshkoppaka@gmail.com - 2013-05-16 20:20 -0700
    Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-17 03:51 +0000
    Re: How to write fast into a file in python? lokeshkoppaka@gmail.com - 2013-05-16 21:35 -0700
      Re: How to write fast into a file in python? Dave Angel <davea@davea.name> - 2013-05-17 07:58 -0400
      RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 18:20 +0300
        Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-17 16:42 +0000
          RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 20:25 +0300
        Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-17 17:47 +0000
          RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 21:18 +0300
            Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-18 04:01 +0000
              Re: How to write fast into a file in python? Chris Angelico <rosuav@gmail.com> - 2013-05-18 15:28 +1000
              Re: How to write fast into a file in python? 88888 Dihedral <dihedral88888@googlemail.com> - 2013-05-18 04:09 -0700
          RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-17 21:33 +0300
          RE: How to write fast into a file in python? Fábio Santos <fabiosantosart@gmail.com> - 2013-05-18 08:49 +0100
          Re: How to write fast into a file in python? Chris Angelico <rosuav@gmail.com> - 2013-05-19 00:29 +1000
          RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-18 20:00 +0300
            Re: How to write fast into a file in python? Tim Roberts <timr@probo.com> - 2013-05-19 19:04 -0700
          Re: How to write fast into a file in python? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-05-18 15:14 -0400
            Re: How to write fast into a file in python? Roy Smith <roy@panix.com> - 2013-05-18 15:37 -0400
            Re: How to write fast into a file in python? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-05-18 22:23 +0000
          Re: How to write fast into a file in python? Fábio Santos <fabiosantosart@gmail.com> - 2013-05-18 22:19 +0100
          Re: How to write fast into a file in python? Dave Angel <davea@davea.name> - 2013-05-18 22:41 -0400
          RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 06:53 +0300
          Re: How to write fast into a file in python? MRAB <python@mrabarnett.plus.com> - 2013-05-19 16:44 +0100
          RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-20 13:34 +0300
      Re: How to write fast into a file in python? Dan Stromberg <drsalists@gmail.com> - 2013-05-18 12:38 -0700
      RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 08:31 +0300
      RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 08:42 +0300
      Re: How to write fast into a file in python? Chris Angelico <rosuav@gmail.com> - 2013-05-19 19:21 +1000
      RE: How to write fast into a file in python? Carlos Nepomuceno <carlosnepomuceno@outlook.com> - 2013-05-19 12:41 +0300

Page 2 of 2 — ← Prev page 1 [2]


#45531

FromFábio Santos <fabiosantosart@gmail.com>
Date2013-05-18 22:19 +0100
Message-ID<mailman.1820.1368912002.3114.python-list@python.org>
In reply to#45481
On 18 May 2013 20:19, "Dennis Lee Bieber" <wlfraed@ix.netcom.com> wrote:
>
> tOn Sat, 18 May 2013 08:49:55 +0100, Fábio Santos
> <fabiosantosart@gmail.com> declaimed the following in
> gmane.comp.python.general:
>
>
> > You mentioned "\n" translating to two lines, but this won't happen. Windows
> > will not mess with what you write to your file. It's just that
> > traditionally windows and windows programs use \r\n instead of just \n. I
> > think it was for compatibility with os/2 or macintosh (I don't remember
> > which), which used \r for newlines.
> >
>         Neither... It goes back to Teletype machines where one sent a
> carriage return to move the printhead back to the left, then sent a line
> feed to advance the paper (while the head was still moving left), and in
> some cases also provided a rub-out character (a do-nothing) to add an
> additional character time delay.
>
>         TRS-80 Mod 1-4 used <cr> for "new line", I believe Apple used <lf>
> for "new line"... And both lost the ability to move down the page
> without also resetting the carriage to the left. In a world where both
> <cr><lf> is used, one could draw a vertical line of | by just spacing
> across the first line, printing |, then repeat <lf><bkspc>| until done.
> To do the same with conventional <lf> is "new line/return" one has to
> transmit all those spaces for each line...
>
>         At 300baud, that took time....



On Sat, May 18, 2013 at 6:00 PM, Carlos Nepomuceno
<carlosnepomuceno@outlook.com> wrote:
> Python really writes '\n\r' on Windows. Just check the files.
>
> Internal representations only keep '\n' for simplicity, but if you wanna keep track of the file length you have to take that into account. ;)


On Sat, May 18, 2013 at 3:29 PM, Chris Angelico <rosuav@gmail.com> wrote:
> Into two characters, not two lines, but yes. A file opened in text
> mode on Windows will have its lines terminated with two characters.
> (And it's old Macs that used to use \r. OS/2 follows the DOS
> convention of \r\n, but again, many apps these days are happy with
> Unix newlines there too.)
>
> ChrisA

Thanks for your corrections and explanations. I stand corrected and
have learned something.

[toc] | [prev] | [next] | [standalone]


#45544

FromDave Angel <davea@davea.name>
Date2013-05-18 22:41 -0400
Message-ID<mailman.1829.1368931296.3114.python-list@python.org>
In reply to#45481
On 05/18/2013 01:00 PM, Carlos Nepomuceno wrote:
> Python really writes '\n\r' on Windows. Just check the files.

That's backwards.  '\r\n' on Windows, IF you omit the b in the mode when 
creating the file.



-- 
DaveA

[toc] | [prev] | [next] | [standalone]


#45547

FromCarlos Nepomuceno <carlosnepomuceno@outlook.com>
Date2013-05-19 06:53 +0300
Message-ID<mailman.1831.1368935651.3114.python-list@python.org>
In reply to#45481
----------------------------------------
> Date: Sat, 18 May 2013 22:41:32 -0400
> From: davea@davea.name
> To: python-list@python.org
> Subject: Re: How to write fast into a file in python?
>
> On 05/18/2013 01:00 PM, Carlos Nepomuceno wrote:
>> Python really writes '\n\r' on Windows. Just check the files.
>
> That's backwards. '\r\n' on Windows, IF you omit the b in the mode when
> creating the file.

Indeed! My mistake just made me find out that Acorn used that inversion on Acorn MOS.

According to this[1] (at page 449) the OSNEWL routine outputs '\n\r'.

What the hell those guys were thinking??? :p

"OSNEWL
This call issues an LF CR (line feed, carriage return) to the currently selected
output stream. The routine is entered at &FFE7."

[1] http://regregex.bbcmicro.net/BPlusUserGuide-1.07.pdf 		 	   		  

[toc] | [prev] | [next] | [standalone]


#45563

FromMRAB <python@mrabarnett.plus.com>
Date2013-05-19 16:44 +0100
Message-ID<mailman.1847.1368978295.3114.python-list@python.org>
In reply to#45481
On 19/05/2013 04:53, Carlos Nepomuceno wrote:
> ----------------------------------------
>> Date: Sat, 18 May 2013 22:41:32 -0400
>> From: davea@davea.name
>> To: python-list@python.org
>> Subject: Re: How to write fast into a file in python?
>>
>> On 05/18/2013 01:00 PM, Carlos Nepomuceno wrote:
>>> Python really writes '\n\r' on Windows. Just check the files.
>>
>> That's backwards. '\r\n' on Windows, IF you omit the b in the mode when
>> creating the file.
>
> Indeed! My mistake just made me find out that Acorn used that inversion on Acorn MOS.
>
> According to this[1] (at page 449) the OSNEWL routine outputs '\n\r'.
>
> What the hell those guys were thinking??? :p
>
Doing it that way saved a few bytes.

Code was something like this:

FFE3    .OSASCI CMP #&0D
FFE5            BNE OSWRCH
FFE7    .OSNEWL LDA #&0A
FFE9            JSR OSWRCH
FFEC            LDA #&0D
FFEE    .OSWRCH ...

This means that the contents of the accumulator would always be
preserved by a call to OSASCI.

> "OSNEWL
> This call issues an LF CR (line feed, carriage return) to the currently selected
> output stream. The routine is entered at &FFE7."
>
> [1] http://regregex.bbcmicro.net/BPlusUserGuide-1.07.pdf 		 	   		
>

[toc] | [prev] | [next] | [standalone]


#45616

FromCarlos Nepomuceno <carlosnepomuceno@outlook.com>
Date2013-05-20 13:34 +0300
Message-ID<mailman.1885.1369046165.3114.python-list@python.org>
In reply to#45481
Oh well! Just got a flashback from the old times at the 8-bit assembly line.

Dirty deeds done dirt cheap! lol

----------------------------------------
> Date: Sun, 19 May 2013 16:44:55 +0100
> From: python@mrabarnett.plus.com
> To: python-list@python.org
> Subject: Re: How to write fast into a file in python?
>
> On 19/05/2013 04:53, Carlos Nepomuceno wrote:
>> ----------------------------------------
>>> Date: Sat, 18 May 2013 22:41:32 -0400
>>> From: davea@davea.name
>>> To: python-list@python.org
>>> Subject: Re: How to write fast into a file in python?
>>>
>>> On 05/18/2013 01:00 PM, Carlos Nepomuceno wrote:
>>>> Python really writes '\n\r' on Windows. Just check the files.
>>>
>>> That's backwards. '\r\n' on Windows, IF you omit the b in the mode when
>>> creating the file.
>>
>> Indeed! My mistake just made me find out that Acorn used that inversion on Acorn MOS.
>>
>> According to this[1] (at page 449) the OSNEWL routine outputs '\n\r'.
>>
>> What the hell those guys were thinking??? :p
>>
> Doing it that way saved a few bytes.
>
> Code was something like this:
>
> FFE3 .OSASCI CMP #&0D
> FFE5 BNE OSWRCH
> FFE7 .OSNEWL LDA #&0A
> FFE9 JSR OSWRCH
> FFEC LDA #&0D
> FFEE .OSWRCH ...
>
> This means that the contents of the accumulator would always be
> preserved by a call to OSASCI.
>
>> "OSNEWL
>> This call issues an LF CR (line feed, carriage return) to the currently selected
>> output stream. The routine is entered at &FFE7."
>>
>> [1] http://regregex.bbcmicro.net/BPlusUserGuide-1.07.pdf
>>
>
> --
> http://mail.python.org/mailman/listinfo/python-list 		 	   		  

[toc] | [prev] | [next] | [standalone]


#45527

FromDan Stromberg <drsalists@gmail.com>
Date2013-05-18 12:38 -0700
Message-ID<mailman.1816.1368905919.3114.python-list@python.org>
In reply to#45467

[Multipart message — attachments visible in raw view] — view raw

With CPython 2.7.3:
./t
time taken to write a file of size 52428800  is  15.86 seconds

time taken to write a file of size 52428800  is  7.91 seconds

time taken to write a file of size 52428800  is  9.64 seconds


With pypy-1.9:
./t
time taken to write a file of size 52428800  is  3.708232 seconds

time taken to write a file of size 52428800  is  4.868304 seconds

time taken to write a file of size 52428800  is  1.93612 seconds


Here's the code:
#!/usr/local/pypy-1.9/bin/pypy
#!/usr/bin/python

import sys
import time
import StringIO

sys.path.insert(0, '/usr/local/lib')
import bufsock

def create_file_numbers_old(filename, size):
    start = time.clock()

    value = 0
    with open(filename, "w") as f:
        while f.tell() < size:
            f.write("{0}\n".format(value))
            value += 1

    end = time.clock()

    print "time taken to write a file of size", size, " is ", (end -start),
"seconds \n"

def create_file_numbers_bufsock(filename, intended_size):
    start = time.clock()

    value = 0
    with open(filename, "w") as f:
        bs = bufsock.bufsock(f)
        actual_size = 0
        while actual_size < intended_size:
            string = "{0}\n".format(value)
            actual_size += len(string) + 1
            bs.write(string)
            value += 1
        bs.flush()

    end = time.clock()

    print "time taken to write a file of size", intended_size, " is ", (end
-start), "seconds \n"


def create_file_numbers_file_like(filename, intended_size):
    start = time.clock()

    value = 0
    with open(filename, "w") as f:
        file_like = StringIO.StringIO()
        actual_size = 0
        while actual_size < intended_size:
            string = "{0}\n".format(value)
            actual_size += len(string) + 1
            file_like.write(string)
            value += 1
        file_like.seek(0)
        f.write(file_like.read())

    end = time.clock()

    print "time taken to write a file of size", intended_size, " is ", (end
-start), "seconds \n"

create_file_numbers_old('output.txt', 50 * 2**20)
create_file_numbers_bufsock('output2.txt', 50 * 2**20)
create_file_numbers_file_like('output3.txt', 50 * 2**20)




On Thu, May 16, 2013 at 9:35 PM, <lokeshkoppaka@gmail.com> wrote:

> On Friday, May 17, 2013 8:50:26 AM UTC+5:30, lokesh...@gmail.com wrote:
> > I need to write numbers into a file upto 50mb and it should be fast
> >
> > can any one help me how to do that?
> >
> > i had written the following code..
> >
> >
> -----------------------------------------------------------------------------------------------------------
> >
> > def create_file_numbers_old(filename, size):
> >
> > start = time.clock()
> >
> >
> >
> > value = 0
> >
> > with open(filename, "w") as f:
> >
> > while f.tell()< size:
> >
> > f.write("{0}\n".format(value))
> >
> > value += 1
> >
> >
> >
> > end = time.clock()
> >
> >
> >
> > print "time taken to write a file of size", size, " is ", (end -start),
> "seconds \n"
> >
> >
> ------------------------------------------------------------------------------------------------------------------
> >
> > it takes about 20sec i need 5 to 10 times less than that.
> size = 50mb
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]


#45549

FromCarlos Nepomuceno <carlosnepomuceno@outlook.com>
Date2013-05-19 08:31 +0300
Message-ID<mailman.1833.1368941537.3114.python-list@python.org>
In reply to#45467
Thanks Dan! I've never used CPython or PyPy. Will try them later.

I think the main difference between your create_file_numbers_file_like()
 and the fastwrite5.py I sent earlier is that I've used cStringIO 
instead of StringIO. It took 12s less using cStringIO.

My numbers are much greater, but I've used Python 2.7.5 instead:

C:\src\Python>python create_file_numbers.py
time taken to write a file of size 52428800  is  39.1199457743 seconds

time taken to write a file of size 52428800  is  14.8704800436 seconds

time taken to write a file of size 52428800  is  23.0011990985 seconds


I've downloaded bufsock.py and python2x3.py. The later one was hard to remove the source code from the web page.

Can I use them on my projects? I'm not used to the UCI license[1]. What's the difference to the GPL?




[1] http://stromberg.dnsalias.org/~dstromberg/UCI-license.html

________________________________
> Date: Sat, 18 May 2013 12:38:30 -0700 
> Subject: Re: How to write fast into a file in python? 
> From: drsalists@gmail.com 
> To: lokeshkoppaka@gmail.com 
> CC: python-list@python.org 
>  
>  
> With CPython 2.7.3: 
> ./t 
> time taken to write a file of size 52428800  is  15.86 seconds 
>  
> time taken to write a file of size 52428800  is  7.91 seconds 
>  
> time taken to write a file of size 52428800  is  9.64 seconds 
>  
>  
> With pypy-1.9: 
> ./t 
> time taken to write a file of size 52428800  is  3.708232 seconds 
>  
> time taken to write a file of size 52428800  is  4.868304 seconds 
>  
> time taken to write a file of size 52428800  is  1.93612 seconds 
>  

> Here's the code: 
> #!/usr/local/pypy-1.9/bin/pypy 
> #!/usr/bin/python 
>  
> import sys 
> import time 
> import StringIO 
>  
> sys.path.insert(0, '/usr/local/lib') 
> import bufsock 
>  
> def create_file_numbers_old(filename, size): 
>      start = time.clock() 
>  
>      value = 0 
>      with open(filename, "w") as f: 
>          while f.tell() < size: 
>              f.write("{0}\n".format(value)) 
>              value += 1 
>  
>      end = time.clock() 
>  
>      print "time taken to write a file of size", size, " is ", (end  
> -start), "seconds \n" 
>  
> def create_file_numbers_bufsock(filename, intended_size): 
>      start = time.clock() 
>  
>      value = 0 
>      with open(filename, "w") as f: 
>          bs = bufsock.bufsock(f) 
>          actual_size = 0 
>          while actual_size < intended_size: 
>              string = "{0}\n".format(value) 
>              actual_size += len(string) + 1 
>              bs.write(string) 
>              value += 1 
>          bs.flush() 
>  
>      end = time.clock() 
>  
>      print "time taken to write a file of size", intended_size, " is ",  
> (end -start), "seconds \n" 
>  
>  
> def create_file_numbers_file_like(filename, intended_size): 
>      start = time.clock() 
>  
>      value = 0 
>      with open(filename, "w") as f: 
>          file_like = StringIO.StringIO() 
>          actual_size = 0 
>          while actual_size < intended_size: 
>              string = "{0}\n".format(value) 
>              actual_size += len(string) + 1 
>              file_like.write(string) 
>              value += 1 
>          file_like.seek(0) 
>          f.write(file_like.read()) 
>  
>      end = time.clock() 
>  
>      print "time taken to write a file of size", intended_size, " is ",  
> (end -start), "seconds \n" 
>  
> create_file_numbers_old('output.txt', 50 * 2**20) 
> create_file_numbers_bufsock('output2.txt', 50 * 2**20) 
> create_file_numbers_file_like('output3.txt', 50 * 2**20) 
>  
>  
>  
>  
> On Thu, May 16, 2013 at 9:35 PM,  
> <lokeshkoppaka@gmail.com<mailto:lokeshkoppaka@gmail.com>> wrote: 
> On Friday, May 17, 2013 8:50:26 AM UTC+5:30,  
> lokesh...@gmail.com<mailto:lokesh...@gmail.com> wrote: 
> > I need to write numbers into a file upto 50mb and it should be fast 
> > 
> > can any one help me how to do that? 
> > 
> > i had written the following code.. 
> > 
> >  
> ----------------------------------------------------------------------------------------------------------- 
> > 
> > def create_file_numbers_old(filename, size): 
> > 
> > start = time.clock() 
> > 
> > 
> > 
> > value = 0 
> > 
> > with open(filename, "w") as f: 
> > 
> > while f.tell()< size: 
> > 
> > f.write("{0}\n".format(value)) 
> > 
> > value += 1 
> > 
> > 
> > 
> > end = time.clock() 
> > 
> > 
> > 
> > print "time taken to write a file of size", size, " is ", (end  
> -start), "seconds \n" 
> > 
> >  
> ------------------------------------------------------------------------------------------------------------------ 
> > 
> > it takes about 20sec i need 5 to 10 times less than that. 
> size = 50mb 
> -- 
> http://mail.python.org/mailman/listinfo/python-list 
>  
>  
> -- http://mail.python.org/mailman/listinfo/python-list 		 	   		  

[toc] | [prev] | [next] | [standalone]


#45550

FromCarlos Nepomuceno <carlosnepomuceno@outlook.com>
Date2013-05-19 08:42 +0300
Message-ID<mailman.1834.1368942232.3114.python-list@python.org>
In reply to#45467
BTW, I've downloaded from the following places:

http://stromberg.dnsalias.org/svn/bufsock/trunk/bufsock.py
http://stromberg.dnsalias.org/~dstromberg/backshift/documentation/html/python2x3-pysrc.html

Are those the latest versions?

----------------------------------------
> From: carlosnepomuceno@outlook.com
> To: python-list@python.org
> Subject: RE: How to write fast into a file in python?
> Date: Sun, 19 May 2013 08:31:08 +0300
> CC: lokeshkoppaka@gmail.com
>
> Thanks Dan! I've never used CPython or PyPy. Will try them later.
>
> I think the main difference between your create_file_numbers_file_like()
> and the fastwrite5.py I sent earlier is that I've used cStringIO
> instead of StringIO. It took 12s less using cStringIO.
>
> My numbers are much greater, but I've used Python 2.7.5 instead:
>
> C:\src\Python>python create_file_numbers.py
> time taken to write a file of size 52428800  is  39.1199457743 seconds
>
> time taken to write a file of size 52428800  is  14.8704800436 seconds
>
> time taken to write a file of size 52428800  is  23.0011990985 seconds
>
>
> I've downloaded bufsock.py and python2x3.py. The later one was hard to remove the source code from the web page.
>
> Can I use them on my projects? I'm not used to the UCI license[1]. What's the difference to the GPL?
>
>
>
>
> [1] http://stromberg.dnsalias.org/~dstromberg/UCI-license.html
>
> ________________________________
>> Date: Sat, 18 May 2013 12:38:30 -0700
>> Subject: Re: How to write fast into a file in python?
>> From: drsalists@gmail.com
>> To: lokeshkoppaka@gmail.com
>> CC: python-list@python.org
>>
>>
>> With CPython 2.7.3:
>> ./t
>> time taken to write a file of size 52428800 is 15.86 seconds
>>
>> time taken to write a file of size 52428800 is 7.91 seconds
>>
>> time taken to write a file of size 52428800 is 9.64 seconds
>>
>>
>> With pypy-1.9:
>> ./t
>> time taken to write a file of size 52428800 is 3.708232 seconds
>>
>> time taken to write a file of size 52428800 is 4.868304 seconds
>>
>> time taken to write a file of size 52428800 is 1.93612 seconds
>>
>
>> Here's the code:
>> #!/usr/local/pypy-1.9/bin/pypy
>> #!/usr/bin/python
>>
>> import sys
>> import time
>> import StringIO
>>
>> sys.path.insert(0, '/usr/local/lib')
>> import bufsock
>>
>> def create_file_numbers_old(filename, size):
>> start = time.clock()
>>
>> value = 0
>> with open(filename, "w") as f:
>> while f.tell() < size:
>> f.write("{0}\n".format(value))
>> value += 1
>>
>> end = time.clock()
>>
>> print "time taken to write a file of size", size, " is ", (end
>> -start), "seconds \n"
>>
>> def create_file_numbers_bufsock(filename, intended_size):
>> start = time.clock()
>>
>> value = 0
>> with open(filename, "w") as f:
>> bs = bufsock.bufsock(f)
>> actual_size = 0
>> while actual_size < intended_size:
>> string = "{0}\n".format(value)
>> actual_size += len(string) + 1
>> bs.write(string)
>> value += 1
>> bs.flush()
>>
>> end = time.clock()
>>
>> print "time taken to write a file of size", intended_size, " is ",
>> (end -start), "seconds \n"
>>
>>
>> def create_file_numbers_file_like(filename, intended_size):
>> start = time.clock()
>>
>> value = 0
>> with open(filename, "w") as f:
>> file_like = StringIO.StringIO()
>> actual_size = 0
>> while actual_size < intended_size:
>> string = "{0}\n".format(value)
>> actual_size += len(string) + 1
>> file_like.write(string)
>> value += 1
>> file_like.seek(0)
>> f.write(file_like.read())
>>
>> end = time.clock()
>>
>> print "time taken to write a file of size", intended_size, " is ",
>> (end -start), "seconds \n"
>>
>> create_file_numbers_old('output.txt', 50 * 2**20)
>> create_file_numbers_bufsock('output2.txt', 50 * 2**20)
>> create_file_numbers_file_like('output3.txt', 50 * 2**20)
>>
>>
>>
>>
>> On Thu, May 16, 2013 at 9:35 PM,
>> <lokeshkoppaka@gmail.com<mailto:lokeshkoppaka@gmail.com>> wrote:
>> On Friday, May 17, 2013 8:50:26 AM UTC+5:30,
>> lokesh...@gmail.com<mailto:lokesh...@gmail.com> wrote:
>>> I need to write numbers into a file upto 50mb and it should be fast
>>>
>>> can any one help me how to do that?
>>>
>>> i had written the following code..
>>>
>>>
>> -----------------------------------------------------------------------------------------------------------
>>>
>>> def create_file_numbers_old(filename, size):
>>>
>>> start = time.clock()
>>>
>>>
>>>
>>> value = 0
>>>
>>> with open(filename, "w") as f:
>>>
>>> while f.tell()< size:
>>>
>>> f.write("{0}\n".format(value))
>>>
>>> value += 1
>>>
>>>
>>>
>>> end = time.clock()
>>>
>>>
>>>
>>> print "time taken to write a file of size", size, " is ", (end
>> -start), "seconds \n"
>>>
>>>
>> ------------------------------------------------------------------------------------------------------------------
>>>
>>> it takes about 20sec i need 5 to 10 times less than that.
>> size = 50mb
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>>
>>
>> -- http://mail.python.org/mailman/listinfo/python-list
> --
> http://mail.python.org/mailman/listinfo/python-list 		 	   		  

[toc] | [prev] | [next] | [standalone]


#45555

FromChris Angelico <rosuav@gmail.com>
Date2013-05-19 19:21 +1000
Message-ID<mailman.1842.1368955322.3114.python-list@python.org>
In reply to#45467
On Sun, May 19, 2013 at 3:31 PM, Carlos Nepomuceno
<carlosnepomuceno@outlook.com> wrote:
> Thanks Dan! I've never used CPython or PyPy. Will try them later.

CPython is the "classic" interpreter, written in C. It's the one
you'll get from the obvious download links on python.org.

ChrisA

[toc] | [prev] | [next] | [standalone]


#45556

FromCarlos Nepomuceno <carlosnepomuceno@outlook.com>
Date2013-05-19 12:41 +0300
Message-ID<mailman.1843.1368956490.3114.python-list@python.org>
In reply to#45467
ooops! I meant to say Cython. nevermind...

----------------------------------------
> Date: Sun, 19 May 2013 19:21:54 +1000
> Subject: Re: How to write fast into a file in python?
> From: rosuav@gmail.com
> To: python-list@python.org
>
> On Sun, May 19, 2013 at 3:31 PM, Carlos Nepomuceno
> <carlosnepomuceno@outlook.com> wrote:
>> Thanks Dan! I've never used CPython or PyPy. Will try them later.
>
> CPython is the "classic" interpreter, written in C. It's the one
> you'll get from the obvious download links on python.org.
>
> ChrisA
> --
> http://mail.python.org/mailman/listinfo/python-list 		 	   		  

[toc] | [prev] | [standalone]


Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python


csiph-web