Path: csiph.com!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!newsfeed.eweka.nl!eweka.nl!feeder3.eweka.nl!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; '16,': 0.03; 'cpython': 0.05; 'that?': 0.05; 'subject:file': 0.07; 'sys': 0.07; 'string': 0.09; 'friday,': 0.09; 'skip:# 30': 0.09; 'skip:- 110': 0.09; 'subject:into': 0.09; 'subject:How': 0.10; 'python': 0.11; 'def': 0.12; '"seconds': 0.16; '"w")': 0.16; 'cstringio': 0.16; 'f.tell()': 0.16; 'instead:': 0.16; 'pypy.': 0.16; 'size):': 0.16; 'stringio': 0.16; 'to:name:python list': 0.16; 'url:py': 0.16; 'url:svn': 0.16; 'subject:python': 0.16; 'sat,': 0.16; 'size,': 0.16; 'wrote:': 0.18; 'skip:f 30': 0.19; 'thu,': 0.19; 'later': 0.20; 'written': 0.21; '>>>': 0.22; 'import': 0.22; 'python?': 0.22; 'print': 0.22; 'received:65.55.116': 0.24; 'earlier': 0.24; "i've": 0.25; 'source': 0.25; 'code:': 0.26; 'downloaded': 0.26; 'skip:" 20': 0.27; 'header:In-Reply-To:1': 0.27; 'skip:- 40': 0.29; '[1]': 0.29; "i'm": 0.30; 'url:mailman': 0.30; 'code': 0.31; 'page.': 0.31; "skip:' 10": 0.31; 'that.': 0.31; 'skip:- 100': 0.31; 'file': 0.32; 'skip:c 30': 0.32; 'url:python': 0.33; 'skip:# 10': 0.33; 'date:': 0.34; 'but': 0.35; 'url:listinfo': 0.36; 'thanks': 0.36; 'subject:?': 0.36; 'url:org': 0.36; 'should': 0.36; 'seconds': 0.37; 'email addr:python.org': 0.37; 'to:addr :python-list': 0.38; 'pm,': 0.38; 'skip:_ 30': 0.39; 'subject:': 0.39; 'to:addr:python.org': 0.39; 'url:mail': 0.40; 'how': 0.40; 'remove': 0.60; 'numbers': 0.61; 'took': 0.61; 'times': 0.62; 'email addr:gmail.com': 0.63; 're:': 0.63; 'email name:python- list': 0.65; 'latest': 0.67; 'between': 0.67; '2013': 0.98 X-TMN: [FG03/Fg/JhuA9rdLgbh3u8iVCSvqUAwY] X-Originating-Email: [carlosnepomuceno@outlook.com] From: Carlos Nepomuceno To: Python List Subject: RE: How to write fast into a file in python? Date: Sun, 19 May 2013 08:42:41 +0300 Importance: Normal In-Reply-To: References: , , <87f9a3d4-427e-472f-bee7-9501ba842b36@googlegroups.com>, , , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 19 May 2013 05:42:41.0789 (UTC) FILETIME=[ADF582D0:01CE5453] X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 195 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1368942232 news.xs4all.nl 15955 [2001:888:2000:d::a6]:55289 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:45550 BTW=2C I've downloaded from the following places:=0A= =0A= http://stromberg.dnsalias.org/svn/bufsock/trunk/bufsock.py=0A= http://stromberg.dnsalias.org/~dstromberg/backshift/documentation/html/pyth= on2x3-pysrc.html=0A= =0A= Are those the latest versions?=0A= =0A= ----------------------------------------=0A= > From: carlosnepomuceno@outlook.com=0A= > To: python-list@python.org=0A= > Subject: RE: How to write fast into a file in python?=0A= > Date: Sun=2C 19 May 2013 08:31:08 +0300=0A= > CC: lokeshkoppaka@gmail.com=0A= >=0A= > Thanks Dan! I've never used CPython or PyPy. Will try them later.=0A= >=0A= > I think the main difference between your create_file_numbers_file_like()= =0A= > and the fastwrite5.py I sent earlier is that I've used cStringIO=0A= > instead of StringIO. It took 12s less using cStringIO.=0A= >=0A= > My numbers are much greater=2C but I've used Python 2.7.5 instead:=0A= >=0A= > C:\src\Python>python create_file_numbers.py=0A= > time taken to write a file of size 52428800 is 39.1199457743 seconds=0A= >=0A= > time taken to write a file of size 52428800 is 14.8704800436 seconds=0A= >=0A= > time taken to write a file of size 52428800 is 23.0011990985 seconds=0A= >=0A= >=0A= > I've downloaded bufsock.py and python2x3.py. The later one was hard to re= move the source code from the web page.=0A= >=0A= > Can I use them on my projects? I'm not used to the UCI license[1]. What's= the difference to the GPL?=0A= >=0A= >=0A= >=0A= >=0A= > [1] http://stromberg.dnsalias.org/~dstromberg/UCI-license.html=0A= >=0A= > ________________________________=0A= >> Date: Sat=2C 18 May 2013 12:38:30 -0700=0A= >> Subject: Re: How to write fast into a file in python?=0A= >> From: drsalists@gmail.com=0A= >> To: lokeshkoppaka@gmail.com=0A= >> CC: python-list@python.org=0A= >>=0A= >>=0A= >> With CPython 2.7.3:=0A= >> ./t=0A= >> time taken to write a file of size 52428800 is 15.86 seconds=0A= >>=0A= >> time taken to write a file of size 52428800 is 7.91 seconds=0A= >>=0A= >> time taken to write a file of size 52428800 is 9.64 seconds=0A= >>=0A= >>=0A= >> With pypy-1.9:=0A= >> ./t=0A= >> time taken to write a file of size 52428800 is 3.708232 seconds=0A= >>=0A= >> time taken to write a file of size 52428800 is 4.868304 seconds=0A= >>=0A= >> time taken to write a file of size 52428800 is 1.93612 seconds=0A= >>=0A= >=0A= >> Here's the code:=0A= >> #!/usr/local/pypy-1.9/bin/pypy=0A= >> #!/usr/bin/python=0A= >>=0A= >> import sys=0A= >> import time=0A= >> import StringIO=0A= >>=0A= >> sys.path.insert(0=2C '/usr/local/lib')=0A= >> import bufsock=0A= >>=0A= >> def create_file_numbers_old(filename=2C size):=0A= >> start =3D time.clock()=0A= >>=0A= >> value =3D 0=0A= >> with open(filename=2C "w") as f:=0A= >> while f.tell() < size:=0A= >> f.write("{0}\n".format(value))=0A= >> value +=3D 1=0A= >>=0A= >> end =3D time.clock()=0A= >>=0A= >> print "time taken to write a file of size"=2C size=2C " is "=2C (end=0A= >> -start)=2C "seconds \n"=0A= >>=0A= >> def create_file_numbers_bufsock(filename=2C intended_size):=0A= >> start =3D time.clock()=0A= >>=0A= >> value =3D 0=0A= >> with open(filename=2C "w") as f:=0A= >> bs =3D bufsock.bufsock(f)=0A= >> actual_size =3D 0=0A= >> while actual_size < intended_size:=0A= >> string =3D "{0}\n".format(value)=0A= >> actual_size +=3D len(string) + 1=0A= >> bs.write(string)=0A= >> value +=3D 1=0A= >> bs.flush()=0A= >>=0A= >> end =3D time.clock()=0A= >>=0A= >> print "time taken to write a file of size"=2C intended_size=2C " is "=2C= =0A= >> (end -start)=2C "seconds \n"=0A= >>=0A= >>=0A= >> def create_file_numbers_file_like(filename=2C intended_size):=0A= >> start =3D time.clock()=0A= >>=0A= >> value =3D 0=0A= >> with open(filename=2C "w") as f:=0A= >> file_like =3D StringIO.StringIO()=0A= >> actual_size =3D 0=0A= >> while actual_size < intended_size:=0A= >> string =3D "{0}\n".format(value)=0A= >> actual_size +=3D len(string) + 1=0A= >> file_like.write(string)=0A= >> value +=3D 1=0A= >> file_like.seek(0)=0A= >> f.write(file_like.read())=0A= >>=0A= >> end =3D time.clock()=0A= >>=0A= >> print "time taken to write a file of size"=2C intended_size=2C " is "=2C= =0A= >> (end -start)=2C "seconds \n"=0A= >>=0A= >> create_file_numbers_old('output.txt'=2C 50 * 2**20)=0A= >> create_file_numbers_bufsock('output2.txt'=2C 50 * 2**20)=0A= >> create_file_numbers_file_like('output3.txt'=2C 50 * 2**20)=0A= >>=0A= >>=0A= >>=0A= >>=0A= >> On Thu=2C May 16=2C 2013 at 9:35 PM=2C=0A= >> > wrote:=0A= >> On Friday=2C May 17=2C 2013 8:50:26 AM UTC+5:30=2C=0A= >> lokesh...@gmail.com wrote:=0A= >>> I need to write numbers into a file upto 50mb and it should be fast=0A= >>>=0A= >>> can any one help me how to do that?=0A= >>>=0A= >>> i had written the following code..=0A= >>>=0A= >>>=0A= >> ------------------------------------------------------------------------= -----------------------------------=0A= >>>=0A= >>> def create_file_numbers_old(filename=2C size):=0A= >>>=0A= >>> start =3D time.clock()=0A= >>>=0A= >>>=0A= >>>=0A= >>> value =3D 0=0A= >>>=0A= >>> with open(filename=2C "w") as f:=0A= >>>=0A= >>> while f.tell()< size:=0A= >>>=0A= >>> f.write("{0}\n".format(value))=0A= >>>=0A= >>> value +=3D 1=0A= >>>=0A= >>>=0A= >>>=0A= >>> end =3D time.clock()=0A= >>>=0A= >>>=0A= >>>=0A= >>> print "time taken to write a file of size"=2C size=2C " is "=2C (end=0A= >> -start)=2C "seconds \n"=0A= >>>=0A= >>>=0A= >> ------------------------------------------------------------------------= ------------------------------------------=0A= >>>=0A= >>> it takes about 20sec i need 5 to 10 times less than that.=0A= >> size =3D 50mb=0A= >> --=0A= >> http://mail.python.org/mailman/listinfo/python-list=0A= >>=0A= >>=0A= >> -- http://mail.python.org/mailman/listinfo/python-list=0A= > --=0A= > http://mail.python.org/mailman/listinfo/python-list =