Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; '16,': 0.03; 'cpython': 0.05; 'that?': 0.05; 'subject:file': 0.07; 'sys': 0.07; 'string': 0.09; 'friday,': 0.09; 'skip:# 30': 0.09; 'skip:- 110': 0.09; 'subject:into': 0.09; 'subject:How': 0.10; 'python': 0.11; 'def': 0.12; '"seconds': 0.16; '"w")': 0.16; 'cstringio': 0.16; 'f.tell()': 0.16; 'instead:': 0.16; 'pypy.': 0.16; 'size):': 0.16; 'stringio': 0.16; 'to:name:python list': 0.16; 'subject:python': 0.16; 'sat,': 0.16; 'size,': 0.16; 'wrote:': 0.18; 'skip:f 30': 0.19; 'thu,': 0.19; 'later': 0.20; 'written': 0.21; 'import': 0.22; 'cc:addr:gmail.com': 0.22; 'python?': 0.22; 'print': 0.22; 'received:65.55.116': 0.24; 'earlier': 0.24; 'cc:2**0': 0.24; "i've": 0.25; 'source': 0.25; 'code:': 0.26; 'downloaded': 0.26; 'skip:" 20': 0.27; 'header:In-Reply-To:1': 0.27; '[1]': 0.29; "i'm": 0.30; 'url:mailman': 0.30; 'code': 0.31; 'page.': 0.31; "skip:' 10": 0.31; 'that.': 0.31; 'skip:- 100': 0.31; 'file': 0.32; 'skip:c 30': 0.32; 'url:python': 0.33; 'skip:# 10': 0.33; 'date:': 0.34; 'but': 0.35; 'url:listinfo': 0.36; 'thanks': 0.36; 'subject:?': 0.36; 'url:org': 0.36; 'should': 0.36; 'seconds': 0.37; 'email addr:python.org': 0.37; 'to:addr:python-list': 0.38; 'pm,': 0.38; 'skip:_ 30': 0.39; 'subject:': 0.39; 'to:addr:python.org': 0.39; 'url:mail': 0.40; 'how': 0.40; 'remove': 0.60; 'numbers': 0.61; 'took': 0.61; 'times': 0.62; 'email addr:gmail.com': 0.63; 're:': 0.63; 'email name:python- list': 0.65; 'between': 0.67; 'is\xa0': 0.84; '2013': 0.98 X-TMN: [slRKimDVV2OFeWazbp9oPVS6yEyGV41Y] X-Originating-Email: [carlosnepomuceno@outlook.com] From: Carlos Nepomuceno To: Python List Subject: RE: How to write fast into a file in python? Date: Sun, 19 May 2013 08:31:08 +0300 Importance: Normal In-Reply-To: References: , <87f9a3d4-427e-472f-bee7-9501ba842b36@googlegroups.com>, Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 19 May 2013 05:31:08.0242 (UTC) FILETIME=[1092A320:01CE5452] Cc: "lokeshkoppaka@gmail.com" X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 182 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1368941537 news.xs4all.nl 15947 [2001:888:2000:d::a6]:52401 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:45549 Thanks Dan! I've never used CPython or PyPy. Will try them later.=0A= =0A= I think the main difference between your create_file_numbers_file_like()=0A= and the fastwrite5.py I sent earlier is that I've used cStringIO =0A= instead of StringIO. It took 12s less using cStringIO.=0A= =0A= My numbers are much greater=2C but I've used Python 2.7.5 instead:=0A= =0A= C:\src\Python>python create_file_numbers.py=0A= time taken to write a file of size 52428800=A0 is=A0 39.1199457743 seconds= =0A= =0A= time taken to write a file of size 52428800=A0 is=A0 14.8704800436 seconds= =0A= =0A= time taken to write a file of size 52428800=A0 is=A0 23.0011990985 seconds= =0A= =0A= =0A= I've downloaded bufsock.py and python2x3.py. The later one was hard to remo= ve the source code from the web page.=0A= =0A= Can I use them on my projects? I'm not used to the UCI license[1]. What's t= he difference to the GPL?=0A= =0A= =0A= =0A= =0A= [1] http://stromberg.dnsalias.org/~dstromberg/UCI-license.html=0A= =0A= ________________________________=0A= > Date: Sat=2C 18 May 2013 12:38:30 -0700 =0A= > Subject: Re: How to write fast into a file in python? =0A= > From: drsalists@gmail.com =0A= > To: lokeshkoppaka@gmail.com =0A= > CC: python-list@python.org =0A= > =0A= > =0A= > With CPython 2.7.3: =0A= > ./t =0A= > time taken to write a file of size 52428800 is 15.86 seconds =0A= > =0A= > time taken to write a file of size 52428800 is 7.91 seconds =0A= > =0A= > time taken to write a file of size 52428800 is 9.64 seconds =0A= > =0A= > =0A= > With pypy-1.9: =0A= > ./t =0A= > time taken to write a file of size 52428800 is 3.708232 seconds =0A= > =0A= > time taken to write a file of size 52428800 is 4.868304 seconds =0A= > =0A= > time taken to write a file of size 52428800 is 1.93612 seconds =0A= > =0A= =0A= > Here's the code: =0A= > #!/usr/local/pypy-1.9/bin/pypy =0A= > #!/usr/bin/python =0A= > =0A= > import sys =0A= > import time =0A= > import StringIO =0A= > =0A= > sys.path.insert(0=2C '/usr/local/lib') =0A= > import bufsock =0A= > =0A= > def create_file_numbers_old(filename=2C size): =0A= > start =3D time.clock() =0A= > =0A= > value =3D 0 =0A= > with open(filename=2C "w") as f: =0A= > while f.tell() < size: =0A= > f.write("{0}\n".format(value)) =0A= > value +=3D 1 =0A= > =0A= > end =3D time.clock() =0A= > =0A= > print "time taken to write a file of size"=2C size=2C " is "=2C (end= =0A= > -start)=2C "seconds \n" =0A= > =0A= > def create_file_numbers_bufsock(filename=2C intended_size): =0A= > start =3D time.clock() =0A= > =0A= > value =3D 0 =0A= > with open(filename=2C "w") as f: =0A= > bs =3D bufsock.bufsock(f) =0A= > actual_size =3D 0 =0A= > while actual_size < intended_size: =0A= > string =3D "{0}\n".format(value) =0A= > actual_size +=3D len(string) + 1 =0A= > bs.write(string) =0A= > value +=3D 1 =0A= > bs.flush() =0A= > =0A= > end =3D time.clock() =0A= > =0A= > print "time taken to write a file of size"=2C intended_size=2C " is = "=2C =0A= > (end -start)=2C "seconds \n" =0A= > =0A= > =0A= > def create_file_numbers_file_like(filename=2C intended_size): =0A= > start =3D time.clock() =0A= > =0A= > value =3D 0 =0A= > with open(filename=2C "w") as f: =0A= > file_like =3D StringIO.StringIO() =0A= > actual_size =3D 0 =0A= > while actual_size < intended_size: =0A= > string =3D "{0}\n".format(value) =0A= > actual_size +=3D len(string) + 1 =0A= > file_like.write(string) =0A= > value +=3D 1 =0A= > file_like.seek(0) =0A= > f.write(file_like.read()) =0A= > =0A= > end =3D time.clock() =0A= > =0A= > print "time taken to write a file of size"=2C intended_size=2C " is = "=2C =0A= > (end -start)=2C "seconds \n" =0A= > =0A= > create_file_numbers_old('output.txt'=2C 50 * 2**20) =0A= > create_file_numbers_bufsock('output2.txt'=2C 50 * 2**20) =0A= > create_file_numbers_file_like('output3.txt'=2C 50 * 2**20) =0A= > =0A= > =0A= > =0A= > =0A= > On Thu=2C May 16=2C 2013 at 9:35 PM=2C =0A= > > wrote: =0A= > On Friday=2C May 17=2C 2013 8:50:26 AM UTC+5:30=2C =0A= > lokesh...@gmail.com wrote: =0A= > > I need to write numbers into a file upto 50mb and it should be fast =0A= > > =0A= > > can any one help me how to do that? =0A= > > =0A= > > i had written the following code.. =0A= > > =0A= > > =0A= > -------------------------------------------------------------------------= ---------------------------------- =0A= > > =0A= > > def create_file_numbers_old(filename=2C size): =0A= > > =0A= > > start =3D time.clock() =0A= > > =0A= > > =0A= > > =0A= > > value =3D 0 =0A= > > =0A= > > with open(filename=2C "w") as f: =0A= > > =0A= > > while f.tell()< size: =0A= > > =0A= > > f.write("{0}\n".format(value)) =0A= > > =0A= > > value +=3D 1 =0A= > > =0A= > > =0A= > > =0A= > > end =3D time.clock() =0A= > > =0A= > > =0A= > > =0A= > > print "time taken to write a file of size"=2C size=2C " is "=2C (end = =0A= > -start)=2C "seconds \n" =0A= > > =0A= > > =0A= > -------------------------------------------------------------------------= ----------------------------------------- =0A= > > =0A= > > it takes about 20sec i need 5 to 10 times less than that. =0A= > size =3D 50mb =0A= > -- =0A= > http://mail.python.org/mailman/listinfo/python-list =0A= > =0A= > =0A= > -- http://mail.python.org/mailman/listinfo/python-list =