Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <CABRP1o_ab1w91jHQN_9cFMAj=rpY3GVUMcKOA9+_TskRMUu=CQ@mail.gmail.com>
References: <CABRP1o_ab1w91jHQN_9cFMAj=rpY3GVUMcKOA9+_TskRMUu=CQ@mail.gmail.com>
Date: Tue, 23 Apr 2013 11:38:17 +1000
Subject: Re: optomizations
From: Chris Angelico <rosuav@gmail.com>
To: python-list@python.org
Content-Type: text/plain; charset=ISO-8859-1
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.945.1366681100.3114.python-list@python.org>
Lines: 33
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:44131

On Tue, Apr 23, 2013 at 11:19 AM, Rodrick Brown <rodrick.brown@gmail.com> wrote:
>             with gzip.open(args.inputfile) as datafile:
>                 for line in datafile:
>                     outfile = '{}{}{}_combined.log'.format(dateobj.year,
> dateobj.month, dateobj.day)
>                     outdir = (args.outputdir + os.sep + siteurl)
>
>                     with open(outdir + os.sep + outfile, 'w+') as outf:
>                         outf.write(line)

You're opening files and closing them again for every line. This
wouldn't cause you to spin the CPU (more likely it'd thrash the hard
disk - unless you have an SSD), but it is certainly an optimization
target.

Can you know in advance what files you need? If not, I'd try something
like this:

outf = {} # Might want a better name though

.....
               outfile = ...
               if outfile not in outf:
                   os.makedirs(...)
                   outf[outfile] = open(...)
               outf[outfile].write(line)

for f in outf.values():
  f.close()

Open files only as needed, close 'em all at the end.

ChrisA