Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #44131

Re: optomizations

Show all headers | View raw

On Tue, Apr 23, 2013 at 11:19 AM, Rodrick Brown <rodrick.brown@gmail.com> wrote:
>             with gzip.open(args.inputfile) as datafile:
>                 for line in datafile:
>                     outfile = '{}{}{}_combined.log'.format(dateobj.year,
> dateobj.month, dateobj.day)
>                     outdir = (args.outputdir + os.sep + siteurl)
>
>                     with open(outdir + os.sep + outfile, 'w+') as outf:
>                         outf.write(line)

You're opening files and closing them again for every line. This
wouldn't cause you to spin the CPU (more likely it'd thrash the hard
disk - unless you have an SSD), but it is certainly an optimization
target.

Can you know in advance what files you need? If not, I'd try something
like this:

outf = {} # Might want a better name though

.....
               outfile = ...
               if outfile not in outf:
                   os.makedirs(...)
                   outf[outfile] = open(...)
               outf[outfile].write(line)

for f in outf.values():
  f.close()

Open files only as needed, close 'em all at the end.

ChrisA

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: optomizations Chris Angelico <rosuav@gmail.com> - 2013-04-23 11:38 +1000

csiph-web