Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90953 > unrolled thread

Re: Best approach to create humongous amount of files

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Best approach to create humongous amount of files Chris Angelico <rosuav@gmail.com> - 2015-05-20 22:58 +1000
    Re: Best approach to create humongous amount of files paul.anton.letnes@gmail.com - 2015-05-20 08:32 -0700

#90953 — Re: Best approach to create humongous amount of files

On Wed, May 20, 2015 at 9:44 PM, Parul Mogra <scoria.799@gmail.com> wrote:
> My objective is to create large amount of data files (say a million *.json
> files), using a pre-existing template file (*.json). Each file would have a
> unique name, possibly by incorporating time stamp information. The files
> have to be generated in a folder specified.
>
> What is the best strategy to achieve this task, so that the files will be
> generated in the shortest possible time? Say within an hour.

If they're to be created within an hour, timestamp of creation
probably isn't useful, but if you have internal data to use as the
file name, that would work. Otherwise, try a simple sequential
integer.

All you'd need would be a loop that creates a bunch of files... most
of your code will be figuring out what parts of the template need to
change. Not too difficult.

ChrisA

[toc] | [next] | [standalone]

#90960

There's a module called "template" that I've used before, for the find/replace part. I never investigated its performance, but my script used less than 1 s for 100 files IIRC :-)

Paul

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web