Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90959 > unrolled thread

Re: Best approach to create humongous amount of files

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Best approach to create humongous amount of files Tim Chase <python.list@tim.thechases.com> - 2015-05-20 10:07 -0500

#90959 — Re: Best approach to create humongous amount of files

On 2015-05-20 22:58, Chris Angelico wrote:
> On Wed, May 20, 2015 at 9:44 PM, Parul Mogra <scoria.799@gmail.com>
> wrote:
> > My objective is to create large amount of data files (say a
> > million *.json files), using a pre-existing template file
> > (*.json). Each file would have a unique name, possibly by
> > incorporating time stamp information. The files have to be
> > generated in a folder specified.
[snip] 
> try a simple sequential integer.
> 
> All you'd need would be a loop that creates a bunch of files... most
> of your code will be figuring out what parts of the template need to
> change. Not too difficult.

If you store your template as a Python string-formatting template,
you can just use string-formatting to do your dirty work:


  import random
  HOW_MANY = 1000000
  template = """{
    "some_string": "%(string)s",
    "some_int": %(int)i
    }
    """

  wordlist = [
    word.rstrip()
    for word in open('/usr/share/dict/words')
    ]
  wordlist[:] = [ # just lowercase all-alpha words
    word
    for word in wordlist
    if word.isalpha() and word.islower()
    ]

  for i in xrange(HOW_MANY):
    fname = "data_%08i.json" % i
    with open(fname, "w") as f:
      f.write(template % {
        "string_value": random.choice(wordlist),
        "int_value": random.randint(0, 1000),
        })


-tkc

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web