Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Wed, 20 May 2015 10:07:23 -0500
From: Tim Chase <python.list@tim.thechases.com>
To: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Subject: Re: Best approach to create humongous amount of files
In-Reply-To: <CAPTjJmppiMpVjTBt5CH_6DGSdCWw5aoDU+jY-3wMs5Ai7tPdKw@mail.gmail.com>
References: <CAPkZ3MS5SiGH9OCe9RSTmakF681O+qM572y49FuDBmBix=aiFg@mail.gmail.com> <CAPTjJmppiMpVjTBt5CH_6DGSdCWw5aoDU+jY-3wMs5Ai7tPdKw@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.173.1432135585.17265.python-list@python.org>
Lines: 52
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:90959

On 2015-05-20 22:58, Chris Angelico wrote:
> On Wed, May 20, 2015 at 9:44 PM, Parul Mogra <scoria.799@gmail.com>
> wrote:
> > My objective is to create large amount of data files (say a
> > million *.json files), using a pre-existing template file
> > (*.json). Each file would have a unique name, possibly by
> > incorporating time stamp information. The files have to be
> > generated in a folder specified.
[snip] 
> try a simple sequential integer.
> 
> All you'd need would be a loop that creates a bunch of files... most
> of your code will be figuring out what parts of the template need to
> change. Not too difficult.

If you store your template as a Python string-formatting template,
you can just use string-formatting to do your dirty work:


  import random
  HOW_MANY = 1000000
  template = """{
    "some_string": "%(string)s",
    "some_int": %(int)i
    }
    """

  wordlist = [
    word.rstrip()
    for word in open('/usr/share/dict/words')
    ]
  wordlist[:] = [ # just lowercase all-alpha words
    word
    for word in wordlist
    if word.isalpha() and word.islower()
    ]

  for i in xrange(HOW_MANY):
    fname = "data_%08i.json" % i
    with open(fname, "w") as f:
      f.write(template % {
        "string_value": random.choice(wordlist),
        "int_value": random.randint(0, 1000),
        })


-tkc