Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.037
X-Spam-Evidence: '*H*': 0.93; '*S*': 0.00; 'hour.': 0.09; 'subject:create': 0.09; 'subject:files': 0.09; 'cc:addr:python- list': 0.11; 'creates': 0.14; 'template': 0.14; 'useful,': 0.14; '(say': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'integer.': 0.16; 'sequential': 0.16; 'timestamp': 0.16; 'folder': 0.16; 'wrote:': 0.18; 'wed,': 0.18; 'otherwise,': 0.22; 'cc:addr:python.org': 0.22; 'cc:2**0': 0.24; 'possibly': 0.26; 'header:In-Reply-To:1': 0.27; 'message- id:@mail.gmail.com': 0.30; 'work.': 0.31; 'code': 0.31; 'bunch': 0.31; 'file': 0.32; 'probably': 0.32; 'created': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'possible': 0.36; 'too': 0.37; 'files': 0.38; 'pm,': 0.38; 'most': 0.60; 'simple': 0.61; 'strategy': 0.64; 'within': 0.65; '20,': 0.68; 'million': 0.74; '2015': 0.84; 'hour,': 0.84; 'stamp': 0.91; 'subject:Best': 0.91; 'task,': 0.91; 'to:none': 0.92
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=aHE88QANXL9wQXHcURwz45UshTTy8mkNu0b+P9sGGZY=; b=aw68e4I3a0MT/TvrvCfF8+SfrTXeAJ1kk0skJArmevPFicM3lALiulwbvloRH9yKcE NW80OcxPTgX24gpzzgr6qEyM40XfgJYMYjTN/5sihan3KWMGile2G/MDZwsGx5lBPSnY PWYux/lDK4L8Pnzj32c46aYY30w46UPRg5l3t7hiHeHVU0vSuFIcItC8AJSVcENSMMzf BKzDiJNLYe0vpSDbd5DlxA95uVNuCRZUoh8chbpnntBDxwThZLVGXVfw/NVe9Mib3rVp qznvrY+KnCX9Kiw85jajIcyLeGEbunS3TLQ9cqhShdt2TtjS/urhJRbbVUvvjDhy7JMt poyA==
MIME-Version: 1.0
X-Received: by 10.107.16.32 with SMTP id y32mr42410676ioi.53.1432126685975; Wed, 20 May 2015 05:58:05 -0700 (PDT)
In-Reply-To: <CAPkZ3MS5SiGH9OCe9RSTmakF681O+qM572y49FuDBmBix=aiFg@mail.gmail.com>
References: <CAPkZ3MS5SiGH9OCe9RSTmakF681O+qM572y49FuDBmBix=aiFg@mail.gmail.com>
Date: Wed, 20 May 2015 22:58:05 +1000
Subject: Re: Best approach to create humongous amount of files
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.20+
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.168.1432126695.17265.python-list@python.org>
Lines: 19
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1432126695 news.xs4all.nl 2887 [2001:888:2000:d::a6]:41932
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:90953

On Wed, May 20, 2015 at 9:44 PM, Parul Mogra <scoria.799@gmail.com> wrote:
> My objective is to create large amount of data files (say a million *.json
> files), using a pre-existing template file (*.json). Each file would have a
> unique name, possibly by incorporating time stamp information. The files
> have to be generated in a folder specified.
>
> What is the best strategy to achieve this task, so that the files will be
> generated in the shortest possible time? Say within an hour.

If they're to be created within an hour, timestamp of creation
probably isn't useful, but if you have internal data to use as the
file name, that would work. Otherwise, try a simple sequential
integer.

All you'd need would be a loop that creates a bunch of files... most
of your code will be figuring out what parts of the template need to
change. Not too difficult.

ChrisA