Path: csiph.com!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!194.109.133.86.MISMATCH!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <cameron@cskk.homeip.net>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'true,': 0.05; 'url:bitbucket': 0.05; 'context': 0.07; 'modify': 0.07; 'subject:file': 0.07; 'subject:same': 0.07; 'caller': 0.09; 'logic': 0.09; 'changes': 0.15; 'windows': 0.15; '(assuming': 0.16; 'accesses': 0.16; 'csv': 0.16; 'defined.': 0.16; 'excludes': 0.16; 'flush': 0.16; 'from:addr:cs': 0.16; 'from:addr:zip.com.au': 0.16; 'from:name:cameron simpson': 0.16; 'message- id:@cskk.homeip.net': 0.16; 'necessary;': 0.16; 'received:211.29': 0.16; 'received:211.29.132': 0.16; 'received:optusnet.com.au': 0.16; 'received:syd.optusnet.com.au': 0.16; 'simpson': 0.16; 'sure.': 0.16; 'url:css': 0.16; 'url:py': 0.16; 'write,': 0.16; 'wrote:': 0.18; 'file,': 0.19; 'example': 0.22; 'header:User- Agent:1': 0.23; 'cheers,': 0.24; 'file.': 0.24; "i've": 0.25; 'options': 0.25; 'holds': 0.26; 'shown': 0.26; 'header:In-Reply- To:1': 0.27; 'database,': 0.30; 'code': 0.31; 'file': 0.32; 'probably': 0.32; 'open': 0.33; 'url:python': 0.33; 'to:name :python-list': 0.33; 'subject:the': 0.34; 'basic': 0.35; 'but': 0.35; 'there': 0.35; 'received:com.au': 0.36; 'scheme': 0.36; 'charset:us-ascii': 0.36; 'url:org': 0.36; 'two': 0.37; 'conditions.': 0.38; 'jason': 0.38; 'mine': 0.38; 'received:211': 0.38; 'needed': 0.38; 'to:addr:python-list': 0.38; 'files': 0.38; 'to:addr:python.org': 0.39; 'release': 0.40; 'how': 0.40; 'ensure': 0.60; 'wife': 0.60; 'here:': 0.62; 'content- disposition:inline': 0.62; 'show': 0.63; 'personal': 0.63; 'such': 0.63; 'more': 0.64; 'close': 0.67; 'approaches': 0.68; 'wish': 0.70; 'cases;': 0.84; 'modify.': 0.84; 'want:': 0.84; 'race': 0.95
Date: Thu, 11 Jul 2013 16:31:58 +1000
From: Cameron Simpson <cs@zip.com.au>
To: python-list <python-list@python.org>
Subject: Re: Concurrent writes to the same file
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CANy1k1iCT3O=o6AwrpaZemzup1dQRwq-vpoperegamtB2psbcA@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
References: <CANy1k1iCT3O=o6AwrpaZemzup1dQRwq-vpoperegamtB2psbcA@mail.gmail.com>
X-Optus-CM-Score: 0
X-Optus-CM-Analysis: v=2.0 cv=Q6eKePKa c=1 sm=1 a=wom5GMh1gUkA:10 a=rn_v9wUuToUA:10 a=kj9zAlcOel0A:10 a=vrnE16BAAAAA:8 a=ZtCCktOnAAAA:8 a=jUyegM5cxxMA:10 a=pGLkceISAAAA:8 a=xtERp6CFAAAA:8 a=R4_8bbkxw8WXxVSbmasA:9 a=CjuIK1q_8ugA:10 a=MSl-tDqOz04A:10 a=OD7RDU0+DEATlv/dZntIcw==:117
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.4568.1373524405.3114.python-list@python.org>
Lines: 57
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1373524405 news.xs4all.nl 15980 [2001:888:2000:d::a6]:42846
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:50418

On 10Jul2013 22:57, Jason Friedman <jsf80238@gmail.com> wrote:
| Other than using a database, what are my options for allowing two processes
| to edit the same file at the same time?  When I say same time, I can accept
| delays.  I considered lock files, but I cannot conceive of how I avoid race
| conditions.

Sure. (Assuming UNIX. Windows probably excludes both processes
having the file open at the one time, but that's not a show stopper.)

You need to use a lock file to ensure only one process accesses the file at a time.

While a process holds the lock, access the file.

There are two basic approaches here: 

  take lock
    open file for write, modify the file, close it
  release lock

  both processes have the file open for update
    take lock
      modify file, flush buffers
    release lock

The code I use to take a lockfile is my "lockfile" context manager:

  https://bitbucket.org/cameron_simpson/css/src/374f650025f156554a986fb3fd472003d2a2519a/lib/python/cs/fileutils.py?at=default#cl-408

An example caller goes:

    with lockfile(self.csvpath):
        backup = "%s.bak-%s" % (self.csvpath, datetime.datetime.now().isoformat())
        copyfile(self.csvpath, backup)
        write_csv_file(self.csvpath, self.nodedb.nodedata())
    if not self.keep_backups:
        os.remove(backup)

as shown here:

  https://bitbucket.org/cameron_simpson/css/src/374f650025f156554a986fb3fd472003d2a2519a/lib/python/cs/nodedb/csvdb.py?at=default#cl-171

Simplify as necessary; you just want:

  with lockfile(mainfile):
    modify(mainfile)
if you have no wish for backups in case of a failed modify.

If both processes need to see each other's changes then there's
some more logic needed to monitor the file for changes. I've got
such a scheme for CSV files in beta at present for a personal
project. It won't suit all use cases; mine is well defined.

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

Is it true, Sen. Bedfellow, that your wife rides with bikers?   - Milo Bloom