Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #65736 > unrolled thread

system wide mutex

Back to article view | Back to comp.lang.python

Contents

  system wide mutex Asaf Las <roegltd@gmail.com> - 2014-02-09 02:39 -0800
    Re: system wide mutex Asaf Las <roegltd@gmail.com> - 2014-02-09 03:00 -0800
    Re: system wide mutex Skip Montanaro <skip@pobox.com> - 2014-02-09 05:00 -0600
      Re: system wide mutex Asaf Las <roegltd@gmail.com> - 2014-02-09 03:07 -0800
      Re: system wide mutex Roy Smith <roy@panix.com> - 2014-02-09 09:45 -0500
        Re: system wide mutex Skip Montanaro <skip@pobox.com> - 2014-02-09 09:40 -0600
        Re: system wide mutex Chris Angelico <rosuav@gmail.com> - 2014-02-10 09:09 +1100
    Re: system wide mutex Marko Rauhamaa <marko@pacujo.net> - 2014-02-09 18:14 +0200
      Re: system wide mutex Asaf Las <roegltd@gmail.com> - 2014-02-09 09:39 -0800
    Re: system wide mutex Grant Edwards <invalid@invalid.invalid> - 2014-02-10 17:00 +0000
    Re: system wide mutex Miki Tebeka <miki.tebeka@gmail.com> - 2014-02-10 09:46 -0800

#65736 — system wide mutex

Hi 

Which one is most recommended to use for mutex alike locking to 
achieve atomic access to single resource:

- fcntl.lockf
- os.open() with O_SHLOCK and O_EXLOCK 
- https://pypi.python.org/pypi/lockfile/0.9.1
- https://pypi.python.org/pypi/zc.lockfile/1.1.0
- any other ?

Thanks 

/Asaf

[toc] | [next] | [standalone]

#65739

Forget to mentioned - CentOS 6.5 Python v3.3.

[toc] | [prev] | [next] | [standalone]

#65740

> Which one is most recommended to use for mutex alike locking to
> achieve atomic access to single resource:
>
> - fcntl.lockf
> - os.open() with O_SHLOCK and O_EXLOCK
> - https://pypi.python.org/pypi/lockfile/0.9.1
> - https://pypi.python.org/pypi/zc.lockfile/1.1.0
> - any other ?

As the author of lockfile, I can tell you it only implements advisory
locking. All programs needing to access the locked resources must
cooperate. It also has bugs which have been reported which I have yet
to spend any time fixing.

Beyond that, your question isn't really detailed enough to answer
completely. You don't identify what sort of systems you need this to
work on (Windows, Mac, various flavors of Unix?), whether programs
written in other languages will be involved (you did say "system
wide"), and whether you need to use it in the face of network file
systems like NFS or Samba.

Skip

[toc] | [prev] | [next] | [standalone]

#65742

On Sunday, February 9, 2014 1:00:39 PM UTC+2, Skip Montanaro wrote:
> > Which one is most recommended to use for mutex alike locking to
> > achieve atomic access to single resource:
> >
> > - fcntl.lockf
> > - os.open() with O_SHLOCK and O_EXLOCK
> > - https://pypi.python.org/pypi/lockfile/0.9.1
> > - https://pypi.python.org/pypi/zc.lockfile/1.1.0
> > - any other ?
> 
> As the author of lockfile, I can tell you it only implements advisory
> locking. All programs needing to access the locked resources must
> cooperate. It also has bugs which have been reported which I have yet
> to spend any time fixing.
> Beyond that, your question isn't really detailed enough to answer
> completely. You don't identify what sort of systems you need this to
> work on (Windows, Mac, various flavors of Unix?), whether programs
> written in other languages will be involved (you did say "system
> wide"), and whether you need to use it in the face of network file
> systems like NFS or Samba.
> Skip

Hi Yes i realized that missed info. The lock is needed
for 2 independently spawned processes to update set of files and non 
non thread safe db 

/Asaf

[toc] | [prev] | [next] | [standalone]

#65763

In article <mailman.6578.1391943647.18130.python-list@python.org>,
 Skip Montanaro <skip@pobox.com> wrote:

> > Which one is most recommended to use for mutex alike locking to
> > achieve atomic access to single resource:
> >
> > - fcntl.lockf
> > - os.open() with O_SHLOCK and O_EXLOCK
> > - https://pypi.python.org/pypi/lockfile/0.9.1
> > - https://pypi.python.org/pypi/zc.lockfile/1.1.0
> > - any other ?
> 
> As the author of lockfile, I can tell you it only implements advisory
> locking. All programs needing to access the locked resources must
> cooperate.

This is true of all mutexes, no?

We needed a mutex that worked across multiple machines, so we ended up 
implementing a mutex in memcache.  We based ours on 
http://tinyurl.com/lybx2kl, but if you google for "memcache mutex 
python", you'll find a number of implementations.

[toc] | [prev] | [next] | [standalone]

#65765

On Sun, Feb 9, 2014 at 8:45 AM, Roy Smith <roy@panix.com> wrote:
> This is true of all mutexes, no?

Hmmm... You might well be right. I thought that use of the O_EXLOCK
flag in the open(2) system call would prevent other processes from
opening the file, but (at least on my Mac) it just blocks until the
first process closes the file. Perhaps there are some other flags you
can use to cause the second open(2) call to error out.

Skip

[toc] | [prev] | [next] | [standalone]

#65783

On Mon, Feb 10, 2014 at 1:45 AM, Roy Smith <roy@panix.com> wrote:
> In article <mailman.6578.1391943647.18130.python-list@python.org>,
>  Skip Montanaro <skip@pobox.com> wrote:
>
>> As the author of lockfile, I can tell you it only implements advisory
>> locking. All programs needing to access the locked resources must
>> cooperate.
>
> This is true of all mutexes, no?
>

Not quite all; mandatory locks actually prevent other access. But they
require support from whatever makes the resource available - most
commonly, the file system - where cooperative locking doesn't. I've
written a number of cooperative locking systems in the past (eg to
protect a network resource, when the mounted file system didn't offer
the sort of locking I needed), and they're usually the way to go when
there's anything complicated going on.

What, all mutexes?
Yes, all mutexes!
What, all?
Well.... nearly all!

-- with apologies to WS Gilbert, HMS Pinafore, and the Pirates of Penzance

ChrisA

[toc] | [prev] | [next] | [standalone]

#65770

Asaf Las <roegltd@gmail.com>:

> Which one is most recommended to use for mutex alike locking to 
> achieve atomic access to single resource:
>
> - fcntl.lockf

I recommend fcntl.flock:

========================================================================
#!/usr/bin/python3

import sys, fcntl, time

with open("test.lock", "w+") as lock:
    fcntl.flock(lock.fileno(), fcntl.LOCK_EX)
    # begin critical
    sys.stdout.write("hello\n")
    time.sleep(3)
    sys.stdout.write("world\n")
    # end critical
========================================================================


Marko

[toc] | [prev] | [next] | [standalone]

#65774

Hi 

Thanks for replies. It would be good to have blocking implementation.
I have to check fcntl if it works in blocking mdoe on CentOS.

Meanwhile there is Posix Semaphore made for Python:
http://semanchuk.com/philip/posix_ipc/

will try it as well.

/Asaf

[toc] | [prev] | [next] | [standalone]

#65833

On 2014-02-09, Asaf Las <roegltd@gmail.com> wrote:
> Hi 
>
> Which one is most recommended to use for mutex alike locking to 
> achieve atomic access to single resource:
>
> - fcntl.lockf
> - os.open() with O_SHLOCK and O_EXLOCK 
> - https://pypi.python.org/pypi/lockfile/0.9.1
> - https://pypi.python.org/pypi/zc.lockfile/1.1.0
> - any other ?

Posix "pthread" mutexes shared memory.

-- 
Grant Edwards               grant.b.edwards        Yow! Give them RADAR-GUIDED
                                  at               SKEE-BALL LANES and
                              gmail.com            VELVEETA BURRITOS!!

[toc] | [prev] | [next] | [standalone]

#65835

IIRC creating a directory is atomic in most environments.

On Sunday, February 9, 2014 2:39:51 AM UTC-8, Asaf Las wrote:
> Hi 
> 
> 
> 
> Which one is most recommended to use for mutex alike locking to 
> 
> achieve atomic access to single resource:
> 
> 
> 
> - fcntl.lockf
> 
> - os.open() with O_SHLOCK and O_EXLOCK 
> 
> - https://pypi.python.org/pypi/lockfile/0.9.1
> 
> - https://pypi.python.org/pypi/zc.lockfile/1.1.0
> 
> - any other ?
> 
> 
> 
> Thanks 
> 
> 
> 
> /Asaf

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web