Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #88385 > unrolled thread

Lockfile hanling

Back to article view | Back to comp.lang.python

Contents

  Lockfile hanling Ervin Hegedüs <airween@gmail.com> - 2015-03-31 16:50 +0200
    Re: Lockfile hanling Christian Gollwitzer <auriocus@gmx.de> - 2015-03-31 20:58 +0200
      Re: Lockfile hanling Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-31 13:17 -0600
        Re: Lockfile hanling Christian Gollwitzer <auriocus@gmx.de> - 2015-03-31 21:59 +0200
          Re: Lockfile hanling Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-31 14:12 -0600
          Re: Lockfile hanling Sturla Molden <sturla.molden@gmail.com> - 2015-04-01 23:51 +0000

#88385 — Lockfile hanling

Hello,

there is an app, written in Python, which stores few bytes of
datas in a single file. The application uses threads. Every
thread can modify the file, but only one at a time. I'm using a
lock file to prevent the multiple access.

Here is the lock method:

  while True:
      try:
          fl = os.open(self.lockfile, os.O_CREAT | os.O_EXCL | os.O_RDWR)
      except OSError, e:
          if e.errno != errno.EEXIST:
              raise
          time.sleep(0.2)
          continue
      except:
          syslog.syslog(syslog.LOG_DEBUG, "Sync error: " + str(sys.exc_info()[1]))
      else:
          break

This works as well for me - about 3-4 weeks. After some weeks, I
got this error:

OSError: [Errno 24] Too many open files: '/var/spool/myapp/queue.lock'


Today the app had been restarted, about 3-4 hours ago. Now I see
under the proc/PID/fd:

lrwx------ 1 root     root     64 márc 31 16.45 5 -> /var/spool/myapp/queue.lock (deleted)

there are about 50 deleted FD's. After few weeks the process
reaches the number if max fd's.

How can I prevent or avoid this issue? What's the correct way to
handle the lockfile in Python?


Thanks,


Ervin

[toc] | [next] | [standalone]

#88398

Am 31.03.15 um 16:50 schrieb Ervin Hegedüs:
> there is an app, written in Python, which stores few bytes of
> datas in a single file. The application uses threads. Every
> thread can modify the file, but only one at a time. I'm using a
> lock file to prevent the multiple access.
>
> Here is the lock method:
>
>[...]
> This works as well for me - about 3-4 weeks. After some weeks, I
> got this error:
>
> OSError: [Errno 24] Too many open files: '/var/spool/myapp/queue.lock'
>

Can you try if fcntl.flock() does what you want? Should be better than 
inventing your own locking mechanism.

	Christian

[toc] | [prev] | [next] | [standalone]

#88399

On Tue, Mar 31, 2015 at 12:58 PM, Christian Gollwitzer <auriocus@gmx.de> wrote:
> Am 31.03.15 um 16:50 schrieb Ervin Hegedüs:
>>
>> there is an app, written in Python, which stores few bytes of
>> datas in a single file. The application uses threads. Every
>> thread can modify the file, but only one at a time. I'm using a
>> lock file to prevent the multiple access.
>>
>> Here is the lock method:
>>
>> [...]
>> This works as well for me - about 3-4 weeks. After some weeks, I
>> got this error:
>>
>> OSError: [Errno 24] Too many open files: '/var/spool/myapp/queue.lock'
>>
>
> Can you try if fcntl.flock() does what you want? Should be better than
> inventing your own locking mechanism.

flock locks are per-process; they won't help for synchronizing access
between competing threads in the same process.

[toc] | [prev] | [next] | [standalone]

#88401

Am 31.03.15 um 21:17 schrieb Ian Kelly:
> On Tue, Mar 31, 2015 at 12:58 PM, Christian Gollwitzer <auriocus@gmx.de> wrote:
>> Am 31.03.15 um 16:50 schrieb Ervin Hegedüs:
>>>
>>> there is an app, written in Python, which stores few bytes of
>>> datas in a single file. The application uses threads. Every
>>> thread can modify the file, but only one at a time. I'm using a
>>> lock file to prevent the multiple access.
>>>
>>> Here is the lock method:
>>>
>>> [...]
>>> This works as well for me - about 3-4 weeks. After some weeks, I
>>> got this error:
>>>
>>> OSError: [Errno 24] Too many open files: '/var/spool/myapp/queue.lock'
>>>
>>
>> Can you try if fcntl.flock() does what you want? Should be better than
>> inventing your own locking mechanism.
>
> flock locks are per-process; they won't help for synchronizing access
> between competing threads in the same process.
>

Ok. But if it is really all in one process, then a classical mutex would 
do, wouldn't it? I'm not experienced with threading in Python, but it 
seems from the documentation, that threading.Lock() works like a mutex.

	Christian

[toc] | [prev] | [next] | [standalone]

#88402

On Tue, Mar 31, 2015 at 1:59 PM, Christian Gollwitzer <auriocus@gmx.de> wrote:
> Am 31.03.15 um 21:17 schrieb Ian Kelly:
>> flock locks are per-process; they won't help for synchronizing access
>> between competing threads in the same process.
>>
>
> Ok. But if it is really all in one process, then a classical mutex would do,
> wouldn't it? I'm not experienced with threading in Python, but it seems from
> the documentation, that threading.Lock() works like a mutex.

As long as there's not *also* some other external process that needs
to access the file occasionally. :-)

[toc] | [prev] | [next] | [standalone]

#88436

Ian Kelly <ian.g.kelly@gmail.com> wrote:

> As long as there's not *also* some other external process that needs
> to access the file occasionally. :-)

Then there is multiprocessing.Lock :)

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web