Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #37045 > unrolled thread

Safely add a key to a dict only if it does not already exist?

Back to article view | Back to comp.lang.python

Contents

  Safely add a key to a dict only if it does not already exist? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-19 04:07 +0000
    Re: Safely add a key to a dict only if it does not already exist? Chris Rebert <clp2@rebertia.com> - 2013-01-18 20:15 -0800
      Re: Safely add a key to a dict only if it does not already exist? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-19 10:31 +0000
    Re: Safely add a key to a dict only if it does not already exist? Lie Ryan <lie.1296@gmail.com> - 2013-01-19 16:00 +1100
    Re: Safely add a key to a dict only if it does not already exist? Vito De Tullio <vito.detullio@gmail.com> - 2013-01-19 08:25 +0100
    Re: Safely add a key to a dict only if it does not already exist? Vito De Tullio <vito.detullio@gmail.com> - 2013-01-19 08:27 +0100
    Re: Safely add a key to a dict only if it does not already exist? Mitya Sirenef <msirenef@lightbird.net> - 2013-01-19 02:35 -0500
    Re: Safely add a key to a dict only if it does not already exist? Peter Otten <__peter__@web.de> - 2013-01-19 09:02 +0100
    Re: Safely add a key to a dict only if it does not already exist? Vito De Tullio <vito.detullio@gmail.com> - 2013-01-19 09:16 +0100
    Re: Safely add a key to a dict only if it does not already exist? Mitya Sirenef <msirenef@lightbird.net> - 2013-01-19 03:18 -0500
    Re: Safely add a key to a dict only if it does not already exist? Vito De Tullio <vito.detullio@gmail.com> - 2013-01-19 09:19 +0100
    Re: Safely add a key to a dict only if it does not already exist? Chris Angelico <rosuav@gmail.com> - 2013-01-19 19:23 +1100
    Re: Safely add a key to a dict only if it does not already exist? Roy Smith <roy@panix.com> - 2013-01-19 08:04 -0500

#37045 — Safely add a key to a dict only if it does not already exist?

I wish to add a key to a dict only if it doesn't already exist, but do it 
in a thread-safe manner.

The naive code is:

if key not in dict:
    dict[key] = value


but of course there is a race condition there: it is possible that 
another thread may have added the same key between the check and the 
store.

How can I add a key in a thread-safe manner?

Any solutions must work with built-in dicts, I cannot use a custom dict 
subclass.


-- 
Steven

[toc] | [next] | [standalone]

#37046

[Multipart message — attachments visible in raw view] — view raw

On Friday, January 18, 2013, Steven D'Aprano wrote:

> I wish to add a key to a dict only if it doesn't already exist, but do it
> in a thread-safe manner.
>
> The naive code is:
>
> if key not in dict:
>     dict[key] = value
>
>
> but of course there is a race condition there: it is possible that

another thread may have added the same key between the check and the
> store.
>
> How can I add a key in a thread-safe manner?
>

I'm not entirely sure, but have you investigated dict.setdefault() ?


-- 
Cheers,
Chris
--
http://rebertia.com

[toc] | [prev] | [next] | [standalone]

#37080

On Fri, 18 Jan 2013 20:15:26 -0800, Chris Rebert wrote:

> On Friday, January 18, 2013, Steven D'Aprano wrote:
> 
>> I wish to add a key to a dict only if it doesn't already exist, but do
>> it in a thread-safe manner.
[...]
> I'm not entirely sure, but have you investigated dict.setdefault() ?

Great! That's exactly what I need, thanks to everyone who responded.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#37049

On 19/01/13 15:15, Chris Rebert wrote:
> On Friday, January 18, 2013, Steven D'Aprano wrote:
>
>     I wish to add a key to a dict only if it doesn't already exist, but
>     do it
>     in a thread-safe manner.
>
>     The naive code is:
>
>     if key not in dict:
>          dict[key] = value
>
>
>     but of course there is a race condition there: it is possible that
>
>     another thread may have added the same key between the check and the
>     store.
>
>     How can I add a key in a thread-safe manner?
>
>
> I'm not entirely sure, but have you investigated dict.setdefault() ?

dict.setdefault() was not atomic on older python version, they were made 
atomic in Python 2.7.3 and Python 3.2.3.

See bug13521 in the issue tracker http://bugs.python.org/issue13521

PS: The bug tracker seems down at the moment, so pulled this from 
Google's cache:
https://webcache.googleusercontent.com/search?q=cache:59PO_F-VEfwJ:bugs.python.org/issue13521+&cd=1&hl=en&ct=clnk&client=ubuntu

[toc] | [prev] | [next] | [standalone]

#37060

Steven D'Aprano wrote:

> I wish to add a key to a dict only if it doesn't already exist, but do it
> in a thread-safe manner.
> 
> The naive code is:
> 
> if key not in dict:
>     dict[key] = value
> 
> 
> but of course there is a race condition there: it is possible that
> another thread may have added the same key between the check and the
> store.
> 
> How can I add a key in a thread-safe manner?

using locks?

import threading                                                                                                                                                                                                     

lock = threading.Lock()
with lock:
    if key not in dict:
        dict[key] = value


-- 
ZeD

[toc] | [prev] | [next] | [standalone]

#37061

Chris Rebert wrote:

>> How can I add a key in a thread-safe manner?
> I'm not entirely sure, but have you investigated dict.setdefault() ?

but how setdefault makes sense in this context? It's used to set a default 
value when you try to retrieve an element from the dict, not when you try to 
set a new one ...

-- 
ZeD

[toc] | [prev] | [next] | [standalone]

#37063

On 01/19/2013 02:27 AM, Vito De Tullio wrote:
> Chris Rebert wrote:
>
>>> How can I add a key in a thread-safe manner?
>> I'm not entirely sure, but have you investigated dict.setdefault() ?
> but how setdefault makes sense in this context? It's used to set a default
> value when you try to retrieve an element from the dict, not when you try to
> set a new one ...
>

I guess setdefault with a sentinel default value, then set to your
new value if d[k] is sentinel?

  - mitya


-- 
Lark's Tongue Guide to Python: http://lightbird.net/larks/

[toc] | [prev] | [next] | [standalone]

#37066

Vito De Tullio wrote:

> Chris Rebert wrote:
> 
>>> How can I add a key in a thread-safe manner?
>> I'm not entirely sure, but have you investigated dict.setdefault() ?
> 
> but how setdefault makes sense in this context? It's used to set a default
> value when you try to retrieve an element from the dict, not when you try
> to set a new one ...

But it also sets the value if the key is not found:

"""
setdefault(...)
    D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D
"""

>>> d = {}
>>> d.setdefault(1, 2)
2
>>> d
{1: 2}
>>> d.setdefault(1, 3)
2
>>> d
{1: 2}


It has been suggested that get_or_set() would have been a better name for 
that functionality...

[toc] | [prev] | [next] | [standalone]

#37068

Peter Otten wrote:

>>>> How can I add a key in a thread-safe manner?
>>> I'm not entirely sure, but have you investigated dict.setdefault() ?
>> 
>> but how setdefault makes sense in this context? It's used to set a
>> default value when you try to retrieve an element from the dict, not when
>> you try to set a new one ...
> 
> But it also sets the value if the key is not found:

yeah, sure, but with a fixed value :)

I mean: if the value is not important, why bother at all trying not to 
override it with an if or a lock or other tecniques? doing

    d['key'] = 'fixed_value'

multiple times in different threads is not a problem in my eyes...

-- 
ZeD

[toc] | [prev] | [next] | [standalone]

#37069

On 01/19/2013 02:35 AM, Mitya Sirenef wrote:
> On 01/19/2013 02:27 AM, Vito De Tullio wrote:
>> Chris Rebert wrote:
>>
>>>> How can I add a key in a thread-safe manner?
>>> I'm not entirely sure, but have you investigated dict.setdefault() ?
>> but how setdefault makes sense in this context? It's used to set a 
>> default
>> value when you try to retrieve an element from the dict, not when you 
>> try to
>> set a new one ...
>>
>
> I guess setdefault with a sentinel default value, then set to your
> new value if d[k] is sentinel?
>
>  - mitya
>
>

Er, that makes no sense.. just setdefault to desired value. -m



-- 
Lark's Tongue Guide to Python: http://lightbird.net/larks/

[toc] | [prev] | [next] | [standalone]

#37070

Peter Otten wrote:

uhhmm.. I think I misread the example

>>>> d = {}
>>>> d.setdefault(1, 2)
> 2
>>>> d
> {1: 2}
>>>> d.setdefault(1, 3)
> 2
>>>> d
> {1: 2}

yeah, sure it can be useful for the OP...

-- 
ZeD

[toc] | [prev] | [next] | [standalone]

#37071

On Sat, Jan 19, 2013 at 7:16 PM, Vito De Tullio <vito.detullio@gmail.com> wrote:
> yeah, sure, but with a fixed value :)
>
> I mean: if the value is not important, why bother at all trying not to
> override it with an if or a lock or other tecniques? doing
>
>     d['key'] = 'fixed_value'
>
> multiple times in different threads is not a problem in my eyes...

How fixed is fixed?

>>> d={}
>>> d.setdefault("foo",1)
1
>>> d.setdefault("bar",2)
2
>>> d
{'bar': 2, 'foo': 1}

If list append is guaranteed atomic, and since setdefault is
apparently atomic, then this would be a thread-safe way to maintain a
dictionary of lists:

>>> d={}
>>> lst=d.setdefault("foo",[])
>>> lst.append(1)

Are those guarantees made?

ChrisA

[toc] | [prev] | [next] | [standalone]

#37085

In article <50fa1bf1$0$30003$c3e8da3$5496439d@news.astraweb.com>,
 Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> I wish to add a key to a dict only if it doesn't already exist, but do it 
> in a thread-safe manner.
> 
> The naive code is:
> 
> if key not in dict:
>     dict[key] = value
> 
> 
> but of course there is a race condition there: it is possible that 
> another thread may have added the same key between the check and the 
> store.
> 
> How can I add a key in a thread-safe manner?

You want something along the lines of:

from threading import Lock
lock = Lock()
[...]
lock.acquire()
if key not in dict:
   dict[key] = value
lock.release()

You probably want to wrap that up in a context manager to ensure the 
lock is released if you get an exception.  You don't want your entire 
program to hang just because somebody handed you a key which wasn't 
hashable, for example.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web