Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #51147 > unrolled thread

Re: Python 3: dict & dict.keys()

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Python 3: dict & dict.keys() Ethan Furman <ethan@stoneleaf.us> - 2013-07-24 08:57 -0700
    Re: Python 3: dict & dict.keys() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-25 05:48 +0000
      Re: Python 3: dict & dict.keys() Chris Angelico <rosuav@gmail.com> - 2013-07-25 16:02 +1000
        Re: Python 3: dict & dict.keys() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-25 07:27 +0000
          Re: Python 3: dict & dict.keys() Chris Angelico <rosuav@gmail.com> - 2013-07-25 18:15 +1000
            Re: Python 3: dict & dict.keys() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-25 09:44 +0000
              Re: Python 3: dict & dict.keys() Chris Angelico <rosuav@gmail.com> - 2013-07-25 20:34 +1000
                Re: Python 3: dict & dict.keys() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-25 14:57 +0000
                  Re: Python 3: dict & dict.keys() Chris Angelico <rosuav@gmail.com> - 2013-07-26 01:07 +1000
      Re: Python 3: dict & dict.keys() Johannes Bauer <dfnsonfsduifb@gmx.de> - 2013-07-25 15:22 +0200
      Re: Python 3: dict & dict.keys() Ethan Furman <ethan@stoneleaf.us> - 2013-07-25 06:57 -0700

#51147 — Re: Python 3: dict & dict.keys()

On 07/24/2013 05:51 AM, Oscar Benjamin wrote:
>
> On Jul 24, 2013 7:25 AM, "Peter Otten" <__peter__@web.de <mailto:peter__@web.de>> wrote:
>>
>> Ethan Furman wrote:
>>
>> > So, my question boils down to:  in Python 3 how is dict.keys() different
>> > from dict?  What are the use cases?
>>
>> I just grepped through /usr/lib/python3, and could not identify a single
>> line where some_object.keys() wasn't either wrapped in a list (or set,
>> sorted, max) call, or iterated over.
>>
>> To me it looks like views are a solution waiting for a problem.
>
> What do you mean? Why would you want to create a temporary list just to iterate over it explicitly or implicitly (set,
> sorted, max,...)?

You wouldn't.  But you don't need .keys() for that either as you can just use the dict itself.

My point is that in 2.x .keys() did something different from the dict, while in 3.x it appears to me that they are the same.

Peter's point is that in the stdlib the new functionality of .keys() is never used, not even once.

--
~Ethan~

[toc] | [next] | [standalone]

#51187

On Wed, 24 Jul 2013 08:57:11 -0700, Ethan Furman wrote:

> My point is that in 2.x .keys() did something different from the dict,
> while in 3.x it appears to me that they are the same.

Then you aren't looking very closely. d.keys() returns a set-like view 
into the dict, which is great for comparing elements:


py> d1 = dict.fromkeys([1, 2, 3, 4])
py> d2 = dict.fromkeys([3, 4, 5, 6])
py> d1.keys() & d2.keys()  # keys that are in both
{3, 4}
py> d1.keys() ^ d2.keys()  # keys not in both
{1, 2, 5, 6}
py> d1.keys() - d2.keys()  # keys only in d1
{1, 2}
py> d2.keys() - d1.keys()  # keys only in d2
{5, 6}


Dicts aren't sets, and don't support set methods:

py> d1 - d2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for -: 'dict' and 'dict'


> Peter's point is that in the stdlib the new functionality of .keys() is
> never used, not even once.

The standard library is not the universe of Python code, and most of the 
standard library predates Python 3. Some of it goes back to Python 1 
idioms. In general, working code doesn't get updated until it stops 
working.

I have code that manually walks over each dict and extracts keys that are 
in both, or one but not the other. Once I drop support for Python 2.6, I 
throw that code away and just use views. But until then, I'm stuck doing 
it the horrible way. Judging by a naive grep of my code, you might think 
I never used views. I do, I just have to reinvent the wheel.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#51193

On Thu, Jul 25, 2013 at 3:48 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Dicts aren't sets, and don't support set methods:
>
> py> d1 - d2
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: unsupported operand type(s) for -: 'dict' and 'dict'

I wouldn't take this as particularly significant, though. A future
version of Python could add that support (and it might well be very
useful), without breaking any of the effects of views.

ChrisA

[toc] | [prev] | [next] | [standalone]

#51201

On Thu, 25 Jul 2013 16:02:42 +1000, Chris Angelico wrote:

> On Thu, Jul 25, 2013 at 3:48 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> Dicts aren't sets, and don't support set methods:
>>
>> py> d1 - d2
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> TypeError: unsupported operand type(s) for -: 'dict' and 'dict'
> 
> I wouldn't take this as particularly significant, though. A future
> version of Python could add that support (and it might well be very
> useful), without breaking any of the effects of views.

I don't think dicts can ever support set methods, since *they aren't 
sets*. Every element consists of both a key and a value, so you have to 
consider both. Set methods are defined in terms of singleton elements, 
not binary elements, so before you even begin, you have to decide what 
does it mean when two elements differ in only one of the two parts?

Given dicts {1: 'a'}, {1: 'b'}, what is the union of them? I can see five 
possibilities:

{1: 'a'}
{1: 'b'}
{1: ['a', 'b']}
{1: set(['a', 'b'])}
Error

Each of the five results may be what you want in some circumstances. It 
would be a stupid thing for dict.union to pick one behaviour and make it 
the One True Way to perform union on two dicts.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#51206

On Thu, Jul 25, 2013 at 5:27 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Thu, 25 Jul 2013 16:02:42 +1000, Chris Angelico wrote:
>
>> On Thu, Jul 25, 2013 at 3:48 PM, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote:
>>> Dicts aren't sets, and don't support set methods:
>>>
>>> py> d1 - d2
>>> Traceback (most recent call last):
>>>   File "<stdin>", line 1, in <module>
>>> TypeError: unsupported operand type(s) for -: 'dict' and 'dict'
>>
>> I wouldn't take this as particularly significant, though. A future
>> version of Python could add that support (and it might well be very
>> useful), without breaking any of the effects of views.
>
> I don't think dicts can ever support set methods, since *they aren't
> sets*. Every element consists of both a key and a value, so you have to
> consider both. Set methods are defined in terms of singleton elements,
> not binary elements, so before you even begin, you have to decide what
> does it mean when two elements differ in only one of the two parts?
>
> Given dicts {1: 'a'}, {1: 'b'}, what is the union of them? I can see five
> possibilities:
>
> {1: 'a'}
> {1: 'b'}
> {1: ['a', 'b']}
> {1: set(['a', 'b'])}
> Error
>
> Each of the five results may be what you want in some circumstances. It
> would be a stupid thing for dict.union to pick one behaviour and make it
> the One True Way to perform union on two dicts.

That's true, but we already have that issue with sets. What's the
union of {0} and {0.0}? Python's answer: It depends on the order of
the operands.

>>> i={0}
>>> f={0.0}
>>> i | f
{0}
>>> f | i
{0.0}

I would say that Python can freely pick from the first two options you
offered (either keep-first or keep-last), most likely the first one,
and it'd make good sense. Your third option would be good for a few
specific circumstances, but then you probably would also want the
combination of {1:'a'} and {1:'a'} to be {1:['a','a']} for
consistency. This would make a good utility function, but isn't what
I'd normally see set union doing. Similarly with the fourth option,
though there it's a little more arguable. Raising an error would work,
but is IMO unnecessary.

(Pike has dictionary operations, but has made different choices. For
instance, 0 and 0.0 are considered distinct, so a set can contain
both. Mappings (dicts) merge by keeping the last, not the first. But
the specifics don't much matter.)

A Python set already has to distinguish between object value and
object identity; a dict simply adds a bit more distinction between
otherwise-considered-identical keys, namely their values.

>>> a="This is a test."
>>> b="This is a test."
>>> a is b
False
>>> id(a)
16241512
>>> id(b)
16241392
>>> id(({a}|{b}).pop())
16241512

Assuming a and b have different ids, which is true in the above
example of Py3.3 on Windows, the set union *must* be different from
one of them. Suppose you do a dictionary of id(key) -> value, and a
set of the keys themselves. You could then do set operations on the
keys, and then go and retrieve the values.

Sure, maybe the way of doing things won't be exactly what everyone
expects... but it works, and it makes plausible sense. And as a
theoretical "might be implemented in Python 3.5", it still has no
impact on views, beyond that there are some operations which must be
done with views in <=3.3 that could be done on the dicts themselves in
future.

ChrisA

[toc] | [prev] | [next] | [standalone]

#51210

On Thu, 25 Jul 2013 18:15:22 +1000, Chris Angelico wrote:

> On Thu, Jul 25, 2013 at 5:27 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> On Thu, 25 Jul 2013 16:02:42 +1000, Chris Angelico wrote:
>>
>>> On Thu, Jul 25, 2013 at 3:48 PM, Steven D'Aprano
>>> <steve+comp.lang.python@pearwood.info> wrote:
>>>> Dicts aren't sets, and don't support set methods:
>>>>
>>>> py> d1 - d2
>>>> Traceback (most recent call last):
>>>>   File "<stdin>", line 1, in <module>
>>>> TypeError: unsupported operand type(s) for -: 'dict' and 'dict'
>>>
>>> I wouldn't take this as particularly significant, though. A future
>>> version of Python could add that support (and it might well be very
>>> useful), without breaking any of the effects of views.
>>
>> I don't think dicts can ever support set methods, since *they aren't
>> sets*. Every element consists of both a key and a value, so you have to
>> consider both. Set methods are defined in terms of singleton elements,
>> not binary elements, so before you even begin, you have to decide what
>> does it mean when two elements differ in only one of the two parts?
>>
>> Given dicts {1: 'a'}, {1: 'b'}, what is the union of them? I can see
>> five possibilities:
>>
>> {1: 'a'}
>> {1: 'b'}
>> {1: ['a', 'b']}
>> {1: set(['a', 'b'])}
>> Error
>>
>> Each of the five results may be what you want in some circumstances. It
>> would be a stupid thing for dict.union to pick one behaviour and make
>> it the One True Way to perform union on two dicts.
> 
> That's true, but we already have that issue with sets. What's the union
> of {0} and {0.0}? Python's answer: It depends on the order of the
> operands.

That's a side-effect of how numeric equality works in Python. Since 0 == 
0.0, you can't have both as keys in the same dict, or set. Indeed, the 
same numeric equality issue occurs here:

py> from fractions import Fraction
py> [0, 2.5] == [0.0, Fraction(5, 2)]
True

So nothing really to do with sets or dicts specifically. 

Aside: I think the contrary behaviour is, well, contrary. It would be 
strange and disturbing to do this:

for key in some_dict:
    if key == 0:
        print("found")
        print(some_dict[key])

and have the loop print "found" and then have the key lookup fail, but 
apparently that's how things work in Pike :-(


> I would say that Python can freely pick from the first two options you
> offered (either keep-first or keep-last), most likely the first one, and
> it'd make good sense. Your third option would be good for a few specific
> circumstances, but then you probably would also want the combination of
> {1:'a'} and {1:'a'} to be {1:['a','a']} for consistency.

Okay, that's six variations. And no, I don't think the "consistency" 
argument is right -- the idea is that you can have multiple values per 
key. Since 'a' == 'a', that's only one value, not two.

The variation using a list, versus the set, depends on whether you care 
about order or hashability.


[...]
> Raising an error would work, but is IMO unnecessary.

I believe that's the only reasonable way for a dict union method to work. 
As the Zen says:

In the face of ambiguity, refuse the temptation to guess.

Since there is ambiguity which value should be associated with the key, 
don't guess.


[...]
> A Python set already has to distinguish between object value and object
> identity; a dict simply adds a bit more distinction between
> otherwise-considered-identical keys, namely their values.

Object identity is a red herring. It would be perfectly valid for a 
Python implementation to create new instances of each element in the set 
union, assuming such creation was free of side-effects (apart from memory 
usage and time, naturally). set.union() makes no promise about the 
identity of elements, and it is defined the same way for languages where 
object identity does not exist (say, old-school Pascal).


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#51213

On Thu, Jul 25, 2013 at 7:44 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Thu, 25 Jul 2013 18:15:22 +1000, Chris Angelico wrote:
>> That's true, but we already have that issue with sets. What's the union
>> of {0} and {0.0}? Python's answer: It depends on the order of the
>> operands.
>
> That's a side-effect of how numeric equality works in Python. Since 0 ==
> 0.0, you can't have both as keys in the same dict, or set. Indeed, the
> same numeric equality issue occurs here:
>
> py> from fractions import Fraction
> py> [0, 2.5] == [0.0, Fraction(5, 2)]
> True
>
> So nothing really to do with sets or dicts specifically.

Here's how I imagine set/dict union:
1) Take a copy of the first object
2) Iterate through the second. If the key doesn't exist in the result, add it.

This works just fine even when "add it" means "store this value
against this key". The dict's value and the object's identity are both
ignored, and you simply take the first one you find.

> Aside: I think the contrary behaviour is, well, contrary. It would be
> strange and disturbing to do this:
>
> for key in some_dict:
>     if key == 0:
>         print("found")
>         print(some_dict[key])
>
> and have the loop print "found" and then have the key lookup fail, but
> apparently that's how things work in Pike :-(

I agree, that would be very strange and disturbing. I mentioned that
aspect merely in passing, but the reason for the difference is not an
oddity of key lookup, but a different decision about float and int: in
Pike, 0 and 0.0 are not equal. (Nor are 1 and 1.0, in case you thought
this was a weirdness of zero.) It's a debatable point; are we trying
to say that all numeric types represent real numbers, and are equal if
they represent the same real number? Or are different representations
distinct, just as much as the string "0" is different from the integer
0? Pike took the latter approach. PHP took the former approach to its
illogical extreme, that the string "0001E1" is equal to "000010" (both
strings). No, the dictionary definitely needs to use object equality
to do its lookup, although I could well imagine an implementation that
runs orders of magnitude faster when object identity can be used.

>> I would say that Python can freely pick from the first two options you
>> offered (either keep-first or keep-last), most likely the first one, and
>> it'd make good sense. Your third option would be good for a few specific
>> circumstances, but then you probably would also want the combination of
>> {1:'a'} and {1:'a'} to be {1:['a','a']} for consistency.
>
> Okay, that's six variations. And no, I don't think the "consistency"
> argument is right -- the idea is that you can have multiple values per
> key. Since 'a' == 'a', that's only one value, not two.

Well, it depends what you're doing with the merging of the dicts. But
all of these extra ways to do things would be explicitly-named
functions with much rarer usage (and quite possibly not part of the
standard library, they'd be snippets shared around and put directly in
application code).

>> Raising an error would work, but is IMO unnecessary.
>
> I believe that's the only reasonable way for a dict union method to work.
> As the Zen says:
>
> In the face of ambiguity, refuse the temptation to guess.
>
> Since there is ambiguity which value should be associated with the key,
> don't guess.

There's already ambiguity as to which of two equal values should be
retained by the set. Python takes the first. Is that guessing? Is that
violating the zen? I don't see a problem with the current set
implementation, and I also don't see a problem with using that for
dict merging.

> Object identity is a red herring. It would be perfectly valid for a
> Python implementation to create new instances of each element in the set
> union, assuming such creation was free of side-effects (apart from memory
> usage and time, naturally). set.union() makes no promise about the
> identity of elements, and it is defined the same way for languages where
> object identity does not exist (say, old-school Pascal).

That still doesn't deal with the "which type should the new object
be". We're back to this question: What is the union of 0 and 0.0?

>>> {0} | {0.0}
{0}
>>> {0.0} | {0}
{0.0}

Maybe Python could create a brand new object, but would it be an int
or a float? The only way I could imagine this working is with a
modified-set class that takes an object constructor, and passes every
object through it. That way, you could have set(float) that coerces
everything to float on entry, which would enforce what you're saying
(even down to potentially creating a new object with a new id, though
float() seems to return a float argument unchanged in CPython 3.3).
Would that really help anything, though? Do we gain anything by not
simply accepting, in the manner of Colonel Fairfax, the first that
comes?

ChrisA

[toc] | [prev] | [next] | [standalone]

#51230

On Thu, 25 Jul 2013 20:34:23 +1000, Chris Angelico wrote:

> On Thu, Jul 25, 2013 at 7:44 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> On Thu, 25 Jul 2013 18:15:22 +1000, Chris Angelico wrote:
>>> That's true, but we already have that issue with sets. What's the
>>> union of {0} and {0.0}? Python's answer: It depends on the order of
>>> the operands.
>>
>> That's a side-effect of how numeric equality works in Python. Since 0
>> == 0.0, you can't have both as keys in the same dict, or set. Indeed,
>> the same numeric equality issue occurs here:
>>
>> py> from fractions import Fraction
>> py> [0, 2.5] == [0.0, Fraction(5, 2)] True
>>
>> So nothing really to do with sets or dicts specifically.
> 
> Here's how I imagine set/dict union:
> 1) Take a copy of the first object
> 2) Iterate through the second. If the key doesn't exist in the result,
> add it.

That's because you're too much of a programmer to step away from the 
implementation. Fundamentally, set union has nothing to do with objects, 
or bit strings, or any concrete implementation. Sets might be infinite, 
and "take a copy" impossible or meaningless.

Logically, the union of set A and set B is the set containing every 
element which is in A, every element in B, and no element which is not. 
How you assemble those elements in a concrete implementation is, in a 
sense, irrelevant. In old-school Pascal, the universe of possible 
elements is taken from the 16-bit, or 32-bit if you're lucky, integers; 
in Python, it's taken from hashable objects. Even using your suggested 
algorithm above, since union is symmetric, it should make no difference 
whether you start with the first, or with the second.


> This works just fine even when "add it" means "store this value against
> this key". The dict's value and the object's identity are both ignored,
> and you simply take the first one you find.

I don't believe that "works", since the whole point of dicts is to store 
the values. In practice, the values are more important than the keys. The 
key only exists so you can get to the value -- the key is equivalent to 
the index in a list, the value to the value at that index. We normally 
end up doing something like "print adict[key]", not "print key". So 
throwing away the values just because they happen to have the same key is 
a fairly dubious thing to do, at least for union or intersection.

(In contrast, that's exactly what you want an update method to do. 
Different behaviour for different methods.)


[...]
>>> Raising an error would work, but is IMO unnecessary.
>>
>> I believe that's the only reasonable way for a dict union method to
>> work. As the Zen says:
>>
>> In the face of ambiguity, refuse the temptation to guess.
>>
>> Since there is ambiguity which value should be associated with the key,
>> don't guess.
> 
> There's already ambiguity as to which of two equal values should be
> retained by the set. 

In an ideal world of Platonic Ideals, it wouldn't matter, since 
everything is equal to itself, and to nothing else. There's only one 
number "two", whether you write it as 2 or 2.0 or 800/400 or Ⅱ or 0b10, 
and it is *impossible even in principle* to distinguish them since there 
is no "them" to distinguish between. Things that are equal shouldn't be 
distinguishable, not by value, not by type, not by identity.

But that would be *too abstract* to be useful, and so we allow some of 
the abstractness leak away, to the benefit of all. But the consequence of 
this is that we sometimes have to make hard decisions, like, which one of 
these various "twos" do we want to keep? Or more often, we stumble into a 
decision by allowing the implementation specify the behaviour, rather 
than choosing the behaviour and the finding an implementation to match 
it. Given the two behaviours:

{2} | {2.0} => {2} or {2.0}, which should it be? Why not Fraction(2) or 
Decimal(2) or 2+0j?

there's no reason to prefer Python's answer, "the value on the left", 
except that it simplifies the implementation. The union operator ought to 
be symmetrical, a ∪ b should be identical to b ∪ a, but isn't. Another 
leaky abstraction.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#51231

On Fri, Jul 26, 2013 at 12:57 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> [ snip lengthy explanation of sets ]
> The union operator ought to
> be symmetrical, a ∪ b should be identical to b ∪ a, but isn't. Another
> leaky abstraction.

Right. I agree with all your theory, which is fine and good. If we had
a "set of real numbers", then each one would be both equal to and
indistinguishable from any other representation of itself. But Python
doesn't work with real numbers. It works with ints and floats and
Fractions and Decimals and Guido-knows-what. (Sorry, I don't normally
talk like that, but the expression begged to be said. :) )

So since Python already lets its abstraction leak a bit for
usefulness, why not retain the exact same leak when working with a
dict? A set is a dict with no values... a dict is a set with extra
payload. They're given very similar literal notation; if they were
meant to be more distinct, why was no alternative symbol used?

(I love how a random side comment can become a topic of its own.)

ChrisA

[toc] | [prev] | [next] | [standalone]

#51214

On 25.07.2013 07:48, Steven D'Aprano wrote:

> Then you aren't looking very closely. d.keys() returns a set-like view 
> into the dict, which is great for comparing elements:
> 
> py> d1 = dict.fromkeys([1, 2, 3, 4])
> py> d2 = dict.fromkeys([3, 4, 5, 6])
> py> d1.keys() & d2.keys()  # keys that are in both
> {3, 4}
> py> d1.keys() ^ d2.keys()  # keys not in both
> {1, 2, 5, 6}
> py> d1.keys() - d2.keys()  # keys only in d1
> {1, 2}
> py> d2.keys() - d1.keys()  # keys only in d2
> {5, 6}

I know this is completely off-topic, but I really must thank you for
showing that neat trick. I didn't know set()'s operators &, ^, - were
overloaded (and always used difference/intersection, etc). That's
really, really neat.

Thanks again,
Joe

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos über Rüdiger Thomas in dsa <hidbv3$om2$1@speranza.aioe.org>

[toc] | [prev] | [next] | [standalone]

#51229

On 07/24/2013 10:48 PM, Steven D'Aprano wrote:
> On Wed, 24 Jul 2013 08:57:11 -0700, Ethan Furman wrote:
>
>> My point is that in 2.x .keys() did something different from the dict,
>> while in 3.x it appears to me that they are the same.
>
> Then you aren't looking very closely.

Actually, I am.  That's why I started this thread.

Thank you for the insights.

--
~Ethan~

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web