Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #18910 > unrolled thread

copy on write

Back to article view | Back to comp.lang.python

Contents

  copy on write Eduardo Suarez-Santana <esuarez@itccanarias.org> - 2012-01-13 11:33 +0000
    Re: copy on write Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-01-13 12:10 +0000
      Re: copy on write Chris Angelico <rosuav@gmail.com> - 2012-01-13 23:30 +1100
        Re: copy on write Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-01-13 13:04 +0000
          Re: copy on write Ethan Furman <ethan@stoneleaf.us> - 2012-01-13 10:40 -0800
            Re: copy on write 88888 Dihedral <dihedral88888@googlemail.com> - 2012-01-13 14:26 -0800
            Re: copy on write 88888 Dihedral <dihedral88888@googlemail.com> - 2012-01-13 14:26 -0800
          Re: copy on write John O'Hagan <research@johnohagan.com> - 2012-02-02 14:18 +1100
          Re: copy on write Devin Jeanpierre <jeanpierreda@gmail.com> - 2012-02-02 01:34 -0500
          Re: copy on write John O'Hagan <research@johnohagan.com> - 2012-02-02 19:11 +1100
            Re: copy on write Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-02-02 09:16 +0000
              Re: copy on write Hrvoje Niksic <hniksic@xemacs.org> - 2012-02-02 11:53 +0100
                Re: copy on write MRAB <python@mrabarnett.plus.com> - 2012-02-02 16:28 +0000
                Re: copy on write Devin Jeanpierre <jeanpierreda@gmail.com> - 2012-02-02 12:21 -0500
              Re: copy on write John O'Hagan <research@johnohagan.com> - 2012-02-03 01:17 +1100
              Re: copy on write Terry Reedy <tjreedy@udel.edu> - 2012-02-02 12:25 -0500
              Re: copy on write John O'Hagan <research@johnohagan.com> - 2012-02-03 14:08 +1100
                Re: copy on write Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-02-03 05:04 +0000
                  Re: copy on write Chris Angelico <rosuav@gmail.com> - 2012-02-03 16:28 +1100
                    Re: copy on write Rick Johnson <rantingrickjohnson@gmail.com> - 2012-02-03 07:35 -0800
                  Re: copy on write Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2012-02-03 10:08 +0100
                  Re: copy on write John O'Hagan <research@johnohagan.com> - 2012-02-03 21:47 +1100
                    Re: copy on write Wolfram Hinderer <wolfram.hinderer@googlemail.com> - 2012-02-05 06:09 -0800
                  Re: copy on write "OKB (not okblacke)" <brenNOSPAMbarn@NObrenSPAMbarn.net> - 2012-02-03 16:15 +0000
        Re: copy on write Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2012-02-02 11:42 +0100
      Re: copy on write Devin Jeanpierre <jeanpierreda@gmail.com> - 2012-01-13 08:50 -0500
        Re: copy on write Grant Edwards <invalid@invalid.invalid> - 2012-01-13 15:13 +0000
          Re: copy on write Devin Jeanpierre <jeanpierreda@gmail.com> - 2012-01-13 11:48 -0500
            Re: copy on write Neil Cerutti <neilc@norwich.edu> - 2012-01-13 16:54 +0000
              Re: copy on write Grant Edwards <invalid@invalid.invalid> - 2012-01-13 18:15 +0000
                Re: copy on write Chris Angelico <rosuav@gmail.com> - 2012-01-14 05:26 +1100
                  Re: copy on write Grant Edwards <invalid@invalid.invalid> - 2012-01-13 19:30 +0000
                    Re: copy on write Neil Cerutti <neilc@norwich.edu> - 2012-01-13 20:11 +0000
              Re: copy on write Evan Driscoll <edriscoll@wisc.edu> - 2012-01-13 13:24 -0600
                Re: copy on write Neil Cerutti <neilc@norwich.edu> - 2012-01-13 21:20 +0000
                  Re: copy on write Evan Driscoll <edriscoll@wisc.edu> - 2012-01-13 16:48 -0600
                    Re: copy on write 88888 Dihedral <dihedral88888@googlemail.com> - 2012-02-02 05:33 -0800
                      Re: copy on write Evan Driscoll <edriscoll@wisc.edu> - 2012-02-02 15:20 -0600
                    Re: copy on write 88888 Dihedral <dihedral88888@googlemail.com> - 2012-02-02 05:33 -0800
                    Re: copy on write 88888 Dihedral <dihedral88888@googlemail.com> - 2012-02-03 14:16 -0800
                    Re: copy on write 88888 Dihedral <dihedral88888@googlemail.com> - 2012-02-03 14:16 -0800
            Re: copy on write Rick Johnson <rantingrickjohnson@gmail.com> - 2012-02-01 19:51 -0800
              Re: copy on write Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-02-02 05:31 +0000

Page 1 of 3  [1] 2 3  Next page →

#18910 — copy on write

I wonder whether this is normal behaviour.

I would expect equal sign to copy values from right to left. However, it 
seems there is a copy-on-write mechanism that is not working.

Anyone can explain and provide a working example?

Thanks,
-Eduardo

$ python
Python 2.7.2 (default, Oct 31 2011, 11:54:55)
[GCC 4.5.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> class n:
...     def __init__(self, id, cont):
...         self.id = id;
...         self.cont = cont;
...
 >>> r={'a':1};
 >>> d={};
 >>> d['x']=r;
 >>> d['y']=r;
 >>> x1 = n('x',d['x']);
 >>> y1 = n('y',d['y']);
 >>> x1.cont['a']=2;
 >>> y1.cont
{'a': 2}
 >>>

[toc] | [next] | [standalone]

#18912

On Fri, 13 Jan 2012 11:33:24 +0000, Eduardo Suarez-Santana wrote:

> I wonder whether this is normal behaviour.
> 
> I would expect equal sign to copy values from right to left. 

Assignment in Python never copies values.

> However, it
> seems there is a copy-on-write mechanism that is not working.

There is no copy-on-write.

Assignment in Python is name binding: the name on the left hand side is 
bound to the object on the right. An object can have zero, one or many 
names. If the object is mutable, changes to the object will be visible 
via any name:

>>> x = []  # lists are mutable objects
>>> y = x  # not a copy of x, but x and y point to the same object
>>> x.append(42)  # mutates the object in place
>>> print y
[42]

The same rules apply not just to names, but also to list items and dict 
items, as well as attributes, and any other reference:

>>> z = [x, y]  # z is a list containing the same sublist twice
>>> z[0].append(23)
>>> print z
[[42, 23], [42, 23]]

When you work with floats, ints or strings, you don't notice this because 
those types are immutable: you can't modify those objects in place. So 
for example:

>>> a = 42  # binds the name 'a' to the object 42
>>> b = a  # a and b point to the same object
>>> a += 1  # creates a new object, and binds it to a
>>> print b  # leaving b still pointing to the old object
42


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#18914

On Fri, Jan 13, 2012 at 11:10 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
>>>> z = [x, y]  # z is a list containing the same sublist twice
>>>> z[0].append(23)
>>>> print z
> [[42, 23], [42, 23]]
>
> When you work with floats, ints or strings, you don't notice this because
> those types are immutable: you can't modify those objects in place. So
> for example:
>
>>>> a = 42  # binds the name 'a' to the object 42
>>>> b = a  # a and b point to the same object
>>>> a += 1  # creates a new object, and binds it to a
>>>> print b  # leaving b still pointing to the old object
> 42

I was about to say that it's a difference between ".append()" which is
a method on the object, and "+=" which is normally a rebinding, but
unfortunately:

>>> a=[]
>>> b=a
>>> a+=[1]
>>> a
[1]
>>> b
[1]
>>> b+=[2]
>>> a
[1, 2]
>>> a
[1, 2]
>>> a=a+[3]
>>> a
[1, 2, 3]
>>> b
[1, 2]

(tested in Python 3.2 on Windows)

It seems there's a distinct difference between a+=b (in-place
addition/concatenation) and a=a+b (always rebinding), which is sorely
confusing to C programmers. But then, there's a lot about Python
that's sorely confusing to C programmers.

ChrisA

[toc] | [prev] | [next] | [standalone]

#18915

On Fri, 13 Jan 2012 23:30:56 +1100, Chris Angelico wrote:

> It seems there's a distinct difference between a+=b (in-place
> addition/concatenation) and a=a+b (always rebinding), 

Actually, both are always rebinding. It just happens that sometimes a+=b 
rebinds to the same object that it was originally bound to.

In the case of ints, a+=b creates a new object (a+b) and rebinds a to it. 
In the case of lists, a+=b nominally creates a list a+b, but in fact it 
implements that as an in-place operation a.extend(b), and then rebinds 
the name a to the list already bound to a. 

It does that because the Python VM doesn't know at compile time whether 
a+=b will be in-place or not, and so it has to do the rebinding in order 
to support the fall-back case of a+=b => a=a+b. Or something -- go read 
the PEP if you really care :)

Normally this is harmless, but there is one interesting little glitch you 
can get:

>>> t = ('a', [23])
>>> t[1] += [42]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> t
('a', [23, 42])




> which is sorely
> confusing to C programmers. But then, there's a lot about Python
> that's sorely confusing to C programmers.

I prefer to think of it as "there's a lot about C that is sorely 
confusing to anyone who isn't a C programmer" <wink>



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#18935

Steven D'Aprano wrote:
> Normally this is harmless, but there is one interesting little glitch you 
> can get:
> 
>>>> t = ('a', [23])
>>>> t[1] += [42]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: 'tuple' object does not support item assignment
>>>> t
> ('a', [23, 42])


There is one other glitch, and possibly my only complaint:

--> a = [1, 2, 3]
--> b = 'hello, world'
--> a = a + b
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: can only concatenate list (not "str") to list
--> a += b
--> a
[1, 2, 3, 'h', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']

IMO, either both + and += should succeed, or both should fail.

~Ethan~

[toc] | [prev] | [next] | [standalone]

#18950

Ethan Furman於 2012年1月14日星期六UTC+8上午2時40分47秒寫道：
> Steven D'Aprano wrote:
> > Normally this is harmless, but there is one interesting little glitch you 
> > can get:
> > 
> >>>> t = ('a', [23])
> >>>> t[1] += [42]
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > TypeError: 'tuple' object does not support item assignment
> >>>> t
> > ('a', [23, 42])
> 
> 
> There is one other glitch, and possibly my only complaint:
> 
> --> a = [1, 2, 3]
> --> b = 'hello, world'
> --> a = a + b
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: can only concatenate list (not "str") to list
> --> a += b
> --> a
> [1, 2, 3, 'h', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']
> 
> IMO, either both + and += should succeed, or both should fail.
> 
> ~Ethan~

The += operator is not  only for value types in the above example. 
 
An operator of two operands and an operator of three operands  of 
general object types are two different operators. 

[toc] | [prev] | [next] | [standalone]

#18951

Ethan Furman於 2012年1月14日星期六UTC+8上午2時40分47秒寫道：
> Steven D'Aprano wrote:
> > Normally this is harmless, but there is one interesting little glitch you 
> > can get:
> > 
> >>>> t = ('a', [23])
> >>>> t[1] += [42]
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > TypeError: 'tuple' object does not support item assignment
> >>>> t
> > ('a', [23, 42])
> 
> 
> There is one other glitch, and possibly my only complaint:
> 
> --> a = [1, 2, 3]
> --> b = 'hello, world'
> --> a = a + b
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: can only concatenate list (not "str") to list
> --> a += b
> --> a
> [1, 2, 3, 'h', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']
> 
> IMO, either both + and += should succeed, or both should fail.
> 
> ~Ethan~

The += operator is not  only for value types in the above example. 
 
An operator of two operands and an operator of three operands  of 
general object types are two different operators. 

[toc] | [prev] | [next] | [standalone]

#19765

On Fri, 13 Jan 2012 10:40:47 -0800
Ethan Furman <ethan@stoneleaf.us> wrote:

> Steven D'Aprano wrote:
> > Normally this is harmless, but there is one interesting little
> > glitch you can get:
> > 
> >>>> t = ('a', [23])
> >>>> t[1] += [42]
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > TypeError: 'tuple' object does not support item assignment
> >>>> t
> > ('a', [23, 42])

IMHO, this is worthy of bug-hood: shouldn't we be able to conclude from the TypeError that the assignment failed?

> There is one other glitch, and possibly my only complaint:
> 
> --> a = [1, 2, 3]
> --> b = 'hello, world'
> --> a = a + b
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: can only concatenate list (not "str") to list
> --> a += b
> --> a
> [1, 2, 3, 'h', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']
> 
> IMO, either both + and += should succeed, or both should fail.
> 
> ~Ethan~


This also happens for tuples, sets, generators and range objects (probably any iterable), AFAIK only when the left operand is a list. Do lists get special treatment in terms of implicitly converting the right-hand operand?

The behaviour of the "in-place" operator could be more consistent across types:

>>> a=[1,2]
>>> a+=(3,4)
>>> a
[1, 2, 3, 4]
>>> a=(1,2)
>>> a+=(3,4)
>>> a
(1, 2, 3, 4)
>>> a=(1,2)
>>> a+=[3,4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate tuple (not "list") to tuple


John

[toc] | [prev] | [next] | [standalone]

#19769

On Wed, Feb 1, 2012 at 10:18 PM, John O'Hagan <research@johnohagan.com> wrote:
> On Fri, 13 Jan 2012 10:40:47 -0800
> Ethan Furman <ethan@stoneleaf.us> wrote:
>
>> Steven D'Aprano wrote:
>> > Normally this is harmless, but there is one interesting little
>> > glitch you can get:
>> >
>> >>>> t = ('a', [23])
>> >>>> t[1] += [42]
>> > Traceback (most recent call last):
>> >   File "<stdin>", line 1, in <module>
>> > TypeError: 'tuple' object does not support item assignment
>> >>>> t
>> > ('a', [23, 42])
>
> IMHO, this is worthy of bug-hood: shouldn't we be able to conclude from the TypeError that the assignment failed?

It did fail. The mutation did not.

I can't think of any way out of this misleadingness, although if you
can that would be pretty awesome.

-- Devin

[toc] | [prev] | [next] | [standalone]

#19779

On Thu, 2 Feb 2012 01:34:48 -0500
Devin Jeanpierre <jeanpierreda@gmail.com> wrote:

> On Wed, Feb 1, 2012 at 10:18 PM, John O'Hagan
> <research@johnohagan.com> wrote:
> > On Fri, 13 Jan 2012 10:40:47 -0800
> > Ethan Furman <ethan@stoneleaf.us> wrote:
> >
> >> Steven D'Aprano wrote:
> >> > Normally this is harmless, but there is one interesting little
> >> > glitch you can get:
> >> >
> >> >>>> t = ('a', [23])
> >> >>>> t[1] += [42]
> >> > Traceback (most recent call last):
> >> >   File "<stdin>", line 1, in <module>
> >> > TypeError: 'tuple' object does not support item assignment
> >> >>>> t
> >> > ('a', [23, 42])
> >
> > IMHO, this is worthy of bug-hood: shouldn't we be able to conclude
> > from the TypeError that the assignment failed?
> 
> It did fail. The mutation did not.

You're right, in fact, for me the surprise is that "t[1] +=" is interpreted as an assignment at all, given that for lists (and other mutable objects which use "+=") it is a mutation. Although as Steven says elsewhere, it actually is an assignment, but one which ends up reassigning to the same object. 
 
But it shouldn't be both. I can't think of another example of (what appears to be but is not) a single operation failing with an exception, but still doing exactly what you intended.

> 
> I can't think of any way out of this misleadingness, although if you
> can that would be pretty awesome.

In the case above, the failure of the assignment is of no consequence. I think it would make more sense if applying "+=" to a tuple element were treated (by the interpreter I suppose) only on the merits of the element, and not as an assignment to the tuple. 

John

[toc] | [prev] | [next] | [standalone]

#19781

On Thu, 02 Feb 2012 19:11:53 +1100, John O'Hagan wrote:

> You're right, in fact, for me the surprise is that "t[1] +=" is
> interpreted as an assignment at all, given that for lists (and other
> mutable objects which use "+=") it is a mutation. Although as Steven
> says elsewhere, it actually is an assignment, but one which ends up
> reassigning to the same object.
>  
> But it shouldn't be both.

Do you expect that x += 1 should succeed? After all, "increment and 
decrement numbers" is practically THE use-case for the augmented 
assignment operators.

How can you expect x += 1 to succeed without an assignment? Numbers in 
Python are immutable, and they have to stay immutable. It would cause 
chaos and much gnashing of teeth if you did this:

x = 2
y = 7 - 5
x += 1
print y * 100
=> prints 300

So if you want x += 1 to succeed, += must do an assignment.

Perhaps you are thinking that Python could determine ahead of time 
whether x[1] += y involved a list or a tuple, and not perform the finally 
assignment if x was a tuple. Well, maybe, but such an approach (if 
possible!) is fraught with danger and mysterious errors even harder to 
debug than the current situation. And besides, what should Python do 
about non-built-in types? There is no way in general to predict whether 
x[1] = something will succeed except to actually try it.

> I can't think of another example of (what
> appears to be but is not) a single operation failing with an exception,
> but still doing exactly what you intended.

Neither can I, but that doesn't mean that the current situation is not 
the least-worst alternative.


>> I can't think of any way out of this misleadingness, although if you
>> can that would be pretty awesome.
> 
> In the case above, the failure of the assignment is of no consequence. I
> think it would make more sense if applying "+=" to a tuple element were
> treated (by the interpreter I suppose) only on the merits of the
> element, and not as an assignment to the tuple.

How should the interpreter deal with other objects which happen to raise 
TypeError? By always ignoring it?

x = [1, None, 3]
x[1] += 2  # apparently succeeds

Or perhaps by hard-coding tuples and only ignoring errors for tuples? So 
now you disguise one error but not others?



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#19785

Steven D'Aprano <steve+comp.lang.python@pearwood.info> writes:

> Perhaps you are thinking that Python could determine ahead of time
> whether x[1] += y involved a list or a tuple, and not perform the
> finally assignment if x was a tuple. Well, maybe, but such an approach
> (if possible!) is fraught with danger and mysterious errors even
> harder to debug than the current situation. And besides, what should
> Python do about non-built-in types? There is no way in general to
> predict whether x[1] = something will succeed except to actually try
> it.

An alternative approach is to simply not perform the final assignment if
the in-place method is available on the contained object.  No prediction
is needed to do it, because the contained object has to be examined
anyway.  No prediction is needed, just don't.  Currently,
lhs[ind] += rhs is implemented like this:

item = lhs[ind]
if hasattr(item, '__iadd__'):
    lhs.__setitem__(ind, item.__iadd__(rhs))
else:
    lhs.__setitem__(ind, item + rhs)
# (Note item assignment in both "if" branches.)

It could, however, be implemented like this:

item = lhs[ind]
if hasattr(item, '__iadd__'):
    item += rhs                  # no assignment, item supports in-place change
else:
    lhs.__setitem__(ind, lhs[ind] + rhs)

This would raise the exact same exception in the tuple case, but without
executing the in-place assignment.  On the other hand, some_list[ind] += 1
would continue working exactly the same as it does now.

In the same vein, in-place methods should not have a return value
(i.e. they should return None), as per Python convention that functions
called for side effect don't return values.

The alternative behavior is unfortunately not backward-compatible (it
ignores the return value of augmented methods), so I'm not seriously
proposing it, but I believe it would have been a better implementation
of augmented assignments than the current one.  The present interface
doesn't just bite those who try to use augmented assignment on tuples
holding mutable objects, but also those who do the same with read-only
properties, which is even more reasonable.  For example, obj.list_attr
being a list, one would expect that obj.list_attr += [1, 2, 3] does the
same thing as obj.list_attr.extend([1, 2, 3]).  And it almost does,
except it also follows up with an assignment after the list has already
been changed, and the assignment to a read-only property raises an
exception.  Refusing to modify the list would have been fine, modifying
it without raising an exception (as described above) would have been
better, but modifying it and *then* raising an exception is a surprise
that takes some getting used to.

[toc] | [prev] | [next] | [standalone]

#19797

On 02/02/2012 10:53, Hrvoje Niksic wrote:
> Steven D'Aprano<steve+comp.lang.python@pearwood.info>  writes:
>
>>  Perhaps you are thinking that Python could determine ahead of time
>>  whether x[1] += y involved a list or a tuple, and not perform the
>>  finally assignment if x was a tuple. Well, maybe, but such an approach
>>  (if possible!) is fraught with danger and mysterious errors even
>>  harder to debug than the current situation. And besides, what should
>>  Python do about non-built-in types? There is no way in general to
>>  predict whether x[1] = something will succeed except to actually try
>>  it.
>
> An alternative approach is to simply not perform the final assignment if
> the in-place method is available on the contained object.  No prediction
> is needed to do it, because the contained object has to be examined
> anyway.  No prediction is needed, just don't.  Currently,
> lhs[ind] += rhs is implemented like this:
>
> item = lhs[ind]
> if hasattr(item, '__iadd__'):
>      lhs.__setitem__(ind, item.__iadd__(rhs))
> else:
>      lhs.__setitem__(ind, item + rhs)
> # (Note item assignment in both "if" branches.)
>
> It could, however, be implemented like this:
>
> item = lhs[ind]
> if hasattr(item, '__iadd__'):
>      item += rhs                  # no assignment, item supports in-place change
> else:
>      lhs.__setitem__(ind, lhs[ind] + rhs)
>
> This would raise the exact same exception in the tuple case, but without
> executing the in-place assignment.  On the other hand, some_list[ind] += 1
> would continue working exactly the same as it does now.
>
> In the same vein, in-place methods should not have a return value
> (i.e. they should return None), as per Python convention that functions
> called for side effect don't return values.
>
> The alternative behavior is unfortunately not backward-compatible (it
> ignores the return value of augmented methods), so I'm not seriously
> proposing it, but I believe it would have been a better implementation
> of augmented assignments than the current one.
 >
[snip]
Could it not perform the assignment if the reference returned by
__iadd__ is the same as the current reference?

For example:

     t[0] += x

would do:

     r = t[0].__iadd__(x)
     if t[0] is not r:
         t[0] = r

Should failed assignment be raising TypeError? Is it really a type
error?

[toc] | [prev] | [next] | [standalone]

#19803

On Thu, Feb 2, 2012 at 11:28 AM, MRAB <python@mrabarnett.plus.com> wrote:
> Should failed assignment be raising TypeError? Is it really a type
> error?

A failed setitem should be a TypeError as much as a failed getitem
should. Should 1[0] be a TypeError?

-- Devin

[toc] | [prev] | [next] | [standalone]

#19794

On 02 Feb 2012 09:16:40 GMT
Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> On Thu, 02 Feb 2012 19:11:53 +1100, John O'Hagan wrote:
> 
> > You're right, in fact, for me the surprise is that "t[1] +=" is
> > interpreted as an assignment at all, given that for lists (and other
> > mutable objects which use "+=") it is a mutation. Although as Steven
> > says elsewhere, it actually is an assignment, but one which ends up
> > reassigning to the same object.
> >  
> > But it shouldn't be both.
> 
> Do you expect that x += 1 should succeed? After all, "increment and 
> decrement numbers" is practically THE use-case for the augmented 
> assignment operators.
> 
> How can you expect x += 1 to succeed without an assignment? 

I don't; obviously, for immutable objects assignment is the only possibility.

[...]
> 
> Perhaps you are thinking that Python could determine ahead of time 
> whether x[1] += y involved a list or a tuple, and not perform the
> finally assignment if x was a tuple. Well, maybe, but such an
> approach (if possible!) is fraught with danger and mysterious errors
> even harder to debug than the current situation. And besides, what
> should Python do about non-built-in types? There is no way in general
> to predict whether x[1] = something will succeed except to actually
> try it.

It's not so much about the type of x but that of x[1]. Wouldn't it be possible to omit the assignment simply if the object referred to by x[1] uses "+=" without creating a new object? That way, some_tuple[i] += y will succeed if some_tuple[i] is a list but not with, say, an int. That seems reasonable to me.

[...]

> > 
> > In the case above, the failure of the assignment is of no
> > consequence. I think it would make more sense if applying "+=" to a
> > tuple element were treated (by the interpreter I suppose) only on
> > the merits of the element, and not as an assignment to the tuple.
> 
> How should the interpreter deal with other objects which happen to
> raise TypeError? By always ignoring it?
> 
> x = [1, None, 3]
> x[1] += 2  # apparently succeeds
> 
> Or perhaps by hard-coding tuples and only ignoring errors for tuples?
> So now you disguise one error but not others?

I'm not suggesting either of those. None can't be modified in place. But for objects which can, wouldn't omitting the final assignment prevent the TypeError in the first place?

John

[toc] | [prev] | [next] | [standalone]

#19804

On 2/2/2012 9:17 AM, John O'Hagan wrote:

> It's not so much about the type of x but that of x[1]. Wouldn't it be
> possible to omit the assignment simply if the object referred to by
> x[1] uses "+=" without creating a new object? That way, some_tuple[i]
> += y will succeed if some_tuple[i] is a list but not with, say, an
> int. That seems reasonable to me.

There was considerable discussion of the exact semantics of augmented 
operations when they were introduced. I do not remember if that 
particular idea was suggested (and rejected) or not. You could try to 
look at the PEP, if there is one, or the dicussion ( probably on pydev 
list).

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#19820

On Thu, 02 Feb 2012 12:25:00 -0500
Terry Reedy <tjreedy@udel.edu> wrote:

> On 2/2/2012 9:17 AM, John O'Hagan wrote:
> 
> > It's not so much about the type of x but that of x[1]. Wouldn't it
> > be possible to omit the assignment simply if the object referred to
> > by x[1] uses "+=" without creating a new object? That way,
> > some_tuple[i] += y will succeed if some_tuple[i] is a list but not
> > with, say, an int. That seems reasonable to me.
> 
> There was considerable discussion of the exact semantics of augmented 
> operations when they were introduced. I do not remember if that 
> particular idea was suggested (and rejected) or not. You could try to 
> look at the PEP, if there is one, or the dicussion ( probably on
> pydev list).
> 

I think we're 12 years late on this one. It's PEP 203 from 2000 and the key phrase was:

"The in-place function should always return a new reference, either
to the old `x' object if the operation was indeed performed
in-place, or to a new object."

If this had read:

"The in-place function should return a reference to a new object
if the operation was not performed in-place."

or something like that, we wouldn't be discussing this.

The discussion on py-dev at the time was quite limited but there was some lively debate on this list the following year (in the context of widespread controversy over new-fangled features which also included list comprehensions and generators), to which the BDFL's response was:

"You shouldn't think "+= is confusing because sometimes it modifies an
object and sometimes it does".  Gee, there are lots of places where
something that's *spelled* the same has a different effect depending
on the object types involved."


That's true, but I don't think there should be a different effect depending on what _name_ we use for an operand:

>>> t=([],)
>>> l=t[0]
>>> l is t[0]
True
>>> l+=[1]
>>> t
([1],)
>>> t[0]+=[1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> t
([1, 1],)
>>> l is t[0]
True

Same object, same operator, different name, different outcome. Maybe that was obvious from the foregoing discussion, but it shocked me when put that way.

John

[toc] | [prev] | [next] | [standalone]

#19823

On Fri, 03 Feb 2012 14:08:06 +1100, John O'Hagan wrote:

> I think we're 12 years late on this one. It's PEP 203 from 2000 and the
> key phrase was:
> 
> "The in-place function should always return a new reference, either to
> the old `x' object if the operation was indeed performed in-place, or to
> a new object."
> 
> If this had read:
> 
> "The in-place function should return a reference to a new object if the
> operation was not performed in-place."
> 
> or something like that, we wouldn't be discussing this.

And what should it return if the operation *is* performed in-place? 
"Don't return anything" is not an option, Python doesn't have procedures. 
That implies that __iadd__ etc. should return None. But two problems come 
to mind:

1) Using None as an out-of-band signal to the interpreter to say "don't 
perform the assignment" makes it impossible for the augmented assignment 
method to return None as the result. If we only think about numeric 
operations like x += 1 then we might not care, but once you consider the 
situation more widely the problem is clear:

x = Fact(foo)
y = Fact(bar)
x & y  # Returns a composite Fact, or None if they are contradictory

With your suggestion, x &= y fails to work, but only sometimes. And when 
it fails, it doesn't fail with an explicit exception, but silently fails 
and then does the wrong thing. This makes debugging a horror.

2) And speaking of debugging, sometimes people forget to include the 
return statement in methods. Normally, the left hand side of the 
assignment then gets set to None, and the error is pretty obvious as soon 
as you try to do something with it. But with your suggestion, instead of 
getting an exception, it silently fails, and your code does the wrong 
thing.

I suppose that they could have invented a new sentinel, or a special 
exception to be raised as a signal, but that's piling complication on top 
of complication, and it isn't clear to me that it's worth it for an 
obscure corner case.

Yes, the current behaviour is a Gotcha, but it's a Gotcha that makes good 
sense compared to the alternatives.

Ultimately, augmented assignment is *assignment*, just like it says on 
the tin. t[1] += x is syntactic sugar for t[1] = t[1].__iadd__(x). It 
can't and shouldn't fail to raise an exception if t is a tuple, because 
tuple item assignment *must* fail.

The problem is that lists treat __iadd__ as an in-place optimization, and 
this clashes with tuple immutability. But if lists *didn't* treat 
__iadd__ as in-place, people would complain when they used it directly 
without a tuple wrapper. 

Perhaps lists shouldn't define += at all, but then people will complain 
that mylist += another_list is slow. Telling them to use mylist.extend 
instead just makes them cranky. After all, mylist + another_list works, 
so why shouldn't += work?

Ultimately, there is no right answer, because the multitude of 
requirements are contradictory. No matter what Python did, somebody would 
complain.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#19826

On Fri, Feb 3, 2012 at 4:04 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> No matter what Python did, somebody would complain.

+1

This is, I think, the ultimate truth of the matter.

ChrisA

[toc] | [prev] | [next] | [standalone]

#19919

On Feb 2, 11:28 pm, Chris Angelico <ros...@gmail.com> wrote:
> On Fri, Feb 3, 2012 at 4:04 PM, Steven D'Aprano
>
> <steve+comp.lang.pyt...@pearwood.info> wrote:
> > No matter what Python did, somebody would complain.
>
> +1
>
> This is, I think, the ultimate truth of the matter.

People would not complain if they did not care. The only useless
complaint is people complaining about other people complaining. And
the only thing worse than that is rabid fanboi brown-nosing!

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web