Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #32748 > unrolled thread

Multi-dimensional list initialization

Back to article view | Back to comp.lang.python

Contents

  Multi-dimensional list initialization Demian Brecht <demianbrecht@gmail.com> - 2012-11-04 22:27 -0800
    Re: Multi-dimensional list initialization Hans Mulder <hansmu@xs4all.nl> - 2012-11-05 10:13 +0100
      Re: Multi-dimensional list initialization Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-06 01:32 +0000
      Re: Multi-dimensional list initialization Chris Angelico <rosuav@gmail.com> - 2012-11-06 13:01 +1100
      Re: Multi-dimensional list initialization Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-06 02:30 +0000
      Re: Multi-dimensional list initialization Andrew Robinson <andrew3@r3dsolutions.com> - 2012-11-05 21:51 -0800
        Re: Multi-dimensional list initialization Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-06 09:04 +0000
          RE: Multi-dimensional list initialization "Shambhu Rajak" <shambhu.rajak@calsoftinc.com> - 2012-11-06 18:57 +0530
          Re: Multi-dimensional list initialization Andrew Robinson <andrew3@r3dsolutions.com> - 2012-11-06 14:41 -0800
            Re: Multi-dimensional list initialization Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-07 01:55 +0000
              Re: Multi-dimensional list initialization Demian Brecht <demianbrecht@gmail.com> - 2012-11-06 22:56 -0800
              Re: Multi-dimensional list initialization wxjmfauth@gmail.com - 2012-11-07 00:57 -0800
              Re: Multi-dimensional list initialization Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-11-07 22:27 +0000
              Re: Multi-dimensional list initialization Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-07 16:39 -0700
              Re: Multi-dimensional list initialization Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-08 00:09 -0700
              Re: Multi-dimensional list initialization Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-08 08:58 -0700
              Re: Multi-dimensional list initialization Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-11-09 01:39 +0000
              Re: Multi-dimensional list initialization Chris Angelico <rosuav@gmail.com> - 2012-11-09 17:07 +1100
                Re: Multi-dimensional list initialization Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-09 06:37 +0000
                  Re: Multi-dimensional list initialization Chris Angelico <rosuav@gmail.com> - 2012-11-09 17:59 +1100
                  Re: Multi-dimensional list initialization Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-11-09 07:27 +0000
                  Re: Multi-dimensional list initialization rusi <rustompmody@gmail.com> - 2012-11-09 07:05 -0800
                    Re: Multi-dimensional list initialization Chris Angelico <rosuav@gmail.com> - 2012-11-10 02:23 +1100
              Re: Multi-dimensional list initialization Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-11-09 14:34 -0500
              RE: Multi-dimensional list initialization "Prasad, Ramit" <ramit.prasad@jpmorgan.com> - 2012-11-09 20:31 +0000
              Re: Multi-dimensional list initialization Ethan Furman <ethan@stoneleaf.us> - 2012-11-09 13:49 -0800
          RE: Multi-dimensional list initialization "Prasad, Ramit" <ramit.prasad@jpmorgan.com> - 2012-11-06 23:39 +0000
          Re: Multi-dimensional list initialization Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-06 16:52 -0700
          Re: Multi-dimensional list initialization MRAB <python@mrabarnett.plus.com> - 2012-11-07 00:23 +0000
            Re: Multi-dimensional list initialization rusi <rustompmody@gmail.com> - 2012-11-06 20:11 -0800
            Re: Multi-dimensional list initialization Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-07 05:05 +0000
              Re: Multi-dimensional list initialization Roy Smith <roy@panix.com> - 2012-11-07 00:12 -0500
                Re: Multi-dimensional list initialization Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2012-11-07 18:32 +1300
                  RE: Multi-dimensional list initialization "Prasad, Ramit" <ramit.prasad@jpmorgan.com> - 2012-11-07 15:57 +0000
              Re: Multi-dimensional list initialization Jussi Piitulainen <jpiitula@ling.helsinki.fi> - 2012-11-07 10:52 +0200
              Re: Multi-dimensional list initialization MRAB <python@mrabarnett.plus.com> - 2012-11-07 17:17 +0000
                Re: Multi-dimensional list initialization Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-07 22:36 +0000
          Re: Multi-dimensional list initialization Ethan Furman <ethan@stoneleaf.us> - 2012-11-07 07:23 -0800
          Re: Multi-dimensional list initialization Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-07 14:01 -0700
            Re: Multi-dimensional list initialization Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-08 00:00 +0000
              Re: Multi-dimensional list initialization Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-08 00:30 +0000
                Re: Multi-dimensional list initialization Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-08 03:47 +0000
              Re: Multi-dimensional list initialization Andrew Robinson <andrew3@r3dsolutions.com> - 2012-11-07 20:51 -0800
              Re: Multi-dimensional list initialization wrw@mac.com - 2012-11-08 08:26 -0500
          Re: Multi-dimensional list initialization Andrew Robinson <andrew3@r3dsolutions.com> - 2012-11-07 16:24 -0800
            Re: Multi-dimensional list initialization Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-08 04:20 +0000
      Re: Multi-dimensional list initialization Chris Angelico <rosuav@gmail.com> - 2012-11-06 17:07 +1100
      Re: Multi-dimensional list initialization Andrew Robinson <andrew3@r3dsolutions.com> - 2012-11-06 00:21 -0800
      Re: Multi-dimensional list initialization Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-06 02:19 -0700
      RE: Multi-dimensional list initialization "Prasad, Ramit" <ramit.prasad@jpmorgan.com> - 2012-11-06 17:32 +0000
      Re: Multi-dimensional list initialization Andrew Robinson <andrew3@r3dsolutions.com> - 2012-11-06 13:14 -0800
      Re: Multi-dimensional list initialization Andrew Robinson <andrew3@r3dsolutions.com> - 2012-11-06 13:19 -0800
      Re: Multi-dimensional list initialization Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-06 15:46 -0700
        Re: Multi-dimensional list initialization Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2012-11-07 18:34 +1300
          Re: Multi-dimensional list initialization Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-07 14:00 +0000
          Re: Multi-dimensional list initialization Ethan Furman <ethan@stoneleaf.us> - 2012-11-07 06:47 -0800
          Re: Multi-dimensional list initialization Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-07 23:06 +0000
          Re: Multi-dimensional list initialization Greg Ewing <greg.ewing@canterbury.ac.nz> - 2012-11-08 14:29 +1300
    Re: Multi-dimensional list initialization wxjmfauth@gmail.com - 2012-11-05 01:55 -0800

Page 1 of 3  [1] 2 3  Next page →

#32748 — Multi-dimensional list initialization

So, here I was thinking "oh, this is a nice, easy way to initialize a 4D matrix" (running 2.7.3, non-core libs not allowed):

m = [[None] * 4] * 4

The way to get what I was after was:

m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]] 

(Obviously, I could have just hardcoded the initialization, but I'm too lazy to type all that out ;))

The behaviour I encountered seems a little contradictory to me. [None] * 4 creates four distinct elements in a single array while [[None] * 4] * 4 creates one distinct array of four distinct elements, with three references to it:

>>> a = [None] * 4
>>> a[0] = 'a'
>>> a
['a', None, None, None]

>>> m = [[None] * 4] * 4
>>> m[0][0] = 'm'
>>> m
[['m', None, None, None], ['m', None, None, None], ['m', None, None, None], ['m', None, None, None]]

Is this expected behaviour and if so, why? In my mind either result makes sense, but the inconsistency is what throws me off.

Demian Brecht
@demianbrecht
http://demianbrecht.github.com

[toc] | [next] | [standalone]

#32762

On 5/11/12 07:27:52, Demian Brecht wrote:
> So, here I was thinking "oh, this is a nice, easy way to initialize a 4D matrix"
> (running 2.7.3, non-core libs not allowed):
> 
> m = [[None] * 4] * 4
> 
> The way to get what I was after was:
> 
> m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]] 

Or alternateively:

m = [[None] * 4 for _ in range(4)]

> (Obviously, I could have just hardcoded the initialization, but I'm too
> lazy to type all that out ;))
> 
> The behaviour I encountered seems a little contradictory to me.
> [None] * 4 creates four distinct elements in a single array

Actually, it creates a list with four references to the same object.
But then, this object is immutable, so you won't notice that it's the
same object.

> while [[None] * 4] * 4 creates one distinct array of four distinct
> elements, with three references to it:

We usually phrase that as "a list with four references to the
same list".  The first reference is not special in any way.

>>>> a = [None] * 4
>>>> a[0] = 'a'
>>>> a
> ['a', None, None, None]
> 
>>>> m = [[None] * 4] * 4
>>>> m[0][0] = 'm'
>>>> m
> [['m', None, None, None], ['m', None, None, None], ['m', None, None, None], ['m', None, None, None]]
> 
> Is this expected behaviour

Yes.

> and if so, why? In my mind either result makes sense, but the
> inconsistency is what throws me off.

There's no inconsistency: in both cases you get a list with four
references to the same object.  The only difference is that in the
fist case, the references are to an immutable object, so the fact
that it's the same object won't hurt you.


Hope this helps,

-- HansM

[toc] | [prev] | [next] | [standalone]

#32791

On 5 November 2012 09:13, Hans Mulder <hansmu@xs4all.nl> wrote:
> On 5/11/12 07:27:52, Demian Brecht wrote:
>> So, here I was thinking "oh, this is a nice, easy way to initialize a 4D matrix"
>> (running 2.7.3, non-core libs not allowed):
>>
>> m = [[None] * 4] * 4
>>
>> The way to get what I was after was:
>>
>> m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]]
>
> Or alternateively:
>
> m = [[None] * 4 for _ in range(4)]

That's the way to do it.

I've seen this question many times between here and the python-tutor
list. It does seem to be a common gotcha.

I was just thinking to myself that it would be a hard thing to change
because the list would need to know how to instantiate copies of all
the different types of the elements in the list. Then I realised it
doesn't. It is simply a case of how the list multiplication operator
is implemented and whether it chooses to use a reference to the same
list or make a copy of that list. Since all of this is implemented
within the same list type it is a relatively easy change to make
(ignoring backward compatibility concerns).

I don't see this non-copying list multiplication behaviour as
contradictory but has anyone ever actually found a use for it?


Oscar

[toc] | [prev] | [next] | [standalone]

#32792

On Tue, Nov 6, 2012 at 12:32 PM, Oscar Benjamin
<oscar.j.benjamin@gmail.com> wrote:
> I was just thinking to myself that it would be a hard thing to change
> because the list would need to know how to instantiate copies of all
> the different types of the elements in the list. Then I realised it
> doesn't. It is simply a case of how the list multiplication operator
> is implemented and whether it chooses to use a reference to the same
> list or make a copy of that list. Since all of this is implemented
> within the same list type it is a relatively easy change to make
> (ignoring backward compatibility concerns).
>
> I don't see this non-copying list multiplication behaviour as
> contradictory but has anyone ever actually found a use for it?

Stupid example of why it can't copy:

bad = [open("test_file")] * 4

How do you clone something that isn't Plain Old Data? Ultimately,
that's where the problem comes from. It's easy enough to clone
something that's all scalars (strings, integers, None, etc) and
non-recursive lists/dicts of scalars, but anything more complicated
than that is rather harder.

If you want a deep copy and are prepared to handle any issues that
might result, you can do this:

>>> import copy
>>> a=[[2,3,4]]
>>> a.extend(copy.deepcopy(a))
>>> a[0][1]=10
>>> a
[[2, 10, 4], [2, 3, 4]]

And some things just won't work:
>>> bad.extend(copy.deepcopy(bad))
Traceback (most recent call last):
  File "<pyshell#17>", line 1, in <module>
    bad.extend(copy.deepcopy(bad))
  File "C:\Python32\lib\copy.py", line 147, in deepcopy
    y = copier(x, memo)
  File "C:\Python32\lib\copy.py", line 209, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "C:\Python32\lib\copy.py", line 166, in deepcopy
    rv = reductor(2)
TypeError: cannot serialize '_io.TextIOWrapper' object

The default behaviour is safe and reliable. When you want something
other than the default, there are ways of doing it.

ChrisA

[toc] | [prev] | [next] | [standalone]

#32793

On 6 November 2012 02:01, Chris Angelico <rosuav@gmail.com> wrote:
> On Tue, Nov 6, 2012 at 12:32 PM, Oscar Benjamin
> <oscar.j.benjamin@gmail.com> wrote:
>> I was just thinking to myself that it would be a hard thing to change
>> because the list would need to know how to instantiate copies of all
>> the different types of the elements in the list. Then I realised it
>> doesn't. It is simply a case of how the list multiplication operator
>> is implemented and whether it chooses to use a reference to the same
>> list or make a copy of that list. Since all of this is implemented
>> within the same list type it is a relatively easy change to make
>> (ignoring backward compatibility concerns).
>>
>> I don't see this non-copying list multiplication behaviour as
>> contradictory but has anyone ever actually found a use for it?
>
> Stupid example of why it can't copy:
>
> bad = [open("test_file")] * 4
>
> How do you clone something that isn't Plain Old Data? Ultimately,
> that's where the problem comes from. It's easy enough to clone
> something that's all scalars (strings, integers, None, etc) and
> non-recursive lists/dicts of scalars, but anything more complicated
> than that is rather harder.

That's not what I meant. But now you've made me realise that I was
wrong about what I did mean. In the case of

   stuff = [[obj] * n] * m

I thought that the multiplication of the inner list ([obj] * n) by m
could create a new list of lists using copies. On closer inspection I
see that the list being multiplied is in fact [[obj] * n] and that
this list can only know that it is a list of lists by inspecting its
element(s) which makes things more complicated.

I retract my claim that this change would be easy to implement.


Oscar

[toc] | [prev] | [next] | [standalone]

#32806

On 11/05/2012 06:30 PM, Oscar Benjamin wrote:
> On 6 November 2012 02:01, Chris Angelico<rosuav@gmail.com>  wrote:
>> On Tue, Nov 6, 2012 at 12:32 PM, Oscar Benjamin
>> <oscar.j.benjamin@gmail.com>  wrote:
>>> I was just thinking to myself that it would be a hard thing to change
>>> because the list would need to know how to instantiate copies of all
>>> the different types of the elements in the list. Then I realised it
>>> doesn't. It is simply a case of how the list multiplication operator
>>> is implemented and whether it chooses to use a reference to the same
>>> list or make a copy of that list. Since all of this is implemented
>>> within the same list type it is a relatively easy change to make
>>> (ignoring backward compatibility concerns).
>>>
>>> I don't see this non-copying list multiplication behaviour as
>>> contradictory but has anyone ever actually found a use for it?
>> Stupid example of why it can't copy:
>>
>> bad = [open("test_file")] * 4
>>
>> How do you clone something that isn't Plain Old Data? Ultimately,
>> that's where the problem comes from. It's easy enough to clone
>> something that's all scalars (strings, integers, None, etc) and
>> non-recursive lists/dicts of scalars, but anything more complicated
>> than that is rather harder.
> That's not what I meant. But now you've made me realise that I was
> wrong about what I did mean. In the case of
>
>     stuff = [[obj] * n] * m
>
> I thought that the multiplication of the inner list ([obj] * n) by m
> could create a new list of lists using copies. On closer inspection I
> see that the list being multiplied is in fact [[obj] * n] and that
> this list can only know that it is a list of lists by inspecting its
> element(s) which makes things more complicated.
>
> I retract my claim that this change would be easy to implement.
>
>
> Oscar
Hi Oscar,

In general, people don't use element multiplication (that I have *ever* 
seen) to make lists where all elements of the outer most list point to 
the same sub-*list* by reference.  The most common use of the 
multiplication is to fill an array with a constant, or short list of 
constants;  Hence, almost everyone has  to work around the issue as the 
initial poster did by using a much longer construction.

The most compact notation in programming really ought to reflect the 
most *commonly* desired operation.  Otherwise, we're really just making 
people do extra typing for no reason.

Further, list comprehensions take quite a bit longer to run than low 
level copies; by a factor of roughly 10. SO, it really would be worth 
implementing the underlying logic -- even if it wasn't super easy.

I really don't think doing a shallow copy of lists would break anyone's 
program.
The non-list elements, whatever they are, can be left as reference 
copies -- but any element which is a list ought to be shallow copied.  
The behavior observed in the opening post where modifying one element of 
a sub-list, modifies all elements of all sub-lists is never desired as 
far as I have ever witnessed.

The underlying implementation of Python can check an object type 
trivially, and the only routine needed is a shallow list copy.  So, no 
it really isn't a complicated operation to do shallow copies of lists.

:)

[toc] | [prev] | [next] | [standalone]

#32811

On Mon, 05 Nov 2012 21:51:24 -0800, Andrew Robinson wrote:

> The most compact notation in programming really ought to reflect the
> most *commonly* desired operation.  Otherwise, we're really just making
> people do extra typing for no reason.

There are many reasons not to put minimizing of typing ahead of all other 
values:

* Typically, code is written once and read many times. Minimizing 
  typing might save you a second or two once, and then cost you many
  seconds every time you read the code. That's why we tell people to
  choose meaningful variable names, instead of naming everything "a" 
  and "b".

* Consistency of semantics is better than a plethora of special 
  cases. Python has a very simple and useful rule: objects should
  not be copied unless explicitly requested to be copied. This is
  much better than having to remember whether this operation or
  that operation makes a copy. The answer is consistent: 

  (pardon me for belabouring the point here)

    Q: Does [0]*10 make ten copies of the integer object?
    A: No, list multiplication doesn't make copies of elements.

    Q: How about [0.0]*10?
    A: No, the elements are never copied.

    Q: What if I use a singleton? Does [None]*10 try to copy?
    A: No, the elements are never copied.

    Q: How about things like file objects that can't be copied?
    A: No, the elements are never copied.

    Q: What about [[]]*10?
    A: No, the elements are never copied.

    Q: How about if the elements are subclasses of list?
    A: No, the elements are never copied.

    Q: What about other mutable objects like sets or dicts?
    A: No, the elements are never copied.

    Q: What about instances of custom classes?
    A: No, the elements are never copied.

    Q: What if they are old-style Classic classes?
    A: No, the elements are never copied.

    Q: What if I do some funny tricks with the metaclass?
    A: No, the elements are never copied.

    Q: How about on Tuesdays? I bet they're copied on Tuesdays.
    A: No, the elements are never copied.



Your proposal throws away consistency for a trivial benefit on a rare use-
case, and replaces it with a bunch of special cases:

    Q: What about [[]]*10?
    A: Oh yeah, I forgot about lists, they're copied.

    Q: How about if the elements are subclasses of list?
    A: Hmmm, that's a good one, I'm not actually sure.

    Q: How about if I use delegation to proxy a list?
    A: Oh no, they definitely won't be copied.

    Q: What about other mutable objects like sets or dicts?
    A: No, definitely not. Unless people complain enough.

    Q: What about instances of custom classes?
    A: That's a definite maybe.

    Q: How about on Tuesdays? I bet they're copied on Tuesdays.
    A: Only if you're in Belgium.


Losing consistency in favour of saving a few characters for something as 
uncommon as list multiplication is a poor tradeoff. That's why this 
proposal has been rejected again and again and again every time it has 
been suggested.

List multiplication [x]*n is conceptually equivalent to:

newlist = []
for i in range(n):
    newlist.append(x)

or if you prefer a list comp:

[x for i in range(n)]

This is nice and simple and efficient. Some objects cannot be copied at 
all. Copying other objects is slow and inefficient. Keeping list 
multiplication consistent, and fast, is MUCH more important than making 
it work as expected for the rare case of 2D arrays:

[[0]*n]*m

where there are other alternatives.


> Further, list comprehensions take quite a bit longer to run than low
> level copies; by a factor of roughly 10. SO, it really would be worth
> implementing the underlying logic -- even if it wasn't super easy.

Copying those elements does not come for free.

It is true that list multiplication can be much faster than a list comp. 
But that's because the list multiply doesn't have to inspect the 
elements, copy them, or engage the iteration machinery. Avoiding copying 
gives you a big saving:


[steve@ando ~]$ python3.3 -m timeit -s "x = range(1000)" 
"[x for _ in range(100)]"  # not copied
100000 loops, best of 3: 11.9 usec per loop

[steve@ando utilities]$ python3.3 -m timeit -s "x = range(1000)" 
"[x[:] for _ in range(100)]"  # copied
10000 loops, best of 3: 103 usec per loop

So there's a factor of ten difference right there. If list multiplication 
had to make copies, it would lose much of its speed advantage. For large 
enough lists, or complicated enough objects, it would become slower than 
a list comprehension.

It would be even slower if list multiplication had to inspect each 
element first and decide whether or not to copy.



> I really don't think doing a shallow copy of lists would break anyone's
> program.

Anyone who is currently using list multiplication with mutable objects is 
expecting that they will be the same object, and relying on that fact. 
Otherwise they wouldn't be using list multiplication.

You're suggesting a semantic change. Therefore they will be expecting 
something different from what actually happens. Result: broken code.

It's not just mutable objects. It's also objects that can't be copied. 
Result: mylist*3 used to work, now it raises an exception. And 
performance issues: what used to be fast is now slow.

Even if this change was allowed, it would have to go through a multi-year 
process. Python 3.3 is too late -- the absolute earliest would be Python 
3.4, which is scheduled for about 18 months from now. So in Python 3.4 
you could write:

from __future__ import list_multiplication_copying

to get the behaviour you want, and then in Python 3.5 it would become the 
default. That's three years until it becomes the standard. Meanwhile, 
there will still be millions of people using Python 2.7 or 3.2, and their 
code will behave differently from your code.

Conservatively, if you write code to support three previous releases, 
that means you can't use this feature until Python 3.7. So that's about 
six years before it can be used widely.

If the problem being solved was big enough, this would be worth doing. 
But it's not.


> The non-list elements, whatever they are, can be left as reference
> copies -- but any element which is a list ought to be shallow copied.

That's even worse than "list multiplication always copies". At least that 
is simple and consistent, even if it isn't consistent with the rest of 
the language, at least it is self-consistent. You are proposing something 
much worse: special cases to remember. "Objects aren't copied, except for 
lists, which are copied."

And then people will wonder why sets aren't copied, and dicts. People 
will make a 2D array like so:

[[0]*5]*10

and it will work. Then they'll write this:

[{}]*5

and wonder why it doesn't work the way they expect. Consistency is *much* 
more valuable than ad hoc DWIM semantics. Languages that try to Do What I 
Mean somehow end up Doing What Somebody Else Meant, But Not Me.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#32817

Well said Steve, I agree with you...
-Shambhu

-----Original Message-----
From: Steven D'Aprano [mailto:steve+comp.lang.python@pearwood.info] 
Sent: Tuesday, November 06, 2012 2:35 PM
To: python-list@python.org
Subject: Re: Multi-dimensional list initialization

On Mon, 05 Nov 2012 21:51:24 -0800, Andrew Robinson wrote:

> The most compact notation in programming really ought to reflect the 
> most *commonly* desired operation.  Otherwise, we're really just 
> making people do extra typing for no reason.

There are many reasons not to put minimizing of typing ahead of all other
values:

* Typically, code is written once and read many times. Minimizing
  typing might save you a second or two once, and then cost you many
  seconds every time you read the code. That's why we tell people to
  choose meaningful variable names, instead of naming everything "a" 
  and "b".

* Consistency of semantics is better than a plethora of special
  cases. Python has a very simple and useful rule: objects should
  not be copied unless explicitly requested to be copied. This is
  much better than having to remember whether this operation or
  that operation makes a copy. The answer is consistent: 

  (pardon me for belabouring the point here)

    Q: Does [0]*10 make ten copies of the integer object?
    A: No, list multiplication doesn't make copies of elements.

    Q: How about [0.0]*10?
    A: No, the elements are never copied.

    Q: What if I use a singleton? Does [None]*10 try to copy?
    A: No, the elements are never copied.

    Q: How about things like file objects that can't be copied?
    A: No, the elements are never copied.

    Q: What about [[]]*10?
    A: No, the elements are never copied.

    Q: How about if the elements are subclasses of list?
    A: No, the elements are never copied.

    Q: What about other mutable objects like sets or dicts?
    A: No, the elements are never copied.

    Q: What about instances of custom classes?
    A: No, the elements are never copied.

    Q: What if they are old-style Classic classes?
    A: No, the elements are never copied.

    Q: What if I do some funny tricks with the metaclass?
    A: No, the elements are never copied.

    Q: How about on Tuesdays? I bet they're copied on Tuesdays.
    A: No, the elements are never copied.



Your proposal throws away consistency for a trivial benefit on a rare use- case, and replaces it with a bunch of special cases:

    Q: What about [[]]*10?
    A: Oh yeah, I forgot about lists, they're copied.

    Q: How about if the elements are subclasses of list?
    A: Hmmm, that's a good one, I'm not actually sure.

    Q: How about if I use delegation to proxy a list?
    A: Oh no, they definitely won't be copied.

    Q: What about other mutable objects like sets or dicts?
    A: No, definitely not. Unless people complain enough.

    Q: What about instances of custom classes?
    A: That's a definite maybe.

    Q: How about on Tuesdays? I bet they're copied on Tuesdays.
    A: Only if you're in Belgium.


Losing consistency in favour of saving a few characters for something as uncommon as list multiplication is a poor tradeoff. That's why this proposal has been rejected again and again and again every time it has been suggested.

List multiplication [x]*n is conceptually equivalent to:

newlist = []
for i in range(n):
    newlist.append(x)

or if you prefer a list comp:

[x for i in range(n)]

This is nice and simple and efficient. Some objects cannot be copied at all. Copying other objects is slow and inefficient. Keeping list multiplication consistent, and fast, is MUCH more important than making it work as expected for the rare case of 2D arrays:

[[0]*n]*m

where there are other alternatives.


> Further, list comprehensions take quite a bit longer to run than low 
> level copies; by a factor of roughly 10. SO, it really would be worth 
> implementing the underlying logic -- even if it wasn't super easy.

Copying those elements does not come for free.

It is true that list multiplication can be much faster than a list comp. 
But that's because the list multiply doesn't have to inspect the elements, copy them, or engage the iteration machinery. Avoiding copying gives you a big saving:


[steve@ando ~]$ python3.3 -m timeit -s "x = range(1000)" 
"[x for _ in range(100)]"  # not copied
100000 loops, best of 3: 11.9 usec per loop

[steve@ando utilities]$ python3.3 -m timeit -s "x = range(1000)" 
"[x[:] for _ in range(100)]"  # copied
10000 loops, best of 3: 103 usec per loop

So there's a factor of ten difference right there. If list multiplication had to make copies, it would lose much of its speed advantage. For large enough lists, or complicated enough objects, it would become slower than a list comprehension.

It would be even slower if list multiplication had to inspect each element first and decide whether or not to copy.



> I really don't think doing a shallow copy of lists would break anyone's
> program.

Anyone who is currently using list multiplication with mutable objects is 
expecting that they will be the same object, and relying on that fact. 
Otherwise they wouldn't be using list multiplication.

You're suggesting a semantic change. Therefore they will be expecting 
something different from what actually happens. Result: broken code.

It's not just mutable objects. It's also objects that can't be copied. 
Result: mylist*3 used to work, now it raises an exception. And 
performance issues: what used to be fast is now slow.

Even if this change was allowed, it would have to go through a multi-year 
process. Python 3.3 is too late -- the absolute earliest would be Python 
3.4, which is scheduled for about 18 months from now. So in Python 3.4 
you could write:

from __future__ import list_multiplication_copying

to get the behaviour you want, and then in Python 3.5 it would become the 
default. That's three years until it becomes the standard. Meanwhile, 
there will still be millions of people using Python 2.7 or 3.2, and their 
code will behave differently from your code.

Conservatively, if you write code to support three previous releases, 
that means you can't use this feature until Python 3.7. So that's about 
six years before it can be used widely.

If the problem being solved was big enough, this would be worth doing. 
But it's not.


> The non-list elements, whatever they are, can be left as reference
> copies -- but any element which is a list ought to be shallow copied.

That's even worse than "list multiplication always copies". At least that 
is simple and consistent, even if it isn't consistent with the rest of 
the language, at least it is self-consistent. You are proposing something 
much worse: special cases to remember. "Objects aren't copied, except for 
lists, which are copied."

And then people will wonder why sets aren't copied, and dicts. People 
will make a 2D array like so:

[[0]*5]*10

and it will work. Then they'll write this:

[{}]*5

and wonder why it doesn't work the way they expect. Consistency is *much* 
more valuable than ad hoc DWIM semantics. Languages that try to Do What I 
Mean somehow end up Doing What Somebody Else Meant, But Not Me.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#32845

On 11/06/2012 01:04 AM, Steven D'Aprano wrote:
> On Mon, 05 Nov 2012 21:51:24 -0800, Andrew Robinson wrote:
>
>> The most compact notation in programming really ought to reflect the
>> most *commonly* desired operation.  Otherwise, we're really just making
>> people do extra typing for no reason.
> There are many reasons not to put minimizing of typing ahead of all other
> values:
I didn't.  I put it ahead of *some* values for the sake of practicality 
and human psychology.
" Practicality beats purity. "

>
> * Typically, code is written once and read many times. Minimizing
>    typing might save you a second or two once, and then cost you many
>    seconds every time you read the code. That's why we tell people to
>    choose meaningful variable names, instead of naming everything "a"
>    and "b".
Yes.  But this isn't going to cost any more time than figuring out 
whether or not the list multiplication is going to cause quirks, 
itself.  Human psychology *tends* (it's a FAQ!) to automatically assume 
the purpose of the list multiplication is to pre-allocate memory for the 
equivalent (using lists) of a multi-dimensional array.  Note the OP even 
said "4d array".

The OP's original construction was simple, elegant, easy to read and 
very commonly done by newbies learning the language because it's 
*intuitive*.  His second try was still intuitive, but less easy to read, 
and not as elegant.

>
> * Consistency of semantics is better than a plethora of special
>    cases. Python has a very simple and useful rule: objects should
>    not be copied unless explicitly requested to be copied. This is
>    much better than having to remember whether this operation or
>    that operation makes a copy. The answer is consistent:
Bull.  Even in the last thread I noted the range() object produces 
special cases.
 >>> range(0,5)[1]
1
 >>> range(0,5)[1:3]
range(1, 3)
 >>>

The principle involved is that it gives you what you *usually* want;  I 
read some of the documentation on why Python 3 chose to implement it 
this way.

>
>    (pardon me for belabouring the point here)
>
>      Q: Does [0]*10 make ten copies of the integer object?
>      A: No, list multiplication doesn't make copies of elements.
Neither would my idea for the vast majority of things on your first list.

     Q: What about [[]]*10?
     A: No, the elements are never copied.

YES! For the obvious reason that such a construction is making mutable 
lists that the user wants to populate later.  If they *didn't* want to 
populate them later, they ought to have used tuples -- which take less 
overhead.  Who even does this thing you are suggesting?!

 >>> a=[[]]*10
 >>> a
[[], [], [], [], [], [], [], [], [], []]
 >>> a[0].append(1)
 >>> a
[[1], [1], [1], [1], [1], [1], [1], [1], [1], [1]]

Oops! Damn, not what anyone normal wants....

>      Q: How about if the elements are subclasses of list?
>      A: No, the elements are never copied.
Another poster brought that point up -- it's something I would have to 
study before answering.
It's a valid objection.

>
>      Q: What about other mutable objects like sets or dicts?
>      A: No, the elements are never copied.
They aren't list multiplication compatible in any event! It's a total 
nonsense objection.

If these are inconsistent in my idea -- OBVIOUSLY -- they are 
inconsistent in Python's present implementation.  You can't even 
reference duplicate them NOW.

 >>> { 1:'a', 2:'b', 3:'c' } * 2
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for *: 'dict' and 'int'

>      Q: How about on Tuesdays? I bet they're copied on Tuesdays.
>      A: No, the elements are never copied.
That's really a stupid objection, and everyone knows it.
" Although that way may not be obvious at first unless you're Dutch. "

>
> Your proposal throws away consistency for a trivial benefit on a rare use-
> case, and replaces it with a bunch of special cases:
RARE!!!! You are NUTS!!!!

>      Q: What about [[]]*10?
>      A: Oh yeah, I forgot about lists, they're copied.
Yup.

>      Q: How about if the elements are subclasses of list?
>      A: Hmmm, that's a good one, I'm not actually sure.
>
>      Q: How about if I use delegation to proxy a list?
>      A: Oh no, they definitely won't be copied.
Give an example usage of why someone would want to do this.  Then we can 
discuss it.
>      Q: What about other mutable objects like sets or dicts?
>      A: No, definitely not. Unless people complain enough.
now you're just repeating yourself to make your contrived list longer -- 
but there's no new objections...

> Losing consistency in favour of saving a few characters for something as
> uncommon as list multiplication is a poor tradeoff. That's why this
> proposal has been rejected again and again and again every time it has
> been suggested.
Please link to the objection being proposed to the developers, and their 
reasoning for rejecting it.
I think you are exaggerating.

> List multiplication [x]*n is conceptually equivalent to:
> <snip>
> This is nice and simple and efficient.
No it isn't efficient. It's *slow* when done as in your example.

> Copying other objects is slow and inefficient. Keeping list
> multiplication consistent, and fast, is MUCH more important than making
> it work as expected for the rare case of 2D arrays:
I don't think so -- again, look at range(); it was made to work 
inconsistent for a "common" case.

Besides, 2D arrays are *not* rare and people *have* to copy internals of 
them very often.
The copy speed will be the same or *faster*, and the typing less -- and 
the psychological mistakes *less*, the elegance more.

It's hardly going to confuse anyone to say that lists are copied with 
list multiplication, but the elements are not.

Every time someone passes a list to a function, they *know* that the 
list is passed by value -- and the elements are passed by reference.  
People in Python are USED to lists being "the" way to weird behavior 
that other languages don't do.

>
> Copying those elements does not come for free.
>
> It is true that list multiplication can be much faster than a list comp.
> But that's because the list multiply doesn't have to inspect the
> elements, copy them, or engage the iteration machinery. Avoiding copying
> gives you a big saving:
>
>
> [steve@ando ~]$ python3.3 -m timeit -s "x = range(1000)"
> "[x for _ in range(100)]"  # not copied
> 100000 loops, best of 3: 11.9 usec per loop
>
> [steve@ando utilities]$ python3.3 -m timeit -s "x = range(1000)"
> "[x[:] for _ in range(100)]"  # copied
> 10000 loops, best of 3: 103 usec per loop
>
> So there's a factor of ten difference right there. If list multiplication
> had to make copies, it would lose much of its speed advantage.
And when multiplication doesn't make copies of *lists*, it's going 
"nowhere fast", because people don't want the results that gives.

So what difference does it make?  People won't make the construction 
unless they wanted to make the copies in the first place.  If they want 
the copies, well -- copies are *slow*.  Big deal.

>   For large
> enough lists, or complicated enough objects, it would become slower than
> a list comprehension.
Huh? You're nuts.

> It would be even slower if list multiplication had to inspect each
> element first and decide whether or not to copy.
A single pointer comparison in a 'C' for loop takes less than 5 nano 
seconds on a 1Ghz machine.
(I'll bet yours is faster than that...!)
Consider: list objects have a pointer which points back to the generic 
list object -- that's all it takes to determine what the "type" is.

Your measured loop times, doing list comprehensions takes over 10 
microseconds *per loop*.
Compared to what you're proposing -- The pointer compare is a mere 0.05% 
change;  You can't even measure that with "timeit!".  BUT: The increase 
in speed for not running tokenized "for" loops is *much* bigger than the 
loss for a single pointer compare; so it will *usually* be a *serious* 
net gain.

>> I really don't think doing a shallow copy of lists would break anyone's
>> program.
> Anyone who is currently using list multiplication with mutable objects is
> expecting that they will be the same object, and relying on that fact.
> Otherwise they wouldn't be using list multiplication.
yes, and I'm not changing that -- except for lists; and *no* one is 
using that.
Find two examples of it from existing non contrived web examples of 
Python code.
*ask* around.

>
> You're suggesting a semantic change. Therefore they will be expecting
> something different from what actually happens. Result: broken code.
Even if it was;  So are many semantic changes happening between python 2 
and python 3.
Look at what python 2 did:

 >>> range(0,5)[0]
0
 >>> range(0,5)[1:3]
[1, 2]

That's a *semantic* change.
Also; if you complain that xrange has been renamed range; then look:

 >>> xrange(0,5)[0]
0
 >>> xrange(0,5)[1:3]
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: sequence index must be integer, not 'slice'

WOW. WOW. WOW.  An even BIGGER semantic change.


> It's not just mutable objects. It's also objects that can't be copied.
> Result: mylist*3 used to work, now it raises an exception. And
> performance issues: what used to be fast is now slow.
Where do you get off??; a list can be copied -- the contents might not.

> Even if this change was allowed, it would have to go through a multi-year
> process.
Fine.  if that's normal -- then let them process it the normal way.  
That's not my concern in the slightest.

> to get the behaviour you want, and then in Python 3.5 it would become the
> default. That's three years until it becomes the standard. Meanwhile,
> there will still be millions of people using Python 2.7 or 3.2, and their
> code will behave differently from your code.
Uh, they aren't *using* the construction I am proposing now -- they are 
avoiding it like the plague.
Hence, it will merely become a new ability in a few years -- not 
'differently' behaving code.

The rest of your repetitive nonsense has been deleted.
:(

[toc] | [prev] | [next] | [standalone]

#32860

On Tue, 06 Nov 2012 14:41:24 -0800, Andrew Robinson wrote:

> Yes.  But this isn't going to cost any more time than figuring out
> whether or not the list multiplication is going to cause quirks, itself.
>  Human psychology *tends* (it's a FAQ!) to automatically assume the
> purpose of the list multiplication is to pre-allocate memory for the
> equivalent (using lists) of a multi-dimensional array.  Note the OP even
> said "4d array".

I'm not entirely sure what your point is here. The OP screwed up -- he 
didn't generate a 4-dimensional array. He generated a 2-dimensional 
array. If his intuition about the number of dimensions is so poor, why 
should his intuition about list multiplication be treated as sacrosanct?

As they say, the only truly intuitive interface is the nipple. There are 
many places where people's intuition about programming fail. And many 
places where Fred's intuition is the opposite of Barney's intuition.

Even more exciting, there are places where people's intuition is 
*inconsistent*, where they expect a line of code to behave differently 
depending on their intention, rather than on the code. And intuition is 
often sub-optimal: e.g. isn't it intuitively obvious that "42" + 1 should 
give 43? (Unless it is intuitively obvious that it should give 421.)

So while I prefer intuitively obvious behaviour where possible, it is not 
the holy grail, and I am quite happy to give it up.


> The OP's original construction was simple, elegant, easy to read and
> very commonly done by newbies learning the language because it's
> *intuitive*.  His second try was still intuitive, but less easy to read,
> and not as elegant.

Yes. And list multiplication is one of those areas where intuition is 
suboptimal -- it produces a worse outcome overall, even if one minor use-
case gets a better outcome.

I'm not disputing that [[0]*n]*m is intuitively obvious and easy. I'm 
disputing that this matters. Python would be worse off if list 
multiplication behaved intuitively.

An analogy: the intuitively obvious thing to do with a screw is to bang 
it in with a hammer. It's long, thin, has a point at the end, and a flat 
head that just screams "hit me". But if you do the intuitive thing, your 
carpentry will be *much worse* than the alternatives -- a hammered in 
screw holds much less strongly than either a nail or a screwed in screw. 
The surface area available for gripping is about 2% compared to a nail 
and about 0.01% compared to a screw used correctly.

Having list multiplication copy has consequences beyond 2D arrays. Those 
consequences make the intuitive behaviour you are requesting a negative 
rather than a positive. If that means that newbie programmers have to 
learn not to hammer screws in, so be it. It might be harder, slower, and 
less elegant to drill a pilot hole and then screw the screw in, but the 
overall result is better.


>> * Consistency of semantics is better than a plethora of special
>>    cases. Python has a very simple and useful rule: objects should not
>>    be copied unless explicitly requested to be copied. This is much
>>    better than having to remember whether this operation or that
>>    operation makes a copy. The answer is consistent:
> 
> Bull.  Even in the last thread I noted the range() object produces
> special cases.
>  >>> range(0,5)[1]
> 1
>  >>> range(0,5)[1:3]
> range(1, 3)

What's the special case here? What do you think is copied?

You take a slice of a tuple, you get a new tuple.

You take a slice of a list, you get a new list.

You take a slice of a range object, you get a new range object.

I'm honestly not getting what you think is inconsistent about this.



> The principle involved is that it gives you what you *usually* want;

Who is the "you" that decides what "you" usually want? And how do they 
know what is "usual"?

Two-dimensional arrays in Python using lists are quite rare. Anyone who 
is doing serious numeric work where they need 2D arrays is using numpy, 
not lists. There are millions of people using Python, so it's hardly 
surprising that once or twice a year some newbie trips over this. But 
it's not something that people tend to trip over again and again and 
again, like C's "assignment is an expression" misfeature.


> I read some of the documentation on why Python 3 chose to implement it
> this way.

What documentation is this? Because this is a design decision that goes 
all the way back to at least Python 1.5:

[steve@ando ~]$ python1.5
Python 1.5.2 (#1, Aug 27 2012, 09:09:18)  [GCC 4.1.2 20080704 (Red Hat 
4.1.2-52)] on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> x = [[0]*5]*3
>>> x[0][1] = 99
>>> x
[[0, 99, 0, 0, 0], [0, 99, 0, 0, 0], [0, 99, 0, 0, 0]]


So I expect the design decision for Python 3 was "we made the right 
decision before, there's no need to change it".



>>    (pardon me for belabouring the point here)
>>
>>      Q: Does [0]*10 make ten copies of the integer object? A: No, list
>>      multiplication doesn't make copies of elements.
>
> Neither would my idea for the vast majority of things on your first
> list.

Um, yes? The point is that "vast majority" is not "everything". Hence, 
your suggested behaviour is inconsistent.



>      Q: What about [[]]*10?
>      A: No, the elements are never copied.
> 
> YES! For the obvious reason that such a construction is making mutable
> lists that the user wants to populate later.  If they *didn't* want to
> populate them later, they ought to have used tuples -- which take less
> overhead.  Who even does this thing you are suggesting?!

Who knows? Who cares? Nobody does:

n -= n

instead of just n=0, but that doesn't mean that we should give it some 
sort of special meaning different from n -= m. If it turns out that the 
definition of list multiplication is such that NOBODY, EVER, uses [[]]*n, 
that is *still* not a good reason for special-casing it. All it means is 
that this will be a less-obscure example of the billions of things which 
can be done in Python but nobody wants to.

You have quoted from the Zen of Python a few times in this post. Perhaps 
you missed one of the most critical ones?

Special cases aren't special enough to break the rules.

There are perfectly good ways to generate a 2D array out of lists, and 
even better reasons not to use lists for that in the first place. (Numpy 
arrays are much better suited for serious work.)


>>      Q: What about other mutable objects like sets or dicts? A: No, the
>>      elements are never copied.
>
> They aren't list multiplication compatible in any event! It's a total
> nonsense objection.

I'm afraid you've just lost an awful lot of credibility there.

py> x = [{}]*5
py> x
[{}, {}, {}, {}, {}]
py> x[0]['key'] = 1
py> x
[{'key': 1}, {'key': 1}, {'key': 1}, {'key': 1}, {'key': 1}]

And similarly for any other mutable object.

If you don't understand that lists can contain other mutable objects 
apart from lists, then you really shouldn't be discussing this issue.


>> Your proposal throws away consistency for a trivial benefit on a rare
>> use- case, and replaces it with a bunch of special cases:
>
> RARE!!!! You are NUTS!!!!

Yes, rare. I base that on about 15 years of Python coding and many 
thousands (tens of thousands?) of hours on Python forums like this one. 
What's your opinion based on?

List multiplication is rare enough, but when it is used, it is usually 
used to generate a 1D array like this:

values = [None]*n  # or 0 is another popular starting value

Using it twice to generate a 2D array is even rarer.


>>      Q: How about if I use delegation to proxy a list? A: Oh no, they
>>      definitely won't be copied.
>
> Give an example usage of why someone would want to do this.  Then we can
> discuss it.

Proxying objects is hardly a rare scenario. Delegation is less common 
since you can subclass built-ins, but it is still used. It is a standard 
design pattern.


>> Losing consistency in favour of saving a few characters for something
>> as uncommon as list multiplication is a poor tradeoff. That's why this
>> proposal has been rejected again and again and again every time it has
>> been suggested.
>
> Please link to the objection being proposed to the developers, and their
> reasoning for rejecting it.
> I think you are exaggerating.

Python is a twenty year old language. Do you really think this is the 
first time somebody has noticed it?

It's hard to search for discussions on the dev list, because the obvious 
search terms bring up many false positives. But here are a couple of bug 
reports closed as "won't fix":

http://bugs.python.org/issue1408
http://bugs.python.org/issue12597

I suspect it is long past time for a PEP so this can be rejected once and 
for all.


>> List multiplication [x]*n is conceptually equivalent to: <snip>
>> This is nice and simple and efficient.
> No it isn't efficient. It's *slow* when done as in your example.

Well of course it is slow*er* when you move it from low-level C to high 
level Python, but it is still fast.

>> Copying other objects is slow and inefficient. Keeping list
>> multiplication consistent, and fast, is MUCH more important than making
>> it work as expected for the rare case of 2D arrays:
>
> I don't think so -- again, look at range(); it was made to work
> inconsistent for a "common" case.

You mentioned range before, but it isn't clear to me what you think is 
inconsistent about it.


> Besides, 2D arrays are *not* rare and people *have* to copy internals of
> them very often.

So you say.


> The copy speed will be the same or *faster*, and the typing less -- and
> the psychological mistakes *less*, the elegance more.

You think that it is *faster* to copy a list than to make a new pointer 
to it? Your credibility is not looking too good here.


> It's hardly going to confuse anyone to say that lists are copied with
> list multiplication, but the elements are not.

Well, that confuses me. What about a list where the elements are lists? 
Are they copied?

What about other mutable objects? Are they copied?

What about mutable objects which are uncopyable, like file objects?


> Every time someone passes a list to a function, they *know* that the
> list is passed by value -- and the elements are passed by reference.

And there goes the last of your credibility. *You* might "know" this, but 
that doesn't make it so.

Python does not use either call-by-value or call-by-reference, and it 
certainly doesn't use different calling conventions for different 
arguments or parts of arguments. Everything is passed using the same 
calling convention. Start here:

http://mail.python.org/pipermail/tutor/2010-December/080505.html


> People in Python are USED to lists being "the" way to weird behavior
> that other languages don't do.

Python's calling behaviour is identical to that used by languages 
including Java (excluding unboxed primitives) and Ruby, to mention only 
two.

You're starting to shout and yell, so perhaps it's best if I finish this 
here.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#32876

On 2012-11-06, at 5:55 PM, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> I'm not entirely sure what your point is here. The OP screwed up -- he 
> didn't generate a 4-dimensional array. He generated a 2-dimensional 
> array. If his intuition about the number of dimensions is so poor, why 
> should his intuition about list multiplication be treated as sacrosanct?

Yep, I may have mis-worded the explanation a bit (although I *did* express that it was a 4D matrix in the OP). I was using a 2D list to represent a 4D matrix in order to easily iterate over 90 degree rotations with zip(*matrix[::-1]). It wasn't for production code (otherwise I *would* be using numpy), it was for an online programming challenge in which external libs are not supported.

> As they say, the only truly intuitive interface is the nipple. There are 
> many places where people's intuition about programming fail. And many 
> places where Fred's intuition is the opposite of Barney's intuition.

I couldn't agree more with this. My question was *not* based on what I perceive to be intuitive (although most of this thread has now seemed to devolve into that and become more of a philosophical debate), but was based on what I thought may have been inconsistent behaviour (which was quickly cleared up with None being immutable and causing it to *seem* that the behaviour was inconsistent to the forgetful mind). As you touch on here, "intuition" is entirely subjective. If you're coming from a C/C++ background, I'd think that your intuition would be that everything's passed by value unless explicitly stated. Someone coming from another background (Lua perhaps?) would likely have entirely different intuition.

> So while I prefer intuitively obvious behaviour where possible, it is not 
> the holy grail, and I am quite happy to give it up.

I fail to see where there has been any giving up on intuitiveness in the context of this particular topic. In my mind, intuitiveness is generally born of repetitiveness and consistency. As everything in Python is a reference, it would seem to me to be inconsistent to treat expressions such as [[obj]*4]*4 un-semantically (Pythonically speaking) and making it *less* intuitive. I agree that Python would definitely be worse off.

Demian Brecht
@demianbrecht
http://demianbrecht.github.com

[toc] | [prev] | [next] | [standalone]

#32880

Le mercredi 7 novembre 2012 02:55:10 UTC+1, Steven D'Aprano a écrit :

> 
> 
> 
> 
> 
> 
> Two-dimensional arrays in Python using lists are quite rare. Anyone who 
> 
> is doing serious numeric work where they need 2D arrays is using numpy, 
> 
> not lists. There are millions of people using Python, so it's hardly 
> 
> surprising that once or twice a year some newbie trips over this. But 
> 
> it's not something that people tend to trip over again and again and 
> 
> again, like C's "assignment is an expression" misfeature.
> 
> 

--------------------


>>> from vecmat6 import *
>>> from vmio5 import *
Traceback (most recent call last):
  File "<eta last command>", line 1, in <module>
ImportError: No module named vmio5
>>> from vmio6 import *
>>> from svdecomp6 import *
>>> mm = NewMat(3, 3)
>>> mm
[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]
>>> mm[0][0] = 1.0; mm[0][1] = 2.0; mm[0][2] = 3.0
>>> mm[1][0] = 11.0; mm[1][1] = 12.0; mm[1][2] = 13.0
>>> mm[2][0] = 21.0; mm[2][1] = 22.0; mm[2][2] = 23.0
>>> pr(mm, 'mm=')
mm=
(   1.00000e+000  2.00000e+000  3.00000e+000 )
(   1.10000e+001  1.20000e+001  1.30000e+001 )
(   2.10000e+001  2.20000e+001  2.30000e+001 )
>>> aa, b, cc = SVDecomp(mm)
>>> pr(aa, 'aa=')
aa=
(  -8.08925e-002 -9.09280e-001  4.08248e-001 )
(  -4.77811e-001 -3.24083e-001 -8.16497e-001 )
(  -8.74730e-001  2.61114e-001  4.08248e-001 )
>>> pr(b, 'b=')
b=
(   4.35902e+001  1.37646e+000  1.93953e-016 )
>>> pr(cc, 'cc=')
cc=
(  -5.43841e-001  7.33192e-001  4.08248e-001 )
(  -5.76726e-001  2.68499e-002 -8.16497e-001 )
(  -6.09610e-001 -6.79492e-001  4.08248e-001 )
>>> bb = VecToDiagMat(b)
>>> cct = TransposeMat(cc)
>>> oo = MatMulMatMulMat(aa, bb, cct)
>>> pr(oo, 'aa * bb * cct=')
aa * bb * cct=
(   1.00000e+000  2.00000e+000  3.00000e+000 )
(   1.10000e+001  1.20000e+001  1.30000e+001 )
(   2.10000e+001  2.20000e+001  2.30000e+001 )
>>>
>>> # or
>>> oo
[[0.9999999999999991, 1.9999999999999993, 2.9999999999999982],
[10.999999999999995, 11.99999999999999, 12.999999999999996],
[20.999999999999986, 21.999999999999975, 22.999999999999986]]



jmf

[toc] | [prev] | [next] | [standalone]

#32909

On 07/11/2012 22:02, Andrew Robinson wrote:
>

You're doing extremely well, you've overtaken Xah Lee as the biggest 
waste of space on this list.

-- 
Cheers.

Mark Lawrence.

[toc] | [prev] | [next] | [standalone]

#32919

On Wed, Nov 7, 2012 at 3:02 PM, Andrew Robinson
<andrew3@r3dsolutions.com> wrote:
> Draw up some use cases for the multiplication operator (I'm calling on your
> experience, let's not trust mine, right?);  What are all the Typical ways
> people *Do* to use it now?
>
> If those use cases do not *primarily* center around *wanting* an effect
> explicitly caused by reference duplication -- then it may be better to
> abolish list multiplication all together; and rather, improve the list
> comprehensions to overcome the memory, clarity, and speed pitfalls in the
> most common case of initializing a list.

Why?  Just to get rid of an FAQ?

Here's one of the more interesting uses from my own code:

    values = zip(samples, times * num_groups)
    if len(values) < len(times) * num_groups:
        # raise an error

Converting that multiplication to a generator expression would look like this:

    values = zip(samples, (t for _ in range(num_groups) for t in times))

That's not particularly hairy, but I do assert that it is
substantially less readable, and more so because it loses the symmetry
with the following if condition.

The recipes in the itertools docs also include this example, which
notably depends on the list containing multiple references to the same
iterator:

def grouper(n, iterable, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

Replacing the list multiplication in that function with a list
comprehension would be awkward, as the obvious replacement of
[iter(iterable) for _ in range(n)] would produce different results.


> For example, in initialization use cases; often the variable of a for loop
> isn't needed and all the initializers have parameters which only need to be
> evaluated *once* (no side effects).
>
> Hence, there is an opportunity for speed and memory gains,while maintaining
> clarity and *consistency*.
>
> Some ideas of use cases:
> [ (0) in xrange(10) ]  # The function to create a tuple cache's the
> parameter '0', makes 10 (0)'s
> [ dict.__new__(dict) in xrange(10) ]  # dict.__new__, The dict parameter is
> cached -- makes 10 dicts.
> [ lambda x:(0) in xrange(10) ] # lambda caches (0), returns a *reference* to
> it multiple times.

How exactly do you propose to indicate to the compiler which parts of
the expressions are meant to be cached, and which are not?

>>> Bull.  Even in the last thread I noted the range() object produces
>>> special cases.
>>>  >>> range(0,5)[1]
>>> 1
>>>  >>> range(0,5)[1:3]
>>> range(1, 3)
>>
>> What's the special case here? What do you think is copied?
>>
>>
>> You take a slice of a range object, you get a new range object.
>
> You were'nt paying attention, OCCASIONALLY, get an integer, or a list.
>>>> range(3)[2]
> 2
>
> LOOOOK! That's not a range object, that's an integer.  Use Python 3.2 and
> try it.

Of course you got an integer.  You took an index of the range object,
not a slice.  The rule is that taking an index of a sequence returns
an element; taking a slice of a sequence returns a sub-sequence.  You
still have not shown any inconsistency here.

> Game programmers routinely use 2D lists to represent the screen layout;
> For example, they might use 'b' to represent a brick tile, and 'w' to
> represent a water tile.

In many cases it may be simpler to use a plain list of strings:

screen = [
    "sssss",
    "ssbss",
    "sbbbs",
    "bbbbb",
]

> py> x = [{}]*5
> py> x
> [{}, {}, {}, {}, {}]
>
> No, I showed what happed when you do {}*3;
> That *DOESN'T* work;  You aren't multiplying the dictionary, you are
> multiplying the LIST of dictionaries.  Very different things.
> You were complaining that my method doesn't multiply them -- well, gee --
> either mine DOES or python DOESN'T.  Double standards are *crap*.

No, he wasn't.  He was talking about multiplying lists of dicts, and
whether the dicts are then copied or not, just like every other Q&A
item in that dialogue was concerning whether item X in a list should
expect to be copied when the containing list is multiplied.

You are the only one talking about applying the multiplication
operator to dicts.

> Huh?
> I'm not yelling any more than you are.  Are ???YOU??? yelling?

Perhaps you're not aware that on the Internet, TYPING IN ALL CAPS is
commonly construed as SHOUTING.

[toc] | [prev] | [next] | [standalone]

#32933

On Wed, Nov 7, 2012 at 8:13 PM, Andrew Robinson
<andrew3@r3dsolutions.com> wrote:
> OK, and is this a main use case?  (I'm not saying it isn't I'm asking.)

I have no idea what is a "main" use case.

> There is a special keyword which signals the new type of comprehension;  A
> normal comprehension would say eg: '[ foo for i in xrange ]'; but when the
> 'for i in' is reduced to a specific keyword such as 'ini' (instead of
> problematic 'in') the caching form of list comprehension would start.

FYI, the Python devs are not very fond of adding new keywords.  Any
time a new keyword is added, existing code that uses that word as a
name is broken.  'ini' is particularly bad, because 1) it's not a
word, and 2) it's the name of a common type of configuration file and
is probably frequently used as a variable name in relation to such
files.

> So, then, just like a comprehension -- the interpreter will begin to
> evaluate the code from the opening bracket '['; But anything other than a
> function/method will raise a type error (people might want to change that,
> but it's safe).
>
> The interpreter then caches all functions/initialiser methods it comes into
> contact with.
> Since every function/method has a parameter list (even if empty);  The
> interpreter would evaluate the parameter list on the first pass through the
> comprehension, and cache each parameter list with it's respective function.
>
> When the 'ini' keyword is parsed a second time, Python would then evaluate
> each cached function on its cached parameter list; and the result would be
> stored in the created list.
> This cached execution would be repeated as many times as is needed.
>
> Now, for your example:
>
> values = zip(samples, times * num_groups)
>     if len(values) < len(times) * num_groups:
>         # raise an error
>
> Might be done with:
>
> values = zip(   samples, [ lambda:times, ini xrange(num_groups) ]   )
>
>     if len(values) < len(times) * num_groups
>
> The comma after the lambda is questionable, and this construction would be
> slower since lambda automatically invokes the interpreter; but it's correct.

How is this any better than the ordinary list comprehension I already
suggested as a replacement?  For that matter, how is this any better
than list multiplication?  Your basic complaint about list
multiplication as I understand it is that the non-copying semantics
are unintuitive.  Well, the above is even less intuitive.  It is
excessively complicated and almost completely opaque.  If I were to
come across it outside the context of this thread, I would have no
idea what it is meant to be doing.

> As an aside, how would you do the lambda inside a list comprehension?

As a general rule, I wouldn't.  I would use map instead.

> [lambda:6 for i in xrange(10) ] # Nope.

Thak constructs a list of 10 functions and never calls them.  If you
want to actually call the lambda, then:

[(lambda: 6)() for i in range(10)]

or:

map(lambda i: 6, range(10))

But note that the former creates equivalent 10 functions and calls
each of them once, whereas the latter creates one function and calls
it ten times.

>> Of course you got an integer. You took an index of the range object, not a
>> slice. The rule is that taking an index of a sequence returns an element;
>> taking a slice of a sequence returns a sub-sequence. You still have not
>> shown any inconsistency here.
>
>
> Because it's an arbitrary rule which operates differently than the
> traditional idea shown in python docs?
>
> slice.indices()  is *for* (QUOTE)"representing the set of indices specified
> by range(start, stop, step)"
> http://docs.python.org/2/library/functions.html#slice

slice.indices() has nothing to do with it.  Indexing a sequence and
calling the .indices() method on a slice are entirely different
operations.  The slice.indices method is a utility method meant to be
called by __getitem__ implementations when doing slicing, not an
implementation of indexing.  When a sequence is indexed, there is no
slice.  That method is not related in any way to the semantics of
indexing a sequence.

[toc] | [prev] | [next] | [standalone]

#32957

On Thu, Nov 8, 2012 at 1:26 AM, Andrew Robinson
<andrew3@r3dsolutions.com> wrote:
> OK: Then copy by reference using map....:
>
> values = zip(   map( lambda:times, xrange(num_groups) )   )
> if len(values) < len(times) * num_groups ...
>
> Done.  It's clearer than a list comprehension and you still really don't
> need a list multiply.

That is not equivalent to the original.  Even had you not omitted some parts:

values = zip(samples, map(lambda i: times, range(num_groups)))

This still has the problem that map returns a list of num_groups
elements, each of which is times.  The desired value to be passed into
zip is a *single* sequence containing len(times) * num_groups
elements.  This is easily handled by list multiplication, but not so
easily by map or by a single list comprehension.  Looking back at the
'ini' solution you proposed before, I see that this also would be a
problem there.  Fixing the above, it would have to be something like:

values = zip(samples, reduce(operator.add, map(lambda i: times,
range(num_groups)), []))

Or from how I understand the 'ini' syntax to work:

values = zip(samples, reduce(operator.add, [lambda: times, ini
xrange(num_groups)], []))

Which brings to mind another point that I want to get to in a moment.
But when I said that I would use map instead, I meant that *if* the
body of the list comprehension is just a function application, then I
would prefer to use map over the list comprehension.  But in the above
I see no benefit in using a lambda in the first place.

Getting back to that other point, notice what we ended up doing in
both of those constructions above: repeated list concatenation as a
substitute for multiplication.  In fact, when we multiply (aList * 5),
this should be the equivalent of (aList + aList + aList + aList
+aList), should it not?  Clearly, however, there should be no implicit
copying involved in mere list concatenation.  For one thing, if the
user wants to concatenate copies, that is quite easily done
explicitly: (aList[:] + aList[:]) instead of (aList + aList).  For
another, list concatenation is less likely to be used for an
initialization process.  If list multiplication were to copy nested
lists, then, this would break the intuitive notion that list
multiplication is equivalent to repeated list concatenation.

> Yes, but you're very blind to history and code examples implementing the
> slice operation.
> slice usually depends on index; index does not depend on slice.
> Slice is suggested to be implemented by multiple calls to single indexes in
> traditional usage and documentation.

...and then by composing the elements located at those indexes into a
subsequence.

> The xrange(,,)[:] implementation breaks the tradition, because it doesn't
> call index multiple times; nor does it return a result equivalent identical
> to doing that.

Whether any given __getitem__ slicing implementation recursively calls
__getitem__ with a series of indexes or not is an implementation
detail.  If it were possible to index a range object multiple times
and then stuff the results into another range object, then the slicing
result would be equivalent.  The only reason it is not is that you
cannot construct a range object in that fashion.

I think that what you're expecting is that range(5)[:] should return a
list in Python 3 because it returns a list in Python 2.  This does not
represent a change in slicing behavior -- in fact, all you got by
slicing an xrange object in Python 2 was a TypeError.  This represents
an intentional break in backward compatibility between Python 2 and
Python 3, which was the purpose of Python 3 -- to fix a lot of
existing warts in Python by breaking them all at once, rather than
progressively over a long string of versions.  Users porting their
scripts from Python 2 to Python 3 are advised to replace "range(...)"
with "list(range(...))" if what they actually want is a list, and I
believe the 2to3 tool does this automatically.  Once the range object
is converted to a list, there is no further break with Python 2 --
slicing a list gives you a list, just as it always has.

In a nutshell, yes: range(...)[:] produces a different result in
Python 3 than in Python 2, just as it does without the slicing
operation tacked on.  It was never intended that scripts written for
Python 2 should be able to run in Python 3 unchanged without careful
attention to detail.

[toc] | [prev] | [next] | [standalone]

#32991

On 07/11/2012 01:55, Steven D'Aprano wrote:
>
> Who knows? Who cares? Nobody does:
>
> n -= n
>

But I've seen this scattered through code:

x := x - x - x

-- 
Cheers.

Mark Lawrence.

[toc] | [prev] | [next] | [standalone]

#32995

On Fri, Nov 9, 2012 at 12:39 PM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
> On 07/11/2012 01:55, Steven D'Aprano wrote:
>>
>>
>> Who knows? Who cares? Nobody does:
>>
>> n -= n
>>
>
> But I've seen this scattered through code:
>
> x := x - x - x

Can you enlighten us as to how this is better than either:
 x := -x
or
 x := 0 - x
? I'm not seeing it. And I'm not seeing any nonnumeric that would
benefit from being subtracted from itself twice (strings, arrays,
sets, you can subtract them from one another but not usefully more
than once).

ChrisA

[toc] | [prev] | [next] | [standalone]

#32998

On Fri, 09 Nov 2012 17:07:09 +1100, Chris Angelico wrote:

> On Fri, Nov 9, 2012 at 12:39 PM, Mark Lawrence <breamoreboy@yahoo.co.uk>
> wrote:
>> On 07/11/2012 01:55, Steven D'Aprano wrote:
>>>
>>>
>>> Who knows? Who cares? Nobody does:
>>>
>>> n -= n
>>>
>>>
>> But I've seen this scattered through code:
>>
>> x := x - x - x
> 
> Can you enlighten us as to how this is better than either:
>  x := -x
> or
>  x := 0 - x
> ? I'm not seeing it. 

I'm hoping that Mark intended it as an example of crappy code he has 
spotted in some other language rather than a counter-example of something 
you would do.

To be pedantic... there may very well be some (rare) cases where you 
actually do want x -= x rather than just x = 0. Consider the case where x 
could be an INF or NAN. Then x -= x should give x = NAN rather than zero. 
That may be desirable in some cases.

At the very least, the compiler should NOT optimize away x = x - x to 
x = 0 if x could be a float, complex or Decimal.


> And I'm not seeing any nonnumeric that would
> benefit from being subtracted from itself twice (strings, arrays, sets,
> you can subtract them from one another but not usefully more than once).

How do you subtract strings?



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#32999

On Fri, Nov 9, 2012 at 5:37 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Fri, 09 Nov 2012 17:07:09 +1100, Chris Angelico wrote:
>> Can you enlighten us as to how this is better than either:
>>  x := -x
>> or
>>  x := 0 - x
>> ? I'm not seeing it.
>
> I'm hoping that Mark intended it as an example of crappy code he has
> spotted in some other language rather than a counter-example of something
> you would do.

Ohh. Yeah, that figures. Huh.

> To be pedantic... there may very well be some (rare) cases where you
> actually do want x -= x rather than just x = 0. Consider the case where x
> could be an INF or NAN. Then x -= x should give x = NAN rather than zero.
> That may be desirable in some cases.
>
> At the very least, the compiler should NOT optimize away x = x - x to
> x = 0 if x could be a float, complex or Decimal.

Yep. In the specific case of integers, though, and in the specific
instance of CPU registers in assembly language, it's reasonable to
optimize it the *other* way - MOV reg,0 is a one-byte opcode and 1, 2,
or 4 bytes of immediate data, while SUB reg,reg (or XOR reg,reg) is a
two-byte operation regardless of data size. But that's
microoptimization that makes, uhh, itself-subtracted-from-itself sense
in Python.

>> And I'm not seeing any nonnumeric that would
>> benefit from being subtracted from itself twice (strings, arrays, sets,
>> you can subtract them from one another but not usefully more than once).
>
> How do you subtract strings?

The same way you subtract sets. Same with arrays. Python doesn't do
either, but Python also doesn't do the ":=" operator that the example
code demonstrated, so I didn't assume Python.

Pike v7.8 release 700 running Hilfe v3.5 (Incremental Pike Frontend)
> "Hello, world!"-"l";
(1) Result: "Heo, word!"
> ({1,2,3,3,2,3,1,2,1})-({2});
(2) Result: ({ /* 6 elements */
                1,
                3,
                3,
                3,
                1,
                1
            })

Python spells it differently:
>>> "Hello, world!".replace("l","")
'Heo, word!'

Not sure how to do array subtraction other than with filter:
>>> list(filter(lambda x: x!=2,[1,2,3,3,2,3,1,2,1]))
[1, 3, 3, 3, 1, 1]
But there's probably a way (list.remove only takes out the first
occurrence, so it's not equivalent).

In any case, subtracting something from _itself_ is only going to give
you an empty string, array, set, or whatever, and doing so a second
time is going to achieve nothing. Hence my comment.

But poor code we will always have with us, to paraphrase the Gospel of Matthew.

ChrisA

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web