Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #32329 > unrolled thread

Negative array indicies and slice()

Back to article view | Back to comp.lang.python

Contents

  Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-28 20:12 -0700
    Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-28 21:42 -0600
      Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-28 21:00 -0700
        Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-28 22:25 -0600
          Re: Negative array indicies and slice() Andrew <andrewr3mail@gmail.com> - 2012-10-29 00:54 -0700
            Re: Negative array indicies and slice() Chris Rebert <clp2@rebertia.com> - 2012-10-29 01:18 -0700
            Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-29 11:19 +0000
              Re: Negative array indicies and slice() Chris Angelico <rosuav@gmail.com> - 2012-10-29 22:32 +1100
              Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-28 21:52 -0700
              Re: Negative array indicies and slice() Chris Angelico <rosuav@gmail.com> - 2012-10-29 23:40 +1100
                Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-29 22:02 +0000
              Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-28 23:01 -0700
                Re: Negative array indicies and slice() Roy Smith <roy@panix.com> - 2012-10-29 09:52 -0400
                  Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 08:20 -0700
                  Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 17:01 -0600
                  Re: Negative array indicies and slice() Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-10-30 00:04 +0000
                  Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 16:54 -0700
                  Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 02:15 -0600
              Re: Negative array indicies and slice() Chris Angelico <rosuav@gmail.com> - 2012-10-30 00:53 +1100
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 11:09 -0600
              Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-29 22:14 +0000
              Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 08:42 -0700
                Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-30 00:02 +0000
                  Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 12:34 -0700
                    Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-30 08:17 +0000
                      Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-30 08:47 -0700
                      Re: Negative array indicies and slice() Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-10-30 23:48 +0000
                      Re: Negative array indicies and slice() Michael Torrie <torriem@gmail.com> - 2012-10-30 23:29 -0600
                  Re: Negative array indicies and slice() Michael Torrie <torriem@gmail.com> - 2012-10-29 23:53 -0600
                  Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 17:04 -0700
              Re: Negative array indicies and slice() Chris Angelico <rosuav@gmail.com> - 2012-10-30 09:55 +1100
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 17:07 -0600
                Re: Negative array indicies and slice() Roy Smith <roy@panix.com> - 2012-10-29 19:24 -0400
                  Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 17:43 -0600
                    Re: Negative array indicies and slice() Roy Smith <roy@panix.com> - 2012-10-29 20:17 -0400
                  Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 18:05 -0600
              Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 11:00 -0700
              Re: Negative array indicies and slice() Chris Kaynor <ckaynor@zindagigames.com> - 2012-10-29 18:49 -0700
              Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 15:39 -0700
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 23:55 -0600
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 00:51 -0600
              Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 17:17 -0700
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 01:21 -0600
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 01:32 -0600
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 02:46 -0600
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 12:02 -0600
              Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-30 07:21 -0700
              Re: Negative array indicies and slice() Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-10-30 21:33 +0000
                Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-31 10:07 +0000
                  Re: Negative array indicies and slice() Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-10-31 16:01 +0000
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 15:47 -0600
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 15:55 -0600
              Re: Negative array indicies and slice() Chris Angelico <rosuav@gmail.com> - 2012-10-31 09:00 +1100
              Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 16:02 -0600
              Re: Negative array indicies and slice() Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-10-30 23:30 +0000
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 11:27 -0600
            Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 16:33 -0700
          Re: Negative array indicies and slice() Andrew <andrewr3mail@gmail.com> - 2012-10-29 00:54 -0700
      Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-28 21:00 -0700
      Re: Negative array indicies and slice() Andrew <andrewr3mail@gmail.com> - 2012-10-28 21:09 -0700
        Re: Negative array indicies and slice() alex23 <wuwei23@gmail.com> - 2012-10-28 21:44 -0700
          Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-29 01:24 -0700
            Re: Negative array indicies and slice() Chris Rebert <clp2@rebertia.com> - 2012-10-29 01:37 -0700
              Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-29 01:59 -0700
                Re: Negative array indicies and slice() Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-10-29 09:36 +0000
                Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-29 10:34 +0000
              Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-29 01:59 -0700
        Re: Negative array indicies and slice() Paul Rubin <no.email@nospam.invalid> - 2012-10-28 22:14 -0700
          Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-29 01:08 -0700
            Re: Negative array indicies and slice() Chris Rebert <clp2@rebertia.com> - 2012-10-29 01:26 -0700
      Re: Negative array indicies and slice() Andrew <andrewr3mail@gmail.com> - 2012-10-28 21:09 -0700
    Re: Negative array indicies and slice() MRAB <python@mrabarnett.plus.com> - 2012-10-29 03:45 +0000
    Re: Negative array indicies and slice() 88888 Dihedral <dihedral88888@googlemail.com> - 2012-11-01 18:08 -0700

Page 1 of 4  [1] 2 3 4  Next page →

#32329 — Negative array indicies and slice()

The slice operator does not give any way (I can find!) to take slices from negative to positive indexes, although the range is not empty, nor the expected indexes out of range that I am supplying.

Many programs that I write would require introducing variables and logical statements to correct the problem which is very lengthy and error prone unless there is a simple work around.

I *hate* replicating code every time I need to do this!

I also don't understand why slice() is not equivalent to an iterator, but can replace an integer in __getitem__() whereas xrange() can't.


Here's an example for Linux shell, otherwise remove /bin/env...
{{{#!/bin/env python
a=[1,2,3,4,5,6,7,8,9,10]
print a[-4:3]  # I am interested in getting [7,8,9,10,1,2] but I get [].
}}}

[toc] | [next] | [standalone]

#32331

On Sun, Oct 28, 2012 at 9:12 PM,  <andrewr3mail@gmail.com> wrote:
> The slice operator does not give any way (I can find!) to take slices from negative to positive indexes, although the range is not empty, nor the expected indexes out of range that I am supplying.
>
> Many programs that I write would require introducing variables and logical statements to correct the problem which is very lengthy and error prone unless there is a simple work around.
>
> I *hate* replicating code every time I need to do this!
>
> I also don't understand why slice() is not equivalent to an iterator, but can replace an integer in __getitem__() whereas xrange() can't.
>
>
> Here's an example for Linux shell, otherwise remove /bin/env...
> {{{#!/bin/env python
> a=[1,2,3,4,5,6,7,8,9,10]
> print a[-4:3]  # I am interested in getting [7,8,9,10,1,2] but I get [].
> }}}


For a sequence of length 10, "a[-4:3]" is equivalent to "a[6:3]",
which is an empty slice since index 6 is after index 3.

If you want it to wrap around, then take two slices and concatenate
them with "a[-4:] + a[:3]".

[toc] | [prev] | [next] | [standalone]

#32333

On Sunday, October 28, 2012 8:43:30 PM UTC-7, Ian wrote:
> On Sun, Oct 28, 2012 at 9:12 PM, andrew wrote:
> 
> > The slice operator does not give any way (I can find!) to take slices from negative to positive indexes, although the range is not empty, nor the expected indexes out of range that I am supplying.
> 
> >
> 
> > Many programs that I write would require introducing variables and logical statements to correct the problem which is very lengthy and error prone unless there is a simple work around.
> 
> >
> 
> > I *hate* replicating code every time I need to do this!
> 
> >
> 
> > I also don't understand why slice() is not equivalent to an iterator, but can replace an integer in __getitem__() whereas xrange() can't.
> 
> >
> 
> >
> 
> > Here's an example for Linux shell, otherwise remove /bin/env...
> 
> > {{{#!/bin/env python
> 
> > a=[1,2,3,4,5,6,7,8,9,10]
> 
> > print a[-4:3]  # I am interested in getting [7,8,9,10,1,2] but I get [].
> 
> > }}}
> 
> 
> 
> 
> 
> For a sequence of length 10, "a[-4:3]" is equivalent to "a[6:3]",
> 
> which is an empty slice since index 6 is after index 3.
> 
> 
> 
> If you want it to wrap around, then take two slices and concatenate
> 
> them with "a[-4:] + a[:3]".

Hi Ian,
Well, no it really isn't equivalent.
Consider a programmer who writes:
xrange(-4,3) *wants* [-4,-3,-2,-1,0,1,2]

That is the "idea" of a range; for what reason would anyone *EVER* want -4 to +3 to be 6:3???

I do agree that the data held in -4 is equivalent to the data in 6, but the index is not the same.

So: Why does python choose to convert them to positive indexes, and have slice operate differently than xrange -- for the slice() object can't possibly know the size of the array when it is passed in to __getitem__;  They are totally separate classes.

I realize I can concat. two slice ranges, BUT, the ranges do not always span from negative to positive.

eg: a line in my program reads:
a[x-5:x]

if x is 7, then this is a positive index to a positive index.
So, there is no logic to using two slices concatd !

I use this arbitrary range code *often* so I need a general purpose solution.
I looked up slice() but the help is of no use, I don't even know how I might overload it to embed some logic to concatenate ranges of data; nor even if it is possible.

[toc] | [prev] | [next] | [standalone]

#32337

On Sun, Oct 28, 2012 at 10:00 PM,  <andrewr3mail@gmail.com> wrote:
> Hi Ian,
> Well, no it really isn't equivalent.
> Consider a programmer who writes:
> xrange(-4,3) *wants* [-4,-3,-2,-1,0,1,2]
>
> That is the "idea" of a range; for what reason would anyone *EVER* want -4 to +3 to be 6:3???

That is what ranges do, but your question was about slices, not ranges.

> So: Why does python choose to convert them to positive indexes, and have slice operate differently than xrange -- for the slice() object can't possibly know the size of the array when it is passed in to __getitem__;  They are totally separate classes.

Ranges can contain negative integers.  However, sequences do not have
negative indices.  Therefore, negative indices in slices are used to
count from the end instead of from the start.  As stated in the
language docs, "If either bound is negative, the sequence’s length is
added to it."  Therefore, "a[-4:3]" does not wrap around the end of
the sequence because "a[6:3]" does not wrap around the end of the
sequence.

> I realize I can concat. two slice ranges, BUT, the ranges do not always span from negative to positive.

def wrapping_slice(seq, start, stop):
    start, stop, _ = slice(start, stop).indices(len(seq))
    if start <= stop:
        return seq[start:stop]
    else:
        return seq[start:] + seq[:stop]

You'll have to decide for yourself whether you want it to return an
empty list or the entire list if start == stop.

[toc] | [prev] | [next] | [standalone]

#32345

On Sunday, October 28, 2012 9:26:01 PM UTC-7, Ian wrote:
> On Sun, Oct 28, 2012 at 10:00 PM,  Andrew wrote:
> 
> > Hi Ian,
> 
> > Well, no it really isn't equivalent.
> 
> > Consider a programmer who writes:
> 
> > xrange(-4,3) *wants* [-4,-3,-2,-1,0,1,2]
> 
> >
> 
> > That is the "idea" of a range; for what reason would anyone *EVER* want -4 to +3 to be 6:3???
> 
> 
> 
> That is what ranges do, but your question was about slices, not ranges.

Actually, I said in the OP:

"I also don't understand why slice() is not equivalent to an iterator, but can replace an integer in __getitem__() whereas xrange() can't."

=========================

Thank you for the code snippet; I don't think it likely that existing programs depend on nor use a negative index and a positive index expecting to take a small chunk in the center... hence, I would return the whole array; Or if someone said [-len(listX) : len(listX)+1 ] I would return the whole array twice.
That's the maximum that is possible.
If someone could show me a normal/reasonable script which *would* expect the other behavior, I'd like to know; compatibility is important.

=========================

My intended inferences about the iterator vs. slice question was perhaps not obvious to you; Notice: an iterator is not *allowed* in __getitem__().

The slice class when passed to __getitem__()  was created to merely pass two numbers and a stride to __getitem__;  As far as I know slice() itself does *nothing* in the actual processing of the elements.  So, it's *redundant* functionality, and far worse, it's restrictive.

The philosophy of Python is to have exactly one way to do something when possible; so, why create a stand alone class that does nothing an existing class could already do, and do it better ?

A simple list of three values would be just as efficient as slice()!
xrange is more flexible, and can be just as efficient.

So, Have I misunderstood the operation of slice()?  I think I might have... but I don't know.

In 'C', where Python is written, circularly linked lists -- and arrays are both very efficient ways of accessing data.  Arrays can, in fact, have negative indexes -- perhaps contrary to what you thought.  One merely defines a variable to act as the base pointer to the array and initialize it to the *end* of the array. Nor is the size of the data elements an issue, since in Python all classes are accessed by pointers which are of uniform size. I routinely do this in C.

Consider, also, that xrange() does not actually create a list -- but merely an iterator generating integers which is exactly what __getitem__ works on.
So, xrange() does not need to incur a memory or noticeable time penalty.

From micro-python, it's clear that their implementation of xrange() is at the 'C' level; which is extremely fast.

[toc] | [prev] | [next] | [standalone]

#32348

On Mon, Oct 29, 2012 at 12:54 AM, Andrew <andrewr3mail@gmail.com> wrote:
> On Sunday, October 28, 2012 9:26:01 PM UTC-7, Ian wrote:
>> On Sun, Oct 28, 2012 at 10:00 PM,  Andrew wrote:
<snip>
> The slice class when passed to __getitem__()  was created to merely pass two numbers and a stride to __getitem__;  As far as I know slice() itself does *nothing* in the actual processing of the elements.  So, it's *redundant* functionality, and far worse, it's restrictive.
>
> The philosophy of Python is to have exactly one way to do something when possible; so, why create a stand alone class that does nothing an existing class could already do, and do it better ?
>
> A simple list of three values would be just as efficient as slice()!
> xrange is more flexible, and can be just as efficient.
>
> So, Have I misunderstood the operation of slice()?  I think I might have... but I don't know.

`slice` is intentionally lenient about the types of the start, stop, and step:
>>> class Foo:
...     def __getitem__(self, slice_):
...         print(slice_)
...         return 42
...
>>> Foo()["a":"b":"c"]
slice('a', 'b', 'c')
42
>>>
Thus, the thing being sliced is free to interpret the parts of the
slice however it wishes; hence, slice() is unable to contain the
"processing" you speak of.
By contrast, xrange() limits itself to integers.
To support the more general case, the slice syntax thus produces a
`slice` rather than an `xrange`.
Doubtlessly, there are also historical issues involved. As implied by
the ugliness of its name, `xrange` was added to the language
relatively later.

Cheers,
Chris

[toc] | [prev] | [next] | [standalone]

#32359

On Mon, 29 Oct 2012 00:54:29 -0700, Andrew wrote:

> Actually, I said in the OP:
> 
> "I also don't understand why slice() is not equivalent to an iterator,
> but can replace an integer in __getitem__() whereas xrange() can't."

Slices and iterators have different purposes and therefore have not been 
made interchangeable. Yes, there are certain similarities between a slice 
and xrange, but there are also significant differences.


> Thank you for the code snippet; I don't think it likely that existing
> programs depend on nor use a negative index and a positive index
> expecting to take a small chunk in the center... 

On the contrary. That is the most straightforward and useful idea of 
slicing, to grab a contiguous slice of items.

Why would you want to grab a slice from the end of the list, and a slice 
from the start of the list, and swap them around? Apart from simulating 
card shuffles and cuts, who does that?


> hence, I would return
> the whole array; Or if someone said [-len(listX) : len(listX)+1 ] I
> would return the whole array twice.

Well, that's one possible interpretation, but it is not the one that 
Python uses. When you create your own language, you can choose whatever 
interpretation seems most useful to you too.



> That's the maximum that is possible.
> If someone could show me a normal/reasonable script which *would* expect
> the other behavior, I'd like to know; compatibility is important.

I'm not entirely sure I understand what you are asking here. 


> My intended inferences about the iterator vs. slice question was perhaps
> not obvious to you; Notice: an iterator is not *allowed* in
> __getitem__().

Actually, you can write __getitem__ for your own classes to accept 
anything you like.

py> class Test:
...     def __getitem__(self, index):
...             return index
...
py> t = Test()
py> t["Hello world"]
'Hello world'
py> t[{'x': None}]
{'x': None}


> The slice class when passed to __getitem__()  was created to merely pass
> two numbers and a stride to __getitem__;  As far as I know slice()
> itself does *nothing* in the actual processing of the elements.  So,
> it's *redundant* functionality, and far worse, it's restrictive.

You say that as if it were a bad thing.


> The philosophy of Python is to have exactly one way to do something when
> possible; so, why create a stand alone class that does nothing an
> existing class could already do, and do it better ?

What existing class is that? It certainly isn't xrange.

Because xrange represents a concrete sequence of numbers, all three of 
start, end and stride must be concrete, known, integers:

py> xrange(4, None, 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: an integer is required

Whereas slices can trivially include blanks that get filled in only when 
actually used:

py> "hello world"[aslice]
'owrd'
py> "NOBODY expects the Spanish Inquisition!"[aslice]
'D xet h pns nusto!'


So, no, xrange is no substitute for slices. Not even close.


> A simple list of three values would be just as efficient as slice()!

On the contrary, a simple list of three values not only could not do 
everything a slice does, but it's over twice the size!

py> sys.getsizeof([1, 2, 3])
44
py> sys.getsizeof(slice(1, 2, 3))
20


> xrange is more flexible, and can be just as efficient.

Less flexible, less efficient.


[snip]
> In 'C', where Python is written, 

That's a popular misapprehension. Python is written in Java, or Lisp, or 
Haskell, or CLR (dot Net), or RPython, or Ocaml, or Parrot. Each of those 
languages have, or had, at least one Python implementation. Oh, there's 
also a version written in C, or so I have heard.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#32360

On Mon, Oct 29, 2012 at 10:19 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
>> In 'C', where Python is written,
>
> That's a popular misapprehension. Python is written in Java, or Lisp, or
> Haskell, or CLR (dot Net), or RPython, or Ocaml, or Parrot. Each of those
> languages have, or had, at least one Python implementation. Oh, there's
> also a version written in C, or so I have heard.

And that's not including the human-brain implementation, perhaps the
most important of all. Although the current port of Python to my brain
isn't quite a complete implementation, lacking a few bits that I
should probably get to at some point, but even so, it's as useful to
me as firing up IDLE.

I wonder if what the OP is looking for is not slicing, but something
more akin to map. Start with a large object and an iterator that
produces keys, and create an iterator/list of their corresponding
values. Something like:

a=[1,2,3,4,5,6,7,8,9,10]
b=[a[i] for i in xrange(-4,3)]

It's not strictly a slice operation, but it's a similar sort of thing,
and it can do the wraparound quite happily.

ChrisA

[toc] | [prev] | [next] | [standalone]

#32363

On 10/29/2012 04:32 AM, Chris Angelico wrote:
> I wonder if what the OP is looking for is not slicing, but something 
> more akin to map. Start with a large object and an iterator that 
> produces keys, and create an iterator/list of their corresponding 
> values. Something like: a=[1,2,3,4,5,6,7,8,9,10] b=[a[i] for i in 
> xrange(-4,3)] It's not strictly a slice operation, but it's a similar 
> sort of thing, and it can do the wraparound quite happily. ChrisA 

A list comprehension ?
That does do what I am interested in, *very* much so.  Quite a gem, Chris!

:-\
I am curious as to how quickly it constructs the result compared to a 
slice operation.

Eg:
a[1:5]
vs.
[ a[i] for i in xrange[1:5] ]

But, unless it were grossly slower -- so that if/then logic and slices 
were generally faster -- I will use it.
Thanks.

--Andrew.

[toc] | [prev] | [next] | [standalone]

#32366

On Mon, Oct 29, 2012 at 3:52 PM, Andrew Robinson
<andrew3@r3dsolutions.com> wrote:
> I am curious as to how quickly it constructs the result compared to a slice
> operation.
>
> Eg:
> a[1:5]
> vs.
> [ a[i] for i in xrange[1:5] ]

For the most part, don't concern yourself with performance. Go with
functionality and readability. In the trivial case shown here, the
slice is WAY clearer, so it should definitely be the one used; in
other cases, the slice might simply be insufficient, so you go with
whatever achieves your goal. Performance will usually be "good
enough", even if there's a marginally faster way.

ChrisA

[toc] | [prev] | [next] | [standalone]

#32425

On Mon, 29 Oct 2012 23:40:53 +1100, Chris Angelico wrote:

> On Mon, Oct 29, 2012 at 3:52 PM, Andrew Robinson
> <andrew3@r3dsolutions.com> wrote:
>> I am curious as to how quickly it constructs the result compared to a
>> slice operation.
>>
>> Eg:
>> a[1:5]
>> vs.
>> [ a[i] for i in xrange[1:5] ]
> 
> For the most part, don't concern yourself with performance. Go with
> functionality and readability. In the trivial case shown here, the slice
> is WAY clearer, so it should definitely be the one used; in other cases,
> the slice might simply be insufficient, so you go with whatever achieves
> your goal. Performance will usually be "good enough", even if there's a
> marginally faster way.


Slicing is about an order of magnitude faster:


[steve@ando ~]$ python2.7 -m timeit -s "x = range(100, 1000, 2)" "x
[20:40]"
1000000 loops, best of 3: 0.342 usec per loop
[steve@ando ~]$ python2.7 -m timeit -s "x = range(100, 1000, 2)" "[x[i] 
for i in xrange(20, 40)]"
100000 loops, best of 3: 3.43 usec per loop


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#32370

On 10/29/2012 04:19 AM, Steven D'Aprano wrote:
> On Mon, 29 Oct 2012 00:54:29 -0700, Andrew wrote:
>
> Slices and iterators have different purposes and therefore have not been
> made interchangeable. Yes, there are certain similarities between a slice
> and xrange, but there are also significant differences.
Aha, now were getting to the actual subject.

> [snip]
>> In 'C', where Python is written,
> That's a popular misapprehension. Python is written in Java, or Lisp, or
> Haskell, or CLR (dot Net), or RPython, or Ocaml, or Parrot. Each of those
> languages have, or had, at least one Python implementation. Oh, there's
> also a version written in C, or so I have heard.
:-P
I didn't say it was only written in "C",  but in "C" where it is 
implemented.
I will be porting Python 3.xx to a super low power embedded processor 
(MSP430), both space and speed are at a premium.
Running Python on top of Java would be a *SERIOUS* mistake.  .NET won't 
even run on this system. etc.

>
>
>> Thank you for the code snippet; I don't think it likely that existing
>> programs depend on nor use a negative index and a positive index
>> expecting to take a small chunk in the center...
> On the contrary. That is the most straightforward and useful idea of
> slicing, to grab a contiguous slice of items.
Show me an example where someone would write a slice with a negative and 
a positive index (both in the same slice);
and have that slice grab a contiguous slice in the *middle* of the list 
with orientation of lower index to greater index.
I have asked before;  It's not that I don't think it possible -- it's 
that I can't imagine a common situation.

> Why would you want to grab a slice from the end of the list, and a slice
> from the start of the list, and swap them around? Apart from simulating
> card shuffles and cuts, who does that?
Advanced statistics programmers using lookup tables that are 
symmetrical.  Try Physicists too -- but they're notably weird.

>> My intended inferences about the iterator vs. slice question was perhaps
>> not obvious to you; Notice: an iterator is not *allowed* in
>> __getitem__().
> Actually, you can write __getitem__ for your own classes to accept
> anything you like.
Yes, I realize that.
But, why can't I just overload the existing __getitem__ for lists and 
not bother writing an entire class?
Everything in Python is supposed to be an object and one of the big 
supposed "selling" points is the ability to overload "any" object's methods.
The lists aren't special -- they're just a bunch of constant decimal 
numbers, typically given as a large tuple.

>
> py>  class Test:
> ...     def __getitem__(self, index):
> ...             return index
> ...
Better:
 >>> class Test:
...     def __getitem__( self, *index ):
...             return index

No extra curlies required...

> You say that as if it were a bad thing.

hmmm... and you as if sarcastic? :-)
It is a bad thing to have any strictly un-necessary and non-code saving 
objects where memory is restricted.

> What existing class is that? It certainly isn't xrange.
>
> Because xrange represents a concrete sequence of numbers, all three of
> start, end and stride must be concrete, known, integers:
>

Let's up the ante.  I'll admit xrange() won't do "later" fill in the 
blank -- BUT --
xrange() is a subclass of an *existing* class called iterator.
Iterators are very general.  They can even be made random.

>> The philosophy of Python is to have exactly one way to do something when
>> possible; so, why create a stand alone class that does nothing an
>> existing class could already do, and do it better ?
> py>  xrange(4, None, 2)
> Traceback (most recent call last):
>    File "<stdin>", line 1, in<module>
> TypeError: an integer is required

Hmmm..
Let's try your example exactly as shown...

"hello world"[aslice]
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
NameError: name 'aslice' is not defined

WOW. Cool.
Where did the blanks *actually* get filled in?  Or HOW WILL they in your 
next post?
>
> On the contrary, a simple list of three values not only could not do 
> everything a slice does, but it's over twice the size! 
Yeah, There is a definite issue there.  But the case isn't decided by 
that number alone.
A slice is storing three integers -- and an integer is size is 12.
So, slices don't use integers.  If the type that *IS* used happens to be 
a real Python type, we may merely typecast integers to that type -- 
insert them in a tuple and by definition, they must be the same size.

  Looking at some of the online programming notes -- a slice apparently 
doesn't use an integer storage variable that is capable of arbitrary 
expansion. =-O -- and hence, won't work for very large sized lists.  
That actually explains some crashes I have noted in the past when 
working with 20 million element lists that I wanted a slice of.  I had 
*plenty* of ram on that system.
Besides: The program code to implement slice() is undoubtedly larger 
than 12 bytes of savings!
How many slices() are typically found in memory simultaneously?

You're on the way to convincing me -- but you need to show off this fill 
in the blank issue.
That's a difference of kind -- and not of size.
An iterator, could be an object with methods...  So what common 
application uses this "fill" in the blank stuff?

[toc] | [prev] | [next] | [standalone]

#32376

In article <mailman.3009.1351516065.27098.python-list@python.org>,
 Andrew Robinson <andrew3@r3dsolutions.com> wrote:

> Show me an example where someone would write a slice with a negative and 
> a positive index (both in the same slice);
> and have that slice grab a contiguous slice in the *middle* of the list 
> with orientation of lower index to greater index.

It's possible in bioinformatics.  Many organisms have circular 
chromosomes.  It's a single DNA molecule spliced into a loop.  There's 
an "origin", but it's more a convenience thing for people to assign some 
particular base-pair to be location 0.  From the organism's point of 
view, the origin isn't anything special (but there *is* a fixed 
orientation).

It's entirely plausible for somebody to want to extract the sub-sequence 
from 100 bp (base-pairs) before the origin to 100 bp after the origin.  
If you were storing the sequence in Python string (or list), the most 
convenient way to express this would be seq[-100:100].  Likewise, if you 
wanted the *other* fragment, you would write seq[100:-100].

There is a minor confounding factor here in that biologists number 
sequences starting with 1, not 0.  At least that was the way when I was 
doing this stuff mumble years ago.  I don't know what the current 
convention is.

[toc] | [prev] | [next] | [standalone]

#32428

On 10/29/2012 06:52 AM, Roy Smith wrote:
> Show me an example where someone would write a slice with a negative and
> a positive index (both in the same slice);
> and have that slice grab a contiguous slice in the *middle* of the list
> with orientation of lower index to greater index.
> It's possible in bioinformatics. ...
> eq[100:-100].
I decided to go to bed... I was starting to write very badly worded 
responses. :)

Thanks, Roy,  what you have just shown is another example that agrees 
with what I am trying to do.
FYI: I was asking for a reason why Python's present implementation is 
desirable...

I wonder, for example:

Given an arbitrary list:
a=[1,2,3,4,5,6,7,8,9,10,11,12]

Why would someone *want* to do:
a[-7,10]
Instead of saying
a[5:10] or a[-7:-2] ?

eg:
What algorithm would naturally *desire* the default behavior of slicing 
when using *mixed* negative and positive indexes?
In the case of a bacterial circular DNA/RNA ring, asking for codons[ 
-10: 10 ]  would logically desire codon[-10:] + codon[:10] not an empty 
list, right?

I think your example is a very reasonable thing the scientific community 
would want to do with Python.
:)

[toc] | [prev] | [next] | [standalone]

#32437

On Mon, Oct 29, 2012 at 9:20 AM, Andrew Robinson
<andrew3@r3dsolutions.com> wrote:
> FYI: I was asking for a reason why Python's present implementation is
> desirable...
>
> I wonder, for example:
>
> Given an arbitrary list:
> a=[1,2,3,4,5,6,7,8,9,10,11,12]
>
> Why would someone *want* to do:
> a[-7,10]
> Instead of saying
> a[5:10] or a[-7:-2] ?

A quick search of local code turns up examples like this:

if name.startswith('{') and name.endswith('}'):
    name = name[1:-1]

If slices worked like ranges, then the result of that would be empty,
which is obviously not desirable.

I don't know of a reason why one might need to use a negative start
with a positive stop, though.

[toc] | [prev] | [next] | [standalone]

#32445

On 29 October 2012 23:01, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> On Mon, Oct 29, 2012 at 9:20 AM, Andrew Robinson
> <andrew3@r3dsolutions.com> wrote:
>> FYI: I was asking for a reason why Python's present implementation is
>> desirable...
>>
>> I wonder, for example:
>>
>> Given an arbitrary list:
>> a=[1,2,3,4,5,6,7,8,9,10,11,12]
>>
>> Why would someone *want* to do:
>> a[-7,10]
>> Instead of saying
>> a[5:10] or a[-7:-2] ?
>
> A quick search of local code turns up examples like this:
>
> if name.startswith('{') and name.endswith('}'):
>     name = name[1:-1]
>
> If slices worked like ranges, then the result of that would be empty,
> which is obviously not desirable.
>
> I don't know of a reason why one might need to use a negative start
> with a positive stop, though.

It's useful when getting a reversed slice:

>>> a = [1,2,3,4,5,6,7,8,9,10]
>>> a[-3:3:-1]
[8, 7, 6, 5]


Oscar

[toc] | [prev] | [next] | [standalone]

#32470

On 10/29/2012 04:01 PM, Ian Kelly wrote:
> On Mon, Oct 29, 2012 at 9:20 AM, Andrew Robinson
> <andrew3@r3dsolutions.com>  wrote:
>> FYI: I was asking for a reason why Python's present implementation is
>> desirable...
>>
>> I wonder, for example:
>>
>> Given an arbitrary list:
>> a=[1,2,3,4,5,6,7,8,9,10,11,12]
>>
>> Why would someone *want* to do:
>> a[-7,10]
>> Instead of saying
>> a[5:10] or a[-7:-2] ?
> A quick search of local code turns up examples like this:
>
> if name.startswith('{') and name.endswith('}'):
>      name = name[1:-1]
Which is done to avoid explicitly calling the len() operator.
> If slices worked like ranges, then the result of that would be empty,
> which is obviously not desirable.
Yes, and that's an excellent point -- but note what I am showing in the 
example.  It is that example, which I am specifying.  There are only two 
cases where I think the default behavior of Python gives undesirable 
results:

The step is positive, and the pair of indexes goes from negative to 
positive.
Likewise, If the pair went from positive to negative, and the step was 
negative.

In all other combinations, the default behavior of python ought to 
remain intact.
I apologize for not making this crystal clear -- I thought you would 
focus on the specific example I gave.

> I don't know of a reason why one might need to use a negative start
> with a positive stop, though.
I've already given several examples; and another poster did too -- eg: 
Gene sequences for bacteria.  It's not uncommon to need this.  If I do 
some digging, I can also show some common graphics operations that 
benefit greatly from this ability -- NOTE: in another thread I just 
showed someone how to operate on RGBA values...  Slicing becomes THE 
major operation done when converting, or blitting, graphics data. etc.

Another example -- Jpeg, for example, uses discrete cosines -- which are 
a naturally cyclic data type.  They repeat with a fixed period.  I know 
there are "C" libraries already made for Jpeg -- but that doesn't mean 
many other applications with no "C" library aren't plagued by this problem.

I don't know how to make this point more clear.  There really *ARE* 
applications that uses cyclic lists of data; or which can avoid extra 
logic to fix problems encountered from linear arrays which *end* at a 
particular point.

sometimes it is desirable for a truncation to occur, sometimes it's 
NOT.  The sign convention test I outlined, I believe, clearly detects 
when a cyclic data set is desired. If there are normal examples where my 
tests fail -- that's what's important to me.

[toc] | [prev] | [next] | [standalone]

#32475

On Mon, Oct 29, 2012 at 5:54 PM, Andrew Robinson
<andrew3@r3dsolutions.com> wrote:
>> I don't know of a reason why one might need to use a negative start
>> with a positive stop, though.
>
> I've already given several examples; and another poster did too

I meant that I don't know of a reason to do that given the existing
semantics, which is what you were asking for.  :-)

I understand and agree that there are potential applications for
allowing slices to wrap around from negative to positive.  What I'm
not convinced of is that these applications need or should be handled
by the slicing operation -- which is more commonly understood as
producing subsequences -- especially since they already can be handled
relatively easily by iteration.  I think that more users would find
the behavior surprising than useful.

[toc] | [prev] | [next] | [standalone]

#32379

On Mon, Oct 29, 2012 at 5:01 PM, Andrew Robinson
<andrew3@r3dsolutions.com> wrote:
>  Looking at some of the online programming notes -- a slice apparently
> doesn't use an integer storage variable that is capable of arbitrary
> expansion. =-O -- and hence, won't work for very large sized lists.  That
> actually explains some crashes I have noted in the past when working with 20
> million element lists that I wanted a slice of.  I had *plenty* of ram on
> that system.

Can you provide links to these notes? I'm looking at
cpython/Include/sliceobject.h that has this comment:

/*

A slice object containing start, stop, and step data members (the
names are from range).  After much talk with Guido, it was decided to
let these be any arbitrary python type.  Py_None stands for omitted values.
*/

Also, the code for slice objects in CPython works with Py_ssize_t (a
signed quantity of the same length as size_t), which will allow at
least 2**31 for an index. I would guess that your crashes were nothing
to do with 20 million elements and slices.

ChrisA

[toc] | [prev] | [next] | [standalone]

#32405

On Oct 29, 2012 7:10 AM, "Andrew Robinson" <andrew3@r3dsolutions.com> wrote:
> I will be porting Python 3.xx to a super low power embedded processor (MSP430), both space and speed are at a premium.
> Running Python on top of Java would be a *SERIOUS* mistake.  .NET won't even run on this system. etc.

If that's the case, then running Python at all is probably a mistake.
You know the interpreter alone has an overhead of nearly 6 MB?

> Yes, I realize that.
> But, why can't I just overload the existing __getitem__ for lists and not bother writing an entire class?

You can just overload that one method in a subclass of list.  Being
able to monkey-patch __getitem__ for the list class itself would not
be advisable, as it would affect all list slicing anywhere in your
program and possibly lead to some unexpected behaviors.

> Hmmm..
> Let's try your example exactly as shown...
>
> "hello world"[aslice]
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> NameError: name 'aslice' is not defined
>
> WOW. Cool.
> Where did the blanks *actually* get filled in?  Or HOW WILL they in your next post?

It appears that Steven omitted the definition of aslice by mistake.
It looks like it should have been:

aslice = slice(4, None, 2)

>  Looking at some of the online programming notes -- a slice apparently doesn't use an integer storage variable that is capable of arbitrary expansion. =-O -- and hence, won't work for very large sized lists.  That actually explains some crashes I have noted in the past when working with 20 million element lists that I wanted a slice of.  I had *plenty* of ram on that system.

20 million is nothing.  On a 32-bit system, sys.maxsize == 2 ** 31 -
1.  If the error you were seeing was MemoryError, then more likely you
were running into dynamic allocation issues due to fragmentation of
virtual memory.

[toc] | [prev] | [next] | [standalone]

Page 1 of 4  [1] 2 3 4  Next page →

Back to top | Article view | comp.lang.python

csiph-web