Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #89638 > unrolled thread

l = range(int(1E9))

Back to article view | Back to comp.lang.python

Contents

  l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-04-30 18:06 +0200
    Re: l = range(int(1E9)) Grant Edwards <invalid@invalid.invalid> - 2015-04-30 16:33 +0000
      Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-04-30 19:26 +0200
        Re: l = range(int(1E9)) Ben Finney <ben+python@benfinney.id.au> - 2015-05-01 03:41 +1000
          Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-04-30 20:44 +0200
      Re: l = range(int(1E9)) Roel Schroeven <roel@roelschroeven.net> - 2015-04-30 21:40 +0200
    Re: l = range(int(1E9)) Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-04-30 16:55 +0000
      Re: l = range(int(1E9)) Ben Finney <ben+python@benfinney.id.au> - 2015-05-01 03:20 +1000
        Re: l = range(int(1E9)) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-01 15:20 +1000
          Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-05-01 08:08 +0200
        Re: l = range(int(1E9)) BartC <bc@freeuk.com> - 2015-05-02 16:26 +0100
          Re: l = range(int(1E9)) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-02 16:40 +0100
            Re: l = range(int(1E9)) BartC <bc@freeuk.com> - 2015-05-02 17:17 +0100
              Re: l = range(int(1E9)) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-02 17:39 +0100
                Re: l = range(int(1E9)) BartC <bc@freeuk.com> - 2015-05-02 19:34 +0100
                  Re: l = range(int(1E9)) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-02 20:15 +0100
                    Re: l = range(int(1E9)) Marko Rauhamaa <marko@pacujo.net> - 2015-05-02 22:26 +0300
                    Re: l = range(int(1E9)) BartC <bc@freeuk.com> - 2015-05-02 20:51 +0100
                      Re: l = range(int(1E9)) Joel Goldstick <joel.goldstick@gmail.com> - 2015-05-02 17:21 -0400
                      Re: l = range(int(1E9)) Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-02 15:31 -0600
                      Re: l = range(int(1E9)) Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-05-02 21:40 +0000
                        Re: l = range(int(1E9)) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-02 23:26 +0100
                        Re: l = range(int(1E9)) BartC <bc@freeuk.com> - 2015-05-02 23:33 +0100
                          Re: l = range(int(1E9)) Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-05-03 00:13 +0000
                          Re: l = range(int(1E9)) Michael Torrie <torriem@gmail.com> - 2015-05-02 20:07 -0600
                          Re: l = range(int(1E9)) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-03 03:16 +0100
                            Re: l = range(int(1E9)) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-03 21:16 +1000
                              Re: l = range(int(1E9)) Chris Angelico <rosuav@gmail.com> - 2015-05-03 21:30 +1000
                              Re: l = range(int(1E9)) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-03 19:36 +0100
                          Re: l = range(int(1E9)) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-03 21:15 +1000
                            Re: l = range(int(1E9)) Chris Angelico <rosuav@gmail.com> - 2015-05-03 21:29 +1000
                        Re: l = range(int(1E9)) Terry Reedy <tjreedy@udel.edu> - 2015-05-02 19:26 -0400
                          Re: l = range(int(1E9)) Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-05-03 00:01 +0000
                            Re: l = range(int(1E9)) Terry Reedy <tjreedy@udel.edu> - 2015-05-02 23:18 -0400
                              Re: l = range(int(1E9)) Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-05-03 03:40 +0000
                        Re: l = range(int(1E9)) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-03 21:05 +1000
                      Re: l = range(int(1E9)) Terry Reedy <tjreedy@udel.edu> - 2015-05-02 19:51 -0400
                      Re: l = range(int(1E9)) Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-02 22:51 -0600
                        Re: l = range(int(1E9)) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-03 21:21 +1000
                  Re: l = range(int(1E9)) Marko Rauhamaa <marko@pacujo.net> - 2015-05-02 22:15 +0300
              Re: l = range(int(1E9)) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-03 20:32 +1000
                Re: l = range(int(1E9)) Chris Angelico <rosuav@gmail.com> - 2015-05-03 20:59 +1000
          Re: l = range(int(1E9)) Terry Reedy <tjreedy@udel.edu> - 2015-05-02 18:41 -0400
      Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-04-30 19:28 +0200
    Re: l = range(int(1E9)) Terry Reedy <tjreedy@udel.edu> - 2015-04-30 13:02 -0400
    Re: l = range(int(1E9)) Gary Herron <gherron@digipen.edu> - 2015-04-30 10:05 -0700
      Re: l = range(int(1E9)) Rob Gaddi <rgaddi@technologyhighland.invalid> - 2015-04-30 17:12 +0000
        Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-04-30 20:50 +0200
          Re: l = range(int(1E9)) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-04-30 21:53 +0100
            Re: l = range(int(1E9)) ElChino <elchino@cnn.cn> - 2015-04-30 23:23 +0200
              Re: l = range(int(1E9)) Chris Angelico <rosuav@gmail.com> - 2015-05-01 08:45 +1000
                Re: l = range(int(1E9)) ElChino <elchino@cnn.cn> - 2015-05-01 01:03 +0200
              Re: l = range(int(1E9)) Ben Finney <ben+python@benfinney.id.au> - 2015-05-01 09:12 +1000
                Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-05-01 07:04 +0200
                  Re: l = range(int(1E9)) Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-05-01 13:15 +0200
                    Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-05-01 14:22 +0200
                  Re: l = range(int(1E9)) Terry Reedy <tjreedy@udel.edu> - 2015-05-01 22:00 -0400
              Use ‘python2’ or ‘python3’, explicit is better than implicit  (was: l = range(int(1E9))) Ben Finney <ben+python@benfinney.id.au> - 2015-05-01 09:19 +1000
                Re: Use 'python2' or 'python3', explicit is better than implicit  (was: l = range(int(1E9))) Rustom Mody <rustompmody@gmail.com> - 2015-04-30 19:24 -0700
              Re: l = range(int(1E9)) Chris Angelico <rosuav@gmail.com> - 2015-05-01 09:23 +1000
            Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-05-01 06:19 +0200
              Re: l = range(int(1E9)) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-01 05:41 +0100
              Re: l = range(int(1E9)) Michael Torrie <torriem@gmail.com> - 2015-05-01 07:25 -0600
                Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-05-01 16:12 +0200
    Re: l = range(int(1E9)) Gisle Vanem <gvanem@yahoo.no> - 2015-04-30 20:23 +0200
      Re: l = range(int(1E9)) alister <alister.nospam.ware@ntlworld.com> - 2015-04-30 18:48 +0000
        Re: l = range(int(1E9)) Dave Angel <davea@davea.name> - 2015-04-30 14:59 -0400
          Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-04-30 22:18 +0200
            Re: l = range(int(1E9)) Tim Chase <python.list@tim.thechases.com> - 2015-04-30 16:41 -0500
              Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-05-01 07:19 +0200
    Re: l = range(int(1E9)) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-01 14:42 +1000
      Re: l = range(int(1E9)) Cecil Westerhof <Cecil@decebal.nl> - 2015-05-01 07:13 +0200
      Re: l = range(int(1E9)) Tony the Tiger <tony@tiger.invalid> - 2015-05-02 21:28 +0000
        Re: l = range(int(1E9)) Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-02 15:32 -0600
        Re: l = range(int(1E9)) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-03 20:56 +1000

Page 1 of 4  [1] 2 3 4  Next page →

#89638 — l = range(int(1E9))

If I execute:
    l = range(int(1E9)

The python process gobbles up all the memory and is killed. The
problem is that after this my swap is completely used, because other
processes have swapped to it. This make those programs more slowly. Is
there a way to circumvent Python claiming all the memory?

By the way: this is CPython 2.7.8.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [next] | [standalone]

#89640

On 2015-04-30, Cecil Westerhof <Cecil@decebal.nl> wrote:
> If I execute:
>     l = range(int(1E9)
>
> The python process gobbles up all the memory and is killed. The
> problem is that after this my swap is completely used, because other
> processes have swapped to it. This make those programs more slowly.
> Is there a way to circumvent Python claiming all the memory?

I presume "don't do that" has already occured to you?

You can always use ulimit to limit the memory allowed for the process
running Python.

-- 
Grant Edwards               grant.b.edwards        Yow! Should I get locked
                                  at               in the PRINCICAL'S
                              gmail.com            OFFICE today -- or have
                                                   a VASECTOMY??

[toc] | [prev] | [next] | [standalone]

#89649

Op Thursday 30 Apr 2015 18:33 CEST schreef Grant Edwards:

> On 2015-04-30, Cecil Westerhof <Cecil@decebal.nl> wrote:
>> If I execute:
>> l = range(int(1E9)
>>
>> The python process gobbles up all the memory and is killed. The
>> problem is that after this my swap is completely used, because
>> other processes have swapped to it. This make those programs more
>> slowly. Is there a way to circumvent Python claiming all the
>> memory?
>
> I presume "don't do that" has already occured to you?

Well, it is just playing with the possibilities/limits of Python.
Better getting the problem now, then when I can not afford to lose
time. ;-)


> You can always use ulimit to limit the memory allowed for the
> process running Python.

That works, yes. Now I get a MemoryError and the other processes are
left alone. Now determining what are the best values.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#89651

Cecil Westerhof <Cecil@decebal.nl> writes:

> That works, yes. Now I get a MemoryError and the other processes are
> left alone. Now determining what are the best values.

I would strongly recommend that “best values” includes “run Python
version >= 3”.

One of the many problems you avoid by leaving Python 2 behind is that
many functions which used to return entire collections, now return lazy
evaluators (such as generators or views) which will not consume memory
the way you're describing.

Please try to learn Python using only the currently-developed Python 3.

-- 
 \     “Listen: we are here on Earth to fart around. Don't let anybody |
  `\                  tell you otherwise.” —_Timequake_, Kurt Vonnegut |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#89661

Op Thursday 30 Apr 2015 19:41 CEST schreef Ben Finney:

> Cecil Westerhof <Cecil@decebal.nl> writes:
>
>> That works, yes. Now I get a MemoryError and the other processes
>> are left alone. Now determining what are the best values.
>
> I would strongly recommend that “best values” includes “run Python
> version >= 3”.
>
> One of the many problems you avoid by leaving Python 2 behind is
> that many functions which used to return entire collections, now
> return lazy evaluators (such as generators or views) which will not
> consume memory the way you're describing.
>
> Please try to learn Python using only the currently-developed Python
> 3.

The problem with that is that in my neighbourhood 2 is still mostly
exclusively used. I want to learn also 3, but 2 has more priority.

The modules I publish on GitHub work with both. :-)

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#89665

Grant Edwards schreef op 2015-04-30 18:33:
> On 2015-04-30, Cecil Westerhof <Cecil@decebal.nl> wrote:
>> If I execute:
>>     l = range(int(1E9)
>>
>> The python process gobbles up all the memory and is killed. The
>> problem is that after this my swap is completely used, because other
>> processes have swapped to it. This make those programs more slowly.
>> Is there a way to circumvent Python claiming all the memory?
> 
> I presume "don't do that" has already occured to you?

A few weeks ago, I had a bug in a Python script which caused it to 
consume all memory, thrashed my computer; the system become unresponsive 
  and it was very difficult to kill the process and get everything back 
in working order.

Then I thought (foolishly) that the bug was fixed, so I ran the script 
again, with the same disastrous results. And then again. Stupid, I know.

A way to limit memory usage would have been nice: the script would have 
been killed before it could grind the whole system to a halt.

> You can always use ulimit to limit the memory allowed for the process
> running Python.

Sadly in my case the OS is Windows, and as far as I know it doesn't have 
a ulimit equivalent for limiting memory usage.

I guess I could have used 32-bit Python instead of 64-bit Python to 
limit available memory.


-- 
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
   -- Isaac Asimov

Roel Schroeven

[toc] | [prev] | [next] | [standalone]

#89642

On 2015-04-30, Cecil Westerhof <Cecil@decebal.nl> wrote:
> If I execute:
>     l = range(int(1E9)
>
> The python process gobbles up all the memory and is killed. The
> problem is that after this my swap is completely used, because other
> processes have swapped to it. This make those programs more slowly. Is
> there a way to circumvent Python claiming all the memory?
>
> By the way: this is CPython 2.7.8.

It's your operating system's job to handle processes.

If you use xrange() instead of range() then you will get an iterator
which will return each of the numbers in turn without any need to
create an enormous list of all of them.

[toc] | [prev] | [next] | [standalone]

#89648

Jon Ribbens <jon+usenet@unequivocal.co.uk> writes:

> On 2015-04-30, Cecil Westerhof <Cecil@decebal.nl> wrote:
> > If I execute:
> >     l = range(int(1E9)
> >
> > The python process gobbles up all the memory and is killed. […] Is
> > there a way to circumvent Python claiming all the memory?

You seem to be asking for a way to stop a program doing exactly what
it's written to do. I don't know what kind of answer you expect.

> It's your operating system's job to handle processes.

Indeed. In this case, the program is written to gobble up memory, and
the operating system kills it. To “circumvent” that behaviour surely
reveals the problem: that the operating system isn't handling processes
very well.

> > By the way: this is CPython 2.7.8.
>
> If you use xrange() instead of range() then you will get an iterator
> which will return each of the numbers in turn without any need to
> create an enormous list of all of them.

If you use Python 3 instead of the obsolescent Python 2, the ‘range’
callable has this sensible behaviour by default.

-- 
 \       “Corporation, n. An ingenious device for obtaining individual |
  `\       profit without individual responsibility.” —Ambrose Bierce, |
_o__)                                   _The Devil's Dictionary_, 1906 |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#89696

On Fri, 1 May 2015 03:20 am, Ben Finney wrote:

> Jon Ribbens <jon+usenet@unequivocal.co.uk> writes:
> 
>> On 2015-04-30, Cecil Westerhof <Cecil@decebal.nl> wrote:
>> > If I execute:
>> > l = range(int(1E9)
>> >
>> > The python process gobbles up all the memory and is killed. […] Is
>> > there a way to circumvent Python claiming all the memory?
> 
> You seem to be asking for a way to stop a program doing exactly what
> it's written to do. I don't know what kind of answer you expect.

How about a way to stop a program from doing what you didn't intend it to
do? As we all know quite well, computers are quite stupid, they do what
they are told, not what we want.

It's not just to protect against bugs and mistakes. It's also to protect one
user from another user, or a single user from hostile code. Just because
Cecil tells the computer "use up 12GB of RAM in one great big list" doesn't
mean that the other users of that computer have to acquiesce to that
request.


>> It's your operating system's job to handle processes.
> 
> Indeed. In this case, the program is written to gobble up memory, and
> the operating system kills it. To “circumvent” that behaviour surely
> reveals the problem: that the operating system isn't handling processes
> very well.

Indeed indeed :-) 

Some programming language virtual machines limit how much memory they will
use. The CPython VM isn't one of those, although I understand that both
Jython and IronPython are. (I may be wrong -- corrections are welcome.)
That means you have to use the OS to limit how much memory CPython will
use, if you can.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#89700

Op Friday 1 May 2015 07:20 CEST schreef Steven D'Aprano:

> Some programming language virtual machines limit how much memory
> they will use. The CPython VM isn't one of those, although I
> understand that both Jython and IronPython are. (I may be wrong --

Jython runs in the JVM, so Jython is.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#89771

On 30/04/2015 18:20, Ben Finney wrote:
> Jon Ribbens <jon+usenet@unequivocal.co.uk> writes:

>> If you use xrange() instead of range() then you will get an iterator
>> which will return each of the numbers in turn without any need to
>> create an enormous list of all of them.
>
> If you use Python 3 instead of the obsolescent Python 2, the ‘range’
> callable has this sensible behaviour by default.

When I first looked at Python 20 or so years ago this seemed to be the 
standard way of writing a for-loop:

for i in range(N):
    ....

I remember being completely astonished at the time that 'range' actually 
created a list of values from 0 to N-1.

Python was already known to be slow yet it deliberately crippled itself 
by using just about the slowest method imaginable of executing a simple 
iterative loop? By first creating a list of a million objects!

And sometimes you weren't even interested in the values but just wanted 
to execute something N times so it was a wasted effort.

That was eventually fixed with xrange, but why do it like that in the 
first place?

(At the time, I was creating bytecode languages as part of the 
applications I was writing. An empty for-loop executed just one bytecode 
per iteration, and it was also the fastest bytecode instruction. An 
empty for-loop executed three Python bytecodes per iteration last time I 
looked. It seems that Python used to like making a rod for its own back.)

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#89774

On 02/05/2015 16:26, BartC wrote:
> On 30/04/2015 18:20, Ben Finney wrote:
>> Jon Ribbens <jon+usenet@unequivocal.co.uk> writes:
>
>>> If you use xrange() instead of range() then you will get an iterator
>>> which will return each of the numbers in turn without any need to
>>> create an enormous list of all of them.
>>
>> If you use Python 3 instead of the obsolescent Python 2, the ‘range’
>> callable has this sensible behaviour by default.
>
> When I first looked at Python 20 or so years ago this seemed to be the
> standard way of writing a for-loop:
>
> for i in range(N):
>     ....
>
> I remember being completely astonished at the time that 'range' actually
> created a list of values from 0 to N-1.
>
> Python was already known to be slow yet it deliberately crippled itself
> by using just about the slowest method imaginable of executing a simple
> iterative loop? By first creating a list of a million objects!
>
> And sometimes you weren't even interested in the values but just wanted
> to execute something N times so it was a wasted effort.
>
> That was eventually fixed with xrange, but why do it like that in the
> first place?
>
> (At the time, I was creating bytecode languages as part of the
> applications I was writing. An empty for-loop executed just one bytecode
> per iteration, and it was also the fastest bytecode instruction. An
> empty for-loop executed three Python bytecodes per iteration last time I
> looked. It seems that Python used to like making a rod for its own back.)
>

I first started maybe 14 years ago and the standard way of writing a for 
loop was, and still is:-

for item in items:

When did this change, or has it always been this way and you were simply 
using an idiom from other languages?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#89777

On 02/05/2015 16:40, Mark Lawrence wrote:
> On 02/05/2015 16:26, BartC wrote:
>> On 30/04/2015 18:20, Ben Finney wrote:
>>> Jon Ribbens <jon+usenet@unequivocal.co.uk> writes:
>>
>>>> If you use xrange() instead of range() then you will get an iterator
>>>> which will return each of the numbers in turn without any need to
>>>> create an enormous list of all of them.
>>>
>>> If you use Python 3 instead of the obsolescent Python 2, the ‘range’
>>> callable has this sensible behaviour by default.
>>
>> When I first looked at Python 20 or so years ago this seemed to be the
>> standard way of writing a for-loop:
>>
>> for i in range(N):
>>     ....
>>
>> I remember being completely astonished at the time that 'range' actually
>> created a list of values from 0 to N-1.

> I first started maybe 14 years ago and the standard way of writing a for
> loop was, and still is:-
>
> for item in items:
>
> When did this change, or has it always been this way and you were simply
> using an idiom from other languages?

Your example is the equivalent of 'forall' in other languages, where you 
iterate over the values of some collection of data.

I agree that most for-loops in Pythonic code probably fall into that 
category.

But for looping over a simple integer range, then using 'range' to 
denote the range (and build a list as it used to do), was how it was 
done. And earlier on people would have been porting coding code to 
Python at which point a straightforward 'for i=a to b' loop suddenly 
acquired a substantial overhead it didn't have before!

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#89778

On 02/05/2015 17:17, BartC wrote:
> On 02/05/2015 16:40, Mark Lawrence wrote:
>> On 02/05/2015 16:26, BartC wrote:
>>> On 30/04/2015 18:20, Ben Finney wrote:
>>>> Jon Ribbens <jon+usenet@unequivocal.co.uk> writes:
>>>
>>>>> If you use xrange() instead of range() then you will get an iterator
>>>>> which will return each of the numbers in turn without any need to
>>>>> create an enormous list of all of them.
>>>>
>>>> If you use Python 3 instead of the obsolescent Python 2, the ‘range’
>>>> callable has this sensible behaviour by default.
>>>
>>> When I first looked at Python 20 or so years ago this seemed to be the
>>> standard way of writing a for-loop:
>>>
>>> for i in range(N):
>>>     ....
>>>
>>> I remember being completely astonished at the time that 'range' actually
>>> created a list of values from 0 to N-1.
>
>> I first started maybe 14 years ago and the standard way of writing a for
>> loop was, and still is:-
>>
>> for item in items:
>>
>> When did this change, or has it always been this way and you were simply
>> using an idiom from other languages?
>
> Your example is the equivalent of 'forall' in other languages, where you
> iterate over the values of some collection of data.
>
> I agree that most for-loops in Pythonic code probably fall into that
> category.
>
> But for looping over a simple integer range, then using 'range' to
> denote the range (and build a list as it used to do), was how it was
> done. And earlier on people would have been porting coding code to
> Python at which point a straightforward 'for i=a to b' loop suddenly
> acquired a substantial overhead it didn't have before!
>

All you are saying is that they didn't bother reading the docs and 
learning how to write a *PYTHON* for loop.  Failing that don't bother 
using range, just directly convert your (say) C loop into Python.  I 
really don't see any issue here at all.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#89786

On 02/05/2015 17:39, Mark Lawrence wrote:
> On 02/05/2015 17:17, BartC wrote:
>> On 02/05/2015 16:40, Mark Lawrence wrote:


>>> for item in items:
>>>
>>> When did this change, or has it always been this way and you were simply
>>> using an idiom from other languages?
>>
>> Your example is the equivalent of 'forall' in other languages, where you
>> iterate over the values of some collection of data.
>>
>> I agree that most for-loops in Pythonic code probably fall into that
>> category.
>>
>> But for looping over a simple integer range, then using 'range' to
>> denote the range (and build a list as it used to do), was how it was
>> done. And earlier on people would have been porting coding code to
>> Python at which point a straightforward 'for i=a to b' loop suddenly
>> acquired a substantial overhead it didn't have before!
>>
>
> All you are saying is that they didn't bother reading the docs and
> learning how to write a *PYTHON* for loop.  Failing that don't bother
> using range, just directly convert your (say) C loop into Python.  I
> really don't see any issue here at all.

OK, so it's the programmer's fault if as fundamental a concept as a 
for-loop ranging over integers is implemented inefficiently. He has to 
transform it into high-level terms, or has to reconstruct it somehow 
using a while-loop and an incrementing loop index.

Now I understand (why Python has been beset for so long with performance 
problems, if that is a typical attitude!).

BTW, why did they introduce the 'xrange' thing; for fun? Or did someone 
realise there might have been an issue after all?

(I tried an empty loop counting to 1 billion in Python 2.x, using 'for i 
in range'. It ran out of memory. Counting to 100 million instead, it 
worked, but still used a massive 1.5GB RAM while doing so (and took 6 
seconds to count to 100M, not too bad for Python)

Outside Python, it might typically take a few seconds to count to 1 
billion, and would use virtually no memory.

Why would anyone want to loop over all those numbers instead of 
iterating over actual data? For any number of reasons. Counting how many 
happy numbers are in that range for one!)

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#89787

On 02/05/2015 19:34, BartC wrote:
> On 02/05/2015 17:39, Mark Lawrence wrote:
>> On 02/05/2015 17:17, BartC wrote:
>>> On 02/05/2015 16:40, Mark Lawrence wrote:
>
>
>>>> for item in items:
>>>>
>>>> When did this change, or has it always been this way and you were
>>>> simply
>>>> using an idiom from other languages?
>>>
>>> Your example is the equivalent of 'forall' in other languages, where you
>>> iterate over the values of some collection of data.
>>>
>>> I agree that most for-loops in Pythonic code probably fall into that
>>> category.
>>>
>>> But for looping over a simple integer range, then using 'range' to
>>> denote the range (and build a list as it used to do), was how it was
>>> done. And earlier on people would have been porting coding code to
>>> Python at which point a straightforward 'for i=a to b' loop suddenly
>>> acquired a substantial overhead it didn't have before!
>>>
>>
>> All you are saying is that they didn't bother reading the docs and
>> learning how to write a *PYTHON* for loop.  Failing that don't bother
>> using range, just directly convert your (say) C loop into Python.  I
>> really don't see any issue here at all.
>
> OK, so it's the programmer's fault if as fundamental a concept as a
> for-loop ranging over integers is implemented inefficiently. He has to
> transform it into high-level terms, or has to reconstruct it somehow
> using a while-loop and an incrementing loop index.
>
> Now I understand (why Python has been beset for so long with performance
> problems, if that is a typical attitude!).
>
> BTW, why did they introduce the 'xrange' thing; for fun? Or did someone
> realise there might have been an issue after all?
>
> (I tried an empty loop counting to 1 billion in Python 2.x, using 'for i
> in range'. It ran out of memory. Counting to 100 million instead, it
> worked, but still used a massive 1.5GB RAM while doing so (and took 6
> seconds to count to 100M, not too bad for Python)
>
> Outside Python, it might typically take a few seconds to count to 1
> billion, and would use virtually no memory.
>
> Why would anyone want to loop over all those numbers instead of
> iterating over actual data? For any number of reasons. Counting how many
> happy numbers are in that range for one!)
>

I give up.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#89790

Mark Lawrence <breamoreboy@yahoo.co.uk>:

> On 02/05/2015 19:34, BartC wrote:
>> [...]
>> [...]
>> [...]
>> [...]
>> [...]
>> [...]
>> [...]
>> [...]
>> [...]
>> [...]
>> [...]
>> [...]
>> [...]
>> [...]
[etc etc etc]
>
> I give up.

Mark, you do a commendable job admonishing the forum participants
against top-posting. Let me bring up another highly recommended
practice: trimming, or pruning, your quotes.

   The proper order is
 
      > Quote 1 (properly pruned)
 
      Your response 1
 
      > Quote 2 (properly pruned)
 
      Your response 2

   <URL: http://lipas.uwasa.fi/~ts/http/quote.html>


Marko

[toc] | [prev] | [next] | [standalone]

#89794

On 02/05/2015 20:15, Mark Lawrence wrote:
> On 02/05/2015 19:34, BartC wrote:

>> OK, so it's the programmer's fault if as fundamental a concept as a
>> for-loop ranging over integers is implemented inefficiently. He has to
>> transform it into high-level terms, or has to reconstruct it somehow
>> using a while-loop and an incrementing loop index.

> I give up.

So do I, I think, if no-one is willing to admit that the original way of 
implementing range() was a glaring mistake.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#89799

>> I give up.
>
>
> So do I, I think, if no-one is willing to admit that the original way of
> implementing range() was a glaring mistake.
>
>
> --
> Bartc
>
It doesn't matter for small ranges.  Anyway it was fixed.
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Joel Goldstick
http://joelgoldstick.com

[toc] | [prev] | [next] | [standalone]

#89802

On Sat, May 2, 2015 at 1:51 PM, BartC <bc@freeuk.com> wrote:
> On 02/05/2015 20:15, Mark Lawrence wrote:
>>
>> On 02/05/2015 19:34, BartC wrote:
>
>
>>> OK, so it's the programmer's fault if as fundamental a concept as a
>>> for-loop ranging over integers is implemented inefficiently. He has to
>>> transform it into high-level terms, or has to reconstruct it somehow
>>> using a while-loop and an incrementing loop index.
>
>
>> I give up.
>
>
> So do I, I think, if no-one is willing to admit that the original way of
> implementing range() was a glaring mistake.

range() was and is a *convenience* function. In the real world, the
vast majority of for loops are over arrays or other containers, not
integers, and those that aren't are usually very small. In non-toy
code, using a for loop to count to a billion is highly unusual.

So yeah, for a programmer porting code to Python who needed to loop
over an array, the correct approach would be to actually loop over the
*array* in place of the indices of the array. I don't know why you
make this out to be such a big deal; it's a simple conversion.

Would it have been better if range() had been implemented as xrange()
from the beginning? Sure, that would have been great. Except for one
small detail: the iterator protocol didn't exist back then. That
wasn't introduced until PEP 234 in Python 2.1, which means that the
xrange() function wasn't even *possible* before then.

I don't think anybody would claim that Python was perfect when it was
first introduced (nor is it perfect now). Like all other software, it
has improved over time as a result of iterative refinement.

[toc] | [prev] | [next] | [standalone]

Page 1 of 4  [1] 2 3 4  Next page →

Back to top | Article view | comp.lang.python

csiph-web