Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #19222 > unrolled thread

Looking under Python's hood: Will we find a high performance or clunky engine?

Back to article view | Back to comp.lang.python

Contents

  Looking under Python's hood: Will we find a high performance or clunky engine? Rick Johnson <rantingrickjohnson@gmail.com> - 2012-01-22 07:50 -0800
    Re: Looking under Python's hood: Will we find a high performance or clunky engine? Heiko Wundram <modelnine@modelnine.org> - 2012-01-22 18:52 +0100
    Re: Looking under Python's hood: Will we find a high performance or clunky engine? Robert Kern <robert.kern@gmail.com> - 2012-01-22 18:01 +0000
      Re: Looking under Python's hood: Will we find a high performance or clunky engine? 88888 Dihedral <dihedral88888@googlemail.com> - 2012-01-23 10:56 -0800
      Re: Looking under Python's hood: Will we find a high performance or clunky engine? 88888 Dihedral <dihedral88888@googlemail.com> - 2012-01-23 10:56 -0800
        Re: Looking under Python's hood: Will we find a high performance or clunky engine? alex23 <wuwei23@gmail.com> - 2012-01-23 22:44 -0800
          Re: Looking under Python's hood: Will we find a high performance or clunky engine? Chris Angelico <rosuav@gmail.com> - 2012-01-24 19:14 +1100
    Re: Looking under Python's hood: Will we find a high performance or clunky engine? Michael Torrie <torriem@gmail.com> - 2012-01-22 17:38 -0700
      Re: Looking under Python's hood: Will we find a high performance or clunky engine? Rick Johnson <rantingrickjohnson@gmail.com> - 2012-01-22 17:04 -0800
        Re: Looking under Python's hood: Will we find a high performance or clunky engine? Michael Torrie <torriem@gmail.com> - 2012-01-22 18:32 -0700
    Re: Looking under Python's hood: Will we find a high performance or clunky engine? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-01-23 09:11 +0000
    Re: Looking under Python's hood: Will we find a high performance or clunky engine? Grant Edwards <invalid@invalid.invalid> - 2012-01-23 16:08 +0000

#19222 — Looking under Python's hood: Will we find a high performance or clunky engine?

What does Python do when presented with this code?

py> [line.strip('\n') for line in f.readlines()]

If Python reads all the file lines first and THEN iterates AGAIN to do
the strip; we are driving a Fred flintstone mobile. If however Python
strips each line of the lines passed into readlines in one fell swoop,
we made the correct choice.

Which is it Pythonistas? Which is it?

[toc] | [next] | [standalone]

#19226

Am 22.01.2012 16:50, schrieb Rick Johnson:
> What does Python do when presented with this code?
>
> py>  [line.strip('\n') for line in f.readlines()]
>
> If Python reads all the file lines first and THEN iterates AGAIN to do
> the strip; we are driving a Fred flintstone mobile. If however Python
> strips each line of the lines passed into readlines in one fell swoop,
> we made the correct choice.
>
> Which is it Pythonistas? Which is it?

You aren't one (considering how vocal you are in arguing for changes to 
the language)?

So: shouldn't you be able to answer your own question?

-- 
--- Heiko.

[toc] | [prev] | [next] | [standalone]

#19227

On 1/22/12 3:50 PM, Rick Johnson wrote:
>
> What does Python do when presented with this code?
>
> py>  [line.strip('\n') for line in f.readlines()]
>
> If Python reads all the file lines first and THEN iterates AGAIN to do
> the strip; we are driving a Fred flintstone mobile. If however Python
> strips each line of the lines passed into readlines in one fell swoop,
> we made the correct choice.
>
> Which is it Pythonistas? Which is it?

The .readlines() method is an old API that predates the introduction of 
iterators to Python. The modern way to do this in one iteration is to use the 
file object as an iterator:

   [line.strip('\n') for line in f]

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#19275

在 2012年1月23日星期一UTC+8上午2时01分11秒，Robert Kern写道：
> On 1/22/12 3:50 PM, Rick Johnson wrote:
> >
> > What does Python do when presented with this code?
> >
> > py>  [line.strip('\n') for line in f.readlines()]
> >
> > If Python reads all the file lines first and THEN iterates AGAIN to do
> > the strip; we are driving a Fred flintstone mobile. If however Python
> > strips each line of the lines passed into readlines in one fell swoop,
> > we made the correct choice.
> >
> > Which is it Pythonistas? Which is it?
> 
> The .readlines() method is an old API that predates the introduction of 
> iterators to Python. The modern way to do this in one iteration is to use the 
> file object as an iterator:
> 
>    [line.strip('\n') for line in f]

This is more powerful by turning an object to be iterable.

But the list comprehension violates the basic operating 
principle of the iteratee chaining rule in programming.
 
I know manny python programmers just abandon the list comprehension
in non-trivial processes. 

[toc] | [prev] | [next] | [standalone]

#19276

在 2012年1月23日星期一UTC+8上午2时01分11秒，Robert Kern写道：
> On 1/22/12 3:50 PM, Rick Johnson wrote:
> >
> > What does Python do when presented with this code?
> >
> > py>  [line.strip('\n') for line in f.readlines()]
> >
> > If Python reads all the file lines first and THEN iterates AGAIN to do
> > the strip; we are driving a Fred flintstone mobile. If however Python
> > strips each line of the lines passed into readlines in one fell swoop,
> > we made the correct choice.
> >
> > Which is it Pythonistas? Which is it?
> 
> The .readlines() method is an old API that predates the introduction of 
> iterators to Python. The modern way to do this in one iteration is to use the 
> file object as an iterator:
> 
>    [line.strip('\n') for line in f]

This is more powerful by turning an object to be iterable.

But the list comprehension violates the basic operating 
principle of the iteratee chaining rule in programming.
 
I know manny python programmers just abandon the list comprehension
in non-trivial processes. 

[toc] | [prev] | [next] | [standalone]

#19324

On Jan 24, 4:56 am, 88888 Dihedral <dihedral88...@googlemail.com>
wrote:
> 在 2012年1月23日星期一UTC+8上午2时01分11秒，Robert Kern写道：
> >    [line.strip('\n') for line in f]
>
> This is more powerful by turning an object to be iterable.
> But the list comprehension violates the basic operating
> principle of the iteratee chaining rule in programming.

Thankfully, the syntax is almost identical for generators, which are
chain-able:

    noEOLs = (line.strip('\n') for line in f)
    txtSuffix = (line for line in noEOLs if line.endswith('txt'))
    ...etc

> I know manny python programmers just abandon the list comprehension
> in non-trivial processes.

Really? Observation of the python mailing list indicates the opposite:
people seem inclined to use them no matter what.

Also: PLEASE STOP DOUBLE POSTING.

[toc] | [prev] | [next] | [standalone]

#19325

On Tue, Jan 24, 2012 at 5:44 PM, alex23 <wuwei23@gmail.com> wrote:
> On Jan 24, 4:56 am, 88888 Dihedral <dihedral88...@googlemail.com>
> wrote:
>> I know manny python programmers just abandon the list comprehension
>> in non-trivial processes.
>
> Really? Observation of the python mailing list indicates the opposite:
> people seem inclined to use them no matter what.

You're responding to a bot, although I think a human must be
post-processing and selecting for humour.

ChrisA

[toc] | [prev] | [next] | [standalone]

#19241

On 01/22/2012 08:50 AM, Rick Johnson wrote:
> 
> What does Python do when presented with this code?
> 
> py> [line.strip('\n') for line in f.readlines()]
> 
> If Python reads all the file lines first and THEN iterates AGAIN to do
> the strip; we are driving a Fred flintstone mobile. If however Python
> strips each line of the lines passed into readlines in one fell swoop,
> we made the correct choice.
> 
> Which is it Pythonistas? Which is it?

You're doing it wrong, obviously.  I'm actually surprised that an expert
such as yourself would read a file in this way.  In any language.
Surely you would iterate over the file object which is the obvious way
to do it.

I guess we'll chalk this up as another python pitfall.  Looking forward
to your programming language which will prevent such things while
maintaining the purity and beauty of Python's ideals.

[toc] | [prev] | [next] | [standalone]

#19245

On Jan 22, 6:38 pm, Michael Torrie <torr...@gmail.com> wrote:
> On 01/22/2012 08:50 AM, Rick Johnson wrote:
>
>
>
> > What does Python do when presented with this code?
>
> > py> [line.strip('\n') for line in f.readlines()]
>
> > If Python reads all the file lines first and THEN iterates AGAIN to do
> > the strip; we are driving a Fred flintstone mobile. If however Python
> > strips each line of the lines passed into readlines in one fell swoop,
> > we made the correct choice.
>
> > Which is it Pythonistas? Which is it?
>
> You're doing it wrong, obviously.  I'm actually surprised that an expert
> such as yourself would read a file in this way.  In any language.
> Surely you would iterate over the file object which is the obvious way
> to do it.

That's just the point. If an expert such as myself can make a simple
mistake as this, then one can only expect that the neophytes are going
to suffer greatly. I wonder how many tutorials are out there in WWW
still teaching old ways of writing Python code? Old ways that have
been superseded by newer versions.

> I guess we'll chalk this up as another python pitfall.  Looking forward
> to your programming language which will prevent such things while
> maintaining the purity and beauty of Python's ideals.

Purity is a myth perpetrated by those who are blinded by their own
sanctimonious ideals of self worth. Any language that vows to be
"pure" (in a backwards compatible sense) is doomed to bitrot.

Programming languages and creators have a father and daughter
relationship.

The creator (father) wants his language (daughter) to be pure forever
and ever. You can lock her away in a tower with a titanium chastity
belt and stick your head in the sand but sooner or later a prince
charming (evolution) is going to come along and defile her. Will you
have the strength to realize that your daughters life is more
important than your feeble attachments to nostalgic purity?

The fact is, Python's purity has been defiled already. And although we
must give GvR the credit he deserves for making these long overdue
changes, he did not go far enough. Essentially he has accepted his
daughter will not longer be pure but he refuses to acknowledge the new
communion. His internal drive to evolve hath waxed cold. He has the
license plate and the job title and that seems to be enough. Where is
the passion?

[toc] | [prev] | [next] | [standalone]

#19246

On 01/22/2012 06:04 PM, Rick Johnson wrote:
> That's just the point. If an expert such as myself can make a simple
> mistake as this, then one can only expect that the neophytes are going
> to suffer greatly. I wonder how many tutorials are out there in WWW
> still teaching old ways of writing Python code? Old ways that have
> been superseded by newer versions.

Well that's that then.  Everything is good.


> Programming languages and creators have a father and daughter
> relationship.
> 
> The creator (father) wants his language (daughter) to be pure forever
> and ever.

Good thing you are so funny, or back into the plonk file you would go.
You're welcome to fork and deflower our pure Python if you want (you
seem to be quite stuck on sex metaphors).  Please feel free.  Looking
forward to your less-virginal take on the language.  Seems like you know
what's best, and only you can save and ravish Python at the same time.

[toc] | [prev] | [next] | [standalone]

#19256

On Sun, 22 Jan 2012 07:50:59 -0800, Rick Johnson wrote:

> What does Python do when presented with this code?
> 
> py> [line.strip('\n') for line in f.readlines()]
> 
> If Python reads all the file lines first and THEN iterates AGAIN to do
> the strip; we are driving a Fred flintstone mobile.

Nonsense. File-like objects offer two APIs: there is a lazy iterator 
approach, using the file-like object itself as an iterator, and an eager 
read-it-all-at-once approach, offered by the venerable readlines() 
method. readlines *deliberately* reads the entire file, and if you as a 
developer do so by accident, you have no-one to blame but yourself. Only 
a poor tradesman blames his tools instead of taking responsibility for 
learning how to use them himself.

You should use whichever approach is more appropriate for your situation. 
You might want to consider reading from the file as quickly as possible, 
in one big chunk if you can, so you can close it again and let other 
applications have access to it. Or you might not care. The choice is 
yours.

For small files, readlines() will probably be faster, although for small 
files it won't make much practical difference. Who cares whether it takes 
0.01ms or 0.02ms? For medium sized files, say, a few thousand lines, it 
could go either way, depending on memory use, the size of the internal 
file buffer, and implementation details. Only for files large enough that 
allocating memory for all the lines at once becomes significant will lazy 
iteration be a clear winner.

But if the file is that big, are you sure that a list comprehension is 
the right tool in the first place?

In general, you should not care greatly which of the two you use, unless 
profiling your application shows that this is the bottleneck.

But it is extremely unlikely that copying even a few thousands lines 
around memory will be slower than reading them from disk in the first 
place. Unless you expect to be handling truly large files, you've got 
more important things to optimize before wasting time caring about this.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#19268

On 2012-01-22, Rick Johnson <rantingrickjohnson@gmail.com> wrote:

> What does Python do when presented with this code?

It does what you tell it to.  What else would you expect?

-- 
Grant Edwards               grant.b.edwards        Yow! Are we wet yet?
                                  at               
                              gmail.com            

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web