Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #82076 > unrolled thread

Re: Python Iterables struggling using map() built-in

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Python Iterables struggling using map() built-in Chris Angelico <rosuav@gmail.com> - 2014-12-08 05:29 +1100
    Re: Python Iterables struggling using map() built-in Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-12-08 10:33 +1100
      Re: Python Iterables struggling using map() built-in Chris Angelico <rosuav@gmail.com> - 2014-12-08 10:50 +1100
        Re: Python Iterables struggling using map() built-in Roy Smith <roy@panix.com> - 2014-12-07 19:27 -0500
          Re: Python Iterables struggling using map() built-in Chris Angelico <rosuav@gmail.com> - 2014-12-08 11:35 +1100
            Re: Python Iterables struggling using map() built-in Steven D'Aprano <steve@pearwood.info> - 2014-12-08 09:40 +0000
              Re: Python Iterables struggling using map() built-in Chris Angelico <rosuav@gmail.com> - 2014-12-08 20:50 +1100
              Re: Python Iterables struggling using map() built-in Roy Smith <roy@panix.com> - 2014-12-08 19:02 -0500
      Re: Python Iterables struggling using map() built-in Roy Smith <roy@panix.com> - 2014-12-07 19:12 -0500
        Re: Python Iterables struggling using map() built-in Chris Angelico <rosuav@gmail.com> - 2014-12-08 11:34 +1100
          Re: Python Iterables struggling using map() built-in Roy Smith <roy@panix.com> - 2014-12-07 19:45 -0500
            Re: Python Iterables struggling using map() built-in Chris Angelico <rosuav@gmail.com> - 2014-12-08 12:00 +1100
            Re: Python Iterables struggling using map() built-in MRAB <python@mrabarnett.plus.com> - 2014-12-08 03:09 +0000
        Re: Python Iterables struggling using map() built-in Terry Reedy <tjreedy@udel.edu> - 2014-12-07 19:38 -0500
          Re: Python Iterables struggling using map() built-in Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-12-08 18:06 +1300
        Re: Python Iterables struggling using map() built-in Ned Batchelder <ned@nedbatchelder.com> - 2014-12-07 20:06 -0500
        Re: Python Iterables struggling using map() built-in Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-12-09 13:50 +1100
          Re: Python Iterables struggling using map() built-in Terry Reedy <tjreedy@udel.edu> - 2014-12-09 00:03 -0500
            Re: Python Iterables struggling using map() built-in Steven D'Aprano <steve@pearwood.info> - 2014-12-09 06:40 +0000
          Re: Python Iterables struggling using map() built-in Terry Reedy <tjreedy@udel.edu> - 2014-12-09 15:06 -0500
            Re: Python Iterables struggling using map() built-in Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-12-10 11:10 +1100
              Re: Python Iterables struggling using map() built-in Chris Angelico <rosuav@gmail.com> - 2014-12-10 11:20 +1100
              Re: Python Iterables struggling using map() built-in Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-12-10 00:24 +0000
              Re: Python Iterables struggling using map() built-in Roy Smith <roy@panix.com> - 2014-12-09 21:44 -0500
                Re: Python Iterables struggling using map() built-in Steven D'Aprano <steve@pearwood.info> - 2014-12-10 09:24 +0000
                  Re: Python Iterables struggling using map() built-in Chris Angelico <rosuav@gmail.com> - 2014-12-10 20:32 +1100
                  Re: Python Iterables struggling using map() built-in Marko Rauhamaa <marko@pacujo.net> - 2014-12-10 12:58 +0200
                  Re: Python Iterables struggling using map() built-in Ian Kelly <ian.g.kelly@gmail.com> - 2014-12-10 09:46 -0700
                    Re: Python Iterables struggling using map() built-in Steven D'Aprano <steve@pearwood.info> - 2014-12-11 04:01 +0000
                      Re: Python Iterables struggling using map() built-in Ian Kelly <ian.g.kelly@gmail.com> - 2014-12-11 15:00 -0700
                        Re: Python Iterables struggling using map() built-in Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-12-12 10:28 +1100
                          Re: Python Iterables struggling using map() built-in Ian Kelly <ian.g.kelly@gmail.com> - 2014-12-11 16:57 -0700
                            Re: Python Iterables struggling using map() built-in Roy Smith <roy@panix.com> - 2014-12-11 20:55 -0500
                              Re: Python Iterables struggling using map() built-in Ian Kelly <ian.g.kelly@gmail.com> - 2014-12-12 00:36 -0700
                          Re: Python Iterables struggling using map() built-in Chris Angelico <rosuav@gmail.com> - 2014-12-12 11:44 +1100
                  Re: Python Iterables struggling using map() built-in Terry Reedy <tjreedy@udel.edu> - 2014-12-10 12:48 -0500
                  Re: Python Iterables struggling using map() built-in Ian Kelly <ian.g.kelly@gmail.com> - 2014-12-10 13:32 -0700
                  Re: Python Iterables struggling using map() built-in Terry Reedy <tjreedy@udel.edu> - 2014-12-10 23:17 -0500
      Re: Python Iterables struggling using map() built-in Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-12-08 18:00 +1300

Page 1 of 2  [1] 2  Next page →

#82076 — Re: Python Iterables struggling using map() built-in

On Mon, Dec 8, 2014 at 5:27 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> On Dec 7, 2014 8:31 AM, "Ned Batchelder" <ned@nedbatchelder.com> wrote:
>> NOTE: THIS EXAMPLE IS HORRIBLE.  This code is crazy-confusing, and should
>> never have been used as an example of iteration. It layers at least three
>> iterations on top of each other, making it very difficult to see what is
>> going on.  It uses "while iters" where "while True" would do exactly the
>> same thing (iters will never be false).
>
> That's not quite correct; the "while iters" actually guards against the case
> where the passed args are empty. With no sub-iterators, no StopIteration
> would ever be raised, and the result would be an infinite generator of empty
> tuples. The "while iters" makes it return immediately instead.
>
> So it seems this example is even more confusing than you thought.

I'm actually glad PEP 479 will break this kind of code. Gives a good
excuse for rewriting it to be more readable.

ChrisA

[toc] | [next] | [standalone]

#82080

Chris Angelico wrote:

> On Mon, Dec 8, 2014 at 5:27 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
>> On Dec 7, 2014 8:31 AM, "Ned Batchelder" <ned@nedbatchelder.com> wrote:
>>> NOTE: THIS EXAMPLE IS HORRIBLE.  This code is crazy-confusing, and
>>> should never have been used as an example of iteration. It layers at
>>> least three iterations on top of each other, making it very difficult to
>>> see what is
>>> going on.  It uses "while iters" where "while True" would do exactly the
>>> same thing (iters will never be false).
>>
>> That's not quite correct; the "while iters" actually guards against the
>> case where the passed args are empty. With no sub-iterators, no
>> StopIteration would ever be raised, and the result would be an infinite
>> generator of empty tuples. The "while iters" makes it return immediately
>> instead.
>>
>> So it seems this example is even more confusing than you thought.
> 
> I'm actually glad PEP 479 will break this kind of code. Gives a good
> excuse for rewriting it to be more readable.

What kind of code is that? Short, simple, Pythonic and elegant? :-)

Here's the code again, with indentation fixed:


def myzip(*args):
    iters = map(iter, args)
    while iters:
        res = [next(i) for i in iters]
        yield tuple(res)


That is *beautiful code*. It's written for Python 2, where map returns a
list, so the "while iters" line is morally equivalent to:

    while iters != [] and True


It would be even more beautiful if we get rid of the unnecessary temporary
variable:

def myzip(*args):
    iters = map(iter, args)
    while iters:
        yield tuple([next(i) for i in iters])


I think this function makes a good test to separate the masters from the
apprentices. No offence intended to Ned, who is a master, anyone can have a
bad day or a blind spot. 

If you can read this function and instantly tell how it works, that it is
bug-free and duplicates the behaviour of the built-in zip(), you're
probably Raymond Hettinger. If you can tell what it does but you have to
think about it for a minute or two before you understand why it works, you
can call yourself a Python master. If you have to sit down with the
interactive interpreter and experiment for a bit to understand it, you're
doing pretty well.

I do not believe that good code must be obviously right. It's okay for code
to be subtly right. Either is better than complicated code which contains
no obvious bugs.

How would we re-write this to work in the future Python 3.7? Unless I have
missed something, I think we could write it like this:

def myzip37(*args):
    iters = list(map(iter, args))
    while iters:
        try:
            yield tuple([next(i) for i in iters])
        except StopIteration:
            return


which I guess is not too horrible. If Python had never supported the current
behaviour, I'd probably be happy with this. But having seen how elegant
generators *can* be, the post-PEP 479 version will always look bloated and
clumsy to me, like Garfield next to a cheetah.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#82082

On Mon, Dec 8, 2014 at 10:33 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> How would we re-write this to work in the future Python 3.7? Unless I have
> missed something, I think we could write it like this:
>
> def myzip37(*args):
>     iters = list(map(iter, args))
>     while iters:
>         try:
>             yield tuple([next(i) for i in iters])
>         except StopIteration:
>             return
>
> which I guess is not too horrible.

It's not horrible, and there are other ways it could be written too,
which also aren't horrible. Yes, it's not quite as short as the other
version; but more importantly, it's explicit about how StopIteration
affects it. It's clear that this exception, if raised by _any_ of the
iterators (even after consuming values from some of them, perhaps),
will silently terminate the generator.

The current behaviour favours a handful of cases like this, although
personally I think the termination of zip() is simply "this is what
happens if we have no code here, so let's document it" rather than
being something inherently ideal; the new behaviour favours the
debugging of many obscure cases and some less-obscure ones as well.
Most importantly, if you run the old version of myzip on Python 3.7,
you'll get an immediate and noisy RuntimeError when it terminates, and
you'll know exactly where to go fix stuff; if you run a buggy
generator on Python 3.4, you simply see no more results, without any
explanation of why. I'd rather debug the RuntimeError that can be
easily and trivially fixed in 99% of cases.

ChrisA

[toc] | [prev] | [next] | [standalone]

#82085

In article <mailman.16689.1417996247.18130.python-list@python.org>,
 Chris Angelico <rosuav@gmail.com> wrote:

> On Mon, Dec 8, 2014 at 10:33 AM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
> > How would we re-write this to work in the future Python 3.7? Unless I have
> > missed something, I think we could write it like this:
> >
> > def myzip37(*args):
> >     iters = list(map(iter, args))
> >     while iters:
> >         try:
> >             yield tuple([next(i) for i in iters])
> >         except StopIteration:
> >             return

I'm still not liking this use of while.  Yes, of course, it handles the 
special case of no arguments, but I'd be in-your-face about that (not 
tested):

def myzip37(*args):
    iters = list(map(iter, args))
    if not iters:
        return None
    while True:
        try:
            yield tuple([next(i) for i in iters])
        except StopIteration:
            return

This makes it really obvious that there's something going on inside the 
loop other than exhausting the control variable to cause it to exit.

Although, to be honest, I'm wondering if this is more straight-forward 
(also not tested):

def myzip37(*args):
    if not args:
        return
    iters = list(map(iter, args))
    while True:
        try:
            yield tuple(map(next, iters))
        except StopIteration:
            return

[toc] | [prev] | [next] | [standalone]

#82087

On Mon, Dec 8, 2014 at 11:27 AM, Roy Smith <roy@panix.com> wrote:
> Although, to be honest, I'm wondering if this is more straight-forward
> (also not tested):
>
> def myzip37(*args):
>     if not args:
>         return
>     iters = list(map(iter, args))

Yes, I prefer this too. It's explicit and clear that passing no
arguments will yield no values.

ChrisA

[toc] | [prev] | [next] | [standalone]

#82117

On Mon, 08 Dec 2014 11:35:36 +1100, Chris Angelico wrote:

> On Mon, Dec 8, 2014 at 11:27 AM, Roy Smith <roy@panix.com> wrote:
>> Although, to be honest, I'm wondering if this is more straight-forward
>> (also not tested):
>>
>> def myzip37(*args):
>>     if not args:
>>         return
>>     iters = list(map(iter, args))
> 
> Yes, I prefer this too. It's explicit and clear that passing no
> arguments will yield no values.

The first version is explicit and clear too. I'm sorry to say this, but 
it is true: if you (generic you) don't recognise that

    while iters:
        ...


skips the while block if iters is an empty list, then *you* have a 
problem, not the code. You're simply not fluent with the language. (Or 
you have a strange blind-spot, in which case you have my sympathy but 
that's your problem to deal with, not mine.) This is not an obscure 
corner of some rarely-used library that has a strange and weird API, it 
is a fundamental part of Python.

I *guarantee* that there are people who will bitch and moan about your 
example too, and insist that the only "right" way to write it is to be 
"explicit" that an empty tuple is falsey:

    if not args == ():
        return

and there will be some who are convinced that operator precedence is too 
"implicit":

    if not (args == ()):   # args != () is okay too
        return


Dumbing down code is an anti-pattern, because there's always somebody 
just a little less fluent in the language who will complain that it isn't 
dumbed down enough.

Sometimes we have this meme that if you can't take in code at a glance 
and instantly understand it, it's bad code. But that meme is untrue, and 
we all know that it is untrue: real world code is often hard to 
understand because it can't be any simpler. Try understanding the code 
for a web server, or code that calculates the square root of a number, or 
code for importing a module.

Occasionally, very occasionally, we have a combination of programming 
language and algorithm which combines in such a way that you can actually 
implement a useful algorithm in a simple, elegant, minimalist way. As 
programmers, we all know how fecking rare this is: we start with an 
elegant five line function, and by the time we cover all the corners and 
fix the bugs it's twenty lines and you can't tell what it does any more 
without studying it for ten minutes. zip() is an exception, you can write 
zip() in Python beautifully. It breaks my heart that there are people who 
think that it is improved by adding unnecessary guard clauses to it.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#82118

On Mon, Dec 8, 2014 at 8:40 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> The first version is explicit and clear too. I'm sorry to say this, but
> it is true: if you (generic you) don't recognise that
>
>     while iters:
>         ...
>
>
> skips the while block if iters is an empty list, then *you* have a
> problem, not the code.

Of course it skips the body if iters starts out empty. The argument is
whether or not it makes sense to use this to mean "if iters: while
True:" because iters will never be changed. Is it abusing syntax or a
valid way to spell that condition?

ChrisA

[toc] | [prev] | [next] | [standalone]

#82162

In article <5485721c$0$2817$c3e8da3$76491128@news.astraweb.com>,
 Steven D'Aprano <steve@pearwood.info> wrote:

> On Mon, 08 Dec 2014 11:35:36 +1100, Chris Angelico wrote:
> 
> > On Mon, Dec 8, 2014 at 11:27 AM, Roy Smith <roy@panix.com> wrote:
> >> Although, to be honest, I'm wondering if this is more straight-forward
> >> (also not tested):
> >>
> >> def myzip37(*args):
> >>     if not args:
> >>         return
> >>     iters = list(map(iter, args))
> > 
> > Yes, I prefer this too. It's explicit and clear that passing no
> > arguments will yield no values.
> 
> The first version is explicit and clear too. I'm sorry to say this, but 
> it is true: if you (generic you) don't recognise that
> 
>     while iters:
>         ...
> 
> 
> skips the while block if iters is an empty list, then *you* have a 
> problem, not the code.

The problem is not that the while body is skipped if iters is falsey.  
Thats quite clear.

The problem is that the looping termination isn't actually controlled by 
the control variable being exhausted.  It's controlled by an exception 
getting thrown.  I'm OK with breaking out of a loop by throwing an 
exception, but if that's what you're going to do, then make it clear 
that's the case by doing "while 1" or "while True".  Those screams out, 
"Hey, look at me, I'm an infinite loop", which is your clue that there's 
something else going on.

[toc] | [prev] | [next] | [standalone]

#82084

Chris Angelico wrote:
> > I'm actually glad PEP 479 will break this kind of code. Gives a good
> > excuse for rewriting it to be more readable.

Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> What kind of code is that? Short, simple, Pythonic and elegant? :-)
> 
> Here's the code again, with indentation fixed:
> 
> 
> def myzip(*args):
>     iters = map(iter, args)
>     while iters:
>         res = [next(i) for i in iters]
>         yield tuple(res)

Ugh.  When I see "while foo", my brain says, "OK, you're about to see a 
loop which is controlled by the value of foo being changed inside the 
loop".  That's not at all what's happening here, so my brain runs into a 
wall.

Next problem, what the heck is "res"?  We're not back in the punch-card 
days.  We don't have to abbreviate variable names to save columns.  
Variable names are supposed to describe what they hold, and thus help 
you understand the code.  I have no idea what "res" is supposed to be.  
Residue?  Result?  Rest_of_items?  Response?  None of these make much 
sense here, so I'm just left befuddled.

> It would be even more beautiful if we get rid of the unnecessary temporary
> variable:
> 
> def myzip(*args):
>     iters = map(iter, args)
>     while iters:
>         yield tuple([next(i) for i in iters])

Well, that's one way to solve the mystery of what "res" means, but it 
doesn't actually make it easier to understand.

> I think this function makes a good test to separate the masters from the
> apprentices.

The goal of good code is NOT to separate the masters from the 
apprentices.  The goal of good code is to be correct and easy to 
understand by the next guy who comes along to maintain it.
 
> If you can read this function and instantly tell how it works, that it is
> bug-free and duplicates the behaviour of the built-in zip(), you're
> probably Raymond Hettinger. If you can tell what it does but you have to
> think about it for a minute or two before you understand why it works, you
> can call yourself a Python master. If you have to sit down with the
> interactive interpreter and experiment for a bit to understand it, you're
> doing pretty well.

That pretty much is the point I'm trying to make.  If the code is so 
complicated that masters can only understand it after a couple of 
minutes of thought, and those of us who are just "doing pretty well" 
need to sit down and puzzle it out in the REPL, then it's too 
complicated for most people to understand.  KISS beats elegant.

[toc] | [prev] | [next] | [standalone]

#82086

On Mon, Dec 8, 2014 at 11:12 AM, Roy Smith <roy@panix.com> wrote:
> Ugh.  When I see "while foo", my brain says, "OK, you're about to see a
> loop which is controlled by the value of foo being changed inside the
> loop".  That's not at all what's happening here, so my brain runs into a
> wall.

I agree, with the caveat that this kind of thing makes a fine infinite loop:

while "No exception raised":
    # do stuff that can raise an exception
while "Playing more games":
    play_game()
    if not still_playing: break
    reset_game_board()

Nobody expects a string literal to actually become false inside the
loop. With a local name, yes, I would expect it to at least have a
chance of becoming false.

> Next problem, what the heck is "res"?  We're not back in the punch-card
> days.  We don't have to abbreviate variable names to save columns.
> Variable names are supposed to describe what they hold, and thus help
> you understand the code.  I have no idea what "res" is supposed to be.
> Residue?  Result?  Rest_of_items?  Response?  None of these make much
> sense here, so I'm just left befuddled.

I take it as "result", which makes plenty of sense to me. It's the
thing that's about to be yielded. Given that there's not much else you
can say in a meta-function like zip(), I have no problem with that.
Here's a slightly different example:

def mark_last(it):
    it = iter(it)
    lastres = sentinel = object()
    while "more values coming":
        res = next(it, sentinel)
        if lastres is not sentinel: yield (lastres, res is sentinel)
        if res is sentinel: return
        lastres = res

Use of "res" for "result" and "lastres" to mean "res as of the
previous iteration of the loop" seems fine to me.

ChrisA

[toc] | [prev] | [next] | [standalone]

#82089

In article <mailman.16690.1417998873.18130.python-list@python.org>,
 Chris Angelico <rosuav@gmail.com> wrote:

> > Next problem, what the heck is "res"?  We're not back in the punch-card
> > days.  We don't have to abbreviate variable names to save columns.
> > Variable names are supposed to describe what they hold, and thus help
> > you understand the code.  I have no idea what "res" is supposed to be.
> > Residue?  Result?  Rest_of_items?  Response?  None of these make much
> > sense here, so I'm just left befuddled.
> 
> I take it as "result", which makes plenty of sense to me.

OK, so spell it out.  Three more keystrokes (well, plus another three 
when you use it on the next line).  And one of them is a vowel; they 
don't even cost much.  The next guy who has to read your code will thank 
you for it.

[toc] | [prev] | [next] | [standalone]

#82090

On Mon, Dec 8, 2014 at 11:45 AM, Roy Smith <roy@panix.com> wrote:
>> I take it as "result", which makes plenty of sense to me.
>
> OK, so spell it out.  Three more keystrokes (well, plus another three
> when you use it on the next line).  And one of them is a vowel; they
> don't even cost much.  The next guy who has to read your code will thank
> you for it.

Maybe. Personally, I don't mind the odd abbreviation; they keep the
code small enough to eyeball, rather than spelling everything out
everywhere. Using "cur" (or "curr") for current, "next" for next,
"prev" for previous, as prefixes to a short word saying *what* they're
the current/next/previous of, is sufficiently obvious IMO to justify
the repeated use of the abbreviation. Why does Python have "int" and
"str" rather than "integer" and "string"? Or, worse,
"arbitrary_precision_integer" and "unicode_codepoint_string"? Common
words get shortened - it's a legit form of Huffman compression.

ChrisA

[toc] | [prev] | [next] | [standalone]

#82096

On 2014-12-08 01:00, Chris Angelico wrote:
> On Mon, Dec 8, 2014 at 11:45 AM, Roy Smith <roy@panix.com> wrote:
>>> I take it as "result", which makes plenty of sense to me.
>>
>> OK, so spell it out.  Three more keystrokes (well, plus another three
>> when you use it on the next line).  And one of them is a vowel; they
>> don't even cost much.  The next guy who has to read your code will thank
>> you for it.
>
> Maybe. Personally, I don't mind the odd abbreviation; they keep the
> code small enough to eyeball, rather than spelling everything out
> everywhere. Using "cur" (or "curr") for current, "next" for next,
> "prev" for previous, as prefixes to a short word saying *what* they're
> the current/next/previous of, is sufficiently obvious IMO to justify
> the repeated use of the abbreviation. Why does Python have "int" and
> "str" rather than "integer" and "string"? Or, worse,
> "arbitrary_precision_integer" and "unicode_codepoint_string"? Common
> words get shortened - it's a legit form of Huffman compression.
>
Not to mention "len", "def", "iter", etc.

[toc] | [prev] | [next] | [standalone]

#82088

On 12/7/2014 7:12 PM, Roy Smith wrote:
> Chris Angelico wrote:
>>> I'm actually glad PEP 479 will break this kind of code. Gives a good
>>> excuse for rewriting it to be more readable.
>
> Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
>> What kind of code is that? Short, simple, Pythonic and elegant? :-)
>>
>> Here's the code again, with indentation fixed:
>>
>>
>> def myzip(*args):
>>      iters = map(iter, args)
>>      while iters:
>>          res = [next(i) for i in iters]
>>          yield tuple(res)
>
> Ugh.  When I see "while foo", my brain says, "OK, you're about to see a
> loop which is controlled by the value of foo being changed inside the
> loop".  That's not at all what's happening here, so my brain runs into a
> wall.

I agree.  Too tricky.  The code should have been

def myzip(*args):
     if args:
         iters = map(iter, args)
         while True:
             res = [next(i) for i in iters]
             yield tuple(res)

However, this 'beautiful' code has a trap.  If one gets rid of the 
seemingly unneeded temporary list res by telescoping the last two lines 
into a bit too much into

             yield tuple(next(i) for i in iters)

we now have an infinite generator, because tuple() swallows the 
StopIteration raised as a side-effect of next calls.

def myzip(*args):
     if args:
         iters = map(iter, args)
         while True:
             try:
                 result = [next(i) for i in iters]
             except StopIteration
                 return
             yield tuple(res)

makes the side-effect dependence of stopping clearer.  Putting
                  yield tuple([next(i) for i in iters])
in the try would also work.


-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#82106

Terry Reedy wrote:
> However, this 'beautiful' code has a trap.  If one gets rid of the 
> seemingly unneeded temporary list res by telescoping the last two lines 
> into a bit too much into
> 
>             yield tuple(next(i) for i in iters)
> 
> we now have an infinite generator, because tuple() swallows the 
> StopIteration raised as a side-effect of next calls.

An excellent example of the kind of thing that PEP 479
is designed to catch! There couldn't be a better
advertisement for it. :-)

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#82091

On 12/7/14 7:12 PM, Roy Smith wrote:
> Chris Angelico wrote:
>>> I'm actually glad PEP 479 will break this kind of code. Gives a good
>>> excuse for rewriting it to be more readable.
>
> Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
>> What kind of code is that? Short, simple, Pythonic and elegant? :-)
>>
>> Here's the code again, with indentation fixed:
>>
>>
>> def myzip(*args):
>>      iters = map(iter, args)
>>      while iters:
>>          res = [next(i) for i in iters]
>>          yield tuple(res)
>
> Ugh.  When I see "while foo", my brain says, "OK, you're about to see a
> loop which is controlled by the value of foo being changed inside the
> loop".  That's not at all what's happening here, so my brain runs into a
> wall.
>
> Next problem, what the heck is "res"?  We're not back in the punch-card
> days.  We don't have to abbreviate variable names to save columns.
> Variable names are supposed to describe what they hold, and thus help
> you understand the code.  I have no idea what "res" is supposed to be.
> Residue?  Result?  Rest_of_items?  Response?  None of these make much
> sense here, so I'm just left befuddled.
>
>> It would be even more beautiful if we get rid of the unnecessary temporary
>> variable:
>>
>> def myzip(*args):
>>      iters = map(iter, args)
>>      while iters:
>>          yield tuple([next(i) for i in iters])
>
> Well, that's one way to solve the mystery of what "res" means, but it
> doesn't actually make it easier to understand.
>
>> I think this function makes a good test to separate the masters from the
>> apprentices.
>
> The goal of good code is NOT to separate the masters from the
> apprentices.  The goal of good code is to be correct and easy to
> understand by the next guy who comes along to maintain it.
>
>> If you can read this function and instantly tell how it works, that it is
>> bug-free and duplicates the behaviour of the built-in zip(), you're
>> probably Raymond Hettinger. If you can tell what it does but you have to
>> think about it for a minute or two before you understand why it works, you
>> can call yourself a Python master. If you have to sit down with the
>> interactive interpreter and experiment for a bit to understand it, you're
>> doing pretty well.
>
> That pretty much is the point I'm trying to make.  If the code is so
> complicated that masters can only understand it after a couple of
> minutes of thought, and those of us who are just "doing pretty well"
> need to sit down and puzzle it out in the REPL, then it's too
> complicated for most people to understand.  KISS beats elegant.
>

Now that I understand all the intricacies (thanks everyone!), this is 
how I would write it:

     def zip(*args):
         if not args:
             return
         iters = list(map(iter, args))
         while True:
             try:
                 result = [next(it) for it in iters]
             except StopIteration:
                 return
             yield tuple(result)

The implicit use of StopIteration to end the entire generator is far too 
implicit for my taste.  This code expresses the intent much better.

And good call on not being able to use:

     tuple(next(it) for it in iters)

Again, the tricky implicit hidden StopIterations are confusing.

One last tweak: why do we use map to make iters, but a list 
comprehension to make result?  OK, let's try this:

     result = map(next, iters)

Oops, another infinite loop on Py3, right, because map is lazy, so the 
StopIteration doesn't happen until the tuple() call.  OK, try this:

     result = list(map(next, iters))

Nope, still infinite because now list() consumes the StopIteration 
again. Ugh.

-- 
Ned Batchelder, http://nedbatchelder.com

[toc] | [prev] | [next] | [standalone]

#82173

Roy Smith wrote:

> Chris Angelico wrote:
>> > I'm actually glad PEP 479 will break this kind of code. Gives a good
>> > excuse for rewriting it to be more readable.
> 
> Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
>> What kind of code is that? Short, simple, Pythonic and elegant? :-)
>> 
>> Here's the code again, with indentation fixed:
>> 
>> 
>> def myzip(*args):
>>     iters = map(iter, args)
>>     while iters:
>>         res = [next(i) for i in iters]
>>         yield tuple(res)
> 
> Ugh.  When I see "while foo", my brain says, "OK, you're about to see a
> loop which is controlled by the value of foo being changed inside the
> loop".

Yes. Me too. 99% of the time when you see "while foo", that's what you'll
get, so it's the safe assumption. But it's only an assumption, not a
requirement. When you read a bit more of the code and see that iters isn't
being modified, your reaction ought to be closer "oh wow, that's neat"
than "oh noes it's different from what I expected".

"while foo" is logically equivalent to "if foo: while foo:". The if is
completely redundant.


> That's not at all what's happening here, so my brain runs into a 
> wall.

I hope you are exaggerating for effect, because if you genuinely mean that
reading that code causes major mental trauma (perhaps the equivalent of a
mental BSOD) then you've probably picked the wrong industry to be working
in. Imagine how you would cope reading genuinely obfuscated code. You would
probably have a nervous breakdown :-)

It's okay to read code which forces you to reevaluate your initial
assumption about the code. People, especially (allegedly) smart people like
programmers, are intelligent and flexible. If you can't do that, you're
going to hate Python:

- Python has no "repeat N times" loop, we have to use "for i in range(...)"
instead, so seeing a for-loop doesn't necessarily mean that the loop
variable will be used. Sometimes it isn't.


- I cannot count the number of times I've read, or written, a method that
doesn't use "self", but doesn't bother to declare it as a staticmethod.


- The official way to get a single arbitrary value from a set without
removing it is:

    for value in the_set:
        return value

GvR recently gave an example of how to process a single element in a
possibly-empty iterator:

    for x in it:
        print(x)
        break
    else:
        print('nothing')


so there are two examples of using a for-loop to *not* loop over something.


- Ducktyping. Just because some code is using a goose, doesn't mean that a
goose is required. Perhaps a duck is required but a goose is close enough.


> Next problem, what the heck is "res"?  We're not back in the punch-card
> days.  We don't have to abbreviate variable names to save columns.

*shrug* I didn't pick the name. But "res" is a standard abbreviation
for "result" or "resource", and from context it clearly should be "result".


[...]
>> I think this function makes a good test to separate the masters from the
>> apprentices.
> 
> The goal of good code is NOT to separate the masters from the
> apprentices.  The goal of good code is to be correct 

So far I agree.


> and easy to 
> understand by the next guy who comes along to maintain it.

No. That is *one secondary goal*. Efficiency is another secondary goal.
Sometimes education is even more important, and in this specific case the
function is being used to teach people.

Another secondary goal ought to be beauty and elegance over ugliness. It's
not often a beautiful function actually is good enough for production use.
It's usually surrounded by an inelegant if not downright ugly pile of code
testing arguments, checking for error conditions, handling corner cases,
etc. That ugliness can obfuscate the underlying algorithm and make the
function harder to understand. If it isn't *necessary*, take it out.

    "Perfection is achieved, not when there is nothing more to add, 
    but when there is nothing left to take away."

While ease of maintenance is an important goal, think about what we are
discussing. zip() is a primitive function. Once you have decided on the
public API, it will probably never need any maintenance. It's not like the
requirements will change -- "yeah, we used to want to zip the items
together, but now we need to add a 9% superannuation surcharge to them
first".

The beauty of this code is that it is so simple that it cannot fail to be
bug-free. (Am I wrong? Have I missed a corner-case?) There's no unnecessary
code in the function that can hide bugs and obfuscate what it does.

And it is so simple that it's hard to see anything written in pure Python
being more efficient. So as far as maintenance goes, that's irrelevant.
That function is simply finished, done, complete. Anything you do to
improve it can only make it slower, buggier, or uglier.

(And that's a rare and beautiful thing in code.)


>> If you can read this function and instantly tell how it works, that it is
>> bug-free and duplicates the behaviour of the built-in zip(), you're
>> probably Raymond Hettinger. If you can tell what it does but you have to
>> think about it for a minute or two before you understand why it works,
>> you can call yourself a Python master. If you have to sit down with the
>> interactive interpreter and experiment for a bit to understand it, you're
>> doing pretty well.
> 
> That pretty much is the point I'm trying to make.  If the code is so
> complicated that masters can only understand it after a couple of
> minutes of thought, 

No, you have misunderstood me. Yet again, it seems, my use of language is
too subtle and I fail to get my point across :-(

If you can read that function and immediately understand that it zips up
items, then you have achieved mastery over Python. You are no longer a
beginner who has to puzzle over questions like "what does [next(i) for i in
iters] do?". That's the first step.

The second step is to tell that the code is correct. It doesn't just zip up
items, but it handles the corner cases correctly: what if there are no
arguments at all? What if the arguments are of different sizes? What if
they're all infinite iterators? If some of the arguments are not iterable,
will it raise correctly? It's okay if you need to think for a minute or two
to convince yourself that this is the case. You have still achieved mastery
over them.


> and those of us who are just "doing pretty well" 
> need to sit down and puzzle it out in the REPL, then it's too
> complicated for most people to understand.  KISS beats elegant.

I love the REPL as much, if not more, than most programmers. It pains me to
think about using any language without one. But I don't believe for a
second that you *needed* to use the REPL to understand this function. You
might have chosen to do so, "just to be sure", but I'm confident that if I
had asked you to analyse the code just in your head you could have done so
correctly.

That's what I did, and I don't think I'm a better Python programmer than
you.

And for what it's worth, I too tried it in the REPL. Why? Not because I
couldn't understand the function, but because I feared I may have missed
some odd corner case. I'm pretty sure I have got them all: no arguments, a
single argument, different sized arguments. That's part of the "minute or
two" to be sure that the function is correct.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#82175

On 12/8/2014 9:50 PM, Steven D'Aprano wrote:
> Roy Smith wrote:
>
>> Chris Angelico wrote:

>>> def myzip(*args):
>>>      iters = map(iter, args)
>>>      while iters:
>>>          res = [next(i) for i in iters]
>>>          yield tuple(res)
>>
>> Ugh.  When I see "while foo", my brain says, "OK, you're about to see a
>> loop which is controlled by the value of foo being changed inside the
>> loop".
>
> Yes. Me too. 99% of the time when you see "while foo", that's what you'll
> get, so it's the safe assumption. But it's only an assumption, not a
> requirement. When you read a bit more of the code and see that iters isn't
> being modified, your reaction ought to be closer "oh wow, that's neat"

To me it is a code smell.  iters is empty if and only if args is empty. 
  If args is empty, iters should not be created.

if args:
   iters = ...
   while True
     ... (return on exception)

makes the logic clear.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#82180

On Tue, 09 Dec 2014 00:03:33 -0500, Terry Reedy wrote:

> On 12/8/2014 9:50 PM, Steven D'Aprano wrote:
>> Roy Smith wrote:
>>
>>> Chris Angelico wrote:
> 
>>>> def myzip(*args):
>>>>      iters = map(iter, args)
>>>>      while iters:
>>>>          res = [next(i) for i in iters]
>>>>          yield tuple(res)
>>>
>>> Ugh.  When I see "while foo", my brain says, "OK, you're about to see
>>> a loop which is controlled by the value of foo being changed inside
>>> the loop".
>>
>> Yes. Me too. 99% of the time when you see "while foo", that's what
>> you'll get, so it's the safe assumption. But it's only an assumption,
>> not a requirement. When you read a bit more of the code and see that
>> iters isn't being modified, your reaction ought to be closer "oh wow,
>> that's neat"
> 
> To me it is a code smell.  iters is empty if and only if args is empty.
>   If args is empty, iters should not be created.
> 
> if args:
>    iters = ...
>    while True
>      ... (return on exception)
> 
> makes the logic clear.

When you understand why this body of code is inelegant and ugly, you 
should understand why the above is inelegant and ugly:

def double(x):
    if x != 0:
        x *= 2
    return x




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#82235

On 12/9/2014 12:03 AM, Terry Reedy wrote:
>> Roy Smith wrote:
>>
>>> Chris Angelico wrote:
>
>>>> def myzip(*args):
>>>>      iters = map(iter, args)
>>>>      while iters:
>>>>          res = [next(i) for i in iters]
>>>>          yield tuple(res)
>>>
>>> Ugh.  When I see "while foo", my brain says, "OK, you're about to see a
>>> loop which is controlled by the value of foo being changed inside the
>>> loop".

What is nasty to me is that to understand the loop, one must do a whole 
program analysis to determine both that 'iters' is not rebound and that 
the list it is bound to is not mutated.  To do the later, one must not 
only read the loop body, but also preceding code to make sure the list 
is not aliased.

 > iters is empty if and only if args is empty.
> If args is empty, iters should not be created.
>
> if args:
>    iters = ...
>    while True
>      ... (return on exception)
>
> makes the logic clear.

Once the logic is clear and 'localized', even a simple compiler like 
CPython's can see that this is a loop-forever construct and that the 
loop test is unnecessary.  So it can be removed.

 >>> dis("while a: b+=1")
   1           0 SETUP_LOOP              20 (to 23)
         >>    3 LOAD_NAME                0 (a)
               6 POP_JUMP_IF_FALSE       22
               9 LOAD_NAME                1 (b)
              12 LOAD_CONST               0 (1)
              15 INPLACE_ADD
              16 STORE_NAME               1 (b)
              19 JUMP_ABSOLUTE            3
         >>   22 POP_BLOCK
         >>   23 LOAD_CONST               1 (None)
              26 RETURN_VALUE

 >>> dis("while True: b+=1")
   1           0 SETUP_LOOP              13 (to 16)
         >>    3 LOAD_NAME                0 (b)
               6 LOAD_CONST               0 (1)
               9 INPLACE_ADD
              10 STORE_NAME               0 (b)
              13 JUMP_ABSOLUTE            3
         >>   16 LOAD_CONST               1 (None)
              19 RETURN_VALUE

'while 1' and 'while "exception not raised"' are similarly condensed. 
This leaves only the initial test of the argument.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web