Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #15438 > unrolled thread

all() is slow?

Back to article view | Back to comp.lang.python

Contents

  all() is slow? "OKB (not okblacke)" <brenNOSPAMbarn@NObrenSPAMbarn.net> - 2011-11-07 21:00 +0000
    Re: all() is slow? Chris Rebert <clp2@rebertia.com> - 2011-11-07 13:39 -0800
    Re: all() is slow? david vierra <codewarrior0@gmail.com> - 2011-11-07 13:46 -0800
      Re: all() is slow? Chris Angelico <rosuav@gmail.com> - 2011-11-08 09:06 +1100
        Re: all() is slow? Henrik Faber <hfaber@invalid.net> - 2011-11-08 13:09 +0100
          Re: all() is slow? Chris Angelico <rosuav@gmail.com> - 2011-11-08 23:14 +1100
          Re: all() is slow? John Posner <jjposner@optimum.net> - 2011-11-08 17:51 -0500
          Re: all() is slow? Tim Chase <python.list@tim.thechases.com> - 2011-11-08 17:56 -0600
      Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-08 19:44 -0500
        Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-09 02:47 +0000
          Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-09 18:01 -0500
            Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-09 23:11 +0000
              Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-09 20:26 -0500
                Re: all() is slow? alex23 <wuwei23@gmail.com> - 2011-11-09 19:50 -0800
                  Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-09 23:40 -0500
                  Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-10 07:35 +0000
                Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-10 07:48 +0000
                  Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-10 03:51 -0500
                  Re: all() is slow? gene heskett <gheskett@wdtv.com> - 2011-11-10 08:20 -0500
                  Re: all() is slow? Terry Reedy <tjreedy@udel.edu> - 2011-11-10 14:25 -0500
              Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-09 20:35 -0500
              Re: all() is slow? Ian Kelly <ian.g.kelly@gmail.com> - 2011-11-09 19:10 -0700
              Re: all() is slow? Ethan Furman <ethan@stoneleaf.us> - 2011-11-10 10:43 -0800
              Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-10 15:37 -0500
                Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-10 23:07 +0000
                  Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-10 23:35 -0500
                    Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-12 01:18 +0000
                      Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-13 01:28 -0500
                        Re: all() is slow? alex23 <wuwei23@gmail.com> - 2011-11-13 18:50 -0800
                          Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-13 23:48 -0500
                Re: all() is slow? "OKB (not okblacke)" <brenNOSPAMbarn@NObrenSPAMbarn.net> - 2011-11-11 05:40 +0000
              Re: all() is slow? Ian Kelly <ian.g.kelly@gmail.com> - 2011-11-10 14:19 -0700
                Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-10 22:56 +0000
              Re: all() is slow? Ethan Furman <ethan@stoneleaf.us> - 2011-11-10 13:47 -0800
          Re: all() is slow? Chris Angelico <rosuav@gmail.com> - 2011-11-10 10:15 +1100
      Re: all() is slow? Chris Angelico <rosuav@gmail.com> - 2011-11-09 12:19 +1100
      Re: all() is slow? Chris Rebert <clp2@rebertia.com> - 2011-11-08 17:30 -0800
        Re: all() is slow? Hans Mulder <hansmu@xs4all.nl> - 2011-11-09 16:41 +0100
          Re: all() is slow? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2011-11-09 09:07 -0800
    Re: all() is slow? John Nagle <nagle@animats.com> - 2011-11-09 14:16 -0800
      Re: all() is slow? alex23 <wuwei23@gmail.com> - 2011-11-09 19:52 -0800
      Re: all() is slow? "OKB (not okblacke)" <brenNOSPAMbarn@NObrenSPAMbarn.net> - 2011-11-10 19:35 +0000
    Re: all() is slow? BOOK-AZ <amalguseynov@gmail.com> - 2011-11-15 08:02 -0800

Page 1 of 3  [1] 2 3  Next page →

#15438 — all() is slow?

    	I noticed this (Python 2.6.5 on Windows XP):

>>> import random, timeit
>>> def myAll(x):
...     for a in x:
...         if a not in (True, False):
...             return False
...     return True
>>> x = [random.choice([True, False]) for a in xrange(0, 5000000)]
>>> timeit.timeit('myAll(x)', 'from __main__ import myAll, x', 
number=10)
0: 9.7685158309226452
>>> timeit.timeit('all(a in (True, False) for a in x)', 'from __main__ 
import x', number=10)
1: 12.348196768024984
>>> x = [random.randint(0,100) for a in xrange(0, 5000000)]
>>> def myAll(x):
...     for a in x:
...         if not a <= 100:
...             return False
...     return True
>>> timeit.timeit('myAll(x)', 'from __main__ import myAll, x', 
number=10)
4: 2.8248207523582209
>>> timeit.timeit('all(a <= 100 for a in x)', 'gc.enable(); from 
__main__ import x', number=10)
5: 4.6433557896324942

    	What is the point of the all() function being a builtin if it's 
slower than writing a function to do the check myself?

-- 
--OKB (not okblacke)
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is
no path, and leave a trail."
	--author unknown

[toc] | [next] | [standalone]

#15440

On Mon, Nov 7, 2011 at 1:00 PM, OKB (not okblacke)
<brenNOSPAMbarn@nobrenspambarn.net> wrote:
<snip>
>        What is the point of the all() function being a builtin if it's
> slower than writing a function to do the check myself?

Regardless of whether it's slower (which I expect someone will be
along to debunk or explain shortly), do you really want to have to
write an additional boilerplate function or block of code /every
single time/ you want to do such a check? The runtime speed difference
is unlikely to be worth your time as a developer in many cases. And by
Murphy's Law, you *will* make errors writing these repetitive code
blocks (e.g. forget to negate the conditional), whereas reusing all()
makes that much less likely.

The trade-off is run-time speed for developer
productivity/convenience; Python tends to lean towards the latter (to
varying degrees).

Cheers,
Chris

[toc] | [prev] | [next] | [standalone]

#15442

On Nov 7, 11:00 am, "OKB (not okblacke)"
<brenNOSPAMb...@NObrenSPAMbarn.net> wrote:

>         What is the point of the all() function being a builtin if it's
> slower than writing a function to do the check myself?
>

But, you didn't write an all() function.  You wrote a more specialized
allBoolean() function. I think this comparison is more fair to the
builtin all():

>>> def myAll(x):
...     for a in x:
...         if not a: return False
...     return True
...
>>> timeit.timeit('myAll(a in (True, False) for a in x)', 'from __main__ import myAll, x', number=10)
14.510986388627998
>>> timeit.timeit('all(a in (True, False) for a in x)', 'from __main__ import x', number=10)
12.209779342432576

[toc] | [prev] | [next] | [standalone]

#15443

On Tue, Nov 8, 2011 at 8:46 AM, david vierra <codewarrior0@gmail.com> wrote:
> But, you didn't write an all() function.  You wrote a more specialized
> allBoolean() function. I think this comparison is more fair to the
> builtin all():

So really, it's not "all() is slow" but "function calls are slow".
Maybe it'd be worthwhile making an all-factory:

def my_all(code,lst):
    exec("""def tmp_all(x):
        for a in x:
            if not ("""+code+"""): return False
        return True
""")
    return tmp_all(lst)
timeit.timeit('my_all("a in (True, False)",x)','from __main__ import
my_all,x',number=10)

Bad code imho, but it _is_ faster than both the original and the builtin.

ChrisA

[toc] | [prev] | [next] | [standalone]

#15464

On 07.11.2011 23:06, Chris Angelico wrote:
> On Tue, Nov 8, 2011 at 8:46 AM, david vierra <codewarrior0@gmail.com> wrote:
>> But, you didn't write an all() function.  You wrote a more specialized
>> allBoolean() function. I think this comparison is more fair to the
>> builtin all():
> 
> So really, it's not "all() is slow" but "function calls are slow".
> Maybe it'd be worthwhile making an all-factory:

PLEASE say you're joking. If I saw code like that on any of our project,
this would definitely qualify for a DailyWTF.

Regards, Henrik

[toc] | [prev] | [next] | [standalone]

#15465

On Tue, Nov 8, 2011 at 11:09 PM, Henrik Faber <hfaber@invalid.net> wrote:
> On 07.11.2011 23:06, Chris Angelico wrote:
>> So really, it's not "all() is slow" but "function calls are slow".
>> Maybe it'd be worthwhile making an all-factory:
>
> PLEASE say you're joking. If I saw code like that on any of our project,
> this would definitely qualify for a DailyWTF.

For the benefit of anyone who was actually in doubt: YES, I was
joking. Do *not* put code like this into any production project.

But it's still fun to write bad code once in a while. It's like
Mythbusters getting to crash cars. Fun partly _because_ it's something
you normally don't want to do.

ChrisA

[toc] | [prev] | [next] | [standalone]

#15484

On 2:59 PM, Chris Angelico wrote:

>>> So really, it's not "all() is slow" but "function calls are slow".
>>> Maybe it'd be worthwhile making an all-factory:
>> PLEASE say you're joking. If I saw code like that on any of our project,
>> this would definitely qualify for a DailyWTF.
> For the benefit of anyone who was actually in doubt: YES, I was
> joking. Do *not* put code like this into any production project.

Because an all-factory would produce code smell? :-)

[toc] | [prev] | [next] | [standalone]

#15486

On 11/08/2011 04:51 PM, John Posner wrote:
> On 2:59 PM, Chris Angelico wrote:
>
>>>> So really, it's not "all() is slow" but "function calls are slow".
>>>> Maybe it'd be worthwhile making an all-factory:
>>> PLEASE say you're joking. If I saw code like that on any of our project,
>>> this would definitely qualify for a DailyWTF.
>> For the benefit of anyone who was actually in doubt: YES, I was
>> joking. Do *not* put code like this into any production project.
>
> Because an all-factory would produce code smell? :-)

Groan...that word-play is the pits! :)

-tkc

[toc] | [prev] | [next] | [standalone]

#15488

Clearly what we need is a modern hygienic macro system to avoid this exec mess!

defmacro defall(name, cond):
    def name(lst):
        for a in lst:
            if not cond:
                return False
        return True

invoke defall(all, cond)

Only slightly tongue in cheek.

We have that stupid exec trick in the Python stdlib. It has to die.

http://hg.python.org/cpython/file/6bf07db23445/Lib/collections/__init__.py#l240

Devin

On Mon, Nov 7, 2011 at 5:06 PM, Chris Angelico <rosuav@gmail.com> wrote:
> On Tue, Nov 8, 2011 at 8:46 AM, david vierra <codewarrior0@gmail.com> wrote:
>> But, you didn't write an all() function.  You wrote a more specialized
>> allBoolean() function. I think this comparison is more fair to the
>> builtin all():
>
> So really, it's not "all() is slow" but "function calls are slow".
> Maybe it'd be worthwhile making an all-factory:
>
> def my_all(code,lst):
>    exec("""def tmp_all(x):
>        for a in x:
>            if not ("""+code+"""): return False
>        return True
> """)
>    return tmp_all(lst)
> timeit.timeit('my_all("a in (True, False)",x)','from __main__ import
> my_all,x',number=10)
>
> Bad code imho, but it _is_ faster than both the original and the builtin.
>
> ChrisA
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#15493

On Tue, 08 Nov 2011 19:44:18 -0500, Devin Jeanpierre wrote:

> We have that stupid exec trick in the Python stdlib. It has to die.
> 
> http://hg.python.org/cpython/file/6bf07db23445/Lib/collections/__init__.py#l240


http://bugs.python.org/issue3974
http://blog.ccpgames.com/kristjan/2011/05/28/namedtuple-and-exec/

If it were someone other than Raymond Hettinger responsible for the use 
of exec in namedtuple, I'd be a lot more suspicious of it.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#15516

> If it were someone other than Raymond Hettinger responsible for the use
> of exec in namedtuple, I'd be a lot more suspicious of it.

I'm not going to be less suspicious based on a name. It reads like
insanity, and the justification was terrible.

Devin

On Tue, Nov 8, 2011 at 9:47 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Tue, 08 Nov 2011 19:44:18 -0500, Devin Jeanpierre wrote:
>
>> We have that stupid exec trick in the Python stdlib. It has to die.
>>
>> http://hg.python.org/cpython/file/6bf07db23445/Lib/collections/__init__.py#l240
>
>
> http://bugs.python.org/issue3974
> http://blog.ccpgames.com/kristjan/2011/05/28/namedtuple-and-exec/
>
> If it were someone other than Raymond Hettinger responsible for the use
> of exec in namedtuple, I'd be a lot more suspicious of it.
>
>
>
> --
> Steven
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#15517

On Wed, 09 Nov 2011 18:01:16 -0500, Devin Jeanpierre wrote:

>> If it were someone other than Raymond Hettinger responsible for the use
>> of exec in namedtuple, I'd be a lot more suspicious of it.
> 
> I'm not going to be less suspicious based on a name.

Neither am I. I am less suspicious based on a reputation. Raymond is a 
well-known, trusted senior Python developer who knows what he is doing.


> It reads like
> insanity, and the justification was terrible.

It reads fine, and the justification is perfectly valid.

You're right to be cautious of exec. You're wrong to be phobic about it. 
What do you think is going to happen? The exec call inside namedtuple is 
going to creep out of the module in the wee hours of the night, 
contaminating other functions and modules while you sleep? Be serious. If 
you have an actual concrete security vulnerability caused by the use of 
exec inside namedtuple, or some other bug, then say so. Otherwise, your 
paranoia is unjustified.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#15520

> Neither am I. I am less suspicious based on a reputation. Raymond is a
> well-known, trusted senior Python developer who knows what he is doing.

I don't really know anything about him or why people respect him, so I
have no reason to share your faith.

> It reads fine, and the justification is perfectly valid.

Well. It reads fine in a certain sense, in that I can figure out
what's going on (although I have some troubles figuring out why the
heck certain things are in the code). The issue is that what's going
on is otherworldly: this is not a Python pattern, this is not a normal
approach. To me, that means it does not read fine.

The use of exec also results in (seemingly) arbitrary constraints on
the input. Like, why can't "--" be a name? Because exec? Is there some
other reason?

I don't like the use of exec, and I don't like the justification (it
seems handwavy). I pointed this out in a thread full of people saying
"never EVER use exec this way", so it's obviously not just me that
thinks this is awful.

> You're right to be cautious of exec. You're wrong to be phobic about it.
> What do you think is going to happen?

I think somebody will read it and think this is a good idea.

Devin


On Wed, Nov 9, 2011 at 6:11 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Wed, 09 Nov 2011 18:01:16 -0500, Devin Jeanpierre wrote:
>
>>> If it were someone other than Raymond Hettinger responsible for the use
>>> of exec in namedtuple, I'd be a lot more suspicious of it.
>>
>> I'm not going to be less suspicious based on a name.
>
> Neither am I. I am less suspicious based on a reputation. Raymond is a
> well-known, trusted senior Python developer who knows what he is doing.
>
>
>> It reads like
>> insanity, and the justification was terrible.
>
> It reads fine, and the justification is perfectly valid.
>
> You're right to be cautious of exec. You're wrong to be phobic about it.
> What do you think is going to happen? The exec call inside namedtuple is
> going to creep out of the module in the wee hours of the night,
> contaminating other functions and modules while you sleep? Be serious. If
> you have an actual concrete security vulnerability caused by the use of
> exec inside namedtuple, or some other bug, then say so. Otherwise, your
> paranoia is unjustified.
>
>
>
> --
> Steven
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#15525

On Nov 10, 11:26 am, Devin Jeanpierre <jeanpierr...@gmail.com> wrote:
> I don't really know anything about him or why people respect him, so I
> have no reason to share your faith.

But you're happy to accept the opinions of random posters saying "exec
is evil"? (And it's really not a good idea to be proud of your
ignorance...)

> Like, why can't "--" be a name?

Why would you ever want it to be?

> I don't like the use of exec, and I don't like the justification (it
> seems handwavy).

As opposed to your in-depth critique?

> I pointed this out in a thread full of people saying
> "never EVER use exec this way", so it's obviously not just me that
> thinks this is awful.

No, instead you have a thread full of people happy to criticise
something for which they're providing no alternative implementation.
You can't exactly say _why_ it's bad, other than other people have
echoed it, but you won't actually do anything about it.

> I think somebody will read it and think this is a good idea.

Just as I thought.

[toc] | [prev] | [next] | [standalone]

#15527

> (And it's really not a good idea to be proud of your
> ignorance...)

I wasn't bragging.

> But you're happy to accept the opinions of random posters saying "exec
> is evil"?
[...]
> As opposed to your in-depth critique?
[...]
> No, instead you have a thread full of people happy to criticise
> something for which they're providing no alternative implementation.
> You can't exactly say _why_ it's bad, other than other people have
> echoed it, but you won't actually do anything about it.

I said it was bad because I found it difficult to read and it was
"weird". I also mentioned that it's conceivable that it has security
flaws, but that's not as big a deal. I believe I also said that it was
bad because it resulted in """arbitrary""" limitations in
functionality. So, yes, I did say why it's bad, and it's not just
because other people say so. My reasons are weak, but that's a
different story.

I also mentioned the alternative implementation, which uses a dict.
There was even already a patch submitted to make namedtuple work this
way, so I don't think I had to be too specific. R. Hettinger rejected
this patch, which was what I was referring to when I was referring to
handwaviness.

So, no.

> Just as I thought.

Woo condescension.

Devin

On Wed, Nov 9, 2011 at 10:50 PM, alex23 <wuwei23@gmail.com> wrote:
> On Nov 10, 11:26 am, Devin Jeanpierre <jeanpierr...@gmail.com> wrote:
>> I don't really know anything about him or why people respect him, so I
>> have no reason to share your faith.
>
> But you're happy to accept the opinions of random posters saying "exec
> is evil"? (And it's really not a good idea to be proud of your
> ignorance...)
>
>> Like, why can't "--" be a name?
>
> Why would you ever want it to be?
>
>> I don't like the use of exec, and I don't like the justification (it
>> seems handwavy).
>
> As opposed to your in-depth critique?
>
>> I pointed this out in a thread full of people saying
>> "never EVER use exec this way", so it's obviously not just me that
>> thinks this is awful.
>
> No, instead you have a thread full of people happy to criticise
> something for which they're providing no alternative implementation.
> You can't exactly say _why_ it's bad, other than other people have
> echoed it, but you won't actually do anything about it.
>
>> I think somebody will read it and think this is a good idea.
>
> Just as I thought.
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#15529

On Wed, 09 Nov 2011 19:50:42 -0800, alex23 wrote:

>> I pointed this out in a thread full of people saying "never EVER use
>> exec this way", so it's obviously not just me that thinks this is
>> awful.
> 
> No, instead you have a thread full of people happy to criticise
> something for which they're providing no alternative implementation. You
> can't exactly say _why_ it's bad, other than other people have echoed
> it, but you won't actually do anything about it.

In fairness there are alternative implementations. They are not identical 
to the version using exec. There is at least one use-case for *not* using 
exec, even at the cost of functionality: a restricted Python environment 
without exec.

On the other hand, a restricted Python without exec is not actually 
Python.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#15530

On Wed, 09 Nov 2011 20:26:56 -0500, Devin Jeanpierre wrote:

>> Neither am I. I am less suspicious based on a reputation. Raymond is a
>> well-known, trusted senior Python developer who knows what he is doing.
> 
> I don't really know anything about him or why people respect him, so I
> have no reason to share your faith.

That's fine.

I don't expect you to take my word on it (and why should you, I could be 
an idiot or a sock-puppet), but you could always try googling for 
"Raymond Hettinger python" and see what comes up. He is not some fly-by 
Python coder who snuck some dubious n00b code into the standard library 
when no-one was looking :)

The mere fact that it was accepted into the standard library should tell 
you that core Python developers consider it an acceptable technique. 
That's not to say the technique is uncontroversial. But there are still 
people who dislike "x if flag else y" and @decorator syntax -- 
controversy, in and of itself, isn't necessarily a reason to avoid 
certain idioms.


Are you familiar with the idea of "code smell"?

http://www.codinghorror.com/blog/2006/05/code-smells.html
http://www.joelonsoftware.com/articles/Wrong.html

I would agree that the use of exec is a code smell. But that doesn't mean 
it is wrong or bad, merely that it needs a second look before accepting 
it. There's a world of difference between "You MUST NOT use exec" and 
"You SHOULD NOT use exec".

See RFC 2119 if you are unclear on the difference:

http://www.ietf.org/rfc/rfc2119.txt



>> It reads fine, and the justification is perfectly valid.
> 
> Well. It reads fine in a certain sense, in that I can figure out what's
> going on (although I have some troubles figuring out why the heck
> certain things are in the code). The issue is that what's going on is
> otherworldly: this is not a Python pattern, this is not a normal
> approach. To me, that means it does not read fine.

There's nothing inside the template being exec'ed that couldn't be found 
in non-exec code. So if you're having trouble figuring out parts of the 
code, the presence of the exec is not the problem.

Having said that, dynamic code generation is well known for often being 
harder to read than "ordinary" code. But then, pointers are hard too.



> The use of exec also results in (seemingly) arbitrary constraints on the
> input. Like, why can't "--" be a name? Because exec? Is there some other
> reason?

Because Python doesn't allow "--" to be an attribute name, and so 
namedtuple doesn't let you try:

t = namedtuple("T", "foo -- bar")(1, 2, 3)
print(t.foo)
print(t.--)
print(t.bar)




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#15531

> I don't expect you to take my word on it (and why should you, I could be
> an idiot or a sock-puppet), but you could always try googling for
> "Raymond Hettinger python" and see what comes up. He is not some fly-by
> Python coder who snuck some dubious n00b code into the standard library
> when no-one was looking :)

Alright, I know *something* about him. I knew he was a core developer,
and that he was responsible for namedtuple. I also discovered (when I
looked up his activestate profile) some other stuff he wrote. I don't
really know anything about him outside of that -- i.e. I have no idea
what parts of Python he's contributed things to in the past that could
make me go, "oh, wow, _he_ did that?" and so on. I don't really feel
like a few minutes research would give me the right feel, it generally
has to come up organically.

Anyway, if we step back, for a trustworthy developer who wrote
something seemingly-crazy, I should be willing to suspend judgement
until I see the relevant facts about something that the developer
might have and I don't. But he did give the facts,
( http://bugs.python.org/issue3974 again) , and I'm not convinced.

Things can go terribly wrong when abusing exec e.g.
http://www.gossamer-threads.com/lists/python/bugs/568206 . That
shouldn't ever happen with a function such as this. exec opens doors
that should not be opened without a really good reason, and those
reasons don't strike me that way.

> The mere fact that it was accepted into the standard library should tell
> you that core Python developers consider it an acceptable technique.

I've seen core developers rail against the namedtuple source code. In
fairness, I don't believe exec was the subject of the rant --
nonetheless its presence isn't evidence of general support, and even
if it were, my tastes have always differed from that of the core
developers.

> That's not to say the technique is uncontroversial. But there are still
> people who dislike "x if flag else y" and @decorator syntax --
> controversy, in and of itself, isn't necessarily a reason to avoid
> certain idioms.

I think there's somewhat a difference in magnitude of objections
between using exec as a hacked-together macro system, and using "x if
flag else y" when if statements would do.

If the exec trick is reasonable, we should normalize it in the form of
a real, useful macro system, that can protect us against exec's many
flaws (code injection, accidental syntax errors, etc.) and tell future
programmers how to do this safely and in a semi-approvable way.

> I would agree that the use of exec is a code smell. But that doesn't mean
> it is wrong or bad, merely that it needs a second look before accepting
> it. There's a world of difference between "You MUST NOT use exec" and
> "You SHOULD NOT use exec".

Do I really need a second look? I see exec, I wonder what it's doing.
It isn't doing anything that couldn't be done subjectively better with
e.g. a dict, so I disapprove of the usage of exec.

> There's nothing inside the template being exec'ed that couldn't be found
> in non-exec code. So if you're having trouble figuring out parts of the
> code, the presence of the exec is not the problem.

There's more overhead going back and forth to the template, and
there's related things that I can't be sure are because of exec or
because of design decisions, etc.  It makes code reading more
challenging, even if it's still possible. That said, sure, some of
these are problems with whatever else he's done.

> Having said that, dynamic code generation is well known for often being
> harder to read than "ordinary" code. But then, pointers are hard too.

And on the other other hand, Python lacks explicit support for both
pointers and code generation (unless you count strings and ctypes).

> Because Python doesn't allow "--" to be an attribute name, and so
> namedtuple doesn't let you try:
>
> t = namedtuple("T", "foo -- bar")(1, 2, 3)
> print(t.foo)
> print(t.--)
> print(t.bar)

'--' is a valid attribute name on virtually any object that supports
attribute setting (e.g. function objects). Of course, you need to use
setattr() and getattr(). Is this really the reason, or is it a
limitation caused primarily by the usage of exec and the need to
prevent code injection? If somebody added this feature later on, would
this create a security vulnerability in certain projects that used
namedtuple in certain ways?

Devin

On Thu, Nov 10, 2011 at 2:48 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Wed, 09 Nov 2011 20:26:56 -0500, Devin Jeanpierre wrote:
>
>>> Neither am I. I am less suspicious based on a reputation. Raymond is a
>>> well-known, trusted senior Python developer who knows what he is doing.
>>
>> I don't really know anything about him or why people respect him, so I
>> have no reason to share your faith.
>
> That's fine.
>
> I don't expect you to take my word on it (and why should you, I could be
> an idiot or a sock-puppet), but you could always try googling for
> "Raymond Hettinger python" and see what comes up. He is not some fly-by
> Python coder who snuck some dubious n00b code into the standard library
> when no-one was looking :)
>
> The mere fact that it was accepted into the standard library should tell
> you that core Python developers consider it an acceptable technique.
> That's not to say the technique is uncontroversial. But there are still
> people who dislike "x if flag else y" and @decorator syntax --
> controversy, in and of itself, isn't necessarily a reason to avoid
> certain idioms.
>
>
> Are you familiar with the idea of "code smell"?
>
> http://www.codinghorror.com/blog/2006/05/code-smells.html
> http://www.joelonsoftware.com/articles/Wrong.html
>
> I would agree that the use of exec is a code smell. But that doesn't mean
> it is wrong or bad, merely that it needs a second look before accepting
> it. There's a world of difference between "You MUST NOT use exec" and
> "You SHOULD NOT use exec".
>
> See RFC 2119 if you are unclear on the difference:
>
> http://www.ietf.org/rfc/rfc2119.txt
>
>
>
>>> It reads fine, and the justification is perfectly valid.
>>
>> Well. It reads fine in a certain sense, in that I can figure out what's
>> going on (although I have some troubles figuring out why the heck
>> certain things are in the code). The issue is that what's going on is
>> otherworldly: this is not a Python pattern, this is not a normal
>> approach. To me, that means it does not read fine.
>
> There's nothing inside the template being exec'ed that couldn't be found
> in non-exec code. So if you're having trouble figuring out parts of the
> code, the presence of the exec is not the problem.
>
> Having said that, dynamic code generation is well known for often being
> harder to read than "ordinary" code. But then, pointers are hard too.
>
>
>
>> The use of exec also results in (seemingly) arbitrary constraints on the
>> input. Like, why can't "--" be a name? Because exec? Is there some other
>> reason?
>
> Because Python doesn't allow "--" to be an attribute name, and so
> namedtuple doesn't let you try:
>
> t = namedtuple("T", "foo -- bar")(1, 2, 3)
> print(t.foo)
> print(t.--)
> print(t.bar)
>
>
>
>
> --
> Steven
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#15535

On Thursday, November 10, 2011 08:13:13 AM Devin Jeanpierre did opine:

> > I don't expect you to take my word on it (and why should you, I could
> > be an idiot or a sock-puppet), but you could always try googling for
> > "Raymond Hettinger python" and see what comes up. He is not some
> > fly-by Python coder who snuck some dubious n00b code into the
> > standard library when no-one was looking :)
> 
> Alright, I know *something* about him. I knew he was a core developer,
> and that he was responsible for namedtuple. I also discovered (when I
> looked up his activestate profile) some other stuff he wrote. I don't
> really know anything about him outside of that -- i.e. I have no idea
> what parts of Python he's contributed things to in the past that could
> make me go, "oh, wow, _he_ did that?" and so on. I don't really feel
> like a few minutes research would give me the right feel, it generally
> has to come up organically.
> 
> Anyway, if we step back, for a trustworthy developer who wrote
> something seemingly-crazy, I should be willing to suspend judgement
> until I see the relevant facts about something that the developer
> might have and I don't. But he did give the facts,
> ( http://bugs.python.org/issue3974 again) , and I'm not convinced.
> 
> Things can go terribly wrong when abusing exec e.g.
> http://www.gossamer-threads.com/lists/python/bugs/568206 . That
> shouldn't ever happen with a function such as this. exec opens doors
> that should not be opened without a really good reason, and those
> reasons don't strike me that way.
 
If, in the sense that this python 'exec' essentially duplicates the bash 
version, then I have found it quite useful, it was taught to me several 
years ago by another teacher who was a long time Solaris fan, and if it 
were to go away, I have several bash scripts running here right now that 
would require major re-writes.

> > The mere fact that it was accepted into the standard library should
> > tell you that core Python developers consider it an acceptable
> > technique.

Well, its certainly not a new concept.  All the major 'shell interpreters' 
have it, why not python?

> I've seen core developers rail against the namedtuple source code. In
> fairness, I don't believe exec was the subject of the rant --
> nonetheless its presence isn't evidence of general support, and even
> if it were, my tastes have always differed from that of the core
> developers.
> 
> > That's not to say the technique is uncontroversial. But there are
> > still people who dislike "x if flag else y" and @decorator syntax --
> > controversy, in and of itself, isn't necessarily a reason to avoid
> > certain idioms.
> 
> I think there's somewhat a difference in magnitude of objections
> between using exec as a hacked-together macro system, and using "x if
> flag else y" when if statements would do.
> 
> If the exec trick is reasonable, we should normalize it in the form of
> a real, useful macro system, that can protect us against exec's many
> flaws (code injection, accidental syntax errors, etc.) and tell future
> programmers how to do this safely and in a semi-approvable way.
> 
> > I would agree that the use of exec is a code smell. But that doesn't
> > mean it is wrong or bad, merely that it needs a second look before
> > accepting it. There's a world of difference between "You MUST NOT use
> > exec" and "You SHOULD NOT use exec".
> 
> Do I really need a second look? I see exec, I wonder what it's doing.
> It isn't doing anything that couldn't be done subjectively better with
> e.g. a dict, so I disapprove of the usage of exec.
> 
> > There's nothing inside the template being exec'ed that couldn't be
> > found in non-exec code. So if you're having trouble figuring out
> > parts of the code, the presence of the exec is not the problem.
> 
> There's more overhead going back and forth to the template, and
> there's related things that I can't be sure are because of exec or
> because of design decisions, etc.  It makes code reading more
> challenging, even if it's still possible. That said, sure, some of
> these are problems with whatever else he's done.
> 
> > Having said that, dynamic code generation is well known for often
> > being harder to read than "ordinary" code. But then, pointers are
> > hard too.
> 
> And on the other other hand, Python lacks explicit support for both
> pointers and code generation (unless you count strings and ctypes).
> 
> > Because Python doesn't allow "--" to be an attribute name, and so
> > namedtuple doesn't let you try:
> > 
> > t = namedtuple("T", "foo -- bar")(1, 2, 3)
> > print(t.foo)
> > print(t.--)
> > print(t.bar)
> 
> '--' is a valid attribute name on virtually any object that supports
> attribute setting (e.g. function objects). Of course, you need to use
> setattr() and getattr(). Is this really the reason, or is it a
> limitation caused primarily by the usage of exec and the need to
> prevent code injection? If somebody added this feature later on, would
> this create a security vulnerability in certain projects that used
> namedtuple in certain ways?
> 
> Devin
> 
> On Thu, Nov 10, 2011 at 2:48 AM, Steven D'Aprano
> 
> <steve+comp.lang.python@pearwood.info> wrote:
> > On Wed, 09 Nov 2011 20:26:56 -0500, Devin Jeanpierre wrote:
> >>> Neither am I. I am less suspicious based on a reputation. Raymond is
> >>> a well-known, trusted senior Python developer who knows what he is
> >>> doing.
> >> 
> >> I don't really know anything about him or why people respect him, so
> >> I have no reason to share your faith.
> > 
> > That's fine.
> > 
> > I don't expect you to take my word on it (and why should you, I could
> > be an idiot or a sock-puppet), but you could always try googling for
> > "Raymond Hettinger python" and see what comes up. He is not some
> > fly-by Python coder who snuck some dubious n00b code into the
> > standard library when no-one was looking :)
> > 
> > The mere fact that it was accepted into the standard library should
> > tell you that core Python developers consider it an acceptable
> > technique. That's not to say the technique is uncontroversial. But
> > there are still people who dislike "x if flag else y" and @decorator
> > syntax -- controversy, in and of itself, isn't necessarily a reason
> > to avoid certain idioms.
> > 
> > 
> > Are you familiar with the idea of "code smell"?
> > 
> > http://www.codinghorror.com/blog/2006/05/code-smells.html
> > http://www.joelonsoftware.com/articles/Wrong.html
> > 
> > I would agree that the use of exec is a code smell. But that doesn't
> > mean it is wrong or bad, merely that it needs a second look before
> > accepting it. There's a world of difference between "You MUST NOT use
> > exec" and "You SHOULD NOT use exec".
> > 
> > See RFC 2119 if you are unclear on the difference:
> > 
> > http://www.ietf.org/rfc/rfc2119.txt
> > 
> >>> It reads fine, and the justification is perfectly valid.
> >> 
> >> Well. It reads fine in a certain sense, in that I can figure out
> >> what's going on (although I have some troubles figuring out why the
> >> heck certain things are in the code). The issue is that what's going
> >> on is otherworldly: this is not a Python pattern, this is not a
> >> normal approach. To me, that means it does not read fine.
> > 
> > There's nothing inside the template being exec'ed that couldn't be
> > found in non-exec code. So if you're having trouble figuring out
> > parts of the code, the presence of the exec is not the problem.
> > 
> > Having said that, dynamic code generation is well known for often
> > being harder to read than "ordinary" code. But then, pointers are
> > hard too.
> > 
> >> The use of exec also results in (seemingly) arbitrary constraints on
> >> the input. Like, why can't "--" be a name? Because exec? Is there
> >> some other reason?
> > 
> > Because Python doesn't allow "--" to be an attribute name, and so
> > namedtuple doesn't let you try:
> > 
> > t = namedtuple("T", "foo -- bar")(1, 2, 3)
> > print(t.foo)
> > print(t.--)
> > print(t.bar)
> > 
> > 
> > 
> > 
> > --
> > Steven
> > --
> > http://mail.python.org/mailman/listinfo/python-list


Cheers, Gene
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
My web page: <http://coyoteden.dyndns-free.com:85/gene>
>Ever heard of .cshrc?
That's a city in Bosnia.  Right?
(Discussion in comp.os.linux.misc on the intuitiveness of commands.)

[toc] | [prev] | [next] | [standalone]

#15554

On 11/10/2011 3:51 AM, Devin Jeanpierre wrote:

>> Because Python doesn't allow "--" to be an attribute name, and so
>> namedtuple doesn't let you try:
>>
>> t = namedtuple("T", "foo -- bar")(1, 2, 3)
>> print(t.foo)
>> print(t.--)
>> print(t.bar)
>
> '--' is a valid attribute name on virtually any object that supports
> attribute setting (e.g. function objects).

ob.-- is not valid Python because '--' is not a name.

 > Of course, you need to use setattr() and getattr().

I consider the fact that CPython's setattr accepts non-name strings to 
be a bit of a bug. Or if you will, leniency for speed. (A unicode name 
check in Py3 would be much more expensive than an ascii name check in 
Py2.) I would consider it legitimate for another implementation to only 
accept names and to use a specialized name_dict for attribute dictionaries.

So I consider it quite legitimate for namedtuple to requires real names 
for the fields. The whole point is to allow ob.name access to tuple 
members. Someone who plans to use set/getattr with arbitrary strings 
should just use a dict instead of a tuple.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web