Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #15535

Re: all() is slow?

Show key headers only | View raw

On Thursday, November 10, 2011 08:13:13 AM Devin Jeanpierre did opine:

> > I don't expect you to take my word on it (and why should you, I could
> > be an idiot or a sock-puppet), but you could always try googling for
> > "Raymond Hettinger python" and see what comes up. He is not some
> > fly-by Python coder who snuck some dubious n00b code into the
> > standard library when no-one was looking :)
> 
> Alright, I know *something* about him. I knew he was a core developer,
> and that he was responsible for namedtuple. I also discovered (when I
> looked up his activestate profile) some other stuff he wrote. I don't
> really know anything about him outside of that -- i.e. I have no idea
> what parts of Python he's contributed things to in the past that could
> make me go, "oh, wow, _he_ did that?" and so on. I don't really feel
> like a few minutes research would give me the right feel, it generally
> has to come up organically.
> 
> Anyway, if we step back, for a trustworthy developer who wrote
> something seemingly-crazy, I should be willing to suspend judgement
> until I see the relevant facts about something that the developer
> might have and I don't. But he did give the facts,
> ( http://bugs.python.org/issue3974 again) , and I'm not convinced.
> 
> Things can go terribly wrong when abusing exec e.g.
> http://www.gossamer-threads.com/lists/python/bugs/568206 . That
> shouldn't ever happen with a function such as this. exec opens doors
> that should not be opened without a really good reason, and those
> reasons don't strike me that way.
 
If, in the sense that this python 'exec' essentially duplicates the bash 
version, then I have found it quite useful, it was taught to me several 
years ago by another teacher who was a long time Solaris fan, and if it 
were to go away, I have several bash scripts running here right now that 
would require major re-writes.

> > The mere fact that it was accepted into the standard library should
> > tell you that core Python developers consider it an acceptable
> > technique.

Well, its certainly not a new concept.  All the major 'shell interpreters' 
have it, why not python?

> I've seen core developers rail against the namedtuple source code. In
> fairness, I don't believe exec was the subject of the rant --
> nonetheless its presence isn't evidence of general support, and even
> if it were, my tastes have always differed from that of the core
> developers.
> 
> > That's not to say the technique is uncontroversial. But there are
> > still people who dislike "x if flag else y" and @decorator syntax --
> > controversy, in and of itself, isn't necessarily a reason to avoid
> > certain idioms.
> 
> I think there's somewhat a difference in magnitude of objections
> between using exec as a hacked-together macro system, and using "x if
> flag else y" when if statements would do.
> 
> If the exec trick is reasonable, we should normalize it in the form of
> a real, useful macro system, that can protect us against exec's many
> flaws (code injection, accidental syntax errors, etc.) and tell future
> programmers how to do this safely and in a semi-approvable way.
> 
> > I would agree that the use of exec is a code smell. But that doesn't
> > mean it is wrong or bad, merely that it needs a second look before
> > accepting it. There's a world of difference between "You MUST NOT use
> > exec" and "You SHOULD NOT use exec".
> 
> Do I really need a second look? I see exec, I wonder what it's doing.
> It isn't doing anything that couldn't be done subjectively better with
> e.g. a dict, so I disapprove of the usage of exec.
> 
> > There's nothing inside the template being exec'ed that couldn't be
> > found in non-exec code. So if you're having trouble figuring out
> > parts of the code, the presence of the exec is not the problem.
> 
> There's more overhead going back and forth to the template, and
> there's related things that I can't be sure are because of exec or
> because of design decisions, etc.  It makes code reading more
> challenging, even if it's still possible. That said, sure, some of
> these are problems with whatever else he's done.
> 
> > Having said that, dynamic code generation is well known for often
> > being harder to read than "ordinary" code. But then, pointers are
> > hard too.
> 
> And on the other other hand, Python lacks explicit support for both
> pointers and code generation (unless you count strings and ctypes).
> 
> > Because Python doesn't allow "--" to be an attribute name, and so
> > namedtuple doesn't let you try:
> > 
> > t = namedtuple("T", "foo -- bar")(1, 2, 3)
> > print(t.foo)
> > print(t.--)
> > print(t.bar)
> 
> '--' is a valid attribute name on virtually any object that supports
> attribute setting (e.g. function objects). Of course, you need to use
> setattr() and getattr(). Is this really the reason, or is it a
> limitation caused primarily by the usage of exec and the need to
> prevent code injection? If somebody added this feature later on, would
> this create a security vulnerability in certain projects that used
> namedtuple in certain ways?
> 
> Devin
> 
> On Thu, Nov 10, 2011 at 2:48 AM, Steven D'Aprano
> 
> <steve+comp.lang.python@pearwood.info> wrote:
> > On Wed, 09 Nov 2011 20:26:56 -0500, Devin Jeanpierre wrote:
> >>> Neither am I. I am less suspicious based on a reputation. Raymond is
> >>> a well-known, trusted senior Python developer who knows what he is
> >>> doing.
> >> 
> >> I don't really know anything about him or why people respect him, so
> >> I have no reason to share your faith.
> > 
> > That's fine.
> > 
> > I don't expect you to take my word on it (and why should you, I could
> > be an idiot or a sock-puppet), but you could always try googling for
> > "Raymond Hettinger python" and see what comes up. He is not some
> > fly-by Python coder who snuck some dubious n00b code into the
> > standard library when no-one was looking :)
> > 
> > The mere fact that it was accepted into the standard library should
> > tell you that core Python developers consider it an acceptable
> > technique. That's not to say the technique is uncontroversial. But
> > there are still people who dislike "x if flag else y" and @decorator
> > syntax -- controversy, in and of itself, isn't necessarily a reason
> > to avoid certain idioms.
> > 
> > 
> > Are you familiar with the idea of "code smell"?
> > 
> > http://www.codinghorror.com/blog/2006/05/code-smells.html
> > http://www.joelonsoftware.com/articles/Wrong.html
> > 
> > I would agree that the use of exec is a code smell. But that doesn't
> > mean it is wrong or bad, merely that it needs a second look before
> > accepting it. There's a world of difference between "You MUST NOT use
> > exec" and "You SHOULD NOT use exec".
> > 
> > See RFC 2119 if you are unclear on the difference:
> > 
> > http://www.ietf.org/rfc/rfc2119.txt
> > 
> >>> It reads fine, and the justification is perfectly valid.
> >> 
> >> Well. It reads fine in a certain sense, in that I can figure out
> >> what's going on (although I have some troubles figuring out why the
> >> heck certain things are in the code). The issue is that what's going
> >> on is otherworldly: this is not a Python pattern, this is not a
> >> normal approach. To me, that means it does not read fine.
> > 
> > There's nothing inside the template being exec'ed that couldn't be
> > found in non-exec code. So if you're having trouble figuring out
> > parts of the code, the presence of the exec is not the problem.
> > 
> > Having said that, dynamic code generation is well known for often
> > being harder to read than "ordinary" code. But then, pointers are
> > hard too.
> > 
> >> The use of exec also results in (seemingly) arbitrary constraints on
> >> the input. Like, why can't "--" be a name? Because exec? Is there
> >> some other reason?
> > 
> > Because Python doesn't allow "--" to be an attribute name, and so
> > namedtuple doesn't let you try:
> > 
> > t = namedtuple("T", "foo -- bar")(1, 2, 3)
> > print(t.foo)
> > print(t.--)
> > print(t.bar)
> > 
> > 
> > 
> > 
> > --
> > Steven
> > --
> > http://mail.python.org/mailman/listinfo/python-list


Cheers, Gene
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
My web page: <http://coyoteden.dyndns-free.com:85/gene>
>Ever heard of .cshrc?
That's a city in Bosnia.  Right?
(Discussion in comp.os.linux.misc on the intuitiveness of commands.)

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

all() is slow? "OKB (not okblacke)" <brenNOSPAMbarn@NObrenSPAMbarn.net> - 2011-11-07 21:00 +0000
  Re: all() is slow? Chris Rebert <clp2@rebertia.com> - 2011-11-07 13:39 -0800
  Re: all() is slow? david vierra <codewarrior0@gmail.com> - 2011-11-07 13:46 -0800
    Re: all() is slow? Chris Angelico <rosuav@gmail.com> - 2011-11-08 09:06 +1100
      Re: all() is slow? Henrik Faber <hfaber@invalid.net> - 2011-11-08 13:09 +0100
        Re: all() is slow? Chris Angelico <rosuav@gmail.com> - 2011-11-08 23:14 +1100
        Re: all() is slow? John Posner <jjposner@optimum.net> - 2011-11-08 17:51 -0500
        Re: all() is slow? Tim Chase <python.list@tim.thechases.com> - 2011-11-08 17:56 -0600
    Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-08 19:44 -0500
      Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-09 02:47 +0000
        Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-09 18:01 -0500
          Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-09 23:11 +0000
            Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-09 20:26 -0500
              Re: all() is slow? alex23 <wuwei23@gmail.com> - 2011-11-09 19:50 -0800
                Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-09 23:40 -0500
                Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-10 07:35 +0000
              Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-10 07:48 +0000
                Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-10 03:51 -0500
                Re: all() is slow? gene heskett <gheskett@wdtv.com> - 2011-11-10 08:20 -0500
                Re: all() is slow? Terry Reedy <tjreedy@udel.edu> - 2011-11-10 14:25 -0500
            Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-09 20:35 -0500
            Re: all() is slow? Ian Kelly <ian.g.kelly@gmail.com> - 2011-11-09 19:10 -0700
            Re: all() is slow? Ethan Furman <ethan@stoneleaf.us> - 2011-11-10 10:43 -0800
            Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-10 15:37 -0500
              Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-10 23:07 +0000
                Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-10 23:35 -0500
                Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-12 01:18 +0000
                Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-13 01:28 -0500
                Re: all() is slow? alex23 <wuwei23@gmail.com> - 2011-11-13 18:50 -0800
                Re: all() is slow? Devin Jeanpierre <jeanpierreda@gmail.com> - 2011-11-13 23:48 -0500
              Re: all() is slow? "OKB (not okblacke)" <brenNOSPAMbarn@NObrenSPAMbarn.net> - 2011-11-11 05:40 +0000
            Re: all() is slow? Ian Kelly <ian.g.kelly@gmail.com> - 2011-11-10 14:19 -0700
              Re: all() is slow? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-10 22:56 +0000
            Re: all() is slow? Ethan Furman <ethan@stoneleaf.us> - 2011-11-10 13:47 -0800
        Re: all() is slow? Chris Angelico <rosuav@gmail.com> - 2011-11-10 10:15 +1100
    Re: all() is slow? Chris Angelico <rosuav@gmail.com> - 2011-11-09 12:19 +1100
    Re: all() is slow? Chris Rebert <clp2@rebertia.com> - 2011-11-08 17:30 -0800
      Re: all() is slow? Hans Mulder <hansmu@xs4all.nl> - 2011-11-09 16:41 +0100
        Re: all() is slow? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2011-11-09 09:07 -0800
  Re: all() is slow? John Nagle <nagle@animats.com> - 2011-11-09 14:16 -0800
    Re: all() is slow? alex23 <wuwei23@gmail.com> - 2011-11-09 19:52 -0800
    Re: all() is slow? "OKB (not okblacke)" <brenNOSPAMbarn@NObrenSPAMbarn.net> - 2011-11-10 19:35 +0000
  Re: all() is slow? BOOK-AZ <amalguseynov@gmail.com> - 2011-11-15 08:02 -0800

csiph-web