Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #11476 > unrolled thread

Re: Why no warnings when re-assigning builtin names?

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Why no warnings when re-assigning builtin names? Chris Angelico <rosuav@gmail.com> - 2011-08-15 23:15 +0100
    Re: Why no warnings when re-assigning builtin names? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-08-16 11:32 +1000
      Re: Why no warnings when re-assigning builtin names? Philip Semanchuk <philip@semanchuk.com> - 2011-08-15 23:23 -0400
        Re: Why no warnings when re-assigning builtin names? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-08-16 15:15 +1000
          Re: Why no warnings when re-assigning builtin names? Philip Semanchuk <philip@semanchuk.com> - 2011-08-16 10:13 -0400
            Re: Why no warnings when re-assigning builtin names? rantingrick <rantingrick@gmail.com> - 2011-08-16 08:44 -0700
              Re: Why no warnings when re-assigning builtin names? Andrew Berg <bahamutzero8825@gmail.com> - 2011-08-16 12:23 -0500
          Re: Why no warnings when re-assigning builtin names? Chris Angelico <rosuav@gmail.com> - 2011-08-16 16:12 +0100
          Re: Why no warnings when re-assigning builtin names? Ethan Furman <ethan@stoneleaf.us> - 2011-08-16 08:41 -0700
            Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-16 17:11 +0000
              Re: Why no warnings when re-assigning builtin names? Ethan Furman <ethan@stoneleaf.us> - 2011-08-16 10:48 -0700
              Re: Why no warnings when re-assigning builtin names? Tim Chase <python.list@tim.thechases.com> - 2011-08-16 13:34 -0500
          Re: Why no warnings when re-assigning builtin names? Philip Semanchuk <philip@semanchuk.com> - 2011-08-16 11:31 -0400
          Re: Why no warnings when re-assigning builtin names? Philip Semanchuk <philip@semanchuk.com> - 2011-08-16 11:38 -0400
          Re: Why no warnings when re-assigning builtin names? Ethan Furman <ethan@stoneleaf.us> - 2011-08-16 09:19 -0700
            Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-16 17:11 +0000
              Re: Why no warnings when re-assigning builtin names? Ethan Furman <ethan@stoneleaf.us> - 2011-08-16 10:43 -0700
                Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-16 18:56 +0000
                  Re: Why no warnings when re-assigning builtin names? Terry Reedy <tjreedy@udel.edu> - 2011-08-16 19:32 -0400
                    Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-16 23:49 +0000
                      Re: Why no warnings when re-assigning builtin names? Chris Angelico <rosuav@gmail.com> - 2011-08-17 01:02 +0100
                        Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-17 01:35 +0000
                          Re: Why no warnings when re-assigning builtin names? Chris Angelico <rosuav@gmail.com> - 2011-08-17 08:27 +0100
                            Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-17 16:33 +0000
                      Re: Why no warnings when re-assigning builtin names? Terry Reedy <tjreedy@udel.edu> - 2011-08-16 22:10 -0400
              Re: Why no warnings when re-assigning builtin names? Chris Angelico <rosuav@gmail.com> - 2011-08-16 19:23 +0100
          Re: Why no warnings when re-assigning builtin names? Tim Chase <python.list@tim.thechases.com> - 2011-08-16 11:10 -0500
          Re: Why no warnings when re-assigning builtin names? Philip Semanchuk <philip@semanchuk.com> - 2011-08-16 12:32 -0400
            Re: Why no warnings when re-assigning builtin names? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-08-17 11:29 +1000
              Re: Why no warnings when re-assigning builtin names? Philip Semanchuk <philip@semanchuk.com> - 2011-08-16 22:16 -0400
          Re: Why no warnings when re-assigning builtin names? Ethan Furman <ethan@stoneleaf.us> - 2011-08-16 10:07 -0700
          Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-16 17:11 +0000
            Re: Why no warnings when re-assigning builtin names? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-08-17 10:58 +1000
              Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-17 03:17 +0000
                Re: Why no warnings when re-assigning builtin names? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-08-17 18:15 +1000
                  Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-17 16:33 +0000
                    Re: Why no warnings when re-assigning builtin names? Ethan Furman <ethan@stoneleaf.us> - 2011-08-17 10:36 -0700
                      Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-17 17:36 +0000
                        Re: Why no warnings when re-assigning builtin names? Ethan Furman <ethan@stoneleaf.us> - 2011-08-17 11:13 -0700
                          Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-17 18:09 +0000
                    Re: Why no warnings when re-assigning builtin names? Chris Angelico <rosuav@gmail.com> - 2011-08-17 18:42 +0100
                      Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-17 17:38 +0000
                        Re: Why no warnings when re-assigning builtin names? Chris Angelico <rosuav@gmail.com> - 2011-08-17 19:02 +0100
                    Re: Why no warnings when re-assigning builtin names? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-08-18 10:36 +1000
              Re: Why no warnings when re-assigning builtin names? Chris Angelico <rosuav@gmail.com> - 2011-08-17 08:34 +0100
          RE: Why no warnings when re-assigning builtin names? "Gerrat Rickert" <grickert@coldstorage.com> - 2011-08-16 13:15 -0400
          Re: Why no warnings when re-assigning builtin names? Terry Reedy <tjreedy@udel.edu> - 2011-08-16 19:29 -0400
          Re: Why no warnings when re-assigning builtin names? Philip Semanchuk <philip@semanchuk.com> - 2011-08-16 20:18 -0400
          Re: Why no warnings when re-assigning builtin names? Terry Reedy <tjreedy@udel.edu> - 2011-08-16 22:15 -0400
          Re: Why no warnings when re-assigning builtin names? Philip Semanchuk <philip@semanchuk.com> - 2011-08-16 22:51 -0400
          RE: Why no warnings when re-assigning builtin names? "Gerrat Rickert" <grickert@coldstorage.com> - 2011-08-17 09:30 -0400
      Re: Why no warnings when re-assigning builtin names? Chris Angelico <rosuav@gmail.com> - 2011-08-16 09:16 +0100
    Re: Why no warnings when re-assigning builtin names? Chris Torek <nospam@torek.net> - 2011-08-31 21:30 +0000
      Re: Why no warnings when re-assigning builtin names? Seebs <usenet-nospam@seebs.net> - 2011-08-31 21:57 +0000

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#11628

On Wed, Aug 17, 2011 at 12:49 AM, Seebs <usenet-nospam@seebs.net> wrote:
> Yes, but is it a *significant* cost?  My assumption is that the suppression
> would be of checking, not just of displaying messages.
>

It mightn't be very significant, but there'd still be some cost.
However, IMHO the greatest cost is the spamminess; forcing the user to
deal with lines and lines of warnings is not a useful way to design a
language.

ChrisA

[toc] | [prev] | [next] | [standalone]

#11647

On 2011-08-17, Chris Angelico <rosuav@gmail.com> wrote:
> On Wed, Aug 17, 2011 at 12:49 AM, Seebs <usenet-nospam@seebs.net> wrote:
>> Yes, but is it a *significant* cost? ?My assumption is that the suppression
>> would be of checking, not just of displaying messages.

> It mightn't be very significant, but there'd still be some cost.
> However, IMHO the greatest cost is the spamminess; forcing the user to
> deal with lines and lines of warnings is not a useful way to design a
> language.

Lines and lines?

I'd say if it's to be allowed to shadow built-ins (and I'm not sure that's
a good thing at all), you'd still be looking at one warning per shadowing,
which shouldn't be too many.

-s
-- 
Copyright 2011, all wrongs reversed.  Peter Seebach / usenet-nospam@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.

[toc] | [prev] | [next] | [standalone]

#11662

On Wed, Aug 17, 2011 at 2:35 AM, Seebs <usenet-nospam@seebs.net> wrote:
> On 2011-08-17, Chris Angelico <rosuav@gmail.com> wrote:
>> It mightn't be very significant, but there'd still be some cost.
>> However, IMHO the greatest cost is the spamminess; forcing the user to
>> deal with lines and lines of warnings is not a useful way to design a
>> language.
>
> Lines and lines?
>
> I'd say if it's to be allowed to shadow built-ins (and I'm not sure that's
> a good thing at all), you'd still be looking at one warning per shadowing,
> which shouldn't be too many.

One warning per shadowing. Okay.

def foo(list):
   """Foo's the list provided and returns True on success or False on
failure."""

def bar(list):
    """Counts the number of bars in the list, assuming it to be made
of music."""
    if not foo(list): return

You call foo() once and bar() twice. How many shadowings have there
been? How many warnings do you get?

A simple implementation would give five warnings for this case - once
for each invocation that shadows a builtin. Another simple
implementation would give two warnings, at the time that the def
statements are executed; this is preferred, but it's still two
warnings, and if you have a huge set of functions that do this, that
can easily be "lines and lines" of warnings. Or should it set a flag
and say "I've already warned this session about shadowing 'list', so
suppress all others"? That seems confusing.

ChrisA

[toc] | [prev] | [next] | [standalone]

#11694

On 2011-08-17, Chris Angelico <rosuav@gmail.com> wrote:
> def foo(list):
>    """Foo's the list provided and returns True on success or False on
> failure."""
>
> def bar(list):
>     """Counts the number of bars in the list, assuming it to be made
> of music."""
>     if not foo(list): return

> You call foo() once and bar() twice. How many shadowings have there
> been? How many warnings do you get?

I'd say two, one when def foo... is parsed, one when def bar... is parsed.

> A simple implementation would give five warnings for this case - once
> for each invocation that shadows a builtin. Another simple
> implementation would give two warnings, at the time that the def
> statements are executed; this is preferred, but it's still two
> warnings, and if you have a huge set of functions that do this, that
> can easily be "lines and lines" of warnings. Or should it set a flag
> and say "I've already warned this session about shadowing 'list', so
> suppress all others"? That seems confusing.

I guess I don't object to multiple warnings if I do the same thing multiple
times.  I was just thinking in terms of a single parse-time warning for the
actual point at which something is shadowed, rather than, say, a warning
every time a name is hit during execution of statements and refers to a
shadow.

-s
-- 
Copyright 2011, all wrongs reversed.  Peter Seebach / usenet-nospam@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.

[toc] | [prev] | [next] | [standalone]

#11648

On 8/16/2011 7:49 PM, Seebs wrote:
> On 2011-08-16, Terry Reedy<tjreedy@udel.edu>  wrote:
>> On 8/16/2011 2:56 PM, Seebs wrote:
>>> I wonder whether there's a way to mitigate the cost of these things by
>>> messing with -W settings, such that runtime that wants to be fast can
>>> omit the checks, but the default could still be to, well, prevent likely
>>> errors.
>
>> Warning messages have a cost even if suppressed.
>
> Yes, but is it a *significant* cost?  My assumption is that the suppression
> would be of checking, not just of displaying messages.

The -W settings suppress display or printing of warnings, not their 
generation.

import warnings

if some_condition:
   warnings.warn(message, Warning)

The test and warn call are made regardless of disposition. Filters 
determine what is done with each Warning subclass. The logging module 
can grab them too.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#11593

On Tue, Aug 16, 2011 at 6:43 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
> Seebs wrote:
>>
>> On 2011-08-16, Ethan Furman <ethan@stoneleaf.us> wrote:
>>>
>>> I think warnings should be reserved for language changes and such (like
>>> DeprecationWarning, RuntimeWarning, and FutureWarning), not for possible
>>> programmer mistakes.
>>
>> I disagree, on the basis of the following:
>>
>> The quality of C code I have to deal with has increased dramatically as
>> gcc's aggressive use of warnings has spread.
>
> With gcc you pay the cost once, with Python you would pay it with every run.
>  A linter would be more along the lines of 'pay it once'.

Agreed; in addition, it's spam. Spam at compile time isn't much of an
issue; you type ./configure and make, and a billion messages scroll
past you. Spam at run time? Everyone who USES the program has to deal
with it, and in amongst the program's own output. I don't know how to
have Python emit warnings that wouldn't cause these issues. The only
thing I can think of is to have the interactive interpreter default to
showing more warnings, which is far from perfect itself.

As to 'file' specifically, my 3.2 doesn't seem to have it as a
builtin. The return value from open() is _io.TextIOWrapper, so the
whole issue of file=open(...) may have been completely dodged.

ChrisA

[toc] | [prev] | [next] | [standalone]

#11573

On 08/16/2011 10:31 AM, Philip Semanchuk wrote:
> On Aug 16, 2011, at 11:12 AM, Chris Angelico wrote:
>> There are several types of shadowing:
>>
>> 1) Deliberate shadowing because you want to change the behavior of the
>> name. Extremely rare.
>> 2) Shadowing simply by using the name of an unusual builtin (like
>> 'file') in a context where you never use it. Very common.
>> 3) Unintentional shadowing where you create a variable, but then
>> intend to use the builtin. This is the only one that's a
>> problem.
>
> Yes, but before you get to #3 you have to go through #2. The
> way I see it, #2 is setting a trap, #3 is actually stepping
> in it. I don't want to do either. Neither do I like working
> with code that has set trap #2 for me.

Chris succinctly described the times I've done shadowing. 
Fortunately, the shadowing done in #3 (which you appropriately 
describe as being a superset of #2) is fairly remedied with most 
editors...since it usually occurs when you have "oh, I 
accidentally shadowed builtin X", so you just do a global 
search-and-replace for all those places you shadowed X and rename 
it to something like "my_X" and proceed to use X as the builtin.

The bigger issue I have is module shadowing which is trickier to 
catch and produces weird symptoms (i.e. cryptic errors).  The 
most common one I see is creating a local module called 
"email.py" and then having issues when trying to use 
standard-library email calls which find your local email.py 
before they find the email.py file in the standard library.  I 
actually wrote a tool to scan for duplicate modules in 
$PYTHONPATH (pretty dumb tool, could easily have broken on things 
like zipfile imports, DLLs, etc), but it made diagnosing the 
issue easier.

-tkc

[toc] | [prev] | [next] | [standalone]

#11577

On Aug 16, 2011, at 12:19 PM, Ethan Furman wrote:

> Philip Semanchuk wrote:
>> On Aug 16, 2011, at 11:41 AM, Ethan Furman wrote:
>>> Philip Semanchuk wrote:
>>>> If we are to eschew warnings in
>>>> cases where they might be highlighting something harmless, then we would
>>>> have no warnings at all.
> >>
>>> Sounds good to me.  ;)  Keep such things in the IDE's, and then those
> >> who desire such behavior can have it there.  Do not clutter Python with
> >> such.
>> You wink, yet you sound serious. 
> 
> The smiley is an attempt to not sound harsh.

Thanks. It's hard to know on the Internet.


>>> I don't see that as a problem that Python needs to solve.
>> "need" is a strong word. Python will be fine regardless of whether this changes
> > or not. I believe Python could be improved; that's all I'm arguing.
> 
> Python can be improved -- I don't see 'hand-holding' as an improvement.  IDEs and lints can do this.

When you say "hand-holding", I hear a pejorative. That makes "I don't see 'hand-holding' as an improvement" a tautology. Have I misheard you?

I think Python does lots of beneficial hand-holding. Garbage collection is a good example. $DIETY knows, people have been struggling with manual memory management in C and its ilk for a long time. Even though there are good tools to help, memory leaks still happen. Python increases our productivity by allowing us to forget about manual memory management altogether. I can do it with tools like valgrind, but Python's makes the point moot. Is that hand-holding? If so, I'm all for it.

Cheers
Philip

[toc] | [prev] | [next] | [standalone]

#11645

Philip Semanchuk wrote:


> I think Python does lots of beneficial hand-holding. Garbage collection is
> a good example. $DIETY knows, people have been struggling with manual
> memory management in C and its ilk for a long time. Even though there are
> good tools to help, memory leaks still happen. Python increases our
> productivity by allowing us to forget about manual memory management
> altogether. I can do it with tools like valgrind, but Python's makes the
> point moot. Is that hand-holding? If so, I'm all for it.

Hand-holding is not a well-defined term. According to Mel, garbage
collectors would be hand-holding designed for soft nellies.

http://www.cs.utah.edu/~elb/folklore/mel.html

But Mel would probably make his own nails rather than buy them from the
hardware store too. To some secret recipe involving rare metals that most
people have never even heard of. And use a hammer forged from a single
piece of meteor iron, shaped to fit his hand precisely. And it would be too
heavy for mortals to lift.

Of course the term "hand-holding" is a pejorative. The implication is that
the person needing the hand-holding is like a child who needs somebody to
hold their hand while they cross the road. Adults are supposed to know
better (if only it were true in real life!). On the other hand, a real
adult knows their limitations, and if you need a bit of hand-holding in a
certain area of your life, so be it. 

Some errors aren't exactly a matter of inexperience and poor judgement, and
so everyone can benefit from guard rails, seat belts and garbage
collection. In that case, it isn't hand-holding, because there's no
expectation that you should grow up from it. Even Donald Knuth probably
would get benefit from a good garbage collector.

I have no objection to lint tools. But separation of concerns should apply:
the Python compiler should just compile what I tell it to, the linter
should warn me if I'm running with scissors.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#11650

On Aug 16, 2011, at 9:29 PM, Steven D'Aprano wrote:

> I have no objection to lint tools. But separation of concerns should apply:
> the Python compiler should just compile what I tell it to, the linter
> should warn me if I'm running with scissors.

This point (also made by Ethan) I can agree with. I haven't looked through all the warnings the Python compiler emits, but it seems like it currently doesn't dispense advice (unlike, say, gcc). It only warns about changes in the language & standard library. In that context, asking it to warn about shadowing builtins would be an expansion of scope. 

bye,
Philip

[toc] | [prev] | [next] | [standalone]

#11581

Philip Semanchuk wrote:
> I think Python does lots of beneficial hand-holding. Garbage collection
 > is a good example. $DIETY knows, people have been struggling with manual
 > memory management in C and its ilk for a long time. Even though there
 > are good tools to help, memory leaks still happen. Python increases our
 > productivity by allowing us to forget about manual memory management
 > altogether. I can do it with tools like valgrind, but Python's makes
 > the point moot. Is that hand-holding? If so, I'm all for it.

Good point.  There is an important difference, however, between offering 
warnings to newbie programmers (which is how this thread started), and 
memory management.

~Ethan~

[toc] | [prev] | [next] | [standalone]

#11583

On 2011-08-16, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> On Tue, 16 Aug 2011 01:23 pm Philip Semanchuk wrote:
>>> Why should built-ins be treated as more sacred than your own objects?
 
>> Because built-ins are described in the official documentation as having a
>> specific behavior, while my objects are not.

> *My* objects certainly are, because I write documentation for my code. My
> docs are no less official than Python's docs.

Sure they are.  I can't get yours from python.org.

> You can shadow anything. Sometimes shadowing is safe, sometimes it isn't. I
> don't see why we should necessarily fear safe shadowing of built-ins more
> than we fear unsafe shadowing of non-built-ins.

I think largely because anyone coming to your code will have expectations
about the built-ins.  For non-built-ins, they'll have to look around the code
to see what they do, but for built-ins, they come to the table with an
otherwise-reasonable expectation that they already know what that word means.

> A warning that is off by default won't help the people who need it, because
> they don't know enough to turn the warning on. A warning that is on by
> default will be helpful to the newbie programmer for the first week or so,
> and then will be nothing but an annoyance for the rest of their career.

Will it?

I am pretty sure that I'd keep it on and fix anything that triggered it,
because shadowing built-ins strikes me as Asking For Trouble.

> Protecting n00bs from their own errors is an admirable aim, but have you
> considered that warnings for something which may be harmless could do more
> harm than good?

I would distinguish between "may not be causing bugs" and "may be harmless".

I think code which shadows a built-in has a pretty real risk of being
harmful at some unspecified future point when some maintainer who hasn't
memorized every last line of the code makes the totally reasonable assumption
that basic language features are still working and available.

> Perhaps. But I'm not so sure it is worth the cost of extra code to detect
> shadowing and raise a warning. After all, the average coder probably never
> shadows anything, and for those that do, once they get bitten *once* they
> either never do it again or learn how to shadow safely. I don't see it as a
> problem.

I would guess that it happens moderately often entirely by accident.

Not a Python example, but I recently had a heck of a time with some
JavaScript I was trying to maintain.  Couldn't get it to work on a slightly
older Firefox, and the diagnostics from Firebug came across as basically
illucid.

Problem:  I was declaring an array named 'history'.

My thoughts would be:
1.  It's hard to avoid shadowing anything unless you know the entire language
and never forget things.
2.  In particular, Python likes to use clear, obvious, names for things.
Meaning that your choice of a clear, obvious, name for a similar thing
could be the name of a thing in the language.
3.  I am not sure at all that shadowing can be "safe" in code which will
some day be maintained.

The chances of someone coming along and trying to write clean, idiomatic,
Python which happens to blow up interestingly because their code runs in
an environment where part of the standard language has been shadowed
strike me as Too High.

MHO, but speaking only for myself, I'd want that warning on and I'd not
consider it harmless.

-s
-- 
Copyright 2011, all wrongs reversed.  Peter Seebach / usenet-nospam@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.

[toc] | [prev] | [next] | [standalone]

#11639

Seebs wrote:

> On 2011-08-16, Steven D'Aprano <steve+comp.lang.python@pearwood.info>
> wrote:
>> On Tue, 16 Aug 2011 01:23 pm Philip Semanchuk wrote:
>>>> Why should built-ins be treated as more sacred than your own objects?
>  
>>> Because built-ins are described in the official documentation as having
>>> a specific behavior, while my objects are not.
> 
>> *My* objects certainly are, because I write documentation for my code. My
>> docs are no less official than Python's docs.
> 
> Sure they are.  I can't get yours from python.org.

And what makes that unofficial? 

python.org is not the sole arbiter of official documentation in the world.
You can't get docs for Django or Scipy or NLTK from python.org either, but
just try telling the authors of those libraries that their docs are somehow
unofficial and see how far that gets you.


[...]
> I think code which shadows a built-in has a pretty real risk of being
> harmful at some unspecified future point when some maintainer who hasn't
> memorized every last line of the code makes the totally reasonable
> assumption that basic language features are still working and available.

Am I the only person who writes functions and methods any more? *wink*

Modern languages, and by modern I mean most languages more than, oh, about
fifty years old, provide ways to limit the scope of variables. You don't
need to memorise "every last line of the code" to safely edit a function.

def process(list, arg):
    spam(list)
    ham(arg)
    cheese(list, arg)

The scope of parameter "list" is limited to within the function process
itself. Inside, it shadows the built-in list. Outside, it doesn't do squat.


[...]
> My thoughts would be:
> 1.  It's hard to avoid shadowing anything unless you know the entire
> language and never forget things.

Define the "entire language". Does that include the names of all the
plethora of exceptions? How about the standard library?

For what it's worth, I don't care about memorising all the built-ins. I
delegate that job to my editor, which has a syntax highlighter for Python.
It never forgets built-ins. (In fact, sometimes this is a nuisance. When
I'm editing Python 3 code, it still insists that apply and reduce are
built-ins.)



> 2.  In particular, Python likes to use clear, obvious, names for things.
> Meaning that your choice of a clear, obvious, name for a similar thing
> could be the name of a thing in the language.

Yes, and Python also encourages the use of scopes, so that the clear,
obvious name for something in one scope does not clash with the clear,
obvious, identical name for something completely different in another
scope.


> 3.  I am not sure at all that shadowing can be "safe" in code which will
> some day be maintained.

Oh there's no doubt that shadowing *can* be unsafe. But then, very few
things can't be abused.

As I see it, there are (at least) four stages related to shadowing.

(1) At first, you don't even know enough to be concerned by shadowing. You
blithely call variables "list" and "str" without a care in the world...
until something goes wrong.


(2) Second stage, you know enough to realise that shadowing can be bad. You
avoid shadowing everything. Your code is full of variables called "my_list"
and "astr" and "some_tuple". You call your functions things like "izip"
even though it is designed as a replacement for zip, because the caller
might use from itertools import * and accidentally replace the built-in zip
with my zip. 

You even avoid using "string" as a variable name because it might shadow the
string module -- even though you haven't actually imported or used the
string module in the last four years.


(3) Eventually, you get frustrated writing doc strings like this:

    def function(astr, myint=0):
        """function(astr [, myint]) -> list

        Arguments:
        astr - string
        myint - integer

        Do something tool-like to a string and optional integer...
        """

and begin writing them like this:

    def function(astr, myint):
        """function(str [, int]) -> list

        Do something tool-like to a string and optional integer...
        """


(4) And then, after a while, you decide that perhaps shadowing is not always
so bad. (It helps if the Effbot tells you off for objecting to shadowing in
a two-line function.) At first, you limit yourself to using parameter names
that shadow built-ins in small tool-like functions. Then when the world
doesn't implode, you rethink the use of top-level function names
like "izip" and realise that namespaces exist so that you don't need to
care about shadowing functions you don't use, and if people call import *
they have nobody to blame but themselves if things break.



More-or-less-auto-biographically-ly y'rs,


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#11653

On 2011-08-17, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> Seebs wrote:
>> On 2011-08-16, Steven D'Aprano <steve+comp.lang.python@pearwood.info>
>> wrote:
>>> *My* objects certainly are, because I write documentation for my code. My
>>> docs are no less official than Python's docs.

>> Sure they are.  I can't get yours from python.org.

> And what makes that unofficial? 

> python.org is not the sole arbiter of official documentation in the world.

But it is the sole arbiter of official *Python* documentation.  If I am
looking at code in Python, I reasonably expect the Python docs to be
correct about that code.  Incomplete, sure, but *correct*.

>> I think code which shadows a built-in has a pretty real risk of being
>> harmful at some unspecified future point when some maintainer who hasn't
>> memorized every last line of the code makes the totally reasonable
>> assumption that basic language features are still working and available.

> Am I the only person who writes functions and methods any more? *wink*

Yes.  Everyone else converted to single giant blocks of code because they
are easier to develop.  :P

> Modern languages, and by modern I mean most languages more than, oh, about
> fifty years old, provide ways to limit the scope of variables. You don't
> need to memorise "every last line of the code" to safely edit a function.

Only the parts that are in scope, but...

I seem to recall (and I'm pretty Python newbiesh, so I could be wrong)
that Python classes create a scope in which bits of the class become
visible, and some classes are sorta biggish.

> def process(list, arg):
>     spam(list)
>     ham(arg)
>     cheese(list, arg)

> The scope of parameter "list" is limited to within the function process
> itself. Inside, it shadows the built-in list. Outside, it doesn't do squat.

Yes.  But what about the built-in spam?  :)

> Define the "entire language". Does that include the names of all the
> plethora of exceptions? How about the standard library?

I'd think "standard library" or close to it.

> For what it's worth, I don't care about memorising all the built-ins. I
> delegate that job to my editor, which has a syntax highlighter for Python.
> It never forgets built-ins. (In fact, sometimes this is a nuisance. When
> I'm editing Python 3 code, it still insists that apply and reduce are
> built-ins.)

Heh.

I mostly don't use syntax highlighters; at best, they distract me, at worst,
they distract me a lot.  I also don't use one in English, although I am
sure some people would love to have nouns in blue and punctuation in green.

> Yes, and Python also encourages the use of scopes, so that the clear,
> obvious name for something in one scope does not clash with the clear,
> obvious, identical name for something completely different in another
> scope.

"Another" scope is normally a horizontal thing -- you're talking about
a different scope such that you are *either* in this one *or* in that
one.

Built-ins are not in a scope you are never not in.

> Oh there's no doubt that shadowing *can* be unsafe. But then, very few
> things can't be abused.

Yup.

> As I see it, there are (at least) four stages related to shadowing.

> (1) At first, you don't even know enough to be concerned by shadowing. You
> blithely call variables "list" and "str" without a care in the world...
> until something goes wrong.

> (2) Second stage, you know enough to realise that shadowing can be bad. You
> avoid shadowing everything. Your code is full of variables called "my_list"
> and "astr" and "some_tuple". You call your functions things like "izip"
> even though it is designed as a replacement for zip, because the caller
> might use from itertools import * and accidentally replace the built-in zip
> with my zip. 

> You even avoid using "string" as a variable name because it might shadow the
> string module -- even though you haven't actually imported or used the
> string module in the last four years.

Heh.  (I got advised by pylint not to grab something from it, but I no
longer remember why; I seem to recall being totally unable to find a way
to avoid that warning and still have the string processing I needed.)

> (3) Eventually, you get frustrated writing doc strings like this:
>
>     def function(astr, myint=0):
>         """function(astr [, myint]) -> list
>
>         Arguments:
>         astr - string
>         myint - integer
>
>         Do something tool-like to a string and optional integer...
>         """

> and begin writing them like this:

>     def function(astr, myint):
>         """function(str [, int]) -> list
>
>         Do something tool-like to a string and optional integer...
>         """


That seems safe enough to me.  :)

> (4) And then, after a while, you decide that perhaps shadowing is not always
> so bad. (It helps if the Effbot tells you off for objecting to shadowing in
> a two-line function.) At first, you limit yourself to using parameter names
> that shadow built-ins in small tool-like functions. Then when the world
> doesn't implode, you rethink the use of top-level function names
> like "izip" and realise that namespaces exist so that you don't need to
> care about shadowing functions you don't use, and if people call import *
> they have nobody to blame but themselves if things break.

Hmm.  See, I've never reached that, in Python or any other language.  I
figure it creates a new potential for confusion, and that I would rather
avoid any ambiguity.  I don't *like* ambiguity in code.

So I don't shadow stuff that's part of the language, because doing that
makes it possible for a line of code to have a clear and obvious meaning to
someone who looks at that line out of context, and a *completely different*
clear and obvious meaning to someone who looks at it with a bit more
context.  I don't like that!  It means that someone reading my code can
never, ever, assume that a standard language feature is actually itself
and not a local shadow which does something different unless they go
walking back up the scope chain checking.  And that's a pretty big cost
to attach to stuff that is, by design, basic and universally available.

I might feel differently about names which were only visible after you
imported some specific thing, but if we're talking built-ins that are
always visible...

Hmm.  No, actually, even then, I just think it's Bad Mojo to overlap
names that were taken by the language itself.  If a name is in a different
scope which I'm not in, I don't care about it, but if the name would already
be in scope if I didn't declare it, then I am creating a nasty ambiguity
if I use it.

-s
-- 
Copyright 2011, all wrongs reversed.  Peter Seebach / usenet-nospam@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.

[toc] | [prev] | [next] | [standalone]

#11665

On Wed, 17 Aug 2011 01:17 pm Seebs wrote:

[...]
> "Another" scope is normally a horizontal thing -- you're talking about
> a different scope such that you are *either* in this one *or* in that
> one.
> 
> Built-ins are not in a scope you are never not in.

That's technically incorrect. Built-ins are a scope you are never in, if
by "in" you mean "code is executed in this scope".

You have three scopes in general:

Local
Global
Built-ins

(There's also zero or more nonlocal scopes, between local and global, that
applies to closures and nested functions, but never mind that.)

Code is almost(?) always executed in the local scope. (eval and exec let you
mess around with that, somewhat.) E.g. an assignment "x = 1" applies to the
local namespace unless you explicitly declare it global. If you are in the
top level of a module, the local namespace is also the global one, the
global statement does nothing, and the assignment occurs in the global
namespace.

However, name lookup rules are such that while assignments are always local,
unsuccessful lookups may fall through to global then built-in. 

See also: http://www.python.org/dev/peps/pep-0227/

The only way to put something into the built-in namespace is by using the
fully-qualified name:

>>> import builtins  # Use __builtin__ in Python 2
>>> builtins.x = 1  # Kids! Don't try this at home!
>>>
>>> x
1
>>> del x  # prove it isn't a global or local
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined

So a top-level (global) assignment, say:

def sum(list):
    ...

*shadows* the built-ins sum and list, it doesn't replace them. It defines a
local variable "list" and a global variable "sum", but it doesn't touch
either built-in. They are still available via the fully qualified name
builtins.sum and builtins.list.


[...]
>> (4) And then, after a while, you decide that perhaps shadowing is not
>> always so bad. (It helps if the Effbot tells you off for objecting to
>> shadowing in a two-line function.) At first, you limit yourself to using
>> parameter names that shadow built-ins in small tool-like functions. Then
>> when the world doesn't implode, you rethink the use of top-level function
>> names like "izip" and realise that namespaces exist so that you don't
>> need to care about shadowing functions you don't use, and if people call
>> import * they have nobody to blame but themselves if things break.
> 
> Hmm.  See, I've never reached that, in Python or any other language.  I
> figure it creates a new potential for confusion, and that I would rather
> avoid any ambiguity.  I don't *like* ambiguity in code.

Ah, well you see the thing is, this is Python. As soon as you call any
function you don't control, you no longer know what your environment is
with any certainty. For all you know, the harmless-looking function is
monkey-patching builtins like there's no tomorrow. Speaking broadly,
perhaps too broadly, we're incredibly proud of the ability to modify nearly
everything at runtime.

Fortunately, while we are proud of having that ability, actually *using* it
is considered a mortal sin. We're not Ruby developers -- if you actually
monkey-patch something, especially built-ins, you can expect to be taken
outside and slapped around with a fish if you get caught.

http://www.youtube.com/watch?v=IhJQp-q1Y1s

 
> So I don't shadow stuff that's part of the language, because doing that
> makes it possible for a line of code to have a clear and obvious meaning
> to someone who looks at that line out of context, and a *completely
> different* clear and obvious meaning to someone who looks at it with a bit
> more
> context.  I don't like that!  It means that someone reading my code can
> never, ever, assume that a standard language feature is actually itself
> and not a local shadow which does something different unless they go
> walking back up the scope chain checking.  And that's a pretty big cost
> to attach to stuff that is, by design, basic and universally available.

Sure. But they can't have that certainty regardless of whether you shadow
something, because how do they know whether you've shadowed it or not?

In theory, anything could be anything at any time, and you have no
protection. In practice, I worry more about being eaten by
genetically-engineered flying piranhas than about rogue monkey-patching
code.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#11695

On 2011-08-17, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> On Wed, 17 Aug 2011 01:17 pm Seebs wrote:
> [...]
>> "Another" scope is normally a horizontal thing -- you're talking about
>> a different scope such that you are *either* in this one *or* in that
>> one.

>> Built-ins are not in a scope you are never not in.

> That's technically incorrect. Built-ins are a scope you are never in, if
> by "in" you mean "code is executed in this scope".

No, by "in" I mean "lookups from your code will reach this scope if they
don't find something sooner".

>> Hmm.  See, I've never reached that, in Python or any other language.  I
>> figure it creates a new potential for confusion, and that I would rather
>> avoid any ambiguity.  I don't *like* ambiguity in code.

> Ah, well you see the thing is, this is Python. As soon as you call any
> function you don't control, you no longer know what your environment is
> with any certainty. For all you know, the harmless-looking function is
> monkey-patching builtins like there's no tomorrow. Speaking broadly,
> perhaps too broadly, we're incredibly proud of the ability to modify nearly
> everything at runtime.

Heh.

> Fortunately, while we are proud of having that ability, actually *using* it
> is considered a mortal sin. We're not Ruby developers -- if you actually
> monkey-patch something, especially built-ins, you can expect to be taken
> outside and slapped around with a fish if you get caught.

Okay, so.

Here's what I don't get.

If it's such a bad thing, *why is it allowed*?  Why are you proud of the
ability to do something that you are never socially-allowed to do?

I have mixed feelings about Ruby's general tolerance for monkey-patching,
although I've seen it do some pretty awesome things, so I am somewhat
inclined to accept that it may be worth it.  But it's clearly a feature
which is used intentionally.

It sounds to me like Python is very proud of having a feature which no
one would ever use.  ... Why?

> Sure. But they can't have that certainty regardless of whether you shadow
> something, because how do they know whether you've shadowed it or not?

Well, they could trust me.  :)

> In theory, anything could be anything at any time, and you have no
> protection. In practice, I worry more about being eaten by
> genetically-engineered flying piranhas than about rogue monkey-patching
> code.

I do too, if I know that people don't shadow built-ins.  If I know that
people are shadowing built-ins, then some of the time, when I'm editing
other peoples' code, they HAVE effectively monkey-patched it.

-s
-- 
Copyright 2011, all wrongs reversed.  Peter Seebach / usenet-nospam@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.

[toc] | [prev] | [next] | [standalone]

#11700

Seebs wrote:
> On 2011-08-17, Steven D'Aprano wrote:
>> On Wed, 17 Aug 2011 01:17 pm Seebs wrote:
>>> Hmm.  See, I've never reached that, in Python or any other language.  I
>>> figure it creates a new potential for confusion, and that I would rather
>>> avoid any ambiguity.  I don't *like* ambiguity in code.
> 
>> Ah, well you see the thing is, this is Python. As soon as you call any
>> function you don't control, you no longer know what your environment is
>> with any certainty. For all you know, the harmless-looking function is
>> monkey-patching builtins like there's no tomorrow. Speaking broadly,
>> perhaps too broadly, we're incredibly proud of the ability to modify nearly
>> everything at runtime.
> 
> Heh.
> 
>> Fortunately, while we are proud of having that ability, actually *using* it
>> is considered a mortal sin. We're not Ruby developers -- if you actually
>> monkey-patch something, especially built-ins, you can expect to be taken
>> outside and slapped around with a fish if you get caught.
> 
> Okay, so.
> 
> Here's what I don't get.
> 
> If it's such a bad thing, *why is it allowed*?  Why are you proud of the
> ability to do something that you are never socially-allowed to do?

Monkey-patching built-ins would be something along the lines of

     import sys
     sys.modules['__builtin__'].str = my_super_string

and is what stands you in jeopardy of being fish-slapped.  ;)

Merely shadowing a built-in, or stdlib, or whatever, isn't monkey-patching.


~Ethan~

[toc] | [prev] | [next] | [standalone]

#11703

On 2011-08-17, Ethan Furman <ethan@stoneleaf.us> wrote:
> Seebs wrote:
>> On 2011-08-17, Steven D'Aprano wrote:
>>> Ah, well you see the thing is, this is Python. As soon as you call any
>>> function you don't control, you no longer know what your environment is
>>> with any certainty. For all you know, the harmless-looking function is
>>> monkey-patching builtins like there's no tomorrow. Speaking broadly,
>>> perhaps too broadly, we're incredibly proud of the ability to modify nearly
>>> everything at runtime.

>>> Fortunately, while we are proud of having that ability, actually *using* it
>>> is considered a mortal sin. We're not Ruby developers -- if you actually
>>> monkey-patch something, especially built-ins, you can expect to be taken
>>> outside and slapped around with a fish if you get caught.

>> Here's what I don't get.

>> If it's such a bad thing, *why is it allowed*?  Why are you proud of the
>> ability to do something that you are never socially-allowed to do?

> Monkey-patching built-ins would be something along the lines of

>      import sys
>      sys.modules['__builtin__'].str = my_super_string

> and is what stands you in jeopardy of being fish-slapped.  ;)

> Merely shadowing a built-in, or stdlib, or whatever, isn't monkey-patching.

Oh, I know.  I was just noticing that Steven's post is specifically talking
about how Python users are proud of having the ability to monkey-patch.

If monkey-patching like that is a mortal sin, leads to fish-slapping, and
so on..

Why is it possible?  Why not just reject such things as invalid code?

-s
-- 
Copyright 2011, all wrongs reversed.  Peter Seebach / usenet-nospam@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.

[toc] | [prev] | [next] | [standalone]

#11706

Seebs wrote:
> On 2011-08-17, Ethan Furman <ethan@stoneleaf.us> wrote:
>> Seebs wrote:
>>> On 2011-08-17, Steven D'Aprano wrote:
>>>> Ah, well you see the thing is, this is Python. As soon as you call any
>>>> function you don't control, you no longer know what your environment is
>>>> with any certainty. For all you know, the harmless-looking function is
>>>> monkey-patching builtins like there's no tomorrow. Speaking broadly,
>>>> perhaps too broadly, we're incredibly proud of the ability to modify nearly
>>>> everything at runtime.
> 
>>>> Fortunately, while we are proud of having that ability, actually *using* it
>>>> is considered a mortal sin. We're not Ruby developers -- if you actually
>>>> monkey-patch something, especially built-ins, you can expect to be taken
>>>> outside and slapped around with a fish if you get caught.
> 
>>> Here's what I don't get.
> 
>>> If it's such a bad thing, *why is it allowed*?  Why are you proud of the
>>> ability to do something that you are never socially-allowed to do?
> 
>> Monkey-patching built-ins would be something along the lines of
> 
>>      import sys
>>      sys.modules['__builtin__'].str = my_super_string
> 
>> and is what stands you in jeopardy of being fish-slapped.  ;)
> 
>> Merely shadowing a built-in, or stdlib, or whatever, isn't monkey-patching.
> 
> Oh, I know.  I was just noticing that Steven's post is specifically talking
> about how Python users are proud of having the ability to monkey-patch.
> 
> If monkey-patching like that is a mortal sin, leads to fish-slapping, and
> so on..
> 
> Why is it possible?  Why not just reject such things as invalid code?

Well, the mortal sin part is a bit of an exaggeration -- it's more like 
you'd better have a really darn good reason to do it.  And it is 
absolutely one of my favorite parts about Python.  If I want to inject a 
custom Path class into builtins so it's readily available, and then 
change os.listdir to return it instead of normal strings, I can.  If my 
application is truly case-insensitive, I can make my own istr class and 
monkey-patch builtins so it's what is used.  Can this blow-up in my 
face?  Certainly.  But I would rather have the option open to me instead 
of being told "No, I'm sorry, you can't do that because I (developers in 
question) didn't imagine a good use case for it".

Part of the fun of Python is experimentation.  And how much fun is it to 
be told over and over, "No, you can't do that"?

As an example of something that could easily have been outlawed, but 
wasn't, check out http://stackoverflow.com/questions/7068340

~Ethan~

[toc] | [prev] | [next] | [standalone]

#11708

On 2011-08-17, Ethan Furman <ethan@stoneleaf.us> wrote:
> Part of the fun of Python is experimentation.  And how much fun is it to 
> be told over and over, "No, you can't do that"?

Okay, I buy that.

Actually, this sort of fits with my experience of how (sane) people do it
in Ruby.

And I'm really the wrong person to criticize people for monkey-patching:

	http://www.yoctoproject.org/projects/pseudo

(In which I monkeypatch *the entire Unix filesystem API*.  On Linux and
OS X.  For C programs.)

I guess maybe my question shouldn't have been "why is it allowed", but "and
why is it bad to use it?"  It just seemed like "you should never do this,
it's horrible" and "we're proud of being able to do this" were not entirely
compatible.

-s
-- 
Copyright 2011, all wrongs reversed.  Peter Seebach / usenet-nospam@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web