Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #110609 > unrolled thread

__all__ attribute: bug and proposal

Back to article view | Back to comp.lang.python

Contents

  __all__ attribute: bug and proposal Pavel S <pavel@schon.cz> - 2016-06-27 13:56 -0700
    Re: __all__ attribute: bug and proposal Chris Angelico <rosuav@gmail.com> - 2016-06-28 07:32 +1000
      Re: __all__ attribute: bug and proposal Pavel S <pavel@schon.cz> - 2016-06-27 15:29 -0700
        Re: __all__ attribute: bug and proposal Chris Angelico <rosuav@gmail.com> - 2016-06-28 08:41 +1000
        Re: __all__ attribute: bug and proposal Terry Reedy <tjreedy@udel.edu> - 2016-06-27 22:46 -0400
          Re: __all__ attribute: bug and proposal Steven D'Aprano <steve@pearwood.info> - 2016-06-28 13:24 +1000
    Re: __all__ attribute: bug and proposal Random832 <random832@fastmail.com> - 2016-06-27 20:25 -0400
    Re: __all__ attribute: bug and proposal Chris Angelico <rosuav@gmail.com> - 2016-06-28 10:32 +1000
    Re: __all__ attribute: bug and proposal MRAB <python@mrabarnett.plus.com> - 2016-06-28 02:55 +0100
    Re: __all__ attribute: bug and proposal Steven D'Aprano <steve@pearwood.info> - 2016-06-28 11:56 +1000
    Re: __all__ attribute: bug and proposal Chris Angelico <rosuav@gmail.com> - 2016-06-28 12:20 +1000
    Re: __all__ attribute: bug and proposal Random832 <random832@fastmail.com> - 2016-06-27 22:27 -0400
    Re: __all__ attribute: bug and proposal Zachary Ware <zachary.ware+pylist@gmail.com> - 2016-06-27 23:31 -0500
    Re: __all__ attribute: bug and proposal Ethan Furman <ethan@stoneleaf.us> - 2016-06-28 08:29 -0700

#110609 — __all__ attribute: bug and proposal

Hi,
I today uncovered subtle bug and would like to share it with you.

By a mistake, I forgot to put comma into '__all__' tuple of some module. Notice missing comma after 'B'.

# module foo.py
__all__ = (
    'A',
    'B'
    'C',
)

class A: pass
class B: pass
class C: pass

If you try to import * from the module, it will raise an error, because 'B' and 'C' will be concatenated into 'BC'.

>>> from foo import *
AttributeError: 'module' object has no attribute 'BC'

The bug won't be found until someone imports *.
In order to identify problems as soon as possible, here's the proposal.

Porposal: allow putting objects into __all__ directly, so possible problems will be found earlier:

# module foo.py
class A: pass
class B: pass
class C: pass

__all__ = (A, B, C)

Note: this currently don't work.

>>> from foo import *
TypeError: attribute name must be string, not 'type'

[toc] | [next] | [standalone]

#110611

On Tue, Jun 28, 2016 at 6:56 AM, Pavel S <pavel@schon.cz> wrote:
> Porposal: allow putting objects into __all__ directly, so possible problems will be found earlier:
>
> # module foo.py
> class A: pass
> class B: pass
> class C: pass
>
> __all__ = (A, B, C)
>
> Note: this currently don't work.
>
>>>> from foo import *
> TypeError: attribute name must be string, not 'type'

How would it know what names to import them under? It's all very well
with classes and functions (you could mandate that it uses the
canonical name), but what about integers or strings?

ChrisA

[toc] | [prev] | [next] | [standalone]

#110613

> but what about integers or strings?

Can you provide example?

---
No matter if __all__ uses names or objects, I think it should be validated not only when importing '*', but always.

Frankly, do you always unit-test if __all__ works?

[toc] | [prev] | [next] | [standalone]

#110614

On Tue, Jun 28, 2016 at 8:29 AM, Pavel S <pavel@schon.cz> wrote:
>> but what about integers or strings?
>
> Can you provide example?

Sure. I'll spare you the wall of text that is os.__all__, but here's the types:

>>> import os
>>> {type(getattr(os, x)).__name__ for x in os.__all__}
{'bool', 'module', 'function', 'str', '_Environ', 'NoneType', 'type',
'int', 'dict', 'builtin_function_or_method'}
>>> collections.Counter(hasattr(getattr(os, x), "__name__") for x in os.__all__)
Counter({True: 184, False: 122})

So, roughly 60% of them even have canonical names. And you don't
always want the canonical name, anyway:

>>> os.path.__name__
'posixpath'

(Your results may vary. Specifically. os.path.__name__ will be
'ntpath' if you're on Windows.)

> No matter if __all__ uses names or objects, I think it should be validated not only when importing '*', but always.
>
> Frankly, do you always unit-test if __all__ works?

Frankly, I don't unit test enough. But if you have __all__, it
shouldn't be too hard, nor too unobvious, to test it.

>>> def test_all():
...   exec("from os import *")
...
>>> test_all()
>>> os.__all__.append("asdf")
>>> test_all()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in test_all
  File "<string>", line 1, in <module>
AttributeError: module 'os' has no attribute 'asdf'

ChrisA

[toc] | [prev] | [next] | [standalone]

#110637

On 6/27/2016 6:29 PM, Pavel S wrote:

> Frankly, do you always unit-test if __all__ works?

One should.  CPython's test suite includes test___all__.  I believe it 
imports every stdlib module, looks for __all__, and if present, tests 
that it works.  It probably executes 'from module import *'.  A side 
effect is testing that everything imports.  I am pretty sure that 
test___all__ has caught bugs.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#110639

On Tue, 28 Jun 2016 12:46 pm, Terry Reedy wrote:

> On 6/27/2016 6:29 PM, Pavel S wrote:
> 
>> Frankly, do you always unit-test if __all__ works?
> 
> One should.  CPython's test suite includes test___all__.  I believe it
> imports every stdlib module, looks for __all__, and if present, tests
> that it works.  It probably executes 'from module import *'.  A side
> effect is testing that everything imports.  I am pretty sure that
> test___all__ has caught bugs.

I write my own unit tests for __all__, which has certainly caught bugs.



-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[toc] | [prev] | [next] | [standalone]

#110622

On Mon, Jun 27, 2016, at 16:56, Pavel S wrote:
> Porposal: allow putting objects into __all__ directly, so possible
> problems will be found earlier:

Question: What happens if the object in __all__ isn't the same object
that's in the module?

[toc] | [prev] | [next] | [standalone]

#110624

On Tue, Jun 28, 2016 at 6:56 AM, Pavel S <pavel@schon.cz> wrote:
> By a mistake, I forgot to put comma into '__all__' tuple of some module. Notice missing comma after 'B'.
>
> # module foo.py
> __all__ = (
>     'A',
>     'B'
>     'C',
> )
>
> class A: pass
> class B: pass
> class C: pass
>
> If you try to import * from the module, it will raise an error, because 'B' and 'C' will be concatenated into 'BC'.
>
>>>> from foo import *
> AttributeError: 'module' object has no attribute 'BC'
>
> The bug won't be found until someone imports *.

If you're primarily worried about classes and functions, here's a neat
trick you can use:

__all__ = []
def all(thing):
    __all__.append(thing.__name__)
    return thing

@all
class A: pass
@all
class B: pass
@all
class C: pass
@all
def d(): pass

del all # clean up the namespace (optional)


The decorator doesn't change anything (it returns its argument as-is),
but it captures the canonical name into __all__. Obviously you can't
use this if you want a non-canonical name, and you can't use it for
anything other than classes and functions (you can't decorate "pi =
3.14159"), but it might help with your actual problem.

ChrisA

[toc] | [prev] | [next] | [standalone]

#110631

On 2016-06-28 01:32, Chris Angelico wrote:
[snip]

> If you're primarily worried about classes and functions, here's a neat
> trick you can use:
>
> __all__ = []
> def all(thing):
>     __all__.append(thing.__name__)
>     return thing
>
Err... won't that hide the 'all' builtin?

[toc] | [prev] | [next] | [standalone]

#110632

On Tue, 28 Jun 2016 06:56 am, Pavel S wrote:

> Hi,
> I today uncovered subtle bug and would like to share it with you.
> 
> By a mistake, I forgot to put comma into '__all__' tuple of some module.
> Notice missing comma after 'B'.
> 
> # module foo.py
> __all__ = (
>     'A',
>     'B'
>     'C',
> )

Right. That's a language feature: implicit concatenation of string literals.
If you have two string literals separated by nothing but whitespace, the
compiler will concatenate them:

s = 'He said, "Hello, did you see where' " they're" ' going?"'

is equivalent to:

s = 'He said, "Hello, did you see where they\'re going?"'


> If you try to import * from the module, it will raise an error, because
> 'B' and 'C' will be concatenated into 'BC'.

Correct. Exactly the same as if you had written:

__all__ = ['A', 'BC', 'D']


> The bug won't be found until someone imports *.

I always write a unit-test to check that everything in __all__ exists.


> In order to identify problems as soon as possible, here's the proposal.
> 
> Porposal: allow putting objects into __all__ directly, so possible
> problems will be found earlier:
> 
> # module foo.py
> class A: pass
> class B: pass
> class C: pass
> 
> __all__ = (A, B, C)

There are two problems with this idea.

Suppose you have this:


prefs_file = "filename"
__all__ = (prefs_file, A, B, C)  # suppose A, B and C are all defined


Obviously, the intention is that the caller can say:

from themodule import *
print(prefs_file)

and "filename" will be printed. But there's two problems:

(1) the __all__ tuple doesn't know anything about the name "prefs_file". It
only knows about "filename", the object (value) of prefs_file. So there is
no way for the import system to know what name to use for that value:

????? = "filename"


(2) For backwards-compatibility, we still need to support the old way of
writing __all__, using names given as strings:

__all__ = ('prefs_file', 'A', 'B', 'C')

But now you have an ambiguity. If __all__ looks like this:

__all__ = (prefs_file, A, B, C)

does the first entry mean "import the value 'filename'" (new behaviour), or
does it mean "import the value with the name 'filename'" (old behaviour)?




-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[toc] | [prev] | [next] | [standalone]

#110633

On Tue, Jun 28, 2016 at 11:55 AM, MRAB <python@mrabarnett.plus.com> wrote:
> On 2016-06-28 01:32, Chris Angelico wrote:
> [snip]
>
>> If you're primarily worried about classes and functions, here's a neat
>> trick you can use:
>>
>> __all__ = []
>> def all(thing):
>>     __all__.append(thing.__name__)
>>     return thing
>>
> Err... won't that hide the 'all' builtin?
>

Yep. So? If you're not using it *at top level*, it's not a problem. I
clean up the namespace at the end, so if you use it inside a function,
it'll still be available. And most of my scripts don't use 'all'
anyway... it's no worse than using 'id' for a database ID.

ChrisA

[toc] | [prev] | [next] | [standalone]

#110635

On Mon, Jun 27, 2016, at 20:32, Chris Angelico wrote:
> If you're primarily worried about classes and functions, here's a neat
> trick you can use:

How about just

__all__ = [list of stuff including classes and functions]
__all__ = [x if isinstance(x, str) else x.__name__ for x in __all__]

[toc] | [prev] | [next] | [standalone]

#110643

On Mon, Jun 27, 2016 at 7:32 PM, Chris Angelico <rosuav@gmail.com> wrote:
> If you're primarily worried about classes and functions, here's a neat
> trick you can use:
>
> __all__ = []
> def all(thing):
>     __all__.append(thing.__name__)
>     return thing

Barry Warsaw has written a nice decorator called 'public' that can be
installed from PyPI as 'atpublic'[0].  It was proposed for inclusion
as a builtin in 3.6 [1], but a less-than-enthusiastic response at the
2016 Language Summit has put that off indefinitely.  I, for one, would
like to see it happen anyway [2], and public support may make it
possible.

The '@public' decorator works like so:

"""
# spam.py

@public
def spam():
    return ' '.join(['spam']*10)

@public
def eggs():
    return 'bacon'

public(HAM=4)
assert HAM == 4

assert sorted(__all__) == sorted(['spam', 'eggs', 'HAM'])
"""

This strikes me as being a cleaner approach than what the OP
suggested, easier to use than Chris' simple decorator, and a nice way
to bring an end to outdated __all__ lists (which is a problem
currently plaguing the standard library, and probably many other
libraries).

[0] https://pypi.python.org/pypi/atpublic
[1] https://bugs.python.org/issue26632
[2] https://bugs.python.org/issue26632#msg267305

--
Zach

[toc] | [prev] | [next] | [standalone]

#110711

On 06/27/2016 09:31 PM, Zachary Ware wrote:
> On Mon, Jun 27, 2016 at 7:32 PM, Chris Angelico wrote:

>> If you're primarily worried about classes and functions, here's a neat
>> trick you can use:
>>
>> __all__ = []
>> def all(thing):
>>      __all__.append(thing.__name__)
>>      return thing
>
> Barry Warsaw has written a nice decorator called 'public' that can be
> installed from PyPI as 'atpublic'[0].  It was proposed for inclusion
> as a builtin in 3.6 [1], but a less-than-enthusiastic response at the
> 2016 Language Summit has put that off indefinitely.  I, for one, would
> like to see it happen anyway [2], and public support may make it
> possible.
>
> The '@public' decorator works like so:
>
> """
> # spam.py
>
> @public
> def spam():
>      return ' '.join(['spam']*10)
>
> @public
> def eggs():
>      return 'bacon'
>
> public(HAM=4)
> assert HAM == 4
>
> assert sorted(__all__) == sorted(['spam', 'eggs', 'HAM'])
> """
>
> This strikes me as being a cleaner approach than what the OP
> suggested, easier to use than Chris' simple decorator, and a nice way
> to bring an end to outdated __all__ lists (which is a problem
> currently plaguing the standard library, and probably many other
> libraries).

I agree.  But lets all do:

   from atpublic import public as api

as using 'public' for the name is bound to lead to confusion with other 
languages that use public to mean something else, and __all__ is really 
defining the public _API_ of that module.

--
~Ethan~

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web