Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #75361 > unrolled thread

Getting a list of all modules

Back to article view | Back to comp.lang.python

Contents

  Getting a list of all modules Steven D'Aprano <steve@pearwood.info> - 2014-07-30 07:43 +0000
    Re: Getting a list of all modules Chris Angelico <rosuav@gmail.com> - 2014-07-30 18:33 +1000
    Re: Getting a list of all modules Peter Otten <__peter__@web.de> - 2014-07-30 10:46 +0200
    Re: Getting a list of all modules Robert Kern <robert.kern@gmail.com> - 2014-07-30 11:35 +0100
    Re: Getting a list of all modules Leo Jay <python.leojay@gmail.com> - 2014-07-30 21:22 +0800
      Re: Getting a list of all modules Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-07-31 10:41 +0000
        Re: Getting a list of all modules Roy Smith <roy@panix.com> - 2014-07-31 07:55 -0400
          Re: Getting a list of all modules Chris Angelico <rosuav@gmail.com> - 2014-07-31 23:13 +1000
        Re: Getting a list of all modules Terry Reedy <tjreedy@udel.edu> - 2014-07-31 15:59 -0400
        Re: Getting a list of all modules Robert Kern <robert.kern@gmail.com> - 2014-08-01 14:39 +0100
          Re: Getting a list of all modules Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-08-01 15:41 +0000
            Re: Getting a list of all modules jonnicoll11@googlemail.com - 2014-08-02 14:13 -0700
              Re: Getting a list of all modules Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-08-02 22:22 +0100
            Re: Getting a list of all modules Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-08-03 14:38 +0100
    Re: Getting a list of all modules Chris Angelico <rosuav@gmail.com> - 2014-07-30 23:27 +1000
    Re: Getting a list of all modules wxjmfauth@gmail.com - 2014-07-30 06:42 -0700
    Re: Getting a list of all modules Ian Kelly <ian.g.kelly@gmail.com> - 2014-07-30 08:29 -0600
    Re: Getting a list of all modules Skip Montanaro <skip@pobox.com> - 2014-07-30 09:45 -0500
    Re: Getting a list of all modules Akira Li <4kir4.1i@gmail.com> - 2014-07-31 22:55 +0400
    Bug with help (was Re: Getting a list of all modules) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-07-31 20:19 +0100
    Re: Bug with help (was Re: Getting a list of all modules) Terry Reedy <tjreedy@udel.edu> - 2014-07-31 16:15 -0400
    Re: Bug with help (was Re: Getting a list of all modules) Akira Li <4kir4.1i@gmail.com> - 2014-08-02 06:17 +0400
      Re: Bug with help (was Re: Getting a list of all modules) Heinz Schmitz <HeinzSchmitz@gmx.net> - 2014-08-02 10:33 +0200
        Re: Bug with help (was Re: Getting a list of all modules) Robert Kern <robert.kern@gmail.com> - 2014-08-02 13:13 +0100
        Re: Bug with help (was Re: Getting a list of all modules) Terry Reedy <tjreedy@udel.edu> - 2014-08-02 14:17 -0400

Page 1 of 2  [1] 2  Next page →

#75361 — Getting a list of all modules

I'm looking for a programmatic way to get a list of all Python modules 
and packages. Not just those already imported, but all those which 
*could* be imported.

I have a quick-and-dirty function which half does the job:


def get_modules():
    extensions = ('.py', '.pyc', '.pyo', '.so', '.dll')
    matches = set()
    for location in sys.path:
        if location == '': location = '.'
        if os.path.isdir(location):
            for name in os.listdir(location):
                base, ext = os.path.splitext(name)
                if ext in extensions:
                    matches.add(base)
    return sorted(matches)



but I know it's wrong (it doesn't handle packages correctly, or zip 
files, doesn't follow .pth files, has a very naive understanding of cross-
platform issues, fails to include built-in modules that don't live in the 
file system, and probably more).

Is this problem already solved? Can anyone make any suggestions?



-- 
Steven

[toc] | [next] | [standalone]

#75363

On Wed, Jul 30, 2014 at 5:43 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> Is this problem already solved? Can anyone make any suggestions?

I don't know of an actual solution, but I know where I'd look for one,
and that's importlib. If nothing else, you can use
importlib.machinery.all_suffixes() rather than hard-coding them, for
instance.

ChrisA

[toc] | [prev] | [next] | [standalone]

#75364

Steven D'Aprano wrote:

> I'm looking for a programmatic way to get a list of all Python modules
> and packages. Not just those already imported, but all those which
> *could* be imported.
> 
> I have a quick-and-dirty function which half does the job:
> 
> 
> def get_modules():
>     extensions = ('.py', '.pyc', '.pyo', '.so', '.dll')
>     matches = set()
>     for location in sys.path:
>         if location == '': location = '.'
>         if os.path.isdir(location):
>             for name in os.listdir(location):
>                 base, ext = os.path.splitext(name)
>                 if ext in extensions:
>                     matches.add(base)
>     return sorted(matches)
> 
> 
> 
> but I know it's wrong (it doesn't handle packages correctly, or zip
> files, doesn't follow .pth files, has a very naive understanding of cross-
> platform issues, fails to include built-in modules that don't live in the
> file system, and probably more).
> 
> Is this problem already solved? Can anyone make any suggestions?

$ python3 -m pydoc -b

shows a page with modules that I think is more complete than what you have. 
A quick glance at the implementation suggests that the hard work is done by 

pkgutil.iter_modules()

[toc] | [prev] | [next] | [standalone]

#75365

On 2014-07-30 09:46, Peter Otten wrote:
> Steven D'Aprano wrote:
>
>> I'm looking for a programmatic way to get a list of all Python modules
>> and packages. Not just those already imported, but all those which
>> *could* be imported.
>>
>> I have a quick-and-dirty function which half does the job:
>>
>>
>> def get_modules():
>>      extensions = ('.py', '.pyc', '.pyo', '.so', '.dll')
>>      matches = set()
>>      for location in sys.path:
>>          if location == '': location = '.'
>>          if os.path.isdir(location):
>>              for name in os.listdir(location):
>>                  base, ext = os.path.splitext(name)
>>                  if ext in extensions:
>>                      matches.add(base)
>>      return sorted(matches)
>>
>>
>>
>> but I know it's wrong (it doesn't handle packages correctly, or zip
>> files, doesn't follow .pth files, has a very naive understanding of cross-
>> platform issues, fails to include built-in modules that don't live in the
>> file system, and probably more).
>>
>> Is this problem already solved? Can anyone make any suggestions?
>
> $ python3 -m pydoc -b
>
> shows a page with modules that I think is more complete than what you have.
> A quick glance at the implementation suggests that the hard work is done by
>
> pkgutil.iter_modules()

There are two niggles to this answer: it omits builtin modules, but those are 
easily discovered through sys.builtin_module_names. It can also include spurious 
script .py files that cannot be imported because their names are not Python 
identifiers: e.g. check-newconfigs.py. Those are easy to filter out, fortunately.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#75372

On Wed, Jul 30, 2014 at 3:43 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> I'm looking for a programmatic way to get a list of all Python modules
> and packages. Not just those already imported, but all those which
> *could* be imported.
>

If you don't actually import it, how can you know it could be imported?
Not all .so files are valid python modules.
Not all .py files could be imported by all python interpreters.

> I have a quick-and-dirty function which half does the job:
>
>
> def get_modules():
>     extensions = ('.py', '.pyc', '.pyo', '.so', '.dll')
>     matches = set()
>     for location in sys.path:
>         if location == '': location = '.'
>         if os.path.isdir(location):
>             for name in os.listdir(location):
>                 base, ext = os.path.splitext(name)
>                 if ext in extensions:
>                     matches.add(base)
>     return sorted(matches)
>
>
>
> but I know it's wrong (it doesn't handle packages correctly, or zip
> files, doesn't follow .pth files, has a very naive understanding of cross-
> platform issues, fails to include built-in modules that don't live in the
> file system, and probably more).
>
> Is this problem already solved? Can anyone make any suggestions?
>
>
>
> --
> Steven
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Best Regards,
Leo Jay

[toc] | [prev] | [next] | [standalone]

#75408

On Wed, 30 Jul 2014 21:22:18 +0800, Leo Jay wrote:

> On Wed, Jul 30, 2014 at 3:43 PM, Steven D'Aprano <steve@pearwood.info>
> wrote:
>> I'm looking for a programmatic way to get a list of all Python modules
>> and packages. Not just those already imported, but all those which
>> *could* be imported.
>>
>>
> If you don't actually import it, how can you know it could be imported?
> Not all .so files are valid python modules. Not all .py files could be
> imported by all python interpreters.

You're right, of course, but I'm not concerned by whether or not the 
module is error-free and can be imported successfully.

I'm working on tab completion for module names. I have some alpha-quality 
code working, so if I hit TAB after typing "import ma" I get this:


py> import ma
macpath      macurl2path  mailbox      mailcap      mangle       
markupbase   math

For what it's worth, importing "mangle" fails with a SyntaxError. But 
that's okay, I don't expect tab completion to only show *valid* 
modules :-)

Over the next few days I'll make an official announcement, but if anyone 
wants a sneak-peek, check out:

http://code.google.com/p/tabhistory/source/browse/tabhistory.py


where I have indenting, code completion, filename completion, and module 
completion all working to some degree or another.

Thanks to everyone for their help.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#75411

In article <53da1d5a$0$29974$c3e8da3$5496439d@news.astraweb.com>,
 Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> I'm working on tab completion for module names. I have some alpha-quality 
> code working, so if I hit TAB after typing "import ma" I get this:
> 
> 
> py> import ma
> macpath      macurl2path  mailbox      mailcap      mangle       
> markupbase   math

That's kind of neat.  What do you do if the same name appears multiple 
places in your path?  It would be useful to call this out.  More than 
once, I've done "import x" and gotten a different x than I expected.  
Those kinds of problems can be tricky to debug.

[toc] | [prev] | [next] | [standalone]

#75415

On Thu, Jul 31, 2014 at 9:55 PM, Roy Smith <roy@panix.com> wrote:
> In article <53da1d5a$0$29974$c3e8da3$5496439d@news.astraweb.com>,
>  Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
>
>> I'm working on tab completion for module names. I have some alpha-quality
>> code working, so if I hit TAB after typing "import ma" I get this:
>>
>>
>> py> import ma
>> macpath      macurl2path  mailbox      mailcap      mangle
>> markupbase   math
>
> That's kind of neat.  What do you do if the same name appears multiple
> places in your path?  It would be useful to call this out.  More than
> once, I've done "import x" and gotten a different x than I expected.
> Those kinds of problems can be tricky to debug.

Immaterial to the tab completion, as it just means one of them is
shadowed by the other. But yes, that could be extremely useful - or
maybe completely useless, if there's a lot of intentional shadowing
happening. At any rate, it'd be nice to be able to say "Show me
everything that could be found for this name", which the same code
could answer.

ChrisA

[toc] | [prev] | [next] | [standalone]

#75437

On 7/31/2014 6:41 AM, Steven D'Aprano wrote:

> I'm working on tab completion for module names. I have some alpha-quality
> code working, so if I hit TAB after typing "import ma" I get this:
>
>
> py> import ma
> macpath      macurl2path  mailbox      mailcap      mangle
> markupbase   math

This is an interesting idea for Idle. Idle currently completes names 
from builtins and, in interactive mode, the current globals. So in the 
situation above, Idle offers 'map' and 'max' in a listbox with all 
builtins (and globals). But after '.', it completes attribute names 
instead and within strings, filenames, starting from the current 
directory of the user process for relative names.

It should not be too hard to recognize 'import' and 'from' instead of or 
in addition to '.' as context markers.  Even just offering stdlib name 
(or current directory names or . imports) would be a help.  Builtin 
names can never be correct in this context.

> For what it's worth, importing "mangle" fails with a SyntaxError. But
> that's okay, I don't expect tab completion to only show *valid*
> modules :-)
>
> Over the next few days I'll make an official announcement, but if anyone
> wants a sneak-peek, check out:
>
> http://code.google.com/p/tabhistory/source/browse/tabhistory.py
>
>
> where I have indenting, code completion, filename completion, and module
> completion all working to some degree or another.
>
> Thanks to everyone for their help.


-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#75476

On 2014-07-31 11:41, Steven D'Aprano wrote:
> On Wed, 30 Jul 2014 21:22:18 +0800, Leo Jay wrote:
>
>> On Wed, Jul 30, 2014 at 3:43 PM, Steven D'Aprano <steve@pearwood.info>
>> wrote:
>>> I'm looking for a programmatic way to get a list of all Python modules
>>> and packages. Not just those already imported, but all those which
>>> *could* be imported.
>>>
>>>
>> If you don't actually import it, how can you know it could be imported?
>> Not all .so files are valid python modules. Not all .py files could be
>> imported by all python interpreters.
>
> You're right, of course, but I'm not concerned by whether or not the
> module is error-free and can be imported successfully.
>
> I'm working on tab completion for module names. I have some alpha-quality
> code working, so if I hit TAB after typing "import ma" I get this:
>
>
> py> import ma
> macpath      macurl2path  mailbox      mailcap      mangle
> markupbase   math
>
> For what it's worth, importing "mangle" fails with a SyntaxError. But
> that's okay, I don't expect tab completion to only show *valid*
> modules :-)
>
> Over the next few days I'll make an official announcement, but if anyone
> wants a sneak-peek, check out:
>
> http://code.google.com/p/tabhistory/source/browse/tabhistory.py
>
>
> where I have indenting, code completion, filename completion, and module
> completion all working to some degree or another.

Take a look at what has already been implemented in IPython:

https://github.com/ipython/ipython/blob/master/IPython/core/completerlib.py#L208

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#75488

On Fri, 01 Aug 2014 14:39:09 +0100, Robert Kern wrote:

> Take a look at what has already been implemented in IPython:
> 
> https://github.com/ipython/ipython/blob/master/IPython/core/
completerlib.py#L208


Awesome! Thank you!


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#75548

On Friday, 1 August 2014 16:41:41 UTC+1, Steven D'Aprano  wrote:
> On Fri, 01 Aug 2014 14:39:09 +0100, Robert Kern wrote:
> 
> 
> 
> > Take a look at what has already been implemented in IPython:
> 
> > 
> 
> > https://github.com/ipython/ipython/blob/master/IPython/core/
> 
> completerlib.py#L208
> 
> 
> 
> 
> 
> Awesome! Thank you!
> 
> 
> 
> 
> 
> -- 
> 
> Steven

Also this, possibly:

    https://github.com/davidhalter/jedi

    Jon N

[toc] | [prev] | [next] | [standalone]

#75550

On 02/08/2014 22:13, jonnicoll11@googlemail.com wrote:
> On Friday, 1 August 2014 16:41:41 UTC+1, Steven D'Aprano  wrote:
>> On Fri, 01 Aug 2014 14:39:09 +0100, Robert Kern wrote:
>>
>>
>>
>>> Take a look at what has already been implemented in IPython:
>>
>>>
>>
>>> https://github.com/ipython/ipython/blob/master/IPython/core/
>>
>> completerlib.py#L208
>>
>>
>>
>>
>>
>> Awesome! Thank you!
>>
>>
>>
>>
>>
>> --
>>
>> Steven
>
> Also this, possibly:
>
>      https://github.com/davidhalter/jedi
>
>      Jon N
>

Would you please read and action this 
https://wiki.python.org/moin/GoogleGroupsPython to prevent us seeing the 
double line spacing above, thanks.


-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#75597

On 01/08/2014 16:41, Steven D'Aprano wrote:
> On Fri, 01 Aug 2014 14:39:09 +0100, Robert Kern wrote:
>
>> Take a look at what has already been implemented in IPython:
>>
>> https://github.com/ipython/ipython/blob/master/IPython/core/
> completerlib.py#L208
>
> Awesome! Thank you!
>

Is Lib/idlelib/AutoComplete.py of any use?  I found it when looking at 
this http://bugs.python.org/issue18766

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#75373

On Wed, Jul 30, 2014 at 11:22 PM, Leo Jay <python.leojay@gmail.com> wrote:
> On Wed, Jul 30, 2014 at 3:43 PM, Steven D'Aprano <steve@pearwood.info> wrote:
>> I'm looking for a programmatic way to get a list of all Python modules
>> and packages. Not just those already imported, but all those which
>> *could* be imported.
>>
>
> If you don't actually import it, how can you know it could be imported?
> Not all .so files are valid python modules.
> Not all .py files could be imported by all python interpreters.

What if you define it as "modules you could attempt to import"? Sure,
any module might fail during importing, but you've still taken a
statement of "import spam" and turned it into an attempt to read some
specific file from the disk.

ChrisA

[toc] | [prev] | [next] | [standalone]

#75375

Le mercredi 30 juillet 2014 09:43:10 UTC+2, Steven D'Aprano a écrit :
> I'm looking for a programmatic way to get a list of all Python modules 
> 
> and packages. Not just those already imported, but all those which 
> 
> *could* be imported.
> 
> 
> 

>>> 
>>> sys.path.append(r'D:\unicode_in_python_is_a_broken_mess')
>>> import steven
>>> steven.msg
'think a little bit...'
>>>

jmf

[toc] | [prev] | [next] | [standalone]

#75377

[Multipart message — attachments visible in raw view] — view raw

On Jul 30, 2014 4:37 AM, "Robert Kern" <robert.kern@gmail.com> wrote:
>
> On 2014-07-30 09:46, Peter Otten wrote:
>>
>> Steven D'Aprano wrote:
>>
>>> I'm looking for a programmatic way to get a list of all Python modules
>>> and packages. Not just those already imported, but all those which
>>> *could* be imported.
>>>
>>> I have a quick-and-dirty function which half does the job:
>>>
>>>
>>> def get_modules():
>>>      extensions = ('.py', '.pyc', '.pyo', '.so', '.dll')
>>>      matches = set()
>>>      for location in sys.path:
>>>          if location == '': location = '.'
>>>          if os.path.isdir(location):
>>>              for name in os.listdir(location):
>>>                  base, ext = os.path.splitext(name)
>>>                  if ext in extensions:
>>>                      matches.add(base)
>>>      return sorted(matches)
>>>
>>>
>>>
>>> but I know it's wrong (it doesn't handle packages correctly, or zip
>>> files, doesn't follow .pth files, has a very naive understanding of
cross-
>>> platform issues, fails to include built-in modules that don't live in
the
>>> file system, and probably more).
>>>
>>> Is this problem already solved? Can anyone make any suggestions?
>>
>>
>> $ python3 -m pydoc -b
>>
>> shows a page with modules that I think is more complete than what you
have.
>> A quick glance at the implementation suggests that the hard work is done
by
>>
>> pkgutil.iter_modules()
>
>
> There are two niggles to this answer: it omits builtin modules, but those
are easily discovered through sys.builtin_module_names. It can also include
spurious script .py files that cannot be imported because their names are
not Python identifiers: e.g. check-newconfigs.py. Those are easy to filter
out, fortunately.

It will also omit any modules provided by a custom module finder that
doesn't implement iter_modules, which is not a required part of the
interface.

[toc] | [prev] | [next] | [standalone]

#75378

[Multipart message — attachments visible in raw view] — view raw

On Wed, Jul 30, 2014 at 2:43 AM, Steven D'Aprano <steve@pearwood.info>
wrote:

> I'm looking for a programmatic way to get a list of all Python modules
> and packages. Not just those already imported, but all those which
> *could* be imported.
>

I wrote a modified dir(), which I inject into builtins in interactive
sessions. When getting a directory of an object, it attempts to identify
not-yet-imported modules it contains. Such modules are enclosed in square
brackets. Modules that smell like packages also get a trailing '/'. For
example:

% python
Python 2.7.5+ (2.7:2921f6c2009e, Apr 30 2014, 14:00:13)
[GCC 4.4.6 [TWW]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import dateutil
>>> dir(dateutil)
['[easter]', '[parser]', '[relativedelta]', '[rrule]', '[tz]', '[tzwin]',
'[zoneinfo/]', '__author__', '__builtins__', '__doc__', '__file__',
'__license__', '__name__', '__package__', '__path__', '__version__']


It's not perfect, but works for my needs. Perhaps it will give you some
ideas.

Skip

[toc] | [prev] | [next] | [standalone]

#75430

Steven D'Aprano <steve@pearwood.info> writes:

> I'm looking for a programmatic way to get a list of all Python modules 
> and packages. Not just those already imported, but all those which 
> *could* be imported.
...
> Is this problem already solved? Can anyone make any suggestions?

Look at how `help('modules')` is implemented. Though it crashes on my
system.

See also,

How can I get a list of locally installed Python modules? [1]
python - get available modules [2]

[1]
http://stackoverflow.com/questions/739993/how-can-i-get-a-list-of-locally-installed-python-modules
[2]
http://stackoverflow.com/questions/3952513/python-get-available-modules


--
Akira

[toc] | [prev] | [next] | [standalone]

#75436 — Bug with help (was Re: Getting a list of all modules)

On 31/07/2014 19:55, Akira Li wrote:
> Steven D'Aprano <steve@pearwood.info> writes:
>
>> I'm looking for a programmatic way to get a list of all Python modules
>> and packages. Not just those already imported, but all those which
>> *could* be imported.
> ...
>> Is this problem already solved? Can anyone make any suggestions?
>
> Look at how `help('modules')` is implemented. Though it crashes on my
> system.
>

Have you reported this at bugs.python.org or is there already an issue 
for the problem that you see?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web