Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!goblin3!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed4a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.005 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'matches': 0.07; 'python3': 0.07; 'subject:Getting': 0.07; "'.'": 0.09; 'builtin': 0.09; 'imported': 0.09; 'omit': 0.09; 'subject:modules': 0.09; 'python': 0.11; 'def': 0.12; '>>': 0.16; "'':": 0.16; '(it': 0.16; '.py': 0.16; 'finder': 0.16; 'imported.': 0.16; 'modules,': 0.16; 'programmatic': 0.16; 'set()': 0.16; 'spurious': 0.16; 'sys.path:': 0.16; '\xc2\xa0if': 0.16; 'extensions': 0.16; 'wrote:': 0.18; 'module': 0.19; 'packages.': 0.19; '>>>': 0.22; 'email addr:gmail.com>': 0.22; '>>>': 0.24; 'script': 0.25; '>': 0.26; 'header:In-Reply-To:1': 0.27; 'function': 0.29; 'am,': 0.29; "doesn't": 0.30; 'subject:list': 0.30; 'message-id:@mail.gmail.com': 0.30; "i'm": 0.30; 'base,': 0.31; "d'aprano": 0.31; 'steven': 0.31; 'anyone': 0.31; 'file': 0.32; 'probably': 0.32; 'subject:all': 0.32; 'problem': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'ext': 0.36; 'done': 0.36; 'shows': 0.36; 'half': 0.37; 'wrong': 0.37; 'two': 0.37; 'list': 0.37; 'easily': 0.37; 'implement': 0.38; 'skip:o 20': 0.38; 'system,': 0.38; 'e.g.': 0.38; 'filter': 0.38; 'handle': 0.38; 'to:addr:python-list': 0.38; 'files': 0.38; 'does': 0.39; 'to:addr:python.org': 0.39; 'skip:p 20': 0.39; 'skip:u 10': 0.60; 'easy': 0.60; 'skip:\xc2 10': 0.60; 'issues,': 0.61; 'complete': 0.62; 'name': 0.63; 'zip': 0.64; 'more': 0.64; '30,': 0.65; 'jul': 0.74; 'discovered': 0.83; 'answer:': 0.84; 'glance': 0.84; 'otten': 0.84; 'have.': 0.93 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=8q6C3U4Ik3jPWB9KuNeykCJDH6/b2y7vDz7v6xMQZ1U=; b=lXSzx0JuDkcT1h46MRe3rM2Ft8BhPmrmxuhl9iM2p11hKai2ZEjU5uEBlnb/IGNYBF oN1kid/LblmBcxxl6Q3jmU34Cy1crsZxWWBynr5HI5PePD1vET/+mwjEw0KkgvbS/l1s yha5U+ulUX9aZJXLUnM+UBjasGQ3Ja+5JQL+HElxcoCh3pTmXnq8MWrrvjel8W2oA3Rt 8p3eTH1w7dWvcxEUBsuHbmM7f1yvBL4GDdRR+Z1jX318gH7GExkKRuS28Znwzxh46Bjh f62uN6hwCHs09fcVLqGAc5lMytUHvyppw8Ea2QYF759Y2vEP9qisy3RGrX3dDv0F7M7V YnRA== MIME-Version: 1.0 X-Received: by 10.69.17.230 with SMTP id gh6mr5330552pbd.0.1406730544360; Wed, 30 Jul 2014 07:29:04 -0700 (PDT) In-Reply-To: References: <53d8a20e$0$29977$c3e8da3$5496439d@news.astraweb.com> Date: Wed, 30 Jul 2014 08:29:04 -0600 Subject: Re: Getting a list of all modules From: Ian Kelly To: Python Content-Type: multipart/alternative; boundary=089e0158c44464325504ff69faeb X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 136 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1406730553 news.xs4all.nl 2933 [2001:888:2000:d::a6]:56438 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:75377 --089e0158c44464325504ff69faeb Content-Type: text/plain; charset=UTF-8 On Jul 30, 2014 4:37 AM, "Robert Kern" wrote: > > On 2014-07-30 09:46, Peter Otten wrote: >> >> Steven D'Aprano wrote: >> >>> I'm looking for a programmatic way to get a list of all Python modules >>> and packages. Not just those already imported, but all those which >>> *could* be imported. >>> >>> I have a quick-and-dirty function which half does the job: >>> >>> >>> def get_modules(): >>> extensions = ('.py', '.pyc', '.pyo', '.so', '.dll') >>> matches = set() >>> for location in sys.path: >>> if location == '': location = '.' >>> if os.path.isdir(location): >>> for name in os.listdir(location): >>> base, ext = os.path.splitext(name) >>> if ext in extensions: >>> matches.add(base) >>> return sorted(matches) >>> >>> >>> >>> but I know it's wrong (it doesn't handle packages correctly, or zip >>> files, doesn't follow .pth files, has a very naive understanding of cross- >>> platform issues, fails to include built-in modules that don't live in the >>> file system, and probably more). >>> >>> Is this problem already solved? Can anyone make any suggestions? >> >> >> $ python3 -m pydoc -b >> >> shows a page with modules that I think is more complete than what you have. >> A quick glance at the implementation suggests that the hard work is done by >> >> pkgutil.iter_modules() > > > There are two niggles to this answer: it omits builtin modules, but those are easily discovered through sys.builtin_module_names. It can also include spurious script .py files that cannot be imported because their names are not Python identifiers: e.g. check-newconfigs.py. Those are easy to filter out, fortunately. It will also omit any modules provided by a custom module finder that doesn't implement iter_modules, which is not a required part of the interface. --089e0158c44464325504ff69faeb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On Jul 30, 2014 4:37 AM, "Robert Kern" <robert.kern@gmail.com> wrote:
>
> On 2014-07-30 09:46, Peter Otten wrote:
>>
>> Steven D'Aprano wrote:
>>
>>> I'm looking for a programmatic way to get a list of all Py= thon modules
>>> and packages. Not just those already imported, but all those w= hich
>>> *could* be imported.
>>>
>>> I have a quick-and-dirty function which half does the job:
>>>
>>>
>>> def get_modules():
>>> =C2=A0 =C2=A0 =C2=A0extensions =3D ('.py', '.pyc&#= 39;, '.pyo', '.so', '.dll')
>>> =C2=A0 =C2=A0 =C2=A0matches =3D set()
>>> =C2=A0 =C2=A0 =C2=A0for location in sys.path:
>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if location =3D=3D ''= ;: location =3D '.'
>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if os.path.isdir(location):<= br> >>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0for name in os= .listdir(location):
>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= base, ext =3D os.path.splitext(name)
>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= if ext in extensions:
>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0matches.add(base)
>>> =C2=A0 =C2=A0 =C2=A0return sorted(matches)
>>>
>>>
>>>
>>> but I know it's wrong (it doesn't handle packages corr= ectly, or zip
>>> files, doesn't follow .pth files, has a very naive underst= anding of cross-
>>> platform issues, fails to include built-in modules that don= 9;t live in the
>>> file system, and probably more).
>>>
>>> Is this problem already solved? Can anyone make any suggestion= s?
>>
>>
>> $ python3 -m pydoc -b
>>
>> shows a page with modules that I think is more complete than what = you have.
>> A quick glance at the implementation suggests that the hard work i= s done by
>>
>> pkgutil.iter_modules()
>
>
> There are two niggles to this answer: it omits builtin modules, but th= ose are easily discovered through sys.builtin_module_names. It can also inc= lude spurious script .py files that cannot be imported because their names = are not Python identifiers: e.g. check-newconfigs.py. Those are easy to fil= ter out, fortunately.

It will also omit any modules provided by a custom module fi= nder that doesn't implement iter_modules, which is not a required part = of the interface.

--089e0158c44464325504ff69faeb--