Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #74081 > unrolled thread

Module name does not match file name

Back to article view | Back to comp.lang.python

Contents

  Module name does not match file name Steven D'Aprano <steve@pearwood.info> - 2014-07-07 07:57 +0000
    Re: Module name does not match file name Chris Angelico <rosuav@gmail.com> - 2014-07-07 18:04 +1000
      Re: Module name does not match file name Steven D'Aprano <steve@pearwood.info> - 2014-07-07 09:03 +0000
        Re: Module name does not match file name Chris Angelico <rosuav@gmail.com> - 2014-07-07 19:11 +1000
    Re: Module name does not match file name Steven D'Aprano <steve@pearwood.info> - 2014-07-07 08:57 +0000
      Re: Module name does not match file name Chris Angelico <rosuav@gmail.com> - 2014-07-07 19:03 +1000
        Re: Module name does not match file name Steven D'Aprano <steve@pearwood.info> - 2014-07-07 09:28 +0000
      Re: Module name does not match file name Peter Otten <__peter__@web.de> - 2014-07-07 12:41 +0200
      Re: Module name does not match file name Robert Kern <robert.kern@gmail.com> - 2014-07-07 12:15 +0100
        Re: Module name does not match file name Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-07-07 11:56 +0000
          Re: Module name does not match file name Chris Angelico <rosuav@gmail.com> - 2014-07-07 22:04 +1000
          Re: Module name does not match file name Robert Kern <robert.kern@gmail.com> - 2014-07-07 15:25 +0100

#74081 — Module name does not match file name

Cut a long story short... 

I'm trying to debug a Tkinter app written in Python. The author of this 
app designed it as a maze of twisty dependencies, all alike, and his idea 
of source control was to make multiple copies of every file and drop them 
in random places on the hard drive. He also has a whole lot of non-
standard, semi-standard, and just plain weird locations added to the 
PYTHONPATH, some of which are added at run time for extra enjoyment, so 
I'm having lots of fun trying to work out which files are being imported 
and where they are being imported from.

I ran the modulefinder tool to get a list of imported modules:

$ python2.6 -m modulefinder /long/path/script.py


but to my surprise, it claims that "import pg" is *not* reading the pg.py 
file I located, but instead loading a module with a completely different 
name:

$ python2.6 -m modulefinder /long/path/script.py

  Name                  File
  ----                  ----
[...]
m pg                    /usr/local/lib/python2.6/dist-packages/pgmodule.so


Can anyone explain how "import pg" can end up coming from pgmodule.so?


Sure enough:

>>> import pg
>>> pg.__file__
'/usr/local/lib/python2.6/dist-packages/pgmodule.so'


What sorcery is this???


-- 
Steven

[toc] | [next] | [standalone]

#74083

On Mon, Jul 7, 2014 at 5:57 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> Can anyone explain how "import pg" can end up coming from pgmodule.so?

First guess: There's a "pg.py" somewhere that imports the so, then
replaces itself in sys.modules.

# importme.py
import sys
sys.modules["importme"]=sys


>>> import importme
>>> importme
<module 'sys' (built-in)>

Unfortunately, that doesn't help much with actually finding the source
code that's doing this.

ChrisA

[toc] | [prev] | [next] | [standalone]

#74086

On Mon, 07 Jul 2014 18:04:36 +1000, Chris Angelico wrote:

> On Mon, Jul 7, 2014 at 5:57 PM, Steven D'Aprano <steve@pearwood.info>
> wrote:
>> Can anyone explain how "import pg" can end up coming from pgmodule.so?
> 
> First guess: There's a "pg.py" somewhere that imports the so, then
> replaces itself in sys.modules.


Oh, there's a pg.py somewhere alright, but it imports _py and there is no 
_py module anywhere on the system. I don't believe that the pg.py file is 
actually used, even though it is in the site-packages next to pgmodule.py.

I did mention the maze of twisty dependencies, didn't I?



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#74088

On Mon, Jul 7, 2014 at 7:03 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> On Mon, 07 Jul 2014 18:04:36 +1000, Chris Angelico wrote:
>
>> On Mon, Jul 7, 2014 at 5:57 PM, Steven D'Aprano <steve@pearwood.info>
>> wrote:
>>> Can anyone explain how "import pg" can end up coming from pgmodule.so?
>>
>> First guess: There's a "pg.py" somewhere that imports the so, then
>> replaces itself in sys.modules.
>
>
> Oh, there's a pg.py somewhere alright, but it imports _py and there is no
> _py module anywhere on the system. I don't believe that the pg.py file is
> actually used, even though it is in the site-packages next to pgmodule.py.

Huh. Definitely curious as to what 'python -v' says, then.

> I did mention the maze of twisty dependencies, didn't I?

And I winced at your opening paragraph.

ChrisA

[toc] | [prev] | [next] | [standalone]

#74085

Ah, I think I have a partial answer... but not a complete answer.


On Mon, 07 Jul 2014 07:57:21 +0000, Steven D'Aprano wrote:

> Can anyone explain how "import pg" can end up coming from pgmodule.so?
> 
> 
> Sure enough:
> 
>>>> import pg
>>>> pg.__file__
> '/usr/local/lib/python2.6/dist-packages/pgmodule.so'



I've looked inside the pgmodule.c source code, and it includes this 
snippet:


extern void
initpg(void)
{
    char *p;
    int i;

    Py_InitModule("pg", pg_methods);


which suggests that the pgmodule.so file creates a module called "pg". 
What I don't understand is how "import pg" gets turned into "run 
pgmodule.so"?




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#74087

On Mon, Jul 7, 2014 at 6:57 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> which suggests that the pgmodule.so file creates a module called "pg".
> What I don't understand is how "import pg" gets turned into "run
> pgmodule.so"?

What happens if you *don't* import pg? Is there a sys.modules["pg"]
already? If so, I'd go looking at what gets imported on startup - in
fact, either way, "python -v" output might be of value here.

ChrisA

[toc] | [prev] | [next] | [standalone]

#74089

On Mon, 07 Jul 2014 19:03:33 +1000, Chris Angelico wrote:

> On Mon, Jul 7, 2014 at 6:57 PM, Steven D'Aprano <steve@pearwood.info>
> wrote:
>> which suggests that the pgmodule.so file creates a module called "pg".
>> What I don't understand is how "import pg" gets turned into "run
>> pgmodule.so"?
> 
> What happens if you *don't* import pg? Is there a sys.modules["pg"]
> already? 

No.

py> "pg" in sys.modules.keys()
False


But then there's this:


py> import pgmodule
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define init function (initpgmodule)
py> import pg
py> import pgmodule
py> pg.__file__ == pgmodule.__file__
True


o_O


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#74092

Steven D'Aprano wrote:

> Ah, I think I have a partial answer... but not a complete answer.
> 
> 
> On Mon, 07 Jul 2014 07:57:21 +0000, Steven D'Aprano wrote:
> 
>> Can anyone explain how "import pg" can end up coming from pgmodule.so?
>> 
>> 
>> Sure enough:
>> 
>>>>> import pg
>>>>> pg.__file__
>> '/usr/local/lib/python2.6/dist-packages/pgmodule.so'
> 
> 
> 
> I've looked inside the pgmodule.c source code, and it includes this
> snippet:
> 
> 
> extern void
> initpg(void)
> {
>     char *p;
>     int i;
> 
>     Py_InitModule("pg", pg_methods);
> 
> 
> which suggests that the pgmodule.so file creates a module called "pg".
> What I don't understand is how "import pg" gets turned into "run
> pgmodule.so"?

It seems the suffix "module.so" is tried automagically by the import 
machinery:

$ touch alphamodule.so
$ python
Python 2.7.5+ (default, Feb 27 2014, 19:37:08) 
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import alpha
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: ./alphamodule.so: file too short

[toc] | [prev] | [next] | [standalone]

#74093

On 2014-07-07 09:57, Steven D'Aprano wrote:
> Ah, I think I have a partial answer... but not a complete answer.
>
>
> On Mon, 07 Jul 2014 07:57:21 +0000, Steven D'Aprano wrote:
>
>> Can anyone explain how "import pg" can end up coming from pgmodule.so?
>>
>>
>> Sure enough:
>>
>>>>> import pg
>>>>> pg.__file__
>> '/usr/local/lib/python2.6/dist-packages/pgmodule.so'
>
>
>
> I've looked inside the pgmodule.c source code, and it includes this
> snippet:
>
>
> extern void
> initpg(void)
> {
>      char *p;
>      int i;
>
>      Py_InitModule("pg", pg_methods);
>
>
> which suggests that the pgmodule.so file creates a module called "pg".
> What I don't understand is how "import pg" gets turned into "run
> pgmodule.so"?

This has been standard Python behavior for extension modules since forever. It's 
a very old practice and not recommended for new code, though.

[~]
|1> import imp

[~]
|2> imp.get_suffixes()
[('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1), ('.pyc', 'rb', 2)]

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#74096

On Mon, 07 Jul 2014 12:15:51 +0100, Robert Kern wrote:

> On 2014-07-07 09:57, Steven D'Aprano wrote:
>> What I don't understand is how "import pg" gets turned into "run
>> pgmodule.so"?
> 
> This has been standard Python behavior for extension modules since
> forever. It's a very old practice and not recommended for new code,
> though.

Hmmm. Well, that is very special. Is this documented anywhere?

I know very little about extension modules. If I just rename the source 
file from pgmodule.c to pg.c, recompile to pg.so, and use that in place 
of pgmodule.so, is anything likely to break?



> |1> import imp
> 
> [~]
> |2> imp.get_suffixes()
> [('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1), ('.pyc',
> 'rb', 2)]


That seems to be gone in Python 3.3:


py> imp.get_suffixes()
[('.cpython-33m.so', 'rb', 3), ('.abi3.so', 'rb', 3), ('.so', 'rb', 3), 
('.py', 'U', 1), ('.pyc', 'rb', 2)]


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#74097

On Mon, Jul 7, 2014 at 9:56 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Hmmm. Well, that is very special. Is this documented anywhere?

"Special", in the sense of Shepherd Book addressing Mal. "Isn't
that... special."

ChrisA

[toc] | [prev] | [next] | [standalone]

#74111

On 2014-07-07 12:56, Steven D'Aprano wrote:
> On Mon, 07 Jul 2014 12:15:51 +0100, Robert Kern wrote:
>
>> On 2014-07-07 09:57, Steven D'Aprano wrote:
>>> What I don't understand is how "import pg" gets turned into "run
>>> pgmodule.so"?
>>
>> This has been standard Python behavior for extension modules since
>> forever. It's a very old practice and not recommended for new code,
>> though.
>
> Hmmm. Well, that is very special. Is this documented anywhere?

Not that I can find. Maybe the PEP that removed it for Python 3 might document 
it implicitly.

> I know very little about extension modules. If I just rename the source
> file from pgmodule.c to pg.c, recompile to pg.so, and use that in place
> of pgmodule.so, is anything likely to break?

I don't think so, but why would you bother (assuming you aren't the maintainer 
of pgmodule.so)?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web