Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #102483 > unrolled thread

How a module is being marked as imported?

Back to article view | Back to comp.lang.python

Contents

  How a module is being marked as imported? Jean-Charles Lefebvre <polyvertex@gmail.com> - 2016-02-04 02:19 -0800
    Re: How a module is being marked as imported? Jean-Charles Lefebvre <polyvertex@gmail.com> - 2016-02-04 07:35 -0800
      Re: How a module is being marked as imported? Kevin Conway <kevinjacobconway@gmail.com> - 2016-02-04 16:11 +0000

#102483 — How a module is being marked as imported?

Hi all,

The short version: How CPython marks a module as being fully imported, if it does, so that the same import statement ran from another C thread at the same time does not collide? Or, reversely, does not think the module is not already fully imported?

The full version: I'm running CPython 3.5.1, embedded into a C++ application on Windows. The application is heavily multi-threaded so several C threads call some Python code at the same time (different Python modules), sharing interpreter's resources by acquiring/releasing the GIL frequently DURING the calls, at language boundaries.

Sometimes (but always only once per application instance), a call to os.path.expandvars raises the AttributeError exception with message: module 'string' has no attribute 'ascii_letters'. It is raised by the ntpath.expandvars function (line 372). When I noticed the late import statement of the 'string' module at the line above, I thought that MAYBE, it could be because the interpreter is ran in an heavily multi-threaded environment and that the GIL acquiring/releasing occurred at a bad timing? Making me wonder how the import mechanism interacts with the GIL, if it does?

[toc] | [next] | [standalone]

#102500

So far, I've been advised to:

1/ Double-check that the GIL was correctly acquired
2/ Ensure there's no 'string' module in my project
3/ Manually pre-import commonly used standard modules at interpreter's init-time to avoid race conditions due to the multi-threaded nature of the running environment

No problem found for 1/ & 2/ (double-checked). I tried 3/ before posting and could not reproduce the problem at all which is probably the patch I will apply due to the lack of a better solution. I guess I'll have to dig into __import__'s code and related.


On Thursday, February 4, 2016 at 11:20:05 AM UTC+1, Jean-Charles Lefebvre wrote:
> Hi all,
> 
> The short version: How CPython marks a module as being fully imported, if it does, so that the same import statement ran from another C thread at the same time does not collide? Or, reversely, does not think the module is not already fully imported?
> 
> The full version: I'm running CPython 3.5.1, embedded into a C++ application on Windows. The application is heavily multi-threaded so several C threads call some Python code at the same time (different Python modules), sharing interpreter's resources by acquiring/releasing the GIL frequently DURING the calls, at language boundaries.
> 
> Sometimes (but always only once per application instance), a call to os.path.expandvars raises the AttributeError exception with message: module 'string' has no attribute 'ascii_letters'. It is raised by the ntpath.expandvars function (line 372). When I noticed the late import statement of the 'string' module at the line above, I thought that MAYBE, it could be because the interpreter is ran in an heavily multi-threaded environment and that the GIL acquiring/releasing occurred at a bad timing? Making me wonder how the import mechanism interacts with the GIL, if it does?

[toc] | [prev] | [next] | [standalone]

#102501

As an attempt to answer your original question, Python doesn't explicitly
mark a module as done. It does keep imports cached in sys.modules, though.
The behaviour you describe where later imports get the same module object
is driven by that cache.

There are cases, such as cyclical imports, where the object in sys.modules
is empty for a period of time and acts as a placeholder until the code from
that module is loaded. During that time, any usage of the module would
result in a similar failure to what you have described.

Additionally, if you are writing multithreaded code then acquiring the GIL
may not be enough. There is another lock used for imports.

On Thu, Feb 4, 2016, 09:43 Jean-Charles Lefebvre <polyvertex@gmail.com>
wrote:

> So far, I've been advised to:
>
> 1/ Double-check that the GIL was correctly acquired
> 2/ Ensure there's no 'string' module in my project
> 3/ Manually pre-import commonly used standard modules at interpreter's
> init-time to avoid race conditions due to the multi-threaded nature of the
> running environment
>
> No problem found for 1/ & 2/ (double-checked). I tried 3/ before posting
> and could not reproduce the problem at all which is probably the patch I
> will apply due to the lack of a better solution. I guess I'll have to dig
> into __import__'s code and related.
>
>
> On Thursday, February 4, 2016 at 11:20:05 AM UTC+1, Jean-Charles Lefebvre
> wrote:
> > Hi all,
> >
> > The short version: How CPython marks a module as being fully imported,
> if it does, so that the same import statement ran from another C thread at
> the same time does not collide? Or, reversely, does not think the module is
> not already fully imported?
> >
> > The full version: I'm running CPython 3.5.1, embedded into a C++
> application on Windows. The application is heavily multi-threaded so
> several C threads call some Python code at the same time (different Python
> modules), sharing interpreter's resources by acquiring/releasing the GIL
> frequently DURING the calls, at language boundaries.
> >
> > Sometimes (but always only once per application instance), a call to
> os.path.expandvars raises the AttributeError exception with message: module
> 'string' has no attribute 'ascii_letters'. It is raised by the
> ntpath.expandvars function (line 372). When I noticed the late import
> statement of the 'string' module at the line above, I thought that MAYBE,
> it could be because the interpreter is ran in an heavily multi-threaded
> environment and that the GIL acquiring/releasing occurred at a bad timing?
> Making me wonder how the import mechanism interacts with the GIL, if it
> does?
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web