Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #95357 > unrolled thread

Re: Module load times

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Module load times "Joseph L. Casale" <jcasale@activenetwerx.com> - 2015-08-13 21:12 +0000
    Re: Module load times Steven D'Aprano <steve@pearwood.info> - 2015-08-14 12:10 +1000
      RE: Module load times "Joseph L. Casale" <jcasale@activenetwerx.com> - 2015-08-14 02:25 +0000
      Re: Module load times Chris Angelico <rosuav@gmail.com> - 2015-08-14 12:38 +1000

#95357 — Re: Module load times

Hi Stefan,

> How is the DLL binding implemented? Using "ctypes"? Or something else?

It is through ctypes.

> Obviously, instantiating a large ctypes wrapper will take some time. A
> binary module would certainly be quicker here, both in terms of import time
> and execution time. Since you're generating the code anyway, generating
> Cython code instead shouldn't be difficult but would certainly yield faster
> code.

True, I was using XSLT to auto generate the module. I will however explore this.

> What makes you think the import might be a problem? That's a one-time
> thing. Or is your application a command-line tool or so that needs to start
> and terminate quickly?

The code is used within plugin points and soon to be asynchronous code
(once  the original broken implementation is fixed) where in some cases it
will be instantiated 100's of 1000's of times etc...

Thanks,
jlc

[toc] | [next] | [standalone]

#95365

On Fri, 14 Aug 2015 07:12 am, Joseph L. Casale wrote:

>> What makes you think the import might be a problem? That's a one-time
>> thing. Or is your application a command-line tool or so that needs to
>> start and terminate quickly?
> 
> The code is used within plugin points and soon to be asynchronous code
> (once  the original broken implementation is fixed) where in some cases it
> will be instantiated 100's of 1000's of times etc...

Importing is not the same as instantiation.

When you import a module, the code is only read from disk and instantiated
the first time. Then it is cached. Subsequent imports in the same Python
session use the cached version.

So the answer will depend on your application. If your application runs for
a long time (relatively speaking), and imports the plugins 100s or 1000s of
times, it should not matter. Only the first import is likely to be slow.

But if the application starts up, imports the plugin, then shuts down, then
repeats 100s or 1000s of times, that may be slow. Or if it launches
separate processes, each of which imports the plugin.

Without understanding your application, and the plugin model, it is
difficult to predict the impact of importing.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#95366

> Importing is not the same as instantiation.
>
> When you import a module, the code is only read from disk and instantiated
> the first time. Then it is cached. Subsequent imports in the same Python
> session use the cached version.

I do mean imported, in the original design there were many ctype function
prototypes, the c developer had ran some tests and showed quite a bit of
time taken up just the import over 100 clean imports. I don't have his test
harness and hence why I am trying to validate the results.

> So the answer will depend on your application. If your application runs for
> a long time (relatively speaking), and imports the plugins 100s or 1000s of
> times, it should not matter. Only the first import is likely to be slow.
>
> But if the application starts up, imports the plugin, then shuts down, then
> repeats 100s or 1000s of times, that may be slow. Or if it launches
> separate processes, each of which imports the plugin.

Yeah that wasn't clear. The plugins are invoked in fresh interpreter processes
and hence modules with import side effects or simply large modules can
manifest over time.

Thanks Steven
jlc

[toc] | [prev] | [next] | [standalone]

#95367

On Fri, Aug 14, 2015 at 12:25 PM, Joseph L. Casale
<jcasale@activenetwerx.com> wrote:
> Yeah that wasn't clear. The plugins are invoked in fresh interpreter processes
> and hence modules with import side effects or simply large modules can
> manifest over time.

If they're invoked in fresh processes, then you're looking at process
startup time, and I would recommend using OS-level tools to play
around with that. Unix-like systems will usually have a 'time' command
which will tell you exactly how much wall and CPU time a process took
needed; just create a system that starts up and promptly shuts down,
and then you can test iterations that way.

Of course, there are a million and one variables (disk caches, other
activity on the system, etc), but it's better than nothing.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web