Re: reloading code and multiprocessing

Path	csiph.com!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<andrea.crotti.0@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.005
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'subject:code': 0.07; 'python': 0.09; 'imported': 0.09; 'inherited': 0.09; 'subject:skip:m 10': 0.09; 'cc:addr:python-list': 0.10; 'anyway': 0.11; 'language': 0.14; 'chris,': 0.16; 'correctly,': 0.16; 'expert.': 0.16; 'reimport': 0.16; 'reload': 0.16; 'say.': 0.16; 'sys.modules': 0.16; 'think?': 0.16; 'wrote:': 0.17; 'issue,': 0.17; 'thu,': 0.17; 'import': 0.21; 'fairly': 0.21; 'runs': 0.22; "i'd": 0.22; 'cc:20': 0.23; "i've": 0.23; 'seems': 0.23; 'cc:no real name:20': 0.24; 'cc:addr:python.org': 0.25; 'header:In- Reply-To:1': 0.25; 'message-id:@mail.gmail.com': 0.27; "doesn't": 0.28; 'there.': 0.28; 'chris': 0.28; 'url:mailman': 0.29; 'probably': 0.29; "i'm": 0.29; 'maybe': 0.29; 'function': 0.30; 'code': 0.31; '(and': 0.32; 'url:python': 0.32; 'running': 0.32; 'url:listinfo': 0.32; 'right?': 0.33; 'problem': 0.33; 'received:google.com': 0.34; 'wrong': 0.34; 'done': 0.34; 'thanks': 0.34; 'needed': 0.35; 'pm,': 0.35; 'received:209.85': 0.35; 'something': 0.35; 'there': 0.35; 'but': 0.36; 'url:org': 0.36; 'depends': 0.36; 'expensive': 0.36; 'modules': 0.36; "didn't": 0.36; 'problems': 0.36; 'possible': 0.37; 'received:209': 0.37; 'far': 0.37; 'subject:: ': 0.38; 'files': 0.38; 'things': 0.38; 'sure': 0.38; 'system.': 0.39; 'short': 0.39; 'where': 0.40; 'header:Received:5': 0.40; 'url:mail': 0.40; 'skip:u 10': 0.60; 'remove': 0.61; "you've": 0.61; 'safe': 0.63; 'more': 0.63; 'jul': 0.65; 'andrea': 0.84; 'job;': 0.84; 'safe.': 0.93
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=/DDgi4nnDUFAHfWRasYdl62Y/77lSFdgdN2+WEiE52M=; b=OX4Hr5PEQXafsl9Pm1Bo9a27MeItvoTRqS0C0JKrlzy9KruUXc+8ro0TSIYB66wIZE TvgOf2njZqlxpZ0WiDLeJDK7LD2r1DzKh33CM6M2iA7IuBZwne/wm/IXn/SeQLaaMXSH 51DO4xD4zeoO4e8cy9rOI5XCHzkYjeUTE3shMd4ZKhtN9ScdWKIAorOuQ96W4a9GcMGY knK+GoNsgAsL4J2goniZeb16AfXx76Nflu+LPBOi1LFYY5hKMFmeBolAqCFi/ILP49V1 mPyNhvRMV2HrjXxnkwpfJCEtxJ49cSazqKoN/rHbMncJKXAwb/yvm874mIl0IwQ1q3n7 uSQQ==
MIME-Version	1.0
In-Reply-To	<CAPTjJmoUkKCe3q_qa7iACw0B0oVdZB-_Mo7xvgJT0nEyaOTdwg@mail.gmail.com>
References	<CAF_E5Jb=_H1D9YwLWGzE+D40U9ji1kFzo1hz=VwB3yzb8=43CA@mail.gmail.com> <CAPTjJmoUkKCe3q_qa7iACw0B0oVdZB-_Mo7xvgJT0nEyaOTdwg@mail.gmail.com>
Date	Mon, 23 Jul 2012 14:51:05 +0100
Subject	Re: reloading code and multiprocessing
From	andrea crotti <andrea.crotti.0@gmail.com>
To	Chris Angelico <rosuav@gmail.com>
Content-Type	text/plain; charset=ISO-8859-1
Cc	python-list@python.org
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.2469.1343051468.4697.python-list@python.org> (permalink)
Lines	39
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1343051468 news.xs4all.nl 6887 [2001:888:2000:d::a6]:35767
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:25868

Show key headers only | View raw

2012/7/20 Chris Angelico <rosuav@gmail.com>:
> On Thu, Jul 19, 2012 at 8:15 PM, andrea crotti
> <andrea.crotti.0@gmail.com> wrote:
>> We need to be able to reload code on a live system.  This live system
>> has a daemon process always running but it runs many subprocesses with
>> multiprocessing, and the subprocesses might have a short life...
>> ...
>> As long as I import the code in the function and make sure to remove the
>> "pyc" files everything seems to work..
>> Are there any possible problems which I'm not seeing in this approach or
>> it's safe?
>
> Python never promises reloading reliability, but from my understanding
> of what you've done here, it's probably safe. However, you may find
> that you're using the wrong language for the job; it depends on how
> expensive it is to spin off all those processes and ship their work to
> them. But if that's not an issue, I'd say you have something safe
> there. (Caveat: I've given this only a fairly cursory examination, and
> I'm not an expert. Others may have more to say. I just didn't want the
> resident Markov chainer to be the only one to respond!!)
>
> ChrisA
> --
> http://mail.python.org/mailman/listinfo/python-list


Thanks Chris, always nice to get a "human" answer ;)

Anyway the only other problem which I found is that if I start the
subprocesses after many other things are initialised, it might happen
that the reloading doesn't work correctly, is that right?

Because sys.modules will get inherited from the subprocesses and it
will not reimport what has been already imported as far as I
understood..

So or I make sure I import everything only where it is needed or (and
maybe better and more explicit) I remove manually from sys.modules all
the modules that I want to reload, what do you think?

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: reloading code and multiprocessing andrea crotti <andrea.crotti.0@gmail.com> - 2012-07-23 14:51 +0100

csiph-web