Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Peter Otten <__peter__@web.de>
Newsgroups: comp.lang.python
Subject: Re: Technique for safely reloading dynamically generated module
Date: Thu, 21 Jul 2016 17:40:16 +0200
Organization: None
Lines: 58
Message-ID: <mailman.28.1469115627.22221.python-list@python.org>
References: <1469112141.379402.672830825.05659985@webmail.messagingengine.com> <nmqqd2$lcd$1@ger.gmane.org>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.13.3
Precedence: list
Xref: csiph.com comp.lang.python:111718

Malcolm Greene wrote:

> We're designing a server application that parses a custom DSL (domain
> specific language) source file, generates a Python module with the
> associated logic, and runs the associated code. Since this is a server
> application, we need to reload the module after each regeneration. Is
> this process is simple as the following pseudo code or are there other
> issues we need to be aware of? Are there better techniques for this
> workflow (eval, compile, etc)?
>  
> We're working in Python 3.5.1.
> 
> import importlib
>  
> # custom_code is the module our code will generate - a version of this
> # file will always be present
> # if custom_code.py is missing, a blank version of this file is created
> # before this step
> import custom_code
>  
> while True:
>     # (re)generates custom_code.py visible in sys.path
>    generate_custom_code( source_file )
>  
>    # reload the module whose source we just generated
>    importlib.reload( custom_code )
>  
>    # run the main code in generated module
>    custom_code.go()

If the go() function in that loop is the only place where the generated 
module is used you can avoid writing code to the file system altogether.

Here's a really simple alternative suggestion based on exec():

SOURCEFILE = "whatever.dsl"


def make_func(sourcefile):
    ns = {}
    exec(generated_code(sourcefile), ns)
    return ns["go"]


def generated_code(sourcefile):
    # placeholder for your actual source generation
    return """
def go():
    print("Generated from", {!r})
""".format(sourcefile)


while True:
    # XXX should we wait for updates of sourcefile here?
    go = make_func(SOURCEFILE)
    go()