Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python.announce > #3551

[ANN] pyastinterp - Tree-walking interpreter for Python (+ "advent of code" idea)

Show key headers only | View raw

Hello,

pyastinterp is a tree-walking (aka meta-circular) interpreter for
Python. It was the most complete tree-walking interpreter as of a year
ago. As it was written as a personal "advent of code" project during
the winter holidays of 2019/2020.

I remembered about it this holiday season, and decided to post about
it, both as a holiday eyecandy, and as a challenge to other Python
hackers (such winter holidays challenges are known as "advent of
code" (of course, unless you believe there's only one of them)): can you
write a more complete implementation? Or maybe you did during this
year? Please post about it somewhere (perhaps, python-ideas mailing
list?).

https://github.com/pfalcon/pyastinterp


README with more details follows:

pyastinterp
===========

This is an AST interpreter (also known as tree-walking interpreter) for
the Python programming language, written in Python itself. It is thus
a meta-circular interpreter. While Python implementations of a Python VM
(i.e. bytecode interpreters) are relatively well known (starting with
PyPy), AST interpreters are much less common beyond those which implement
a very limited subset of the syntax.

Pyastinterp strives to fulfil 2 goals:

1. Be as complete as possible (and see how much effort that takes and
   if there're any limits).
2. Yet be as minimal as possible, avoiding useless boilerplate and
   reimplementing what may be already available.

First goal is pretty obvious: ideally, we'd like Pyastinterp to be able
to interpret (run) any application a native Python interpreter can.

Second goal is rooted in many reasons. To start with, Pyastinterp is
a spiritual part of the [Pycopy](https://github.com/pfalcon/pycopy)
project, whose goal is the development of minimalist Python-compatible
ecosystem. Besides, "bloated" meta-circular interpreters are well out
there (the same PyPy) - there's lack on the opposite side of spectrum.
Pyastinterp is also intended to be a "teaching" and "for-research"
project. Surely, it's much easier to learn and reuse/repurpose a small,
rather than large, project. These last requirements however also put
bounds on "minimality": `pyastinterp` tries to have a clear focus
(AST interpretation) and avoid unneeded boilerplate, but not be
obfuscated, and actually tries to be clear and easily hackable (provide
some reusable infrastructure), even at the expense of somewhat less
optimality/performance.

To achieve the goals of minimality, `pyastinterp` does following:

1. Is concerned with syntax, not semantics (largely).
2. That means that it tries to interpret Python AST tree, but
   relies on the underlying Python runtime to implement the actual
   operations (semantics). In some cases `pyastinterp` actually
   has to deal with runtime semantics, but the point is that it
   should be done only when there's no other choice (i.e. rarely).
3. This in particular means that `pyastinterp` itself requires a
   (pretty) full Python implementation to run, it's not the case
   that it's written in "some subset of Python". (Contrast that
   with e.g. PyPy's RPython, which is a Python subset language,
   in which interpreters are written).
4. Another consequence is that there's no semantic separation
   between "metalevel" (level of the interpreter) and target
   application level. This is again unlike PyPy, where 2 are explicitly
   separated. Lack of separation allows to save on "marshalling"
   values between the 2 levels, but also make it possible to have
   "leaks" between levels, e.g. unexpected and unhandled exception
   in the interpreter to be delivered to the application, causing
   havoc and confusion. Pyastinterp's response to this concern
   is a pledge that there shouldn't be such unhandled interpreter-level
   exceptions. Of course, catching all such cases may take quite
   a lot of testing and debugging, and fixing them may affect code
   minimality and clarity. We'll see how it goes.
5. However, due to choices described above, implementation of many
   syntactic constructs is very clear, concise, and isomorphic
   between the target and meta levels. See e.g. how "if" statement
   is implemented: the implementation looks almost exactly how the
   usage of the "if" statement is.


Supported Python subset
-----------------------

Pyastinterp should implement almost all Python3.5 features, stopping
at generators, which are not that easy to implement in a tree-walking
interpreter. It also does not include import hook, so the imported
modules are interpreted by the underlying Python VM, not by
Pyastinterp.


Running
-------

Quick way to run something interesting:

    wget https://raw.githubusercontent.com/python/cpython/3.6/Lib/test/pystone.py
    python3 -m astinterp pystone.my 10000

You can pass any other script on the command line and see how it goes.


Credits and licensing
---------------------

Pyastinterp is (c) [Paul Sokolovsky](https://github.com/pfalcon) and
is released under the MIT license.



-- 
Best regards,
 Paul                          mailto:pmiscml@gmail.com

Back to comp.lang.python.announce | Previous | Next | Find similar | Unroll thread

Thread

[ANN] pyastinterp - Tree-walking interpreter for Python (+ "advent of code" idea) Paul Sokolovsky <pmiscml@gmail.com> - 2020-12-23 19:01 +0300

csiph-web