Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #13932 > unrolled thread

Advice wanted: handling weird vendoring situation

Back to article view | Back to linux.debian.maint.python

Contents

  Advice wanted: handling weird vendoring situation Julian Gilbey <julian@d-and-j.net> - 2022-02-07 22:30 +0100
    Re: Advice wanted: handling weird vendoring situation Julian Gilbey <julian@d-and-j.net> - 2022-02-08 08:10 +0100
      Re: Advice wanted: handling weird vendoring situation Gregor Riepl <onitake@gmail.com> - 2022-02-09 19:40 +0100
        Re: Advice wanted: handling weird vendoring situation Julian Gilbey <julian@d-and-j.net> - 2022-02-12 22:30 +0100
          Re: Advice wanted: handling weird vendoring situation Gregor Riepl <onitake@gmail.com> - 2022-02-13 12:10 +0100
            Re: Advice wanted: handling weird vendoring situation Julian Gilbey <julian@d-and-j.net> - 2022-02-16 17:30 +0100

#13932 — Advice wanted: handling weird vendoring situation

Hi,

I'm working towards packaging pydevd (which is a recursive dependency
of Spyder via IPython), and it's a somewhat challenging package!  I
have hit an issue and would appreciate some thoughts on how best to
handle it.

Background:

pydevd is a debugging package which can attach to running scripts.  It
is used by PyDev, PyCharm, VSCode and Spyder, and with Spyder, it is
imported through debugpy, which in turn is imported into IPython.

In order to ensure that the libraries it is using have not been
modified by gevent, it uses vendored copies of various standard
library modules rather than just saying "import inspect" etc.

I thought I could address this issue by replacing the vendored copies
of the library modules by symlinks to /usr/lib/python3.X/, but now
I've hit another snag: some of these modules import other modules.
For example:

pydev_imps/_pydev_SimpleXMLRPCServer.py
is a very old version of /usr/lib/python3.X/xmlrpc/server.py.  It
contains within it the following lines:

from _pydev_imps import _pydev_xmlrpclib as xmlrpclib
from _pydev_imps._pydev_xmlrpclib import Fault
from _pydev_imps import _pydev_SocketServer as SocketServer
from _pydev_imps import _pydev_BaseHTTPServer as BaseHTTPServer

These libraries are:
_pydev_imps._pydev_xmlrpclib -> xmlrpc.client
_pydev_imps._pydev_SocketServer -> socketserver
_pydev_imps._pydev_BaseHTTPServer -> http.server

So what should I do?

One solution is just to symlink from _pydev_SimpleXMLRPCServer.py to
/usr/lib/python3.X/xmlrpc/server.py and not worry about the other
modules.  But that might break things in non-obvious ways, so I don't
want to do that.

Another possible solution is to update all of the vendored copies in
_pydev_imps using the relevant /usr/lib/python3.X/* modules and making
the same modifications to the imports to load local copies.  But then
we will have duplicate copies of standard library modules, which I
also don't want to do.

Perhaps another possibility is to have symlinks in the _pydev_imps
directory to the standard library versions and then temporarily modify
sys.path to look in _pydev_imps before looking in standard locations.
I don't know whether this will work, though.

(There is another snag, too, which is that the path will depend on the
Python version.  So I will probably have
_pydev_imps/python_39/socketserver.py -> /usr/lib/python3.9/socketserver.py
and
_pydev_imps/python_310/socketserver.py -> /usr/lib/python3.10/socketserver.py
But that's a simpler issue.)

Any thoughts on these ideas would be much appreciated!

Best wishes,

   Julian

[toc] | [next] | [standalone]

#13935

On Mon, Feb 07, 2022 at 09:27:28PM +0000, Julian Gilbey wrote:
> [...]
> 
> I thought I could address this issue by replacing the vendored copies
> of the library modules by symlinks to /usr/lib/python3.X/, but now
> I've hit another snag: some of these modules import other modules.
> For example:
> 
> pydev_imps/_pydev_SimpleXMLRPCServer.py
> is a very old version of /usr/lib/python3.X/xmlrpc/server.py.  It
> contains within it the following lines:
> 
> from _pydev_imps import _pydev_xmlrpclib as xmlrpclib
> from _pydev_imps._pydev_xmlrpclib import Fault
> from _pydev_imps import _pydev_SocketServer as SocketServer
> from _pydev_imps import _pydev_BaseHTTPServer as BaseHTTPServer
> [...]
> Perhaps another possibility is to have symlinks in the _pydev_imps
> directory to the standard library versions and then temporarily modify
> sys.path to look in _pydev_imps before looking in standard locations.
> I don't know whether this will work, though.

I realise now that this "nice" solution won't work, as the standard
library code says:

import socketserver

so modifying sys.path will just change the value of
sys.modules["socketserver"].  However, the vendored code instead loads
this module to sys.modules["_pydev_imps._pydev_SocketServer"] or
something like that, deliberately avoiding interfering with
sys.modules["socketserver"].

Ho hum.

   Julian

[toc] | [prev] | [next] | [standalone]

#13936

> I realise now that this "nice" solution won't work, as the standard
> library code says:
> 
> import socketserver
> 
> so modifying sys.path will just change the value of
> sys.modules["socketserver"].  However, the vendored code instead loads
> this module to sys.modules["_pydev_imps._pydev_SocketServer"] or
> something like that, deliberately avoiding interfering with
> sys.modules["socketserver"].

It seems to me that the "correct" solution would be to motivate upstream
not to vendor anything in their source tree. If they really need
vendoring to avoid compatibility issues with various environments, they
should do so only when building releases. It still wouldn't solve the
problem of incompatible system modules, but at least it would make it
clearer which versions they require and why.

Perhaps they have a maintenance script for updating the vendored
dependencies? You could use that to find out how to reverse the changes,
or start from a clean slate?

[toc] | [prev] | [next] | [standalone]

#13943

On Wed, Feb 09, 2022 at 07:39:40PM +0100, Gregor Riepl wrote:
> > I realise now that this "nice" solution won't work, as the standard
> > library code says:
> > 
> > import socketserver
> > 
> > so modifying sys.path will just change the value of
> > sys.modules["socketserver"].  However, the vendored code instead loads
> > this module to sys.modules["_pydev_imps._pydev_SocketServer"] or
> > something like that, deliberately avoiding interfering with
> > sys.modules["socketserver"].
> 
> It seems to me that the "correct" solution would be to motivate upstream
> not to vendor anything in their source tree. If they really need
> vendoring to avoid compatibility issues with various environments, they
> should do so only when building releases. It still wouldn't solve the
> problem of incompatible system modules, but at least it would make it
> clearer which versions they require and why.

Hi Gregor,

Thanks!

And indeed, that would be a good idea, but I doubt it's going to
happen :( They don't need "vendoring" as far as I can tell, but they
just need a pristine copy of the system library that can be loaded
independently of the system library, so that if gevent patches the
system library at runtime, this package still has access to an
unpatched copy.

So the solution I'm currently in the process of trying is to copy the
version from the oldest supported Python version at Debian package
build time.  We'll see how this goes....

> Perhaps they have a maintenance script for updating the vendored
> dependencies? You could use that to find out how to reverse the changes,
> or start from a clean slate?

Unlikely; some of their vendored dependencies date back to Python 2.5!

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#13944

> So the solution I'm currently in the process of trying is to copy the
> version from the oldest supported Python version at Debian package
> build time.  We'll see how this goes....
> 
>> Perhaps they have a maintenance script for updating the vendored
>> dependencies? You could use that to find out how to reverse the changes,
>> or start from a clean slate?
> 
> Unlikely; some of their vendored dependencies date back to Python 2.5!

In that case, I think this is the issue that must be solved first:
Ensuring that their code is compatible with the *latest* published
version, and vendoring the system version at build time.

Not a good solution, since it will still leave the system vulnerable
when one of the dependencies gets a security update, but better than
shipping a static version that might have numerous issues and will no
longer receive any patches.

The alternative would be that they take full responsibility for their
vendored code, but then it will be much harder to detect when they're
affected by a vulnerability.

[toc] | [prev] | [next] | [standalone]

#13950

On Sun, Feb 13, 2022 at 12:07:44PM +0100, Gregor Riepl wrote:
> > So the solution I'm currently in the process of trying is to copy the
> > version from the oldest supported Python version at Debian package
> > build time.  We'll see how this goes....
> > 
> >> Perhaps they have a maintenance script for updating the vendored
> >> dependencies? You could use that to find out how to reverse the changes,
> >> or start from a clean slate?
> > 
> > Unlikely; some of their vendored dependencies date back to Python 2.5!
> 
> In that case, I think this is the issue that must be solved first:
> Ensuring that their code is compatible with the *latest* published
> version, and vendoring the system version at build time.

Hi Gregor,

Thanks for your thoughts.

Yes, and that is what I'm going with.  Unfortunately, upstream are
trying to keep pydevd compatible with almost every version of Python
that exists (Python 2.7, Python 3.x, PyPy, IronPython and Jython), so
they cannot necessarily use the latest versions of the library.  But
I'm only going to aim to support the supported versions of CPython in
Debian, so I can use the latest versions of the library.  (Actually, I
am using the version in the lowest Debian-supported version of Python,
in case a newer version of the library uses newly introduced syntax.)

> Not a good solution, since it will still leave the system vulnerable
> when one of the dependencies gets a security update, but better than
> shipping a static version that might have numerous issues and will no
> longer receive any patches.

Indeed.

> The alternative would be that they take full responsibility for their
> vendored code, but then it will be much harder to detect when they're
> affected by a vulnerability.

Quite.

Best wishes,

   Julian

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web