Path: csiph.com!weretis.net!feeder8.news.weretis.net!news.mixmin.net!aioe.org!bofh.it!news.nic.it!robomod
From: Wookey <wookey@wookware.org>
Newsgroups: linux.debian.maint.python
Subject: Python C-library import paths
Date: Fri, 01 Apr 2022 03:40:01 +0200
Message-ID: <E7eqB-5z0H-3@gated-at.bofh.it>
X-Mailbox-Line: From debian-python-request@lists.debian.org  Fri Apr  1 01:33:17 2022
Old-Return-Path: <wookey@wookware.org>
X-Amavis-Spam-Status: No, score=-14.21 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, FOURLA=0.1, LDO_WHITELIST=-5, PGPSIGNATURE=-5, RCVD_IN_DNSWL_MED=-2.3, T_SCC_BODY_TEXT_LINE=-0.01] autolearn=unavailable autolearn_force=no
X-Policyd-Weight: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 CL_IP_EQ_FROM_MX=-3.1; rate: -4.6
Mail-Followup-To: debian-python@lists.debian.org, Debian DL/AI list <debian-ai@lists.debian.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="THYEXwetZJOK3OLY"
Content-Disposition: inline
Organization: Wookware
User-Agent: Mutt/1.10.1 (2018-07-13)
X-Mailing-List: <debian-python@lists.debian.org> archive/latest/19859
List-ID: <debian-python.lists.debian.org>
List-URL: <https://lists.debian.org/debian-python/>
List-Archive: https://lists.debian.org/msgid-search/20220401013257.GS23856@mail.wookware.org
Approved: robomod@news.nic.it
Lines: 158
Sender: robomod@news.nic.it
X-Original-Cc: Debian DL/AI list <debian-ai@lists.debian.org>
X-Original-Date: Fri, 1 Apr 2022 02:32:58 +0100
X-Original-Message-ID: <20220401013257.GS23856@mail.wookware.org>
Xref: csiph.com linux.debian.maint.python:14037


--THYEXwetZJOK3OLY
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

I am packaging apache tvm. It builds a C-library libtvm.so (and libtvm_runt=
ime.so).
It also has a python interface which is how most people use it, so I've bui=
lt that into python3-tvm

It has a /usr/bin/tvmc which fails if you run it due to not being able to f=
ind the installed c-libraries.

I have no idea how the python c-library-finding mechanism constructs
its path list, so I'm not sure where to prod this to make it
work. There is presumably a right place to add a path to look on, or
maybe to enable the 'it's in the standard debian system path - just do what=
 /etc/ld.so.conf.d/* says'
functionality.

Currently I get this:
$ tvmc
Traceback (most recent call last):
  File "/usr/bin/tvmc", line 33, in <module>
    sys.exit(load_entry_point('tvm=3D=3D0.8.0', 'console_scripts', 'tvmc')(=
))
  File "/usr/bin/tvmc", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
    module =3D import_module(match.group('module'))
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_modu=
le
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_remo=
ved
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_remo=
ved
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_remo=
ved
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_remo=
ved
  File "/usr/lib/python3/dist-packages/tvm/__init__.py", line 26, in <modul=
e>
    from ._ffi.base import TVMError, __version__, _RUNTIME_ONLY
  File "/usr/lib/python3/dist-packages/tvm/_ffi/__init__.py", line 28, in <=
module>
    from .base import register_error
  File "/usr/lib/python3/dist-packages/tvm/_ffi/base.py", line 71, in <modu=
le>
    _LIB, _LIB_NAME =3D _load_lib()
  File "/usr/lib/python3/dist-packages/tvm/_ffi/base.py", line 51, in _load=
_lib
    lib_path =3D libinfo.find_lib_path()
  File "/usr/lib/python3/dist-packages/tvm/_ffi/libinfo.py", line 146, in f=
ind_lib_path
    raise RuntimeError(message)
RuntimeError: Cannot find the files.
List of candidates:
/home/wookey/bin/libtvm.so
/usr/local/bin/libtvm.so
/usr/bin/libtvm.so
/bin/libtvm.so
/usr/local/games/libtvm.so
/usr/games/libtvm.so
/usr/lib/python3/dist-packages/tvm/libtvm.so
/usr/lib/libtvm.so
/home/wookey/bin/libtvm_runtime.so
/usr/local/bin/libtvm_runtime.so
/usr/bin/libtvm_runtime.so
/bin/libtvm_runtime.so
/usr/local/games/libtvm_runtime.so
/usr/games/libtvm_runtime.so
/usr/lib/python3/dist-packages/tvm/libtvm_runtime.so
/usr/lib/libtvm_runtime.so

So it tries quite hard to find it, but doesn't know about multiarch and thu=
s fails to look in the right place:
/usr/lib/<triplet>/   (/usr/lib/x86_64-linux-gnu/ on this box)

Also does python really think that /usr/local/games/ should be cheked befor=
e /usr/lib/ ? That just seems wrong.

Clues about where to prod gratefully received.=20

I see that /usr/lib/python3/dist-packages/tvm/_ffi/libinfo.py contains
a function 'get_dll_directories' which looks promising and adds
TVM_LIBRARY_PATH to the search list and if I run tvmc like this:
 TVM_LIBRARY_PATH=3D/usr/lib/x86_64-linux-gnu/ tvmc
then that path is at the top of the list.

OK, but that mostly reveals a second issue: it's looking for
libtvm.so, but that unversioned link is only provoded in the dev package
libtvm-dev. The library package has the versioned filenames
/usr/lib/x86_64-linux-gnu/libtvm.so.0
/usr/lib/x86_64-linux-gnu/libtvm_runtime.so.0

So I also have to persuade it to look for libtvm.so.0 not
libtvm.so. Where does that info live?  OK, a bit more research shows
that that is in /usr/lib/python3/dist-packages/tvm/_ffi/libinfo.py
which is in the source as python/tvm/_ffi_libinfo.py, in find_lib_path
and that's easy to fix, and probably even the right place to fix it?

The paths is harder though. get_dll_directories in
python/tvm/_ffi_libinfo.py adds $PATH after $LD_LIBRARY_PATH to make
it's search list. Is searching $PATH for libraries ever right?

What it should actually be adding is what's in /etc/ld.so.conf.d/*
That can be listed with
/sbin/ldconfig -v 2>/dev/null | grep -v ^$'\t' | cut -d: -f1
(yuk? is there really no better way?)

How does one do that in python to get that set of path added in the
libinfo.py function?
https://github.com/apache/tvm/blob/main/python/tvm/_ffi/libinfo.py

Am I barking up the right tree here or is there a better way?

Wookey
--=20
Principal hats:  Debian, Wookware, ARM
http://wookware.org/

--THYEXwetZJOK3OLY
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEER4nvI8Pe/wVWh5yq+4YyUahvnkcFAmJGVkQACgkQ+4YyUahv
nkeTNRAAlfz/w5bg2pZhh1uqrI7gJLnVLRJ+BwTo2eXJcxYQyXGfsJ4kyQlYHcSi
OnHKSlOiQKHrfHjKB2J5GwgXs+NNXECbUnAfXn6jW1vYHIAZ8CUlLAFnv89ZHavh
jgpCx5r/Brh5jXUBms4wq2j50s7PEjoiYRfXfgxILHn9nhpwaMbBlLgU4gvB0u/y
mvk8MQjpSq8Q/RDYL4rd4YnpPHJ77amN2Bpt8DTkEONREbmKKOFHKagL0ILeE3Br
LxcWQ883uiuibSJmLR2N/10/t+XVeIWE5buvfPrFKR+lkXzNPxT+91fbgqcDQ3KW
2ZBucOphl5NmPfGqwLgbAjE5yzDs5xn6FY5Hgv24VpPe0REz5EFGpOgspFWgx0mH
xOid7xrSmuXHI2q3mrbYXwIJ9q4l2SOSVjb5TuyHstCQbeceZpOv1DlDCkPWYraP
+gGhl/wfyA3CYXC1NCaZDc13Pm+WwEESFv6PaCOXPrJvXKZhvVNqG5hFakQjwL3F
qxiZEBNruw4MjJWloioHD0HWofkjK4J2vL0IjADCvTz5Fvu5cKYgIKZ2bkdO8jv4
o7IYapEoTt5tFZx8iXMJVB1qRXzM51qOiiHZqSDifsyHuTTISgGshbuYMdqu6kDR
IwDVjAiRZMTm54YnlCntIeh+q77tgZbT/1DtniT5mDAcn3rOuyA=
=Ctex
-----END PGP SIGNATURE-----

--THYEXwetZJOK3OLY--