Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #14037

Python C-library import paths

Show key headers only | View raw

[Multipart message — attachments visible in raw view] - view raw

I am packaging apache tvm. It builds a C-library libtvm.so (and libtvm_runtime.so).
It also has a python interface which is how most people use it, so I've built that into python3-tvm

It has a /usr/bin/tvmc which fails if you run it due to not being able to find the installed c-libraries.

I have no idea how the python c-library-finding mechanism constructs
its path list, so I'm not sure where to prod this to make it
work. There is presumably a right place to add a path to look on, or
maybe to enable the 'it's in the standard debian system path - just do what /etc/ld.so.conf.d/* says'
functionality.

Currently I get this:
$ tvmc
Traceback (most recent call last):
  File "/usr/bin/tvmc", line 33, in <module>
    sys.exit(load_entry_point('tvm==0.8.0', 'console_scripts', 'tvmc')())
  File "/usr/bin/tvmc", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/tvm/__init__.py", line 26, in <module>
    from ._ffi.base import TVMError, __version__, _RUNTIME_ONLY
  File "/usr/lib/python3/dist-packages/tvm/_ffi/__init__.py", line 28, in <module>
    from .base import register_error
  File "/usr/lib/python3/dist-packages/tvm/_ffi/base.py", line 71, in <module>
    _LIB, _LIB_NAME = _load_lib()
  File "/usr/lib/python3/dist-packages/tvm/_ffi/base.py", line 51, in _load_lib
    lib_path = libinfo.find_lib_path()
  File "/usr/lib/python3/dist-packages/tvm/_ffi/libinfo.py", line 146, in find_lib_path
    raise RuntimeError(message)
RuntimeError: Cannot find the files.
List of candidates:
/home/wookey/bin/libtvm.so
/usr/local/bin/libtvm.so
/usr/bin/libtvm.so
/bin/libtvm.so
/usr/local/games/libtvm.so
/usr/games/libtvm.so
/usr/lib/python3/dist-packages/tvm/libtvm.so
/usr/lib/libtvm.so
/home/wookey/bin/libtvm_runtime.so
/usr/local/bin/libtvm_runtime.so
/usr/bin/libtvm_runtime.so
/bin/libtvm_runtime.so
/usr/local/games/libtvm_runtime.so
/usr/games/libtvm_runtime.so
/usr/lib/python3/dist-packages/tvm/libtvm_runtime.so
/usr/lib/libtvm_runtime.so

So it tries quite hard to find it, but doesn't know about multiarch and thus fails to look in the right place:
/usr/lib/<triplet>/   (/usr/lib/x86_64-linux-gnu/ on this box)

Also does python really think that /usr/local/games/ should be cheked before /usr/lib/ ? That just seems wrong.

Clues about where to prod gratefully received. 

I see that /usr/lib/python3/dist-packages/tvm/_ffi/libinfo.py contains
a function 'get_dll_directories' which looks promising and adds
TVM_LIBRARY_PATH to the search list and if I run tvmc like this:
 TVM_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ tvmc
then that path is at the top of the list.

OK, but that mostly reveals a second issue: it's looking for
libtvm.so, but that unversioned link is only provoded in the dev package
libtvm-dev. The library package has the versioned filenames
/usr/lib/x86_64-linux-gnu/libtvm.so.0
/usr/lib/x86_64-linux-gnu/libtvm_runtime.so.0

So I also have to persuade it to look for libtvm.so.0 not
libtvm.so. Where does that info live?  OK, a bit more research shows
that that is in /usr/lib/python3/dist-packages/tvm/_ffi/libinfo.py
which is in the source as python/tvm/_ffi_libinfo.py, in find_lib_path
and that's easy to fix, and probably even the right place to fix it?

The paths is harder though. get_dll_directories in
python/tvm/_ffi_libinfo.py adds $PATH after $LD_LIBRARY_PATH to make
it's search list. Is searching $PATH for libraries ever right?

What it should actually be adding is what's in /etc/ld.so.conf.d/*
That can be listed with
/sbin/ldconfig -v 2>/dev/null | grep -v ^$'\t' | cut -d: -f1
(yuk? is there really no better way?)

How does one do that in python to get that set of path added in the
libinfo.py function?
https://github.com/apache/tvm/blob/main/python/tvm/_ffi/libinfo.py

Am I barking up the right tree here or is there a better way?

Wookey
-- 
Principal hats:  Debian, Wookware, ARM
http://wookware.org/

Back to linux.debian.maint.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Python C-library import paths Wookey <wookey@wookware.org> - 2022-04-01 03:40 +0200
  Re: Python C-library import paths "M. Zhou" <lumin@debian.org> - 2022-04-01 06:50 +0200
    Re: Python C-library import paths Wookey <wookey@wookware.org> - 2022-04-02 14:00 +0200
      Re: Python C-library import paths Simon McVittie <smcv@debian.org> - 2022-04-02 17:10 +0200
        Re: Python C-library import paths Wookey <wookey@wookware.org> - 2022-04-03 04:40 +0200

csiph-web