Path: csiph.com!weretis.net!feeder8.news.weretis.net!news.uzoreto.com!newsfeed.xs4all.nl!newsfeed7.news.xs4all.nl!bofh.it!news.nic.it!robomod From: "M. Zhou" Newsgroups: linux.debian.maint.python Subject: Re: Python C-library import paths Date: Fri, 01 Apr 2022 06:50:01 +0200 Message-ID: References: X-Mailbox-Line: From debian-python-request@lists.debian.org Fri Apr 1 04:48:09 2022 Old-Return-Path: X-Amavis-Spam-Status: No, score=-6.971 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, FOURLA=0.1, LDO_WHITELIST=-5, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SARE_MSGID_LONG40=0.637, T_SCC_BODY_TEXT_LINE=-0.01] autolearn=unavailable autolearn_force=no X-Policyd-Weight: using cached result; rate: -4.6 X-Riseup-User-ID: 83B3FAB5016E7D0159A7EE946B3B50770F0CB63E3E30D7556E51F76E17A5363D Content-Type: text/plain; charset="UTF-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Mailing-List: archive/latest/19860 List-ID: List-URL: List-Archive: https://lists.debian.org/msgid-search/33a7e6fb1bd2b913e60ce4e56554acfa3fcb9d0e.camel@debian.org Approved: robomod@news.nic.it Lines: 80 Organization: linux.* mail to news gateway Sender: robomod@news.nic.it X-Original-Cc: Debian DL/AI list X-Original-Date: Fri, 01 Apr 2022 00:30:23 -0400 X-Original-Message-ID: <33a7e6fb1bd2b913e60ce4e56554acfa3fcb9d0e.camel@debian.org> X-Original-References: <20220401013257.GS23856@mail.wookware.org> Xref: csiph.com linux.debian.maint.python:14038 On Fri, 2022-04-01 at 02:32 +0100, Wookey wrote: > > RuntimeError: Cannot find the files. > List of candidates: > /home/wookey/bin/libtvm.so > /usr/local/bin/libtvm.so > /usr/bin/libtvm.so > /bin/libtvm.so > /usr/local/games/libtvm.so > /usr/games/libtvm.so > /usr/lib/python3/dist-packages/tvm/libtvm.so > /usr/lib/libtvm.so > /home/wookey/bin/libtvm_runtime.so > /usr/local/bin/libtvm_runtime.so > /usr/bin/libtvm_runtime.so > /bin/libtvm_runtime.so > /usr/local/games/libtvm_runtime.so > /usr/games/libtvm_runtime.so > /usr/lib/python3/dist-packages/tvm/libtvm_runtime.so > /usr/lib/libtvm_runtime.so > > So it tries quite hard to find it, but doesn't know about multiarch > and thus fails to look in the right place: > /usr/lib//   (/usr/lib/x86_64-linux-gnu/ on this box) dlopen should know the multiarch triplet on debian. They have written their own ffi loader, so I think it is an upstream bug. The upstream should detect and add multiarch directory to the paths. > > I see that /usr/lib/python3/dist-packages/tvm/_ffi/libinfo.py > contains > a function 'get_dll_directories' which looks promising and adds > TVM_LIBRARY_PATH to the search list and if I run tvmc like this: >  TVM_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ tvmc > then that path is at the top of the list. User specified stuff goes first. That makes sense. > OK, but that mostly reveals a second issue: it's looking for > libtvm.so, but that unversioned link is only provoded in the dev > package > libtvm-dev. The library package has the versioned filenames > /usr/lib/x86_64-linux-gnu/libtvm.so.0 > /usr/lib/x86_64-linux-gnu/libtvm_runtime.so.0 I think it is fine to let it dlopen the libtvm.so, as it says itself as some sort of "compiler". Take pytorch as example, python3-torch has some functionalities for extending itself with C++. As a result, libtorch-dev is a dependency of python3-torch. > So I also have to persuade it to look for libtvm.so.0 not > libtvm.so. Where does that info live?  OK, a bit more research shows > that that is in /usr/lib/python3/dist-packages/tvm/_ffi/libinfo.py > which is in the source as python/tvm/_ffi_libinfo.py, in > find_lib_path > and that's easy to fix, and probably even the right place to fix it? If the upstream's own ffi loader does not look at the SOVERSION, then it should be designed to work without it. > The paths is harder though. get_dll_directories in > python/tvm/_ffi_libinfo.py adds $PATH after $LD_LIBRARY_PATH to make > it's search list. Is searching $PATH for libraries ever right? > > What it should actually be adding is what's in /etc/ld.so.conf.d/* > That can be listed with > /sbin/ldconfig -v 2>/dev/null | grep -v ^$'\t' | cut -d: -f1 > (yuk? is there really no better way?) > > How does one do that in python to get that set of path added in the > libinfo.py function? > https://github.com/apache/tvm/blob/main/python/tvm/_ffi/libinfo.py > > Am I barking up the right tree here or is there a better way? > This really looked like an ffi loader bug.