Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.bugs.dist > #1170758 > unrolled thread
| Started by | Nilesh Patra <nilesh@riseup.net> |
|---|---|
| First post | 2023-10-03 23:00 +0200 |
| Last post | 2023-10-10 10:10 +0200 |
| Articles | 5 — 3 participants |
Back to article view | Back to linux.debian.bugs.dist
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Bug#1053314: Depends: python3-h5py-mpi without python3-h5py Nilesh Patra <nilesh@riseup.net> - 2023-10-03 23:00 +0200
Bug#1053314: Depends: python3-h5py-mpi without python3-h5py Nilesh Patra <nilesh@riseup.net> - 2023-10-08 17:50 +0200
Bug#1053314: Depends: python3-h5py-mpi without python3-h5py Rafael Laboissière <rafael@debian.org> - 2023-10-09 08:40 +0200
Bug#1053314: Depends: python3-h5py-mpi without python3-h5py Rafael Laboissière <rafael@debian.org> - 2023-10-10 09:40 +0200
Bug#1053314: Depends: python3-h5py-mpi without python3-h5py Drew Parsons <dparsons@debian.org> - 2023-10-10 10:10 +0200
| From | Nilesh Patra <nilesh@riseup.net> |
|---|---|
| Date | 2023-10-03 23:00 +0200 |
| Subject | Bug#1053314: Depends: python3-h5py-mpi without python3-h5py |
| Message-ID | <HkUEN-dksC-1@gated-at.bofh.it> |
On Sun, 01 Oct 2023 15:25:43 +0200 Drew Parsons <dparsons@debian.org> wrote: > Package: xmds2 > Version: 3.1.0+dfsg2-4 > Severity: serious > Justification: debci > > xmds2 Depends: python3-h5py-mpi without depending on python3-h5py > > python3-h5py-mpi only provides the h5py._debian_h5py_mpi namespace, > not h5py. Hence tests using h5py without specifying _debian_h5py_mpi > fail. This is the case with h5py 3.9.0. (Marking Severity: serious due > to debci failure) > > python3-h5py provides the h5py namespace, so if xmds2 strictly needs > the mpi build, but accessing via the generic h5py namespace, then the > Depends should specify both packages > Depends: python3-h5py python3-h5py-mpi > > > Note there seems to be an inconsistency in the xmds2 package > configuration. It has MPI dependencies (python3-h5py-mpi, also > mpi-default-dev, libfftw3-mpi-dev), but with respect to hdf5 it has > Depends: libhdf5-serial-dev > Should that instead be > Depends: libhdf5-mpi-dev ? Seems you're right, taking a brief look at it. I've CC'ed Rafael to further comment/upload a fix. @Rafael, this seems to be the last blocker on h5py transition to testing, so prompt action would be really cool! Best, Nilesh
[toc] | [next] | [standalone]
| From | Nilesh Patra <nilesh@riseup.net> |
|---|---|
| Date | 2023-10-08 17:50 +0200 |
| Message-ID | <HmEcy-epKp-11@gated-at.bofh.it> |
| In reply to | #1170758 |
Hello,
On 10/8/23 17:22, Rafael Laboissière wrote:
> Ok, I tried to fix the building problem by including python3-h5py, alongside with python3-h5py-mpi, into Build-Depends, as suggested by Drew, but the xmds2 package FTBFS.
>
> Here is a way to reproduce the problem without building the package:
>
> $ dpkg -l python3-h5py\*
> Desired=Unknown/Install/Remove/Purge/Hold
> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name Version Architecture Description
> +++-===================-============-============-=======================================================
> ii python3-h5py 3.9.0-3 all general-purpose Python interface to hdf5
> ii python3-h5py-mpi 3.9.0-3 amd64 general-purpose Python interface to hdf5 (Python 3 MPI)
> un python3-h5py-serial <none> <none> (no description available)
> $ echo 'import h5py' | python3
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib/python3/dist-packages/h5py/__init__.py", line 21, in <module>
> from . import _debian_h5py_serial as _h5py
> ImportError: cannot import name '_debian_h5py_serial' from partially initialized module 'h5py' (most likely due to a circular import) (/usr/lib/python3/dist-packages/h5py/__init__.py)
>
> Is there a way to force h5py to import _debian_h5py_serial instead of _debian_h5py_mpi, via the generic h5py namespace?
Drew would probably answer that question better but from taking a brief look, it seems to be on expected lines.
This should work if you run it explicitly with mpi.
$ mpirun -n 1 python3 -c "import h5py" && echo "true"
true
or with setting the MPI var manually.
$ OMPI_COMM_WORLD_SIZE=1 python3 -c "import h5py" && echo "true"
true
If you want the _debian_h5py_serial interface then you need python3-h5py-serial and the B-D (and Depends) on h5py-mpi
should be dropped which would mean this package does not need the -mpi package.
Otherwise, a (unreliable) hack that you could do it that add a B-D on h5py *before* mpi and then -serial should also be installed (at least on my env).
If the code really needs h5py-mpi, then it should be running the build/tests with mpi enabled (via openmpi).
At least that's the impression I get from reading.
https://sources.debian.org/src/h5py/3.9.0-3/debian/README.Debian/
This patch gets the package building for me with h5py-mpi+h5py, but not sure if it is the right thing to do -- please verify for yourself as package maintainer :)
--- a/xpdeint/XSILFile.py
+++ b/xpdeint/XSILFile.py
@@ -31,6 +31,9 @@
numpy = None
+# Set env var to use h5py-mpi
+os.environ['OMPI_COMM_WORLD_SIZE'] = '1'
+
def require_h5py():
global h5py
if not h5py:
[toc] | [prev] | [next] | [standalone]
| From | Rafael Laboissière <rafael@debian.org> |
|---|---|
| Date | 2023-10-09 08:40 +0200 |
| Message-ID | <HmS5P-eyt1-5@gated-at.bofh.it> |
| In reply to | #1171176 |
Thanks for this detailed explanation, Drew. I released version 3.1.0+dfsg2-5 of the xmds2 package before reading it. I added python3-h5py to Build-Depends and libhdf5-mpi-dev to Depends, as you suggested (even though there is a typo in the debian/changelog entry, stating eroneaously that libhdf5-serial-dev has been added; I will fix this in the next release). I also used H5PY_ALWAYS_USE_MPI=1, as you mentioned. As regards adding also python3-h5py-serial to Depends and putting a fallback code in place, I will have to give it a little thought. Maybe, I should discuss this with the upstream authors, to know what they thing. Let us see how things evolve. At least, I hope that version 3.1.0+dfsg2-5 will really fix Bug#1053314 and the h5py transition will be completed. Best, Rafael * Drew Parsons <dparsons@debian.org> [2023-10-09 02:23]: > Nilesh explained most of the situation correctly. I can give some > more background. It made sense (to me) to have h5py built against > hdf5-mpi, since I figured that if you need the complexity of the hdf5 > file format then you probably want to use it in an MPI environment. > > There was a complaint from a user though, who wanted to make use of a > massive ensemble of HDF5 (h5py) serial jobs, and the small cost of > loading up MPI support was interfering with their throughput. > > So the compromise solution was to provide both builds, with a custom > __init__.py to select the serial or MPI build depending on runtime > environment. If an MPI environment is detected then the h5py MPI > build is loaded, otherwise the serial build is loaded. > > If you want to run h5py in a serial process, then one might say you'd > normally want the serial build. As Nilesh noted, I put in a mechanism > to load the MPI build if you really want to access the mpi build in a > serial process (mpirun -n 1 is not a "serial" process as such, it's > still an MPI environment even though using only 1 process). > > The mechanism to force MPI loading is NOT to set OMPI_COMM_WORLD_SIZE. > I recommend NOT doing that. I couldn't promise it won't mess up other > things, certainly it will get in the way of an MPICH environment. No, > the mechanism for handling this for h5py is described in > /usr/share/doc/python3-h5py/README.Debian: set H5PY_ALWAYS_USE_MPI=1 > >> Is there a way to force h5py to import _debian_h5py_serial instead of >> _debian_h5py_mpi, via the generic h5py namespace? > > It sounds like there is some confusion about how xmds2 should be used. > Is it intended to be used as a serial process or MPI? I noted in the > bug report that xmds2 Depends: libhdf5-serial-dev. Is it even using > MPI? If you want it to be using h5py-serial, then why does xmsd2 > depend on python3-h5py-mpi? > > It seems to me that xmds2's h5py dependency should be the same as its > hdf5 dependency. If it uses libhdf5-serial then should it be > depending on just python3-h5py (implying python3-h5py-serial, make it > explicit if needed) and not depend on python3-h5py-mpi? > > If xmds2 is intended to be flexible, equally happy in serial and MPI > environments (and can actually make use of h5py-mpi) then perhaps the > dependency should cover all cases, > Depends: python3-h5py, python3-h5py-serial, python3-h5py-mpi > all three explicitly, since otherwise one or the other of -serial or > -mpi would be missed. > > The problem raises interesting questions about h5py configuration. I > set up it so you could choose how you want it to work, with or without > MPI support. But it looks like an edge case is missing: it's failing > in serial jobs if you chose to set up your installation with > python3-h5py-mpi and explicitly don't want python3-h5py-serial (unless > you always set H5PY_ALWAYS_USE_MPI). Perhaps I should add an > additional fallback to try h5py-mpi if h5py-serial is not found (in a > serial job), the same way that h5py-serial is loaded as a fallback in > an MPI job if h5py-mpi is not found. On the other hand maybe that just > hides the real problem, that h5py-serial was not installed when > actually it was wanted after all. The ImportError correctly > identifies that case. > > > > > On 2023-10-08 17:38, Nilesh Patra wrote: >> Hello, >> >> On 10/8/23 17:22, Rafael Laboissière wrote: >>> Ok, I tried to fix the building problem by including python3-h5py, >>> alongside with python3-h5py-mpi, into Build-Depends, as suggested >>> by Drew, but the xmds2 package FTBFS. >>> >>> Here is a way to reproduce the problem without building the package: >>> >>> $ dpkg -l python3-h5py\* >>> Desired=Unknown/Install/Remove/Purge/Hold >>> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend >>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) >>> ||/ Name Version Architecture Description >>> +++-===================-============-============-======================================================= >>> ii python3-h5py 3.9.0-3 all >>> general-purpose Python interface to hdf5 >>> ii python3-h5py-mpi 3.9.0-3 amd64 >>> general-purpose Python interface to hdf5 (Python 3 MPI) >>> un python3-h5py-serial <none> <none> (no >>> description available) >>> $ echo 'import h5py' | python3 >>> Traceback (most recent call last): >>> File "<stdin>", line 1, in <module> >>> File "/usr/lib/python3/dist-packages/h5py/__init__.py", line >>> 21, in <module> >>> from . import _debian_h5py_serial as _h5py >>> ImportError: cannot import name '_debian_h5py_serial' from >>> partially initialized module 'h5py' (most likely due to a circular >>> import) (/usr/lib/python3/dist-packages/h5py/__init__.py) >>> >>> Is there a way to force h5py to import _debian_h5py_serial instead >>> of _debian_h5py_mpi, via the generic h5py namespace? >> >> Drew would probably answer that question better but from taking a >> brief look, it seems to be on expected lines. >> This should work if you run it explicitly with mpi. >> >> $ mpirun -n 1 python3 -c "import h5py" && echo "true" >> true >> >> or with setting the MPI var manually. >> >> $ OMPI_COMM_WORLD_SIZE=1 python3 -c "import h5py" && echo "true" >> true >> >> If you want the _debian_h5py_serial interface then you need >> python3-h5py-serial and the B-D (and Depends) on h5py-mpi >> should be dropped which would mean this package does not need the >> -mpi package. >> >> Otherwise, a (unreliable) hack that you could do it that add a B-D on >> h5py *before* mpi and then -serial should also be installed (at least >> on my env). >> >> If the code really needs h5py-mpi, then it should be running the >> build/tests with mpi enabled (via openmpi). >> At least that's the impression I get from reading. >> >> https://sources.debian.org/src/h5py/3.9.0-3/debian/README.Debian/ >> >> This patch gets the package building for me with h5py-mpi+h5py, but >> not sure if it is the right thing to do -- please verify for yourself >> as package maintainer :) >> >> --- a/xpdeint/XSILFile.py >> +++ b/xpdeint/XSILFile.py >> @@ -31,6 +31,9 @@ >> numpy = None >> +# Set env var to use h5py-mpi >> +os.environ['OMPI_COMM_WORLD_SIZE'] = '1' >> + >> def require_h5py(): >> global h5py >> if not h5py: >
[toc] | [prev] | [next] | [standalone]
| From | Rafael Laboissière <rafael@debian.org> |
|---|---|
| Date | 2023-10-10 09:40 +0200 |
| Message-ID | <Hnfvr-eOFV-1@gated-at.bofh.it> |
| In reply to | #1171216 |
* Rafael Laboissière <rafael@debian.org> [2023-10-09 08:30]: > […] At least, I hope that version 3.1.0+dfsg2-5 will really fix > Bug#1053314 and the h5py transition will be completed. Unfortunately, it is still not working: https://ci.debian.net/data/autopkgtest/testing/arm64/x/xmds2/38836874/log.gz I guess that the problem comes from the absence of /usr/bin/h5cc, which is in the hdf5-helpers package. This package, which is a dependency of libhdf5-dev, is not installed at the autopkgtest run. I confess that I am confused and need your advice here. Currently, the xmds2 package depends on libhdf5-mpi-dev. Should hdf5-helpers or libhdf5-dev be added to Depends? Best, Rafael
[toc] | [prev] | [next] | [standalone]
| From | Drew Parsons <dparsons@debian.org> |
|---|---|
| Date | 2023-10-10 10:10 +0200 |
| Message-ID | <HnfYt-eP4H-1@gated-at.bofh.it> |
| In reply to | #1171318 |
On 2023-10-10 09:26, Rafael Laboissière wrote: > * Rafael Laboissière <rafael@debian.org> [2023-10-09 08:30]: > >> […] At least, I hope that version 3.1.0+dfsg2-5 will really fix >> Bug#1053314 and the h5py transition will be completed. > > Unfortunately, it is still not working: > https://ci.debian.net/data/autopkgtest/testing/arm64/x/xmds2/38836874/log.gz > > I guess that the problem comes from the absence of /usr/bin/h5cc, > which is in the hdf5-helpers package. This package, which is a > dependency of libhdf5-dev, is not installed at the autopkgtest run. > > I confess that I am confused and need your advice here. Currently, the > xmds2 package depends on libhdf5-mpi-dev. Should hdf5-helpers or > libhdf5-dev be added to Depends? The hdf5 packages have some complex symlink management to allow for mpi to be either openmpi or mpich. Possibly xdms2 got caught by that. Sometimes the symlink goes wrong and the hdf-mpi packages need to be reinstalled to refresh it. But that shouldn't affect a buildd (or debci) run, it should already have a fresh installation of the packages. I'll take a closer look. Drew
[toc] | [prev] | [standalone]
Back to top | Article view | linux.debian.bugs.dist
csiph-web