Path: csiph.com!aioe.org!bofh.it!news.nic.it!robomod From: Neil Williams Newsgroups: linux.debian.maint.python Subject: Re: dxtbx pytest issue Date: Thu, 01 Sep 2022 14:10:01 +0200 Message-ID: References: X-Mailbox-Line: From debian-python-request@lists.debian.org Thu Sep 1 12:03:36 2022 Old-Return-Path: X-Amavis-Spam-Status: No, score=-113.421 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, DIGITS_LETTERS=1, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FOURLA=0.1, LDO_WHITELIST=-5, PGPSIGNATURE=-5, RCVD_IN_DNSWL_MED=-2.3, T_SCC_BODY_TEXT_LINE=-0.01, USER_IN_DKIM_WELCOMELIST=-0.01, USER_IN_DKIM_WHITELIST=-100] autolearn=ham autolearn_force=no X-Policyd-Weight: using cached result; rate: -4.6 X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/4_Lx9QRqGH8virppBAwRV_v"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Mailing-List: archive/latest/20261 List-ID: List-URL: List-Archive: https://lists.debian.org/msgid-search/20220901130312.728e52fd@felix.codehelp Approved: robomod@news.nic.it Lines: 297 Organization: linux.* mail to news gateway Sender: robomod@news.nic.it X-Original-Cc: debian-python X-Original-Date: Thu, 1 Sep 2022 13:03:12 +0100 X-Original-Message-ID: <20220901130312.728e52fd@felix.codehelp> X-Original-References: <605521306.3759500.1662031956039.JavaMail.zimbra@synchrotron-soleil.fr> Xref: csiph.com linux.debian.maint.python:14422 --Sig_/4_Lx9QRqGH8virppBAwRV_v Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Thu, 1 Sep 2022 13:32:36 +0200 (CEST) PICCA Frederic-Emmanuel wrote: > Hello, >=20 > I am still trying to package[2] dxtbx[1], and now I end up with > something strange. When I run the test suite during the build I have > a failure like this >=20 > tests/test_dataset_as_flex.py ..F..F..F > [ 2%] Looks like you need a -v option to see more detail. =20 > I put the error message bellow, it is quite long >=20 > now If I execute by hand only the failing test like this, it works >=20 >=20 > $ pytest-3 tests/test_dataset_as_flex.py=20 The 2% indicates that other tests have executed before the test series gets to test_dataset_as_flex.py. When you run it directly, no other tests are run. This could indicate that there is interference between tests (an upstream problem) in that the tests are not sufficiently discrete and are affected by which other tests are executed before the failing test. pytest has support to solve these problems, if the test suite is written correctly. It could also be a version problem with any of the build-dependencies (like h5py), so check for various requirements*.txt and see if upstream have a .gitlabci.yml or .github config which clarifies exactly what versions are used. You may need to exclude certain tests in case, for example, h5py in Debian is built with options that dxtbx cannot handle. > before investigating further, I would like your advice in order to > debug this sort of issue. >=20 > first what is the difference between >=20 > pytest and pytest pytest scans the directory tree to collect all the tests it can find. pytest only collects the tests from that file. Check for a pytest.ini in the upstream source tree and also setup.cfg Use the -v option to see more of what is going on and the --collect-only option is useful too. See pytest --help as other filtering options are available as well. You'll need to have a local build environment (fakeroot debian/rules build etc.) & dig into the Python unit test methods and see what is going on. > thanks for your help >=20 > Frederic >=20 > [1] https://github.com/cctbx/dxtbx > [2] https://salsa.debian.org/science-team/dxtbx >=20 >=20 >=20 > full error message >=20 > ___________ test_dataset_as_flex[int-dataset_as_flex_int-bshuf_lz4] > ____________ >=20 > type_name =3D 'int', creator =3D > converter =3D >=20 > @pytest.mark.parametrize( > "creator", > [ > uncompressed, > gzip, > bshuf_lz4, > ], > ) > @pytest.mark.parametrize( > "type_name,converter", > [ > ("int", dataset_as_flex_int), > ("float", dataset_as_flex_float), > ("double", dataset_as_flex_double), > ], > ) > def test_dataset_as_flex(type_name, creator, converter): > # Create an in-memory HDF5 dataset with unique name > f =3D h5py.File(type_name + ".h5", "w", driver=3D"core", > backing_store=3DFalse)=20 > shape =3D (20, 20, 20) > > dataset =3D creator(f, shape, type_name) =20 >=20 > tests/test_dataset_as_flex.py:64:=20 > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ tests/test_dataset_as_flex.py:34: in bshuf_lz4 > return file.create_dataset( > /usr/lib/python3/dist-packages/h5py/_debian_h5py_serial/_hl/group.py:161: > in create_dataset dsid =3D dataset.make_new_dset(group, shape, dtype, > data, name, **kwds) > /usr/lib/python3/dist-packages/h5py/_debian_h5py_serial/_hl/dataset.py:10= 6: > in make_new_dset dcpl =3D filters.fill_dcpl( _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _=20 >=20 > plist =3D 0x7fccdbbd9760> shape =3D (20, 20, 20), dtype =3D dtype('int64'), chunks > 0x7fccdbbd9760> =3D (10, 10, 10) > compression =3D 32008, compression_opts =3D (0, 2), shuffle =3D None > fletcher32 =3D None, maxshape =3D None, scaleoffset =3D None, external = =3D [] > allow_unknown_filter =3D False >=20 > def fill_dcpl(plist, shape, dtype, chunks, compression, > compression_opts, shuffle, fletcher32, maxshape, scaleoffset, > external, allow_unknown_filter=3DFalse): > """ Generate a dataset creation property list. > =20 > Undocumented and subject to change without warning. > """ > =20 > if shape is None or shape =3D=3D (): > shapetype =3D 'Empty' if shape is None else 'Scalar' > if any((chunks, compression, compression_opts, shuffle, > fletcher32, scaleoffset is not None)): > raise TypeError( > f"{shapetype} datasets don't support chunk/filter > options" ) > if maxshape and maxshape !=3D (): > raise TypeError(f"{shapetype} datasets cannot be > extended") return h5p.create(h5p.DATASET_CREATE) > =20 > def rq_tuple(tpl, name): > """ Check if chunks/maxshape match dataset rank """ > if tpl in (None, True): > return > try: > tpl =3D tuple(tpl) > except TypeError: > raise TypeError('"%s" argument must be None or a > sequence object' % name) if len(tpl) !=3D len(shape): > raise ValueError('"%s" must have same rank as dataset > shape' % name)=20 > rq_tuple(chunks, 'chunks') > rq_tuple(maxshape, 'maxshape') > =20 > if compression is not None: > if isinstance(compression, FilterRefBase): > compression_opts =3D compression.filter_options > compression =3D compression.filter_id > =20 > if compression not in encode and not > isinstance(compression, int): raise ValueError('Compression filter > "%s" is unavailable' % compression)=20 > if compression =3D=3D 'gzip': > if compression_opts is None: > gzip_level =3D DEFAULT_GZIP > elif compression_opts in range(10): > gzip_level =3D compression_opts > else: > raise ValueError("GZIP setting must be an integer > from 0-9, not %r" % compression_opts)=20 > elif compression =3D=3D 'lzf': > if compression_opts is not None: > raise ValueError("LZF compression filter accepts > no options")=20 > elif compression =3D=3D 'szip': > if compression_opts is None: > compression_opts =3D DEFAULT_SZIP > =20 > err =3D "SZIP options must be a 2-tuple ('ec'|'nn', > even integer 0-32)" try: > szmethod, szpix =3D compression_opts > except TypeError: > raise TypeError(err) > if szmethod not in ('ec', 'nn'): > raise ValueError(err) > if not (0 raise ValueError(err) > =20 > elif compression_opts is not None: > # Can't specify just compression_opts by itself. > raise TypeError("Compression method must be specified") > =20 > if scaleoffset is not None: > # scaleoffset must be an integer when it is not None or > False, # except for integral data, for which scaleoffset =3D=3D True is > # permissible (will use SO_INT_MINBITS_DEFAULT) > =20 > if scaleoffset < 0: > raise ValueError('scale factor must be >=3D 0') > =20 > if dtype.kind =3D=3D 'f': > if scaleoffset is True: > raise ValueError('integer scaleoffset must be > provided for ' 'floating point types') > elif dtype.kind in ('u', 'i'): > if scaleoffset is True: > scaleoffset =3D h5z.SO_INT_MINBITS_DEFAULT > else: > raise TypeError('scale/offset filter only supported > for integer ' 'and floating-point types') > =20 > # Scale/offset following fletcher32 in the filter chain > will (almost?) # always triggers a read error, as most scale/offset > settings are # lossy. Since fletcher32 must come first (see comment > below) we # simply prohibit the combination of fletcher32 and > scale/offset. if fletcher32: > raise ValueError('fletcher32 cannot be used with > potentially lossy' ' scale/offset filter') > =20 > external =3D _normalize_external(external) > # End argument validation > =20 > if (chunks is True) or \ > (chunks is None and any((shuffle, fletcher32, compression, > maxshape, scaleoffset is not None))): > chunks =3D guess_chunk(shape, maxshape, dtype.itemsize) > =20 > if maxshape is True: > maxshape =3D (None,)*len(shape) > =20 > if chunks is not None: > plist.set_chunk(chunks) > plist.set_fill_time(h5d.FILL_TIME_ALLOC) # prevent > resize glitch=20 > # scale-offset must come before shuffle and compression > if scaleoffset is not None: > if dtype.kind in ('u', 'i'): > plist.set_scaleoffset(h5z.SO_INT, scaleoffset) > else: # dtype.kind =3D=3D 'f' > plist.set_scaleoffset(h5z.SO_FLOAT_DSCALE, > scaleoffset)=20 > for item in external: > plist.set_external(*item) > =20 > if shuffle: > plist.set_shuffle() > =20 > if compression =3D=3D 'gzip': > plist.set_deflate(gzip_level) > elif compression =3D=3D 'lzf': > plist.set_filter(h5z.FILTER_LZF, h5z.FLAG_OPTIONAL) > elif compression =3D=3D 'szip': > opts =3D {'ec': h5z.SZIP_EC_OPTION_MASK, 'nn': > h5z.SZIP_NN_OPTION_MASK} plist.set_szip(opts[szmethod], szpix) > elif isinstance(compression, int): > if not allow_unknown_filter and not > h5z.filter_avail(compression): > > raise ValueError("Unknown compression filter number: > > %s" % compression) =20 > E ValueError: Unknown compression filter number: 32008 >=20 > /usr/lib/python3/dist-packages/h5py/_debian_h5py_serial/_hl/filters.py:28= 1: > ValueError >=20 --=20 Neil Williams =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D https://linux.codehelp.co.uk/ --Sig_/4_Lx9QRqGH8virppBAwRV_v Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEf3HB6ceOc10DYMbM8WfkPIFDtoIFAmMQn4AACgkQ8WfkPIFD toKCpxAAv3QWRu0Qcpj/7TsVQ8h7+dUacyAjR1jf/y/FT35O0lrmbGpjewgFOqRF FSSJTOcKEjFO9YuYVmIuSJxxK6ax7Z0DIl+Y6IM7P9dw35Ys9cVuBFSi9VhWpwzS GO5VKj/3A27/bD0H4yyzmc0oh86bD2Wcu1Z49rARf5v2lvJpUMGki7ck3o8wto+Q 3eHhGM0Pj+CePN3BZdI6ersCMQosgnHL22tAtHvAkC45uW77iN2T6Se3ZC1O0OOK M4bpFDJnv51M1sfflR511SpsrDFkwB0ozxLIgP59YBPvmMo76aOxI8/u78om2MDu JEcdX1wVjX0rVmSavDe165MSS/KVUZJMhpAcd7Jwk6Rx5Hv/C5fFtjYmZpNL6/A2 ER8J6VhqFruK90/2DFr8niqTFH9eJGP7j8zXwzRZ7gN/KqaW5jYmBlv+KkkWnmcr WscuWv47ASS+FS6dNAiBZVM0RL9HOmZG0bxH2eN4pnMlFzZE4/cbwfqwI/EWa5Lk +PCcONF/aL7n8VGl63oEW7WdRkhfcvDu9wxg+C0W3dq9ZJhhdcO0V2ncIJnE6fs5 YViid6wRVuTZSuEsJlnNsGHY5lB9z2NXvuMMp4sJ7Sx8QCOyWKQhOGMDvmcRGyLa JjyKpaMS+dVwvrR9jb9HP2gmGoR03+XMuiJiDm8hTVufPPZBmF8= =gvkT -----END PGP SIGNATURE----- --Sig_/4_Lx9QRqGH8virppBAwRV_v--