Path: csiph.com!newsfeed.xs4all.nl!newsfeed8.news.xs4all.nl!bofh.it!news.nic.it!robomod From: Antonio Terceiro Newsgroups: linux.debian.bugs.dist,linux.debian.maint.python Subject: Bug#1000803: autopkgtest: avoid duplication with autopkgtests and how unittests are run at build-time Date: Thu, 02 Dec 2021 00:30:03 +0100 Message-ID: References: X-Mailbox-Line: From debian-bugs-dist-request@lists.debian.org Wed Dec 1 23:27:09 2021 Old-Return-Path: X-Spam-Flag: NO X-Spam-Score: -4.2 Reply-To: Antonio Terceiro , 1000803@bugs.debian.org Resent-To: debian-bugs-dist@lists.debian.org Resent-Cc: Debian CI team X-Debian-Pr-Message: followup 1000803 X-Debian-Pr-Package: autopkgtest X-Debian-Pr-Source: autopkgtest MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="5xkTkK06NWRzUbbm" Content-Disposition: inline X-Debian-Message: from BTS X-Mailing-List: archive/latest/1693629 List-ID: List-URL: Approved: robomod@news.nic.it Lines: 104 Organization: linux.* mail to news gateway Sender: robomod@news.nic.it X-Original-Cc: 1000803@bugs.debian.org, debian-python@lists.debian.org X-Original-Date: Wed, 1 Dec 2021 20:19:20 -0300 X-Original-Message-ID: X-Original-References: <163816099309.1882314.457076046272823426.reportbug@zion.matrix.int> <163816099309.1882314.457076046272823426.reportbug@zion.matrix.int> Xref: csiph.com linux.debian.bugs.dist:1083870 linux.debian.maint.python:13756 --5xkTkK06NWRzUbbm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, Adding debian-python@l.d.o The context is #1000803 where Sandro asked about reducing duplication when running upstream test suites both during the build and under autopkgtest. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D1000803 On Tue, Nov 30, 2021 at 12:47:42AM -0500, Sandro Tosi wrote: > > This is usually solved outside of autopkgtest itself. > > > > For example Ruby packages have a single place where tests are specified > > (debian/ruby-tests.*{rb,rake,yaml}, and the appropriate > > debian/tests/control file is autogenerated by autodep8. I would say 99% > > of Ruby packages require no duplication, or even explicitly specifying > > commands to run tests at all. > > > > In general, this is most efficiently solved by package type (e.g. > > programming language), because the way the tests are run at build-time > > usually is tied to the build system. You then usually need some test > > runner, probably extracted from, or provided by, the build system, to > > also provide the same funcionality in an autopkgtest-compatible way. > > > > All the package types that are well supported in autodep8 have their > > specialized test runner tools that avoid this type of duplication. >=20 > I'm mostly interested in the python ecosystem, and the provided > autodep8 test are extremely superficial (essentially only importing > the main python module packaged), which is better than nothing but not > the same level as what's in d/rules. >=20 > Are you aware of any effort to expand the Python autodep8 tests to > uniform them into a comprehensive, and unique place to run tests at > build-time and via autopkgtest? I'm not. > If no such effort currently exists, what would one have to do to start > developing it? Not necessarily volunteering, just gathering > information. In general terms, I see this being implemented like this: 0) modify pybuild so one can call e.g. `pybuild --autopkgtest`. That should work almost the same as `pybuild --test`, but would copy the test files under ${AUTOPKGTEST_TMP} and run the tests from there, instead of assuming a previously built tree and trying to chdir into =2Epybuild/*/build. 1) add a way of being able to specify pybuild parameters outside of debian/rules that can be used both during the build and under autopkgtest For Ruby, gem2deb-test-runner is driven by debian/ruby-tests.*, both during the build, and under autopkgtest. In pybuild, some aspects of the test run can already be done this way, e.g. debian/pybuild.testfiles. Maybe we could have debian/pybuild.env to read options in the same way as they are set in debian/rules today (with environment variables). 2) update autodep8 to generate the appropriate control file to call `pybuild --autopkgtest`. This needs to be "smart" enough to not automatically add this call to packages that are not ready for it. At the moment, almost 1000 source packages have `Testsuite: autopkgtest-pkg-python`. We would probably need a new value, or (probably better) to additionally check for something else in the source tree. Each item has quite some details to be figured out, but this is the general idea. If someone is willing to work on this I could help with guidance along the way. --5xkTkK06NWRzUbbm Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEst7mYDbECCn80PEM/A2xu81GC94FAmGoAvMACgkQ/A2xu81G C97vzA//avVa50a83IveAuzWsdOfMflZIzR8nKKqFgqvizPyii4RaLXnGycVCeRq T20cg2T5v9pfV1jcpijffej3IWDXHcf78EbR3nWEm8yGONsysU9YxhqEaijmU9gM NxEyoMcJnIS3x6eEgpHwZQmlnWcTpgnbNW4Q/2d5FlVHZzHvuoUB185ZC+HZDfQB HehI18Ne7qi9M9Z82bWBP3pfqI8P8ULcV0VfMbhisyeGsUi3no8YCvk01j0v8c5o jDtyK8LPeVZL9obQxONEhe0soquA3HE2hPAB4weVYeiyDhglh1cTZyOBZvjuPAud 9e16OjOu8rx4K0bP81DjP2R9cIxOREZgguXGTT/xm9ojpTuDVtGjNoSD/gWAur0l cpAM5ZYep6hv7tlB8czAzSmH4IRLQ5zNcxwuvSH8WfvlsU10gj4EyiIcev6Srsao tfzN0DkzbFihl/q0KzibMR3cM/S16SDZ4rN4QPJAAMVo0BRw8V4ZR5jB1Hrwmpza XR86F7g4XoWF7Cm8O7W1noMq18qIOhAUZslLpMqR9TDKJqWk8f/6HKVqtymbKOrt +LVMMCjnweQGXwH4uG+gLqrdrxDV7qed1j2o1PUwxIgBWmuWhSykCafxPYBA3otD W2xKZNu2yUfdV5R483KKUhcY5JxqXI7p6wIbG9wYUprx2Rw/OeI= =qO7E -----END PGP SIGNATURE----- --5xkTkK06NWRzUbbm--