Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #13756 > unrolled thread

Bug#1000803: autopkgtest: avoid duplication with autopkgtests and how unittests are run at build-time

Back to article view | Back to linux.debian.maint.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1000803: autopkgtest: avoid duplication with autopkgtests and how unittests are run at build-time Antonio Terceiro <terceiro@debian.org> - 2021-12-02 00:30 +0100
    Bug#1000803: autopkgtest: avoid duplication with autopkgtests and how unittests are run at build-time Antonio Terceiro <terceiro@debian.org> - 2021-12-20 20:10 +0100
      Bug#1000803: dh-python: add pybuild-autopkgtest, a test runner Antonio Terceiro <terceiro@debian.org> - 2021-12-25 14:30 +0100

#13756 — Bug#1000803: autopkgtest: avoid duplication with autopkgtests and how unittests are run at build-time

[Multipart message — attachments visible in raw view] — view raw

Hi,

Adding debian-python@l.d.o

The context is #1000803 where Sandro asked about reducing duplication
when running upstream test suites both during the build and under
autopkgtest.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000803

On Tue, Nov 30, 2021 at 12:47:42AM -0500, Sandro Tosi wrote:
> > This is usually solved outside of autopkgtest itself.
> >
> > For example Ruby packages have a single place where tests are specified
> > (debian/ruby-tests.*{rb,rake,yaml}, and the appropriate
> > debian/tests/control file is autogenerated by autodep8. I would say 99%
> > of Ruby packages require no duplication, or even explicitly specifying
> > commands to run tests at all.
> >
> > In general, this is most efficiently solved by package type (e.g.
> > programming language), because the way the tests are run at build-time
> > usually is tied to the build system. You then usually need some test
> > runner, probably extracted from, or provided by, the build system, to
> > also provide the same funcionality in an autopkgtest-compatible way.
> >
> > All the package types that are well supported in autodep8 have their
> > specialized test runner tools that avoid this type of duplication.
> 
> I'm mostly interested in the python ecosystem, and the provided
> autodep8 test are extremely superficial (essentially only importing
> the main python module packaged), which is better than nothing but not
> the same level as what's in d/rules.
> 
> Are you aware of any effort to expand the Python autodep8 tests to
> uniform them into a comprehensive, and unique place to run tests at
> build-time and via autopkgtest?

I'm not.

> If no such effort currently exists, what would one have to do to start
> developing it? Not necessarily volunteering, just gathering
> information.

In general terms, I see this being implemented like this:

0) modify pybuild so one can call e.g. `pybuild --autopkgtest`. That
should work almost the same as `pybuild --test`, but would copy the test
files under ${AUTOPKGTEST_TMP} and run the tests from there, instead of
assuming a previously built tree and trying to chdir into
.pybuild/*/build.

1) add a way of being able to specify pybuild parameters outside of
debian/rules that can be used both during the build and under
autopkgtest

   For Ruby, gem2deb-test-runner is driven by debian/ruby-tests.*, both
   during the build, and under autopkgtest.

   In pybuild, some aspects of the test run can already be done this
   way, e.g. debian/pybuild.testfiles. Maybe we could have
   debian/pybuild.env to read options in the same way as they are set in
   debian/rules today (with environment variables).

2) update autodep8 to generate the appropriate control file to call
   `pybuild --autopkgtest`. This needs to be "smart" enough to not
   automatically add this call to packages that are not ready for it. At
   the moment, almost 1000 source packages have
   `Testsuite: autopkgtest-pkg-python`. We would probably need a new
   value, or (probably better) to additionally check for something else
   in the source tree.

Each item has quite some details to be figured out, but this is the
general idea.

If someone is willing to work on this I could help with guidance along
the way.

[toc] | [next] | [standalone]

#13787

[Multipart message — attachments visible in raw view] — view raw

On Wed, Dec 01, 2021 at 08:19:20PM -0300, Antonio Terceiro wrote:
> Hi,
> 
> Adding debian-python@l.d.o
> 
> The context is #1000803 where Sandro asked about reducing duplication
> when running upstream test suites both during the build and under
> autopkgtest.
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000803
> 
> On Tue, Nov 30, 2021 at 12:47:42AM -0500, Sandro Tosi wrote:
> > > This is usually solved outside of autopkgtest itself.
> > >
> > > For example Ruby packages have a single place where tests are specified
> > > (debian/ruby-tests.*{rb,rake,yaml}, and the appropriate
> > > debian/tests/control file is autogenerated by autodep8. I would say 99%
> > > of Ruby packages require no duplication, or even explicitly specifying
> > > commands to run tests at all.
> > >
> > > In general, this is most efficiently solved by package type (e.g.
> > > programming language), because the way the tests are run at build-time
> > > usually is tied to the build system. You then usually need some test
> > > runner, probably extracted from, or provided by, the build system, to
> > > also provide the same funcionality in an autopkgtest-compatible way.
> > >
> > > All the package types that are well supported in autodep8 have their
> > > specialized test runner tools that avoid this type of duplication.
> > 
> > I'm mostly interested in the python ecosystem, and the provided
> > autodep8 test are extremely superficial (essentially only importing
> > the main python module packaged), which is better than nothing but not
> > the same level as what's in d/rules.
> > 
> > Are you aware of any effort to expand the Python autodep8 tests to
> > uniform them into a comprehensive, and unique place to run tests at
> > build-time and via autopkgtest?
> 
> I'm not.
> 
> > If no such effort currently exists, what would one have to do to start
> > developing it? Not necessarily volunteering, just gathering
> > information.
> 
> In general terms, I see this being implemented like this:
> 
> 0) modify pybuild so one can call e.g. `pybuild --autopkgtest`. That
> should work almost the same as `pybuild --test`, but would copy the test
> files under ${AUTOPKGTEST_TMP} and run the tests from there, instead of
> assuming a previously built tree and trying to chdir into
> .pybuild/*/build.
> 
> 1) add a way of being able to specify pybuild parameters outside of
> debian/rules that can be used both during the build and under
> autopkgtest
> 
>    For Ruby, gem2deb-test-runner is driven by debian/ruby-tests.*, both
>    during the build, and under autopkgtest.
> 
>    In pybuild, some aspects of the test run can already be done this
>    way, e.g. debian/pybuild.testfiles. Maybe we could have
>    debian/pybuild.env to read options in the same way as they are set in
>    debian/rules today (with environment variables).
> 
> 2) update autodep8 to generate the appropriate control file to call
>    `pybuild --autopkgtest`. This needs to be "smart" enough to not
>    automatically add this call to packages that are not ready for it. At
>    the moment, almost 1000 source packages have
>    `Testsuite: autopkgtest-pkg-python`. We would probably need a new
>    value, or (probably better) to additionally check for something else
>    in the source tree.
> 
> Each item has quite some details to be figured out, but this is the
> general idea.

I have written a prototype implementation of this, and published it as
https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/27

[toc] | [prev] | [next] | [standalone]

#13811 — Bug#1000803: dh-python: add pybuild-autopkgtest, a test runner

[Multipart message — attachments visible in raw view] — view raw

Control: reassign -1 dh-python
Control: retitle -1 dh-python: add pybuild-autopkgtest, a test runner
Control: severity -1 wishlist
Control: tag -1 + patch
Control: forwarded -1 https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/27

On Mon, Dec 20, 2021 at 04:03:17PM -0300, Antonio Terceiro wrote:
> On Wed, Dec 01, 2021 at 08:19:20PM -0300, Antonio Terceiro wrote:
> > Hi,
> > 
> > Adding debian-python@l.d.o
> > 
> > The context is #1000803 where Sandro asked about reducing duplication
> > when running upstream test suites both during the build and under
> > autopkgtest.
> > 
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000803
> > 
> > On Tue, Nov 30, 2021 at 12:47:42AM -0500, Sandro Tosi wrote:
> > > > This is usually solved outside of autopkgtest itself.
> > > >
> > > > For example Ruby packages have a single place where tests are specified
> > > > (debian/ruby-tests.*{rb,rake,yaml}, and the appropriate
> > > > debian/tests/control file is autogenerated by autodep8. I would say 99%
> > > > of Ruby packages require no duplication, or even explicitly specifying
> > > > commands to run tests at all.
> > > >
> > > > In general, this is most efficiently solved by package type (e.g.
> > > > programming language), because the way the tests are run at build-time
> > > > usually is tied to the build system. You then usually need some test
> > > > runner, probably extracted from, or provided by, the build system, to
> > > > also provide the same funcionality in an autopkgtest-compatible way.
> > > >
> > > > All the package types that are well supported in autodep8 have their
> > > > specialized test runner tools that avoid this type of duplication.
> > > 
> > > I'm mostly interested in the python ecosystem, and the provided
> > > autodep8 test are extremely superficial (essentially only importing
> > > the main python module packaged), which is better than nothing but not
> > > the same level as what's in d/rules.
> > > 
> > > Are you aware of any effort to expand the Python autodep8 tests to
> > > uniform them into a comprehensive, and unique place to run tests at
> > > build-time and via autopkgtest?
> > 
> > I'm not.
> > 
> > > If no such effort currently exists, what would one have to do to start
> > > developing it? Not necessarily volunteering, just gathering
> > > information.
> > 
> > In general terms, I see this being implemented like this:
> > 
> > 0) modify pybuild so one can call e.g. `pybuild --autopkgtest`. That
> > should work almost the same as `pybuild --test`, but would copy the test
> > files under ${AUTOPKGTEST_TMP} and run the tests from there, instead of
> > assuming a previously built tree and trying to chdir into
> > .pybuild/*/build.
> > 
> > 1) add a way of being able to specify pybuild parameters outside of
> > debian/rules that can be used both during the build and under
> > autopkgtest
> > 
> >    For Ruby, gem2deb-test-runner is driven by debian/ruby-tests.*, both
> >    during the build, and under autopkgtest.
> > 
> >    In pybuild, some aspects of the test run can already be done this
> >    way, e.g. debian/pybuild.testfiles. Maybe we could have
> >    debian/pybuild.env to read options in the same way as they are set in
> >    debian/rules today (with environment variables).
> > 
> > 2) update autodep8 to generate the appropriate control file to call
> >    `pybuild --autopkgtest`. This needs to be "smart" enough to not
> >    automatically add this call to packages that are not ready for it. At
> >    the moment, almost 1000 source packages have
> >    `Testsuite: autopkgtest-pkg-python`. We would probably need a new
> >    value, or (probably better) to additionally check for something else
> >    in the source tree.
> > 
> > Each item has quite some details to be figured out, but this is the
> > general idea.
> 
> I have written a prototype implementation of this, and published it as
> https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/27

This is now ready to be seriously considered, please take a look.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web