Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #14352

Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF)

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

On 2022-07-27 16 h 32, Julian Gilbey wrote:
> On Wed, Jul 27, 2022 at 07:45:12PM +0100, Julian Gilbey wrote:
>> On Wed, Jul 27, 2022 at 10:26:33AM -0400, Louis-Philippe Véronneau wrote:
>>> The way I see it:
>>>
>>> 1. We should have a Lintian tag for packages not using the new
>>> pybuild-autodep8 autopkgtest. It would be even better if this tag would be a
>>> pointed hint that identified 'manually' written unit test autopkgtests that
>>> could be replaced.
>>>
>>> This way, you get something like:
>>>
>>> python-foo source: not-using-pybuil-autodep8 [debian/tests/unittests]
>>>
>>> for python packages that have old 'manually' written unit test autopkgtests
>>> and:
>>>
>>> python-foo source: not-using-pybuild-autodep8 [no-autopkgtest]
>>>
>>> for python packages without any autopkgtest.
>>>
>>> 2. lintian-brush (or something else, but I think lintian-brush is the right
>>> tool) would go over these packages to:
>>>
>>> 2.1 Add the new autodep8 autopkgtests and build the package to see if they
>>> pass
>>> 2.2 Remove the "manual" unit test autopkgtests if 2.1 succeeds
>>> 2.3 Open a bug report if 2.1 fails
>>
>> I'd be wary about 2.2 and 2.3.  I have several packages where I know
>> that an automated test will fail; there are all sorts of weird cases
>> where I've had to write tests manually.  I would also be quite cross
>> if manually crafted tests were automatically removed, especially in
>> cases such as Simon mentioned where they do things that that
>> automatically generated test does not do.  Another thing I could
>> imagine happening is that the automated test succeeds in a trivial way
>> - it succeeds but doesn't actually test much because of the nature of
>> the package.
>>
>> On the other hand, a bug report saying something like the following
>> seems much more reasonable: "We've tested this package using the
>> automated autopkgtest system and it seems to work by adding the line
>> 'Testsuite: autopkgtest-pkg-pybuild'; please check that the automated
>> tests cover all of the tests of your manually written debian/tests/*
>> and if so, then please remove them.  The autopkgtest-pkg-pybuild logs
>> are attached."  This would give the maintainer the chance to decide
>> how best to proceed.
> 
> Here's another alternative to steps 2.1-2.3 based on this:
> 
>   For packages which currently have manually-written autopkgtests:
> 
>   2.A Try removing debian/tests and adding Testsuite:
>       autopkgtest-pkg-pybuild to debian/control, then building the
>       package and running autopkgtest.
> 
>   2.B If this works, then submit a bug report to the BTS as I suggested
>       above.
> 
>   2.C If this does not work, don't do anything more; trust that the
>       maintainer knew what they were doing when they wrote the manual
>       autopkgtests.
> 
>   For packages which don't currently have manually-written
>   autopkgtests:
> 
>   2.A' Try adding Testsuite: autopkgtest-pkg-pybuild to debian/control,
>        then building the package and running autopkgtest.
> 
>   2.B' If this works, then either Janitor adds this line to
>        debian/control or submit a bug report to the BTS to recommend
>        this.  (But we would not expect Janitor to do step 2.1', so this
>        might have to be a different setup, or maybe the script doing
>        2.A' could leave a list of packages for Janitor, or something
>        like that.)
> 
>   2.C' If this does not work, submit a wishlist bug to the BTS to
>        recommend that the maintainer adds some autopkgtest tests,
>        either by making the autodep8 system work or by writing some
>        manual tests.
> 
> I'd be wary about adding lintian tags for this, though: with so many
> packages not being able to use the autodep8 system (I vaguely recall
> someone suggesting that a third of Python packages would not be able
> to use the system), that's a lot of packages getting false positives.
> In particular, for the suggested first version of
> not-using-pybuild-autodep8 tag (which would probably be better named
> manual-autopkgtest-could-be-pybuild-autodep8), how would lintian go
> about identifying which packages fall into category 2.B and which into
> 2.C?  The second version of the tag (better named something like
> no-autopkgtest-could-use-pybuild-autodep8, but that's still not very
> good) is less problematic.

This all makes a lot of sense to me and I think this is the way forward, 
once the MR is merged. Thanks for the discussion.

-- 
   ⢀⣴⠾⠻⢶⣦⠀
   ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
   ⢿⡄⠘⠷⠚⠋   pollo@debian.org / veronneau.org
   ⠈⠳⣄

Back to linux.debian.maint.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Notes from the DC22 Python Team BoF Louis-Philippe Véronneau <pollo@debian.org> - 2022-07-23 20:00 +0200
  Re: Notes from the DC22 Python Team BoF Julian Gilbey <julian@d-and-j.net> - 2022-07-25 09:40 +0200
    Re: Notes from the DC22 Python Team BoF Thomas Goirand <zigo@debian.org> - 2022-07-27 10:00 +0200
      Re: Notes from the DC22 Python Team BoF Louis-Philippe Véronneau <pollo@debian.org> - 2022-07-27 16:20 +0200
  pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Julian Gilbey <julian@d-and-j.net> - 2022-07-25 10:10 +0200
    Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Antonio Terceiro <terceiro@debian.org> - 2022-07-26 17:00 +0200
      Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Scott Kitterman <debian@kitterman.com> - 2022-07-26 17:50 +0200
        Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Antonio Terceiro <terceiro@debian.org> - 2022-07-26 23:00 +0200
          Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Paul Wise <pabs@debian.org> - 2022-07-27 02:20 +0200
            Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Louis-Philippe Véronneau <pollo@debian.org> - 2022-07-27 05:50 +0200
      Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Julian Gilbey <julian@d-and-j.net> - 2022-07-27 10:20 +0200
        Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Louis-Philippe Véronneau <pollo@debian.org> - 2022-07-27 16:30 +0200
          Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Julian Gilbey <julian@d-and-j.net> - 2022-07-27 20:50 +0200
            Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Julian Gilbey <julian@d-and-j.net> - 2022-07-27 22:40 +0200
              Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Julian Gilbey <julian@d-and-j.net> - 2022-07-28 20:50 +0200
              Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Louis-Philippe Véronneau <pollo@debian.org> - 2022-08-18 02:00 +0200
        Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF) Simon McVittie <smcv@debian.org> - 2022-07-27 18:50 +0200

csiph-web