Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #16114 > unrolled thread

Policy Change Proposal: Running the upstream test suite

Back to article view | Back to linux.debian.maint.python

Contents

  Policy Change Proposal: Running the upstream test suite Louis-Philippe Véronneau <pollo@debian.org> - 2024-07-29 11:00 +0200
    Re: Policy Change Proposal: Running the upstream test suite PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2024-07-29 11:30 +0200
      Re: Policy Change Proposal: Running the upstream test suite Martin <debacle@debian.org> - 2024-07-29 14:00 +0200
        Re: Policy Change Proposal: Running the upstream test suite PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2024-07-29 15:50 +0200
          Re: Policy Change Proposal: Running the upstream test suite Andrey Rakhmatullin <wrar@debian.org> - 2024-07-29 16:00 +0200
            Re: Policy Change Proposal: Running the upstream test suite PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2024-07-29 17:40 +0200
              Re: Policy Change Proposal: Running the upstream test suite Scott Kitterman <debian@kitterman.com> - 2024-07-29 18:10 +0200
              Re: Policy Change Proposal: Running the upstream test suite Andrey Rakhmatullin <wrar@debian.org> - 2024-07-30 10:40 +0200
                Re: Policy Change Proposal: Running the upstream test suite PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2024-07-30 11:10 +0200
                  Re: Policy Change Proposal: Running the upstream test suite Andrey Rakhmatullin <wrar@debian.org> - 2024-07-31 16:20 +0200
                Re: Policy Change Proposal: Running the upstream test suite Julian Gilbey <julian@d-and-j.net> - 2024-07-30 17:00 +0200
                  Re: Policy Change Proposal: Running the upstream test suite Andrey Rakhmatullin <wrar@debian.org> - 2024-07-31 16:10 +0200
            Re: Policy Change Proposal: Running the upstream test suite Simon McVittie <smcv@debian.org> - 2024-07-31 10:50 +0200
              Re: Policy Change Proposal: Running the upstream test suite Andrey Rakhmatullin <wrar@debian.org> - 2024-07-31 16:10 +0200
                Re: Policy Change Proposal: Running the upstream test suite Simon McVittie <smcv@debian.org> - 2024-07-31 16:20 +0200
    Re: Policy Change Proposal: Running the upstream test suite Scott Kitterman <debian@kitterman.com> - 2024-07-29 14:10 +0200
      Re: Policy Change Proposal: Running the upstream test suite Simon McVittie <smcv@debian.org> - 2024-07-29 16:40 +0200
      Re: Policy Change Proposal: Running the upstream test suite Julian Gilbey <julian@d-and-j.net> - 2024-07-29 23:50 +0200
      Re: Policy Change Proposal: Running the upstream test suite Louis-Philippe Véronneau <pollo@debian.org> - 2024-07-30 03:00 +0200
        Re: Policy Change Proposal: Running the upstream test suite Scott Kitterman <debian@kitterman.com> - 2024-07-30 04:20 +0200
          Re: Policy Change Proposal: Running the upstream test suite Louis-Philippe Véronneau <pollo@debian.org> - 2024-07-30 04:50 +0200
            Re: Policy Change Proposal: Running the upstream test suite Scott Kitterman <debian@kitterman.com> - 2024-07-30 06:00 +0200
              Re: Policy Change Proposal: Running the upstream test suite Louis-Philippe Véronneau <pollo@debian.org> - 2024-07-30 06:30 +0200
                Re: Policy Change Proposal: Running the upstream test suite Scott Kitterman <debian@kitterman.com> - 2024-07-30 06:50 +0200
                  Re: Policy Change Proposal: Running the upstream test suite thomas@goirand.fr - 2024-07-30 09:20 +0200
    Re: Policy Change Proposal: Running the upstream test suite Pierre-Elliott Bécue <peb@debian.org> - 2024-07-29 16:10 +0200
      Re: Policy Change Proposal: Running the upstream test suite Louis-Philippe Véronneau <pollo@debian.org> - 2024-07-30 03:00 +0200
    Re: Policy Change Proposal: Running the upstream test suite Louis-Philippe Véronneau <pollo@debian.org> - 2024-08-15 00:10 +0200
      Re: Policy Change Proposal: Running the upstream test suite Andreas Tille <andreas@an3as.eu> - 2024-08-16 17:00 +0200
      Re: Policy Change Proposal: Running the upstream test suite Pierre-Elliott Bécue <peb@debian.org> - 2024-08-17 00:50 +0200
        Re: Policy Change Proposal: Running the upstream test suite Louis-Philippe Véronneau <pollo@debian.org> - 2024-08-18 23:20 +0200
          Re: Policy Change Proposal: Running the upstream test suite Graham Inggs <ginggs@debian.org> - 2024-08-19 15:40 +0200

Page 1 of 2  [1] 2  Next page →

#16114 — Policy Change Proposal: Running the upstream test suite

Hello,

As discussed during the DebConf24 Python BoF, I'm submitting this change 
to the policy to require the use of the upstream test suite, both during 
the build process and as an autopkgtest.

You can find the MR here:

https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24

People present at the BoF seemed to think this was a good idea. If you 
don't please reply to this message and make yourself heard :)

Cheers,

-- 
   ⢀⣴⠾⠻⢶⣦⠀
   ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
   ⢿⡄⠘⠷⠚⠋   pollo@debian.org / veronneau.org
   ⠈⠳⣄

[toc] | [next] | [standalone]

#16115

I have one concerne when using pybuild autopkgtest.

It install by default the build dependencies. which is not what peoples are doing when they install the packages.
It should be possible to define the autopkgtest dependencies. This way we could catch missing dependencies in python-<modules> dependencies.

Is it possible ?


Cheers

Frederic

[toc] | [prev] | [next] | [standalone]

#16117

Quoting PICCA Frederic-Emmanuel  
<frederic-emmanuel.picca@synchrotron-soleil.fr>:
> It install by default the build dependencies. which is not what  
> peoples are doing when they install the packages.
> It should be possible to define the autopkgtest dependencies. This  
> way we could catch missing dependencies in python-<modules>  
> dependencies.
>
> Is it possible ?

I believe (didn't check so many packages), in most cases
installing the build dependencies is the better option:

1. There might be testing frameworks to install as b-d, but
    not as Depend.

2. Many unit test suites check various variants and optional
    behaviour, that might be separated in different binary
    packages (think of myapp-postgres and myapp-mariadb, etc.)

Maybe as an option for some packages?

Or a different kind of test, running on top of piuparts?

[toc] | [prev] | [next] | [standalone]

#16119

My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.

I think about package whcih provide oiptional dependencies via the extra. In that case the extra should be declare when calling dh_python3.

So I must install only the python3-<module> and the test framework to be sure that it is fonctionnal as installed.

It seems to me that the autopkgtest-pybuild plugin is not well suited for this purpose. (as it is now).

Cheers

Frederic

For piupart, I alsao have some test that should be perfore with one of my package

the upgrade of a database.

I should install the package, upgrade it and check that the new version has the right database upgraded.

Is it possible to test upgrade scenario with autopkgtests ?

[toc] | [prev] | [next] | [standalone]

#16120

[Multipart message — attachments visible in raw view] — view raw

On Mon, Jul 29, 2024 at 03:30:39PM +0200, PICCA Frederic-Emmanuel wrote:
> My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.

Maybe we indeed want a "minimal" autopkgtest environment, but many
upstream tests will fail in those and I don't see an automatic way to test
a random package in this way.

-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#16123

> Maybe we indeed want a "minimal" autopkgtest environment, but many
> upstream tests will fail in those and I don't see an automatic way to test
> a random package in this way.

Even if not minimal, at least correspond to the upstream declares dependencies.

by 'declare' I am not even sure of the meaning.

dependencies of the test can be different from the required installed ones.

Maybe we should  install only the python binaries and the dependencies marked <!nocheck>.

Is there a standard in the Python community for the test dependencies ?

[toc] | [prev] | [next] | [standalone]

#16124

On July 29, 2024 3:14:33 PM UTC, PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> wrote:
>> Maybe we indeed want a "minimal" autopkgtest environment, but many
>> upstream tests will fail in those and I don't see an automatic way to test
>> a random package in this way.
>
>Even if not minimal, at least correspond to the upstream declares dependencies.
>
>by 'declare' I am not even sure of the meaning.
>
>dependencies of the test can be different from the required installed ones.
>
>Maybe we should  install only the python binaries and the dependencies marked <!nocheck>.
>
>Is there a standard in the Python community for the test dependencies ?
>
As usual, there are several.

Scott K

[toc] | [prev] | [next] | [standalone]

#16134

[Multipart message — attachments visible in raw view] — view raw

On Mon, Jul 29, 2024 at 05:14:33PM +0200, PICCA Frederic-Emmanuel wrote:
> > Maybe we indeed want a "minimal" autopkgtest environment, but many
> > upstream tests will fail in those and I don't see an automatic way to test
> > a random package in this way.
> 
> Even if not minimal, at least correspond to the upstream declares dependencies.

Those will correspond to build dependencies in a typical simple case, as
the upstream tests need all those optional deps, assuming you mean the
tests dependencies (and not the runtime dependencies which should already
be in Depends).

> Maybe we should  install only the python binaries and the dependencies marked <!nocheck>.

In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
as you don't need anything beyond the build system to "build" a pure
Python module.

So this is exactly what you wanted to avoid.

-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#16135

> Those will correspond to build dependencies in a typical simple case, as
> the upstream tests need all those optional deps, assuming you mean the
> tests dependencies (and not the runtime dependencies which should already
> be in Depends).

Yes the runtime dependencies should be listed in the python3-<module> Depends:

> 
>> Maybe we should  install only the python binaries and the dependencies marked
>> <!nocheck>.
> 
> In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
> as you don't need anything beyond the build system to "build" a pure
> Python module.

> So this is exactly what you wanted to avoid.

Yes, I want to be sure that all the runtime dependencies are rightfully declared in the Depends of the python module package.

sometimes the upstream forgot about dependencies, or mark them as optional, but they are not when running the tests...

Is it possible to achieve this automatically ? lintian ? dh-python ? python3-stdeb ?

Fred.

PS: In haskell we have a cabal-debian package which compute automatically the list of the Dependencies, from the cabal file, then it update the control file to reflect the new status of the package. 

[toc] | [prev] | [next] | [standalone]

#16142

[Multipart message — attachments visible in raw view] — view raw

On Tue, Jul 30, 2024 at 10:47:59AM +0200, PICCA Frederic-Emmanuel wrote:
> Yes, I want to be sure that all the runtime dependencies are rightfully declared in the Depends of the python module package.

Depending on what do you mean by "all" this is not always what we want. We
often don't put all modules imported by all parts of the shipped code into
Depends, instead using Recommends, Suggests or nothing.

> sometimes the upstream forgot about dependencies, or mark them as optional, but they are not when running the tests...

Similarly, all modules installed for tests shouldn't always go into
Depends ("all modules installed for tests" is ideally the same as "all
modules imported by all parts of the shipped code", plus test system
deps, and minus deps for e.g. Windows code that we ship but don't run or
test).


> Is it possible to achieve this automatically ? lintian ? dh-python ? python3-stdeb ?

Well our tools can look at the deps declared by the upstream, and normally
do that as a part of the build process. But that won't help when those
deps are not what we want.


-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#16136

On Tue, Jul 30, 2024 at 01:30:57PM +0500, Andrey Rakhmatullin wrote:
> > Maybe we should  install only the python binaries and the dependencies marked <!nocheck>.
> 
> In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
> as you don't need anything beyond the build system to "build" a pure
> Python module.

Erm, no, you need to B-D on all runtime dependencies so they are
correctly included in the computed python3:Depends substvar, unless
you list these runtime dependencies explictly by hand.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#16141

[Multipart message — attachments visible in raw view] — view raw

On Tue, Jul 30, 2024 at 03:59:04PM +0100, Julian Gilbey wrote:
> > > Maybe we should  install only the python binaries and the dependencies marked <!nocheck>.
> > 
> > In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
> > as you don't need anything beyond the build system to "build" a pure
> > Python module.
> 
> Erm, no, you need to B-D on all runtime dependencies so they are
> correctly included in the computed python3:Depends substvar, unless
> you list these runtime dependencies explictly by hand.

Those will be installed anyway via Depends, by definition, but also in my
experience you don't always need to do one of these two things. Maybe I'm
doing it wrong and my python3:Depends are not 100% correct though.

-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#16137

On Mon, 29 Jul 2024 at 18:55:26 +0500, Andrey Rakhmatullin wrote:
> On Mon, Jul 29, 2024 at 03:30:39PM +0200, PICCA Frederic-Emmanuel wrote:
> > My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.
> 
> Maybe we indeed want a "minimal" autopkgtest environment

For a simple smoke-test that the Python packages import succesfully, isn't
this exactly autopkgtest-pkg-python?

For thorough test coverage, I don't think this is possible: in general,
the test suite will legitimately require installation of packages that are
only needed during testing (pytest or similar), or packages that are
optional for basic use of the code but mandatory for full test coverage
(those might be Recommends or Suggests).

> but many
> upstream tests will fail in those and I don't see an automatic way to test
> a random package in this way

If we want to run upstream test-suites as autopkgtests, then I think
ideally we want the same test coverage as autopkgtest-pkg-python,
*and* the same test coverage as autopkgtest-pkg-pybuild, as separate
autopkgtest test-cases with different dependencies.

But I don't think we can easily have that, because the Testsuite field
only takes one value.

Would it be possible to make autopkgtest-pkg-pybuild absorb
the functionality of autopkgtest-pkg-python, so that it
generates a debian/control that does some simple smoke-tests (like
autopkgtest-pkg-python does), *and* runs the upstream test suite (like
autopkgtest-pkg-pybuild currently does)?

If not, then the proposed policy change would result in us gaining
coverage but also losing coverage.

    smcv

[toc] | [prev] | [next] | [standalone]

#16140

[Multipart message — attachments visible in raw view] — view raw

On Wed, Jul 31, 2024 at 09:46:27AM +0100, Simon McVittie wrote:
> > > My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.
> > 
> > Maybe we indeed want a "minimal" autopkgtest environment
> 
> For a simple smoke-test that the Python packages import succesfully, isn't
> this exactly autopkgtest-pkg-python?

Yes.


> For thorough test coverage, I don't think this is possible: in general,
> the test suite will legitimately require installation of packages that are
> only needed during testing (pytest or similar), or packages that are
> optional for basic use of the code but mandatory for full test coverage
> (those might be Recommends or Suggests).

I was thinking about "running a subset of the test suite that doesn't need
those optional deps" (I didn't consider that this needs pytest and likely
various pytest plugins and their deps so it isn't strictly minimal) and
automatically identifying this subset is what I later said I think isn't
possible.

> 
> > but many
> > upstream tests will fail in those and I don't see an automatic way to test
> > a random package in this way
> 
> If we want to run upstream test-suites as autopkgtests, then I think
> ideally we want the same test coverage as autopkgtest-pkg-python,
> *and* the same test coverage as autopkgtest-pkg-pybuild, as separate
> autopkgtest test-cases with different dependencies.

I always thought the smoke test is redundant if there is the upstream
tests one.
What do you think?

> But I don't think we can easily have that, because the Testsuite field
> only takes one value.

It was easy (and achieved by accident) before autopkgtest-pkg-pybuild: you
could have upstream tests in debian/tests and the smoke test via Testsuite
or automatic discovery (I could never remember which combination gives you
both). But, again, I never thought this is desirable and was removing the
smoke test.


-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#16143

On Wed, 31 Jul 2024 at 19:01:05 +0500, Andrey Rakhmatullin wrote:
> On Wed, Jul 31, 2024 at 09:46:27AM +0100, Simon McVittie wrote:
> > If we want to run upstream test-suites as autopkgtests, then I think
> > ideally we want the same test coverage as autopkgtest-pkg-python,
> > *and* the same test coverage as autopkgtest-pkg-pybuild, as separate
> > autopkgtest test-cases with different dependencies.
> 
> I always thought the smoke test is redundant if there is the upstream
> tests one.
> What do you think?

I think it is not redundant, because it's a quick way to catch packaging
errors where the python3-foo binary package is missing a mandatory
dependency - which seems to be a somewhat common failure mode, because
the maintainer (who presumably has the Build-Depends installed) will
never notice it by testing on their own system.

For instance in this (hypothetical) package:

    Source: python-foo
    Build-Depends: dh-python, python3-all, python3-bar, ...

    Package: python3-foo
    Depends: ${misc:Depends}, ${python3:Depends}

Suppose python3-foo does an "import bar", therefore there should have
been a "Depends: python3-bar" which is missing here. The smoke test
would detect that.

(I know that ideally python3-bar would be part of the ${python3:Depends},
and with some upstream build systems under some phases of the moon,
it is; but I'm thinking here about the packages where that information
doesn't work or isn't propagated.)

I'm a fan of putting a small amount of effort into automated tests, and
then having running the tests before uploading save you from a class of
Severity: grave bugs forever :-)

> > But I don't think we can easily have that, because the Testsuite field
> > only takes one value.

Actually it seems it's meant to be able to take more than one value, and
the fact that autodep8 doesn't support that appears to be considered a bug.
(#1042717, #1061620, #1077645)

    smcv

[toc] | [prev] | [next] | [standalone]

#16118

On July 29, 2024 8:53:11 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
>Hello,
>
>As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
>
>You can find the MR here:
>
>https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
>
>People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)

I understand the theory and why it's a good idea to run the test suite.  I don't think it ought to be a hard requirement.  I have several packages where there's a test suite, but I don't run it:

1.  The largest set is packages that need test only dependencies which are not packaged.  When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged, I generally stop and don't enable tests for things that are only in the archives to support tests.  Noseofyeti is an example of this.

2.  There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems.  If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.

3.  I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.

While I agree that running tests is good, it's not a universally reasonable requirement.

If we add this as a hard requirement for team packages, what do people think will happen?  Will people invest more volunteer time that they otherwise wouldn't have to add running tests?  Will they leave it for someone else on the team to address?  Will they remove packages from team maintenance?

Python policy is only enforceable within the team.  For the rest of the archive, it's a statement of best practices for people to consider.  Making this a hard requirement for team packages is a statement that we would rather such packages not be team maintained.  I don't think that's a good idea.

Scott K

[toc] | [prev] | [next] | [standalone]

#16122

On Mon, 29 Jul 2024 at 12:07:27 +0000, Scott Kitterman wrote:
> While I agree that running tests is good, it's not a universally
> reasonable requirement.

I agree. In a project as large as Debian, most requirements similar to
this one at least need the qualifier "... unless there is a reason why
we can't".

> 2.  There's at least one case where Debian has customizations that
> cause the test suite to fail, but the failures don't seem to cause any
> real problems.  If anyone wants to make it so the weasyprint test suite
> works on Debian, please knock yourself out.

This seems like the situation is:

- a build-time test or autopkgtest test failure would be a
  release-critical bug

- upstream tests failing always indicate a bug, but that bug might not
  be RC:
    - the tests might be broken, flaky or make assumptions that aren't
      true in Debian (the tests have a bug, the package under test does not)
    - or the tests might be showing us a genuine bug in the package
      under test, but its severity might be anywhere between wishlist
      and critical

- in the case of this specific package, Scott has assessed the severity of
  the bug that is implied by the tests failing, and has decided that it's
  Severity: normal or less

and that seems fine.

Ideally there would be a bug open in the BTS to document this limitation,
possibly only wishlist, possibly tagged help.

If part of the test suite is fine but a different part of the test suite
is problematic, I'd personally prefer to flag the problematic parts as
skipped/ignored, and run the rest (for example src:python3-mpd skips some
of its tests like this); but it's reasonable for opinions on this to vary,
and I don't know whether there would be anything left in this particular
package's test suite after skipping the problematic parts.

> 3.  I also maintain multiple packages which require network access
> to run their test suite, so they can't run tests during build, only
> autopkgtests.

Running the test suite for these would be a Debian Policy violation
(usually a RC bug), and if Python team policy conflicts with Debian Policy,
it's Debian Policy that wins.

> If we add this as a hard requirement for team packages, what do people
> think will happen?  Will people invest more volunteer time that they
> otherwise wouldn't have to add running tests?  Will they leave it for
> someone else on the team to address?  Will they remove packages from
> team maintenance?

These are all valid questions. It's very easy to create new QA
requirements that are simple to achieve for simple packages, but for the
packages where they are not so simple, someone who feels that a particular
level of QA coverage is valuable will need to step up to do that work,
and threatening contributors with sanctions if they don't do particular
work ("we'll throw you out of the team" or "we'll delete the package
you've invested time in from Debian" or similar) should be rare and a
last resort.

We often say that Debian Policy is (and should be) a tool for getting
a better distribution, and not a stick to beat contributors with. I
think the same should be equally true for individual teams' self-imposed
policies.

    smcv

[toc] | [prev] | [next] | [standalone]

#16125

On Mon, Jul 29, 2024 at 12:07:27PM +0000, Scott Kitterman wrote:
> I understand the theory and why it's a good idea to run the test suite.  I don't think it ought to be a hard requirement.  I have several packages where there's a test suite, but I don't run it:
> [...]

Here's another case: the package has to be fully installed for the
test suite to work.  So the tests can't be run at build time, only at
autopkgtest time.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#16127

On 2024-07-29 21:07, Scott Kitterman wrote:
> 
> 
> On July 29, 2024 8:53:11 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
>> Hello,
>>
>> As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
>>
>> You can find the MR here:
>>
>> https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
>>
>> People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)
> 
> I understand the theory and why it's a good idea to run the test suite.  I don't think it ought to be a hard requirement.  I have several packages where there's a test suite, but I don't run it:
> 
> 1.  The largest set is packages that need test only dependencies which are not packaged.  When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged, I generally stop and don't enable tests for things that are only in the archives to support tests.  Noseofyeti is an example of this.

That sounds like a valid technical reason not to run the tests to me :)

> 2.  There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems.  If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.

Again, as long as you document that, I don't think it would go against 
the proposed policy change.

> 3.  I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.

Same.


-- 
   ⢀⣴⠾⠻⢶⣦⠀
   ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
   ⢿⡄⠘⠷⠚⠋   pollo@debian.org / veronneau.org
   ⠈⠳⣄

[toc] | [prev] | [next] | [standalone]

#16128

On July 30, 2024 12:49:50 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
>On 2024-07-29 21:07, Scott Kitterman wrote:
>> 
>> 
>> On July 29, 2024 8:53:11 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
>>> Hello,
>>> 
>>> As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
>>> 
>>> You can find the MR here:
>>> 
>>> https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
>>> 
>>> People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)
>> 
>> I understand the theory and why it's a good idea to run the test suite.  I don't think it ought to be a hard requirement.  I have several packages where there's a test suite, but I don't run it:
>> 
>> 1.  The largest set is packages that need test only dependencies which are not packaged.  When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged, I generally stop and don't enable tests for things that are only in the archives to support tests.  Noseofyeti is an example of this.
>
>That sounds like a valid technical reason not to run the tests to me :)
>
>> 2.  There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems.  If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.
>
>Again, as long as you document that, I don't think it would go against the proposed policy change.
>
>> 3.  I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.
>
>Same.
>

Except for #3, I don't get that from the wording in the MR.  I don't think not worth the trouble is a technical reason.  I think the real rule that's being proposed is that packages must run the test suit or document why not.  I don't have a problem with that, but I don't think that's what it actually says now.  I think if you were to change must to should and strike the word technical before reason, it would accomplish the same thing and be clearer.  I could support that.

Scott K

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.maint.python

csiph-web