Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #16258 > unrolled thread

Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Back to article view | Back to linux.debian.maint.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Stefano Rivera <stefanor@debian.org> - 2024-09-08 16:00 +0200
    Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s" Scott Kitterman <debian@kitterman.com> - 2024-09-08 19:30 +0200
      Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Louis-Philippe Véronneau <pollo@debian.org> - 2024-09-12 17:40 +0200
    Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Julian Gilbey <julian@d-and-j.net> - 2024-09-08 23:40 +0200
      Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Stefano Rivera <stefanor@debian.org> - 2024-09-09 10:40 +0200
        Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Julian Gilbey <julian@d-and-j.net> - 2024-09-09 17:30 +0200
          Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Stefano Rivera <stefanor@debian.org> - 2024-09-10 17:10 +0200
            Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Alexandre Detiste <alexandre.detiste@gmail.com> - 2024-09-12 16:30 +0200
              Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Stefano Rivera <stefanor@debian.org> - 2024-09-12 16:30 +0200
      Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Stefano Rivera <stefanor@debian.org> - 2024-09-09 10:40 +0200
    Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Timo Röhling <roehling@debian.org> - 2024-09-13 11:00 +0200

#16258 — Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

[Multipart message — attachments visible in raw view] — view raw

Hi Louis-Philippe (2022.11.28_01:46:58_+0200)
> Too often, a mistake or a misconfiguration leads to no tests being detected
> when trying to run the upstream testsuite.
> 
> When this happens, the result of the test command typically looks like "Ran
> 0 tests in 0.000s".

A couple of years later:

We've implemented the feature in Python 3.12, unittest's runner now
exits with return value 5 if no tests were discovered, like pytest does.
https://github.com/python/cpython/issues/113661

We initially ignored this exit status in dh-python, to allow us to
transition to Python 3.12, without too much pain.

I've just done a rebuild test without ignoring it to see how much effort
it would take to be able to use this feature:

I built 6440 packages build-depending on dh-python in one way or
another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.

To fix these build failures, package maintainers would have these
options:

1. Get the build to run some unit tests (assuming they exist),
2. override_dh_auto_test with something noop,
3. export PYBUILD_DISABLE=test,
4. We could make this failure opt-in in dh-python. Maybe via an explicit
   --test-unittest option that selects the unittest runner. If you don't
   explicitly select this runner, you'd get an attempt to run tests by
   with unittest, and no failure if no tests are found.

The downside of options 2 and 3 is that if the package does start
gaining an upstream test suite, nothing ever will attempt to run it. I
think that's OK?

If you want to experiment yourself, there's a version of dh-python in
experimental that will treat no tests as an error.

Should we file bugs for these packages? Or implement option 4?

Attached is a current dd-list.

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [next] | [standalone]

#16259 — Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

On September 8, 2024 1:43:33 PM UTC, Stefano Rivera <stefanor@debian.org> wrote:
>Hi Louis-Philippe (2022.11.28_01:46:58_+0200)
>> Too often, a mistake or a misconfiguration leads to no tests being detected
>> when trying to run the upstream testsuite.
>> 
>> When this happens, the result of the test command typically looks like "Ran
>> 0 tests in 0.000s".
>
>A couple of years later:
>
>We've implemented the feature in Python 3.12, unittest's runner now
>exits with return value 5 if no tests were discovered, like pytest does.
>https://github.com/python/cpython/issues/113661
>
>We initially ignored this exit status in dh-python, to allow us to
>transition to Python 3.12, without too much pain.
>
>I've just done a rebuild test without ignoring it to see how much effort
>it would take to be able to use this feature:
>
>I built 6440 packages build-depending on dh-python in one way or
>another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.
>
>To fix these build failures, package maintainers would have these
>options:
>
>1. Get the build to run some unit tests (assuming they exist),
>2. override_dh_auto_test with something noop,
>3. export PYBUILD_DISABLE=test,
>4. We could make this failure opt-in in dh-python. Maybe via an explicit
>   --test-unittest option that selects the unittest runner. If you don't
>   explicitly select this runner, you'd get an attempt to run tests by
>   with unittest, and no failure if no tests are found.
>
>The downside of options 2 and 3 is that if the package does start
>gaining an upstream test suite, nothing ever will attempt to run it. I
>think that's OK?
>
>If you want to experiment yourself, there's a version of dh-python in
>experimental that will treat no tests as an error.
>
>Should we file bugs for these packages? Or implement option 4?
>
>Attached is a current dd-list.
>
Thanks.  I think some variation of #4 is the right answer.  Causing a thousand packages to FTBFS isn't great.  I would find it easier to have an environment variable (similar to what you suggested for #3) instead of adding an override, but that's more of an implementation detail.

Scott K

[toc] | [prev] | [next] | [standalone]

#16276

On 2024-09-08 13:21, Scott Kitterman wrote:
> Thanks.  I think some variation of #4 is the right answer.  Causing a thousand packages to FTBFS isn't great.  I would find it easier to have an environment variable (similar to what you suggested for #3) instead of adding an override, but that's more of an implementation detail.
> 
> Scott K
> 

Having an opt-in environment variable means all the packages using 
unittest will eventually need to add it.

What is the end goal? Once we have enough buy-in we flip it around?

I agree breaking tons of packages isn't great, but I don't want us to 
haul crud forever either :(

-- 
   ⢀⣴⠾⠻⢶⣦⠀
   ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
   ⢿⡄⠘⠷⠚⠋   pollo@debian.org / veronneau.org
   ⠈⠳⣄

[toc] | [prev] | [next] | [standalone]

#16260

On Sun, Sep 08, 2024 at 03:43:33PM +0200, Stefano Rivera wrote:
> [...]
> We've implemented the feature in Python 3.12, unittest's runner now
> exits with return value 5 if no tests were discovered, like pytest does.
> https://github.com/python/cpython/issues/113661
> [...]
> 
> I built 6440 packages build-depending on dh-python in one way or
> another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.

My guess is that most of these 1124 have no tests at all, rather than
having a misconfigured setup.  A unittest is the pybuild default test
framework, unittest is used and fails to find any tests, hence all of
these failures.

> To fix these build failures, package maintainers would have these
> options:
> 
> 1. Get the build to run some unit tests (assuming they exist),
> 2. override_dh_auto_test with something noop,
> 3. export PYBUILD_DISABLE=test,
> 4. We could make this failure opt-in in dh-python. Maybe via an explicit
>    --test-unittest option that selects the unittest runner. If you don't
>    explicitly select this runner, you'd get an attempt to run tests by
>    with unittest, and no failure if no tests are found.

I like option 4 for the above reason.  But implementing this would
mean that all of the packages that currently *do* use unittest
(intentionally, but without having to code it explicitly as it's the
default) would suddenly not have any tests running until they
proactively add --test-unittest or set PYBUILD_TEST_UNITTEST = 1 or
similar.  This seems like an unfortunate consequence.

Is there a way of looking at the logs of the packages that passed the
build to identify which ones successfully passed tests using unittest?
There is also the issue of packages that use unittest (as the default)
via autopkgtest-pkg-pybuild but override dh_auto_test during the
build, though that will be much rarer.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#16261

Re-sent, with the bug in CC

Hi Julian (2024.09.08_21:33:49_+0000)
> > I built 6440 packages build-depending on dh-python in one way or
> > another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.
> 
> My guess is that most of these 1124 have no tests at all, rather than
> having a misconfigured setup.  A unittest is the pybuild default test
> framework, unittest is used and fails to find any tests, hence all of
> these failures.

Yes, almost certainly.

> > 4. We could make this failure opt-in in dh-python. Maybe via an explicit
> >    --test-unittest option that selects the unittest runner. If you don't
> >    explicitly select this runner, you'd get an attempt to run tests by
> >    with unittest, and no failure if no tests are found.
> 
> I like option 4 for the above reason.  But implementing this would
> mean that all of the packages that currently *do* use unittest
> (intentionally, but without having to code it explicitly as it's the
> default) would suddenly not have any tests running until they
> proactively add --test-unittest or set PYBUILD_TEST_UNITTEST = 1 or
> similar.  This seems like an unfortunate consequence.

I would leave unittest as the default runner, but without missing test
detection.

That's a slightly unexpected behaviour, but it makes the default case
work.

Downside is that you have to opt-in to missing test detection. Maybe we
can have a lintian tag for that?

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#16267

On Mon, Sep 09, 2024 at 08:31:52AM +0000, Stefano Rivera wrote:
> > Hi Julian (2024.09.08_21:33:49_+0000)
> > > I built 6440 packages build-depending on dh-python in one way or
> > > another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.
> > 
> > My guess is that most of these 1124 have no tests at all, rather than
> > having a misconfigured setup.  A unittest is the pybuild default test
> > framework, unittest is used and fails to find any tests, hence all of
> > these failures.
> 
> Yes, almost certainly.
> [...]
> 
> I would leave unittest as the default runner, but without missing test
> detection.
> 
> That's a slightly unexpected behaviour, but it makes the default case
> work.
> 
> Downside is that you have to opt-in to missing test detection. Maybe we
> can have a lintian tag for that?

That seems a bit heavy to ask for.

Is there any way of identifying those packages that do genuinely use
unittest?  If there are not that many of them, then implementing a
--test-unittest option would be a good way to go.  I would imagine the
following timeline:

(1) --test-unittest is introduced as an option to explicitly select
unittest as the test framework.  When --test-unittest is specified,
the test will fail if no tests are found.  unittest is still used as a
fallback test framework; in this case, the dh_auto_test call will
succeed if no tests are run.

(2) Add some sort of warning for pybuild-using packages that run
dh_auto_test but haven't specified a test framework and for which
autodetection of the test framework fails.  If there aren't any tests
to run, an empty override_dh_auto_test target should be specified.

(3) Stop using unittest as the default test framework, and fail if no
test framework has been specified or autodetected.

But that might be overkill for something which may not actually be
much of a problem.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#16270

Hi Julian (2024.09.09_15:19:51_+0000)
> That seems a bit heavy to ask for.
> 
> Is there any way of identifying those packages that do genuinely use
> unittest?

From 6438 build logs:
- 651 don't call dh_auto_test
- 2180 do something custom
- 1989 use pytest
- 25 use nose
- 18 use nose2
- 23 use tox
- 3 use stestr
- 1561 packages use pybuild's unittest runner
  * 391 pass
  * 1170 fail
    + 1139 NO TESTS RAN
    + 33 the test suite failed

(numbers don't quite add up, because this was a lot of grep | wc -l)

> If there are not that many of them, then implementing a
> --test-unittest option would be a good way to go.  I would imagine the
> following timeline:
> 
> (1) --test-unittest is introduced as an option to explicitly select
> unittest as the test framework.  When --test-unittest is specified,
> the test will fail if no tests are found.  unittest is still used as a
> fallback test framework; in this case, the dh_auto_test call will
> succeed if no tests are run.
> 
> (2) Add some sort of warning for pybuild-using packages that run
> dh_auto_test but haven't specified a test framework and for which
> autodetection of the test framework fails.  If there aren't any tests
> to run, an empty override_dh_auto_test target should be specified.
> 
> (3) Stop using unittest as the default test framework, and fail if no
> test framework has been specified or autodetected.
> 
> But that might be overkill for something which may not actually be
> much of a problem.

Yeah, that can work. We can also just abort after step 2.

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#16274

Hi,

>From my personal experience, the most common cause of missing tests
is because d/watch follow pypi where the tarball is incomplete
and the typical fix is to switch to github tarball.

Could this "1139 NO TESTS RAN" correlated with d/watch ?

> - 25 use nose

Even less as of today ;-)

Greetings


Le mar. 10 sept. 2024 à 17:06, Stefano Rivera <stefanor@debian.org> a écrit :
>
> Hi Julian (2024.09.09_15:19:51_+0000)
> > That seems a bit heavy to ask for.
> >
> > Is there any way of identifying those packages that do genuinely use
> > unittest?
>
> From 6438 build logs:
> - 651 don't call dh_auto_test
> - 2180 do something custom
> - 1989 use pytest
> - 25 use nose
> - 18 use nose2
> - 23 use tox
> - 3 use stestr
> - 1561 packages use pybuild's unittest runner
>   * 391 pass
>   * 1170 fail
>     + 1139 NO TESTS RAN
>     + 33 the test suite failed
>
> (numbers don't quite add up, because this was a lot of grep | wc -l)

[toc] | [prev] | [next] | [standalone]

#16275

Hi Alexandre (2024.09.12_14:26:13_+0000)
> >From my personal experience, the most common cause of missing tests
> is because d/watch follow pypi where the tarball is incomplete
> and the typical fix is to switch to github tarball.
> 
> Could this "1139 NO TESTS RAN" correlated with d/watch ?

It'll be part of them, yes.

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#16262

Hi Julian (2024.09.08_21:33:49_+0000)
> > I built 6440 packages build-depending on dh-python in one way or
> > another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.
> 
> My guess is that most of these 1124 have no tests at all, rather than
> having a misconfigured setup.  A unittest is the pybuild default test
> framework, unittest is used and fails to find any tests, hence all of
> these failures.

Yes, almost certainly.

> > 4. We could make this failure opt-in in dh-python. Maybe via an explicit
> >    --test-unittest option that selects the unittest runner. If you don't
> >    explicitly select this runner, you'd get an attempt to run tests by
> >    with unittest, and no failure if no tests are found.
> 
> I like option 4 for the above reason.  But implementing this would
> mean that all of the packages that currently *do* use unittest
> (intentionally, but without having to code it explicitly as it's the
> default) would suddenly not have any tests running until they
> proactively add --test-unittest or set PYBUILD_TEST_UNITTEST = 1 or
> similar.  This seems like an unfortunate consequence.

I would leave unittest as the default runner, but without missing test
detection.

That's a slightly unexpected behaviour, but it makes the default case
work.

Downside is that you have to opt-in to missing test detection. Maybe we
can have a lintian tag for that?

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#16278

[Multipart message — attachments visible in raw view] — view raw

Hi Stefano,

* Stefano Rivera <stefanor@debian.org> [2024-09-08 15:43]:
>To fix these build failures, package maintainers would have these
>options:
>
>1. Get the build to run some unit tests (assuming they exist),
>2. override_dh_auto_test with something noop,
>3. export PYBUILD_DISABLE=test,
>4. We could make this failure opt-in in dh-python. Maybe via an explicit
>   --test-unittest option that selects the unittest runner. If you don't
>   explicitly select this runner, you'd get an attempt to run tests by
>   with unittest, and no failure if no tests are found.
Maybe we can start by making missing tests in autopkgtest fail? 
As you need to opt-in via "Testsuite: autopkgtest-pkg-pybuild", we 
can be quite certain that missing tests are not intentional.

I think that would work very nicely together with option 4 for the 
build itself. You might even use the "Testsuite" setting as opt-in 
flag, but that is probably taking it too far.


Cheers
Timo

-- 
⢀⣴⠾⠻⢶⣦⠀   ╭────────────────────────────────────────────────────╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling                                       │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄⠀⠀⠀⠀   ╰────────────────────────────────────────────────────╯

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web