Groups > linux.debian.maint.python > #16259

Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

Path	csiph.com!fu-berlin.de!bofh.it!news.nic.it!robomod
From	Scott Kitterman <debian@kitterman.com>
Newsgroups	linux.debian.maint.python
Subject	Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"
Date	Sun, 08 Sep 2024 19:30:01 +0200
Message-ID	<JktTz-aTUa-23@gated-at.bofh.it> (permalink)
References	<FwzIZ-6MUo-9@gated-at.bofh.it> <FwzIZ-6MUo-9@gated-at.bofh.it> <JkqCl-aRPf-9@gated-at.bofh.it>
X-Original-To	debian-python@lists.debian.org
X-Mailbox-Line	From debian-python-request@lists.debian.org Sun Sep 8 17:22:17 2024
Old-Return-Path	<debian@kitterman.com>
X-Amavis-Spam-Status	No, score=-9.51 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, LDO_WHITELIST=-5, RCVD_IN_DNSWL_MED=-2.3, T_SCC_BODY_TEXT_LINE=-0.01] autolearn=ham autolearn_force=no
X-Policyd-Weight	NOT_IN_SBL_XBL_SPAMHAUS=-1.5 CL_IP_EQ_HELO_IP=-2 (check from: .kitterman. - helo: .interserver.kitterman. - helo-domain: .kitterman.) FROM/MX_MATCHES_HELO(DOMAIN)=-2; rate: -5.5
MIME-Version	1.0
Content-Type	text/plain; charset=utf-8
Content-Transfer-Encoding	quoted-printable
X-Mailing-List	<debian-python@lists.debian.org> archive/latest/22319
List-ID	<debian-python.lists.debian.org>
List-URL	<https://lists.debian.org/debian-python/>
List-Archive	https://lists.debian.org/msgid-search/FB98764F-17A6-442B-B25C-3F92E53870D0@kitterman.com
Approved	robomod@news.nic.it
Lines	60
Organization	linux.* mail to news gateway
Sender	robomod@news.nic.it
X-Original-Date	Sun, 08 Sep 2024 17:21:10 +0000
X-Original-Message-ID	<FB98764F-17A6-442B-B25C-3F92E53870D0@kitterman.com>
X-Original-References	<1d836a0a-b926-aaa1-c04c-5a9d6910347a@debian.org> <1d836a0a-b926-aaa1-c04c-5a9d6910347a@debian.org> <6b7zvq5bjmmfp3nogosqrgejexhne6mjyzfsih7iyrlaixa7hv@clcppesyi4m7>
Xref	csiph.com linux.debian.maint.python:16259

Show key headers only | View raw


On September 8, 2024 1:43:33 PM UTC, Stefano Rivera <stefanor@debian.org> wrote:
>Hi Louis-Philippe (2022.11.28_01:46:58_+0200)
>> Too often, a mistake or a misconfiguration leads to no tests being detected
>> when trying to run the upstream testsuite.
>> 
>> When this happens, the result of the test command typically looks like "Ran
>> 0 tests in 0.000s".
>
>A couple of years later:
>
>We've implemented the feature in Python 3.12, unittest's runner now
>exits with return value 5 if no tests were discovered, like pytest does.
>https://github.com/python/cpython/issues/113661
>
>We initially ignored this exit status in dh-python, to allow us to
>transition to Python 3.12, without too much pain.
>
>I've just done a rebuild test without ignoring it to see how much effort
>it would take to be able to use this feature:
>
>I built 6440 packages build-depending on dh-python in one way or
>another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.
>
>To fix these build failures, package maintainers would have these
>options:
>
>1. Get the build to run some unit tests (assuming they exist),
>2. override_dh_auto_test with something noop,
>3. export PYBUILD_DISABLE=test,
>4. We could make this failure opt-in in dh-python. Maybe via an explicit
>   --test-unittest option that selects the unittest runner. If you don't
>   explicitly select this runner, you'd get an attempt to run tests by
>   with unittest, and no failure if no tests are found.
>
>The downside of options 2 and 3 is that if the package does start
>gaining an upstream test suite, nothing ever will attempt to run it. I
>think that's OK?
>
>If you want to experiment yourself, there's a version of dh-python in
>experimental that will treat no tests as an error.
>
>Should we file bugs for these packages? Or implement option 4?
>
>Attached is a current dd-list.
>
Thanks.  I think some variation of #4 is the right answer.  Causing a thousand packages to FTBFS isn't great.  I would find it easier to have an environment variable (similar to what you suggested for #3) instead of adding an override, but that's more of an implementation detail.

Scott K

Thread

Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Stefano Rivera <stefanor@debian.org> - 2024-09-08 16:00 +0200
  Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s" Scott Kitterman <debian@kitterman.com> - 2024-09-08 19:30 +0200
    Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Louis-Philippe Véronneau <pollo@debian.org> - 2024-09-12 17:40 +0200
  Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Julian Gilbey <julian@d-and-j.net> - 2024-09-08 23:40 +0200
    Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Stefano Rivera <stefanor@debian.org> - 2024-09-09 10:40 +0200
      Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Julian Gilbey <julian@d-and-j.net> - 2024-09-09 17:30 +0200
        Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Stefano Rivera <stefanor@debian.org> - 2024-09-10 17:10 +0200
          Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Alexandre Detiste <alexandre.detiste@gmail.com> - 2024-09-12 16:30 +0200
            Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Stefano Rivera <stefanor@debian.org> - 2024-09-12 16:30 +0200
    Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Stefano Rivera <stefanor@debian.org> - 2024-09-09 10:40 +0200
  Re: Bug#1024971: pybuild: should fail when the result of running  tests is "Ran 0 tests in 0.000s" Timo Röhling <roehling@debian.org> - 2024-09-13 11:00 +0200

csiph-web