Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #13047 > unrolled thread

How to run unit tests?

Back to article view | Back to linux.debian.maint.python

Contents

  How to run unit tests? Peter Wienemann <fossdev@posteo.de> - 2020-12-08 21:40 +0100
    Re: How to run unit tests? Andrey Rahmatullin <wrar@debian.org> - 2020-12-08 21:50 +0100
      Re: How to run unit tests? Peter Wienemann <fossdev@posteo.de> - 2020-12-12 16:40 +0100

#13047 — How to run unit tests?

Dear Python experts,

in trying to update the python-lark package [0] to the most recent 
upstream version, an interesting issue regarding unit tests showed up 
[1]. Depending on how one runs unit tests they either fail or not. Tried 
options are:

1. PYTHONWARNINGS=d pythonX -m unittest discover -v tests
2. pythonX setup.py test
3. pythonX -m tests

Upstream uses option 3 in its CI testing [2] and did not see any issues 
with lark release 0.11.1. dh_auto_test seems to use option 2 by default 
(for the pybuild build system). For autopkgtest I have chosen to use 
option 1. The latter two options both fail for version 0.11.1. After 
applying a patch provided by upstream [3], also option 2 works but 
option 1 still fails. Upstream suggested to change the way the 
(autopkgtest) tests are run [4].

I am reluctant to do this without a better understanding of the 
differences between the above three options. Can anybody shed some light 
on the differences between them? Is there something like a general 
recommendation on what is the best way to run Python unit tests?

Peter

[0] https://salsa.debian.org/python-team/packages/python-lark
[1] https://github.com/lark-parser/lark/issues/792
[2] 
https://github.com/lark-parser/lark/blob/0.11.1/.github/workflows/tests.yml#L28
[3] https://github.com/lark-parser/lark/pull/782
[4] https://github.com/lark-parser/lark/issues/792#issuecomment-739504664

[toc] | [next] | [standalone]

#13048

[Multipart message — attachments visible in raw view] — view raw

On Tue, Dec 08, 2020 at 09:33:03PM +0100, Peter Wienemann wrote:
> Dear Python experts,
> 
> in trying to update the python-lark package [0] to the most recent upstream
> version, an interesting issue regarding unit tests showed up [1]. Depending
> on how one runs unit tests they either fail or not. Tried options are:
> 
> 1. PYTHONWARNINGS=d pythonX -m unittest discover -v tests
> 2. pythonX setup.py test
> 3. pythonX -m tests
Option 3 runs
https://github.com/lark-parser/lark/blob/master/tests/__main__.py which
mostly just runs unittest.main().
Option 2 seems to run the same file according to setup.py.
I'm not familiar with unittests enough know the difference between options
1 and 3.
You didn't explain what actual problems do you have, but
https://github.com/lark-parser/lark/issues/792 suggests that some way you
used was skipping some of the tests?

> Upstream uses option 3 in its CI testing [2] and did not see any issues with
> lark release 0.11.1. dh_auto_test seems to use option 2 by default (for the
> pybuild build system). For autopkgtest I have chosen to use option 1. The
> latter two options both fail for version 0.11.1. 
Which is good?

> After applying a patch provided by upstream [3], also option 2 works but option 1 still fails.
Which is good or bad? I'm not sure.

> Upstream suggested to change the way the (autopkgtest) tests are run [4].
So are you asking only about autopkgtests, not build-time tests?

> Is there something like a general recommendation on what is the best way
> to run Python unit tests?
No (except "whatever the upstream CI runs").

-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#13057

Hi Andrey,

thanks for sharing your thoughts.

On 08.12.20 21:43, Andrey Rahmatullin wrote:
> You didn't explain what actual problems do you have, but
> https://github.com/lark-parser/lark/issues/792 suggests that some way you
> used was skipping some of the tests?

I am not familiar enough with Python unit testing for a solid judgement 
but I find it disturbing that the outcome of the test suite depends on 
the way you run it. I wonder whether this is

a) due to a problem in the test suite,
b) due to a problem somewhere in the tested code,
c) by construction because the three different ways to run the test 
suite are not equivalent?

>> Upstream uses option 3 in its CI testing [2] and did not see any issues with
>> lark release 0.11.1. dh_auto_test seems to use option 2 by default (for the
>> pybuild build system). For autopkgtest I have chosen to use option 1. The
>> latter two options both fail for version 0.11.1.
> Which is good?
> 
>> After applying a patch provided by upstream [3], also option 2 works but option 1 still fails.
> Which is good or bad? I'm not sure.

I am asking myself the same questions. That is why I am asking for 
advice here. :-)

>> Upstream suggested to change the way the (autopkgtest) tests are run [4].
> So are you asking only about autopkgtests, not build-time tests?

Rereading it I think one should not constrain the question to 
autopkgtests but also consider build-time tests.

>> Is there something like a general recommendation on what is the best way
>> to run Python unit tests?
> No (except "whatever the upstream CI runs").

This would vote for option 3:

pythonX -m tests

Best regards

Peter

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web