Groups > linux.debian.maint.python > #13057

Re: How to run unit tests?

Path	csiph.com!weretis.net!feeder8.news.weretis.net!news.mixmin.net!aioe.org!bofh.it!news.nic.it!robomod
From	Peter Wienemann <fossdev@posteo.de>
Newsgroups	linux.debian.maint.python
Subject	Re: How to run unit tests?
Date	Sat, 12 Dec 2020 16:40:01 +0100
Message-ID	<Blfa1-3iQ-3@gated-at.bofh.it> (permalink)
References	<BjRWa-1Dl-11@gated-at.bofh.it> <BjS5Q-1GU-21@gated-at.bofh.it>
X-Mailbox-Line	From debian-python-request@lists.debian.org Sat Dec 12 15:35:59 2020
Old-Return-Path	<fossdev@posteo.de>
X-Amavis-Spam-Status	No, score=-9.479 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, LDO_WHITELIST=-5, MURPHY_DRUGS_REL8=0.02, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001] autolearn=ham autolearn_force=no
X-Policyd-Weight	NOT_IN_SBL_XBL_SPAMHAUS=-1.5 CL_IP_EQ_HELO_IP=-2 (check from: .posteo. - helo: .mout01.posteo. - helo-domain: .posteo.) FROM/MX_MATCHES_HELO(DOMAIN)=-2; rate: -5.5
Dkim-Signature	v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.de; s=2017; t=1607787342; bh=r8RoPQRMagTz9R0ItWpz4X9Tr7/Ft5pY3M4363bFQdQ=; h=Subject:To:From:Date:From; b=r7GAu8qT5k/72AbIKPCFdZ3k32K39KqiRRkJcqvaX/FkRu7PIEoNjSXqhDFw8afz1 y6WkB9EwVWVKqpFzWNoCRq1CFooGG4SHHK5WR24mqETSib6nVPVjuy1ruq+WeJBT6q eMDdsk/Rfd8m8wwP6qm31fNpP8XKkOfc7GrYbwcx1m7NzsU03+1sGgJoqotyrg5/uj 1mklhgTiEFp9PDLv3FlM7C4EDKYhlRKchFiBwk9nhyvbDhxfS6ojVsxpMDpUll+s5a 7VS5LNZqlk9P+Iml1aGYBunWBTYn6Ge0p2bGSSZwRJ1IhN9p/subEcYImUeq/CSEOx mFs2vjFNTuhLQ==
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.5.1
MIME-Version	1.0
Content-Type	text/plain; charset=utf-8; format=flowed
Content-Language	en-US
Content-Transfer-Encoding	7bit
X-Mailing-List	<debian-python@lists.debian.org> archive/latest/18830
List-ID	<debian-python.lists.debian.org>
List-URL	<https://lists.debian.org/debian-python/>
List-Archive	https://lists.debian.org/msgid-search/4cd114ec-1ddd-cee3-8f4b-e70a7dc82dad@posteo.de
Approved	robomod@news.nic.it
Lines	47
Organization	linux.* mail to news gateway
Sender	robomod@news.nic.it
X-Original-Date	Sat, 12 Dec 2020 16:35:41 +0100
X-Original-Message-ID	<4cd114ec-1ddd-cee3-8f4b-e70a7dc82dad@posteo.de>
X-Original-References	<ca4e7355-7290-16a7-805f-66e952f99d9d@posteo.de> <X8/lW0aLakA+U7z3@belkar.wrar.name>
Xref	csiph.com linux.debian.maint.python:13057

Show key headers only | View raw

Hi Andrey,

thanks for sharing your thoughts.

On 08.12.20 21:43, Andrey Rahmatullin wrote:
> You didn't explain what actual problems do you have, but
> https://github.com/lark-parser/lark/issues/792 suggests that some way you
> used was skipping some of the tests?

I am not familiar enough with Python unit testing for a solid judgement 
but I find it disturbing that the outcome of the test suite depends on 
the way you run it. I wonder whether this is

a) due to a problem in the test suite,
b) due to a problem somewhere in the tested code,
c) by construction because the three different ways to run the test 
suite are not equivalent?

>> Upstream uses option 3 in its CI testing [2] and did not see any issues with
>> lark release 0.11.1. dh_auto_test seems to use option 2 by default (for the
>> pybuild build system). For autopkgtest I have chosen to use option 1. The
>> latter two options both fail for version 0.11.1.
> Which is good?
> 
>> After applying a patch provided by upstream [3], also option 2 works but option 1 still fails.
> Which is good or bad? I'm not sure.

I am asking myself the same questions. That is why I am asking for 
advice here. :-)

>> Upstream suggested to change the way the (autopkgtest) tests are run [4].
> So are you asking only about autopkgtests, not build-time tests?

Rereading it I think one should not constrain the question to 
autopkgtests but also consider build-time tests.

>> Is there something like a general recommendation on what is the best way
>> to run Python unit tests?
> No (except "whatever the upstream CI runs").

This would vote for option 3:

pythonX -m tests

Best regards

Peter

Back to linux.debian.maint.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

How to run unit tests? Peter Wienemann <fossdev@posteo.de> - 2020-12-08 21:40 +0100
  Re: How to run unit tests? Andrey Rahmatullin <wrar@debian.org> - 2020-12-08 21:50 +0100
    Re: How to run unit tests? Peter Wienemann <fossdev@posteo.de> - 2020-12-12 16:40 +0100

csiph-web