Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #16435 > unrolled thread

Correct command to skip multiple tests in autopkgtest?

Back to article view | Back to linux.debian.maint.python

Contents

  Correct command to skip multiple tests in autopkgtest? Soren Stoutner <soren@debian.org> - 2024-11-02 00:10 +0100
    Re: Correct command to skip multiple tests in autopkgtest? Soren Stoutner <soren@debian.org> - 2024-11-06 22:30 +0100
      Re: Correct command to skip multiple tests in autopkgtest? Julian Gilbey <julian@d-and-j.net> - 2024-11-11 21:20 +0100
        Re: Correct command to skip multiple tests in autopkgtest? Soren Stoutner <soren@debian.org> - 2024-11-12 19:10 +0100
    Re: Correct command to skip multiple tests in autopkgtest? Soren Stoutner <soren@debian.org> - 2024-12-03 01:30 +0100

#16435 — Correct command to skip multiple tests in autopkgtest?

[Multipart message — attachments visible in raw view] — view raw

I am in the process of packaging pyinstaller.

https://pypi.org/project/pyinstaller/

Many of pyinstaller’s tests require the pyinstaller-hooks-contrib package, 
which I am also in the process of packaging.

https://pypi.org/project/pyinstaller-hooks-contrib/

But pyinstaller-hooks-contrib depends on pyinstaller for its tests.

In order to bootstrap the introduction of these packages into Debian, I am 
temporarily disabling the pyinstaller tests that depend on pyinstaller-hooks-
contrib using the following command in debian/rules:

export PYBUILD_TEST_ARGS=-k\
	'not test_find_module\
	and not test_metadata_searching\
	and not test_Qt_QTranslate\
	and not test_zeep\
	and not test_scapy3\
	and not test_zope_interface\
	and not test_gevent\
	and not test_user_preferred_locale\
	and not test_pytz'

This works successfully during the build process:

1004 passed, 286 skipped, 25 deselected, 24 xfailed, 1 xpassed, 17 warnings in 
6969.97s (1:56:09)

https://salsa.debian.org/soren/python-pyinstaller/-/jobs/6514811#L6635

But, for some reason, autopkgtest doesn’t like the format of the arguments.

I: pybuild base:311: cd /tmp/autopkgtest-lxc.ntf_swjm/downtmp/autopkgtest_tmp/
build; python3.12 -m pytest -k 'not test_find_module and not 
test_metadata_searching and not test_Qt_QTranslate and not test_zeep and not 
test_scapy3 and not test_zope_interface and not test_gevent and not 
test_user_preferred_locale and not test_pytz'
usage: __main__.py [-h] [--link-only] {samedir,subdir,parentdir}
__main__.py: error: argument test_type: invalid choice: 'not test_find_module 
and not test_metadata_searching and not test_Qt_QTranslate and not test_zeep 
and not test_scapy3 and not test_zope_interface and not test_gevent and not 
test_user_preferred_locale and not test_pytz' (choose from 'samedir', 
'subdir', ‘parentdir’)

https://salsa.debian.org/soren/python-pyinstaller/-/jobs/6514819#L781

I have tried several syntax variations without success.  Does anyone know the 
correct syntax for skipping several tests in autopgktest?

-- 
Soren Stoutner
soren@debian.org

[toc] | [next] | [standalone]

#16438

[Multipart message — attachments visible in raw view] — view raw

On Friday, November 1, 2024 4:03:45 PM MST Soren Stoutner wrote:
> I: pybuild base:311: cd /tmp/autopkgtest-lxc.ntf_swjm/downtmp/
autopkgtest_tmp/
> build; python3.12 -m pytest -k 'not test_find_module and not
> test_metadata_searching and not test_Qt_QTranslate and not test_zeep and not
> test_scapy3 and not test_zope_interface and not test_gevent and not
> test_user_preferred_locale and not test_pytz'
> usage: __main__.py [-h] [--link-only] {samedir,subdir,parentdir}
> __main__.py: error: argument test_type: invalid choice: 'not test_find_module
> and not test_metadata_searching and not test_Qt_QTranslate and not test_zeep
> and not test_scapy3 and not test_zope_interface and not test_gevent and not
> test_user_preferred_locale and not test_pytz' (choose from 'samedir',
> 'subdir', ‘parentdir’)

My experience with another package is that -k works correctly with autopkgtest 
when skipping only one test.  The problem appears to be something related to 
parsing multiple tests to skip.  I have tried a couple of syntax variations, 
but so far haven’t been able to guess the right one.

-- 
Soren Stoutner
soren@debian.org

[toc] | [prev] | [next] | [standalone]

#16444

On Wed, Nov 06, 2024 at 02:29:34PM -0700, Soren Stoutner wrote:
> On Friday, November 1, 2024 4:03:45 PM MST Soren Stoutner wrote:
> > I: pybuild base:311: cd /tmp/autopkgtest-lxc.ntf_swjm/downtmp/
> autopkgtest_tmp/
> > build; python3.12 -m pytest -k 'not test_find_module and not
> > test_metadata_searching and not test_Qt_QTranslate and not test_zeep and not
> > test_scapy3 and not test_zope_interface and not test_gevent and not
> > test_user_preferred_locale and not test_pytz'
> > usage: __main__.py [-h] [--link-only] {samedir,subdir,parentdir}
> > __main__.py: error: argument test_type: invalid choice: 'not test_find_module
> > and not test_metadata_searching and not test_Qt_QTranslate and not test_zeep
> > and not test_scapy3 and not test_zope_interface and not test_gevent and not
> > test_user_preferred_locale and not test_pytz' (choose from 'samedir',
> > 'subdir', ‘parentdir’)
> 
> My experience with another package is that -k works correctly with autopkgtest 
> when skipping only one test.  The problem appears to be something related to 
> parsing multiple tests to skip.  I have tried a couple of syntax variations, 
> but so far haven’t been able to guess the right one.

You could have a look at what I've done in pydevd (soon to be removed
from the archive), which was a complex case.  But I'd suggest using
the syntax

--deselect=tests/test_file.py::test_testname

instead of -k; it can be given multiple times, and as long as there
are no spaces or weird characters in the test names (for example,
tests/test_file.py::test_testname[this is a parameter]), it works fine
when stored as a shell variable and then expanded without quotes.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#16447

[Multipart message — attachments visible in raw view] — view raw

On Monday, November 11, 2024 1:18:12 PM MST Julian Gilbey wrote:
> You could have a look at what I've done in pydevd (soon to be removed
> from the archive), which was a complex case.  But I'd suggest using
> the syntax
> 
> --deselect=tests/test_file.py::test_testname
> 
> instead of -k; it can be given multiple times, and as long as there
> are no spaces or weird characters in the test names (for example,
> tests/test_file.py::test_testname[this is a parameter]), it works fine
> when stored as a shell variable and then expanded without quotes.

Thanks for the suggestion, but I wasn’t able to get --deselect to work, either 
during the build or in autopkgtest.  I also have been unsuccessful in getting 
--ignore or -k to work with autopkgtest, even though they both work during the 
build.

My sense is that it is some bug related to the implementation of "Testsuite: 
autopkgtest-pkg-pybuild”, but I don’t know enough about it to troubleshoot it.

-- 
Soren Stoutner
soren@debian.org

[toc] | [prev] | [next] | [standalone]

#16516

[Multipart message — attachments visible in raw view] — view raw

On Friday, November 1, 2024 4:03:45 PM MST Soren Stoutner wrote:
> I am in the process of packaging pyinstaller.
> 
> https://pypi.org/project/pyinstaller/
> 
> Many of pyinstaller’s tests require the pyinstaller-hooks-contrib package,
> which I am also in the process of packaging.
> 
> https://pypi.org/project/pyinstaller-hooks-contrib/
> 
> But pyinstaller-hooks-contrib depends on pyinstaller for its tests.
> 
> In order to bootstrap the introduction of these packages into Debian, I am
> temporarily disabling the pyinstaller tests that depend on pyinstaller-
hooks-
> contrib using the following command in debian/rules:
> 
> export PYBUILD_TEST_ARGS=-k\
> 	'not test_find_module\
> 	and not test_metadata_searching\
> 	and not test_Qt_QTranslate\
> 	and not test_zeep\
> 	and not test_scapy3\
> 	and not test_zope_interface\
> 	and not test_gevent\
> 	and not test_user_preferred_locale\
> 	and not test_pytz'
> 
> This works successfully during the build process:
> 
> 1004 passed, 286 skipped, 25 deselected, 24 xfailed, 1 xpassed, 17 warnings 
in
> 6969.97s (1:56:09)
> 
> https://salsa.debian.org/soren/python-pyinstaller/-/jobs/6514811#L6635
> 
> But, for some reason, autopkgtest doesn’t like the format of the arguments.
> 
> I: pybuild base:311: cd /tmp/autopkgtest-lxc.ntf_swjm/downtmp/
autopkgtest_tmp/
> build; python3.12 -m pytest -k 'not test_find_module and not
> test_metadata_searching and not test_Qt_QTranslate and not test_zeep and not
> test_scapy3 and not test_zope_interface and not test_gevent and not
> test_user_preferred_locale and not test_pytz'
> usage: __main__.py [-h] [--link-only] {samedir,subdir,parentdir}
> __main__.py: error: argument test_type: invalid choice: 'not test_find_module
> and not test_metadata_searching and not test_Qt_QTranslate and not test_zeep
> and not test_scapy3 and not test_zope_interface and not test_gevent and not
> test_user_preferred_locale and not test_pytz' (choose from 'samedir',
> 'subdir', ‘parentdir’)
> 
> https://salsa.debian.org/soren/python-pyinstaller/-/jobs/6514819#L781
> 
> I have tried several syntax variations without success.  Does anyone know 
the
> correct syntax for skipping several tests in autopgktest?

In the case of PyInstaller, the answer ended up being to not use `Testsuite: 
autopkgtest-pkg-pybuild` and instead call the tests manually using debian/
tests/control.

https://salsa.debian.org/soren/python-pyinstaller/-/blob/main/debian/tests/
control?ref_type=heads

Although there are some further errors to investigate, this at least allowed 
the tests to run.

https://salsa.debian.org/soren/python-pyinstaller/-/pipelines/772011

I am uncertain why `Testsuite: autopkgtest-pkg-pybuild` was failing, but it 
may relate in some way to the fact that PyInstaller has so many tests that 
they are all organized into subdirectories.  Perhaps not having any tests 
directly in `tests` was an issue.  At least, that is one possible meaning of 
the `choose from 'samedir', 'subdir', ‘parentdir’)` in the error message.

https://github.com/pyinstaller/pyinstaller/tree/develop/tests

-- 
Soren Stoutner
soren@debian.org

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web