Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1131007

Bug#1027466: python-cmaes: flaky autopkgtest: Unreliable test timings

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

Source: python-cmaes
Version: 0.9.0-1
Severity: serious
User: debian-ci@lists.debian.org
Usertags: flaky

Dear maintainer(s),

I looked at the results of the autopkgtest of your package. I noticed 
that it regularly fails on some architectures, particularly on s390x, 
but not exclusively there. Yesterday I triggered 5 or 6 reference runs 
on every architecture and 6/6 failed on s390x and 1/5 failed on ppc64el. 
The error message suggests the solution is to turn deadlines off. Can 
you consider doing that? The same error happened at least once also on 
amd64, so it's not only the "obscure" release architectures.

Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are causing people unrelated to your package to spend time on these
tests.

Don't hesitate to reach out if you need help and some more information
from our infrastructure.

Paul

https://ci.debian.net/data/autopkgtest/testing/ppc64el/p/python-cmaes/29815406/log.gz

Draw 5: [(array([-2.81633907e+14, -1.17549435e-38, -1.17549435e-38]), 
0.0), (array([4.39569613e+009, 2.22507386e-313, 4.39569613e+009]), 
67207168.0), (array([-2.81500763e+14, -2.81500763e+14, 
-2.81500763e+14]), 1537.0), (array([1.70317369e+15, 1.70317369e+15, 
5.62915594e+14]), 486606715.0), (array([-2.25294795e+15, 
2.00001000e+00, -2.25294795e+15]), -4295688511.0), 
(array([-1.75639510e+07,  0.00000000e+00,  2.89334784e+14]), 
117020729737728.0), (array([ 2.561e+03,  2.561e+03, -1.000e-05]), 1.9)]
Draw 6: [(array([4.50843053e+13, 1.09954551e+12, 1.09954551e+12]), 
-4.294967296334234e+16), (array([ 2.85877335e+14, -6.59750009e+13, 
-6.59750009e+13]), -1.2061593328768367e-165), (array([-1.1467774e+12, 
-1.1467774e+12, -1.1467774e+12]), 0.0), (array([-4.22236517e+14, 
2.00001000e+00,  0.00000000e+00]), 496085902557184.0), 
(array([5.62915594e+14, 5.62915594e+14, 5.62915594e+14]), 
288235272011776.0), (array([2.82578968e+14, 2.82578968e+14, 
2.82578968e+14]), -1.7976931348623157e+308), (array([ 1.67774790e+07, 
2.05828577e+15, -2.22044605e-16]), -1103823438096.0)]
Draw 7: [(array([-6.60146835e+12, -1.53700000e+03,  0.00000000e+00]), 
-167839291.0), (array([ 4.31259682e+009, -1.11253693e-308, 
-1.11253693e-308]), 0.0), (array([2.60000000e+02, 3.40282347e+38, 
3.40282347e+38]), 1.1), (array([-8.55704363e+08, -8.55704363e+08, 
2.00001000e+00]), 1.5), (array([-1.33575193e+12, -1.33575193e+12, 
-1.33575193e+12]), -5e-324), (array([66305., 66305., 66305.]), 0.0), 
(array([-9.00719925e+15,             nan,  1.13818803e+12]), 0.0)]
Unreliable test timings! On an initial run, this test took 540.20ms, 
which exceeded the deadline of 200.00ms, but on a subsequent run it took 
5.07 ms, which did not. If you expect this sort of variability in your 
test timings, consider turning deadlines off for this test by setting 
deadline=None.


https://ci.debian.net/data/autopkgtest/testing/s390x/p/python-cmaes/29815257/log.gz

        0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), 0.0), (array([0., 0., 
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), 0.0), (array([0., 0., 
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), 0.0)]
Unreliable test timings! On an initial run, this test took 386.19ms, 
which exceeded the deadline of 200.00ms, but on a subsequent run it took 
27.02 ms, which did not. If you expect this sort of variability in your 
test timings, consider turning deadlines off for this test by setting 
deadline=None.


https://ci.debian.net/data/autopkgtest/testing/amd64/p/python-cmaes/29718110/log.gz

======================================================================
ERROR: test_cma_tell (tests.test_fuzzing.TestFuzzing)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/tmp/autopkgtest-lxc.d1aldrvi/downtmp/autopkgtest_tmp/tests/test_fuzzing.py", 
line 10, in test_cma_tell
     data=st.data(),
   File "/usr/lib/python3/dist-packages/hypothesis/core.py", line 1190, 
in wrapped_test
     raise the_error_hypothesis_found
   File "/usr/lib/python3/dist-packages/hypothesis/core.py", line 586, 
in test
     raise DeadlineExceeded(runtime, self.settings.deadline)
hypothesis.errors.DeadlineExceeded: Test took 206.80ms, which exceeds 
the deadline of 200.00ms

Back to linux.debian.bugs.dist | Previous | Next | Find similar | Unroll thread

Thread

Bug#1027466: python-cmaes: flaky autopkgtest: Unreliable test timings Paul Gevers <elbrus@debian.org> - 2023-01-01 09:10 +0100

csiph-web