Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #96865 > unrolled thread
| Started by | paul.anton.letnes@gmail.com |
|---|---|
| First post | 2015-09-20 00:52 -0700 |
| Last post | 2015-09-20 10:58 +0200 |
| Articles | 2 — 2 participants |
Back to article view | Back to comp.lang.python
Ordering dependent test failures paul.anton.letnes@gmail.com - 2015-09-20 00:52 -0700
Re: Ordering dependent test failures Peter Otten <__peter__@web.de> - 2015-09-20 10:58 +0200
| From | paul.anton.letnes@gmail.com |
|---|---|
| Date | 2015-09-20 00:52 -0700 |
| Subject | Ordering dependent test failures |
| Message-ID | <2663735f-f237-4ed4-ab41-e9068041358b@googlegroups.com> |
Hi!
Fascinated by the concept of ordering dependent test failures [0], I've run the python test suite [1] with 256 different random seeds (took a little more than 12 hours). The results vary a lot - for instance, the number of tests reported as OK varies, the number of skips varies, etc. Since I'm not sure how to report or interpret them, I'll just post a summary below.
The test suite was run on arch linux [2] with gcc 5.2.0, with the source code taken from a clone of the python repo yesterday [3].
What could I do with all this in order to make more sense of it, and could it be of any help what so ever to python development? I'll gladly make the full log files available to whomever is interested, in whatever format is convenient. In the meantime I'll run more random seeds, because why not.
[0] https://docs.python.org/devguide/buildbots.html#ordering-dependent-failures
[1] ./python -Wd -E -bb -m test -uall -rwW --randseed $k > testlog/${k}.log 2>&1
[2] https://www.archlinux.org/
[3] Python 3.6.0a0 (default:3704cea9fd8e, Sep 19 2015, 16:12:53)
The command "grep OK *.log", where e.g. 5.log is the output of command [1] with "--randseed 5":
0.log:380 tests OK.
0.log:OK (skipped=15)
0.log:OK (skipped=15)
0.log:OK (skipped=15)
0.log:OK (skipped=2)
1.log:383 tests OK.
1.log:OK (skipped=2)
2.log:380 tests OK.
2.log:OK (skipped=15)
2.log:OK (skipped=15)
2.log:OK (skipped=15)
2.log:OK (skipped=2)
3.log:380 tests OK.
3.log:OK (skipped=15)
3.log:OK (skipped=15)
3.log:OK (skipped=2)
3.log:OK (skipped=37)
4.log:381 tests OK.
4.log:OK (skipped=15)
4.log:OK (skipped=15)
4.log:OK (skipped=2)
5.log:380 tests OK.
5.log:OK (skipped=15)
5.log:OK (skipped=15)
5.log:OK (skipped=15)
5.log:OK (skipped=2)
6.log:381 tests OK.
6.log:OK (skipped=15)
6.log:OK (skipped=15)
6.log:OK (skipped=2)
7.log:380 tests OK.
7.log:OK (skipped=15)
7.log:OK (skipped=15)
7.log:OK (skipped=15)
7.log:OK (skipped=2)
8.log:381 tests OK.
8.log:OK (skipped=15)
8.log:OK (skipped=15)
8.log:OK (skipped=2)
9.log:382 tests OK.
9.log:OK (skipped=15)
9.log:OK (skipped=2)
10.log:381 tests OK.
10.log:OK (skipped=15)
10.log:OK (skipped=15)
10.log:OK (skipped=2)
11.log:380 tests OK.
11.log:OK (skipped=15)
11.log:OK (skipped=15)
11.log:OK (skipped=2)
11.log:OK (skipped=37)
12.log:380 tests OK.
12.log:OK (skipped=15)
12.log:OK (skipped=15)
12.log:OK (skipped=15)
12.log:OK (skipped=2)
13.log:383 tests OK.
13.log:OK (skipped=2)
14.log:381 tests OK.
14.log:OK (skipped=15)
14.log:OK (skipped=2)
14.log:OK (skipped=39)
15.log:383 tests OK.
15.log:OK (skipped=2)
16.log:381 tests OK.
16.log:OK (skipped=15)
16.log:OK (skipped=15)
16.log:OK (skipped=2)
17.log:381 tests OK.
17.log:OK (skipped=15)
17.log:OK (skipped=15)
17.log:OK (skipped=2)
18.log:381 tests OK.
18.log:OK (skipped=15)
18.log:OK (skipped=15)
18.log:OK (skipped=2)
19.log:381 tests OK.
19.log:OK (skipped=15)
19.log:OK (skipped=15)
19.log:OK (skipped=2)
20.log:381 tests OK.
20.log:OK (skipped=15)
20.log:OK (skipped=15)
20.log:OK (skipped=2)
21.log:382 tests OK.
21.log:OK (skipped=15)
21.log:OK (skipped=2)
22.log:383 tests OK.
22.log:OK (skipped=2)
23.log:383 tests OK.
23.log:OK (skipped=2)
24.log:382 tests OK.
24.log:OK (skipped=15)
24.log:OK (skipped=2)
25.log:381 tests OK.
25.log:OK (skipped=15)
25.log:OK (skipped=15)
25.log:OK (skipped=2)
26.log:380 tests OK.
26.log:OK (skipped=15)
26.log:OK (skipped=15)
26.log:OK (skipped=15)
26.log:OK (skipped=2)
27.log:382 tests OK.
27.log:OK (skipped=15)
27.log:OK (skipped=2)
28.log:381 tests OK.
28.log:OK (skipped=15)
28.log:OK (skipped=15)
28.log:OK (skipped=2)
29.log:383 tests OK.
29.log:OK (skipped=2)
30.log:380 tests OK.
30.log:OK (skipped=15)
30.log:OK (skipped=15)
30.log:OK (skipped=15)
30.log:OK (skipped=2)
31.log:382 tests OK.
31.log:OK (skipped=15)
31.log:OK (skipped=2)
32.log:382 tests OK.
32.log:OK (skipped=15)
32.log:OK (skipped=2)
33.log:383 tests OK.
33.log:OK (skipped=2)
34.log:383 tests OK.
34.log:OK (skipped=2)
35.log:380 tests OK.
35.log:OK (skipped=15)
35.log:OK (skipped=15)
35.log:OK (skipped=15)
35.log:OK (skipped=2)
36.log:382 tests OK.
36.log:OK (skipped=15)
36.log:OK (skipped=2)
37.log:382 tests OK.
37.log:OK (skipped=15)
37.log:OK (skipped=2)
38.log:380 tests OK.
38.log:OK (skipped=15)
38.log:OK (skipped=15)
38.log:OK (skipped=15)
38.log:OK (skipped=2)
39.log:380 tests OK.
39.log:OK (skipped=15)
39.log:OK (skipped=15)
39.log:OK (skipped=15)
39.log:OK (skipped=2)
40.log:381 tests OK.
40.log:OK (skipped=15)
40.log:OK (skipped=15)
40.log:OK (skipped=2)
41.log:383 tests OK.
41.log:OK (skipped=2)
42.log:380 tests OK.
42.log:OK (skipped=15)
42.log:OK (skipped=15)
42.log:OK (skipped=15)
42.log:OK (skipped=2)
43.log:380 tests OK.
43.log:OK (skipped=15)
43.log:OK (skipped=15)
43.log:OK (skipped=15)
43.log:OK (skipped=2)
44.log:383 tests OK.
44.log:OK (skipped=2)
45.log:383 tests OK.
45.log:OK (skipped=2)
46.log:380 tests OK.
46.log:OK (skipped=15)
46.log:OK (skipped=15)
46.log:OK (skipped=15)
46.log:OK (skipped=2)
47.log:383 tests OK.
47.log:OK (skipped=2)
48.log:381 tests OK.
48.log:OK (skipped=15)
48.log:OK (skipped=15)
48.log:OK (skipped=2)
49.log:381 tests OK.
49.log:OK (skipped=15)
49.log:OK (skipped=15)
49.log:OK (skipped=2)
50.log:383 tests OK.
50.log:OK (skipped=2)
51.log:382 tests OK.
51.log:OK (skipped=15)
51.log:OK (skipped=2)
52.log:381 tests OK.
52.log:OK (skipped=15)
52.log:OK (skipped=15)
52.log:OK (skipped=2)
53.log:383 tests OK.
53.log:OK (skipped=2)
54.log:383 tests OK.
54.log:OK (skipped=2)
55.log:382 tests OK.
55.log:OK (skipped=15)
55.log:OK (skipped=2)
56.log:381 tests OK.
56.log:OK (skipped=15)
56.log:OK (skipped=15)
56.log:OK (skipped=2)
57.log:380 tests OK.
57.log:OK (skipped=15)
57.log:OK (skipped=15)
57.log:OK (skipped=15)
57.log:OK (skipped=2)
58.log:383 tests OK.
58.log:OK (skipped=2)
59.log:380 tests OK.
59.log:OK (skipped=15)
59.log:OK (skipped=15)
59.log:OK (skipped=15)
59.log:OK (skipped=2)
60.log:380 tests OK.
60.log:OK (skipped=15)
60.log:OK (skipped=15)
60.log:OK (skipped=15)
60.log:OK (skipped=2)
61.log:381 tests OK.
61.log:OK (skipped=15)
61.log:OK (skipped=15)
61.log:OK (skipped=2)
62.log:382 tests OK.
62.log:OK (skipped=15)
62.log:OK (skipped=2)
63.log:382 tests OK.
63.log:OK (skipped=15)
63.log:OK (skipped=2)
64.log:381 tests OK.
64.log:OK (skipped=15)
64.log:OK (skipped=15)
64.log:OK (skipped=2)
65.log:379 tests OK.
65.log:OK (skipped=15)
65.log:OK (skipped=15)
65.log:OK (skipped=15)
65.log:OK (skipped=2)
66.log:382 tests OK.
66.log:OK (skipped=15)
66.log:OK (skipped=2)
67.log:382 tests OK.
67.log:OK (skipped=15)
67.log:OK (skipped=2)
68.log:383 tests OK.
68.log:OK (skipped=2)
69.log:383 tests OK.
69.log:OK (skipped=2)
70.log:383 tests OK.
70.log:OK (skipped=2)
71.log:383 tests OK.
71.log:OK (skipped=2)
72.log:382 tests OK.
72.log:OK (skipped=15)
72.log:OK (skipped=2)
73.log:382 tests OK.
73.log:OK (skipped=15)
73.log:OK (skipped=2)
74.log:382 tests OK.
74.log:OK (skipped=2)
75.log:382 tests OK.
75.log:OK (skipped=2)
76.log:382 tests OK.
76.log:OK (skipped=15)
76.log:OK (skipped=2)
77.log:381 tests OK.
77.log:OK (skipped=15)
77.log:OK (skipped=15)
77.log:OK (skipped=2)
78.log:383 tests OK.
78.log:OK (skipped=2)
79.log:380 tests OK.
79.log:OK (skipped=15)
79.log:OK (skipped=15)
79.log:OK (skipped=15)
79.log:OK (skipped=2)
80.log:383 tests OK.
80.log:OK (skipped=2)
81.log:382 tests OK.
81.log:OK (skipped=15)
81.log:OK (skipped=2)
82.log:381 tests OK.
82.log:OK (skipped=15)
82.log:OK (skipped=15)
82.log:OK (skipped=2)
83.log:383 tests OK.
83.log:OK (skipped=2)
84.log:380 tests OK.
84.log:OK (skipped=15)
84.log:OK (skipped=15)
84.log:OK (skipped=15)
84.log:OK (skipped=2)
85.log:382 tests OK.
85.log:OK (skipped=15)
85.log:OK (skipped=2)
86.log:383 tests OK.
86.log:OK (skipped=2)
87.log:382 tests OK.
87.log:OK (skipped=15)
87.log:OK (skipped=2)
88.log:382 tests OK.
88.log:OK (skipped=15)
88.log:OK (skipped=2)
89.log:381 tests OK.
89.log:OK (skipped=15)
89.log:OK (skipped=2)
89.log:OK (skipped=37)
90.log:379 tests OK.
90.log:OK (skipped=15)
90.log:OK (skipped=15)
90.log:OK (skipped=15)
90.log:OK (skipped=2)
91.log:380 tests OK.
91.log:OK (skipped=15)
91.log:OK (skipped=15)
91.log:OK (skipped=15)
91.log:OK (skipped=2)
92.log:381 tests OK.
92.log:OK (skipped=15)
92.log:OK (skipped=15)
92.log:OK (skipped=2)
93.log:382 tests OK.
93.log:OK (skipped=2)
94.log:380 tests OK.
94.log:OK (skipped=15)
94.log:OK (skipped=15)
94.log:OK (skipped=15)
94.log:OK (skipped=2)
95.log:381 tests OK.
95.log:OK (skipped=15)
95.log:OK (skipped=15)
95.log:OK (skipped=2)
96.log:380 tests OK.
96.log:OK (skipped=15)
96.log:OK (skipped=15)
96.log:OK (skipped=15)
96.log:OK (skipped=2)
97.log:381 tests OK.
97.log:OK (skipped=15)
97.log:OK (skipped=15)
97.log:OK (skipped=2)
98.log:380 tests OK.
98.log:OK (skipped=15)
98.log:OK (skipped=15)
98.log:OK (skipped=15)
98.log:OK (skipped=2)
99.log:382 tests OK.
99.log:OK (skipped=15)
99.log:OK (skipped=2)
100.log:383 tests OK.
100.log:OK (skipped=2)
101.log:381 tests OK.
101.log:OK (skipped=15)
101.log:OK (skipped=15)
101.log:OK (skipped=2)
102.log:382 tests OK.
102.log:OK (skipped=15)
102.log:OK (skipped=2)
103.log:381 tests OK.
103.log:OK (skipped=15)
103.log:OK (skipped=2)
103.log:OK (skipped=37)
104.log:382 tests OK.
104.log:OK (skipped=2)
105.log:381 tests OK.
105.log:OK (skipped=15)
105.log:OK (skipped=15)
105.log:OK (skipped=2)
106.log:383 tests OK.
106.log:OK (skipped=2)
107.log:380 tests OK.
107.log:OK (skipped=15)
107.log:OK (skipped=15)
107.log:OK (skipped=15)
107.log:OK (skipped=2)
108.log:379 tests OK.
108.log:OK (skipped=15)
108.log:OK (skipped=15)
108.log:OK (skipped=15)
108.log:OK (skipped=2)
109.log:382 tests OK.
109.log:OK (skipped=15)
109.log:OK (skipped=2)
110.log:383 tests OK.
110.log:OK (skipped=2)
111.log:383 tests OK.
111.log:OK (skipped=2)
112.log:383 tests OK.
112.log:OK (skipped=2)
113.log:383 tests OK.
113.log:OK (skipped=2)
114.log:383 tests OK.
114.log:OK (skipped=2)
115.log:382 tests OK.
115.log:OK (skipped=15)
115.log:OK (skipped=2)
116.log:383 tests OK.
116.log:OK (skipped=2)
117.log:381 tests OK.
117.log:OK (skipped=15)
117.log:OK (skipped=15)
117.log:OK (skipped=2)
118.log:382 tests OK.
118.log:OK (skipped=15)
118.log:OK (skipped=2)
119.log:381 tests OK.
119.log:OK (skipped=15)
119.log:OK (skipped=15)
119.log:OK (skipped=2)
120.log:381 tests OK.
120.log:OK (skipped=15)
120.log:OK (skipped=15)
120.log:OK (skipped=2)
121.log:382 tests OK.
121.log:OK (skipped=15)
121.log:OK (skipped=2)
122.log:381 tests OK.
122.log:OK (skipped=15)
122.log:OK (skipped=15)
122.log:OK (skipped=2)
123.log:382 tests OK.
123.log:OK (skipped=15)
123.log:OK (skipped=2)
124.log:382 tests OK.
124.log:OK (skipped=15)
124.log:OK (skipped=2)
125.log:382 tests OK.
125.log:OK (skipped=15)
125.log:OK (skipped=2)
126.log:383 tests OK.
126.log:OK (skipped=2)
127.log:380 tests OK.
127.log:OK (skipped=15)
127.log:OK (skipped=15)
127.log:OK (skipped=15)
127.log:OK (skipped=2)
128.log:383 tests OK.
128.log:OK (skipped=2)
129.log:380 tests OK.
129.log:OK (skipped=15)
129.log:OK (skipped=15)
129.log:OK (skipped=15)
129.log:OK (skipped=2)
130.log:380 tests OK.
130.log:OK (skipped=15)
130.log:OK (skipped=15)
130.log:OK (skipped=15)
130.log:OK (skipped=2)
131.log:381 tests OK.
131.log:OK (skipped=15)
131.log:OK (skipped=15)
131.log:OK (skipped=2)
132.log:381 tests OK.
132.log:OK (skipped=2)
132.log:OK (skipped=37)
133.log:382 tests OK.
133.log:OK (skipped=2)
134.log:380 tests OK.
134.log:OK (skipped=15)
134.log:OK (skipped=15)
134.log:OK (skipped=15)
134.log:OK (skipped=2)
135.log:382 tests OK.
135.log:OK (skipped=2)
135.log:OK (skipped=39)
136.log:380 tests OK.
136.log:OK (skipped=15)
136.log:OK (skipped=15)
136.log:OK (skipped=15)
136.log:OK (skipped=2)
137.log:383 tests OK.
137.log:OK (skipped=2)
138.log:381 tests OK.
138.log:OK (skipped=15)
138.log:OK (skipped=15)
138.log:OK (skipped=2)
139.log:383 tests OK.
139.log:OK (skipped=2)
140.log:383 tests OK.
140.log:OK (skipped=2)
141.log:382 tests OK.
141.log:OK (skipped=15)
141.log:OK (skipped=2)
142.log:380 tests OK.
142.log:OK (skipped=15)
142.log:OK (skipped=15)
142.log:OK (skipped=15)
142.log:OK (skipped=2)
143.log:380 tests OK.
143.log:OK (skipped=15)
143.log:OK (skipped=15)
143.log:OK (skipped=15)
143.log:OK (skipped=2)
144.log:380 tests OK.
144.log:OK (skipped=15)
144.log:OK (skipped=15)
144.log:OK (skipped=15)
144.log:OK (skipped=2)
145.log:380 tests OK.
145.log:OK (skipped=15)
145.log:OK (skipped=15)
145.log:OK (skipped=15)
145.log:OK (skipped=2)
146.log:382 tests OK.
146.log:OK (skipped=15)
146.log:OK (skipped=2)
147.log:382 tests OK.
147.log:OK (skipped=15)
147.log:OK (skipped=2)
148.log:381 tests OK.
148.log:OK (skipped=15)
148.log:OK (skipped=15)
148.log:OK (skipped=2)
149.log:382 tests OK.
149.log:OK (skipped=15)
149.log:OK (skipped=2)
150.log:380 tests OK.
150.log:OK (skipped=15)
150.log:OK (skipped=15)
150.log:OK (skipped=15)
150.log:OK (skipped=2)
151.log:381 tests OK.
151.log:OK (skipped=15)
151.log:OK (skipped=15)
151.log:OK (skipped=2)
152.log:383 tests OK.
152.log:OK (skipped=2)
153.log:381 tests OK.
153.log:OK (skipped=15)
153.log:OK (skipped=15)
153.log:OK (skipped=2)
154.log:382 tests OK.
154.log:OK (skipped=15)
154.log:OK (skipped=2)
155.log:381 tests OK.
155.log:OK (skipped=15)
155.log:OK (skipped=15)
155.log:OK (skipped=2)
156.log:383 tests OK.
156.log:OK (skipped=2)
157.log:380 tests OK.
157.log:OK (skipped=15)
157.log:OK (skipped=15)
157.log:OK (skipped=15)
157.log:OK (skipped=2)
158.log:381 tests OK.
158.log:OK (skipped=15)
158.log:OK (skipped=15)
158.log:OK (skipped=2)
159.log:382 tests OK.
159.log:OK (skipped=15)
159.log:OK (skipped=2)
160.log:382 tests OK.
160.log:OK (skipped=15)
160.log:OK (skipped=2)
161.log:381 tests OK.
161.log:OK (skipped=15)
161.log:OK (skipped=15)
161.log:OK (skipped=2)
162.log:383 tests OK.
162.log:OK (skipped=2)
163.log:383 tests OK.
163.log:OK (skipped=2)
164.log:383 tests OK.
164.log:OK (skipped=2)
165.log:381 tests OK.
165.log:OK (skipped=15)
165.log:OK (skipped=15)
165.log:OK (skipped=2)
166.log:383 tests OK.
166.log:OK (skipped=2)
167.log:382 tests OK.
167.log:OK (skipped=15)
167.log:OK (skipped=2)
168.log:383 tests OK.
168.log:OK (skipped=2)
169.log:380 tests OK.
169.log:OK (skipped=15)
169.log:OK (skipped=15)
169.log:OK (skipped=15)
169.log:OK (skipped=2)
170.log:380 tests OK.
170.log:OK (skipped=15)
170.log:OK (skipped=15)
170.log:OK (skipped=15)
170.log:OK (skipped=2)
171.log:383 tests OK.
171.log:OK (skipped=2)
172.log:382 tests OK.
172.log:OK (skipped=15)
172.log:OK (skipped=2)
173.log:381 tests OK.
173.log:OK (skipped=15)
173.log:OK (skipped=15)
173.log:OK (skipped=2)
174.log:380 tests OK.
174.log:OK (skipped=15)
174.log:OK (skipped=15)
174.log:OK (skipped=15)
174.log:OK (skipped=2)
175.log:381 tests OK.
175.log:OK (skipped=15)
175.log:OK (skipped=15)
175.log:OK (skipped=2)
176.log:380 tests OK.
176.log:OK (skipped=15)
176.log:OK (skipped=15)
176.log:OK (skipped=15)
176.log:OK (skipped=2)
177.log:382 tests OK.
177.log:OK (skipped=15)
177.log:OK (skipped=2)
178.log:381 tests OK.
178.log:OK (skipped=15)
178.log:OK (skipped=15)
178.log:OK (skipped=2)
179.log:381 tests OK.
179.log:OK (skipped=15)
179.log:OK (skipped=15)
179.log:OK (skipped=2)
180.log:381 tests OK.
180.log:OK (skipped=15)
180.log:OK (skipped=15)
180.log:OK (skipped=2)
181.log:382 tests OK.
181.log:OK (skipped=15)
181.log:OK (skipped=2)
182.log:381 tests OK.
182.log:OK (skipped=15)
182.log:OK (skipped=15)
182.log:OK (skipped=2)
183.log:380 tests OK.
183.log:OK (skipped=15)
183.log:OK (skipped=15)
183.log:OK (skipped=15)
183.log:OK (skipped=2)
184.log:381 tests OK.
184.log:OK (skipped=15)
184.log:OK (skipped=15)
184.log:OK (skipped=2)
185.log:381 tests OK.
185.log:OK (skipped=15)
185.log:OK (skipped=15)
185.log:OK (skipped=2)
186.log:380 tests OK.
186.log:OK (skipped=15)
186.log:OK (skipped=15)
186.log:OK (skipped=15)
186.log:OK (skipped=2)
187.log:380 tests OK.
187.log:OK (skipped=15)
187.log:OK (skipped=15)
187.log:OK (skipped=15)
187.log:OK (skipped=2)
188.log:382 tests OK.
188.log:OK (skipped=15)
188.log:OK (skipped=2)
189.log:380 tests OK.
189.log:OK (skipped=15)
189.log:OK (skipped=15)
189.log:OK (skipped=15)
189.log:OK (skipped=2)
190.log:383 tests OK.
190.log:OK (skipped=2)
191.log:380 tests OK.
191.log:OK (skipped=15)
191.log:OK (skipped=15)
191.log:OK (skipped=15)
191.log:OK (skipped=2)
192.log:382 tests OK.
192.log:OK (skipped=15)
192.log:OK (skipped=2)
193.log:383 tests OK.
193.log:OK (skipped=2)
194.log:382 tests OK.
194.log:OK (skipped=2)
195.log:383 tests OK.
195.log:OK (skipped=2)
196.log:383 tests OK.
196.log:OK (skipped=2)
197.log:382 tests OK.
197.log:OK (skipped=15)
197.log:OK (skipped=2)
198.log:380 tests OK.
198.log:OK (skipped=15)
198.log:OK (skipped=15)
198.log:OK (skipped=15)
198.log:OK (skipped=2)
199.log:380 tests OK.
199.log:OK (skipped=15)
199.log:OK (skipped=15)
199.log:OK (skipped=15)
199.log:OK (skipped=2)
200.log:380 tests OK.
200.log:OK (skipped=15)
200.log:OK (skipped=15)
200.log:OK (skipped=15)
200.log:OK (skipped=2)
201.log:382 tests OK.
201.log:OK (skipped=15)
201.log:OK (skipped=2)
202.log:380 tests OK.
202.log:OK (skipped=15)
202.log:OK (skipped=15)
202.log:OK (skipped=15)
202.log:OK (skipped=2)
203.log:380 tests OK.
203.log:OK (skipped=15)
203.log:OK (skipped=15)
203.log:OK (skipped=15)
203.log:OK (skipped=2)
204.log:382 tests OK.
204.log:OK (skipped=15)
204.log:OK (skipped=2)
205.log:382 tests OK.
205.log:OK (skipped=15)
205.log:OK (skipped=2)
206.log:379 tests OK.
206.log:OK (skipped=15)
206.log:OK (skipped=15)
206.log:OK (skipped=15)
206.log:OK (skipped=2)
207.log:382 tests OK.
207.log:OK (skipped=15)
207.log:OK (skipped=2)
208.log:382 tests OK.
208.log:OK (skipped=15)
208.log:OK (skipped=2)
209.log:383 tests OK.
209.log:OK (skipped=2)
210.log:383 tests OK.
210.log:OK (skipped=2)
211.log:382 tests OK.
211.log:OK (skipped=2)
211.log:OK (skipped=39)
212.log:380 tests OK.
212.log:OK (skipped=15)
212.log:OK (skipped=15)
212.log:OK (skipped=15)
212.log:OK (skipped=2)
213.log:382 tests OK.
213.log:OK (skipped=15)
213.log:OK (skipped=2)
214.log:382 tests OK.
214.log:OK (skipped=15)
214.log:OK (skipped=2)
215.log:382 tests OK.
215.log:OK (skipped=15)
215.log:OK (skipped=2)
216.log:383 tests OK.
216.log:OK (skipped=2)
217.log:381 tests OK.
217.log:OK (skipped=15)
217.log:OK (skipped=15)
217.log:OK (skipped=2)
218.log:382 tests OK.
218.log:OK (skipped=15)
218.log:OK (skipped=2)
219.log:382 tests OK.
219.log:OK (skipped=15)
219.log:OK (skipped=2)
220.log:382 tests OK.
220.log:OK (skipped=15)
220.log:OK (skipped=2)
221.log:382 tests OK.
221.log:OK (skipped=15)
221.log:OK (skipped=2)
222.log:383 tests OK.
222.log:OK (skipped=2)
223.log:382 tests OK.
223.log:OK (skipped=15)
223.log:OK (skipped=2)
224.log:381 tests OK.
224.log:OK (skipped=15)
224.log:OK (skipped=15)
224.log:OK (skipped=2)
225.log:382 tests OK.
225.log:OK (skipped=15)
225.log:OK (skipped=2)
226.log:380 tests OK.
226.log:OK (skipped=15)
226.log:OK (skipped=15)
226.log:OK (skipped=15)
226.log:OK (skipped=2)
227.log:381 tests OK.
227.log:OK (skipped=15)
227.log:OK (skipped=15)
227.log:OK (skipped=2)
228.log:383 tests OK.
228.log:OK (skipped=2)
229.log:382 tests OK.
229.log:OK (skipped=15)
229.log:OK (skipped=2)
230.log:381 tests OK.
230.log:OK (skipped=15)
230.log:OK (skipped=15)
230.log:OK (skipped=2)
231.log:382 tests OK.
231.log:OK (skipped=15)
231.log:OK (skipped=2)
232.log:381 tests OK.
232.log:OK (skipped=15)
232.log:OK (skipped=15)
232.log:OK (skipped=2)
233.log:382 tests OK.
233.log:OK (skipped=15)
233.log:OK (skipped=2)
234.log:382 tests OK.
234.log:OK (skipped=15)
234.log:OK (skipped=2)
235.log:382 tests OK.
235.log:OK (skipped=2)
235.log:OK (skipped=37)
236.log:381 tests OK.
236.log:OK (skipped=15)
236.log:OK (skipped=15)
236.log:OK (skipped=2)
237.log:383 tests OK.
237.log:OK (skipped=2)
238.log:381 tests OK.
238.log:OK (skipped=15)
238.log:OK (skipped=15)
238.log:OK (skipped=2)
239.log:381 tests OK.
239.log:OK (skipped=15)
239.log:OK (skipped=15)
239.log:OK (skipped=2)
240.log:381 tests OK.
240.log:OK (skipped=15)
240.log:OK (skipped=15)
240.log:OK (skipped=2)
241.log:381 tests OK.
241.log:OK (skipped=15)
241.log:OK (skipped=15)
241.log:OK (skipped=2)
242.log:380 tests OK.
242.log:OK (skipped=15)
242.log:OK (skipped=15)
242.log:OK (skipped=15)
242.log:OK (skipped=2)
243.log:382 tests OK.
243.log:OK (skipped=15)
243.log:OK (skipped=2)
244.log:381 tests OK.
244.log:OK (skipped=15)
244.log:OK (skipped=15)
244.log:OK (skipped=2)
245.log:380 tests OK.
245.log:OK (skipped=15)
245.log:OK (skipped=15)
245.log:OK (skipped=15)
245.log:OK (skipped=2)
246.log:380 tests OK.
246.log:OK (skipped=15)
246.log:OK (skipped=15)
246.log:OK (skipped=15)
246.log:OK (skipped=2)
247.log:380 tests OK.
247.log:OK (skipped=15)
247.log:OK (skipped=15)
247.log:OK (skipped=15)
247.log:OK (skipped=2)
248.log:380 tests OK.
248.log:OK (skipped=15)
248.log:OK (skipped=15)
248.log:OK (skipped=15)
248.log:OK (skipped=2)
249.log:381 tests OK.
249.log:OK (skipped=15)
249.log:OK (skipped=15)
249.log:OK (skipped=2)
250.log:380 tests OK.
250.log:OK (skipped=15)
250.log:OK (skipped=15)
250.log:OK (skipped=15)
250.log:OK (skipped=2)
251.log:383 tests OK.
251.log:OK (skipped=2)
252.log:383 tests OK.
252.log:OK (skipped=2)
253.log:383 tests OK.
253.log:OK (skipped=2)
254.log:382 tests OK.
254.log:OK (skipped=2)
254.log:OK (skipped=37)
255.log:379 tests OK.
255.log:OK (skipped=15)
255.log:OK (skipped=15)
255.log:OK (skipped=15)
255.log:OK (skipped=2)
255.log:OK (skipped=37)
[toc] | [next] | [standalone]
| From | Peter Otten <__peter__@web.de> |
|---|---|
| Date | 2015-09-20 10:58 +0200 |
| Message-ID | <mailman.35.1442739550.21674.python-list@python.org> |
| In reply to | #96865 |
paul.anton.letnes@gmail.com wrote: > Fascinated by the concept of ordering dependent test failures [0], I've > run the python test suite [1] with 256 different random seeds (took a > little more than 12 hours). The results vary a lot - for instance, the > number of tests reported as OK varies, the number of skips varies, etc. > Since I'm not sure how to report or interpret them, I'll just post a > summary below. > > The test suite was run on arch linux [2] with gcc 5.2.0, with the source > code taken from a clone of the python repo yesterday [3]. > > What could I do with all this in order to make more sense of it, and could > it be of any help what so ever to python development? I'll gladly make the > full log files available to whomever is interested, in whatever format is > convenient. In the meantime I'll run more random seeds, because why not. Given that the most likely interference is between two tests instead of the shotgun approach you could run the unit tests in such a way that for any pair of tests a, b the tests are run at least once with a before b and with a after b. Assuming that a test c does not "destroy" the interference when run between a and b for three tests this could be achieved with 1 abc covers ab, ac, bc 2 cba covers ba, ca, cb If b fails in run 1 you'll assume a to be the culprit. If c fails in run 1 you'll follow up with 1.1 ac 1.2 bc to identify the predecessor causing the failure. Of course there are some complications, e. g. - Neither 1.1 nor 1.2 or both may fail - The typical test suite comprises more than three tests - Should the sequence aa be covered, i. e. a test case interfering with itself?
[toc] | [prev] | [standalone]
Back to top | Article view | comp.lang.python
csiph-web