Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #45808 > unrolled thread

Debugging parallel nose tests?

Back to article view | Back to comp.lang.python

Contents

  Debugging parallel nose tests? Roy Smith <roy@panix.com> - 2013-05-23 09:09 -0400
    Re: Debugging parallel nose tests? Dave Angel <davea@davea.name> - 2013-05-23 14:31 -0400
      Re: Debugging parallel nose tests? Roy Smith <roy@panix.com> - 2013-05-23 21:44 -0400
    Re: Debugging parallel nose tests? Roy Smith <roy@panix.com> - 2013-05-24 23:09 -0400

#45808 — Debugging parallel nose tests?

I've got a suite of about 150 tests written using unittest.  It takes 
5-10 minutes to run, so I'd really like to use nose to run things in 
parallel.  The problem is, when I do that, I get lots of test failures.  
Obviously, we've got some kind of inter-test dependency that I haven't 
been able to locate.

Is there some way to make nose print a report of how it partitioned the 
tests across the various processes?  If I could see that, it might help 
us reproduce the failures.

We're using:

nosetests --process-timeout=60 --processes=40 test_api.py

and

_multiprocess_can_split_ = True

[toc] | [next] | [standalone]

#45831

On 05/23/2013 09:09 AM, Roy Smith wrote:
>
>      <SNIP>
>
> nosetests --process-timeout=60 --processes=40 test_api.py
>

Do you have a 40-processor system?  And do you have enough RAM to run 
all of those processes?


-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#45859

In article <mailman.2027.1369333910.3114.python-list@python.org>,
 Dave Angel <davea@davea.name> wrote:

> On 05/23/2013 09:09 AM, Roy Smith wrote:
> >
> >      <SNIP>
> >
> > nosetests --process-timeout=60 --processes=40 test_api.py
> >
> 
> Do you have a 40-processor system?

No, but many of the tests are I/O bound.

> And do you have enough RAM to run all of those processes?

Yes.  The box I'm on now has 8 gig.  I'd doing a 40 process run right 
now, and I've still got 1/2 gig free.

[toc] | [prev] | [next] | [standalone]

#45941

In article <roy-1CC2EA.09092123052013@news.panix.com>,
 Roy Smith <roy@panix.com> wrote:

> Is there some way to make nose print a report of how it partitioned the 
> tests across the various processes?

I never found such a feature, but I did figure out a way to do what I 
needed.  We use a system of unique id's to track HTTP requests through 
our system.  All you need do is add a X-Unique-Request-ID header and any 
of our server processes will log that.  So, I just had my test suite add:

X-Unique-Request-ID: name-of-test.process-id

to each HTTP request.  That got me what I was looking for (to know what 
tests were grouped into the same process).

> We're using:
> 
> nosetests --process-timeout=60 --processes=40 test_api.py

It turned out, the answer was purely a timing issue.  We had some tests 
that called out to facebook.  Those were (intermittently) so slow they 
caused the 60 second timeout to be exceeded.  The solution there was to 
pull the facebook tests out to their own suite (which we'll run in 
single-process mode).

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web