Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #18596 > unrolled thread

replacing __dict__ with an OrderedDict

Back to article view | Back to comp.lang.python

Contents

  replacing __dict__ with an OrderedDict Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-01-06 10:48 +0100
    Re: replacing __dict__ with an OrderedDict Lie Ryan <lie.1296@gmail.com> - 2012-01-06 22:43 +1100
      Re: replacing __dict__ with an OrderedDict Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-01-06 14:36 +0100
        Re: replacing __dict__ with an OrderedDict Lie Ryan <lie.1296@gmail.com> - 2012-01-07 04:01 +1100
          Re: replacing __dict__ with an OrderedDict Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-01-07 00:45 +0000
        Re: replacing __dict__ with an OrderedDict Ian Kelly <ian.g.kelly@gmail.com> - 2012-01-06 10:20 -0700
          Re: replacing __dict__ with an OrderedDict Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-01-07 00:49 +0000
            Re: replacing __dict__ with an OrderedDict Lie Ryan <lie.1296@gmail.com> - 2012-01-07 12:21 +1100
            Re: replacing __dict__ with an OrderedDict Ian Kelly <ian.g.kelly@gmail.com> - 2012-01-06 18:24 -0700
              Re: replacing __dict__ with an OrderedDict Eelco <hoogendoorn.eelco@gmail.com> - 2012-01-08 14:03 -0800
                Re: replacing __dict__ with an OrderedDict Lie Ryan <lie.1296@gmail.com> - 2012-01-09 23:10 +1100
                  Re: replacing __dict__ with an OrderedDict Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-01-09 14:16 +0100
                    Re: replacing __dict__ with an OrderedDict Lie Ryan <lie.1296@gmail.com> - 2012-01-10 23:31 +1100
                      Re: replacing __dict__ with an OrderedDict Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-01-10 18:21 +0100
          Re: replacing __dict__ with an OrderedDict Roy Smith <roy@panix.com> - 2012-01-09 09:35 -0500
            Re: replacing __dict__ with an OrderedDict Neil Cerutti <neilc@norwich.edu> - 2012-01-09 14:52 +0000
            Re: replacing __dict__ with an OrderedDict Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-01-09 17:59 +0100
              Re: replacing __dict__ with an OrderedDict Ian Kelly <ian.g.kelly@gmail.com> - 2012-01-09 10:30 -0700
              Re: replacing __dict__ with an OrderedDict Roy Smith <roy@panix.com> - 2012-01-09 20:05 -0500
                Re: replacing __dict__ with an OrderedDict Terry Reedy <tjreedy@udel.edu> - 2012-01-09 23:21 -0500
                Re: replacing __dict__ with an OrderedDict Lie Ryan <lie.1296@gmail.com> - 2012-01-10 23:22 +1100
                  Re: replacing __dict__ with an OrderedDict Roy Smith <roy@panix.com> - 2012-01-10 09:05 -0500
                    Re: replacing __dict__ with an OrderedDict Lie Ryan <lie.1296@gmail.com> - 2012-01-11 02:57 +1100
                      Re: replacing __dict__ with an OrderedDict Roy Smith <roy@panix.com> - 2012-01-10 21:47 -0500
                    Re: replacing __dict__ with an OrderedDict Tim Wintle <tim.wintle@teamrubber.com> - 2012-01-10 15:45 +0000
              Re: replacing __dict__ with an OrderedDict Lie Ryan <lie.1296@gmail.com> - 2012-01-10 23:46 +1100
        Re: replacing __dict__ with an OrderedDict Lie Ryan <lie.1296@gmail.com> - 2012-01-07 04:42 +1100
    Re: replacing __dict__ with an OrderedDict Peter Otten <__peter__@web.de> - 2012-01-06 12:44 +0100
      Re: replacing __dict__ with an OrderedDict Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-01-06 14:40 +0100
        Re: replacing __dict__ with an OrderedDict Ian Kelly <ian.g.kelly@gmail.com> - 2012-01-06 09:06 -0700
          Re: replacing __dict__ with an OrderedDict alex23 <wuwei23@gmail.com> - 2012-01-08 20:58 -0800
        Re: replacing __dict__ with an OrderedDict Ian Kelly <ian.g.kelly@gmail.com> - 2012-01-06 09:07 -0700
        Re: replacing __dict__ with an OrderedDict Arnaud Delobelle <arnodel@gmail.com> - 2012-01-06 21:38 +0000
      Re: replacing __dict__ with an OrderedDict Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-01-18 11:29 +0100
    Re: replacing __dict__ with an OrderedDict Ethan Furman <ethan@stoneleaf.us> - 2012-01-06 06:13 -0800
    Re: replacing __dict__ with an OrderedDict Arnaud Delobelle <arnodel@gmail.com> - 2012-01-06 21:32 +0000

Page 1 of 2  [1] 2  Next page →

#18596 — replacing __dict__ with an OrderedDict

Hi!

The topic explains pretty much what I'm trying to do under Python 
2.7[1]. The reason for this is that I want dir(SomeType) to show the 
attributes in the order of their declaration. This in turn should 
hopefully make unittest execute my tests in the order of their 
declaration[2], so that the output becomes more readable and structured, 
just as my test code (hopefully) is.

I've been toying around with metaclasses, trying to replace __dict__ 
directly, or using type() to construct the class, but either I hit 
read-only attributes or the changes seem to be ignored.

As additional thought (but I'm not there yet), I guess that if I want 
dir(some_object) to be ordered similarly, I will also have to perform 
some steps on instance creation, not only on class creation, right?

Thanks for any suggestions!

Uli


[1] I'm stuck with 2.x for now, from what I found it would be easier 
using 3.x.

[2] No, don't tell me that the tests should run in any order. It's not 
the case that my tests depend on each other, but if one basic test for 
the presence of an API fails, the elaborate tests using that API will 
obviously fail, too, and I just want to take the first test that fails 
and analyse that instead of guessing the point to start debugging from 
the N failed tests.

[toc] | [next] | [standalone]

#18597

On 01/06/2012 08:48 PM, Ulrich Eckhardt wrote:
> Hi!
>
> The topic explains pretty much what I'm trying to do under Python
> 2.7[1]. The reason for this is that I want dir(SomeType) to show the
> attributes in the order of their declaration. This in turn should
> hopefully make unittest execute my tests in the order of their
> declaration[2], so that the output becomes more readable and structured,
> just as my test code (hopefully) is.

IMO that's a futile effort, first, because as you already know, the test 
should not rely on the order. If you want the result to be printed in a 
certain order, that's a display issue, not testing order issue. I guess 
you would have better luck doing what you want by customizing the 
TestResult or TestRunner.

[toc] | [prev] | [next] | [standalone]

#18602

Am 06.01.2012 12:43, schrieb Lie Ryan:
> On 01/06/2012 08:48 PM, Ulrich Eckhardt wrote:
>> Hi!
>>
>> The topic explains pretty much what I'm trying to do under Python
>> 2.7[1]. The reason for this is that I want dir(SomeType) to show the
>> attributes in the order of their declaration. This in turn should
>> hopefully make unittest execute my tests in the order of their
>> declaration[2], so that the output becomes more readable and structured,
>> just as my test code (hopefully) is.
>
> IMO that's a futile effort, first, because as you already know, the test
> should not rely on the order. If you want the result to be printed in a
> certain order, that's a display issue, not testing order issue.

I'm not sure if you just -ahem- enjoy Usenet discussions, but please 
read the footnote that explains that I don't want to discuss this part 
and why you are actually wrong with your partial understanding of the 
issue. ;^)


 > I guess you would have better luck doing what you want by customizing
 > the TestResult or TestRunner.

True, perhaps, but doing it this way would be more fun and easier 
reusable in other cases where the default order is not desirable. I can 
also go and name the test functions test_000 to test_009 to get results 
quickly, if that was the only goal.

Cheers!

Uli

[toc] | [prev] | [next] | [standalone]

#18609

On 01/07/2012 12:36 AM, Ulrich Eckhardt wrote:
> Am 06.01.2012 12:43, schrieb Lie Ryan:
>> On 01/06/2012 08:48 PM, Ulrich Eckhardt wrote:
>>> Hi!
>>>
>>> The topic explains pretty much what I'm trying to do under Python
>>> 2.7[1]. The reason for this is that I want dir(SomeType) to show the
>>> attributes in the order of their declaration. This in turn should
>>> hopefully make unittest execute my tests in the order of their
>>> declaration[2], so that the output becomes more readable and structured,
>>> just as my test code (hopefully) is.
>>
>> IMO that's a futile effort, first, because as you already know, the test
>> should not rely on the order. If you want the result to be printed in a
>> certain order, that's a display issue, not testing order issue.
>
> I'm not sure if you just -ahem- enjoy Usenet discussions, but please
> read the footnote that explains that I don't want to discuss this part
> and why you are actually wrong with your partial understanding of the
> issue. ;^)

I fully understand your issue, and I stand by my opinion. I believe 
you're misunderstanding the nature of your problem, your issue is not 
that you want a customized test order execution, but you want a 
customized view of the test result.

That unittest executes its tests in alphabetical order is implementation 
detail for a very good reason, and good unittest practice dictates that 
execution order should never be defined (some even argued that the 
execution order should be randomized). If the test runner turns out to 
execute tests concurrently, that should not cause problems for a 
well-designed test. Displaying the test results in a more convenient 
order for viewing is what you really wanted in 99.99% of the cases.

>  > I guess you would have better luck doing what you want by customizing
>  > the TestResult or TestRunner.
>
> True, perhaps, but doing it this way would be more fun and easier
> reusable in other cases where the default order is not desirable. I can
> also go and name the test functions test_000 to test_009 to get results
> quickly, if that was the only goal.

Fun and easier, perhaps. Except that it solves the wrong problem.

[toc] | [prev] | [next] | [standalone]

#18625

On Sat, 07 Jan 2012 04:01:44 +1100, Lie Ryan wrote:

> On 01/07/2012 12:36 AM, Ulrich Eckhardt wrote:
>> True, perhaps, but doing it this way would be more fun and easier
>> reusable in other cases where the default order is not desirable. I can
>> also go and name the test functions test_000 to test_009 to get results
>> quickly, if that was the only goal.
> 
> Fun and easier, perhaps. Except that it solves the wrong problem.

Fun and easier, and a terrible thing to do. Contrast:

"test_binsearch_tuple is failing. What does that mean?"
"Oh, that tests that binsearch works on a tuple. You need to look at the 
BinSearch.search_tuple method and see what it's doing."

with 

"test_047 is failing. What does that mean?"
"How the hell should I know?"


Debugging is hard enough without obfuscating the test names. You wouldn't 
write a function called "func_009", why write one called "test_009"?



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#18610

On Fri, Jan 6, 2012 at 10:01 AM, Lie Ryan <lie.1296@gmail.com> wrote:
> That unittest executes its tests in alphabetical order is implementation
> detail for a very good reason, and good unittest practice dictates that
> execution order should never be defined (some even argued that the execution
> order should be randomized). If the test runner turns out to execute tests
> concurrently, that should not cause problems for a well-designed test.
> Displaying the test results in a more convenient order for viewing is what
> you really wanted in 99.99% of the cases.

Randomizing the order is not a bad idea, but you also need to be able
to run the tests in a consistent order, from a specific random seed.
In the real world, test conflicts and dependencies do happen, and if
we observe a failure, make a change, rerun the tests and observe
success, we need to be able to be sure that we actually fixed the bug,
and that it didn't pass only because it was run in a different order.

Concurrent testing is a bad idea for this reason -- it's not
repeatable (testing concurrency, OTOH, is a perfectly fine thing to be
thinking about).

Cheers,
Ian

[toc] | [prev] | [next] | [standalone]

#18626

On Fri, 06 Jan 2012 10:20:28 -0700, Ian Kelly wrote:

> On Fri, Jan 6, 2012 at 10:01 AM, Lie Ryan <lie.1296@gmail.com> wrote:
>> That unittest executes its tests in alphabetical order is
>> implementation detail for a very good reason, and good unittest
>> practice dictates that execution order should never be defined (some
>> even argued that the execution order should be randomized). If the test
>> runner turns out to execute tests concurrently, that should not cause
>> problems for a well-designed test. Displaying the test results in a
>> more convenient order for viewing is what you really wanted in 99.99%
>> of the cases.
> 
> Randomizing the order is not a bad idea, but you also need to be able to
> run the tests in a consistent order, from a specific random seed. In the
> real world, test conflicts and dependencies do happen,

In the real world, test conflicts and dependencies are bugs in your test 
suite that should be fixed, like any other bug in code. The fact that it 
is test code that is failing is irrelevant.

Also in the real world, sometimes it is too hard to fix a bug and you 
just live with it.


> and if we observe
> a failure, make a change, rerun the tests and observe success, we need
> to be able to be sure that we actually fixed the bug, and that it didn't
> pass only because it was run in a different order.

Every test should stand alone. You should be able to isolate each and 
every test and run it without the others. If you can't, your test suite 
is buggy. (Chances are good that most test suites are buggy. After all, 
who writes tests for their tests? That's just the start of an infinite 
regress with rapidly diminishing benefit.) 

You may not be able to run tests *simultaneously*, due to clashes 
involving external resources, but you should be able to run them in 
random order.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#18629

On 01/07/2012 11:49 AM, Steven D'Aprano wrote:
> You may not be able to run tests*simultaneously*, due to clashes
> involving external resources, but you should be able to run them in
> random order.

tests that involves external resources should be mocked, although there 
are always a few external resources that cannot be mocked, those are the 
exceptions not the rule; a concurrent test runner might have mechanisms 
to mark them to be run synchronously.

[toc] | [prev] | [next] | [standalone]

#18630

On Fri, Jan 6, 2012 at 5:49 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> In the real world, test conflicts and dependencies are bugs in your test
> suite that should be fixed, like any other bug in code. The fact that it
> is test code that is failing is irrelevant.

I agree 100%, but none of that changes my point, which is that when
that this sort of problem arises, you need to be able to test
consistently to know that the bug is actually fixed.

> Every test should stand alone. You should be able to isolate each and
> every test and run it without the others.

Ideally, yes.  I'm talking about *unintentional* conflicts and dependencies.

[toc] | [prev] | [next] | [standalone]

#18686

i havnt read every post in great detail, but it doesnt seem like your
actual question has been answered, so ill give it a try.

AFAIK, changing __dict__ to be an ordereddict is fundamentally
impossible in python 2. __dict__ is a builtin language construct
hardcoded into the C API. There is no way to mess with it.

Apparently this is different in python 3, but I dont know much about
that.

[toc] | [prev] | [next] | [standalone]

#18706

On 01/09/2012 09:03 AM, Eelco wrote:
> i havnt read every post in great detail, but it doesnt seem like your
> actual question has been answered, so ill give it a try.
>
> AFAIK, changing __dict__ to be an ordereddict is fundamentally
> impossible in python 2. __dict__ is a builtin language construct
> hardcoded into the C API. There is no way to mess with it.
>
> Apparently this is different in python 3, but I dont know much about
> that.

Actually the primary question has been answered by Ian Kelly which 
suggested __prepare__ for Python 3, and Peter Otten posted a code for a 
custom TestLoader that will essentially do what the OP wanted.

I was just suggesting that what the OP thinks he wants is quite likely 
not what he actually wants.

[toc] | [prev] | [next] | [standalone]

#18707

Am 09.01.2012 13:10, schrieb Lie Ryan:
> I was just suggesting that what the OP thinks he wants is quite
> likely  not what he actually wants.

Rest assured that the OP has a rather good idea of what he wants and 
why, the latter being something you don't know, because he never 
bothered to explain it and you never asked. Please don't think he's an 
idiot just because he wants something that doesn't make sense to you.

*le sigh*

Uli

[toc] | [prev] | [next] | [standalone]

#18753

On 01/10/2012 12:16 AM, Ulrich Eckhardt wrote:
> Am 09.01.2012 13:10, schrieb Lie Ryan:
>> I was just suggesting that what the OP thinks he wants is quite
>> likely not what he actually wants.
>
> Rest assured that the OP has a rather good idea of what he wants and
> why, the latter being something you don't know, because he never
> bothered to explain it and you never asked. Please don't think he's an
> idiot just because he wants something that doesn't make sense to you.

The OP explained the "why" clearly in his first post, he wanted to see 
his test results ordered in a certain way to make debugging easier, to 
quote the OP:

"""
... I just want to take the first test that fails and analyse that 
instead of guessing the point to start debugging from the N failed tests.
"""

and then he goes on concluding that he need to reorder the tests itself 
and to replace __dict__ with OrderedDict. While it is possible to 
replace __dict__ with OrderedDict and it is possible to reorder the 
test, those are not his original problem, and the optimal solution to 
his original problem differs from the optimal solution to what he think 
he will need.

I had said this before and I'm saying it again: the problem is a test 
result displaying issue, not testing order issue.

[toc] | [prev] | [next] | [standalone]

#18777

Am 10.01.2012 13:31, schrieb Lie Ryan:
> While it is possible to replace __dict__ with OrderedDict and it is
> possible to reorder the test, those are not his original problem, and
> the optimal solution to his original problem differs from the optimal
> solution to what he think he will need.

Oh, and you know what is optimal in his environment? You really think 
you know better what to do based on the little information provided?


> I had said this before and I'm saying it again: the problem is a test
> result displaying issue, not testing order issue.

There are good reasons for not reordering the results of the tests that 
would be sacrificed by reordering them afterwards.


If you'd just step off your high horse you might actually learn 
something instead of just pissing people off.

Uli

[toc] | [prev] | [next] | [standalone]

#18708

In article <mailman.4488.1325870461.27778.python-list@python.org>,
 Ian Kelly <ian.g.kelly@gmail.com> wrote:

> Randomizing the order is not a bad idea, but you also need to be able
> to run the tests in a consistent order, from a specific random seed.
> In the real world, test conflicts and dependencies do happen, and if
> we observe a failure, make a change, rerun the tests and observe
> success, we need to be able to be sure that we actually fixed the bug,
> and that it didn't pass only because it was run in a different order.

I've seen this argument play out multiple times on this group.  
Executive summary:

OP: "I want to do X"

Peanut Gallery: "You're not supposed to do that"

Here's my commentary on that.

The classic unittest philosophy says that tests should be independent of 
each other, which means they should be able to be run in arbitrary 
order.  Some people advocate that the test framework should 
intentionally randomize the order, to flush out inter-test dependencies 
that the author didn't realize existed (or intend).

Test independence is a good thing.  Many people don't understand this 
when writing tests, and inadvertently write tests that depend on each 
other.  I've worked with those systems.  They're a bear to debug.  
You've got some test suite that runs for 15 minutes and dies at case 37 
of 43.  If you try to run case 37 by itself, it won't run, and you can't 
figure out what state cases 1-36 were supposed to leave the system in to 
make 37 work.  You could sink days or weeks into debugging this kind of 
crap.  BTDT.

That being said, the unittest module, while designed to support the "all 
tests must be independent" philosophy, is a useful piece of software for 
lots of things.  It provides an easy to use framework for writing tests, 
lots of convenient assertions, reporting, test discovery, etc, etc.

If somebody (i.e. the classic "consenting adult" of the Python world) 
wants to take advantage of that to write a test suite where the tests 
*do* depend on each other, and have to be run in a certain order, 
there's nothing wrong with that.  As long as they understand the 
consequences of their actions, don't try to preach unittest religion to 
them.  They're in the best position to know if what they're trying to do 
is the best thing for their particular situation.

[toc] | [prev] | [next] | [standalone]

#18709

On 2012-01-09, Roy Smith <roy@panix.com> wrote:
> If somebody (i.e. the classic "consenting adult" of the Python
> world) wants to take advantage of that to write a test suite
> where the tests *do* depend on each other, and have to be run
> in a certain order, there's nothing wrong with that.  As long
> as they understand the consequences of their actions, don't try
> to preach unittest religion to them.  They're in the best
> position to know if what they're trying to do is the best thing
> for their particular situation.

If a question springs from an idea that is usually a bad
practice, then it should be challenged. The possible broken-nose
of a questioner is a small price to pay for the education of the
peanut gallery.

If a questioner does not wish to defend what they are doing, he
or she has that right, of course.

-- 
Neil Cerutti

[toc] | [prev] | [next] | [standalone]

#18710

Am 09.01.2012 15:35, schrieb Roy Smith:
> The classic unittest philosophy says that tests should be independent of
> each other, which means they should be able to be run in arbitrary
> order.  Some people advocate that the test framework should
> intentionally randomize the order, to flush out inter-test dependencies
> that the author didn't realize existed (or intend).

While I agree with the idea, I'd like to add that independence is an 
illusion. You already have possible dependencies if you run tests in the 
same process/OS-installation/computer/parallel universe. If you now 
happen to influence one test with another and the next run randomizes 
the tests differently, you will never see the fault again. Without this 
reproducability, you don't gain anything but the bad stomach feeling 
that something is wrong.


> Test independence is a good thing.  Many people don't understand this
> when writing tests, and inadvertently write tests that depend on each
> other.  I've worked with those systems.  They're a bear to debug.
> You've got some test suite that runs for 15 minutes and dies at case 37
> of 43.  If you try to run case 37 by itself, it won't run, and you can't
> figure out what state cases 1-36 were supposed to leave the system in to
> make 37 work.  You could sink days or weeks into debugging this kind of
> crap.  BTDT.

I'm sorry to hear that, debugging other peoples' feces is never a task 
to wish for. That said, there are two kinds of dependencies and in at 
least this discussion there hasn't been any mentioning of the 
differences yet, but those differences are important.


Your unfortunate case is where test X creates persistent state that must 
be present in order for test X+1 to produce meaningful results. This 
kind of dependency obviously blows, as it means you can't debug test X+1 
separately. I'd call this operational dependency.

This kind of dependency is IMHO a bug in the tests themselves. The unit 
testing framework could help you find those bugs by allowing random 
order of execution for the test cases.


There is another dependency and that I'd call a logical dependency. This 
occurs when e.g. test X tests for an API presence and test Y tests the 
API behaviour. In other words, Y has no chance to succeed if X already 
failed. Unfortunately, there is no way to express this relation, there 
is no "@unittest.depends(test_X)" to decorate test_Y with (Not yet!). 
Each test would be the root of a tree of tests that it depends on. If a 
dependency fails already, you can either skip the tree or at least mark 
the following test failures as implicit failures, so that you can easily 
distinguish them from the root cause.

This kind of dependency is quite normal although not directly supported 
by the unittest module. As a workaround, being able to define an order 
allows you to move the dependencies further to the top, so they are 
tested first.


To sum it up, in order to catch operational dependencies, you need a 
random order while in order to clearly express logical dependencies in 
the output, you want a fixed order. Neither is the one true way, though 
my gut feeling is that the fixed order is overall more useful.


> As long as they understand the consequences of their actions, don't
> try to preach unittest religion to them.  They're in the best
> position to know if what they're trying to do is the best thing for
> their particular situation.

Amen!

Uli

[toc] | [prev] | [next] | [standalone]

#18711

On Mon, Jan 9, 2012 at 9:59 AM, Ulrich Eckhardt
<ulrich.eckhardt@dominolaser.com> wrote:
> There is another dependency and that I'd call a logical dependency. This
> occurs when e.g. test X tests for an API presence and test Y tests the API
> behaviour. In other words, Y has no chance to succeed if X already failed.
> Unfortunately, there is no way to express this relation, there is no
> "@unittest.depends(test_X)" to decorate test_Y with (Not yet!). Each test
> would be the root of a tree of tests that it depends on. If a dependency
> fails already, you can either skip the tree or at least mark the following
> test failures as implicit failures, so that you can easily distinguish them
> from the root cause.

I can see where that could be useful.  On the other hand, if such a
decorator were included in unittest, I can already envision people
abusing it to explicitly enshrine their operational dependencies,
maybe even taking it as encouragement to write their tests in that
fashion.

[toc] | [prev] | [next] | [standalone]

#18725

In article <11jrt8-l32.ln1@satorlaser.homedns.org>,
 Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> wrote:

> > Some people advocate that the test framework should
> > intentionally randomize the order, to flush out inter-test dependencies
> > that the author didn't realize existed (or intend).
> 
> If you now 
> happen to influence one test with another and the next run randomizes 
> the tests differently, you will never see the fault again. Without this 
> reproducability, you don't gain anything but the bad stomach feeling 
> that something is wrong.

The standard solution to that is to print out the PRNG initialization 
state and provide a way in your test harness to re-initialize it to that 
state.  I've done things like that in test scenarios where it is 
difficult or impossible to cover the problem space deterministically.

> Your unfortunate case is where test X creates persistent state that must 
> be present in order for test X+1 to produce meaningful results. This 
> kind of dependency obviously blows, as it means you can't debug test X+1 
> separately. I'd call this operational dependency.
> 
> This kind of dependency is IMHO a bug in the tests themselves.

For the most part, I'm inclined to agree.  However, there are scenarios 
where having each test build the required state from scratch is 
prohibitively expensive.  Imagine if you worked at NASA wanted to run 
test_booster_ignition(), test_booster_cutoff(), 
test_second_stage_ignition(), and test_self_destruct().  I suppose you 
could run them in random order, but you'd use up a lot of rockets that 
way.

Somewhat more seriously, let's say you wanted to do test queries against 
a database with 100 million records in it.  You could rebuild the 
database from scratch for each test, but doing so might take hours per 
test.  Sometimes, real life is just *so* inconvenient.

> There is another dependency and that I'd call a logical dependency. This 
> occurs when e.g. test X tests for an API presence and test Y tests the 
> API behaviour. In other words, Y has no chance to succeed if X already 
> failed.

Sure.  I run into that all the time.  A trivial example would be the 
project I'm working on now.  I've come to realize that a long unbroken 
string of E's means, "Dummy, you forgot to bring the application server 
up before you ran the tests".  It would be nicer if the test suite could 
have run a single test which proved it could create a TCP connection and 
when that failed, just stop.

[toc] | [prev] | [next] | [standalone]

#18730

On 1/9/2012 8:05 PM, Roy Smith wrote:
> In article<11jrt8-l32.ln1@satorlaser.homedns.org>,
>   Ulrich Eckhardt<ulrich.eckhardt@dominolaser.com>  wrote:
>
>>> Some people advocate that the test framework should
>>> intentionally randomize the order, to flush out inter-test dependencies
>>> that the author didn't realize existed (or intend).
>>
>> If you now
>> happen to influence one test with another and the next run randomizes
>> the tests differently, you will never see the fault again. Without this
>> reproducability, you don't gain anything but the bad stomach feeling
>> that something is wrong.
>
> The standard solution to that is to print out the PRNG initialization
> state and provide a way in your test harness to re-initialize it to that
> state.  I've done things like that in test scenarios where it is
> difficult or impossible to cover the problem space deterministically.
>
>> Your unfortunate case is where test X creates persistent state that must
>> be present in order for test X+1 to produce meaningful results. This
>> kind of dependency obviously blows, as it means you can't debug test X+1
>> separately. I'd call this operational dependency.
>>
>> This kind of dependency is IMHO a bug in the tests themselves.
>
> For the most part, I'm inclined to agree.  However, there are scenarios
> where having each test build the required state from scratch is
> prohibitively expensive.  Imagine if you worked at NASA wanted to run
> test_booster_ignition(), test_booster_cutoff(),
> test_second_stage_ignition(), and test_self_destruct().  I suppose you
> could run them in random order, but you'd use up a lot of rockets that
> way.
>
> Somewhat more seriously, let's say you wanted to do test queries against
> a database with 100 million records in it.  You could rebuild the
> database from scratch for each test, but doing so might take hours per
> test.  Sometimes, real life is just *so* inconvenient.
>
>> There is another dependency and that I'd call a logical dependency. This
>> occurs when e.g. test X tests for an API presence and test Y tests the
>> API behaviour. In other words, Y has no chance to succeed if X already
>> failed.
>
> Sure.  I run into that all the time.  A trivial example would be the
> project I'm working on now.  I've come to realize that a long unbroken
> string of E's means, "Dummy, you forgot to bring the application server
> up before you ran the tests".  It would be nicer if the test suite could
> have run a single test which proved it could create a TCP connection and
> when that failed, just stop.

Many test cases in the Python test suite have multiple asserts. I 
believe both resource sharing and sequential dependencies are reasons. I 
consider 'one assert (test) per testcase' to be on a par with 'one class 
per file'.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web