Path: csiph.com!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail From: Don Y Newsgroups: comp.arch.embedded Subject: Re: Unit Testing: from theory to real case Date: Fri, 30 Aug 2024 05:21:12 -0700 Organization: A noiseless patient Spider Lines: 177 Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Date: Fri, 30 Aug 2024 14:21:22 +0200 (CEST) Injection-Info: dont-email.me; posting-host="abd3e25f7ccb4ae2d2d1b094a7f46280"; logging-data="513153"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19bmK9UYW7Tuevmg9D0cRsG" User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.2.2 Cancel-Lock: sha1:ZZYPgUeNzZXDFa7/50S0B/LDRDM= In-Reply-To: Content-Language: en-US Xref: csiph.com comp.arch.embedded:32210 On 8/30/2024 1:18 AM, pozz wrote: > When you write, test for this, test for that, what happens if the client uses > the module in a wrong way, what happens when the system clock changes a little > or a big, and when the task missed the exact timestamp of an event? > > I was trying to write tests for *all* of those situations, but it seemed to me > a very, VERY, *VERY* big job. The implementation of the calendar module took me > a couple of days, tests seem an infinite job. Because there are lots of ways your code can fail. You have to prove that it doesn't fail in ANY of those ways. uint multiply(multiplicand, multiplier) { return(6) } works well for the test cases: 2,3 3,2 6,1 1,6 but not so well for: 8,5 17,902 1,1 etc. > I have four types of events, for each test I should check the correct behaviour > for each type. > > What happen if the timestamp of an event was already expired when it is added > to the system? I should write 4 tests, one for each type. > > AddOneShotEventWithExpiredTimestamp_NoActions > AddWeeklyEventWithExpiredTimestamp_NoActions > AddMonthlyEventWithExpiredTimestamp_NoActions > AddYearlyEventWithExpiredTimestamp_NoActions Chances are, there is one place in your code that is aware of the fact that the event is scheduled for a PAST time. So, you only need to create one test. (actually, two -- one that proves one behavior for time *almost* NOT past and another for time JUST past) Your goal (having already implemented the modules) is to exercise each path through the code. whatever() { ... if (x > y) { // do something } else { // do something else } ... } Here, there are only two different paths through the code: - one for x > y - one for !(x > y) So, you need to create test cases that will exercise each path. To verify your "x > y" test, you would want to pick an x that is just detectably larger than y. And, another case where x is as large as possible WITHOUT exceeding y. You can view this as defining the "edge" between the two routes. If, for example, you picked x = 5 and x = 3 as your test cases (where y = 4), then you WOULD exercise both paths. But, if you had mistakenly coded this as if (x >= y) { // do something } else { // do something else } you wouldn't be able to detect that fault, whereas using x = 5 and x = 4 would cause you to wonder why "do something else" never got executed! > What does it mean "expired timestamp"? Suppose the event timestamp is A time that is "in the past". If it is time 't', now, what happens if the client specifies an event to happen at time t-1? Should you *immediately* activate the event (because NOW it is t > t-1?) Or, should you discard it because it was SUPPOSED to happen 1 second ago? What if t-495678? Is there a different type of action you expect if the time is "a long time ago" vs. "just recently"? Do events happen at *instants* in time? Or, in CONDITIONS of time? If they happen at instants, then you have to ensure you can discern one instant from another. > "01/01/2024 10:00:00". This timestamp could be expired for a few seconds, a few > minutes or one day or months or years. Maybe the module performs well when the > system time has a different date, but bad if the timestamp expired in the same > day, for example "01/01/2024 11:00:00" or "01/01/2024 10:00:01". > Should I add: > > AddOneShotEventWithExpiredTimestamp1s_NoActions > AddOneShotEventWithExpiredTimestamp1m_NoActions > AddOneShotEventWithExpiredTimestamp1h_NoActions > AddOneShotEventWithExpiredTimestamp1d_NoActions > AddWeeklyEventWithExpiredTimestamp1s_NoActions > AddWeeklyEventWithExpiredTimestamp1m_NoActions > AddWeeklyEventWithExpiredTimestamp1h_NoActions > AddWeeklyEventWithExpiredTimestamp1d_NoActions > AddMonthlyEventWithExpiredTimestamp1s_NoActions > AddMonthlyEventWithExpiredTimestamp1m_NoActions > AddMonthlyEventWithExpiredTimestamp1h_NoActions > AddMonthlyEventWithExpiredTimestamp1d_NoActions > AddYearlyEventWithExpiredTimestamp1s_NoActions > AddYearlyEventWithExpiredTimestamp1m_NoActions > AddYearlyEventWithExpiredTimestamp1h_NoActions > AddYearlyEventWithExpiredTimestamp1d_NoActions > > They are 16 tests for just a single stupid scenario. If I continue this way, I > will thousands of tests. I don't think this is the way to make testing, do I? You declare what scenario you are testing for as a (commentary) preface to the test stanza. If you are testing to ensure "NoActions" is handled correctly, then you look to see how many ways the "NoActions" criteria can tickle the code. If there is only ONE place where "NoActions" alters the flow through the code, then you only need one test (actually, two as you need to cover "SomeAction" to show that "NoAction" is different). In a different test scenaario, you would test that 1s, 1m, 1h, 1d, etc. are all handled correctly IF EACH OF THOSE PASSED THROUGH YOUR CODE OVER DIFFERENT PATHWAYS. And, elsewhere, you might test to see that "repeated" events operate correctly. You "prove" that one scenario is handled correctly and then don't need to reexamine those various tests again in any other scenario UNLESS THEY ALTER THE PATH THROUGH THE CODE. Your understanding of how the code would LIKELY be crafted lets you determine some of these tests before you've written ANY code. E.g., I suggested "expired events" because I am reasonably sure that SOMEWHERE your code is looking at "event time" vs. "now" so you would need to test that comparison. Your knowledge of how the code is *actually* crafted lets you refine your test cases to cover specifics of YOUR implementation. Note that test cases that are applied to version 1 of the code should yield the same results in version 305, even if the implementation changes dramatically. Because the FUNCTIONALITY shouldn't be changing. So, you can just keep adding test cases to your test suite; you don't ever need to remove any. [If a test STOPS working, you have to ask yourself how you have BROKEN/changed the function of the module] For example, I could implement a "long" integer multiplication routine by adding the multiplicand to an accumulator a number of times dictated by the multiplier. I can create test cases for this implementation. Later, I could revise the routine to use a shift-and-add approach. BUT, THE ORIGINAL TEST CASES SHOULD STILL PASS! I might, however, have to add some other tests to identify failures in the shift logic in this new approach (e.g., if I only examined the rightmost 26 of the bits in the multiplier, then a large value multiplier would fail in this approach but could pass in the repeated addition implementation). This would be evident to me in looking at the code because there would be a different path through the code when the "last bit" had been checked.