Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #18898

Re: Regression testing for pointers

Show all headers | View raw

Hi Nick,

>>>>>>> I still think you are testing at too high a level, test the functions
>>>>>>> that manipulate your internal independently.
>>>>>>
>>>>>> There *is* nothing else. the allocator and the deallocator
>>>>>> are the *only* things that massage the internal state of the
>>>>>> heap!
>>>>>
>>>>> The you should extract some of the internal functionality and test it.
>>>>
>>>> I can essentially do this by examining the results of
>>>> successive invocations designed to expose aspects of that
>>>> state.  (see other post to Malcolm, this date)
>>>
>>> seems like doing things the hard way. *And* it's error prone. You
>>> need white-box knowledge to test your function thoughourly- so why
>>> not make it explicit in your test harness?
>>
>> Doing so means you have now determined the *implementation*
>> as well as the functionality.
>
> no. You've tested the implementation as well as the functionality.

If you peek inside AN IMPLEMENTATION and test expecting
some internal aspect of the implementation to comply with
some expectations under <some_set> of conditions, then
you have defined a new interface.

You have to formalize that interface -- otherwise, you can't
KNOW what to expect in any given situation.  You can't *know* what
constitutes "correct" results at that internal level.

>> If you want to split out "sub-functions", do so.  Then,
>> formally specify how they must work.  And, formally
>> specify a test suite for them.  Thereafter, the implementation
>> of the allocator must preserve all of these interfaces.
>
> no. A particular implementation has particualar internal tests.
> Perhaps you provide a call internalDataValid() the high level test
> need never know what the internal details of that are.

How is internalDataValid() defined?  You have now exposed
another interface.  How do I know what that is supposed to
do?  How does SOMEONE ELSE looking at the test suite
assure themselves that "internalDataValid()" is being
tested properly?

>> The whole point of regression testing is to formalize the
>> test cases and ensure changes to a function preserve its
>> interface semantics and functionality.
>
> that's your definition. To me regression tests are to test you
> haven't broken something with your change. A purely functional
> test might not check all the boundaries and special cases.

Then your functional test sucks!

What happens when your test suite reports:
    test string is located at: 0xFFFFFFFC
    the first 4 locations of the string are:  'A' 'B' 'C' 'D'
    the FUT (strlen()) returns a result of: ______
Do you *know* what the strlen() that you probably use *often*
will do in this case?  Will the above test suite SIGSEGV before
it reports any of this??

Or, is your test suite "lame":
    Please type a string, followed by CRLF:  ABCDEFG
    According to strlen(), you typed 7 characters.
    Please type a string, followed by CRLF:

Do you have any faith in how it handles strings with, for example,
32767 characters?  32769??

>> I write a CORDIC sqrt.  I peek inside and notice that iteration
>> X for an input of FOO yields -- and *should* yield -- an
>> approximation having the value BAZ.
>>
>> You rewrite the sqrt using Newton-Rhapson.  Suddenly your test
>> fails because it converges on the result differently.
>
> so testing floating point is difficult.

No.  CORDIC converges on the result differently than a FP
N-R approximation.

Imagine I have an infinitely large table in which all of the values
of sqrt() have been precomputed so sqrt just does a "lookup"
operation.  Obviously, testing it's internals would be very
different from testing one that performs a successive approximation.

>> Or, maybe my peek-inside test looks at some parameter of
>> the CORDIC implementation that simply doesn't EXIST in
>> the Newton-Rhapson version.
>>
>> You can *test* your code by single stepping through its
>> execution or "whatever".  But *formalizing* that portion
>> of the test process is folly.
>
> I'm not sure sqrt() is a really good example. I never suggested
> stepping-through-the-code was a sensible validation step.
>
>>> Only-testing-the-public-interface can be taken to ridiculous
>>> extremes. Do they deduce the internal behaviour of air-traffic
>>> control systems by flying planes at them at different angles?
>>
>> An air traffic control system will have *numerous* layers
>> of FORMAL SPECIFICATION below "system test/specification".
>> All of those lower "interfaces" are tested.  But, they
>> are also points of inflexibility.  Any change to any of them
>> can result in changes to the entire *system* above and below
>> it in the abstraction hierarchy.
>>
>> "Let's make malloc(3) return a pointer to a struct that contains
>> a pointer to the allocated memory and the size of that memory".
>>
>> Suddenly, everything that uses malloc changes.
>
> quite. You've changed the public interface.

Exactly.  Now, imagine changing some *internal* interface of
malloc that is not visible to the user.  YOUR test suite examines
this internal interface.  But, its changed.  Suddenly, your
tests don't work -- even though the rest of the malloc
implementation has been "fixed" so that it still satisfies
IT'S public interface.

>>   (let's make
>> sqrt rely on successive iterations of a CORDIC algorithm;
>> you are free to change the implementation of the CORDIC
>> algorithm -- but, you have to ensure that the results it
>> produces agree with those that are being tested for)
>
> why?

Because YOU want to test this internal interface.  You've
documented it so that you can convince a validation
engineer that your test suite for that INTERNAL interface
functions properly.  Even though I have just changed
the implementation of the function itself (while preserving
*its* public interface SO IT STILL PASSES *IT'S* TEST SUITE)

>> If you FORMALIZE test suites then you formalize interfaces.
>
> why? Why can't you write internal tests? You can call them "formal" if you like. Ok.
>
>      Step 1 "run internalTest() verify that it returns true"
>      Step 2 "run first functional test..."

What is "internalTest()" for strlen()?  for malloc()?  for sqrt()?
A function with *two* entry points??

>> If that's what you want or need, then do so FORMALLY.  Make
>> sure there is a formal document that contractually specifies
>> how the interface is expected to function so that:
>> - implementors know how to implement the function described
>> - consumers know how to *use* that function
>> - validation suites know how to exercise that function
>
> by why does that preclude any other form of testing?

Testing is the process of ensuring an implementation meets
its published goals/criteria.  Anything you want to test
has to be formally specified -- else how do we agree that
your test really *does* validate proper operation?

>> If, OTOH, you don't want to go to all this trouble and tie
>> your future hands, implement whatever ad hoc testing YOU
>> want and keep it on a scrap of paper in your desk drawer
>> since it is not a formal part of the specification for the
>> function/module you are testing!
>
> I find this very strange. Why can't I run internal tests that
> verify correct internal behaviour? These test are not "ad hoc",
> they are not kept in the bottom left hand drawer. They can be reviewed,
> configuration controlled, and results can be recorded.

Are you REQUIRING the function to pass those tests?
If YOU quit and I am assigned the job of finishing your
work, do *I* have to use the same implementation that
you have chosen?  Does the code I implement have to pass
those *internal* tests that you settled on?  How do I
know what they should yield?  *Why* should they yield those
results?  What makes your implementation better than mine?
If I pass a *thorough* set of tests for the published
formal interface, why should I also have to pass your
particular set of internal tests?

I saw an implementation of production software that determined
the size of a file by reading the file and *counting* the
individual bytes.  Sure, the result was correct.  But a foolish
implementation.

How would you test this function's internals -- look at the
count at random points in time and verify that they were
constantly increasing?  Have the loop emit the current
value of the counter periodically (e.g., every 1000 bytes)?

What happens if I rewrite the function and just use fstat()
and return the st_size member of the result?  Your test
(for a 4567 byte file) would expect to see:
   1000
   2000
   3000
   4000
   4567
for example.  *My* implementation would simply produce the
final "4567" result.  It has *failed* your "internal test"
because the implementation didn't rely on an iterative
counter approach.

Yet, it *passes* the test for the function itself!

> It vastly increases confidence in the product over "just run the SAT" .
> I've *seen* systems taht passed their SAT and then exhibited field bugs
> that good module testing would have detected. It also makes debugging
> easier...

If you passed a formal test and failed in the field, then your
formal test sucks.  You clearly aren't testing all of the
conditions that your code is *supposed* to be able to tolerate.

Debugging and testing DURING DEVELOPMENT are different issues.
That's the "desk drawer" comment I made.  You can't *force*
someone else (i.e., The Organization) to embrace your
particular implementation without formally nailing down a
contract for that interface that you are exposing.

>>> To use the over-stretched Civil Engineering analogy. People who
>>> build bridges build them out of known good components which they
>>> assemble into known good sub-systems that they in turn assemble
>>> into complete bridges or whatever.
>>
>> And each of those *components* is formally specified, tested
>> and "vouched for" (by the manufacturer).
>
> yes, but not by the customer of the bridge. He probably writes in the
> contract "shall be constructed in accordance with best industry
> practice" or some such boiler plate. The contract will then specify
> its load capacity and how many days a year it will close due to high
> winds. etc. They don't actually care what grade of steel is used as
> long as someone ensures that the right one was used.

----------^^^^^^^^^^^^^^^^^^^^^^^^^AAAAAAAAA^^^^^^^^^^

A contract, *somewhere* specifies what that "right one" is
and who that "someone" will be.  When the bridge fails,
you can bet these items will be readily identifiable (as
the lawyers sort out who to go after for the losses).

You can create whatever sorts of contracts you (and your
organization) want.  You can hide whatever you want, too.
But, everything that you expose *for* testing ties down
another corner of HOW you implement something -- as well
as how you can reimplement it in the future.

E.g., the is___() character functions are often implemented
as table lookups.  The argument to the function is used
to access a const table whose contents represent the
"classifications" for the "character" corresponding to the
argument.

So, table['c'] encodes information that tells me that 'c'
is alphabetic, NOT numeric, NOT whitespace, NOT punctuation,
graphic, NOT a control character, etc.

*BUT*, table[] and its contents are not formally exposed
in the specifications for any of these functions!  E.g.,
I can reimplement isdigit() to NOT use that table[] but,
instead, examine the argument directly, algebraically.

OTOH, if you expose table[] and try to test it as an
"internal test", you now have to formally define it
*and* mandate its role in the is___() functions.

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-09 16:43 -0700
  Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 12:56 +1300
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-09 19:40 -0700
      Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 17:53 +1300
        Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 18:12 +1300
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 02:05 -0700
          Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 23:17 +1300
            Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-10 12:00 +0000
              Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 11:00 +1300
                Re: Regression testing for pointers Ike Naar <ike@iceland.freeshell.org> - 2012-03-10 22:23 +0000
                Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-10 22:40 +0000
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 12:12 +1300
                Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-11 00:42 +0000
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 14:26 +1300
                Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-11 03:09 +0000
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 18:16 +1300
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 12:01 -0700
              Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 10:35 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 09:19 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 07:23 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 16:23 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 17:25 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 00:19 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 20:34 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 01:00 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 21:22 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 10:19 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 07:25 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 02:00 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 22:18 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 11:32 -0700
                Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-15 05:58 -0700
                Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-15 05:55 -0700
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-15 11:32 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-16 18:14 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-20 01:43 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-20 21:59 +1300
                Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-16 03:05 -0700
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-20 02:33 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-24 11:49 +1300
          Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-11 05:01 -0700
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 09:44 -0700
              Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-12 09:56 +1300
    Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-10 02:28 -0800
      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 11:09 -0700
      Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-10 18:29 +0000
        Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-11 05:22 -0700
          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 09:50 -0700
            Re: Regression testing for pointers "io_x" <a@b.c.invalid> - 2012-03-13 13:09 +0100
  Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-10 03:14 +0000
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 12:42 -0700
  Re: Regression testing for pointers Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-10 08:42 +0000
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 02:19 -0700
      Re: Regression testing for pointers "BartC" <bc@freeuk.com> - 2012-03-10 15:54 +0000
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 11:49 -0700
          Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-11 06:09 -0700
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 12:33 -0700
              Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-12 03:29 -0700
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 10:01 -0700
                Re: Regression testing for pointers "BartC" <bc@freeuk.com> - 2012-03-13 18:56 +0000
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 16:31 -0700
      Re: Regression testing for pointers Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-12 00:30 +0000
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 16:41 -0700
      Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-12 13:39 +1300
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 09:36 -0700
          Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 07:25 +1300
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 16:28 -0700
              Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-13 04:20 +0000
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 17:34 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 00:29 -0700
              Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 17:31 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 01:05 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 21:30 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 10:43 -0700
  Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-10 15:41 -0800
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 11:23 -0700
      Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-13 13:43 -0700
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 01:45 -0700
          Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-14 15:15 -0700
            Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-14 22:57 +0000
              Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-15 13:10 -0700
                Re: Regression testing for pointers jgk@panix.com (Joe keane) - 2012-03-16 22:20 +0000
                Re: Regression testing for pointers "io_x" <a@b.c.invalid> - 2012-03-18 07:15 +0100
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 23:13 -0700
  Re: Regression testing for pointers gordonb.b29ri@burditt.org (Gordon Burditt) - 2012-03-12 19:37 -0500
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 19:23 -0700
  Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 13:12 +1300
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 01:54 -0700
      Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 22:05 +1300
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 11:54 -0700

csiph-web