Re: Regression testing for pointers

From	nick_keighley_nospam@hotmail.com
Newsgroups	comp.lang.c
Subject	Re: Regression testing for pointers
Date	2012-03-16 03:05 -0700
Organization	http://groups.google.com
Message-ID	<16697131.758.1331892337093.JavaMail.geo-discussion-forums@ynbq18> (permalink)
References	(16 earlier) <jjnvii$e7d$1@speranza.aioe.org> <9s9hpbF8igU1@mid.individual.net> <jjpmn4$9j0$1@speranza.aioe.org> <20559117.329.1331816158025.JavaMail.geo-discussion-forums@vbhy1> <jjtcje$t89$1@speranza.aioe.org>

Show all headers | View raw

On Thursday, March 15, 2012 6:32:16 PM UTC, Don Y wrote:
> On 3/15/2012 5:55 AM, nick_keighley_nospam@hotmail.com wrote:
> > On Wednesday, March 14, 2012 9:00:18 AM UTC, Don Y wrote:


> >>>>> I still think you are testing at too high a level, test the functions
> >>>>> that manipulate your internal independently.
> >>>>
> >>>> There *is* nothing else. the allocator and the deallocator
> >>>> are the *only* things that massage the internal state of the
> >>>> heap!
> >>>
> >>> The you should extract some of the internal functionality and test it.
> >>
> >> I can essentially do this by examining the results of
> >> successive invocations designed to expose aspects of that
> >> state.  (see other post to Malcolm, this date)
> >
> > seems like doing things the hard way. *And* it's error prone. You
> > need white-box knowledge to test your function thoughourly- so why
> > not make it explicit in your test harness?
> 
> Doing so means you have now determined the *implementation*
> as well as the functionality.

no. You've tested the implementation as well as the functionality.

> If you want to split out "sub-functions", do so.  Then,
> formally specify how they must work.  And, formally
> specify a test suite for them.  Thereafter, the implementation
> of the allocator must preserve all of these interfaces.

no. A particular implementation has particualar internal tests. 
Perhaps you provide a call internalDataValid() the high level test 
need never know what the internal details of that are.

> The whole point of regression testing is to formalize the
> test cases and ensure changes to a function preserve its
> interface semantics and functionality.

that's your definition. To me regression tests are to test you 
haven't broken something with your change. A purely functional 
test might not check all the boundaries and special cases.

> I write a CORDIC sqrt.  I peek inside and notice that iteration
> X for an input of FOO yields -- and *should* yield -- an
> approximation having the value BAZ.
> 
> You rewrite the sqrt using Newton-Rhapson.  Suddenly your test
> fails because it converges on the result differently.

so testing floating point is difficult.

> Or, maybe my peek-inside test looks at some parameter of
> the CORDIC implementation that simply doesn't EXIST in
> the Newton-Rhapson version.
> 
> You can *test* your code by single stepping through its
> execution or "whatever".  But *formalizing* that portion
> of the test process is folly.

I'm not sure sqrt() is a really good example. I never suggested 
stepping-through-the-code was a sensible validation step.

> > Only-testing-the-public-interface can be taken to ridiculous
> > extremes. Do they deduce the internal behaviour of air-traffic
> > control systems by flying planes at them at different angles?
> 
> An air traffic control system will have *numerous* layers
> of FORMAL SPECIFICATION below "system test/specification".
> All of those lower "interfaces" are tested.  But, they
> are also points of inflexibility.  Any change to any of them
> can result in changes to the entire *system* above and below
> it in the abstraction hierarchy.
> 
> "Let's make malloc(3) return a pointer to a struct that contains
> a pointer to the allocated memory and the size of that memory".
> 
> Suddenly, everything that uses malloc changes.

quite. You've changed the public interface.

>  (let's make
> sqrt rely on successive iterations of a CORDIC algorithm;
> you are free to change the implementation of the CORDIC
> algorithm -- but, you have to ensure that the results it
> produces agree with those that are being tested for)

why?

> If you FORMALIZE test suites then you formalize interfaces.

why? Why can't you write internal tests? You can call them "formal" if you like. Ok. 

    Step 1 "run internalTest() verify that it returns true"
    Step 2 "run first functional test..." 

> If that's what you want or need, then do so FORMALLY.  Make
> sure there is a formal document that contractually specifies
> how the interface is expected to function so that:
> - implementors know how to implement the function described
> - consumers know how to *use* that function
> - validation suites know how to exercise that function

by why does that preclude any other form of testing?

> If, OTOH, you don't want to go to all this trouble and tie
> your future hands, implement whatever ad hoc testing YOU
> want and keep it on a scrap of paper in your desk drawer
> since it is not a formal part of the specification for the
> function/module you are testing!

I find this very strange. Why can't I run internal tests that 
verify correct internal behaviour? These test are not "ad hoc",
they are not kept in the bottom left hand drawer. They can be reviewed,
configuration controlled, and results can be recorded.

It vastly increases confidence in the product over "just run the SAT" .
I've *seen* systems taht passed their SAT and then exhibited field bugs 
that good module testing would have detected. It also makes debugging 
easier...

> > To use the over-stretched Civil Engineering analogy. People who
> > build bridges build them out of known good components which they
> > assemble into known good sub-systems that they in turn assemble
> > into complete bridges or whatever.
> 
> And each of those *components* is formally specified, tested
> and "vouched for" (by the manufacturer).

yes, but not by the customer of the bridge. He probably writes in the 
contract "shall be constructed in accordance with best industry 
practice" or some such boiler plate. The contract will then specify 
its load capacity and how many days a year it will close due to high 
winds. etc. They don't actually care what grade of steel is used as
long as someone ensures that the right one was used.

> So a *lawyer* and
> set of "expert witnesses" can testify that the reason the
> bridge failed was because ACME Asphalt sold defective road
> coating to the builder which didn't meet the specifications
> that the supplier FORMALLY AND LEGALLY AGREED TO in his
> CONTRACT withe the builder.
> 
> You can't go and complain that ACME used *pink* asphalt (while
> you EXPECTED black) -- unless it fails to meet some specified
> criteria in that contract.  Or, that ACME heated the raw material
> using a wood fire instead of a gas oven in preparing the road
> covering (unless the specification says "shall be heated in
> a natural gas fired oven")
> 
> ACME has liberties as to how it approaches each specification
> to which it has *contractually* agreed to be bound.
> 
> > Somwething complex like a memory allocator needs to be broken
> > into chunks and/or expose much of its internals to facilitate testing.
> 
> As I said, *you* can look at individual BITS in the CPU, memory,
> etc. while *you* are testing it.  But, the function formally
> only has ONE interface that it must contractually satisfy.


we seem to have acheived stalemate by repetition. Though "sanity is not statistical" Iwill point out there seem to be two of us that don't agree with you.

Happy programming!

Thread

Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-09 16:43 -0700
  Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 12:56 +1300
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-09 19:40 -0700
      Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 17:53 +1300
        Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 18:12 +1300
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 02:05 -0700
          Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 23:17 +1300
            Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-10 12:00 +0000
              Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 11:00 +1300
                Re: Regression testing for pointers Ike Naar <ike@iceland.freeshell.org> - 2012-03-10 22:23 +0000
                Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-10 22:40 +0000
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 12:12 +1300
                Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-11 00:42 +0000
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 14:26 +1300
                Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-11 03:09 +0000
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 18:16 +1300
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 12:01 -0700
              Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 10:35 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 09:19 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 07:23 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 16:23 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 17:25 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 00:19 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 20:34 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 01:00 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 21:22 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 10:19 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 07:25 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 02:00 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 22:18 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 11:32 -0700
                Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-15 05:58 -0700
                Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-15 05:55 -0700
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-15 11:32 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-16 18:14 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-20 01:43 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-20 21:59 +1300
                Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-16 03:05 -0700
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-20 02:33 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-24 11:49 +1300
          Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-11 05:01 -0700
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 09:44 -0700
              Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-12 09:56 +1300
    Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-10 02:28 -0800
      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 11:09 -0700
      Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-10 18:29 +0000
        Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-11 05:22 -0700
          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 09:50 -0700
            Re: Regression testing for pointers "io_x" <a@b.c.invalid> - 2012-03-13 13:09 +0100
  Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-10 03:14 +0000
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 12:42 -0700
  Re: Regression testing for pointers Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-10 08:42 +0000
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 02:19 -0700
      Re: Regression testing for pointers "BartC" <bc@freeuk.com> - 2012-03-10 15:54 +0000
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 11:49 -0700
          Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-11 06:09 -0700
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 12:33 -0700
              Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-12 03:29 -0700
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 10:01 -0700
                Re: Regression testing for pointers "BartC" <bc@freeuk.com> - 2012-03-13 18:56 +0000
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 16:31 -0700
      Re: Regression testing for pointers Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-12 00:30 +0000
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 16:41 -0700
      Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-12 13:39 +1300
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 09:36 -0700
          Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 07:25 +1300
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 16:28 -0700
              Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-13 04:20 +0000
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 17:34 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 00:29 -0700
              Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 17:31 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 01:05 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 21:30 +1300
                Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 10:43 -0700
  Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-10 15:41 -0800
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 11:23 -0700
      Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-13 13:43 -0700
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 01:45 -0700
          Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-14 15:15 -0700
            Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-14 22:57 +0000
              Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-15 13:10 -0700
                Re: Regression testing for pointers jgk@panix.com (Joe keane) - 2012-03-16 22:20 +0000
                Re: Regression testing for pointers "io_x" <a@b.c.invalid> - 2012-03-18 07:15 +0100
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 23:13 -0700
  Re: Regression testing for pointers gordonb.b29ri@burditt.org (Gordon Burditt) - 2012-03-12 19:37 -0500
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 19:23 -0700
  Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 13:12 +1300
    Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 01:54 -0700
      Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 22:05 +1300
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 11:54 -0700

csiph-web