Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #18420 > unrolled thread

Regression testing for pointers

Back to article view | Back to comp.lang.c

Contents

  Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-09 16:43 -0700
    Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 12:56 +1300
      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-09 19:40 -0700
        Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 17:53 +1300
          Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 18:12 +1300
          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 02:05 -0700
            Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-10 23:17 +1300
              Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-10 12:00 +0000
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 11:00 +1300
                  Re: Regression testing for pointers Ike Naar <ike@iceland.freeshell.org> - 2012-03-10 22:23 +0000
                  Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-10 22:40 +0000
                    Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 12:12 +1300
                      Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-11 00:42 +0000
                        Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 14:26 +1300
                          Re: Regression testing for pointers Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-11 03:09 +0000
                            Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 18:16 +1300
              Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 12:01 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-11 10:35 +1300
                  Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 09:19 -0700
                    Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 07:23 +1300
                      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 16:23 -0700
                        Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 17:25 +1300
                          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 00:19 -0700
                            Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 20:34 +1300
                              Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 01:00 -0700
                                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 21:22 +1300
                                  Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 10:19 -0700
                                    Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 07:25 +1300
                                      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 02:00 -0700
                                        Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 22:18 +1300
                                          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 11:32 -0700
                                          Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-15 05:58 -0700
                                        Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-15 05:55 -0700
                                          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-15 11:32 -0700
                                            Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-16 18:14 +1300
                                              Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-20 01:43 -0700
                                                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-20 21:59 +1300
                                            Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-16 03:05 -0700
                                              Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-20 02:33 -0700
                                                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-24 11:49 +1300
            Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-11 05:01 -0700
              Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 09:44 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-12 09:56 +1300
      Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-10 02:28 -0800
        Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 11:09 -0700
        Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-10 18:29 +0000
          Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-11 05:22 -0700
            Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 09:50 -0700
              Re: Regression testing for pointers "io_x" <a@b.c.invalid> - 2012-03-13 13:09 +0100
    Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-10 03:14 +0000
      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 12:42 -0700
    Re: Regression testing for pointers Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-10 08:42 +0000
      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 02:19 -0700
        Re: Regression testing for pointers "BartC" <bc@freeuk.com> - 2012-03-10 15:54 +0000
          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-10 11:49 -0700
            Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-11 06:09 -0700
              Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 12:33 -0700
                Re: Regression testing for pointers nick_keighley_nospam@hotmail.com - 2012-03-12 03:29 -0700
                  Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 10:01 -0700
                    Re: Regression testing for pointers "BartC" <bc@freeuk.com> - 2012-03-13 18:56 +0000
                      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 16:31 -0700
        Re: Regression testing for pointers Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-12 00:30 +0000
          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 16:41 -0700
        Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-12 13:39 +1300
          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 09:36 -0700
            Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 07:25 +1300
              Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 16:28 -0700
                Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-13 04:20 +0000
                  Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 17:34 +1300
                  Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 00:29 -0700
                Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 17:31 +1300
                  Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 01:05 -0700
                    Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-13 21:30 +1300
                      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-13 10:43 -0700
    Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-10 15:41 -0800
      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-11 11:23 -0700
        Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-13 13:43 -0700
          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 01:45 -0700
            Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-14 15:15 -0700
              Re: Regression testing for pointers Kaz Kylheku <kaz@kylheku.com> - 2012-03-14 22:57 +0000
                Re: Regression testing for pointers Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-15 13:10 -0700
                  Re: Regression testing for pointers jgk@panix.com (Joe keane) - 2012-03-16 22:20 +0000
                  Re: Regression testing for pointers "io_x" <a@b.c.invalid> - 2012-03-18 07:15 +0100
              Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 23:13 -0700
    Re: Regression testing for pointers gordonb.b29ri@burditt.org (Gordon Burditt) - 2012-03-12 19:37 -0500
      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-12 19:23 -0700
    Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 13:12 +1300
      Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 01:54 -0700
        Re: Regression testing for pointers Ian Collins <ian-news@hotmail.com> - 2012-03-14 22:05 +1300
          Re: Regression testing for pointers Don Y <this@isnotme.com> - 2012-03-14 11:54 -0700

Page 3 of 5 — ← Prev page 1 2 [3] 4 5  Next page →

#18527

On Saturday, March 10, 2012 9:05:36 AM UTC, Don Y wrote:
> On 3/9/2012 9:53 PM, Ian Collins wrote:


> >>>> There's rarely a problem generating regression tests for
> >>>> functions/procedures that use concrete *values*:
> >>>>
> >>>> And, it is easy to erect the necessary scaffolding for those
> >>>> sorts of tests without introducing any noticeable uncertainty
> >>>> in the veracity of the results obtained.
> >>>>
> >>>> But, I can't see a way to handle this sort of testing on
> >>>> functions that manipulate *pointers* to objects -- without
> >>>> adding a fair bit of code to the scaffolding that, itself,
> >>>> represents a testing issue! (i.e., does the scaffolding
> >>>> have any bugs that result in false positives or negatives??)
> >>>
> >>> As a general point, it a lot easier and more satisfying for the
> >>> programmer to write the tests before the code under test.
> >>
> >> But that's the whole point -- you can't create test cases
> >> independant of the execution environment!
> >
> > Of course you can. How else would you test embedded code for a device
> > without external communications?
> 
> For an embedded device, you *know* the environment you
> are operating in.  You *know* where the heap resides
> (actual addresses).  You know what the alignment
> restrictions of the processor are.  You know that a
> request for 0x61 bytes with the heap starting at 0x40000000
> having a length of 0x100 in a byte alignable environment
> allocating from the *end* of the free list (as opposed to
> the start) trimming the allocation unit to the requested
> size *should* give you a result of 0x61 bytes located
> at 0x4000009F.  You *then* know that an attempt to
> allocate 0x9E bytes will FAIL under the same calling
> conditions (because a 1 byte crumb will be left in the
> free list -- which "/* CAN'T HAPPEN */".
> 
> I write MyMalloc() and hand it to you WITH A TEST SUITE.
> You run it on <whatever> processor and rely on the test
> suite to prove to you that the code works as advertised.
> 
> Repeat the above example with a heap at 0x2000 (!!) of size
> 0x1000 and you get different results.
> 
> >> What should "malloc(500)" return? 0x1234? 0x12345678?
> >> NULL?
> >
> > What ever *your test harness'* malloc is told to return.
> 
> What "test harness malloc"?  There *is* no "test harness
> malloc!  Recall, I said "malloc and free *types* of functions"
> not "malloc and free" -- in particular, note the semantics of
> myfree() differ from free(3).
> 
> All I can compare against is myself -- and if I have a bug
> then the other instance will faithfully repeat the same
> bug.

if you need to test something as complex as malloc you're going to have to open the box. I'd write validation code for the internal data structures. Code that tested for over-laps in allocated data (probably not possible if the data structures are ok) etc. Then run various test running the validation after each alloc or dealloc. 

<snip>

> I'm going to give you some code to implement a MyMalloc
> (-like) function.  I'm not going to tell you where
> the heap resides in memory.  I'm not going to tell you
> if memory accesses must be byte/word/cache-line aligned.
> Heck, I'm not even going to tell you how *big* the heap
> is!
> 
> What are your test cases?  How do you know if they have
> succeeded or not?  I.e., just because MyMalloc *appears*
> to return a value, are you sure that it is the *correct*
> value?
> 
> Are you sure that the allocation strategy actually enacted
> was "smallest fit" and not "last fit" or "largest fit"?
> (my memory management system is highly parameterized)
> How do you know that the allocated block was *intentionally*
> aligned with whatever the processor wants -- instead of
> *coincidentally* happening to be aligned?  Are you sure
> that any "crumbs" left behind in the free list as a
> result of that alignment are recoverable?  (i.e., if
> you left a single byte behind, then you probably can never
> recover it since its not big enough to contain the list
> overhead necessary to tie it into the free list)

white box rather than black box testing.

[toc] | [prev] | [next] | [standalone]

#18538

Hi Nick,

On 3/11/2012 5:01 AM, nick_keighley_nospam@hotmail.com wrote:
> On Saturday, March 10, 2012 9:05:36 AM UTC, Don Y wrote:

>> I'm going to give you some code to implement a MyMalloc
>> (-like) function.  I'm not going to tell you where
>> the heap resides in memory.  I'm not going to tell you
>> if memory accesses must be byte/word/cache-line aligned.
>> Heck, I'm not even going to tell you how *big* the heap
>> is!
>>
>> What are your test cases?  How do you know if they have
>> succeeded or not?  I.e., just because MyMalloc *appears*
>> to return a value, are you sure that it is the *correct*
>> value?
>>
>> Are you sure that the allocation strategy actually enacted
>> was "smallest fit" and not "last fit" or "largest fit"?
>> (my memory management system is highly parameterized)
>> How do you know that the allocated block was *intentionally*
>> aligned with whatever the processor wants -- instead of
>> *coincidentally* happening to be aligned?  Are you sure
>> that any "crumbs" left behind in the free list as a
>> result of that alignment are recoverable?  (i.e., if
>> you left a single byte behind, then you probably can never
>> recover it since its not big enough to contain the list
>> overhead necessary to tie it into the free list)
>
> white box rather than black box testing.

That still doesn't address the problem.  You don't know,
a priori, what the test environment will be!  So, now you
have replaced "writing a function that can perform correctly
in that unknown environment" with "writing a function that
can TEST ANOTHER FUNCTION in that unknown environment".
You've just moved the issue to another piece of code.

[toc] | [prev] | [next] | [standalone]

#18549

On 03/12/12 05:44 AM, Don Y wrote:
> Hi Nick,
>
> On 3/11/2012 5:01 AM, nick_keighley_nospam@hotmail.com wrote:
>> On Saturday, March 10, 2012 9:05:36 AM UTC, Don Y wrote:
>
>>> I'm going to give you some code to implement a MyMalloc
>>> (-like) function.  I'm not going to tell you where
>>> the heap resides in memory.  I'm not going to tell you
>>> if memory accesses must be byte/word/cache-line aligned.
>>> Heck, I'm not even going to tell you how *big* the heap
>>> is!
>>>
>>> What are your test cases?  How do you know if they have
>>> succeeded or not?  I.e., just because MyMalloc *appears*
>>> to return a value, are you sure that it is the *correct*
>>> value?
>>>
>>> Are you sure that the allocation strategy actually enacted
>>> was "smallest fit" and not "last fit" or "largest fit"?
>>> (my memory management system is highly parameterized)
>>> How do you know that the allocated block was *intentionally*
>>> aligned with whatever the processor wants -- instead of
>>> *coincidentally* happening to be aligned?  Are you sure
>>> that any "crumbs" left behind in the free list as a
>>> result of that alignment are recoverable?  (i.e., if
>>> you left a single byte behind, then you probably can never
>>> recover it since its not big enough to contain the list
>>> overhead necessary to tie it into the free list)
>>
>> white box rather than black box testing.
>
> That still doesn't address the problem.  You don't know,
> a priori, what the test environment will be!  So, now you
> have replaced "writing a function that can perform correctly
> in that unknown environment" with "writing a function that
> can TEST ANOTHER FUNCTION in that unknown environment".
> You've just moved the issue to another piece of code.

Didn't my suggestion address those issues?  If not, what did I miss?

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#18442

On Friday, March 9, 2012 11:56:22 PM UTC, Ian Collins wrote:
> On 03/10/12 12:43 PM, Don Y 

<snip>

> > But, I can't see a way to handle this sort of testing on
> > functions that manipulate *pointers* to objects -- without
> > adding a fair bit of code to the scaffolding that, itself,
> > represents a testing issue!  (i.e., does the scaffolding
> > have any bugs that result in false positives or negatives??)

I think you have to draw the line somewhere. Testing test code leads you into a infinite regression.

> As a general point, it a lot easier and more satisfying for the 
> programmer to write the tests before the code under test.

I know this is the received wisdom in certain circles but I have trouble writing the test before I've at least specified the function's prototype. And certain tests aren't writable until you've seen the function's internals. For instance certain floating point calculations may have divide by zero possibilities which you only know about once you've seen the code.

<snip>

[toc] | [prev] | [next] | [standalone]

#18454

Hi Nick,

On 3/10/2012 3:28 AM, nick_keighley_nospam@hotmail.com wrote:

>>> But, I can't see a way to handle this sort of testing on
>>> functions that manipulate *pointers* to objects -- without
>>> adding a fair bit of code to the scaffolding that, itself,
>>> represents a testing issue!  (i.e., does the scaffolding
>>> have any bugs that result in false positives or negatives??)
>
> I think you have to draw the line somewhere. Testing test code
> leads you into a infinite regression.

Of course!  Hence the desire for your test scaffolding to
be "above suspicion".  The more "thinking" that it has to
do (to adapt to a test environment and particular test
results), then the more opportunities there are for errors
to creep into it.

This is especially the case where the scaffolding is "special"
for (one?) particular function.

>> As a general point, it a lot easier and more satisfying for the
>> programmer to write the tests before the code under test.
>
> I know this is the received wisdom in certain circles but I have
> trouble writing the test before I've at least specified the
> function's prototype. And certain tests aren't writable until
> you've seen the function's internals. For instance certain
> floating point calculations may have divide by zero possibilities
> which you only know about once you've seen the code.

I tend to keep a list of the types of things that I *see* as
potentially "going wrong" while I am writing the code.  At the
very least, I want my test cases to cover every path *through*
the code.  And, test the various boundary conditions that
the code makes obvious.

However, just *thinking* about how you might test something
brings up other cases that you might have to formally
specify in the contract.  E.g., what happens if you "free"
a block that has already been free-d?  (do you silently
ignore it?  or throw an assertion failure??)

I think it important to consider how functions will be used
and ABused in designing your tests.  E.g., I build memory
management functions with scads of invariants within that
verify the caller isn't trying to do something "wrong"
(because callers tend to screw up when it comes to using
memory management routines).  I'd rather have a routine lock
up during testing (where I can *see* it) than worry about
some latent bug in "production".

[toc] | [prev] | [next] | [standalone]

#18456

On 2012-03-10, nick_keighley_nospam@hotmail.com <nick_keighley_nospam@hotmail.com> wrote:
> On Friday, March 9, 2012 11:56:22 PM UTC, Ian Collins wrote:
>> On 03/10/12 12:43 PM, Don Y 
>
><snip>
>
>> > But, I can't see a way to handle this sort of testing on
>> > functions that manipulate *pointers* to objects -- without
>> > adding a fair bit of code to the scaffolding that, itself,
>> > represents a testing issue!  (i.e., does the scaffolding
>> > have any bugs that result in false positives or negatives??)
>
> I think you have to draw the line somewhere. Testing test code leads you into a infinite regression.

That is moot anyway because the point of a /regression/ test suite isn't to
find bugs or validate functionality. It's simply to validate that the
behavior hasn't changed between revisions (including buggy behavior!)

Thus an imperfect regression test suite can still find a regression. 

Some of the output of a regression test case could be due to a bug in the test
case.  If that output changes because of a regression in the software being
tested, the test case has done its job anyway.

The test suite shows (within the limitations of its coverage) that the new
revision of the software duplicates the old revision, not only
feature-for-feature, as well as bug-for-bug.

Sometimes when you fix a bug in the program, you have to update the test suite,
because it has test cases which confirm the buggy behavior.

>> As a general point, it a lot easier and more satisfying for the 
>> programmer to write the tests before the code under test.
>
> I know this is the received wisdom in certain circles but I have trouble

This is just "top down design" under new verbiage. Some programmers like to
write the low-level functions and then express the higher level ones in terms
of the language of these lower level functions. Other programmers are able to
write the top that they want first and fill in the bottom.

> writing the test before I've at least specified the function's prototype. And
> certain tests aren't writable until you've seen the function's internals. For
> instance certain floating point calculations may have divide by zero
> possibilities which you only know about once you've seen the code.

There are obvious limitations to "test first" or "top down".

For instance, imagine it is 1969. Let's write a test case for the awk language!
Okay, let's see now: invent C, Unix, Yacc, the Bourne shell, etc. Finally, awk
arrives, and and we can execute our test case!

Bottom up is the way things really progress; top down is a small-scale, local
aberration.

[toc] | [prev] | [next] | [standalone]

#18528

On Saturday, March 10, 2012 6:29:53 PM UTC, Kaz Kylheku wrote:
> On 2012-03-10, nick_keighley wrote:
> > On Friday, March 9, 2012 11:56:22 PM UTC, Ian Collins wrote:
> >> On 03/10/12 12:43 PM, Don Y 

> >> > But, I can't see a way to handle this sort of testing on
> >> > functions that manipulate *pointers* to objects -- without
> >> > adding a fair bit of code to the scaffolding that, itself,
> >> > represents a testing issue!  (i.e., does the scaffolding
> >> > have any bugs that result in false positives or negatives??)
> >
> > I think you have to draw the line somewhere. Testing test code 
> > leads you into a infinite regression.
> 
> That is moot anyway because the point of a /regression/ test suite isn't to
> find bugs or validate functionality. It's simply to validate that the
> behavior hasn't changed between revisions (including buggy behavior!)

ah. ok. But the first time you run the test it /is/ supposed to be part of your validation.

> Thus an imperfect regression test suite can still find a regression. 

<snip>

> >> As a general point, it a lot easier and more satisfying for the 
> >> programmer to write the tests before the code under test.
> >
> > I know this is the received wisdom in certain circles but I have trouble
> 
> This is just "top down design" under new verbiage.

ah, I thought it was nonsense the first time round! I remember reading a book where a guy constructed an entire program top. Jolly cleaver I thought but how does he know how to write exectly the right procedure at each step. To make no mistakes, no false steps. I suspected then- and more than suspect now! that he didn't actually write the program the way he claimed he'd written it.

> Some programmers like to
> write the low-level functions and then express the higher level ones in terms
> of the language of these lower level functions. Other programmers are able to
> write the top that they want first and fill in the bottom.

I do a bit of both. I like to think of the "outside" of the program. Then go into a tool building phase. These are the sort of things I'll need to get the outside behaviour. The bolt things together, testing as I go. If its very complex then there may be many layers. Oh and these days OO seems to the natural way to do things even if C is the implementaion language. Functional programs I'm still struggling with. 

> > writing the test before I've at least specified the function's 
> > prototype. And certain tests aren't writable until you've seen the 
> > function's internals. For instance certain floating point 
> > calculations may have divide by zero possibilities which you only 
> > know about once you've seen the code.
> 
> There are obvious limitations to "test first" or "top down".

my favourite example was a stream of bytes to hold messages. This could be read from, usually forwards but occaisonally backwards. Internally the stream was a singly linked list of fixed sized blocks. 32 bytes in size if I remeber correctly. This was hurried into "integration test" (put everyting in a big bag and shake it) without any module testing. In the field it failed. That's right when it tried to reverse over a block boundary.

> For instance, imagine it is 1969. Let's write a test case for the 
> awk language!
> Okay, let's see now: invent C, Unix, Yacc, the Bourne shell, etc. 
> Finally, awk arrives, and and we can execute our test case!
> 
> Bottom up is the way things really progress; top down is a small-scale, local
> aberration.

I used to design top down but implement bottom up. Writing mock functions so I can implement and test top down just seems tedious.

top down used to lead to some odd programs. One program I saw seemed to follow a rule of three. The top called three functions each in turn called three more. Eventually he got bored and a big lump of code down in the bowels suddenly did all the work.

[toc] | [prev] | [next] | [standalone]

#18539

Hi Nick,

On 3/11/2012 5:22 AM, nick_keighley_nospam@hotmail.com wrote:
> On Saturday, March 10, 2012 6:29:53 PM UTC, Kaz Kylheku wrote:

>> Bottom up is the way things really progress; top down is a small-scale, local
>> aberration.
>
> I used to design top down but implement bottom up. Writing mock functions so
> I can implement and test top down just seems tedious.

I've a friend who codes entirely top-down.  It is very (psychologically)
straining to watch over his shoulder:

"What's foo() do?"
<shrug>  "I dunno.  I'll figure it out when I write it..."

I design hardware so tend to think of a project in terms of what
it "needs" to perform its function.  Then, build higher levels of
abstraction up from the bottom.

But, you have to know where you are headed, ultimately, so that
you propagate the necessary abstractions upward.  "Do I need to
convey communication errors up to the consumers of that service?"
etc.

I find looking at a problem from the top is the only way to understand
the nature of that goal.  OTOH, I find it easier to determine when
and where to shed detail as building up -- rather than *adding*
detail when drilling down.

[toc] | [prev] | [next] | [standalone]

#18667

"Don Y" <this@isnotme.com> ha scritto nel messaggio
news:jjil48$8rv$1@speranza.aioe.org...
> Hi Nick,
>
> On 3/11/2012 5:22 AM, nick_keighley_nospam@hotmail.com wrote:
>> On Saturday, March 10, 2012 6:29:53 PM UTC, Kaz Kylheku wrote:
>
>>> Bottom up is the way things really progress; top down is a small-scale,
>>> local
>>> aberration.
>>
>> I used to design top down but implement bottom up. Writing mock functions so
>> I can implement and test top down just seems tedious.
>
> I've a friend who codes entirely top-down.  It is very (psychologically)
> straining to watch over his shoulder:
>
> "What's foo() do?"
> <shrug>  "I dunno.  I'll figure it out when I write it..."
>
> I design hardware so tend to think of a project in terms of what
> it "needs" to perform its function.  Then, build higher levels of
> abstraction up from the bottom.
>
> But, you have to know where you are headed, ultimately, so that
> you propagate the necessary abstractions upward.  "Do I need to
> convey communication errors up to the consumers of that service?"
> etc.
>
> I find looking at a problem from the top is the only way to understand
> the nature of that goal.  OTOH, I find it easier to determine when
> and where to shed detail as building up -- rather than *adding*
> detail when drilling down.

for me it is both a top down and a botton up process...
i not identify how and what...; it seems there are changes in code
from the first way of think each code function

each time i'm going from top to down and going from
down to top etc :)

the project start in top down but end going
from top down to botton up and viceversa
at last seems to me that...

Have a good day and good life too :)

[toc] | [prev] | [next] | [standalone]

#18430

On 2012-03-09, Don Y <this@isnotme.com> wrote:
> In particular, I am trying to develop a test suite for my
> dynamic memory management functions (e.g., malloc and free
> types of routines).
>
> I.e., I can't come up with "constant" results against
> which to compare test-time results.  Sticking with the
> traditional malloc/free for example (my routines are
> more heavily parameterized), I can create a malloc.test
> that pushes size_t's to malloc.  But, aside from verifying
> the result is not NULL (or, *is* NULL, in some cases!),
> there's nothing I can do to check that the result is
> actually correct!

A possible regression test case for malloc routines is a program which beats up
the routines, performs certain validation on the inputs and terminates
successfully.

> About the only thing I can check for is if that second
> invocation returned 0x12345699 (which overlaps the previous
> allocation at 0x12345678!).

You could allocate some random blocks and keep them in an array.  All such
blocks should be non-overlapping.

Another thing to test for is that the allocator is not getting confused and
overwriting the data areas of the blocks.

Put data in the blocks and maintain checksums. Check that the allocator
operations do not change anything.

Also, the regression test does not have to use only the public routines.  Give
your allocator routines some debugging functions.

The test could check, for instance, that the allocator's data structures have
returned to some proper state after the blocks have been freed.

This could be repeated for different orders of freeing the blocks: in order, in
reverse order, in scrambled order.

You could have some self-check routines in the allocator.  For instance if it
keeps counters about how much has been allocated, etc, a function could be
provided which performs a check by walking the data structure to validate that
the meta-data matches the structure.

> Furthermore, just examining the results from malloc don't
> tell me that the freelist is being maintained, properly.

See above: expose an internal routine that validates the sanity of the
free list, and use it in the regression test case.

> Of course, I can write code to inspect these objects in
> greater detail.  But, then the test scaffolding starts to
> rival the complexity of the functions being tested!  (bugs)

But that is fine because that scaffolding isn't what is being deployed.

A wind tunnel is going to be larger than the aircraft part being tested.
That's a given.

> I guess the "ideal" I see would be to be able to build a
> "memory image" and verify that the functions cause expected
> and predicted changes in that "image".

That's not a bad idea.

The image will likely contain pointers. Those can be normalized by changing
them to offsets from the base address.

As a substitute for a binary image dump, you could have a
symbolic dump which traverses everything and produces
a printed representation of it.

> But, I can't see any way to make this NOT require some sort
> of active probing of memory by the test scaffolding (which
> means the internals of the functions have to be exposed to
> the test suite).

Exposing internals to a regression test: fine idea.

Would you not test electronic hardware with some probes
to check voltage levels and signals?

[toc] | [prev] | [next] | [standalone]

#18465

Hi Kaz,

On 3/9/2012 8:14 PM, Kaz Kylheku wrote:
> On 2012-03-09, Don Y<this@isnotme.com>  wrote:
>> In particular, I am trying to develop a test suite for my
>> dynamic memory management functions (e.g., malloc and free
>> types of routines).
>>
>> I.e., I can't come up with "constant" results against
>> which to compare test-time results.  Sticking with the
>> traditional malloc/free for example (my routines are
>> more heavily parameterized), I can create a malloc.test
>> that pushes size_t's to malloc.  But, aside from verifying
>> the result is not NULL (or, *is* NULL, in some cases!),
>> there's nothing I can do to check that the result is
>> actually correct!
>
> A possible regression test case for malloc routines is a program which beats up
> the routines, performs certain validation on the inputs and terminates
> successfully.

But that doesn't "prove" that the functions are operating as
*designed*/expected.

E.g., my allocator allows the caller to tell it *how* to select
a chunk of memory from the free list, *where* in the free list
that selection should be made, how the selection should be modified
to fit the specifications of the request, etc.

The allocator can "satisfy" all of the requests that I make
of it (i.e., can return non NULL) yet could have done so
in entirely the wrong "way".

>> About the only thing I can check for is if that second
>> invocation returned 0x12345699 (which overlaps the previous
>> allocation at 0x12345678!).
>
> You could allocate some random blocks and keep them in an array.  All such
> blocks should be non-overlapping.

But that still doesn't tell you that it is returning the *correct*
results.  You're testing an *aspect* of the allocator which doesn;t
guarantee that the allocator itself is functioning properly.

E.g., imagine testing sqrt() with (pseudocode):

    lastvalue = 0.
    lastresult = 0
    do {
       value = lastvalue + 1.
       result = sqrt(value)
       if ! (result > lastresult)
           break;
       lastvalue = value
       lastresult = result
    } while (FOREVER)

I.e., a functioning sqrt() will comply with this test.  But,
a MALfunctioning one will, as well!

> Another thing to test for is that the allocator is not getting confused and
> overwriting the data areas of the blocks.
>
> Put data in the blocks and maintain checksums. Check that the allocator
> operations do not change anything.

Again, that's just poking an *aspect* of the allocation
not testing the allocator's functionality.  Before the blocks
are allocated, their contents are indeterminate.  You don't
*care* if the allocator screws with them prior to allocation.
All you care is that it doesn't alter the contents of ALLOCATED
blocks.

> Also, the regression test does not have to use only the public routines.  Give
> your allocator routines some debugging functions.
>
> The test could check, for instance, that the allocator's data structures have
> returned to some proper state after the blocks have been freed.

The code already is littered with invariants.  I am *sure* it
"works".  :>  The problem is making sure that it continues to
work wen ported to other platforms, when "tweeked" by well
meaning developers, etc.

> This could be repeated for different orders of freeing the blocks: in order, in
> reverse order, in scrambled order.

In my case, its considerably harder than that.  You tell the allocator
(and reclamation function) how you want the free list managed.  So,
you can allocate using a set of criteria (call them "A1") and release
with some criteria R1.  The *next* allocation would yield M1.

OTOH, allocating with A1 and releasing with R2 could cause the next
allocation to yield M2 (!) -- and be operating exactly as designed!

> You could have some self-check routines in the allocator.  For instance if it
> keeps counters about how much has been allocated, etc, a function could be
> provided which performs a check by walking the data structure to validate that
> the meta-data matches the structure.

See above (invariants)

>> Furthermore, just examining the results from malloc don't
>> tell me that the freelist is being maintained, properly.
>
> See above: expose an internal routine that validates the sanity of the
> free list, and use it in the regression test case.
>
>> Of course, I can write code to inspect these objects in
>> greater detail.  But, then the test scaffolding starts to
>> rival the complexity of the functions being tested!  (bugs)
>
> But that is fine because that scaffolding isn't what is being deployed.
>
> A wind tunnel is going to be larger than the aircraft part being tested.
> That's a given.

But the effort to VALIDATE the wind tunnel is amortized over
ALL of the aircraft parts being tested in that tunnel.  Imagine
building a special wind tunnel for the wing.  Of *this* ONE
airframe.  And, yet another for the tail section.  Another for
the fuselage, etc.

Each needs to be validated in its own right before being relied
upon to vouch for the "correctness" of the items tested within.

>> I guess the "ideal" I see would be to be able to build a
>> "memory image" and verify that the functions cause expected
>> and predicted changes in that "image".
>
> That's not a bad idea.
>
> The image will likely contain pointers. Those can be normalized by changing
> them to offsets from the base address.
>
> As a substitute for a binary image dump, you could have a
> symbolic dump which traverses everything and produces
> a printed representation of it.

The problem is building the "template" image.  You can't just
put pointers anywhere.  E.g., how do you create an "unaligned
pointer" in the language?

>> But, I can't see any way to make this NOT require some sort
>> of active probing of memory by the test scaffolding (which
>> means the internals of the functions have to be exposed to
>> the test suite).
>
> Exposing internals to a regression test: fine idea.
>
> Would you not test electronic hardware with some probes
> to check voltage levels and signals?

Sure.  But I wouldn't grab a length of #12AWG and use it to
probe the front end of a satellite receiver!  I'd get a
probe that was VALIDATED AS APPROPRIATE for that sort of
testing.

[toc] | [prev] | [next] | [standalone]

#18438

On Fri, 2012-03-09, Don Y wrote:
> Hi,
>
> There's rarely a problem generating regression tests for
> functions/procedures that use concrete *values*:
>
> sqrt.test
>
> 3 --> 1.732
> 4 --> 2
> 5 --> 2.236
>
> (apologies if I have misremembered any of these values  :> )
>
> And, it is easy to erect the necessary scaffolding for those
> sorts of tests without introducing any noticeable uncertainty
> in the veracity of the results obtained.
>
> But, I can't see a way to handle this sort of testing on
> functions that manipulate *pointers* to objects -- without
> adding a fair bit of code to the scaffolding that, itself,
> represents a testing issue!  (i.e., does the scaffolding
> have any bugs that result in false positives or negatives??)
>
> In particular, I am trying to develop a test suite for my
> dynamic memory management functions (e.g., malloc and free
> types of routines).
>
> I.e., I can't come up with "constant" results against
> which to compare test-time results.  Sticking with the
> traditional malloc/free for example (my routines are
> more heavily parameterized), I can create a malloc.test
> that pushes size_t's to malloc.  But, aside from verifying
> the result is not NULL (or, *is* NULL, in some cases!),
> there's nothing I can do to check that the result is
> actually correct!

I don't see the problem.  The semantics of malloc() are:

- malloc(N) returns either 0 or a pointer to N usable bytes
- that pointer is aligned in some useful way
- active malloc()ed blocks don't overlap

And that's surely something you can test. The number of possible
inputs is staggering, so picking relevant ones is hard ... but the
pass/fail criteria are rather straightforward.

Mixing free() into this is trickier, but still not in the way that
seems to worry you. For starters, how can you make this implementation
fail the tests?

  void free(void *ptr) { ; }

...
> Furthermore, just examining the results from malloc don't
> tell me that the freelist is being maintained, properly.
>
> Of course, I can write code to inspect these objects in
> greater detail.  But, then the test scaffolding starts to
> rival the complexity of the functions being tested!  (bugs)

I think this is just a function of the fact that writing a good
malloc/free is *hard*.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#18440

Hi Jorgen,

On 3/10/2012 1:42 AM, Jorgen Grahn wrote:

>> But, I can't see a way to handle this sort of testing on
>> functions that manipulate *pointers* to objects -- without

---^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

>> adding a fair bit of code to the scaffolding that, itself,
>> represents a testing issue!  (i.e., does the scaffolding
>> have any bugs that result in false positives or negatives??)
>>
>> In particular, I am trying to develop a test suite for my
>> dynamic memory management functions (e.g., malloc and free
>> types of routines).

Note the use of "TYPES of routines" (i.e., NOT "malloc")

>> I.e., I can't come up with "constant" results against
>> which to compare test-time results.  Sticking with the
>> traditional malloc/free for example (my routines are
>> more heavily parameterized), I can create a malloc.test
>> that pushes size_t's to malloc.  But, aside from verifying
>> the result is not NULL (or, *is* NULL, in some cases!),
>> there's nothing I can do to check that the result is
>> actually correct!
>
> I don't see the problem.  The semantics of malloc() are:

See above.  And, the earlier comment regarding "functions
(in general) that manipulate pointers.

> - malloc(N) returns either 0 or a pointer to N usable bytes
> - that pointer is aligned in some useful way
> - active malloc()ed blocks don't overlap
>
> And that's surely something you can test.

No, you can't.  You don't know what valid results *are*!
Can I allocate 64K of memory?  4G bytes of memory?  How
much memory do I really *have* available?  If I allocate
X bytes of memory, where will it end up located (even if
you stick to the conventional malloc, you have no way
of telling if it has actually allocated memory from
a *real* place in memory.  You have no way of knowing
how many of those allocations can/will succeed before the
heap is exhausted.  Or, do you just keep issuing requests
until malloc fails?  Then, examine all of the results
to see if the above conditions hold for all such
allocations?  How do you *know* that the pointer is
aligned properly and not just "coincidence".

2+2 == 2*2

> The number of possible
> inputs is staggering, so picking relevant ones is hard ... but the
> pass/fail criteria are rather straightforward.
>
> Mixing free() into this is trickier, but still not in the way that
> seems to worry you. For starters, how can you make this implementation
> fail the tests?
>
>    void free(void *ptr) { ; }

What if ptr references a location that was NEVER in the heap?
What if it references a location that could never have been
*returned* by malloc()?  E.g., not properly aligned.

>> Furthermore, just examining the results from malloc don't
>> tell me that the freelist is being maintained, properly.
>>
>> Of course, I can write code to inspect these objects in
>> greater detail.  But, then the test scaffolding starts to
>> rival the complexity of the functions being tested!  (bugs)
>
> I think this is just a function of the fact that writing a good
> malloc/free is *hard*.

Writing the subsystem is fairly easy.  But, coming up with
a robust way of testing it so that others can deploy it
on very different hardware is proving difficult.

E.g., I can give you source for sqrt() and you could port it
to <whatever> and then reassure yourself that it is working
properly by running a set of test cases (that I provide
and you potentially augment) through it and verify the results.
sqrt(3) =?= 1.732, etc.  "3" doesn't change from one platform
to another.  Nor does "1.732".

Since you can't nail down actual addresses a priori (i.e., at
test creation time), the only way to design aggressive tests
(for memory allocators) is to do so algorithmically.  But,
then your test suite becomes nontrivial and subject to
bugs (1.732 is ALWAYS 1.732)

[toc] | [prev] | [next] | [standalone]

#18450

"Don Y" <this@isnotme.com> wrote in message
news:jjf6ap$ot3$1@speranza.aioe.org...


> Writing the subsystem is fairly easy.  But, coming up with
> a robust way of testing it so that others can deploy it
> on very different hardware is proving difficult.
>
> E.g., I can give you source for sqrt() and you could port it
> to <whatever> and then reassure yourself that it is working
> properly by running a set of test cases (that I provide
> and you potentially augment) through it and verify the results.
> sqrt(3) =?= 1.732, etc.  "3" doesn't change from one platform
> to another.  Nor does "1.732".

Actually the square root of 3, as returned from such a function, *can*
change between platforms. But is that due to differences in precision, or
because of a bug?

> Since you can't nail down actual addresses a priori (i.e., at
> test creation time), the only way to design aggressive tests
> (for memory allocators) is to do so algorithmically.  But,
> then your test suite becomes nontrivial and subject to
> bugs (1.732 is ALWAYS 1.732)

Maybe you're just trying too hard. Are you trying to produce some software,
which, without benefit of alpha and beta versions, can be released as a
version 1.000 with *no* bugs?

I'm not sure that's possible except for the most trivial examples, and even
then you might just be getting the right answers by luck.

And then there's performance to be considered; it might work properly, but
too slowly. Or it might only slow down after a particular pattern of inputs.
Functions such as allocators are very difficult to test. And it might be 
difficult to just think up suitable tests, without feedback from actual 
trials.

-- 
bartc 

[toc] | [prev] | [next] | [standalone]

#18458

Hi Bart,

On 3/10/2012 8:54 AM, BartC wrote:
> "Don Y" <this@isnotme.com> wrote in message
>
>> Writing the subsystem is fairly easy. But, coming up with
>> a robust way of testing it so that others can deploy it
>> on very different hardware is proving difficult.
>>
>> E.g., I can give you source for sqrt() and you could port it
>> to <whatever> and then reassure yourself that it is working
>> properly by running a set of test cases (that I provide
>> and you potentially augment) through it and verify the results.
>> sqrt(3) =?= 1.732, etc. "3" doesn't change from one platform
>> to another. Nor does "1.732".
>
> Actually the square root of 3, as returned from such a function, *can*
> change between platforms. But is that due to differences in precision, or
> because of a bug?

You can ensure that the routine is good to "N digits"
(where N is related to the minimum conforming "DBL_DIG";
your algorithm could provide more, or less, depending on
its design.  OTOH, saying sqrt(3) == 1.7 is probably not
what the contract guarantees!  :> )

>> Since you can't nail down actual addresses a priori (i.e., at
>> test creation time), the only way to design aggressive tests
>> (for memory allocators) is to do so algorithmically. But,
>> then your test suite becomes nontrivial and subject to
>> bugs (1.732 is ALWAYS 1.732)
>
> Maybe you're just trying too hard. Are you trying to produce some software,
> which, without benefit of alpha and beta versions, can be released as a
> version 1.000 with *no* bugs?

First of all, I consider it sinful to release code that hasn't
been tested.  "Well, I ran it a couple of times and it *looked*
like it was working.  Play with it and tell me what you think..."
is not a valid attitude, IMO.

Second, testing helps clarify how a function/subsystem is
*intended* to work.  It tells users of those functions/subsystems
what they should expect from "real-world examples".  E.g.,
tests expected not to succeed are commented in the test suite
as to WHY they won't "work"  (e.g., sqrt(-1)).

Third, it often helps *you* think of situations that you might
not have addressed -- or, in which the contract may be deficient.

Fourth, they act as REGRESSION tests to safeguard against
the INTRODUCTION of bugs after release:  "This stuff USED TO
WORK, why doesn't it *now*?"

> I'm not sure that's possible except for the most trivial examples, and even
> then you might just be getting the right answers by luck.

"Luck" has no place in an engineering/science discipline:
"Here, Isaac, drop the apple... maybe it will FALL"

You engineer tests to *verify* expected results.  If the test
doesn't turn out the way you *expect*, it should truly be a
surprise to you!

> And then there's performance to be considered; it might work properly, but
> too slowly. Or it might only slow down after a particular pattern of
> inputs.

That, in itself (in this particular case) is actually one of the
things you want to test for!  *If* the code is functioning
properly, then I can expect performance to be X in this situation
(based on the deliberate actions I have taken to fragment the
free list).

Part of the rationale for my abandonment of the traditional
malloc/free approach is because it doesn't give the programmer
any control over runtime *performance* -- it's an opaque box!

> Functions such as allocators are very difficult to test. And it might be
> difficult to just think up suitable tests, without feedback from actual
> trials.

How could a "trial" possibly give me more information than I
have available when examining the source?  If I know my art,
then I should be able to anticipate damn near every possible
case that the code will encounter -- moreso than an application
is likely to stumble upon in the course of pursuing *its* goal.

[toc] | [prev] | [next] | [standalone]

#18533

On Saturday, March 10, 2012 6:49:40 PM UTC, Don Y wrote:
> On 3/10/2012 8:54 AM, BartC wrote:
> > "Don Y" <this@isnotme.com> wrote in message

<snip>

> >> Since you can't nail down actual addresses a priori (i.e., at
> >> test creation time), the only way to design aggressive tests
> >> (for memory allocators) is to do so algorithmically. But,
> >> then your test suite becomes nontrivial and subject to
> >> bugs (1.732 is ALWAYS 1.732)
> >
> > Maybe you're just trying too hard. Are you trying to produce some software,
> > which, without benefit of alpha and beta versions, can be released as a
> > version 1.000 with *no* bugs?
> 
> First of all, I consider it sinful to release code that hasn't
> been tested.  "Well, I ran it a couple of times and it *looked*
> like it was working.  Play with it and tell me what you think..."
> is not a valid attitude, IMO.

that's not quite the same as "no bugs"

<snip>

> > I'm not sure that's possible except for the most trivial examples, and even
> > then you might just be getting the right answers by luck.
> 
> "Luck" has no place in an engineering/science discipline:
> "Here, Isaac, drop the apple... maybe it will FALL"
> 
> You engineer tests to *verify* expected results.  If the test
> doesn't turn out the way you *expect*, it should truly be a
> surprise to you!
> 
> > And then there's performance to be considered; it might work properly, but
> > too slowly. Or it might only slow down after a particular pattern of
> > inputs.
> 
> That, in itself (in this particular case) is actually one of the
> things you want to test for!  *If* the code is functioning
> properly, then I can expect performance to be X in this situation
> (based on the deliberate actions I have taken to fragment the
> free list).

I'll be impressed if you succeed

> Part of the rationale for my abandonment of the traditional
> malloc/free approach is because it doesn't give the programmer
> any control over runtime *performance* -- it's an opaque box!
> 
> > Functions such as allocators are very difficult to test. And it might be
> > difficult to just think up suitable tests, without feedback from actual
> > trials.
> 
> How could a "trial" possibly give me more information than I
> have available when examining the source? 

because the real world is a constant surprise

> If I know my art,
> then I should be able to anticipate damn near every possible
> case that the code will encounter -- moreso than an application
> is likely to stumble upon in the course of pursuing *its* goal.

"should" being the operative word!

"
GLENDOWER
I can call spirits from the vasty deep.

HOTSPUR
  Why, so can I, or so can any man,
  But will they come when you do call for them?
"

This is one of the reasons the formal methods approach has been sluggish. 
In theory you prove a program is correct without ever executing it. But 
people rarely do this on any large scale.

The writers of the Space Shuttle software might have acheived  to what 
you're trying to do. At enourmous cost.

[toc] | [prev] | [next] | [standalone]

#18543

Hi Nick,

On 3/11/2012 6:09 AM, nick_keighley_nospam@hotmail.com wrote:

<snip>

>>> Maybe you're just trying too hard. Are you trying to produce some software,
>>> which, without benefit of alpha and beta versions, can be released as a
>>> version 1.000 with *no* bugs?
>>
>> First of all, I consider it sinful to release code that hasn't
>> been tested.  "Well, I ran it a couple of times and it *looked*
>> like it was working.  Play with it and tell me what you think..."
>> is not a valid attitude, IMO.
>
> that's not quite the same as "no bugs"

Exactly!  Yet that seems to be the most common approach to testing
nowadays (how many folks have formal specifications, test suites,
validation procedures, etc.).

You have to consciously *attack*/challenge a piece of code to
identify deficiencies.  Where are the boundary conditions?
What did the developer *not* expect?  etc.

I tend to be pretty good at coming up with "unexpected"
operating conditions.  When the developer invariably
*whines* saying "but you're not supposed to *do* that!"
I grin and reply, "Then don't LET me!"

>>> And then there's performance to be considered; it might work properly, but
>>> too slowly. Or it might only slow down after a particular pattern of
>>> inputs.
>>
>> That, in itself (in this particular case) is actually one of the
>> things you want to test for!  *If* the code is functioning
>> properly, then I can expect performance to be X in this situation
>> (based on the deliberate actions I have taken to fragment the
>> free list).
>
> I'll be impressed if you succeed

This is really not hard to do.  Since most of my work is for
real-time applications, *how* the algorithm performs (timeliness)
is as important as what it *does*.

This was the motivation for developing this version of the
memory management library -- since an application must be
able to accurately predict how a particular request will
be satisfied INCLUDING HOW LONG IT WILL TAKE.

>>> Functions such as allocators are very difficult to test. And it might be
>>> difficult to just think up suitable tests, without feedback from actual
>>> trials.
>>
>> How could a "trial" possibly give me more information than I
>> have available when examining the source?
>
> because the real world is a constant surprise

Engineering is the art of anticipating those surprises!
In the "ethereal" world of software, this is considerably easier
than in the physical world  ("What happens if a guy whacks it
with a 20lb sledge hammer?")

>> If I know my art,
>> then I should be able to anticipate damn near every possible
>> case that the code will encounter -- moreso than an application
>> is likely to stumble upon in the course of pursuing *its* goal.
>
> "should" being the operative word!

Of course!  That's what makes me good at what I do!  :>

> "
> GLENDOWER
> I can call spirits from the vasty deep.
>
> HOTSPUR
>    Why, so can I, or so can any man,
>    But will they come when you do call for them?
> "
>
> This is one of the reasons the formal methods approach has been sluggish.
> In theory you prove a program is correct without ever executing it. But
> people rarely do this on any large scale.

Actually, this is common on very large scale projects (which,
otherwise, would never be able to run a single invocation to
completion, otherwise).

Where it *doesn't* happen is smaller projects, "lone wolf"
developers, "immature" organizations, etc.  "Pay me now or
pay me later" is a great mantra when it comes to software
quality and testing, in general.

> The writers of the Space Shuttle software might have acheived  to what
> you're trying to do. At enourmous cost.

There are many industries where formal testing and validation
are the norm and "required" practices.  Surprisingly, the
cost for this isn't as great as one would think.  Once a
module has been validated, that cost is behind you -- any
future use of the module comes with an *assurance* of proper
operation (within the constraints of the contract/spec).

In these environments, you really benefit from code AND
design reuse.

OTOH, if you are developing "ad hoc" and testing is left
to the whims of the developer, chances are you can't
afford a proper validation environment and you'll just keep
paying for it in OTHER ways (bugs, feature creep, training,
etc.) without ever being able to wrap your hands around the
entire problem.

[Do you get the impression that I am a big advocate of
formalized methods?  :> ]

[toc] | [prev] | [next] | [standalone]

#18573

On Sunday, March 11, 2012 7:33:27 PM UTC, Don Y wrote:
> On 3/11/2012 6:09 AM, nick_keighley_nospam@hotmail.com wrote:


> [...] Yet that seems to be the most common approach to testing
> nowadays (how many folks have formal specifications, test suites,
> validation procedures, etc.).

there's a system test spec and we have to comply to an international standard. This is verified by an external 3rd party.

> You have to consciously *attack*/challenge a piece of code to
> identify deficiencies.  Where are the boundary conditions?
> What did the developer *not* expect?  etc.
> 
> I tend to be pretty good at coming up with "unexpected"
> operating conditions.  When the developer invariably
> *whines* saying "but you're not supposed to *do* that!"
> I grin and reply, "Then don't LET me!"

I worked in a syetem test department for a while. I collected a list of best excuses "that hardly ever happens", "the customer would never do that" and my all time favourite "but the code says that's what happens!"

<snip>

> >>> Functions such as allocators are very difficult to test. And it 
> >>> might be difficult to just think up suitable tests, without 
> >>> feedback from actual trials.
> >>
> >> How could a "trial" possibly give me more information than I
> >> have available when examining the source?
> >
> > because the real world is a constant surprise
> 
> Engineering is the art of anticipating those surprises!
> In the "ethereal" world of software, this is considerably easier
> than in the physical world  ("What happens if a guy whacks it
> with a 20lb sledge hammer?")

yes but most engineers don't have to cope with millions of potentially interacting parts.

> >> If I know my art,
> >> then I should be able to anticipate damn near every possible
> >> case that the code will encounter -- moreso than an application
> >> is likely to stumble upon in the course of pursuing *its* goal.
> >
> > "should" being the operative word!
> 
> Of course!  That's what makes me good at what I do!  :>

<snip>

> > This is one of the reasons the formal methods approach has been sluggish.
> > In theory you prove a program is correct without ever executing it. But
> > people rarely do this on any large scale.
> 
> Actually, this is common on very large scale projects (which,
> otherwise, would never be able to run a single invocation to
> completion, otherwise).

? why not? Because they don't terminate?

I was talking about actual proofs of correctness. VDM, Z, Sparc. Are these widely used?

> Where it *doesn't* happen is smaller projects, "lone wolf"
> developers, "immature" organizations, etc.  "Pay me now or
> pay me later" is a great mantra when it comes to software
> quality and testing, in general.
> 
> > The writers of the Space Shuttle software might have achieved [] what
> > you're trying to do. At enourmous cost.
> 
> There are many industries where formal testing and validation
> are the norm and "required" practices.

yes. I know. The Space Shuttle software was a step above this. CMM level 5 and all that.

Formal testing is not the same as formal methods

>  Surprisingly, the
> cost for this isn't as great as one would think.  Once a
> module has been validated, that cost is behind you -- any
> future use of the module comes with an *assurance* of proper
> operation (within the constraints of the contract/spec).
> 
> In these environments, you really benefit from code AND
> design reuse.
> 
> OTOH, if you are developing "ad hoc" and testing is left
> to the whims of the developer, chances are you can't
> afford a proper validation environment and you'll just keep
> paying for it in OTHER ways (bugs, feature creep, training,
> etc.) without ever being able to wrap your hands around the
> entire problem.
> 
> [Do you get the impression that I am a big advocate of
> formalized methods?  :> ]

[toc] | [prev] | [next] | [standalone]

#18593

Hi Nick,

On 3/12/2012 3:29 AM, nick_keighley_nospam@hotmail.com wrote:
> On Sunday, March 11, 2012 7:33:27 PM UTC, Don Y wrote:
>> On 3/11/2012 6:09 AM, nick_keighley_nospam@hotmail.com wrote:

>> [...] Yet that seems to be the most common approach to testing
>> nowadays (how many folks have formal specifications, test suites,
>> validation procedures, etc.).
>
> there's a system test spec and we have to comply to an international standard. This is verified by an external 3rd party.

Most folks operate with "none of the above".  How can you know
if you've written what you were *supposed* to write if you
don't have anything DEFINING it and PROVING your compliance??

>> You have to consciously *attack*/challenge a piece of code to
>> identify deficiencies.  Where are the boundary conditions?
>> What did the developer *not* expect?  etc.
>>
>> I tend to be pretty good at coming up with "unexpected"
>> operating conditions.  When the developer invariably
>> *whines* saying "but you're not supposed to *do* that!"
>> I grin and reply, "Then don't LET me!"
>
> I worked in a syetem test department for a while. I collected a
> list of best excuses "that hardly ever happens", "the customer
> would never do that" and my all time favourite "but the code
> says that's what happens!"

When you're moving a few tons of material and "fail to stop"
when expected, people's eyebrows raise!  :>  Or, if you
extract money from a machine and there is no record of
that event.

OTOH, The fact that your dialog box didn't get redrawn on an
expose event tends to get dismissed as unimportant.  ("Um,
if *that* isn't happening as planned/desired/DESIGNED, what
ELSE isn't??")

>>> This is one of the reasons the formal methods approach has been sluggish.
>>> In theory you prove a program is correct without ever executing it. But
>>> people rarely do this on any large scale.
>>
>> Actually, this is common on very large scale projects (which,
>> otherwise, would never be able to run a single invocation to
>> completion, otherwise).
>
> ? why not? Because they don't terminate?

Because they never run long enough without CRASHING!  :>

> I was talking about actual proofs of correctness. VDM, Z, Sparc.
> Are these widely used?

I've never found a case where you could apply such a technique
"across the board".  Rather, you want people who can imagine
the "impossible" and codify those conditions -- in their code
and in their test suites.

E.g., if I am driving a motorized mechanism "to the left",
I don't *just* watch the "left limit switch" and use that
to stop further motion.  What if the motor is wired
*backwards* and I am really moving the mechanism to the
*right*?  What if the limit switches are mixed up and
left is wired to right, etc.

If I am moving left, the RIGHT limit switch should NEVER
be actuated (ignore boundary case where the mechanism is
currently *at* that limit).  So, it should be perfectly
reasonable for me to watch the LEFT limit switch to
stop normal/EXPECTED motion... AND if I see activity on
the RIGHT limit switch, I can panic(), stop the mechanism,
etc.

In reality, this sort of "miswiring" *does* happen.
Especially on prototypes.  In that case, the cost of
ruining a mechanism can be EXTREMELY painful (we don't
have another one!).

But, it takes a perverse sort of mind to think about these
sorts of things.  And, teaching others to think the same
way is difficult.  I resort to having folks sit down and
list the ASSUMPTIONS they are making in any aspect of
design.  Then, ask yourself which of those are *truly*
safe!

E.g., if I have a "string" located at 0xFF...F8 that has
8 non-NUL characters in it, what should strlen() return?

>> Where it *doesn't* happen is smaller projects, "lone wolf"
>> developers, "immature" organizations, etc.  "Pay me now or
>> pay me later" is a great mantra when it comes to software
>> quality and testing, in general.
>>
>>> The writers of the Space Shuttle software might have achieved [] what
>>> you're trying to do. At enourmous cost.
>>
>> There are many industries where formal testing and validation
>> are the norm and "required" practices.
>
> yes. I know. The Space Shuttle software was a step above this. CMM level 5 and all that.
>
> Formal testing is not the same as formal methods

But formal testing WITHOUT formal methods is like buying food
that "Bob" said was prepared healthily.

(Great!  Who's Bob??)

If Bob is a highly respected individual KNOWN (renowned!) for
his dedication to healthy food, that might be a great
endorsement.  OTOH, if Bob is some anonymous guy...

Formal methods tries to take some of the variability out of
individual "preferences"/diligence.

[toc] | [prev] | [next] | [standalone]

#18691

"Don Y" <this@isnotme.com> wrote in message
news:jjla4o$u2h$1@speranza.aioe.org...

>
> When you're moving a few tons of material and "fail to stop"
...
> OTOH, The fact that your dialog box didn't get redrawn on an
...
> E.g., if I am driving a motorized mechanism "to the left",
...
> But, it takes a perverse sort of mind to think about these
> sorts of things.  And, teaching others to think the same
> way is difficult.  I resort to having folks sit down and
> list the ASSUMPTIONS they are making in any aspect of
> design.  Then, ask yourself which of those are *truly*
> safe!

If these sorts of things are typical of what you're testing, then perhaps
you're not using the right language. C is not the safest language in the
world. Perhaps you should be building a domain-specific language on top.

> E.g., if I have a "string" located at 0xFF...F8 that has
> 8 non-NUL characters in it, what should strlen() return?

If you pass strlen() a pointer to any random set of bytes, or any random
pointer, or even just zero, then anything could happen. You don't even need
to think up any tricky corner cases!

(And passing 0 to string functions tends to crash them. You might try
writing a strlen() that is more robust, but the problem will still be having
such low-level functions scattered through your code (plus now having a set 
of string functions that are super-slow because of all the extra checks.)

-- 
Bartc 

[toc] | [prev] | [next] | [standalone]

Page 3 of 5 — ← Prev page 1 2 [3] 4 5  Next page →

Back to top | Article view | comp.lang.c

csiph-web