Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #18841

Re: Method to test a program against different C implementations

Show all headers | View raw

On 03/17/2012 11:26 AM, pozz wrote:
> Il 15/03/2012 23:32, James Kuyper ha scritto:
>> On 03/15/2012 06:09 PM, Jorgen Grahn wrote:
>>> On Sun, 2012-03-11, pozz wrote:
...
>>>> I'm asking if there's a way to test a small C source (a function or a
>>>> small set of functions) against all possible C implementations.  Of
>>>> course I'm thinking of an automatic test, because the number of C
>>>> implementations is high.
...
>>> Most/all of the responses have been surprisingly negative, so here's
>>> one in favor.
>>>
>>> Yes please, I'd like to have something like what you describe. I see
>>> practical problems, and I don't think I'll ever see such a system, but
>>> it's not a stupid idea.
>>
>> It's a stupid idea as long as it's supposed to test "against all
>> possible C implementations", apparently under the misconception that the
>> number of such implementation is not only finite, but indeed small
>> enough to make it possible to imagine that such testing would be
>> feasible using "a small script"!
> 
> Ok, ok, I think I have explained my problem in the my original post in a 
> wrong way (maybe the reason is my poor english language).
> 
> I know the C implementations in the world is a very high number (but 
> this number is surely *finite*), so it could be impossible to implement

The number of actual implementations is finite, but the set of possible
implementations is infinite, and "possible" is what you specified.

Since some compilers allow you to adjust parameters which effectively
makes them a different implementation of C for each different set of
values for those parameters, so even the number of different
implementations of C that can be provided by a single compiler may be
too large to allow practical testing with every single one of them. gcc,
for example, has a huge number of options - I'm not sure it's reasonable
to perform a test using every possible combination of those options,
even if you restrict yourself to the combinations in which it's a fully
conforming implementation of C (which is a small sub-set of the entire
range of possibilities).

> all of them in a single C compiler.  But it's not so absurd to start 
> from a C compiler implementation (even if it is poorly optimized) and 
> include, step by step, others.
> 
> Even gcc supports the option -m32/-m64 that lets to set the desired size 
> of integer type between 32 and 64 bits.
> 
> The "small script" that I mentioned could be:
> 
> gcc -m32 main.c -o main32
> gcc -m64 main.c -o main64
> ./main32 <input.txt >output32.txt
> ./main64 <input.txt >output64.txt
> diff output32.txt output64.txt
> 
> or similar.  Of course the "small script" should consider just a small 
> set of C implementations of interest, depending on the application I'm 
> testing.

Yes, and that's my key point - its the small set of C implementations of
interest to you; it might be of great interest to other people as well,
and of quite negligible interest to a great many other people, because
they use different implementations.

>> As soon as you select a subset of all possible C implementations that is
>> small enough to make such testing feasible, you are talking about an
>> automatic test procedure whose details are highly dependent upon which
>> subset you've chosen. Such a procedure could be of considerable interest
>> to those who would have selected a sufficiently similar subset, while
>> being of little or no interest in those who'd prefer a significantly
>> different subset.
> 
> This is true, but if we have a C compiler that has customizable size of 
> integer types and configurable numerical representation (little-, big- 
> or other representations), it will be a good starting point.
> Of course, there are C implementations with a different size of chars or 
> one's complement negative number representation, but they are very 
> uncommon so they could be ignored.

Except, of course, by those who are using them (if any). one's
complement is pretty rare nowadays; but implementations with 16-bit char
are significantly more common (while still being uncommon).

>> To be useful, the test should also, ideally, cover all possible sets of
>> input data - a space that usually has a very high dimensionality, and a
>> huge volume. For most non-trivial programs it's absolutely, completely,
>> infeasible to cover that entire space. The best that can be done is a
>> finite sample of the possible sets. Unless the automated system has
>> intelligence comparable to that of the professional tester, it's stuck
>> with randomly searching the space of possible input sets, which is
>> unlikely to find all the special cases that a professional tester could
>> identify as worthy of special attention.
> 
> I don't know how you write and test functions.  I know application 
> testing is a different matter of study, but I usually write the function 
> and try to make tests with some (and not all) inputs.  The most evident 
> errors will come out immediately.
> The goal of my test on different C implementation is to catch these 
> evident errors and not to througly test the entire complex application.

Software testing is an entire discipline in it's own right; and
competent software testers can earn good money; more money than my
company's client is willing to pay for, on my current project. I know
enough to put together and execute a decent test plan; but am seldom
given enough time to perform such a test; I usually have to settle for
less testing than I'd like to perform.
-- 
James Kuyper

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Method to test a program against different C implementations pozz <pozzugno@gmail.com> - 2012-03-11 12:27 +0100
  Re: Method to test a program against different C implementations "Joachim Schmitz" <nospam.jojo@schmitz-digital.de> - 2012-03-11 13:48 +0100
    Re: Method to test a program against different C implementations pete <pfiland@mindspring.com> - 2012-03-11 09:03 -0500
      Re: Method to test a program against different C implementations James Kuyper <jameskuyper@verizon.net> - 2012-03-11 10:43 -0400
  Re: Method to test a program against different C implementations Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-03-11 09:08 -0400
    Re: Method to test a program against different C implementations pozz <pozzugno@gmail.com> - 2012-03-11 22:12 +0100
      Re: Method to test a program against different C implementations Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-03-11 17:44 -0400
  Re: Method to test a program against different C implementations Don Y <this@isnotme.com> - 2012-03-11 09:39 -0700
  Re: Method to test a program against different C implementations Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-15 22:09 +0000
    Re: Method to test a program against different C implementations James Kuyper <jameskuyper@verizon.net> - 2012-03-15 18:32 -0400
      Re: Method to test a program against different C implementations pozz <pozzugno@gmail.com> - 2012-03-17 16:26 +0100
        Re: Method to test a program against different C implementations James Kuyper <jameskuyper@verizon.net> - 2012-03-17 11:52 -0400
    Re: Method to test a program against different C implementations Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-03-15 20:31 -0400
      Re: Method to test a program against different C implementations Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-20 22:18 +0000
        Re: Method to test a program against different C implementations Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-03-20 22:38 -0400
          Re: Method to test a program against different C implementations Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-22 21:48 +0000
            Re: Method to test a program against different C implementations James Kuyper <jameskuyper@verizon.net> - 2012-03-22 18:39 -0400
            Re: Method to test a program against different C implementations Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-03-22 21:07 -0400
  Re: Method to test a program against different C implementations Robert Miles <robertmilesxyz@gmail.com> - 2012-03-15 17:41 -0700

csiph-web