Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #18525 > unrolled thread

Method to test a program against different C implementations

Back to article view | Back to comp.lang.c

Contents

  Method to test a program against different C implementations pozz <pozzugno@gmail.com> - 2012-03-11 12:27 +0100
    Re: Method to test a program against different C implementations "Joachim Schmitz" <nospam.jojo@schmitz-digital.de> - 2012-03-11 13:48 +0100
      Re: Method to test a program against different C implementations pete <pfiland@mindspring.com> - 2012-03-11 09:03 -0500
        Re: Method to test a program against different C implementations James Kuyper <jameskuyper@verizon.net> - 2012-03-11 10:43 -0400
    Re: Method to test a program against different C implementations Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-03-11 09:08 -0400
      Re: Method to test a program against different C implementations pozz <pozzugno@gmail.com> - 2012-03-11 22:12 +0100
        Re: Method to test a program against different C implementations Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-03-11 17:44 -0400
    Re: Method to test a program against different C implementations Don Y <this@isnotme.com> - 2012-03-11 09:39 -0700
    Re: Method to test a program against different C implementations Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-15 22:09 +0000
      Re: Method to test a program against different C implementations James Kuyper <jameskuyper@verizon.net> - 2012-03-15 18:32 -0400
        Re: Method to test a program against different C implementations pozz <pozzugno@gmail.com> - 2012-03-17 16:26 +0100
          Re: Method to test a program against different C implementations James Kuyper <jameskuyper@verizon.net> - 2012-03-17 11:52 -0400
      Re: Method to test a program against different C implementations Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-03-15 20:31 -0400
        Re: Method to test a program against different C implementations Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-20 22:18 +0000
          Re: Method to test a program against different C implementations Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-03-20 22:38 -0400
            Re: Method to test a program against different C implementations Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-03-22 21:48 +0000
              Re: Method to test a program against different C implementations James Kuyper <jameskuyper@verizon.net> - 2012-03-22 18:39 -0400
              Re: Method to test a program against different C implementations Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-03-22 21:07 -0400
    Re: Method to test a program against different C implementations Robert Miles <robertmilesxyz@gmail.com> - 2012-03-15 17:41 -0700

#18525 — Method to test a program against different C implementations

We perfectly know that C implementations can be very different in terms of:
- number of bits per byte
- number of bytes per int
- number of bytes per long
- signed or unsigned char
- negative number implementation (two's or one's complement)
- big or little endian or others

I'm asking if there's a way to test a small C source (a function or a 
small set of functions) against all possible C implementations.  Of 
course I'm thinking of an automatic test, because the number of C 
implementations is high.

I think this could be done with an ad-hoc C compiler that can be 
configured (I imagine thorugh command-line parameteres) in several ways. 
  For example, I can say to the compiler to use signed or unsigned char 
(when the signedness is not specified), 16-bit ints and 32-bit longs, 
big-ending representation, two's complement for negative numbers.

So I can write a small script that runs the compiler several times with 
different configurations, producing different executables.  Now I can 
run these exe and compare the results (their outputs) against "the 
correct output" I prepared before.

I think this test could be very useful to verify the correctness of our 
sources not only against the implementation where we are developing, but 
also for other possible implementations that could be used in the future 
by the users of our program.

[toc] | [next] | [standalone]

#18529

pozz wrote:
> We perfectly know that C implementations can be very different in
> terms of: 
> - number of bits per byte

#include <limits.h>
 and use CHAR_BIT

> - number of bytes per int

sizeof(int);

> - number of bytes per long

sizeof(long)

> - signed or unsigned char

#include <limits.h>
and look at CHAR_BIT, CHAR_MAX and CHAR_MIN

> - negative number implementation (two's or one's complement)
> - big or little endian or others

http://c-faq.com/misc/endiantest.html
 
<snip>

Bye, Jojo

[toc] | [prev] | [next] | [standalone]

#18531

Joachim Schmitz wrote:
> 
> pozz wrote:
> > We perfectly know that C implementations can be very different in
> > terms of:
> > - number of bits per byte
> 
> #include <limits.h>
>  and use CHAR_BIT
> 
> > - number of bytes per int
> 
> sizeof(int);
> 
> > - number of bytes per long
> 
> sizeof(long)
> 
> > - signed or unsigned char
> 
> #include <limits.h>
> and look at CHAR_BIT, CHAR_MAX and CHAR_MIN
> 
> > - negative number implementation (two's or one's complement)

    
#include <stdio.h>

    fputs("Negative numbers are ", stdout);
    switch (-1 & 3) {
        case 1: puts ("sign and magnitude.");
                break;
        case 2: puts ("one's complement.");
                break;
        case 3 : puts("two's complement.");
                break;
        default : puts("something strange.");
                break;
    }

> > - big or little endian or others
> 
> http://c-faq.com/misc/endiantest.html
> 
> <snip>
> 
> Bye, Jojo

-- 
pete

[toc] | [prev] | [next] | [standalone]

#18534

On 03/11/2012 10:03 AM, pete wrote:
> Joachim Schmitz wrote:
>>
>> pozz wrote:
>>> We perfectly know that C implementations can be very different in
>>> terms of:
...
>>> - negative number implementation (two's or one's complement)
> 
>     
> #include <stdio.h>
> 
>     fputs("Negative numbers are ", stdout);
>     switch (-1 & 3) {
>         case 1: puts ("sign and magnitude.");
>                 break;
>         case 2: puts ("one's complement.");
>                 break;
>         case 3 : puts("two's complement.");
>                 break;
>         default : puts("something strange.");
>                 break;

It's not merely strange, it's non-conforming - those are the only three
options permitted by the standard.

>     }
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#18532

On 3/11/2012 7:27 AM, pozz wrote:
> We perfectly know that C implementations can be very different in terms of:
> - number of bits per byte
> - number of bytes per int
> - number of bytes per long
> - signed or unsigned char
> - negative number implementation (two's or one's complement)
> - big or little endian or others
>
> I'm asking if there's a way to test a small C source (a function or a
> small set of functions) against all possible C implementations. Of
> course I'm thinking of an automatic test, because the number of C
> implementations is high.

     Infinite. (Proof: sizeof(int) can be 1,2,3,... without bound,
and each of these implies a C implementation that differs in at
least one respect from all those with more or fewer bytes per int.
I was going to say "countably infinite," but then I bethought me
of undefined behavior and decided to delete "countably.")

     So, let's give up on "all possible" and consider just "many"
C implementations.  An important part of the question is then the
choice of which actual or possible implementations to test.

> I think this could be done with an ad-hoc C compiler that can be
> configured (I imagine thorugh command-line parameteres) in several ways.
> For example, I can say to the compiler to use signed or unsigned char
> (when the signedness is not specified), 16-bit ints and 32-bit longs,
> big-ending representation, two's complement for negative numbers.
>
> So I can write a small script that runs the compiler several times with
> different configurations, producing different executables. Now I can run
> these exe and compare the results (their outputs) against "the correct
> output" I prepared before.

     Do the programs take any inputs?  File contents, environment
variables, command-line parameters, ...?  Will you test with "all"
combinations of inputs?

     Further: Suppose your "correct" program outputs 2.718281828459045
and another prints 2.718281828459046 -- is the second "wrong?"  How
did you establish that the first was "correct" to begin with?

> I think this test could be very useful to verify the correctness of our
> sources not only against the implementation where we are developing, but
> also for other possible implementations that could be used in the future
> by the users of our program.

     While I agree that testing for portability is a Good Thing, I
think you've underestimated both the difficulty and the effectiveness.
Testing on one system is already difficult and expensive; testing on
N systems is more so, testing on M^N would be worse still.  If your
target platforms are Windows and MacOS, it seems unlikely that the
folks responsible for the development budget will choose to expend
a lot of resources trying to future-proof the code for hypothetical
platforms with expected total sales of zero ...

     Besides, remember Dijkstra: "Testing shows the presence, not the
absence of bugs."

-- 
Eric Sosman
esosman@ieee-dot-org.invalid

[toc] | [prev] | [next] | [standalone]

#18553

Il 11/03/2012 14:08, Eric Sosman ha scritto:
> On 3/11/2012 7:27 AM, pozz wrote:
>> We perfectly know that C implementations can be very different in
>> terms of:
>> - number of bits per byte
>> - number of bytes per int
>> - number of bytes per long
>> - signed or unsigned char
>> - negative number implementation (two's or one's complement)
>> - big or little endian or others
>>
>> I'm asking if there's a way to test a small C source (a function or a
>> small set of functions) against all possible C implementations. Of
>> course I'm thinking of an automatic test, because the number of C
>> implementations is high.
>
> Infinite. (Proof: sizeof(int) can be 1,2,3,... without bound,
> and each of these implies a C implementation that differs in at
> least one respect from all those with more or fewer bytes per int.
> I was going to say "countably infinite," but then I bethought me
> of undefined behavior and decided to delete "countably.")
>
> So, let's give up on "all possible" and consider just "many"
> C implementations. An important part of the question is then the
> choice of which actual or possible implementations to test.

I know there are an infinite set of C implementations, but usually the 
developer could be interested in a very small set, I think up to ten.
The choice of possible implementations to test is up to the developer.

>> I think this could be done with an ad-hoc C compiler that can be
>> configured (I imagine thorugh command-line parameteres) in several ways.
>> For example, I can say to the compiler to use signed or unsigned char
>> (when the signedness is not specified), 16-bit ints and 32-bit longs,
>> big-ending representation, two's complement for negative numbers.
>>
>> So I can write a small script that runs the compiler several times with
>> different configurations, producing different executables. Now I can run
>> these exe and compare the results (their outputs) against "the correct
>> output" I prepared before.
>
> Do the programs take any inputs? File contents, environment
> variables, command-line parameters, ...? Will you test with "all"
> combinations of inputs?

It depends on the algorithm or program.  Many times I wrote a function 
that I test with another program that automatically calls the function 
with all possible inputs.  The outputs can be automatically tested with 
another program or just visually inspected by the author.
I think it is sufficient to test the function with a small set of most 
representative inputs.

> Further: Suppose your "correct" program outputs 2.718281828459045
> and another prints 2.718281828459046 -- is the second "wrong?" How
> did you establish that the first was "correct" to begin with?

Again it depends on the algorithm.  You have shown a typical example 
where it could be difficult to understand which output is correct (I 
really think it's not so hard), but most of the time the output is fixed 
and is very simple to test its correctness.


>> I think this test could be very useful to verify the correctness of our
>> sources not only against the implementation where we are developing, but
>> also for other possible implementations that could be used in the future
>> by the users of our program.
>
> While I agree that testing for portability is a Good Thing, I
> think you've underestimated both the difficulty and the effectiveness.
> Testing on one system is already difficult and expensive; testing on
> N systems is more so, testing on M^N would be worse still. If your
> target platforms are Windows and MacOS, it seems unlikely that the
> folks responsible for the development budget will choose to expend
> a lot of resources trying to future-proof the code for hypothetical
> platforms with expected total sales of zero ...

Indeed I don't want to test my sources against all possible C 
implementations, but only against interesting (for me, user, sales, 
market...) C implementations or platforms.
The hypothetical "universal" C compiler can be used only for some and 
not all configurations.

Consider the simple problem of MS above: "Write int as a 4 byte 
big-endian to file."  The approach could be different (as the answers 
have shown), but some could work on some and not all or some other 
platforms.
Having a "universal" C compiler, MS could test his algorithm for 
different endianess, int size and so on.  It would be very simple to 
write a test program that calls the function with a small set of inputs 
(0, -10, +10, -65536, +65536, 0x7FFFFFFF, 0xFFFFFFFF...) and manually 
inspect the outputs for errors.

> Besides, remember Dijkstra: "Testing shows the presence, not the
> absence of bugs."

I know, I know ;-)

[toc] | [prev] | [next] | [standalone]

#18556

On 3/11/2012 5:12 PM, pozz wrote:
> [... testing for portability ...]
> It depends on the algorithm or program. Many times I wrote a function
> that I test with another program that automatically calls the function
> with all possible inputs. The outputs can be automatically tested with
> another program or just visually inspected by the author.
> I think it is sufficient to test the function with a small set of most
> representative inputs.

     Spot tests can be reassuring, but they do not "suffice" to
show the correctness of code.  I once stumbled across a bug in a
software floating-point multiplication routine, a bug that only
triggered for one particular pair of sign-plus-mantissa patterns
(the exponents didn't matter).  The famous Pentium FDIV bug was
estimated to affect only one in nine billion randomly-selected
operand pairs.

>> Further: Suppose your "correct" program outputs 2.718281828459045
>> and another prints 2.718281828459046 -- is the second "wrong?" How
>> did you establish that the first was "correct" to begin with?
>
> Again it depends on the algorithm. You have shown a typical example
> where it could be difficult to understand which output is correct (I
> really think it's not so hard), but most of the time the output is fixed
> and is very simple to test its correctness.

     If that's your "most of the time," I'd like to experience it
someday.  Must be pretty relaxing.  ;-)

> Consider the simple problem of MS above: "Write int as a 4 byte
> big-endian to file." The approach could be different (as the answers
> have shown), but some could work on some and not all or some other
> platforms.
> Having a "universal" C compiler, MS could test his algorithm for
> different endianess, int size and so on. It would be very simple to
> write a test program that calls the function with a small set of inputs
> (0, -10, +10, -65536, +65536, 0x7FFFFFFF, 0xFFFFFFFF...) and manually
> inspect the outputs for errors.

     Testing is not simple, and definitely not "very simple."  If you
believe it is so, I can only imagine that you live and code in Eden.

-- 
Eric Sosman
esosman@ieee-dot-org.invalid

[toc] | [prev] | [next] | [standalone]

#18537

On 3/11/2012 4:27 AM, pozz wrote:
> We perfectly know that C implementations can be very different in terms of:
> - number of bits per byte
> - number of bytes per int
> - number of bytes per long
> - signed or unsigned char
> - negative number implementation (two's or one's complement)
> - big or little endian or others

- pointer/object alignment
- layout of struct internals
etc.

(i.e., grep(1) the standard for "implementation defined" and/or
"undefined" and you'll be surprised at the number of "hits"!)

> I'm asking if there's a way to test a small C source (a function or a
> small set of functions) against all possible C implementations. Of
> course I'm thinking of an automatic test, because the number of C
> implementations is high.

The number of *possible* C implementations is high.  But, in
*practical* terms, its usually a much smaller number.  I code
for a wide variety of processors (I probably have 30 *different*
CPU families) and there are liberties within a particular
hardware family (i.e., between compiler vendors).

Yet, I rarely run a piece of code through all of those compilers
for validation.

IME, being *aware* of the portability issues and coding for
them is the biggest return on investment.  Think about where your
code is LIKELY to be compiled instead of where it could *possibly*
be compiled.

> I think this could be done with an ad-hoc C compiler that can be
> configured (I imagine thorugh command-line parameteres) in several ways.
> For example, I can say to the compiler to use signed or unsigned char
> (when the signedness is not specified), 16-bit ints and 32-bit longs,
> big-ending representation, two's complement for negative numbers.

I don't think a compiler exists that can be configured to cover
*all* of the potentialities.

And, you're still faced with the peculiarities of any *particular*
compiler (e.g., "strictly conforming"?) that you ultimately encounter.

> So I can write a small script that runs the compiler several times with
> different configurations, producing different executables. Now I can run
> these exe and compare the results (their outputs) against "the correct
> output" I prepared before.
>
> I think this test could be very useful to verify the correctness of our
> sources not only against the implementation where we are developing, but
> also for other possible implementations that could be used in the future
> by the users of our program.

Spend the effort identifying those issues and *attempting* to
innoculate against them.  Then, plan on doing some "fixup"
work down the road.

This is especially difficult if the "future you" is not, in fact,
*you* (or anyone closely tied to you!)

[toc] | [prev] | [next] | [standalone]

#18815

On Sun, 2012-03-11, pozz wrote:
> We perfectly know that C implementations can be very different in terms of:
> - number of bits per byte
> - number of bytes per int
> - number of bytes per long
> - signed or unsigned char
> - negative number implementation (two's or one's complement)
> - big or little endian or others
>
> I'm asking if there's a way to test a small C source (a function or a 
> small set of functions) against all possible C implementations.  Of 
> course I'm thinking of an automatic test, because the number of C 
> implementations is high.
>
> I think this could be done with an ad-hoc C compiler that can be 
> configured (I imagine thorugh command-line parameteres) in several ways. 
>   For example, I can say to the compiler to use signed or unsigned char 
> (when the signedness is not specified), 16-bit ints and 32-bit longs, 
> big-ending representation, two's complement for negative numbers.
>
> So I can write a small script that runs the compiler several times with 
> different configurations, producing different executables.  Now I can 
> run these exe and compare the results (their outputs) against "the 
> correct output" I prepared before.
>
> I think this test could be very useful to verify the correctness of our 
> sources not only against the implementation where we are developing, but 
> also for other possible implementations that could be used in the future 
> by the users of our program.

Most/all of the responses have been surprisingly negative, so here's
one in favor.

Yes please, I'd like to have something like what you describe. I see
practical problems, and I don't think I'll ever see such a system, but
it's not a stupid idea.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#18816

On 03/15/2012 06:09 PM, Jorgen Grahn wrote:
> On Sun, 2012-03-11, pozz wrote:
>> We perfectly know that C implementations can be very different in terms of:
>> - number of bits per byte
>> - number of bytes per int
>> - number of bytes per long
>> - signed or unsigned char
>> - negative number implementation (two's or one's complement)
>> - big or little endian or others
>>
>> I'm asking if there's a way to test a small C source (a function or a 
>> small set of functions) against all possible C implementations.  Of 
>> course I'm thinking of an automatic test, because the number of C 
>> implementations is high.
>>
>> I think this could be done with an ad-hoc C compiler that can be 
>> configured (I imagine thorugh command-line parameteres) in several ways. 
>>   For example, I can say to the compiler to use signed or unsigned char 
>> (when the signedness is not specified), 16-bit ints and 32-bit longs, 
>> big-ending representation, two's complement for negative numbers.
>>
>> So I can write a small script that runs the compiler several times with 
>> different configurations, producing different executables.  Now I can 
>> run these exe and compare the results (their outputs) against "the 
>> correct output" I prepared before.
>>
>> I think this test could be very useful to verify the correctness of our 
>> sources not only against the implementation where we are developing, but 
>> also for other possible implementations that could be used in the future 
>> by the users of our program.
> 
> Most/all of the responses have been surprisingly negative, so here's
> one in favor.
> 
> Yes please, I'd like to have something like what you describe. I see
> practical problems, and I don't think I'll ever see such a system, but
> it's not a stupid idea.

It's a stupid idea as long as it's supposed to test "against all
possible C implementations", apparently under the misconception that the
number of such implementation is not only finite, but indeed small
enough to make it possible to imagine that such testing would be
feasible using "a small script"!

As soon as you select a subset of all possible C implementations that is
small enough to make such testing feasible, you are talking about an
automatic test procedure whose details are highly dependent upon which
subset you've chosen. Such a procedure could be of considerable interest
to those who would have selected a sufficiently similar subset, while
being of little or no interest in those who'd prefer a significantly
different subset.

To be useful, the test should also, ideally, cover all possible sets of
input data - a space that usually has a very high dimensionality, and a
huge volume. For most non-trivial programs it's absolutely, completely,
infeasible to cover that entire space. The best that can be done is a
finite sample of the possible sets. Unless the automated system has
intelligence comparable to that of the professional tester, it's stuck
with randomly searching the space of possible input sets, which is
unlikely to find all the special cases that a professional tester could
identify as worthy of special attention.

[toc] | [prev] | [next] | [standalone]

#18840

Il 15/03/2012 23:32, James Kuyper ha scritto:
> On 03/15/2012 06:09 PM, Jorgen Grahn wrote:
>> On Sun, 2012-03-11, pozz wrote:
>>> We perfectly know that C implementations can be very different in terms of:
>>> - number of bits per byte
>>> - number of bytes per int
>>> - number of bytes per long
>>> - signed or unsigned char
>>> - negative number implementation (two's or one's complement)
>>> - big or little endian or others
>>>
>>> I'm asking if there's a way to test a small C source (a function or a
>>> small set of functions) against all possible C implementations.  Of
>>> course I'm thinking of an automatic test, because the number of C
>>> implementations is high.
>>>
>>> I think this could be done with an ad-hoc C compiler that can be
>>> configured (I imagine thorugh command-line parameteres) in several ways.
>>>    For example, I can say to the compiler to use signed or unsigned char
>>> (when the signedness is not specified), 16-bit ints and 32-bit longs,
>>> big-ending representation, two's complement for negative numbers.
>>>
>>> So I can write a small script that runs the compiler several times with
>>> different configurations, producing different executables.  Now I can
>>> run these exe and compare the results (their outputs) against "the
>>> correct output" I prepared before.
>>>
>>> I think this test could be very useful to verify the correctness of our
>>> sources not only against the implementation where we are developing, but
>>> also for other possible implementations that could be used in the future
>>> by the users of our program.
>>
>> Most/all of the responses have been surprisingly negative, so here's
>> one in favor.
>>
>> Yes please, I'd like to have something like what you describe. I see
>> practical problems, and I don't think I'll ever see such a system, but
>> it's not a stupid idea.
>
> It's a stupid idea as long as it's supposed to test "against all
> possible C implementations", apparently under the misconception that the
> number of such implementation is not only finite, but indeed small
> enough to make it possible to imagine that such testing would be
> feasible using "a small script"!

Ok, ok, I think I have explained my problem in the my original post in a 
wrong way (maybe the reason is my poor english language).

I know the C implementations in the world is a very high number (but 
this number is surely *finite*), so it could be impossible to implement 
all of them in a single C compiler.  But it's not so absurd to start 
from a C compiler implementation (even if it is poorly optimized) and 
include, step by step, others.

Even gcc supports the option -m32/-m64 that lets to set the desired size 
of integer type between 32 and 64 bits.

The "small script" that I mentioned could be:

gcc -m32 main.c -o main32
gcc -m64 main.c -o main64
./main32 <input.txt >output32.txt
./main64 <input.txt >output64.txt
diff output32.txt output64.txt

or similar.  Of course the "small script" should consider just a small 
set of C implementations of interest, depending on the application I'm 
testing.

For example, suppose I have to write a function that send signed 
integers between -65 and +64 to a file stream of 7-bits blocks.  At last 
I reach an algorithm that works on my 32-bit big-endian machine.  It's 
cool!  Now, how could I be sure this algorithm will work on a 16-bit 
little-endian machine?  If I was a C exepert guru that knew the ANSI C 
standard very good, I could inspect the code and find any error, but I'm 
not a C guru.

It would be nice if I could launch my "quasi-universal C compiler" in 
the following way:

gcc -m16 -little main.c -o main16
gcc -m32 -big main.c -o main32
./main16 <input.txt >output16.txt
./main32 <input.txt >output32.txt
diff output16.txt output32.txt

I know this strategy couldn't detect *each* possible error and couldn't 
be run for *all* possible inputs, but I think some big error could be 
detected.


> As soon as you select a subset of all possible C implementations that is
> small enough to make such testing feasible, you are talking about an
> automatic test procedure whose details are highly dependent upon which
> subset you've chosen. Such a procedure could be of considerable interest
> to those who would have selected a sufficiently similar subset, while
> being of little or no interest in those who'd prefer a significantly
> different subset.

This is true, but if we have a C compiler that has customizable size of 
integer types and configurable numerical representation (little-, big- 
or other representations), it will be a good starting point.
Of course, there are C implementations with a different size of chars or 
one's complement negative number representation, but they are very 
uncommon so they could be ignored.


> To be useful, the test should also, ideally, cover all possible sets of
> input data - a space that usually has a very high dimensionality, and a
> huge volume. For most non-trivial programs it's absolutely, completely,
> infeasible to cover that entire space. The best that can be done is a
> finite sample of the possible sets. Unless the automated system has
> intelligence comparable to that of the professional tester, it's stuck
> with randomly searching the space of possible input sets, which is
> unlikely to find all the special cases that a professional tester could
> identify as worthy of special attention.

I don't know how you write and test functions.  I know application 
testing is a different matter of study, but I usually write the function 
and try to make tests with some (and not all) inputs.  The most evident 
errors will come out immediately.
The goal of my test on different C implementation is to catch these 
evident errors and not to througly test the entire complex application.

[toc] | [prev] | [next] | [standalone]

#18841

On 03/17/2012 11:26 AM, pozz wrote:
> Il 15/03/2012 23:32, James Kuyper ha scritto:
>> On 03/15/2012 06:09 PM, Jorgen Grahn wrote:
>>> On Sun, 2012-03-11, pozz wrote:
...
>>>> I'm asking if there's a way to test a small C source (a function or a
>>>> small set of functions) against all possible C implementations.  Of
>>>> course I'm thinking of an automatic test, because the number of C
>>>> implementations is high.
...
>>> Most/all of the responses have been surprisingly negative, so here's
>>> one in favor.
>>>
>>> Yes please, I'd like to have something like what you describe. I see
>>> practical problems, and I don't think I'll ever see such a system, but
>>> it's not a stupid idea.
>>
>> It's a stupid idea as long as it's supposed to test "against all
>> possible C implementations", apparently under the misconception that the
>> number of such implementation is not only finite, but indeed small
>> enough to make it possible to imagine that such testing would be
>> feasible using "a small script"!
> 
> Ok, ok, I think I have explained my problem in the my original post in a 
> wrong way (maybe the reason is my poor english language).
> 
> I know the C implementations in the world is a very high number (but 
> this number is surely *finite*), so it could be impossible to implement

The number of actual implementations is finite, but the set of possible
implementations is infinite, and "possible" is what you specified.

Since some compilers allow you to adjust parameters which effectively
makes them a different implementation of C for each different set of
values for those parameters, so even the number of different
implementations of C that can be provided by a single compiler may be
too large to allow practical testing with every single one of them. gcc,
for example, has a huge number of options - I'm not sure it's reasonable
to perform a test using every possible combination of those options,
even if you restrict yourself to the combinations in which it's a fully
conforming implementation of C (which is a small sub-set of the entire
range of possibilities).

> all of them in a single C compiler.  But it's not so absurd to start 
> from a C compiler implementation (even if it is poorly optimized) and 
> include, step by step, others.
> 
> Even gcc supports the option -m32/-m64 that lets to set the desired size 
> of integer type between 32 and 64 bits.
> 
> The "small script" that I mentioned could be:
> 
> gcc -m32 main.c -o main32
> gcc -m64 main.c -o main64
> ./main32 <input.txt >output32.txt
> ./main64 <input.txt >output64.txt
> diff output32.txt output64.txt
> 
> or similar.  Of course the "small script" should consider just a small 
> set of C implementations of interest, depending on the application I'm 
> testing.

Yes, and that's my key point - its the small set of C implementations of
interest to you; it might be of great interest to other people as well,
and of quite negligible interest to a great many other people, because
they use different implementations.

>> As soon as you select a subset of all possible C implementations that is
>> small enough to make such testing feasible, you are talking about an
>> automatic test procedure whose details are highly dependent upon which
>> subset you've chosen. Such a procedure could be of considerable interest
>> to those who would have selected a sufficiently similar subset, while
>> being of little or no interest in those who'd prefer a significantly
>> different subset.
> 
> This is true, but if we have a C compiler that has customizable size of 
> integer types and configurable numerical representation (little-, big- 
> or other representations), it will be a good starting point.
> Of course, there are C implementations with a different size of chars or 
> one's complement negative number representation, but they are very 
> uncommon so they could be ignored.

Except, of course, by those who are using them (if any). one's
complement is pretty rare nowadays; but implementations with 16-bit char
are significantly more common (while still being uncommon).

>> To be useful, the test should also, ideally, cover all possible sets of
>> input data - a space that usually has a very high dimensionality, and a
>> huge volume. For most non-trivial programs it's absolutely, completely,
>> infeasible to cover that entire space. The best that can be done is a
>> finite sample of the possible sets. Unless the automated system has
>> intelligence comparable to that of the professional tester, it's stuck
>> with randomly searching the space of possible input sets, which is
>> unlikely to find all the special cases that a professional tester could
>> identify as worthy of special attention.
> 
> I don't know how you write and test functions.  I know application 
> testing is a different matter of study, but I usually write the function 
> and try to make tests with some (and not all) inputs.  The most evident 
> errors will come out immediately.
> The goal of my test on different C implementation is to catch these 
> evident errors and not to througly test the entire complex application.

Software testing is an entire discipline in it's own right; and
competent software testers can earn good money; more money than my
company's client is willing to pay for, on my current project. I know
enough to put together and execute a decent test plan; but am seldom
given enough time to perform such a test; I usually have to settle for
less testing than I'd like to perform.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#18818

On 3/15/2012 6:09 PM, Jorgen Grahn wrote:
> On Sun, 2012-03-11, pozz wrote:
>>[...]
>> I'm asking if there's a way to test a small C source (a function or a
>> small set of functions) against all possible C implementations.  Of
>> course I'm thinking of an automatic test, because the number of C
>> implementations is high.
>>  [...]
> Most/all of the responses have been surprisingly negative, so here's
> one in favor.
>
> Yes please, I'd like to have something like what you describe. I see
> practical problems, and I don't think I'll ever see such a system, but
> it's not a stupid idea.

     It's not stupid, just frighteningly expensive.  Expensive, as in
"If you have to ask, you can't afford it."  Expensive, as in "All the
money in the world wouldn't buy it."  Expensive, as in "If every
proton in the Universe were a tester who'd been working since the
Big Bang was in beta ..."

     So we search for admittedly inferior but attainable alternatives.

-- 
Eric Sosman
esosman@ieee-dot-org.invalid

[toc] | [prev] | [next] | [standalone]

#18916

On Fri, 2012-03-16, Eric Sosman wrote:
> On 3/15/2012 6:09 PM, Jorgen Grahn wrote:
>> On Sun, 2012-03-11, pozz wrote:
>>>[...]
>>> I'm asking if there's a way to test a small C source (a function or a
>>> small set of functions) against all possible C implementations.  Of
>>> course I'm thinking of an automatic test, because the number of C
>>> implementations is high.
>>>  [...]
>> Most/all of the responses have been surprisingly negative, so here's
>> one in favor.
>>
>> Yes please, I'd like to have something like what you describe. I see
>> practical problems, and I don't think I'll ever see such a system, but
>> it's not a stupid idea.
>
>      It's not stupid, just frighteningly expensive.  Expensive, as in
> "If you have to ask, you can't afford it."  Expensive, as in "All the
> money in the world wouldn't buy it."  Expensive, as in "If every
> proton in the Universe were a tester who'd been working since the
> Big Bang was in beta ..."
>
>      So we search for admittedly inferior but attainable alternatives.

Perhaps I should mention that I looked past his unfortunate choice of
words ("all possible C implementations..."). The idea was too
interesting to be shot down that easily.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#18918

On 3/20/2012 6:18 PM, Jorgen Grahn wrote:
> On Fri, 2012-03-16, Eric Sosman wrote:
>> On 3/15/2012 6:09 PM, Jorgen Grahn wrote:
>>> On Sun, 2012-03-11, pozz wrote:
>>>> [...]
>>>> I'm asking if there's a way to test a small C source (a function or a
>>>> small set of functions) against all possible C implementations.  Of
>>>> course I'm thinking of an automatic test, because the number of C
>>>> implementations is high.
>>>>   [...]
>>> Most/all of the responses have been surprisingly negative, so here's
>>> one in favor.
>>>
>>> Yes please, I'd like to have something like what you describe. I see
>>> practical problems, and I don't think I'll ever see such a system, but
>>> it's not a stupid idea.
>>
>>       It's not stupid, just frighteningly expensive.  Expensive, as in
>> "If you have to ask, you can't afford it."  Expensive, as in "All the
>> money in the world wouldn't buy it."  Expensive, as in "If every
>> proton in the Universe were a tester who'd been working since the
>> Big Bang was in beta ..."
>>
>>       So we search for admittedly inferior but attainable alternatives.
>
> Perhaps I should mention that I looked past his unfortunate choice of
> words ("all possible C implementations..."). The idea was too
> interesting to be shot down that easily.

     Even though "all possible" is easily and trivially shot down,
I believe that even "all of interest" is also expensive beyond
practicality.  It is certainly expensive beyond usefulness, that is,
more expensive than alternative approaches for assessing portability.

     And, once all the tests on all the platforms using all the
compilers with all their combinations of option flags have run and
passed, you're still left with Dijkstra's observation.  Pozz' idea
is interesting in a sort of mathematical existence-proof way, but I
don't believe it leads to a practical outcome.

-- 
Eric Sosman
esosman@ieee-dot-org.invalid

[toc] | [prev] | [next] | [standalone]

#18944

On Wed, 2012-03-21, Eric Sosman wrote:
> On 3/20/2012 6:18 PM, Jorgen Grahn wrote:
>> On Fri, 2012-03-16, Eric Sosman wrote:
>>> On 3/15/2012 6:09 PM, Jorgen Grahn wrote:
>>>> On Sun, 2012-03-11, pozz wrote:
>>>>> [...]
>>>>> I'm asking if there's a way to test a small C source (a function or a
>>>>> small set of functions) against all possible C implementations.  Of
>>>>> course I'm thinking of an automatic test, because the number of C
>>>>> implementations is high.
>>>>>   [...]
>>>> Most/all of the responses have been surprisingly negative, so here's
>>>> one in favor.
>>>>
>>>> Yes please, I'd like to have something like what you describe. I see
>>>> practical problems, and I don't think I'll ever see such a system, but
>>>> it's not a stupid idea.
>>>
>>>       It's not stupid, just frighteningly expensive.  Expensive, as in
>>> "If you have to ask, you can't afford it."  Expensive, as in "All the
>>> money in the world wouldn't buy it."  Expensive, as in "If every
>>> proton in the Universe were a tester who'd been working since the
>>> Big Bang was in beta ..."
>>>
>>>       So we search for admittedly inferior but attainable alternatives.
>>
>> Perhaps I should mention that I looked past his unfortunate choice of
>> words ("all possible C implementations..."). The idea was too
>> interesting to be shot down that easily.
>
>      Even though "all possible" is easily and trivially shot down,
> I believe that even "all of interest" is also expensive beyond
> practicality.  It is certainly expensive beyond usefulness, that is,
> more expensive than alternative approaches for assessing portability.

Now we're getting somewhere.

I don't see why "all of interest" would be impractical. I think I can
enumerate them (and I think others did upthread). Let's take 
x86_64 as a base and change these one by one:

- no struct padding
- excessive struct padding
- signed/unsigned 'char'
- 16 bit char
- 16 bit int
- 64 bit int
- one's complement
- big-endian
- unaligned access is fatal

That's less than a dozen, and you don't have to test all combinations.
If you're programming against e.g. POSIX rather than ISO C, the
interesting ones are even fewer.

I don't know how such a system would work in practice. It would
probably be a C interpreter rather than a compiler + emulator, and it
would need OS interfaces so you could at least use things like
malloc() and file I/O in your tests.

>      And, once all the tests on all the platforms using all the
> compilers with all their combinations of option flags have run and
> passed, you're still left with Dijkstra's observation.

You probably mean "testing can only show the presence of bugs".
Of course -- but I don't see that as a reason to skip testing.

  (This discussion is a bit ironic: I strongly dislike test-driven
  development and what seems to me like a trend to hack something
  together without understanding, and modifying it until the tests
  pass. When I argue against such things, I always seem to be in the
  minority ... and now I'm somehow on the other side. Oh well.)

> Pozz' idea
> is interesting in a sort of mathematical existence-proof way, but I
> don't believe it leads to a practical outcome.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#18945

On 03/22/2012 05:48 PM, Jorgen Grahn wrote:
> On Wed, 2012-03-21, Eric Sosman wrote:
>> On 3/20/2012 6:18 PM, Jorgen Grahn wrote:
...
>>> Perhaps I should mention that I looked past his unfortunate choice of
>>> words ("all possible C implementations..."). The idea was too
>>> interesting to be shot down that easily.
>>
>>      Even though "all possible" is easily and trivially shot down,
>> I believe that even "all of interest" is also expensive beyond
>> practicality.  It is certainly expensive beyond usefulness, that is,
>> more expensive than alternative approaches for assessing portability.
> 
> Now we're getting somewhere.
> 
> I don't see why "all of interest" would be impractical. I think I can
> enumerate them (and I think others did upthread). Let's take 
> x86_64 as a base and change these one by one:

x86_64 is not a C compiler; it's a platform on which many different
operating systems can run, and for each operating system, there's many
different C compilers that can run on that platform. Those C compilers
can vary in their characteristics for at least some of the items you
listed (and many others you didn't list).

> - no struct padding
> - excessive struct padding

That's not a single alternative to "no struct padding"; it's an infinite
multidimensional family of alternatives, depending upon the many
different ways padding can be inserted.

> - signed/unsigned 'char'
> - 16 bit char
> - 16 bit int
> - 64 bit int
> - one's complement
> - big-endian
> - unaligned access is fatal
> 
> That's less than a dozen, and you don't have to test all combinations.

Why not? It's trivial to come up with examples that fail only for
particular combinations of those differences, and not for any single
difference from whichever compiler "x86_64" was meant to refer to.

Every item you list has to do with differences in the way types are
represented, and all but the first two items have to do with how
integral types are represented - they come almost exclusively from
section 6.2.6. Are you really suggesting that such variations are the
only ones allowed by the C standard that are "of interest"?  I recommend
reviewing Annex J "Portability Issues" before answering that question.
Are NONE of the items on that list that come from outside section 6.2.6
"of interest"?

[toc] | [prev] | [next] | [standalone]

#18948

On 3/22/2012 5:48 PM, Jorgen Grahn wrote:
> On Wed, 2012-03-21, Eric Sosman wrote:
>> On 3/20/2012 6:18 PM, Jorgen Grahn wrote:
>>> On Fri, 2012-03-16, Eric Sosman wrote:
>>>> On 3/15/2012 6:09 PM, Jorgen Grahn wrote:
>>>>> On Sun, 2012-03-11, pozz wrote:
>>>>>> [...]
>>>>>> I'm asking if there's a way to test a small C source (a function or a
>>>>>> small set of functions) against all possible C implementations.  Of
>>>>>> course I'm thinking of an automatic test, because the number of C
>>>>>> implementations is high.
>>>>>>    [...]
>>>>> Most/all of the responses have been surprisingly negative, so here's
>>>>> one in favor.
>>>>>
>>>>> Yes please, I'd like to have something like what you describe. I see
>>>>> practical problems, and I don't think I'll ever see such a system, but
>>>>> it's not a stupid idea.
>>>>
>>>>        It's not stupid, just frighteningly expensive.  Expensive, as in
>>>> "If you have to ask, you can't afford it."  Expensive, as in "All the
>>>> money in the world wouldn't buy it."  Expensive, as in "If every
>>>> proton in the Universe were a tester who'd been working since the
>>>> Big Bang was in beta ..."
>>>>
>>>>        So we search for admittedly inferior but attainable alternatives.
>>>
>>> Perhaps I should mention that I looked past his unfortunate choice of
>>> words ("all possible C implementations..."). The idea was too
>>> interesting to be shot down that easily.
>>
>>       Even though "all possible" is easily and trivially shot down,
>> I believe that even "all of interest" is also expensive beyond
>> practicality.  It is certainly expensive beyond usefulness, that is,
>> more expensive than alternative approaches for assessing portability.
>
> Now we're getting somewhere.
>
> I don't see why "all of interest" would be impractical. I think I can
> enumerate them (and I think others did upthread). Let's take
> x86_64 as a base and change these one by one:
>
> - no struct padding
> - excessive struct padding

     Two.  (Three?  No matter: I'll ignore "typical" padding,
whatever that might be, giving you the benefit of the doubt.)

> - signed/unsigned 'char'

     Times two is four.

> - 16 bit char

     I presume you mean "as opposed to 8-bit char," so again:
Times two is eight.

> - 16 bit int
> - 64 bit int

     With the assumption of 32-bit int as another possibility:
Times three is twenty-four.

> - one's complement

     Versus two's complement and possibly signed magnitude:
Times two is forty-eight or times three is seventy-two.

> - big-endian

     Times two is ninety-six or one hundred forty-four.

> - unaligned access is fatal

     Here it gets harder to enumerate: What's "unaligned," and
for which types?  But even there's only one type in question,
you've reached one hundred ninety-two to two hundred eighty-eight
combinations.

> That's less than a dozen, and you don't have to test all combinations.

     That's sixteen to twenty-four dozen, if not more, and you've
not yet disclosed a criterion for eliminating the uninteresting 
combinations.  Nor have you even touched on the universe of floating-
point, nor the <time.h> facilities, nor ...

> If you're programming against e.g. POSIX rather than ISO C, the
> interesting ones are even fewer.

     POSIX insists on eight-bit `char', I believe, so that cuts the
possibilities in half: eight to twelve dozen, times "unaligned."
(I'm not sure whether POSIX -- a *family* of standards, BTW --
eliminates other possibilities as well: ones' complement, for example.)

> I don't know how such a system would work in practice. It would
> probably be a C interpreter rather than a compiler + emulator, and it
> would need OS interfaces so you could at least use things like
> malloc() and file I/O in your tests.

     I, too, confess that I don't "know" how such a system would work
in practice.  But look at the combinatorial explosion -- The phrase
"implementation-defined" appears one hundred seventy-seven times in
ISO/IEC 9899:201x by my possibly inaccurate count. Even if two-thirds
of the appearances are non-normative and two-thirds of what remains is
redundant (i.e., deducible from other appearances), the residual one-
ninth leaves almost twenty "degrees of freedom."  Posit just two
alternatives for each (some, quite clearly, have more) and you've got
about a million possibilities.

     Here's the economic proposition: You have a team that can test
your product "adequately" on a budget of X splonders and Y months.
Under the most favorable (i.e., shallow) interpretation, the testing
program you outline takes ~100X splonders and/or ~100Y months. Will
you or will you not look for approaches that take maybe ~10X and/or
~10Y? Or will you devote your resources to the two (three, five,...)
environments that actually appear in the sales forecasts?

     Also, let us not forget that portability is only one desirable
attribute of a code base.  "Correctness" and "features" and "time
to market" and "efficiency" and "saleability" are some others, and
the pursuit of different objectives can often lead to conflict.  Not
all of those conflicts will be resolved in portability's favor -- so
why are you spending 100X splonders on portability, instead of applying
some of it to the other goals?

-- 
Eric Sosman
esosman@ieee-dot-org.invalid

[toc] | [prev] | [next] | [standalone]

#18820

On Sunday, March 11, 2012 6:27:38 AM UTC-5, pozz wrote:
> We perfectly know that C implementations can be very different in terms of:
> - number of bits per byte
> - number of bytes per int
> - number of bytes per long
> - signed or unsigned char
> - negative number implementation (two's or one's complement)
> - big or little endian or others
> 
> I'm asking if there's a way to test a small C source (a function or a 
> small set of functions) against all possible C implementations.  Of 
> course I'm thinking of an automatic test, because the number of C 
> implementations is high.
> 
> I think this could be done with an ad-hoc C compiler that can be 
> configured (I imagine thorugh command-line parameteres) in several ways. 
>   For example, I can say to the compiler to use signed or unsigned char 
> (when the signedness is not specified), 16-bit ints and 32-bit longs, 
> big-ending representation, two's complement for negative numbers.
> 
> So I can write a small script that runs the compiler several times with 
> different configurations, producing different executables.  Now I can 
> run these exe and compare the results (their outputs) against "the 
> correct output" I prepared before.
> 
> I think this test could be very useful to verify the correctness of our 
> sources not only against the implementation where we are developing, but 
> also for other possible implementations that could be used in the future 
> by the users of our program.

Should be possible if you have all possible C compilers available.  That will also mean having most types of computers they can run on.

There are many variations that do not reduce to simple numbers.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web