Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #870 > unrolled thread

Unit testing a compiler

Back to article view | Back to comp.compilers

Contents

  Unit testing a compiler "Rafael R. Sevilla" <dido@imperium.ph> - 2013-03-13 11:50 +0800
    Re: Unit testing a compiler BGB <cr88192@hotmail.com> - 2013-03-13 02:59 -0500
    Re: Unit testing a compiler Robert A Duff <bobduff@shell01.TheWorld.com> - 2013-03-13 18:58 -0400
    Re: Unit testing a compiler Walter Banks <walter@bytecraft.com> - 2013-03-19 18:02 -0500

#870 — Unit testing a compiler

I am currently writing a simple byte compiler for a simple dialect of
Lisp, and am wondering what is the best practice for testing such a
compiler.  It is certainly possible to run the compiler on some test
code and then compare the code it generates with some reference code,
but as the code samples become more complicated that rapidly becomes
unwieldy.  Would it instead be better to test the compiler by actually
running the code it generates, and then comparing the results with what
the code snippet is supposed to evaluate to?  Any other ideas on how to
go about testing the code generator?  What about when optimizations are
being performed?

[toc] | [next] | [standalone]

#871

On 3/12/2013 10:50 PM, Rafael R. Sevilla wrote:
> I am currently writing a simple byte compiler for a simple dialect of
> Lisp, and am wondering what is the best practice for testing such a
> compiler.  It is certainly possible to run the compiler on some test
> code and then compare the code it generates with some reference code,
> but as the code samples become more complicated that rapidly becomes
> unwieldy.  Would it instead be better to test the compiler by actually
> running the code it generates, and then comparing the results with what
> the code snippet is supposed to evaluate to?  Any other ideas on how to
> go about testing the code generator?  What about when optimizations are
> being performed?
>

(mostly writing from personal experience).

generally better I think is rather than testing that the output
matches a specific pattern, is testing that the output code runs,
exhibits the expected behavior, and generates the expected output
values.

for example, if the generated code generates an exception, runs into
an infinite loop, generates bad values, ... then the test can be
considered as having failed.

in this sense, the generated code is to some extent a "black box" as
far as the tests are concerned, but the tests can be more concerned
that it works correctly.

the tests may also contain benchmarks, and maybe any generated code
(such as bytecode and/or ASM), ..., can also be dumped to a log so that
it can be looked over if needed.

IME: determining whether the generated code "makes sense" is generally
something which requires human involvement, and it may also make sense
to dump out the data for each stage in the process, to hopefully make it
easier to figure out at which point things went wrong (is the bug in the
parser, bytecode-compiler, the JIT, or somewhere else?).


granted, there are a few possible drawbacks:
edge cases or "cracks" where bugs can hide, generally due to a specific
feature not being tested sufficiently, or possibly at all;
being lazy and putting off running the tests for too long, and then bugs
creep in.

a drawback is that the more complex a language or compiler becomes, the
more exhaustive the tests that are needed to be able to detect bugs,
which although not necessarily mandating minimalism, is a reason to
refrain from throwing too many extra or unnecessary features onto a
language (a feature which may seem like a good idea at one point may
later turn out to have not been such a good idea).

also, it makes sense to make sure things work before worrying too much
about optimizing them. trying to optimize buggy or broken code may often
lead to a mess which is difficult to debug or to fix. it is generally
better to have a naive implementation than a broken one.


granted, my language isn't really all that simple at this point, having
a fair amount of syntactic and semantic similarity with "mainstream OO
languages". IOW: C-like syntax, class/instance, more-or-less statically
typed (still uses tagged references though), ... though, also with a lot
of script-language features as well.

[toc] | [prev] | [next] | [standalone]

#873

"Rafael R. Sevilla" <dido@imperium.ph> writes:

> I am currently writing a simple byte compiler for a simple dialect of
> Lisp, and am wondering what is the best practice for testing such a
> compiler.  It is certainly possible to run the compiler on some test
> code and then compare the code it generates with some reference code,
> but as the code samples become more complicated that rapidly becomes
> unwieldy.

Right, that won't work.  (Unless you're testing an assembler,
which is a different story.)

>...Would it instead be better to test the compiler by actually
> running the code it generates, and then comparing the results with what
> the code snippet is supposed to evaluate to?

Yes, that's the right answer.

The specification of a high-level language says what programs are
supposed to do, not what machine code is generated by the compiler.
So test what they do -- compare outputs of compiled programs against
expected outputs.  Any difference is a regression worth investigating.

Changes (such as new optimizations) will likely change the machine
code, but if the output of compiled programs is unchanged, that's OK.

>...Any other ideas on how to
> go about testing the code generator?

Lots, but too much to fit in the margin of this post.  ;-)

Here's just one: Every time you fix a bug, add a regression test to
make sure that particular bug doesn't rear its ugly head again.

OK, one more: Every time one of your customers shows you some source
code, turn it into a regression test for your compiler.  Get as much
test code as you can get your hands on.

>...What about when optimizations are
> being performed?

Test with optimization on and off.

Above is about testing that optimization (etc) didn't break something.
That's hard enough, but testing that "optimizations" don't actually
pessimize is even harder.

- Bob

[toc] | [prev] | [next] | [standalone]

#876

"Rafael R. Sevilla" wrote:

> I am currently writing a simple byte compiler for a simple dialect of
> Lisp, and am wondering what is the best practice for testing such a
> compiler.  ...

My compiler regression tests are run through a simulator and a report
generated for each test of go/nogo for compile/link and correct
execution as well as code size, ram requirements and execution time
output in a cvs file. This can then be added to a progress summary
spreadsheet to track progress.

w..

[toc] | [prev] | [standalone]

Back to top | Article view | comp.compilers

csiph-web