Re: Unit testing a compiler

Path	csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!news.misty.com!news.iecc.com!.POSTED!nerds-end
From	Robert A Duff <bobduff@shell01.TheWorld.com>
Newsgroups	comp.compilers
Subject	Re: Unit testing a compiler
Date	Wed, 13 Mar 2013 18:58:42 -0400
Organization	The World Public Access UNIX, Brookline, MA
Lines	48
Sender	johnl@iecc.com
Approved	comp.compilers@iecc.com
Message-ID	<13-03-012@comp.compilers> (permalink)
References	<13-03-009@comp.compilers>
NNTP-Posting-Host	news.iecc.com
Mime-Version	1.0
Content-Type	text/plain; charset=us-ascii
X-Trace	leila.iecc.com 1363216092 8058 64.57.183.58 (13 Mar 2013 23:08:12 GMT)
X-Complaints-To	abuse@iecc.com
NNTP-Posting-Date	Wed, 13 Mar 2013 23:08:12 +0000 (UTC)
Keywords	debug, testing
Posted-Date	13 Mar 2013 19:08:12 EDT
X-submission-address	compilers@iecc.com
X-moderator-address	compilers-request@iecc.com
X-FAQ-and-archives	http://compilers.iecc.com
Xref	csiph.com comp.compilers:873

Show key headers only | View raw

"Rafael R. Sevilla" <dido@imperium.ph> writes:

> I am currently writing a simple byte compiler for a simple dialect of
> Lisp, and am wondering what is the best practice for testing such a
> compiler.  It is certainly possible to run the compiler on some test
> code and then compare the code it generates with some reference code,
> but as the code samples become more complicated that rapidly becomes
> unwieldy.

Right, that won't work.  (Unless you're testing an assembler,
which is a different story.)

>...Would it instead be better to test the compiler by actually
> running the code it generates, and then comparing the results with what
> the code snippet is supposed to evaluate to?

Yes, that's the right answer.

The specification of a high-level language says what programs are
supposed to do, not what machine code is generated by the compiler.
So test what they do -- compare outputs of compiled programs against
expected outputs.  Any difference is a regression worth investigating.

Changes (such as new optimizations) will likely change the machine
code, but if the output of compiled programs is unchanged, that's OK.

>...Any other ideas on how to
> go about testing the code generator?

Lots, but too much to fit in the margin of this post.  ;-)

Here's just one: Every time you fix a bug, add a regression test to
make sure that particular bug doesn't rear its ugly head again.

OK, one more: Every time one of your customers shows you some source
code, turn it into a regression test for your compiler.  Get as much
test code as you can get your hands on.

>...What about when optimizations are
> being performed?

Test with optimization on and off.

Above is about testing that optimization (etc) didn't break something.
That's hard enough, but testing that "optimizations" don't actually
pessimize is even harder.

- Bob

Thread

Unit testing a compiler "Rafael R. Sevilla" <dido@imperium.ph> - 2013-03-13 11:50 +0800
  Re: Unit testing a compiler BGB <cr88192@hotmail.com> - 2013-03-13 02:59 -0500
  Re: Unit testing a compiler Robert A Duff <bobduff@shell01.TheWorld.com> - 2013-03-13 18:58 -0400
  Re: Unit testing a compiler Walter Banks <walter@bytecraft.com> - 2013-03-19 18:02 -0500

csiph-web