Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!news.misty.com!news.iecc.com!.POSTED!nerds-end From: Robert A Duff Newsgroups: comp.compilers Subject: Re: Unit testing a compiler Date: Wed, 13 Mar 2013 18:58:42 -0400 Organization: The World Public Access UNIX, Brookline, MA Lines: 48 Sender: johnl@iecc.com Approved: comp.compilers@iecc.com Message-ID: <13-03-012@comp.compilers> References: <13-03-009@comp.compilers> NNTP-Posting-Host: news.iecc.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: leila.iecc.com 1363216092 8058 64.57.183.58 (13 Mar 2013 23:08:12 GMT) X-Complaints-To: abuse@iecc.com NNTP-Posting-Date: Wed, 13 Mar 2013 23:08:12 +0000 (UTC) Keywords: debug, testing Posted-Date: 13 Mar 2013 19:08:12 EDT X-submission-address: compilers@iecc.com X-moderator-address: compilers-request@iecc.com X-FAQ-and-archives: http://compilers.iecc.com Xref: csiph.com comp.compilers:873 "Rafael R. Sevilla" writes: > I am currently writing a simple byte compiler for a simple dialect of > Lisp, and am wondering what is the best practice for testing such a > compiler. It is certainly possible to run the compiler on some test > code and then compare the code it generates with some reference code, > but as the code samples become more complicated that rapidly becomes > unwieldy. Right, that won't work. (Unless you're testing an assembler, which is a different story.) >...Would it instead be better to test the compiler by actually > running the code it generates, and then comparing the results with what > the code snippet is supposed to evaluate to? Yes, that's the right answer. The specification of a high-level language says what programs are supposed to do, not what machine code is generated by the compiler. So test what they do -- compare outputs of compiled programs against expected outputs. Any difference is a regression worth investigating. Changes (such as new optimizations) will likely change the machine code, but if the output of compiled programs is unchanged, that's OK. >...Any other ideas on how to > go about testing the code generator? Lots, but too much to fit in the margin of this post. ;-) Here's just one: Every time you fix a bug, add a regression test to make sure that particular bug doesn't rear its ugly head again. OK, one more: Every time one of your customers shows you some source code, turn it into a regression test for your compiler. Get as much test code as you can get your hands on. >...What about when optimizations are > being performed? Test with optimization on and off. Above is about testing that optimization (etc) didn't break something. That's hard enough, but testing that "optimizations" don't actually pessimize is even harder. - Bob