Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #27720 > unrolled thread

Hayes tester fails

Back to article view | Back to comp.lang.forth

Contents

  Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-08 02:20 +1100
    Re: Hayes tester fails "Alex McDonald" <blog@rivadpm.com> - 2014-01-07 21:15 +0000
    Re: Hayes tester fails "Alex McDonald" <blog@rivadpm.com> - 2014-01-07 21:20 +0000
    Re: Hayes tester fails anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2014-01-08 14:58 +0000
      Re: Hayes tester fails "Rod Pemberton" <dont_use_email@xnohavenotit.cnm> - 2014-01-08 19:53 -0500
      Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-09 21:49 +1100
        Re: Hayes tester fails anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2014-01-09 12:57 +0000
          Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-11 16:06 +1100
            Re: Hayes tester fails "Rod Pemberton" <dont_use_email@xnohavenotit.cnm> - 2014-01-11 10:28 -0500
            Re: Hayes tester fails anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2014-01-13 16:18 +0000
              Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-15 10:29 +1100
                Re: Hayes tester fails "Alex McDonald" <blog@rivadpm.com> - 2014-01-15 00:00 +0000
                  Re: Hayes tester fails AKK <akk@nospam.org> - 2014-01-15 02:06 +0100
                  Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-19 01:33 +1100
                    Re: Hayes tester fails anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2014-01-18 16:58 +0000
                      Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-21 00:09 +1100
                        Re: Hayes tester fails "Alex McDonald" <blog@rivadpm.com> - 2014-01-20 14:36 +0000
                          Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-25 22:53 +1100
                            Re: Hayes tester fails "Alex McDonald" <blog@rivadpm.com> - 2014-01-25 23:21 +0000
                        Re: Hayes tester fails anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2014-01-21 14:50 +0000
                Re: Hayes tester fails "Rod Pemberton" <dont_use_email@xnohavenotit.cnm> - 2014-01-15 02:40 -0500
                  Re: Hayes tester fails "Elizabeth D. Rather" <erather@forth.com> - 2014-01-14 21:52 -1000
                  Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-19 00:10 +1100
                Re: Hayes tester fails Andrew Haley <andrew29@littlepinkcloud.invalid> - 2014-01-15 03:28 -0600
        Re: Hayes tester fails "Rod Pemberton" <dont_use_email@xnohavenotit.cnm> - 2014-01-11 10:30 -0500
          Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-13 10:45 +1100
    Re: Hayes tester fails Howerd <howerdo@yahoo.co.uk> - 2014-01-09 02:51 -0800
      Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-09 22:12 +1100
        Re: Hayes tester fails Howerd <howerdo@yahoo.co.uk> - 2014-01-09 06:44 -0800
          Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-11 16:14 +1100
            Re: Hayes tester fails Howerd <howerdo@yahoo.co.uk> - 2014-01-11 02:35 -0800
            Re: Hayes tester fails Coos Haak <chforth@hccnet.nl> - 2014-01-11 13:46 +0100
              Re: Hayes tester fails "Ed" <invalid@invalid.com> - 2014-01-13 11:16 +1100

Page 2 of 2 — ← Prev page 1 [2]

#27907

On Tue, 14 Jan 2014 18:29:25 -0500, Ed <invalid@invalid.com> wrote:
> Anton Ertl wrote:
>> "Ed" <invalid@invalid.com> writes:

>>> which will obviously show a positive outcome?
>>
>> What do you mean by that?
>
> What is says.  You ask proposers to supply the tests and the cases to be  
> tested and you are expecting something other than a successful result?
> This is no test but a fait accompli.

I think you've assumed too much.

A test may be successful by design for one group of proposers, and even
another or another, but not necessarily succeed for some others.  That
could indicate an issue with 1) the implementation 2) the test or
3) undefined boundary conditions (etc).

When implementing some of my "sanity" tests, I ran into differences in
boundary issues in a number of Forths.  Did that mean a Forth was incorrect
or my test?  I assumed the latter, although it could've been that Forth.
If the issue is with the Forth, the author should have some way to
encourage him/her/them to ask whether the behavior is correct or not.
A set of tests can do that.


Rod Pemberton

[toc] | [prev] | [next] | [standalone]

#27908

On 1/14/14 9:40 PM, Rod Pemberton wrote:
> On Tue, 14 Jan 2014 18:29:25 -0500, Ed <invalid@invalid.com> wrote:
>> Anton Ertl wrote:
>>> "Ed" <invalid@invalid.com> writes:
> 
>>>> which will obviously show a positive outcome?
>>>
>>> What do you mean by that?
>>
>> What is says.  You ask proposers to supply the tests and the cases to 
>> be tested and you are expecting something other than a successful result?
>> This is no test but a fait accompli.
> 
> I think you've assumed too much.
> 
> A test may be successful by design for one group of proposers, and even
> another or another, but not necessarily succeed for some others.  That
> could indicate an issue with 1) the implementation 2) the test or
> 3) undefined boundary conditions (etc).
> 
> When implementing some of my "sanity" tests, I ran into differences in
> boundary issues in a number of Forths.  Did that mean a Forth was incorrect
> or my test?  I assumed the latter, although it could've been that Forth.
> If the issue is with the Forth, the author should have some way to
> encourage him/her/them to ask whether the behavior is correct or not.
> A set of tests can do that.

The "Hays Test" was originally developed as an adjunct to the Forth94
effort by John Hays, of Johns Hopkins Univ., one of the participants. It
has since been extended and maintained. It's a pretty clean,
comprehensive suite of tests for major features of Standard Forth. So,
it was not developed by proposers per se; it's pretty objective. Any
additions should be (and usually are) subject to group scrutiny.

If you have unexpected results from your independent "sanity tests" you
can always inquire here what expected behavior is and get independent
opinions (possibly more than you want).

Aloha,
Elizabeth

-- 
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

[toc] | [prev] | [next] | [standalone]

#27931

Rod Pemberton wrote:
> On Tue, 14 Jan 2014 18:29:25 -0500, Ed <invalid@invalid.com> wrote:
> > Anton Ertl wrote:
> >> "Ed" <invalid@invalid.com> writes:
>
> >>> which will obviously show a positive outcome?
> >>
> >> What do you mean by that?
> >
> > What is says.  You ask proposers to supply the tests and the cases to be
> > tested and you are expecting something other than a successful result?
> > This is no test but a fait accompli.
>
> I think you've assumed too much.
>
> A test may be successful by design for one group of proposers, and even
> another or another, but not necessarily succeed for some others.  That
> could indicate an issue with 1) the implementation 2) the test or
> 3) undefined boundary conditions (etc).
> ...

To my knowledge '94 never accredited any system or implementation as
that would have required testing, and no test is foolproof.

[toc] | [prev] | [next] | [standalone]

#27909

Ed <invalid@invalid.com> wrote:
> Anton Ertl wrote:
>> "Ed" <invalid@invalid.com> writes:
>> > ...
>> >What does the 200x TC hope to derive from tests written in Hayes format
>> ...
>> The reason why I wanted test cases in proposals where possible is that
>> proposals written in natural language (as is done in the Forth200x
>> process) is not very precise.
> 
> Sometimes that's a good thing.  Faced with a poorly concieved and expressed
> function it may be possible to re-interpret it in a more positive light.  Tests
> kill off that option.

This is either the best or the worst argument for a poorly-worded
specification.  I can't make my mind up.  :-)  

>> >The only rationale I can imagine is that it wants to add the tests
>> >to the Hayes suite.  What message would that send?
>>
>> Sounds like a rethorical question.  So what's your answer?
> 
> Based on what you have said the message from 200x is that tests are
> the way to go and users can rely on them.  

You're committing the fallacy of the excluded middle: the idea that
tests are either completely useless or completely reliable.  In fact,
neither is the case; compliance tests are neither completely reliable
nor useless.  IME they're certainly worth having, and the experience
of writing a torture test forces me to think about boundary
conditions.

> In supplying tests, 200x changes from a language standard to an
> implementation standard.

A test suite does not constrain an implementation any more than a
standard does.

Andrew.

[toc] | [prev] | [next] | [standalone]

#27812

On Thu, 09 Jan 2014 05:49:54 -0500, Ed <invalid@invalid.com> wrote:

> It is my observation Forthers place too much faith in the Hayes tester.

Really?

IMO, most here don't or haven't seem interested in the Hayes tester or
Josh Grams' small-tester.f.

> I make no apology for posting cases wherein the Hayes tester fails.  If  
> it discourages reliance on the tester and its [used] as "proof" that
> systems are Standard compliant, it can only be a good thing.

Of course, compliance with Hayes at least ensures standardized operation
for that which is tested, whether or not it's correct or faulty, i.e.,
all Forths will operate in the same way.

Personally, I found Josh's tester to be useful since it didn't require
as many Forth words to be available to operate as compare to Hayes.
That was useful since my Forth was far from complete at the time.

So, Josh's tester allowed me to do a few things:
1) run Hayes tests,
2) run a few other available test routines in Hayes format such as
those by Gerry Jackson, Anton Ertl, or Dave Williams, and
3) quickly code up a 148 or so very basic "sanity" tests which
compared various standard Forth operations across a few ANS Forths.


Rod Pemberton

[toc] | [prev] | [next] | [standalone]

#27840

Rod Pemberton wrote:
> ...
> Of course, compliance with Hayes at least ensures standardized operation
> for that which is tested, whether or not it's correct or faulty, i.e.,
> all Forths will operate in the same way.

We could call it the Hayes Standard :)

[toc] | [prev] | [next] | [standalone]

#27755

Hi Ed,

It took me a while to understand what you were saying...

How about we add this line to the Hayes test? :

T{ -1 S>D 2 FM/MOD -> -1 1 }T

It covers that case where the divisor has a greater magnitude than the dividend, and the dividend is negative.

I think that any test of a mathematical function should cover all such cases, and maybe the other tests should be reviewed in this light.

Is there a procedure for updating the Hayes test suite?
I notice that it available from Stephen's FLAG repository...

Best regards,
Howerd 

On Tuesday, 7 January 2014 15:20:44 UTC, Ed  wrote:
> -24. 33 FM/MOD . . -1 9  ok
> 
> 
> 
> Now try this buggy definition found in a Forth implementation:
> 
> 
> 
> : FM/MOD ( d1 n1 -- n2 n3 )
> 
>   DUP >R
> 
>   SM/REM
> 
>   OVER 0<> OVER 0< AND IF
> 
>     SWAP R> + SWAP
> 
>     1-
> 
>   ELSE R> DROP THEN ;
> 
> 
> 
> -24. 33 FM/MOD . . 0 -24  ok
> 
> 
> 
> Notice the negative modulus.  That should never happen in floored
> 
> division when the divisor is positive.
> 
> 
> 
> Put the buggy routine through the Hayes tester and watch it pass.

[toc] | [prev] | [next] | [standalone]

#27759

Howerd wrote:
> Hi Ed,
>
> It took me a while to understand what you were saying...
>
> How about we add this line to the Hayes test? :
>
> T{ -1 S>D 2 FM/MOD -> -1 1 }T
>
> It covers that case where the divisor has a greater magnitude than the dividend, and the
> dividend is negative.
> ...

You can do that and Hayes will catch more implementation errors.
What Hayes can't do, is tell you that there are no errors in the functions
it tests.

[toc] | [prev] | [next] | [standalone]

#27765

Hi Ed,

> You can do that and Hayes will catch more implementation errors.
Yes - the case I mentioned is clearly missing at the moment.

> What Hayes can't do, is tell you that there are no errors in the functions
> it tests.
It would be pretty impressive if it could ;-)

Best regards,
Howerd


On Thursday, 9 January 2014 11:12:37 UTC, Ed  wrote:
> Howerd wrote:
> 
> > Hi Ed,
> 
> >
> 
> > It took me a while to understand what you were saying...
> 
> >
> 
> > How about we add this line to the Hayes test? :
> 
> >
> 
> > T{ -1 S>D 2 FM/MOD -> -1 1 }T
> 
> >
> 
> > It covers that case where the divisor has a greater magnitude than the dividend, and the
> 
> > dividend is negative.
> 
> > ...
> 
> 
> 
> You can do that and Hayes will catch more implementation errors.
> 
> What Hayes can't do, is tell you that there are no errors in the functions
> 
> it tests.

[toc] | [prev] | [next] | [standalone]

#27789

Howerd wrote:
> Hi Ed,
>
> > You can do that and Hayes will catch more implementation errors.
> Yes - the case I mentioned is clearly missing at the moment.
>
> > What Hayes can't do, is tell you that there are no errors in the functions
> > it tests.
> It would be pretty impressive if it could ;-)

BTW your test also caught a buggy division I wrote in the 90's which
passed the official Hayes test.  'Two birds with one stone'  Nice.

[toc] | [prev] | [next] | [standalone]

#27803

Hi Ed,

Thanks - glad to be of service :-)

Best regards,
Howerd

On Saturday, 11 January 2014 05:14:27 UTC, Ed  wrote:
> Howerd wrote:
> 
> > Hi Ed,
> 
> >
> 
> > > You can do that and Hayes will catch more implementation errors.
> 
> > Yes - the case I mentioned is clearly missing at the moment.
> 
> >
> 
> > > What Hayes can't do, is tell you that there are no errors in the functions
> 
> > > it tests.
> 
> > It would be pretty impressive if it could ;-)
> 
> 
> 
> BTW your test also caught a buggy division I wrote in the 90's which
> 
> passed the official Hayes test.  'Two birds with one stone'  Nice.

[toc] | [prev] | [next] | [standalone]

#27806

Op Sat, 11 Jan 2014 16:14:27 +1100 schreef Ed:

> Howerd wrote:
>> Hi Ed,
>>
>>> You can do that and Hayes will catch more implementation errors.
>> Yes - the case I mentioned is clearly missing at the moment.
>>
>>> What Hayes can't do, is tell you that there are no errors in the functions
>>> it tests.
>> It would be pretty impressive if it could ;-)
> 
> BTW your test also caught a buggy division I wrote in the 90's which
> passed the official Hayes test.  'Two birds with one stone'  Nice.

In the 1990's I detected a hardware signed division bug in the Cyrix 486DLC 
with the Hayes test ...

-- 
Coos

CHForth, 16 bit DOS applications
http://home.hccnet.nl/j.j.haak/forth.html 

[toc] | [prev] | [next] | [standalone]

#27841

Coos Haak wrote:
> Op Sat, 11 Jan 2014 16:14:27 +1100 schreef Ed:
>
> > Howerd wrote:
> >> Hi Ed,
> >>
> >>> You can do that and Hayes will catch more implementation errors.
> >> Yes - the case I mentioned is clearly missing at the moment.
> >>
> >>> What Hayes can't do, is tell you that there are no errors in the functions
> >>> it tests.
> >> It would be pretty impressive if it could ;-)
> >
> > BTW your test also caught a buggy division I wrote in the 90's which
> > passed the official Hayes test.  'Two birds with one stone'  Nice.
>
> In the 1990's I detected a hardware signed division bug in the Cyrix 486DLC
> with the Hayes test ...

Intel didn't get it right first time either.  FD 12/4 mentions a bug in 8086 IDIV
which produces an interrupt on certain valid results.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.forth

csiph-web