Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #134869 > unrolled thread

Coroutines in Forth

Back to article view | Back to comp.lang.forth

Contents

  Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-05 23:25 +0100
    Re: Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-05 23:30 +0100
      Re: Coroutines in Forth Paul Rubin <no.email@nospam.invalid> - 2026-04-05 17:16 -0700
        Re: Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-06 19:58 +0100
          Re: Coroutines in Forth Paul Rubin <no.email@nospam.invalid> - 2026-04-22 11:13 -0700
            Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-22 22:05 +0200
        Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-07 13:23 +0200
    Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-06 13:51 +0200
      Re: Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-06 22:20 +0100
        Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-07 13:35 +0200
          Re: Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-07 20:55 +0100
            Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-08 12:34 +0200
              Re: Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-08 12:32 +0100
              non-parsing `TO` considered harmful Ruvim <ruvim.pinka@gmail.com> - 2026-06-04 22:26 +0000
                Re: non-parsing `TO` considered harmful Ruvim <ruvim.pinka@gmail.com> - 2026-06-06 12:54 +0000
                  Re: non-parsing `TO` considered harmful anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-06-06 16:17 +0000
                    Re: non-parsing `TO` considered harmful Ruvim <ruvim.pinka@gmail.com> - 2026-06-06 21:09 +0000
                      Re: non-parsing `TO` considered harmful anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-06-07 13:23 +0000
                        Re: non-parsing `TO` considered harmful Ruvim <ruvim.pinka@gmail.com> - 2026-06-07 17:44 +0000
                          Re: non-parsing `TO` considered harmful anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-06-08 05:04 +0000
                            text translator (was: non-parsing `TO` considered harmful) Ruvim <ruvim.pinka@gmail.com> - 2026-06-08 15:53 +0000
                            Recognizers vs Parsing words (was: non-parsing `TO` considered harmful) Ruvim <ruvim.pinka@gmail.com> - 2026-06-09 10:25 +0000
                              Re: Recognizers vs Parsing words Ruvim <ruvim.pinka@gmail.com> - 2026-06-10 19:45 +0000
                        compilation of compilation (was: non-parsing `TO` considered harmful) Ruvim <ruvim.pinka@gmail.com> - 2026-06-08 07:28 +0000
                        Recognizer API v.2025-09-11 (was: non-parsing `TO` considered harmful) Ruvim <ruvim.pinka@gmail.com> - 2026-06-14 12:41 +0000
                          Re: Recognizer API v.2025-09-11 (was: non-parsing `TO` considered harmful) anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-06-15 17:17 +0000
                            Re: Recognizer API v.2025-09-11 Ruvim <ruvim.pinka@gmail.com> - 2026-06-16 17:34 +0000
                              Re: Recognizer API v.2025-09-11 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-06-17 05:05 +0000
                                Setters and getters in APIs (was: Recognizer API v.2025-09-11) Ruvim <ruvim.pinka@gmail.com> - 2026-06-19 16:17 +0000
                                  Re: Setters and getters in APIs (was: Recognizer API v.2025-09-11) anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-06-20 15:51 +0000
                    Re: non-parsing `TO` considered harmful Stephen Pelc <stephen@vfxforth.com> - 2026-06-07 11:59 +0000
                      Re: non-parsing `TO` considered harmful anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-06-07 15:25 +0000
            Re: Coroutines in Forth Stephen Pelc <stephen@vfxforth.com> - 2026-04-09 10:12 +0000
              non-parsing `TO` considered harmful Ruvim <ruvim.pinka@gmail.com> - 2026-06-04 22:30 +0000
                VFX Forth problems Ruvim <ruvim.pinka@gmail.com> - 2026-06-05 16:24 +0000
              Re: Coroutines in Forth marcel hendrix <mhx@iae.nl> - 2026-07-09 09:50 +0200
                Re: Coroutines in Forth albert@SPENARNC.XS4ALL.NL - 2026-07-10 01:48 +0200
            Re: Coroutines in Forth Stephen Pelc <stephen@vfxforth.com> - 2026-06-07 11:53 +0000
              non-parsing `TO` considered harmful (was: Coroutines in Forth) Ruvim <ruvim.pinka@gmail.com> - 2026-06-07 13:31 +0000
                Re: non-parsing `TO` considered harmful Travis Bemann <tabemann@gmail.com> - 2026-08-28 20:36 -0500
          Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-10 19:01 +0200
            Re: Coroutines in Forth dxf <dxforth@gmail.com> - 2026-04-11 11:54 +1000
              Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-05-01 13:07 +0200
            Re: Coroutines in Forth peter <peter.noreply@tin.it> - 2026-04-11 09:49 +0200
            Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-11 22:03 +0200
              Re: Coroutines in Forth dxf <dxforth@gmail.com> - 2026-04-12 12:49 +1000
                Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-12 12:13 +0200
                  Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-12 17:39 +0200
                    Re: Coroutines in Forth dxf <dxforth@gmail.com> - 2026-04-13 10:54 +1000
                      Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-13 19:24 +0200
                Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-12 13:48 +0200
                Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-13 12:13 +0200
                Re: Coroutines in Forth Paul Rubin <no.email@nospam.invalid> - 2026-04-22 11:18 -0700
                  Re: Coroutines in Forth dxf <dxforth@gmail.com> - 2026-04-23 11:13 +1000
                  Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-23 12:37 +0200
                    Re: Coroutines in Forth Paul Rubin <no.email@nospam.invalid> - 2026-04-24 10:36 -0700
                      Re: Coroutines in Forth dxf <dxforth@gmail.com> - 2026-04-25 12:12 +1000
                        Re: Coroutines in Forth Paul Rubin <no.email@nospam.invalid> - 2026-04-24 23:31 -0700
                          Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-25 10:45 +0200
                          Re: Coroutines in Forth dxf <dxforth@gmail.com> - 2026-04-25 22:06 +1000
                            Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-25 15:11 +0200
                              Re: Coroutines in Forth dxf <dxforth@gmail.com> - 2026-04-26 13:33 +1000
                                Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-26 16:28 +0200
                              Re: Coroutines in Forth Paul Rubin <no.email@nospam.invalid> - 2026-04-25 21:46 -0700
                                FP stack depth limitations (was: Coroutines in Forth) anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-26 05:55 +0000
                                  Re: FP stack depth limitations Paul Rubin <no.email@nospam.invalid> - 2026-04-26 00:28 -0700
                                    Re: FP stack depth limitations dxf <dxforth@gmail.com> - 2026-04-26 19:55 +1000
                                  Re: FP stack depth limitations (was: Coroutines in Forth) peter <peter.noreply@tin.it> - 2026-04-26 09:57 +0200
                                    Re: FP stack depth limitations (was: Coroutines in Forth) albert@spenarnc.xs4all.nl - 2026-04-26 14:34 +0200
                          Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-25 15:01 +0200
                      Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-05-01 13:13 +0200
                    Forth, C, hardware, and programming virtues (was: Coroutines in Forth) anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-25 05:26 +0000
                      Re: Forth, C, hardware, and programming virtues Paul Rubin <no.email@nospam.invalid> - 2026-04-24 23:55 -0700
                        Re: Forth, C, hardware, and programming virtues anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-25 08:21 +0000
                        Re: Forth, C, hardware, and programming virtues albert@spenarnc.xs4all.nl - 2026-04-25 11:27 +0200
                      Re: Forth, C, hardware, and programming virtues Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-25 15:43 +0200
                        Re: Forth, C, hardware, and programming virtues anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-25 17:21 +0000
                          Re: Forth, C, hardware, and programming virtues dxf <dxforth@gmail.com> - 2026-04-26 15:21 +1000
                          Re: Forth, C, hardware, and programming virtues Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-26 15:08 +0200
                        Re: Forth, C, hardware, and programming virtues albert@spenarnc.xs4all.nl - 2026-04-26 00:34 +0200
                          Re: Forth, C, hardware, and programming virtues Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-26 15:10 +0200
                  locals (was: Coroutines in Forth) anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-25 04:47 +0000
                    Re: locals Paul Rubin <no.email@nospam.invalid> - 2026-04-24 23:21 -0700
                      Re: locals anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-25 06:43 +0000
                        Re: locals albert@spenarnc.xs4all.nl - 2026-04-25 11:43 +0200
                          Re: locals anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-25 10:22 +0000
                            Re: locals peter <peter.noreply@tin.it> - 2026-04-25 16:07 +0200
                              Re: locals Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-25 17:38 +0200
                                Re: locals albert@spenarnc.xs4all.nl - 2026-04-26 01:13 +0200
                              Re: locals anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-26 14:03 +0000
                                Re: locals peter <peter.noreply@tin.it> - 2026-04-27 09:31 +0200
                                  Re: locals anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-27 07:53 +0000
                                    Re: locals peter <peter.noreply@tin.it> - 2026-04-27 11:52 +0200
                            Re: locals albert@spenarnc.xs4all.nl - 2026-04-26 00:51 +0200
                        Re: locals Paul Rubin <no.email@nospam.invalid> - 2026-04-25 22:40 -0700
                          Re: locals albert@spenarnc.xs4all.nl - 2026-04-26 14:55 +0200
                          Re: locals anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-26 09:50 +0000
                            Re: locals Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-26 16:22 +0200
                              Re: locals anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-26 17:04 +0000
                                Re: locals dxf <dxforth@gmail.com> - 2026-04-27 11:51 +1000
                                Re: locals Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-28 08:21 +0200
                                Re: locals Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-28 08:22 +0200
                            Re: locals dxf <dxforth@gmail.com> - 2026-04-27 11:12 +1000
                              Re: locals Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-28 14:34 +0200
                              Re: locals Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-28 14:34 +0200
                                Re: locals Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-29 12:44 +0100
                                  Re: locals Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-29 14:37 +0200
                                    Re: locals Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-29 14:44 +0200
                            Re: locals Paul Rubin <no.email@nospam.invalid> - 2026-05-01 23:50 -0700
                              Re: locals anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-05-02 10:34 +0000
                            Re: locals Paul Rubin <no.email@nospam.invalid> - 2026-05-01 23:54 -0700
                              Re: locals dxf <dxforth@gmail.com> - 2026-05-02 17:36 +1000
                                Re: locals Paul Rubin <no.email@nospam.invalid> - 2026-05-02 01:11 -0700
                              Re: locals anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-05-02 15:58 +0000
        Re: Coroutines in Forth Krishna Myneni <krishna.myneni@ccreweb.org> - 2026-04-07 09:28 -0500
          Re: Coroutines in Forth anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2026-04-07 16:12 +0000
            Re: Coroutines in Forth Krishna Myneni <krishna.myneni@ccreweb.org> - 2026-04-07 18:06 -0500
            Re: Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-08 11:43 +0100
              Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-08 13:16 +0200
                Re: Coroutines in Forth Krishna Myneni <krishna.myneni@ccreweb.org> - 2026-04-08 11:47 -0500
                  Re: Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-08 21:48 +0100
                    Re: Coroutines in Forth Krishna Myneni <krishna.myneni@ccreweb.org> - 2026-04-09 07:06 -0500
                      Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-09 16:41 +0200
                        Re: Coroutines in Forth Krishna Myneni <krishna.myneni@ccreweb.org> - 2026-04-09 13:34 -0500
                          Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-10 01:23 +0200
                            Re: Coroutines in Forth Krishna Myneni <krishna.myneni@ccreweb.org> - 2026-04-09 21:34 -0500
                  Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-09 13:01 +0200
                    Re: Coroutines in Forth Krishna Myneni <krishna.myneni@ccreweb.org> - 2026-04-09 07:01 -0500
                      Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-09 16:10 +0200
                        Re: Coroutines in Forth Krishna Myneni <krishna.myneni@ccreweb.org> - 2026-04-09 13:29 -0500
                Re: Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-08 21:26 +0100
            Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-08 14:22 +0200
            Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-28 15:31 +0200
              Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-29 10:49 +0200
                Re: Coroutines in Forth Hans Bezemer <the.beez.speaks@gmail.com> - 2026-04-29 15:22 +0200
          Re: Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-08 11:33 +0100
            Re: Coroutines in Forth albert@spenarnc.xs4all.nl - 2026-04-08 13:07 +0200
              Re: Coroutines in Forth Gerry Jackson <do-not-use@swldwa.uk> - 2026-04-08 22:05 +0100
    Re: Coroutines in Forth albert <albert@spenarnc.xs4all.nl> - 2026-06-21 01:26 +0200

Page 1 of 7  [1] 2 3 4 5 6 7  Next page →

#134869 — Coroutines in Forth

Reposted from the discussion "Euroforth 2025 preliminary proceedings" 
where it had been misposted.

On 20/01/2026 08:35, Paul Rubin wrote:
 > albert@spenarnc.xs4all.nl writes:
 >> If you pass an address a as a tail call is it approximately equal
 >> to coroutines:
 >
 > No I don't think so.  The tail call is just a jump to that address
 > (changes the program counter).  A coroutine jump also has to change the
 > stack pointer.  See the section "Knuth's coroutines" here:
 >
 > https://www.chiark.greenend.org.uk/~sgtatham/coroutines.html
 >
 > Some Forths have a CO primitive that I think is similar.  There is
 > something like it on the Greenarrays processor.

Never having looked at coroutines before I tried implementing the 
examples in the above paper. The resulting code is:

-1 constant EOF

: get-char  ( ca u -- ca' u' [ch | EOF] )
    ?dup 0= if drop EOF exit then
    over c@ >r 1 /string r>
;

: is-alpha ( ch -- f )  'a' 'z' 1+ within  ; \ adequate for testing

create token 32 chars allot
2variable tok

: init-token  ( ca u ) token 0 tok 2!  ;

: add-to-token  ( ch -- )  tok 2@ + c! 1 tok +! ;

: got-token  ( type -- )
    cr if ." Word: " else ." Punct: " then
    tok 2@ type
   init-token
;

: ?got-token  ( ca u ) \ To handle tokens left in the buffer e.g. on EOF
    tok @ if 1 got-token then \ Only needed for WORD tokens
;

\ The consumer
: parser  ( ch -- )
    dup is-alpha
    if
       begin
          add-to-token exit
[:] parser.alpha   ( ch2 )  \ Re-enter here
           dup is-alpha 0=
       until
    then
   ?got-token
    add-to-token
    0 got-token       ( Punctuation token )
    exit
[:] parser.end
    ?got-token
;

\ The producer
: decompressor  ( ca u -- )
    init-token
    begin
       get-char                      ( -- ca'u' ch ) \ or ( -- EOF )
       dup EOF =
       if drop parser.end cr ." End of file" cr exit then
       dup
    while
      dup $FF =
      if drop get-char >r get-char   ( -- ca' u' ch ) ( R: -- len )
         dup parser r> 1
         ?do  dup parser.alpha loop
         drop
       else
          parser
       then
    repeat
;

init-token
s\" \xFF\x03abc%\xFF\x02-&xyz@qwerty" decompressor .s
\ Displays
Word: aaabc
Punct: %
Punct: -
Punct: -
Punct: &
Word: xyz
Punct: @
Word: qwerty
End of file

Switching between the routines is achieved by calls to different entry 
points in the Parser, and by EXIT or ; back to the decompressor. This is 
not Standard Forth and uses an extension that I implemented over 10 
years ago, tested but never used. I cannot remember my motivation for 
doing so perhaps somebody mentioned it as an idea. I'd be pleased if 
somebody could point to any previous use of this technique.

This extension is a word called [:] that behaves like : and can only be 
used in the middle of a standard colon definition. Usage e.g.
    : foo ... [:] bar ... [:] baz ... ;
to create named entry points into colon definition FOO.
Such an entry definition compiles no executable code of its own and 
execution of the enclosing definition simply executes the body of the 
definition as if the alternative entry points weren't there e.g.
    : foo 1 . [:] bar 2 . [:] baz 3 . ;
    foo 1 2 3
executing the alternative entry points executes the code following 
definition of the entry point e.g.
    bar 2 3
    baz 3
Using BAR etc in another colon definition calls that entry point just 
like any standard colon definition to run the code following that entry 
point.
An entry point does not use the control stack in any way, therefore it 
can be positioned inside any control structure.
An entry point becomes visible immediately its definition is complete, 
so it can be called from anywhere in the rest of the enclosing definition.
As a pseudo colon definition an entry point has an execution token that 
can be obtained or used by the usual set of words such as ' POSTPONE etc.
Execution of code following an entry point can be terminated at any time 
using EXIT

In the example above in addition to PARSER there are two other entry 
points PARSER.ALPHA (in the middle of a loop) and PARSER.END

There are other uses for [:] entry points:
1. Coroutines (see example above)

2. Recursion by name e.g.
    : foo ... [:] bar ... bar ...;  \ recurses to BAR, or
    : foo [foo] ... foo ... ;
    The second example can be achieved by using SYNONYM
    e.g. SYNONYM FOO RECURSE but the first can include initialisation

3. Obtain the xt of the current definition, which has been requested a 
few times on c.l.f e.g.
    : x [:] my-xt ['] my-xt ... ;  \ Removes the need for LATEST

4. Generators akin to those of Python with next e.g.
    variable n : foo n ! exit [:] foo.next n @ 1 n +! ;
    1 foo
    foo.next   ( -- 1 )
    foo.next   ( -- 2 )

5. Debugging by inserting entry points at which a definition can be run 
with known test data on the stack

6. Nested colon definitions e.g. a possible use - unsure of its utility.
    : a 1 . [: [:] b 2 . 3 . ;] drop 4 . ;
    b 2 3
    b is effectively a nested colon definition - forbidden by the standard

-- 
Gerry

[toc] | [next] | [standalone]

#134870

On 05/04/2026 23:25, Gerry Jackson wrote:
On 05/04/2026 02:02, Paul Rubin wrote:
 >> Gerry Jackson <do-not-use@swldwa.uk> writes:
 >>> Never having looked at coroutines before I tried implementing the
 >>> examples in the above paper. The resulting code is: ...
 >>

 >> It would take me a while to understand that, but [:] is cool (haven't
 >> seen it before),

That's not surprising as I've never posted anything about [:] before. 
That's not to say that it's a new concept it's just that I'm unaware of 
any previous work that is similar.

 >> and it lets you do something similar to protothreads.

I've never heard of protothreads let alone used them. Apparently they 
are used extremely lightweight, stackless threads designed for 
memory-constrained embedded systems. Th+is sounds like they are 
applicable to Forth systems. Do you have any examples of use of 
protothreads.

 >>
 >> For coroutines in general you should also read this:
 >>
 >> https://doi.org/10.1145/1462166.1462167
 >>
 >> It discusses "stackful" coroutines where each coroutine has a separate
 >> call stack that is preserved across coroutine jumps.  It's similar to
 >> Forth's cooperative multitasking.
 > Paul Rubin <no.email@nospam.invalid> writes:
 >> https://doi.org/10.1145/1462166.1462167
 >
 > Ehh, that article is very theoretical.  I had forgotten what it was
 > like, or maybe confused it with a different article.  The below is
 > probably more readable:
 >
 > https://www.lua.org/doc/jucs04.pdf


Thanks for the link, very interesting. According to that paper the Forth 
implementation in my first post is an asymmetric coroutine as are Lua 
coroutines which they justify in an appendix, stating "it is easy to 
demonstrate that symmetric coroutines can be expressed by asymmetric 
facilities"

According to the paper two fundamental characteristics of a coroutine as 
follows:
  – “the values of data local to a coroutine persist between successive 
calls”;
– “the execution of a coroutine is suspended as control leaves it, only 
to carry on where it left off when control re-enters the coroutine at 
some later stage”

For the first point, in my example data local to the parser is retained 
but in global variables. That can be made local by putting them in a 
separate wordlist. Using Forth local variables doesn't make sense when 
[:] is used. If it did make sense then local variable values wouldn't be 
retained.

For the second point, execution of the parser is certainly suspended but 
it doesn't automatically carry on where it left off when control 
re-enters. The re-entry point has to be called by name. I would think 
that makes it more readable.

-- 
Gerry

[toc] | [prev] | [next] | [standalone]

#134871

Gerry Jackson <do-not-use@swldwa.uk> writes:
> I've never heard of protothreads let alone used them. Apparently they

They are basically a fancier version of Simon Tatham's preprocessor
scheme.  In fact I confused the two.  See:

  https://dunkels.com/adam/pt/index.html

> Thanks for the link, very interesting. According to that paper the
> Forth implementation in my first post is an asymmetric coroutine as
> are Lua coroutines which they justify in an appendix,

I think there's a big difference, which is that in Lua, you can call
something as a coroutine, and it can yield from several levels deep in
function calls and later resume where it left off.  Python originally
couldn't do that, but it gained the ability later.  It's how
asynchronous i/o works in Python.  It also lets you implement
multitasking, by having the tasks avoid any blocking operations and
yield frequently to not hog the cpu.

Forth also often implements something like that, with a cooperative
multitasker API where you PAUSE every so often, and communicate through
mailboxes rather than with return values.  It's equivalent though.

> For the first point, in my example data local to the parser is
> retained but in global variables. That can be made local by putting
> them in a separate wordlist. Using Forth local variables doesn't make
> sense when [:] is used. If it did make sense then local variable
> values wouldn't be retained.

Yes, in the C macro system you use static local variables, somewhatlike
Forth globals.

> The re-entry point has to be called by name. I would think that makes
> it more readable.

Does that mean the caller has to know where the coroutine returns from?
That's more bookkeeping.

[toc] | [prev] | [next] | [standalone]

#134873

On 06/04/2026 01:16, Paul Rubin wrote:
> Gerry Jackson <do-not-use@swldwa.uk> writes:
>> I've never heard of protothreads let alone used them. Apparently they
> 
> They are basically a fancier version of Simon Tatham's preprocessor
> scheme.  In fact I confused the two.  See:
> 
>    https://dunkels.com/adam/pt/index.html
> 
>> Thanks for the link, very interesting. According to that paper the
>> Forth implementation in my first post is an asymmetric coroutine as
>> are Lua coroutines which they justify in an appendix,
> 
> I think there's a big difference, which is that in Lua, you can call
> something as a coroutine, and it can yield from several levels deep in
> function calls and later resume where it left off. 

I presume that means that the coroutine yielding has to have its own 
return and data stacks which I've not considered. If that's the case 
then an alternative to call by name suggested below should be able to 
handle that case.

Python originally
> couldn't do that, but it gained the ability later.  It's how
> asynchronous i/o works in Python.  It also lets you implement
> multitasking, by having the tasks avoid any blocking operations and
> yield frequently to not hog the cpu.
> 
> Forth also often implements something like that, with a cooperative
> multitasker API where you PAUSE every so often, and communicate through
> mailboxes rather than with return values.  It's equivalent though.
> 
...
> 
>> The re-entry point has to be called by name. I would think that makes
>> it more readable.
> 
> Does that mean the caller has to know where the coroutine returns from?
> That's more bookkeeping.

Yes and that's presumably undesirable for multi-tasking. A solution to 
that could use the fact that the [:] entry point has an xt that can be 
stored as data prior to returning (yielding). The best way would be to 
have a variant of [:] that saved the xt of the re-entry point in a 
deferred word. On re-entry the saved xt would be executed to take 
control to the correct place. An alternative would be to pass the 
re-entry back to the caller so the it could execute the xt. But that has 
the disadvantage that the xt has to be remembered by the caller.

-- 
Gerry

[toc] | [prev] | [next] | [standalone]

#134942

Gerry Jackson <do-not-use@swldwa.uk> writes:
> I presume that means that the coroutine yielding has to have its own
> return and data stacks which I've not considered. If that's the case 
> then an alternative to call by name suggested below should be able to
> handle that case.

Yes, that's correct about separate stacks.  I'm not sure about the call
by name alternative.  I think Forth does ok with multi-tasking (when
supported) instead of stackful coroutines.

[toc] | [prev] | [next] | [standalone]

#134944

In article <875x5i6eyj.fsf@nightsong.com>,
Paul Rubin  <no.email@nospam.invalid> wrote:
>Gerry Jackson <do-not-use@swldwa.uk> writes:
>> I presume that means that the coroutine yielding has to have its own
>> return and data stacks which I've not considered. If that's the case
>> then an alternative to call by name suggested below should be able to
>> handle that case.
>
>Yes, that's correct about separate stacks.  I'm not sure about the call
>by name alternative.  I think Forth does ok with multi-tasking (when
>supported) instead of stackful coroutines.

If you swap return stacks, then in my opinion they are no longer
coroutines in Forth. PAUSE doesn't implement coroutines in any
meaningful sense.

"
    \ Switch to hex for the duration of the definition.
    VARIABLE BASE-SAVE
    : HEX:    BASE @ BASE-SAVE ! HEX CO BASE-SAVE @ BASE ! ;
    : .x HEX: . ;
    12 .x
    C OK

"

If you swap data stacks,  coroutines are much less useful.

"
    WANT decorated

    : print-stack  .S CO .S ;

    : add + ;

    ' print-stack ' add decorated

    1 3 add
    S[ 1 3 ] OK
    S[ 4 ] OK
"
-- 
The Chinese government is satisfied with its military superiority over USA.
The next 5 year plan has as primary goal to advance life expectancy
over 80 years, like Western Europe.

[toc] | [prev] | [next] | [standalone]

#134876

In article <874ilpaqp1.fsf@nightsong.com>,
Paul Rubin  <no.email@nospam.invalid> wrote:
>I think there's a big difference, which is that in Lua, you can call
>something as a coroutine, and it can yield from several levels deep in
>function calls and later resume where it left off.  Python originally
>couldn't do that, but it gained the ability later.  It's how
>asynchronous i/o works in Python.  It also lets you implement
>multitasking, by having the tasks avoid any blocking operations and
>yield frequently to not hog the cpu.

I have 100+ cores concurrently. I don't want to yield anything.


Groetjes Albert
-- 
The Chinese government is satisfied with its military superiority over USA.
The next 5 year plan has as primary goal to advance life expectancy
over 80 years, like Western Europe.

[toc] | [prev] | [next] | [standalone]

#134872

In article <10qunhm$1nnbt$1@dont-email.me>,
Gerry Jackson  <do-not-use@swldwa.uk> wrote:
<SNIP>
>6. Nested colon definitions e.g. a possible use - unsure of its utility.
>    : a 1 . [: [:] b 2 . 3 . ;] drop 4 . ;
>    b 2 3
>    b is effectively a nested colon definition - forbidden by the standard

The term forbidden sneaks in probably by implementors that have
absolutely no intention to implement this.
Normal it would be phrased as,
"A standard Forth is not required to accommodate nested definitions.
A program relying on such has an environmental dependancy"
Note that [: .. ;] is in fact nesting definitions. 1]



In ciforth you can load an extension { } [ ]
Your example
    : a 1 . [: [:] b 2 . 3 . ;] drop 4 . ;
then becomes
    : a 1 . [ : b 2 . 3 . ; ] b drop 4 . ;
(Maybe with :: ;; that unlinks `b from the dictionary
to make `b private.}

>Gerry

1]
Definition or word, or dictionary entry.
Definition: a dictionary entry is a logical part of the dictionary that
bundles the properties of a word/definition.
Definition: It is identified by a handle, the dea (dictionary entry address)
CDFLN SX
It has a code address: for executing the word, native code.
It has a data address: contains a pointer to data (buffer, ITC, DTC,)
It has a flag address: individual bits identify immediate smudged etc.
It has a link address: that defines a relation its wordlist
It has a name address: contains a (pointer to) the name
Extra fields can be added by an implementation.
Example a source address: points to the source of a word^H^H^H^H d.e..
Each and all of this fields can be modified without disturbing
the identity of a dea, e.g.
    "drop" ALLOCATE$ ' DROP >NFA !

It makes no sense to have a word without a name, old view.
OTOH a d.e. where the name address contains a null pointer
makes perfect sense, new view.
Using dea as an xt :
EXECUTE  ( dea -- ?? ) jump to the address found via the cfa.
This can be accomplished while the d.e. is not (yet) linked
into a wordlist, or floating in ALLOCATEd space, or temporary
in a spare dictionary area that is about to be destroyed,
or even if the "word" has no name.

Groetjes Albert
-- 
The Chinese government is satisfied with its military superiority over USA.
The next 5 year plan has as primary goal to advance life expectancy
over 80 years, like Western Europe.

[toc] | [prev] | [next] | [standalone]

#134874

On 06/04/2026 12:51, albert@spenarnc.xs4all.nl wrote:
> In article <10qunhm$1nnbt$1@dont-email.me>,
> Gerry Jackson  <do-not-use@swldwa.uk> wrote:
> <SNIP>
>> 6. Nested colon definitions e.g. a possible use - unsure of its utility.
>>     : a 1 . [: [:] b 2 . 3 . ;] drop 4 . ;
>>     b 2 3
>>     b is effectively a nested colon definition - forbidden by the standard
> 
> The term forbidden sneaks in probably by implementors that have
> absolutely no intention to implement this.

Hmm I just scanned the Forth 2012 document to see if nested definition 
were ambiguous or had an environmental dependency and I couldn't find 
anything. I always assumed it wasn't permited - perhaps it is.

> Normal it would be phrased as,
> "A standard Forth is not required to accommodate nested definitions.
> A program relying on such has an environmental dependancy"
> Note that [: .. ;] is in fact nesting definitions. 1]
> 
> In ciforth you can load an extension { } [ ]
> Your example
>      : a 1 . [: [:] b 2 . 3 . ;] drop 4 . ;
> then becomes
>      : a 1 . [ : b 2 . 3 . ; ] b drop 4 . ;

That's interesting but I didn't have the call to b and the drop was 
simply to drop the xt of the quotation.

My system can handle nested definitions e.g.
: a 1 . [ : b 2 . 3 . ; ] 4 . ;
a 1 4 ok
b 2 3 ok

It aso handles other types of definitions inside a colon definition i.e. 
variables etc.


> 
> 1]
> Definition or word, or dictionary entry.
> Definition: a dictionary entry is a logical part of the dictionary that
> bundles the properties of a word/definition.
> Definition: It is identified by a handle, the dea (dictionary entry address)
> CDFLN SX
> It has a code address: for executing the word, native code.
> It has a data address: contains a pointer to data (buffer, ITC, DTC,)
> It has a flag address: individual bits identify immediate smudged etc.
> It has a link address: that defines a relation its wordlist
> It has a name address: contains a (pointer to) the name
> Extra fields can be added by an implementation.
> Example a source address: points to the source of a word^H^H^H^H d.e..
> Each and all of this fields can be modified without disturbing
> the identity of a dea, e.g.
>      "drop" ALLOCATE$ ' DROP >NFA !
> 
> It makes no sense to have a word without a name, old view.
> OTOH a d.e. where the name address contains a null pointer
> makes perfect sense, new view.
> Using dea as an xt :
> EXECUTE  ( dea -- ?? ) jump to the address found via the cfa.
> This can be accomplished while the d.e. is not (yet) linked
> into a wordlist, or floating in ALLOCATEd space, or temporary
> in a spare dictionary area that is about to be destroyed,
> or even if the "word" has no name.
> 
> Groetjes Albert


-- 
Gerry

[toc] | [prev] | [next] | [standalone]

#134877

In article <10r1825$2d8j6$1@dont-email.me>,
Gerry Jackson  <do-not-use@swldwa.uk> wrote:
>On 06/04/2026 12:51, albert@spenarnc.xs4all.nl wrote:
>> In article <10qunhm$1nnbt$1@dont-email.me>,
>> Gerry Jackson  <do-not-use@swldwa.uk> wrote:
>> <SNIP>
>>> 6. Nested colon definitions e.g. a possible use - unsure of its utility.
>>>     : a 1 . [: [:] b 2 . 3 . ;] drop 4 . ;
>>>     b 2 3
>>>     b is effectively a nested colon definition - forbidden by the standard
>>
>> The term forbidden sneaks in probably by implementors that have
>> absolutely no intention to implement this.
>
>Hmm I just scanned the Forth 2012 document to see if nested definition
>were ambiguous or had an environmental dependency and I couldn't find
>anything. I always assumed it wasn't permited - perhaps it is.

You have been bamboozled nu the wording.
No one can forbid an implementation to add a facility like this. The
wording is such to scare implementors like me. Note that this doesn't
interfere at all with compliance to the standard. If it is common
place to add nested compilation they have to add this sooner or later.
The prize is a non-standard program. Point me to a program that
does something useful, and doesn't contain implementation dependant
parts.
Now it is a good time to recapitulate my postings
"The four brackets of the apocalypse."

A good test of whether an architecture is sound, is whether
fullfilling such request is easy.
I'm proud that in all programs I designed,reasonable extension were
easy.

A similar situation applies to "TO must scan". It turns out there
is no standard program that can detect this. It steers implementation
towards a scanning TO.

<SNIP>
>--
>Gerry

Groetjes Albert
-- 
The Chinese government is satisfied with its military superiority over USA.
The next 5 year plan has as primary goal to advance life expectancy
over 80 years, like Western Europe.

[toc] | [prev] | [next] | [standalone]

#134883

On 07/04/2026 12:35, albert@spenarnc.xs4all.nl wrote:
> A similar situation applies to "TO must scan". It turns out there
> is no standard program that can detect this. It steers implementation
> towards a scanning TO.

On the contrary, Ruvim posted some code that is a standard program and 
which distinguises between a parsing TO and one that sets a flag for a 
following VALUE to act on.

I can't find the post but the gist of it was (I think):

1 value v1
2 value v2 immediate
: test  to v2 v1  ;
Running test with
3 test
A parsing TO will set v2 to 3
A flagging TO will execute v2 during compilation of test because it is 
immediate. So test will set v1 to 3 leaving v2 unchanged.

-- 
Gerry

[toc] | [prev] | [next] | [standalone]

#134887

In article <10r3nfo$33464$1@dont-email.me>,
Gerry Jackson  <do-not-use@swldwa.uk> wrote:
>On 07/04/2026 12:35, albert@spenarnc.xs4all.nl wrote:
>> A similar situation applies to "TO must scan". It turns out there
>> is no standard program that can detect this. It steers implementation
>> towards a scanning TO.
>
>On the contrary, Ruvim posted some code that is a standard program and
>which distinguises between a parsing TO and one that sets a flag for a
>following VALUE to act on.
>
>I can't find the post but the gist of it was (I think):
>
>1 value v1
>2 value v2 immediate
>: test  to v2 v1  ;
>Running test with
>3 test
>A parsing TO will set v2 to 3
>A flagging TO will execute v2 during compilation of test because it is
>immediate. So test will set v1 to 3 leaving v2 unchanged.

No it doesn't. It leaves garbage on the stack during compilation,
leading to mostly an error.
I leave it up to the reader whether this counts as a standard program.

I overlooked this clever example.
So I guess my VALUE is non compliant, proven by a contrived test.

It still makes no sense to forbid a flagging implementation.
(Also VALUE's don't make sense, anyway.)

>
>--
>Gerry

Groetjes Albert
-- 
The Chinese government is satisfied with its military superiority over USA.
The next 5 year plan has as primary goal to advance life expectancy
over 80 years, like Western Europe.

[toc] | [prev] | [next] | [standalone]

#134891

On 08/04/2026 11:34, albert@spenarnc.xs4all.nl wrote:
> In article <10r3nfo$33464$1@dont-email.me>,
> Gerry Jackson  <do-not-use@swldwa.uk> wrote:
>> On 07/04/2026 12:35, albert@spenarnc.xs4all.nl wrote:
>>> A similar situation applies to "TO must scan". It turns out there
>>> is no standard program that can detect this. It steers implementation
>>> towards a scanning TO.
>>
>> On the contrary, Ruvim posted some code that is a standard program and
>> which distinguises between a parsing TO and one that sets a flag for a
>> following VALUE to act on.
>>
>> I can't find the post but the gist of it was (I think):
>>
>> 1 value v1
>> 2 value v2 immediate
>> : test  to v2 v1  ;
>> Running test with
>> 3 test
>> A parsing TO will set v2 to 3
>> A flagging TO will execute v2 during compilation of test because it is
>> immediate. So test will set v1 to 3 leaving v2 unchanged.
> 
> No it doesn't. It leaves garbage on the stack during compilation,
> leading to mostly an error.

Do you mean your system does that?

My mistake, I should have written:
A flagging TO should ...

> I leave it up to the reader whether this counts as a standard program.

Contrived test or not it is standard.
> 
> I overlooked this clever example.
As did everybody else until Ruvim devised it.

> So I guess my VALUE is non compliant, proven by a contrived test.

Yes

> 
> It still makes no sense to forbid a flagging implementation.

I have some sympathy with that but it does ensure that TO has to be on 
the same line as the value name and that the value name does occur 
immediately after the TO as there can be multiple TOs in succession or 
other intervening code (provided TO doesn't leave garbage on the stack).
Anyway surely a flagging TO is slower as it has to test the flag?

> (Also VALUE's don't make sense, anyway.)

VALUEs make more sense than some other things in the standard!

-- 
Gerry

[toc] | [prev] | [next] | [standalone]

#135128 — non-parsing `TO` considered harmful

On 2026-04-08 10:34, albert@spenarnc.xs4all.nl wrote:
> In article <10r3nfo$33464$1@dont-email.me>,
> Gerry Jackson  <do-not-use@swldwa.uk> wrote:
>> On 07/04/2026 12:35, albert@spenarnc.xs4all.nl wrote:
>>> A similar situation applies to "TO must scan". It turns out there
>>> is no standard program that can detect this. It steers implementation
>>> towards a scanning TO.
>>
>> On the contrary, Ruvim posted some code that is a standard program and
>> which distinguises between a parsing TO and one that sets a flag for a
>> following VALUE to act on.
>>
>> I can't find the post but the gist of it was (I think):
>>
>> 1 value v1
>> 2 value v2 immediate
>> : test  to v2 v1  ;

Yes, something similar.

>> Running test with
>> 3 test
>> A parsing TO will set v2 to 3

Yes, and this is specified by the standard.

>> A flagging TO will execute v2 during compilation of test because it is
>> immediate. So test will set v1 to 3 leaving v2 unchanged.
> 
> No it doesn't. It leaves garbage on the stack during compilation,
> leading to mostly an error.
> I leave it up to the reader whether this counts as a standard program.

The provided program conforms to the Forth-94 standard and later versions.

A system that fails that test does not conform to the standard with 
respect to `to`.


Historically, there were two approaches to implement `to`: "parsing" and 
"non-parsing" [1]. Forth-94 formally specified the "parsing" approach:

     | ANS Forth explicitly requires that TO must parse,
     | so that TO's effect will be predictable when
     | it is used at the end of the parse area.


OTOH, it disallowed applying the words `postpone` and `[compile]` to 
`to` [2]. Perhaps, this was done as a concession to implementations that 
adhered to the "non-parsing" approach, to prevent behavior variations in 
a standard program caused by deviations in implementations of `to`.

[1] 
<https://forthhub.github.io/forth-sf-net/standard/dpans/dpansa6.htm#A.6.2.2295>
[2] 
https://forthhub.github.io/forth-sf-net/standard/dpans/dpans6.htm#6.2.2295


Here is an example of a program that relies on a parsing `to`, but is 
not compliant due to that very ambiguous condition regarding `postpone`.
Let's introduce a multiple assignment construct of the following form:
`1 2 3  to( a b c )`.


   : ?comp ( -- ) state @ if exit then -14 throw ;

   : equals ( sd2 sd1 -- flag )
     dup 3 pick <> if 2drop 2drop false exit then
     compare 0=
   ;
   : source-offset ( -- u )
     >in @
   ;
   : set-source-offset ( u -- )
     source nip over u< invert if >in ! exit then
     -18 throw \ "parsed string overflow"
   ;

   synonym take-lexeme-maybe parse-name

   : take-lexeme ( "ccc" -- sd )
     take-lexeme-maybe dup if exit then -16 throw
   ;

   : to( ( "ccc<rparen>" -- ) \ " a b c )"
     ?comp  source-offset ( u.offset )
     take-lexeme s" )" equals if drop exit then
     ( u.offset ) recurse ( u.offset )
     source-offset  swap set-source-offset
       postpone to
     set-source-offset
   ; immediate

   \ usage example

   0 value a
   0 value b

   : init-foo ( -- ) 2 3  to( a b ) ;

   init-foo  a . b .  \ it should print "2 3"




> 
> I overlooked this clever example.
> So I guess my VALUE is non compliant, proven by a contrived test.
> 
> It still makes no sense to forbid a flagging implementation.
> (Also VALUE's don't make sense, anyway.)


I recently used an immediate value for conditional compilation in a code 
similar to the following.

   0 value [building-target] immediate  ( -- flag )

   : start-building-target
     ...
     true to [building-target]
   ;

   : some-word
     ...
     [building-target] [if]
     ...
     [then]
     ...
   ;



>> Gerry
> 
> Groetjes Albert

--
Ruvim

[toc] | [prev] | [next] | [standalone]

#135133 — Re: non-parsing `TO` considered harmful

On 2026-06-04 22:26, Ruvim wrote:
> On 2026-04-08 10:34, albert@spenarnc.xs4all.nl wrote:
>> In article <10r3nfo$33464$1@dont-email.me>,
>> Gerry Jackson  <do-not-use@swldwa.uk> wrote:
>>> On 07/04/2026 12:35, albert@spenarnc.xs4all.nl wrote:
>>>> A similar situation applies to "TO must scan". It turns out there
>>>> is no standard program that can detect this. It steers implementation
>>>> towards a scanning TO.
>>>
>>> On the contrary, Ruvim posted some code that is a standard program and
>>> which distinguises between a parsing TO and one that sets a flag for a
>>> following VALUE to act on.
>>>
>>> I can't find the post but the gist of it was (I think):
>>>
>>> 1 value v1
>>> 2 value v2 immediate
>>> : test  to v2 v1  ;
> 
> Yes, something similar.
> 
>>> Running test with
>>> 3 test
>>> A parsing TO will set v2 to 3
> 
> Yes, and this is specified by the standard.
> 
>>> A flagging TO will execute v2 during compilation of test because it is
>>> immediate. So test will set v1 to 3 leaving v2 unchanged.
>>
>> No it doesn't. It leaves garbage on the stack during compilation,
>> leading to mostly an error.
>> I leave it up to the reader whether this counts as a standard program.
> 
> The provided program conforms to the Forth-94 standard and later versions.
> 
> A system that fails that test does not conform to the standard with 
> respect to `to`.
> 
> 
> Historically, there were two approaches to implement `to`: "parsing" and 
> "non-parsing" [1]. Forth-94 formally specified the "parsing" approach:
> 
>      | ANS Forth explicitly requires that TO must parse,
>      | so that TO's effect will be predictable when
>      | it is used at the end of the parse area.
> 
> 
> OTOH, it disallowed applying the words `postpone` and `[compile]` to 
> `to` [2]. Perhaps, this was done as a concession to implementations that 
> adhered to the "non-parsing" approach, to prevent behavior variations in 
> a standard program caused by deviations in implementations of `to`.
> 
> [1] <https://forthhub.github.io/forth-sf-net/standard/dpans/ 
> dpansa6.htm#A.6.2.2295>
> [2] https://forthhub.github.io/forth-sf-net/standard/dpans/ 
> dpans6.htm#6.2.2295
> 
> 
> Here is an example of a program that relies on a parsing `to`, but is 
> not compliant due to that very ambiguous condition regarding `postpone`.
> Let's introduce a multiple assignment construct of the following form:
> `1 2 3  to( a b c )`.
> 
> 
>    : ?comp ( -- ) state @ if exit then -14 throw ;
> 
>    : equals ( sd2 sd1 -- flag )
>      dup 3 pick <> if 2drop 2drop false exit then
>      compare 0=
>    ;
>    : source-offset ( -- u )
>      >in @
>    ;
>    : set-source-offset ( u -- )
>      source nip over u< invert if >in ! exit then
>      -18 throw \ "parsed string overflow"
>    ;
> 
>    synonym take-lexeme-maybe parse-name
> 
>    : take-lexeme ( "ccc" -- sd )
>      take-lexeme-maybe dup if exit then -16 throw
>    ;
> 
>    : to( ( "ccc<rparen>" -- ) \ " a b c )"
>      ?comp  source-offset ( u.offset )
>      take-lexeme s" )" equals if drop exit then
>      ( u.offset ) recurse ( u.offset )
>      source-offset  swap set-source-offset
>        postpone to
>      set-source-offset
>    ; immediate
> 
>    \ usage example
> 
>    0 value a
>    0 value b
> 
>    : init-foo ( -- ) 2 3  to( a b ) ;
> 
>    init-foo  a . b .  \ it should print "2 3"



Do you know a Forth system in which `to` parses the parse area and in 
which the definition for `to(` given above *does not* work?


How do you implement a construct `to( ... )` that works both in 
interpretation state and in compilation state?

Obviously, we should remove `?comp` and store the offsets on the return 
stack.

Also, we could replace `postpone to` with
   `state @ if  postpone to  else  ['] to execute  then`

In classic single-xt systems, things are simpler: it suffices to replace 
it with `['] to execute`.


Interestingly, the Recognizer API does not help in implementing this 
construct.



--
Ruvim

[toc] | [prev] | [next] | [standalone]

#135135 — Re: non-parsing `TO` considered harmful

Ruvim <ruvim.pinka@gmail.com> writes:
>> Here is an example of a program that relies on a parsing `to`, but is 
>> not compliant due to that very ambiguous condition regarding `postpone`.
>> Let's introduce a multiple assignment construct of the following form:
>> `1 2 3  to( a b c )`.
>> 
>> 
>>    : ?comp ( -- ) state @ if exit then -14 throw ;
>> 
>>    : equals ( sd2 sd1 -- flag )
>>      dup 3 pick <> if 2drop 2drop false exit then
>>      compare 0=
>>    ;
>>    : source-offset ( -- u )
>>      >in @
>>    ;
>>    : set-source-offset ( u -- )
>>      source nip over u< invert if >in ! exit then
>>      -18 throw \ "parsed string overflow"
>>    ;
>> 
>>    synonym take-lexeme-maybe parse-name
>> 
>>    : take-lexeme ( "ccc" -- sd )
>>      take-lexeme-maybe dup if exit then -16 throw
>>    ;
>> 
>>    : to( ( "ccc<rparen>" -- ) \ " a b c )"
>>      ?comp  source-offset ( u.offset )
>>      take-lexeme s" )" equals if drop exit then
>>      ( u.offset ) recurse ( u.offset )
>>      source-offset  swap set-source-offset
>>        postpone to
>>      set-source-offset
>>    ; immediate
>> 
>>    \ usage example
>> 
>>    0 value a
>>    0 value b
>> 
>>    : init-foo ( -- ) 2 3  to( a b ) ;
>> 
>>    init-foo  a . b .  \ it should print "2 3"
>
>
>
>Do you know a Forth system in which `to` parses the parse area and in 
>which the definition for `to(` given above *does not* work?

Depending on what you mean by "work".  Anything that contains ?COMP is
deficient by design.

>How do you implement a construct `to( ... )` that works both in 
>interpretation state and in compilation state?

I don't.  As for how someone else could do it, see below.

>Also, we could replace `postpone to` with
>   `state @ if  postpone to  else  ['] to execute  then`

Also deficient by design.

>Interestingly, the Recognizer API does not help in implementing this 
>construct.

One way would be to add an immediate word TO( that changes rec-forth
(the default recognizer sequence; damn renamings) to a special
recognizer.  This special recognizer just pushes every string it
should recognizer to a TO(-stack, except when it recognizes ")".

When it recognizes ")", it restores the original REC-FORTH, takes the
top string "<word>" off the TO(-stack, constructs a string "TO
<word>", and EVALUATEs it (you may try to take precautions such that
the right TO is found, but the standard gives us little to play with
here).  Repeat until the TI(-stack is empty.  Not even non-standard
POSTPONE TO is needed, and it also works with non-parsing TO
implementations.  It does not work if the user has defined TO to mean
something else (the curse of EVALUATE).

However, instead of going for recognizers, you might play the same
trick by letting TO( parse the strings up to ")" and push them on the
TO(-stack.  And that's simpler to implement, so yes, the recognizer
API does not help here.

But then recognizers are not designed for more than a word (the string
recognizer is already a stretch).  So what Gforth has is a REC-TO that
recognizes "-><word>".

So here's the implementation (untested):

: to(
  0 0 2>r begin
     parse-name dup 0= abort" unfinished TO("
     2dup 2>r
     s" )" str= until
  2r> 2drop \ get rid of ")"
  begin
     2r> dup while
         [: "to " type ;] >string-execute evaluate
         \ freeing the strings is left as exercise to the reader
  repeat
  2drop \ get rid of 0 0
; immediate

Gforth has an API for defing words with user-defined TO
<https://net2o.de/gforth/Words-with-user_002ddefined-TO-etc_002e.html>,
but there is currently no proper API for defining words that perform
the function of TO or one of its siblings (+TO etc), in particular
there is no API that would support a user-defined REC-TO or TO(.  This
shows in using internal words like TO-SLOTS in REC-TO.

Given the large differences between TO implementations in systems, I
expect that we will have a hard time (as in: it won't happen)
standardizing TO-related APIs.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: https://forth-standard.org/
EuroForth 2025 proceedings: http://www.euroforth.org/ef25/papers/

[toc] | [prev] | [next] | [standalone]

#135136 — Re: non-parsing `TO` considered harmful

On 2026-06-06 16:17, Anton Ertl wrote:
> Ruvim <ruvim.pinka@gmail.com> writes:
>>> Here is an example of a program that relies on a parsing `to`, but is
>>> not compliant due to that very ambiguous condition regarding `postpone`.
>>> Let's introduce a multiple assignment construct of the following form:
>>> `1 2 3  to( a b c )`.
>>>
>>>
>>>     : ?comp ( -- ) state @ if exit then -14 throw ;
>>>
>>>     : equals ( sd2 sd1 -- flag )
>>>       dup 3 pick <> if 2drop 2drop false exit then
>>>       compare 0=
>>>     ;
>>>     : source-offset ( -- u )
>>>       >in @
>>>     ;
>>>     : set-source-offset ( u -- )
>>>       source nip over u< invert if >in ! exit then
>>>       -18 throw \ "parsed string overflow"
>>>     ;
>>>
>>>     synonym take-lexeme-maybe parse-name
>>>
>>>     : take-lexeme ( "ccc" -- sd )
>>>       take-lexeme-maybe dup if exit then -16 throw
>>>     ;
>>>
>>>     : to( ( "ccc<rparen>" -- ) \ " a b c )"
>>>       ?comp  source-offset ( u.offset )
>>>       take-lexeme s" )" equals if drop exit then
>>>       ( u.offset ) recurse ( u.offset )
>>>       source-offset  swap set-source-offset
>>>         postpone to
>>>       set-source-offset
>>>     ; immediate
>>>
>>>     \ usage example
>>>
>>>     0 value a
>>>     0 value b
>>>
>>>     : init-foo ( -- ) 2 3  to( a b ) ;
>>>
>>>     init-foo  a . b .  \ it should print "2 3"
>>
>>
>>
>> Do you know a Forth system in which `to` parses the parse area and in
>> which the definition for `to(` given above *does not* work?
> 
> Depending on what you mean by "work".  

By "does not work" I mean that the provided program does not behave as 
specified in the usage example (a kind of test).



> Anything that contains ?COMP is deficient by design.

Do you mean that preventing accidentally execution of some word in 
interpretation state (by throwing an exception) is deficient?



>> How do you implement a construct `to( ... )` that works both in
>> interpretation state and in compilation state?
> 
> I don't. As for how someone else could do it, see below.>
>> Also, we could replace `postpone to` with
>>    `state @ if  postpone to  else  ['] to execute  then`
> 
> Also deficient by design.

Do you mean an ambiguous condition on postponing and ticking `to`?

Otherwise, if you mean something other than applying `postpone` to 
immediate words, please, clarify.




>> Interestingly, the Recognizer API does not help in implementing this
>> construct.
> 
> One way would be to add an immediate word TO( that changes rec-forth
> (the default recognizer sequence; damn renamings) to a special
> recognizer.  This special recognizer just pushes every string it
> should recognizer to a TO(-stack, except when it recognizes ")".
> 
> When it recognizes ")", it restores the original REC-FORTH, takes the
> top string "<word>" off the TO(-stack, constructs a string "TO
> <word>", and EVALUATEs it (you may try to take precautions such that
> the right TO is found, but the standard gives us little to play with
> here).  Repeat until the TI(-stack is empty.  Not even non-standard
> POSTPONE TO is needed, and it also works with non-parsing TO
> implementations.  It does not work if the user has defined TO to mean
> something else (the curse of EVALUATE).
> 
> However, instead of going for recognizers, you might play the same
> trick by letting TO( parse the strings up to ")" and push them on the
> TO(-stack.  And that's simpler to implement, 
> so yes, the recognizer API does not help here.

Agreed.



> But then recognizers are not designed for more than a word (the string
> recognizer is already a stretch).  So what Gforth has is a REC-TO that
> recognizes "-><word>".
> 
> So here's the implementation (untested):
> 
> : to(
>    0 0 2>r begin
>       parse-name dup 0= abort" unfinished TO("
>       2dup 2>r
>       s" )" str= until
>    2r> 2drop \ get rid of ")"
>    begin
>       2r> dup while
>           [: "to " type ;] >string-execute evaluate
>           \ freeing the strings is left as exercise to the reader
>    repeat
>    2drop \ get rid of 0 0
> ; immediate


Thus, this implementation is more complex and unhygienic [2], and these 
drawbacks are introduced solely for the sake of a few Forth systems that 
provide non-standard `to`. The cost seems unjustified.

[2] It is unhygienic, as it requires `to` to be present in the context.
See: <https://en.wikipedia.org/wiki/Hygienic_macro>



> 
> Gforth has an API for defing words with user-defined TO
> <https://net2o.de/gforth/Words-with-user_002ddefined-TO-etc_002e.html>,
> but there is currently no proper API for defining words that perform
> the function of TO or one of its siblings (+TO etc), in particular
> there is no API that would support a user-defined REC-TO or TO(.  This
> shows in using internal words like TO-SLOTS in REC-TO.


There could be a basic factor similar to `defer!`:

`execute-setter` Execution: ( any1 xt1 -- )
Set `xt1` to return `any1` on execution. An ambiguous condition exists 
if `xt1` cannot be set to return `any1`.
`xt1` is the execution token of a word created with `value`, `2value`, 
or `fvalue`.



> Given the large differences between TO implementations in systems, I
> expect that we will have a hard time (as in: it won't happen)
> standardizing TO-related APIs.

A side note: I think this is another argument against methods based on 
`TO` or `IS` in the Recognizers API, and in APIs in general.


As far as I can see, all implementations that perform parsing (as 
specified in the standard) meet the reasonable expectations.

But the implementations that do not perform parsing, vary in their 
behavior. It is interesting to consider which other modern systems, 
besides VfxForth, iForth, and ciForth, fall into this category.


In most Forth systems `to` is a parsing word:
<https://github.com/search?q=NOT+is%3Afork+language%3Aforth+%2F%28%5E%7C+%29%3A+to+%2F&type=code>



> 
> - anton

--
Ruvim

[toc] | [prev] | [next] | [standalone]

#135142 — Re: non-parsing `TO` considered harmful

Ruvim <ruvim.pinka@gmail.com> writes:
>On 2026-06-06 16:17, Anton Ertl wrote:
>> Ruvim <ruvim.pinka@gmail.com> writes:
>>>> Here is an example of a program that relies on a parsing `to`, but is
>>>> not compliant due to that very ambiguous condition regarding `postpone`.
>>>> Let's introduce a multiple assignment construct of the following form:
>>>> `1 2 3  to( a b c )`.
>>>>
>>>>
>>>>     : ?comp ( -- ) state @ if exit then -14 throw ;
>>>>
>>>>     : equals ( sd2 sd1 -- flag )
>>>>       dup 3 pick <> if 2drop 2drop false exit then
>>>>       compare 0=
>>>>     ;
>>>>     : source-offset ( -- u )
>>>>       >in @
>>>>     ;
>>>>     : set-source-offset ( u -- )
>>>>       source nip over u< invert if >in ! exit then
>>>>       -18 throw \ "parsed string overflow"
>>>>     ;
>>>>
>>>>     synonym take-lexeme-maybe parse-name
>>>>
>>>>     : take-lexeme ( "ccc" -- sd )
>>>>       take-lexeme-maybe dup if exit then -16 throw
>>>>     ;
>>>>
>>>>     : to( ( "ccc<rparen>" -- ) \ " a b c )"
>>>>       ?comp  source-offset ( u.offset )
>>>>       take-lexeme s" )" equals if drop exit then
>>>>       ( u.offset ) recurse ( u.offset )
>>>>       source-offset  swap set-source-offset
>>>>         postpone to
>>>>       set-source-offset
>>>>     ; immediate
>>>>
>>>>     \ usage example
>>>>
>>>>     0 value a
>>>>     0 value b
>>>>
>>>>     : init-foo ( -- ) 2 3  to( a b ) ;
>>>>
>>>>     init-foo  a . b .  \ it should print "2 3"
>>>
>>>
>>>
>>> Do you know a Forth system in which `to` parses the parse area and in
>>> which the definition for `to(` given above *does not* work?
>> 
>> Depending on what you mean by "work".  
>
>By "does not work" I mean that the provided program does not behave as 
>specified in the usage example (a kind of test).

That is already satisfied by:

: to-b-to-a to b to a ;

: to(
  ')' parse 2drop
  postpone to-b-to-a ; immediate

A much simpler implementation that does not have the deficiency
discussed below.  It also demonstrates that you cannot use a test as a
specification.

>> Anything that contains ?COMP is deficient by design.
>
>Do you mean that preventing accidentally execution of some word in 
>interpretation state (by throwing an exception) is deficient?

Any word that is not the text interpreter and that uses "STATE @" is
deficient.

>>> Also, we could replace `postpone to` with
>>>    `state @ if  postpone to  else  ['] to execute  then`
>> 
>> Also deficient by design.
>
>Do you mean an ambiguous condition on postponing and ticking `to`?

I mean the STATE @, but true, that's not the only deficiency.

>> But then recognizers are not designed for more than a word (the string
>> recognizer is already a stretch).  So what Gforth has is a REC-TO that
>> recognizes "-><word>".
>> 
>> So here's the implementation (untested):
>> 
>> : to(
>>    0 0 2>r begin
>>       parse-name dup 0= abort" unfinished TO("
>>       2dup 2>r
>>       s" )" str= until
>>    2r> 2drop \ get rid of ")"
>>    begin
>>       2r> dup while
>>           [: "to " type ;] >string-execute evaluate
>>           \ freeing the strings is left as exercise to the reader
>>    repeat
>>    2drop \ get rid of 0 0
>> ; immediate
>
>
>Thus, this implementation is more complex and unhygienic [2], and these 
>drawbacks are introduced solely for the sake of a few Forth systems that 
>provide non-standard `to`. The cost seems unjustified.
>
>[2] It is unhygienic, as it requires `to` to be present in the context.
>See: <https://en.wikipedia.org/wiki/Hygienic_macro>

Accidental capture of identifiers is not to only problem of
EVALUATE-based macros, even in this case.  Another is accidental
nonvisibility of TO.  And yet, the EVALUATE-based implementation of
TO( is superior to the one you give above in several aspects:

1) Shorter.  And I would say it is less complex (as evidenced by its
shortness), but you claim that it is more complex without explaining
why you think so.

2) No STATE @ deficiency.

3) Implementable in standard Forth (>STRING-EXECUTE can be implemented
in standard Forth), while the implementation you give above uses
non-standard POSTPONE TO and ['] TO.

>> Gforth has an API for defing words with user-defined TO
>> <https://net2o.de/gforth/Words-with-user_002ddefined-TO-etc_002e.html>,
>> but there is currently no proper API for defining words that perform
>> the function of TO or one of its siblings (+TO etc), in particular
>> there is no API that would support a user-defined REC-TO or TO(.  This
>> shows in using internal words like TO-SLOTS in REC-TO.
>
>
>There could be a basic factor similar to `defer!`:
>
>`execute-setter` Execution: ( any1 xt1 -- )
>Set `xt1` to return `any1` on execution. An ambiguous condition exists 
>if `xt1` cannot be set to return `any1`.
>`xt1` is the execution token of a word created with `value`, `2value`, 
>or `fvalue`.

Gforth has an internal word (TO):

method (to) ( val operation xt -- ) \ gforth-internal paren-to
\G @i{xt} is of a value like word @i{name}.  Stores @i{val} @code{to}
\G @i{name}.  @i{operation} selects between @code{to} (0), @code{+to} (1),
\G @code{addr} (2), @code{action-of} (3) and @code{is} (4).

There is a reason why this is not a supported word.

And for the compilation semantics:

: (to), ( xt -- ) ( generated code: v -- )
    \g in compiled @code{to @i{name}}, xt is that of @i{name}.  This
    \g word generates code for storing v (of type appropriate for
    \g @i{name}) there.  This word is a factor of @code{to}.

Again, not a supported word.

One might use these as follows:

variable to(-sem \ true/false for compilation/interpretation semantics

: to(1 ( ... n -- )
   0 >r begin
      parse-name dup 0= abort" unfinished TO("
      s" )" str= 0= while
         find-name dup 0= -13 and throw >r
   repeat
   begin
      r> dup while
         name>interpret \ check for 0 is left as exercise to the reader
         to(-sem @ if (to), else 0 swap (to) then
   repeat
   drop
; immediate

: to(-int  false to(-sem ! to(1 ;
: to(-comp true  to(-sem ! to(1 ;
' to(-int ' to(-comp interpret/compile: to(

If you think that this is too complex, I agree.  Don't do
multi-parsing words like TO(.  Ideally, don't do parsing words at all.

There is one thing that this TO( cannot do: It does not work as
intended inside ]] ... [[.  And if somebody writes

POSTPONE TO( a b )

they will not get the equivalent of

POSTPONE ->b POSTPONE ->a

Can we deal with this by defining a recognizer for TO(, which could
then use the TRANSLATE-TO designed for REC-TO?

'translate-to' ( n xt - translation  ) gforth-experimental
   xt belongs to a value-flavoured (or defer-flavoured) word, n is the
index into the 'to-table:' for xt (see Words with user-defined TO etc.).
Interpreting run-time: '( ... -- ... )'
Perform the to-action with index n in the 'to-table:' of xt.  Additional
stack effects depend on n and xt.

It is probably possible to do that, but it would be complex: The
recognizer would produce a translation that contains all the xts of
the value-like words, and a TRANSLATE-TO(.  The TRANSLATE-TO( action
for interpreting would have to get all of the xts out of the way.  The
it woul get the xt for the last one and perform TRANSLATE-TO
INTERPRETING; repeat for the next one, and repeat until all the xts
are done.  The action for compiling and postponing could be
implemented in a similar way (but call COMPILING and POSTPONING
instead of INTERPRETING), or maybe simpler because there is no need to
get the xts out of the way.

In any case, that's quite a bit of work.  Maybe someone (Stephen
Pelc?) thinks this demonstrates that the recognizer words are too
limited.  I think it demonstrates that TO( is not a good idea.

>> Given the large differences between TO implementations in systems, I
>> expect that we will have a hard time (as in: it won't happen)
>> standardizing TO-related APIs.
>
>A side note: I think this is another argument against methods based on 
>`TO` or `IS` in the Recognizers API, and in APIs in general.

There are no such words in
<https://forth-standard.org/proposals/recognizer-committee-proposal-2025-09-11?hideDiff#reply-1623>.
REC-FORTH is proposed as a deferred word, but there is no requirement
to use IS on it.  And unlike for value-flavoured words, there are
non-parsing words for dealing with deferred words: DEFER@ and DEFER!.
So if you want to define IS( ... ), there is no need to concern
yourself with non-standard words like (TO).

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: https://forth-standard.org/
EuroForth 2025 proceedings: http://www.euroforth.org/ef25/papers/

[toc] | [prev] | [next] | [standalone]

#135144 — Re: non-parsing `TO` considered harmful

On 2026-06-07 13:23, Anton Ertl wrote:
> Ruvim <ruvim.pinka@gmail.com> writes:
>> On 2026-06-06 16:17, Anton Ertl wrote:
>>> Ruvim <ruvim.pinka@gmail.com> writes:
>>>>> Here is an example of a program that relies on a parsing `to`, but is
>>>>> not compliant due to that very ambiguous condition regarding `postpone`.
>>>>> Let's introduce a multiple assignment construct of the following form:
>>>>> `1 2 3  to( a b c )`.
>>>>>
>>>>>
>>>>>      : ?comp ( -- ) state @ if exit then -14 throw ;
>>>>>
>>>>>      : equals ( sd2 sd1 -- flag )
>>>>>        dup 3 pick <> if 2drop 2drop false exit then
>>>>>        compare 0=
>>>>>      ;
>>>>>      : source-offset ( -- u )
>>>>>        >in @
>>>>>      ;
>>>>>      : set-source-offset ( u -- )
>>>>>        source nip over u< invert if >in ! exit then
>>>>>        -18 throw \ "parsed string overflow"
>>>>>      ;
>>>>>
>>>>>      synonym take-lexeme-maybe parse-name
>>>>>
>>>>>      : take-lexeme ( "ccc" -- sd )
>>>>>        take-lexeme-maybe dup if exit then -16 throw
>>>>>      ;
>>>>>
>>>>>      : to( ( "ccc<rparen>" -- ) \ " a b c )"
>>>>>        ?comp  source-offset ( u.offset )
>>>>>        take-lexeme s" )" equals if drop exit then
>>>>>        ( u.offset ) recurse ( u.offset )
>>>>>        source-offset  swap set-source-offset
>>>>>          postpone to
>>>>>        set-source-offset
>>>>>      ; immediate
>>>>>
>>>>>      \ usage example
>>>>>
>>>>>      0 value a
>>>>>      0 value b
>>>>>
>>>>>      : init-foo ( -- ) 2 3  to( a b ) ;
>>>>>
>>>>>      init-foo  a . b .  \ it should print "2 3"
>>>>
>>>>
>>>>
>>>> Do you know a Forth system in which `to` parses the parse area and in
>>>> which the definition for `to(` given above *does not* work?
>>>
>>> Depending on what you mean by "work".
>>
>> By "does not work" I mean that the provided program does not behave as
>> specified in the usage example (a kind of test).
> 
> That is already satisfied by:
> 
> : to-b-to-a to b to a ;
> 
> : to(
>    ')' parse 2drop
>    postpone to-b-to-a ; immediate
> 

No. I referred to the specific definition for `to(`, have a look again:

   | Do you know a Forth system in which
   | `to` parses the parse area and in which
   | the definition for `to(` given above
   | *does not* work?

Whether the definitions works as expected is checked by
`init-foo  a . b .`

To be tested automatically, two last lines can be written as:

   t{ 0 value a  0 value b  -> }t
   t{ :noname 2 3 to( a b ) ; execute  a b -> 2 3 }t


> A much simpler implementation that does not have the deficiency
> discussed below.  It also demonstrates that you cannot use a test as a
> specification.
> 

I don't use a test as a specification. Actually, I use the whole 
program, including the definition for `to(` to test a Forth system.



>>> Anything that contains ?COMP is deficient by design.
>>
>> Do you mean that preventing accidentally execution of some word in
>> interpretation state (by throwing an exception) is deficient?
> 
> Any word that is not the text interpreter and that uses "STATE @" is
> deficient.
> 

I would classify my word `to(` is a text interpreter. Is it still deficient?



[...]
>>>
>>> So here's the implementation (untested):
>>>
>>> : to(
>>>     0 0 2>r begin
>>>        parse-name dup 0= abort" unfinished TO("
>>>        2dup 2>r
>>>        s" )" str= until
>>>     2r> 2drop \ get rid of ")"
>>>     begin
>>>        2r> dup while
>>>            [: "to " type ;] >string-execute evaluate
>>>            \ freeing the strings is left as exercise to the reader
>>>     repeat
>>>     2drop \ get rid of 0 0
>>> ; immediate
>>
>>
>> Thus, this implementation is more complex and unhygienic [2], and these
>> drawbacks are introduced solely for the sake of a few Forth systems that
>> provide non-standard `to`. The cost seems unjustified.
>>
>> [2] It is unhygienic, as it requires `to` to be present in the context.
>> See: <https://en.wikipedia.org/wiki/Hygienic_macro>
> 
> Accidental capture of identifiers is not to only problem of
> EVALUATE-based macros, even in this case.  Another is accidental
> nonvisibility of TO.  And yet, the EVALUATE-based implementation of
> TO( is superior to the one you give above in several aspects:
> 
> 1) Shorter.  And I would say it is less complex (as evidenced by its
> shortness), but you claim that it is more complex without explaining
> why you think so.


YOur definition for `to(` is visually longer: approximately 27 lexemes 
vs 16 lexemes. But this depends on the basis (it is different). If we 
take the standard words as the basis, I expect the approach with two 
loops and `evaluate` will be longer than the approach with recursion and 
`postpone`.

Also, the approach with two loops contains more conditions than the 
approach with one recursion.


> 
> 2) No STATE @ deficiency.

Your definition also contains `STATE @`, but indirectly, inside 
`evaluate`. In this respect, I see no conceptual differences.


> 
> 3) Implementable in standard Forth (>STRING-EXECUTE can be implemented
> in standard Forth), while the implementation you give above uses
> non-standard POSTPONE TO and ['] TO.
This non-standard use was the main purpose of my implementation. I 
wonder which Forth systems (that provide standard compliant `to`) behave 
in unexpected ways under these conditions.



[...]

Further comments later.



--
Ruvim

[toc] | [prev] | [next] | [standalone]

#135146 — Re: non-parsing `TO` considered harmful

Ruvim <ruvim.pinka@gmail.com> writes:
>On 2026-06-07 13:23, Anton Ertl wrote:
>> Ruvim <ruvim.pinka@gmail.com> writes:

>>> On 2026-06-06 16:17, Anton Ertl wrote:
>>>> Ruvim <ruvim.pinka@gmail.com> writes:
>>>>> Do you know a Forth system in which `to` parses the parse area and in
>>>>> which the definition for `to(` given above *does not* work?
>>>>
>>>> Depending on what you mean by "work".
>>>
>>> By "does not work" I mean that the provided program does not behave as
>>> specified in the usage example (a kind of test).
>> 
>> That is already satisfied by:
>> 
>> : to-b-to-a to b to a ;
>> 
>> : to(
>>    ')' parse 2drop
>>    postpone to-b-to-a ; immediate
>> 
[...]
>Whether the definitions works as expected is checked by
>`init-foo  a . b .`
>
>To be tested automatically, two last lines can be written as:
>
>   t{ 0 value a  0 value b  -> }t
>   t{ :noname 2 3 to( a b ) ; execute  a b -> 2 3 }t

My point stands.

But I see your point, too.  You have some program where your test case
does not behave as you intended on some Forth system.

But what's the point of your question.

If there is some system where TO parses and where your program does
not behave as intended, then what?

If every system with parsing TO behaves for your program as you
intended, then what?  This just shows that POSTPONE TO works as you
intend for this program and these systems, not in general.

>> Any word that is not the text interpreter and that uses "STATE @" is
>> deficient.
>> 
>
>I would classify my word `to(` is a text interpreter. Is it still deficient?

Yes, because it is no text interpreter, and playing Humpty-Dumpty does
not change that.  E.g.,

to( : foo 1 + . ; 2 foo )

does not print 3.

>[...]
>>>>
>>>> So here's the implementation (untested):
>>>>
>>>> : to(
>>>>     0 0 2>r begin
>>>>        parse-name dup 0= abort" unfinished TO("
>>>>        2dup 2>r
>>>>        s" )" str= until
>>>>     2r> 2drop \ get rid of ")"
>>>>     begin
>>>>        2r> dup while
>>>>            [: "to " type ;] >string-execute evaluate
>>>>            \ freeing the strings is left as exercise to the reader
>>>>     repeat
>>>>     2drop \ get rid of 0 0
>>>> ; immediate
>>>
>>>
>>> Thus, this implementation is more complex and unhygienic [2], and these
>>> drawbacks are introduced solely for the sake of a few Forth systems that
>>> provide non-standard `to`. The cost seems unjustified.
>>>
>>> [2] It is unhygienic, as it requires `to` to be present in the context.
>>> See: <https://en.wikipedia.org/wiki/Hygienic_macro>
>> 
>> Accidental capture of identifiers is not to only problem of
>> EVALUATE-based macros, even in this case.
...
>If we 
>take the standard words as the basis, I expect the approach with two 
>loops and `evaluate` will be longer than the approach with recursion and 
>`postpone`.

If you take the standard words as the basis, then you cannot use
POSTPONE TO.

>> 2) No STATE @ deficiency.
>
>Your definition also contains `STATE @`, but indirectly, inside 
>`evaluate`.

Yes, good point, as mentioned: Accidental capture of identifiers is
not to only problem of EVALUATE-based macros, even in this case.  The
problem is that the text interpreter inside the EVALUATE invocations
use the STATE at the run-time of TO(, not at the parsing time.

One way to handle that would be to disallow using ', ['], POSTPONE and
[COMPILE] on TO(, but

1) The Forth standard does not give us a way to enforce that.

2) As your TO( implementation shows, such a restriction can become a
hindrance.

Another way to handle that is to have a recognizer that deals with
TO(.  That solves objection 1) above, but not necessarily objection
2).  However, maybe the user who wants to do something which would
require POSTPONEing or ticking a word TO( can do manage to do what
they want with the recognizer, the translator, or the translator's
actions.

In any case, the interpreting action of TRANSLATE-TO( could EVALUATE
"TO b" and "TO a" in interpret state, while the compiling action could
do that in compile state.  For the postponing action one could store
the translation data with POSTPONE SLITERAL and the like, and postpone
the word that performs the compiling action.

This solves the STATE @ problem of EVALUATE, but the accidental
capture of TO or accidental non-visibility of TO would not be solved.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: https://forth-standard.org/
EuroForth 2025 proceedings: http://www.euroforth.org/ef25/papers/

[toc] | [prev] | [next] | [standalone]

Page 1 of 7  [1] 2 3 4 5 6 7  Next page →

Back to top | Article view | comp.lang.forth

csiph-web