Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #18900 > unrolled thread

Parsers for extensible grammars?

Back to article view | Back to comp.arch.embedded

Contents

  Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-22 12:14 -0700
    Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-22 12:17 -0700
    Re: Parsers for extensible grammars? Tim Wescott <seemywebsite@myfooter.really> - 2014-10-22 15:13 -0500
      Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-22 22:54 -0700
        Re: Parsers for extensible grammars? Stefan Reuther <stefan.news@arcor.de> - 2014-10-23 18:36 +0200
          Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-23 12:10 -0700
        Re: Parsers for extensible grammars? Tim Wescott <seemywebsite@myfooter.really> - 2014-10-23 13:45 -0500
          Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-23 12:15 -0700
        Re: Parsers for extensible grammars? jim.brakefield@ieee.org - 2014-10-24 07:05 -0700
          Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-24 08:04 -0700
    Re: Parsers for extensible grammars? Clifford Heath <no.spam@please.net> - 2014-10-23 08:22 +1100
      Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-22 23:25 -0700
      Re: Parsers for extensible grammars? George Neuner <gneuner2@comcast.net> - 2014-10-23 04:45 -0400
        Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-23 04:14 -0700
          Re: Parsers for extensible grammars? George Neuner <gneuner2@comcast.net> - 2014-10-23 16:49 -0400
            Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-24 07:38 -0700
              Re: Parsers for extensible grammars? George Neuner <gneuner2@comcast.net> - 2014-10-26 03:47 -0400
                Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-26 08:41 -0700
                  Re: Parsers for extensible grammars? George Neuner <gneuner2@comcast.net> - 2014-10-26 14:36 -0400
                    Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-26 12:37 -0700
                      Re: Parsers for extensible grammars? George Neuner <gneuner2@comcast.net> - 2014-10-26 18:54 -0400
          Re: Parsers for extensible grammars? Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> - 2014-10-25 02:02 +0000
            Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-25 01:56 -0700
              Re: Parsers for extensible grammars? Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> - 2014-10-25 11:15 +0000
                Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-25 10:21 -0700
                  Re: Parsers for extensible grammars? Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> - 2014-10-26 01:05 +0000
                    Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-25 19:37 -0700
    Re: Parsers for extensible grammars? Andrew Smallshaw <andrews@sdf.lonestar.org> - 2014-10-23 20:04 +0000
    Re: Parsers for extensible grammars? Les Cargill <lcargill99@comcast.com> - 2014-10-23 22:30 -0500
      Re: Parsers for extensible grammars? rickman <gnuarm@gmail.com> - 2014-10-25 13:36 -0400
        Re: Parsers for extensible grammars? Les Cargill <lcargill99@comcast.com> - 2014-10-25 13:14 -0500
          Re: Parsers for extensible grammars? rickman <gnuarm@gmail.com> - 2014-10-25 14:23 -0400
            Re: Parsers for extensible grammars? Les Cargill <lcargill99@comcast.com> - 2014-10-26 00:50 -0500
            Re: Parsers for extensible grammars? albert@spenarnc.xs4all.nl (Albert van der Horst) - 2014-10-26 19:21 +0000
              Re: Parsers for extensible grammars? rickman <gnuarm@gmail.com> - 2014-10-26 15:29 -0400
    Re: Parsers for extensible grammars? Paul E Bennett <Paul_E.Bennett@topmail.co.uk> - 2014-10-24 09:56 +0100
      Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-24 07:46 -0700
        Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-25 19:54 -0700
          Re: Parsers for extensible grammars? Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> - 2014-10-26 12:15 +0000
            Re: Parsers for extensible grammars? Don Y <this@is.not.me.com> - 2014-10-26 09:02 -0700
              Re: Parsers for extensible grammars? rickman <gnuarm@gmail.com> - 2014-10-26 12:13 -0400
      Re: Parsers for extensible grammars? rickman <gnuarm@gmail.com> - 2014-10-25 13:35 -0400
        Re: Parsers for extensible grammars? Paul E Bennett <Paul_E.Bennett@topmail.co.uk> - 2014-10-26 01:29 +0100
          Re: Parsers for extensible grammars? rickman <gnuarm@gmail.com> - 2014-10-25 21:11 -0400

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#18976

On Sun, 26 Oct 2014 12:37:06 -0700, Don Y <this@is.not.me.com> wrote:

>On 10/26/2014 11:36 AM, George Neuner wrote:
>
>[When I offered the "C interpreter" as an alternative to Forth -- to see
>if the resistance was due to the language or the mechanism -- this was
>the reply!  "If you're going to write code, then why create another
>means of writing code OUTSIDE the system??"]

That shows a lack of imagination.  The reason for providing a
scripting language is obvious - assuming there is any possible use for
scripting in the 1st place.


>> As with most things, deep understanding isn't necessary to use [parser]
>> tools, but a programmer does need at least a passing familiarity with
>> the tool's method to understand what it is trying to do and why it is
>> failing.  In this regard parser generators are less forgiving than the
>> average compiler and it doesn't much matter which method(s) the tool
>> uses - you can make as big a mess with LL(n) as with (LA|S)?LR(1) and
>> quite easily hang yourself using PEG or GLR.
>
>My experiences with lex/yacc were far less "pleasant" than writing
>arbitrarily complex code in a procedural/object language!  Too much
>head scratching as there were interactions "across" the grammar
>that were only apparent in hindsight.  The way I thought of the various
>constructs "as a human being" was very different from the way the tools
>NEEDED me to think of them.  This doesn't appear in *most* languages
>(aside from unfortunate issues like "foo =- 2;")

You would have had a much different experience with a recursive decent
LL tool.  You still would have had problems until you understood the
grammar model, but your errors would have been somewhat easier to
locate.  Some tools like Antlr (particularly using the Antlrworks GUI)
are really helpful for locating problems.


>> Learning to use a parser generator isn't really any harder than
>> learning any other programming language, but the documentation tends
>> to be scarier.  This isn't helped by the tendency of tool developers
>> to minimally document and to relate their wares to existing tools that
>> the user is presumed already to be familiar with.
>
>There tends to be far more abstraction involved.  People seem to have
>a problem with abstraction -- always wanting to know "why do you want
>to do that" or "what are you trying to do".
>
>    "Timmy has three apples.  Mary has one.  How many apples would Timmy have
>    to give to Mary in order for them to each have the same number of apples?"
>
>    "Why should Timmy have to give her ANY?  Why couldn't *Mary* bring the
>    right number of apples to school??"
>
>[Of course, the correct answer is that Timmy should quickly EAT two of his
>apples and the problem then disappears!  :> ]

Mary has 5 kittens.  If she gives 2 kittens to Timmy, how many kittens
will Mary have?

5!  Mary won't give Timmy any kittens.


Your point re: abstraction is valid ... but the response to it is that
the abstraction of a formal grammar is the entire point of using a
parser generator.


>> The only really valid exceptions are very simple interface "languages"
>> and very tiny systems 8-) that can't afford the overhead of a
>> generated parser.
>>   :
>> However, I think even the parser size argument is losing weight as the
>> average "small" system keeps getting bigger (32-bit ARM running Linux
>> toasting bread, etc.).
>
>But even big systems often have "regions" in which severe constraints
>are placed on the implementation.  E.g., many of those "big Linux
>implementation" compress the kernel to economize on FLASH/ROM -- this
>suggests an unwillingness to add "unnecessary resources" for a one-time
>event (decompressing the kernel/system on IPL).  The same sorts of
>arguments can apply to "setting configuration options" ("Can't we just
>set some bitmaps in a hidden corner of FLASH?")

True.  But Linux tries to be all things to all people.  In the
"normal" desktop/server environment, there really is no reason for the
kernel to be compressed - it's only ~100KB and damn few distributions
will even start in less than 256MB.  Besides which, the uncompressed
kernel is there on the disk too - used for paging kernel code if that
is enabled.

And how often is compiling for a small system done *on* the small
system?  That's what cross-development is for and I think you can
count on small developers to understand that (at least for the
foreseeable future).

George

[toc] | [prev] | [next] | [standalone]

#18943

On 2014-10-23, Don Y <this@is.not.me.com> wrote:
>
> I hadn't considered the idea of expressing "new" commands in terms
> of "old"/common commands.
>
> What I am looking for is the ability to define a set of commands
> that will "always" be used/useful.  Then, allow others to add
> to that set of commands to address specific needs that are
> necessary/unique to their environment.  (i.e., the environment
> in which these common commands are deployed)
>
> Contrived example:
>
> MESSAGE <text>
> PAUSE <delay>
> STDIN <device>
> STDOUT <device>
>

[snip]

Are you familiar with how the VMS operating system handles a similar
issue for it's CLI command language ?

The VMS CLI is called DCL and DCL has an extendable command language
which allows users/programmers to add new commands, which follow
certain DCL imposed syntax rules, to the process specific command table.

Unlike with Unix and friends, much of a DCL command line, such as if a
qualifier/option is supported for this command, or if required values
for a option is missing, is actually validated by DCL before the
executable behind the command itself is actually run.

This is possible because each command has a command definition file
which describes what the list of options, their types, and what
combinations are disallowed, for each command. This command definition
file is compiled into a binary form which is directly accessible from
DCL.

While this isn't a direct match for what you are describing, it sounds
pretty close and you might get some ideas from it.

The VMS documentation is online at:

	http://h71000.www7.hp.com/doc/os84_index.html

and you want the first part of the "HP OpenVMS Command Definition,
Librarian, and Message Utilities Manual", which is available from:

	http://h71000.www7.hp.com/doc/82final/6100/aa-qsbde-te.PDF

(Ignore the Message Utilities and Librarian sections; they are not
relevant here.)

Any other manuals mentioned in this manual are also available from
the above os84_index.html link. I recommend you follow the PDF link
for each manual; the HTML documentation is not as well done as it
should be.

Simon.

PS: And yes, this is a part of my day job. :-) (For now...)

-- 
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[toc] | [prev] | [next] | [standalone]

#18946

On 10/24/2014 7:02 PM, Simon Clubley wrote:
> On 2014-10-23, Don Y <this@is.not.me.com> wrote:
>>
>> I hadn't considered the idea of expressing "new" commands in terms
>> of "old"/common commands.
>>
>> What I am looking for is the ability to define a set of commands
>> that will "always" be used/useful.  Then, allow others to add
>> to that set of commands to address specific needs that are
>> necessary/unique to their environment.  (i.e., the environment
>> in which these common commands are deployed)

> [snip]
>
> Are you familiar with how the VMS operating system handles a similar
> issue for it's CLI command language ?

I$DISCARDED$MY$3000AXP$A$DECADE$AGO

> The VMS CLI is called DCL and DCL has an extendable command language
> which allows users/programmers to add new commands, which follow
> certain DCL imposed syntax rules, to the process specific command table.
>
> Unlike with Unix and friends, much of a DCL command line, such as if a
> qualifier/option is supported for this command, or if required values
> for a option is missing, is actually validated by DCL before the
> executable behind the command itself is actually run.

Ah, I didn't realize that!  Interesting approach -- assuming there are no
other ways to invoke binaries that bypass this.

> This is possible because each command has a command definition file
> which describes what the list of options, their types, and what
> combinations are disallowed, for each command. This command definition
> file is compiled into a binary form which is directly accessible from
> DCL.
>
> While this isn't a direct match for what you are describing, it sounds
> pretty close and you might get some ideas from it.

It's worth a closer look!

> The VMS documentation is online at:
>
> 	http://h71000.www7.hp.com/doc/os84_index.html
>
> and you want the first part of the "HP OpenVMS Command Definition,
> Librarian, and Message Utilities Manual", which is available from:
>
> 	http://h71000.www7.hp.com/doc/82final/6100/aa-qsbde-te.PDF
>
> (Ignore the Message Utilities and Librarian sections; they are not
> relevant here.)

Thanks!

> Any other manuals mentioned in this manual are also available from
> the above os84_index.html link. I recommend you follow the PDF link
> for each manual; the HTML documentation is not as well done as it
> should be.
>
> Simon.
>
> PS: And yes, this is a part of my day job. :-) (For now...)
>

[toc] | [prev] | [next] | [standalone]

#18947

On 2014-10-25, Don Y <this@is.not.me.com> wrote:
> On 10/24/2014 7:02 PM, Simon Clubley wrote:
>> The VMS CLI is called DCL and DCL has an extendable command language
>> which allows users/programmers to add new commands, which follow
>> certain DCL imposed syntax rules, to the process specific command table.
>>
>> Unlike with Unix and friends, much of a DCL command line, such as if a
>> qualifier/option is supported for this command, or if required values
>> for a option is missing, is actually validated by DCL before the
>> executable behind the command itself is actually run.
>
> Ah, I didn't realize that!  Interesting approach -- assuming there are no
> other ways to invoke binaries that bypass this.
>

Commands can exist outside of the above infrastructure if one wishes;
they are called foreign commands and are treated pretty much like a
Unix command would be - there's no validation of options prior to
the executable starting and hence there's no nicely pre-parsed options
and option values ready to be read from within the program itself.

Foreign commands are used for example when porting Unix tools to VMS as
it means the existing command line parsing code in the tool can be
used pretty much unchanged.

On VMS, if your executable takes no command line input, you can also
just run the executable with the run command.

However it's your choice - you can have DCL do some of the validation
work for you by having your program integrated into the above DCL
infrastructure or you can do it all yourself in your program just as
you do on Unix and friends.

IOW, having foreign commands available as an option doesn't stop you
from also having the native DCL integrated approach. If you try to run
a DCL integrated executable as a foreign command, no values will be
available to read from within your program so you are forced to run it
via the DCL mechanism above which means you also get the DCL level
validation as well.

Simon.

-- 
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[toc] | [prev] | [next] | [standalone]

#18948

On 10/25/2014 4:15 AM, Simon Clubley wrote:
> On 2014-10-25, Don Y <this@is.not.me.com> wrote:
>> On 10/24/2014 7:02 PM, Simon Clubley wrote:
>>> The VMS CLI is called DCL and DCL has an extendable command language
>>> which allows users/programmers to add new commands, which follow
>>> certain DCL imposed syntax rules, to the process specific command table.
>>>
>>> Unlike with Unix and friends, much of a DCL command line, such as if a
>>> qualifier/option is supported for this command, or if required values
>>> for a option is missing, is actually validated by DCL before the
>>> executable behind the command itself is actually run.
>>
>> Ah, I didn't realize that!  Interesting approach -- assuming there are no
>> other ways to invoke binaries that bypass this.
>
> Commands can exist outside of the above infrastructure if one wishes;
> they are called foreign commands and are treated pretty much like a
> Unix command would be - there's no validation of options prior to
> the executable starting and hence there's no nicely pre-parsed options
> and option values ready to be read from within the program itself.

You can *freely* mark any command as either type?  I.e., can I mark
a command that relies on the DCL stuff for option parsing as a FOREIGN
command?  And, thus, screw it up (at runtime)?

> Foreign commands are used for example when porting Unix tools to VMS as
> it means the existing command line parsing code in the tool can be
> used pretty much unchanged.
>
> On VMS, if your executable takes no command line input, you can also
> just run the executable with the run command.
>
> However it's your choice - you can have DCL do some of the validation
> work for you by having your program integrated into the above DCL
> infrastructure or you can do it all yourself in your program just as
> you do on Unix and friends.
>
> IOW, having foreign commands available as an option doesn't stop you
> from also having the native DCL integrated approach. If you try to run
> a DCL integrated executable as a foreign command, no values will be
> available to read from within your program so you are forced to run it
> via the DCL mechanism above which means you also get the DCL level
> validation as well.

So, this (DCL) mechanism is meant as an *aid*/service -- not as a means
of ensuring software integrity (?).

[toc] | [prev] | [next] | [standalone]

#18954

On 2014-10-25, Don Y <this@is.not.me.com> wrote:
> On 10/25/2014 4:15 AM, Simon Clubley wrote:
>>
>> Commands can exist outside of the above infrastructure if one wishes;
>> they are called foreign commands and are treated pretty much like a
>> Unix command would be - there's no validation of options prior to
>> the executable starting and hence there's no nicely pre-parsed options
>> and option values ready to be read from within the program itself.
>
> You can *freely* mark any command as either type?  I.e., can I mark
> a command that relies on the DCL stuff for option parsing as a FOREIGN
> command?  And, thus, screw it up (at runtime)?
>

Yes, but it would fail in a clean way with a status code returned from
the CLI routine called by the program to return the (non-existent)
pre-parsed information.

The idea behind pointing you to the DCL CLD material was to give you
some possible ideas about how extendable CLIs with validated options
are handled in another environment (ie: VMS).

Like I said at the time, it's not an exact match for your requirements
but I thought you might be interested in seeing how a similar problem
was handled in VMS, including the syntax used in the command definition
file (and compare it to how the problem is fully pushed to the executable
itself in Unix land).

>>
>> IOW, having foreign commands available as an option doesn't stop you
>> from also having the native DCL integrated approach. If you try to run
>> a DCL integrated executable as a foreign command, no values will be
>> available to read from within your program so you are forced to run it
>> via the DCL mechanism above which means you also get the DCL level
>> validation as well.
>
> So, this (DCL) mechanism is meant as an *aid*/service -- not as a means
> of ensuring software integrity (?).
>

It's a way of expressing an expected structure for a command line which
can be somewhat validated by DCL before the program even starts and
provides a robust, operating system level, method for a program to
obtain command line parameters and options in a way (and with
functionality) that leaves getopt and friends standing in the dust.

The relevance here is that I've encountered people who have never
been exposed to the VMS way of handling this and who think that ad-hoc
getopt style functionality is the only _possible_ way to parse command
lines. I just wanted to make reference to another way of doing this in
case it gave you some ideas even though there's probably nothing you
can _directly_ use here.

Sometimes Don you leave your questions wide open, presumably in order
to invite a wide range of options in response, including ones you had
never even considered. The difficulty with that is that sometimes it's
hard to understand what additional unspoken constraints might exist. :-)

Simon.

-- 
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[toc] | [prev] | [next] | [standalone]

#18956

On 10/25/2014 6:05 PM, Simon Clubley wrote:
> On 2014-10-25, Don Y <this@is.not.me.com> wrote:
>> On 10/25/2014 4:15 AM, Simon Clubley wrote:
>>>
>>> Commands can exist outside of the above infrastructure if one wishes;
>>> they are called foreign commands and are treated pretty much like a
>>> Unix command would be - there's no validation of options prior to
>>> the executable starting and hence there's no nicely pre-parsed options
>>> and option values ready to be read from within the program itself.
>>
>> You can *freely* mark any command as either type?  I.e., can I mark
>> a command that relies on the DCL stuff for option parsing as a FOREIGN
>> command?  And, thus, screw it up (at runtime)?
>
> Yes, but it would fail in a clean way with a status code returned from
> the CLI routine called by the program to return the (non-existent)
> pre-parsed information.

Understood.  I.e., the DCL-ed executables aren't *fed* arguments
that the front-end has testified as "valid".  Rather, it *fetches*
that information from the front end.  As such, if the front end wasn't
in place when the executable was invoked, the executable is aware of that.

> The idea behind pointing you to the DCL CLD material was to give you
> some possible ideas about how extendable CLIs with validated options
> are handled in another environment (ie: VMS).

Yes.

> Like I said at the time, it's not an exact match for your requirements
> but I thought you might be interested in seeing how a similar problem
> was handled in VMS, including the syntax used in the command definition
> file (and compare it to how the problem is fully pushed to the executable
> itself in Unix land).

Exactly!

>>> IOW, having foreign commands available as an option doesn't stop you
>>> from also having the native DCL integrated approach. If you try to run
>>> a DCL integrated executable as a foreign command, no values will be
>>> available to read from within your program so you are forced to run it
>>> via the DCL mechanism above which means you also get the DCL level
>>> validation as well.
>>
>> So, this (DCL) mechanism is meant as an *aid*/service -- not as a means
>> of ensuring software integrity (?).
>
> It's a way of expressing an expected structure for a command line which
> can be somewhat validated by DCL before the program even starts and
> provides a robust, operating system level, method for a program to
> obtain command line parameters and options in a way (and with
> functionality) that leaves getopt and friends standing in the dust.
>
> The relevance here is that I've encountered people who have never
> been exposed to the VMS way of handling this and who think that ad-hoc
> getopt style functionality is the only _possible_ way to parse command
> lines. I just wanted to make reference to another way of doing this in
> case it gave you some ideas even though there's probably nothing you
> can _directly_ use here.

Understood.  I'm not fond of the VMS syntax.  But, will be studying
the implementation to see if I can extract some ideas and apply them
to a more suitable syntax.

> Sometimes Don you leave your questions wide open, presumably in order
> to invite a wide range of options in response, including ones you had
> never even considered. The difficulty with that is that sometimes it's
> hard to understand what additional unspoken constraints might exist. :-)

I pass along the minimum constraints under which I am operating.  If I
*add* constraints, then I risk steering solutions into a specific direction
that isn't necessarily required for the problem at hand.  Or, discounting
solutions that could be viable.

I engage clients with the same sort of approach:  "Is what you are
telling me a REQUIREMENT or just your idea of how I *might* approach
the problem?"  The fewer the requirements, the more flexibility in
the solution.  This almost always leads to a more optimized solution.

By contrast, wanting extra ARTIFICIAL constraints boxes the design into
a corner.  "Why are you doing it like that?"  "Because that's how everyone
else does it!" or "Because that's how I *thought* it would be done."

In this case, I asked for a "command parser" that was "extensible".  And,
indicated that those extensions would be implemented at compile time
(so, any solution need not accommodate dynamic modifications).  I also
introduced contrived examples for folks who couldn't think entirely in
the abstract:  "Imagine these were the common commands and you wanted
to add commands LIKE these..."  Anything beyond this would change the
problem definition.

Note that I didn't ask for a "programming language" -- or even a
scripting language (yet, at least one proposed solution fits that
description AND fits my requirement!).

OTOH, if I had augmented my question as "an extensible scripting language",
I would risk YOUR solution potentially disqualifying itself.  :-/

[toc] | [prev] | [next] | [standalone]

#18925

Hi Don

On 2014-10-22, Don Y <this@is.not.me.com> wrote:
>
> Ideally, I want a solution that allows folks developing those
> "other" commands to just "bolt onto" what I have done.  E.g.,
> creating a single "grammar definition" (lex/yacc) is probably
> not a good way to go (IME, most small parsers tend to be ad hoc).
>
> [Note:  I can't even guarantee that the extensions will be
> consistent or "harmonious" with the grammar that I implement]

To be honest I'm not clear on what that last part means.  Adding
new "commands" is relatively straightforward but new syntax patterns
are surely impossible in the general case without revising the
grammar directly.

To clarify that anything you could express inside a C function is
probably straightforward but a new language construct such as a
new form of loop or a switch statement that accepts non-constant
case labels is going to be tricky.  Hell in the truly general case
even basic stuff such as determining where a statement begins and
ends is impossible if arbitrary modifications are permissible.

I did write a parser for the former simpler case a year or two ago:
it's easy enough approached from the right direction at the outset.
The key isn't so much in the parser but the lexer.  In the parser
my (slightly simplified) YYSTYPE was something along the lines of:

%union
{
	struct { char *text; } string;
	struct { char *text; int64_t value; } number;
	struct { char *text; int (*function)(YYSTYPE *params); } command;
}

(If you're wondering why every option has a text argument it's
because any token could be converted to the original input if a
string is expected in context.)

The lexer then proceeded more or less as normal but only a single
pattern matching a generic potential command name i.e. a sequence
of all alphabetic characters.  The candidate command name was then
given to a lookup function that did a binary search of a static
array, but obviously how the lookup function works depends on the
context.  

If no match the lexer returned the token as a string, if there was a
match the function pointer is obtained from that array, along with
a token code typing its parameters, e.g. a command with two numeric
parameters, or a command with a numeric and two string parameters.
That approach means that the "plumbing" to add a new command is
simply an extra line in the array definition.

Things get a little more complex with optional parameters but it
remains practical provided the number of possible combinations
remains smallish.  The parameter code is what yylex() returns as
its result so the parser knows what should follow.  In my case
yylex() was handwritten, but that was so it could directly access
my app's internal representation, there's no reason the same approach
couldn't be done in lex.

If you wanted to completely generalise it I suppose you could have
a syntax rule to convert any possible parameter token to a single
"parameter" term and have a single rule for "command (array of
parameters)" but I'd try and avoid that if possible - it moves
checking from the parser to each individual command.  Still, it's
possibly something to have in reserve for oddball commands that
don't fit a general pattern.

-- 
Andrew Smallshaw
andrews@sdf.lonestar.org

[toc] | [prev] | [next] | [standalone]

#18930

Don Y wrote:
> Hi,
>
> [I probably should direct this at George...  :> ]
>
> I'm writing a command parser.  Some set of commands are "common".
> But, other instances of the parser are augmented with additional
> commands/syntax.
>
> [These additions are known at compile time, not "dynamic"]
>
> Ideally, I want a solution that allows folks developing those
> "other" commands to just "bolt onto" what I have done.  E.g.,
> creating a single "grammar definition" (lex/yacc) is probably
> not a good way to go (IME, most small parsers tend to be ad hoc).
>
> [Note:  I can't even guarantee that the extensions will be
> consistent or "harmonious" with the grammar that I implement]
>
> A naive approach (?) might be for my code to take a crack at
> a particular "statement"/command and, in case of FAIL, invoke
> some bolt-on parser to give it a chance to make sense out of
> the input.  If *it* FAILs, then the input is invalid, etc.
>
> This sounds like an incredible kluge.  Any better suggestions?


If all else fails, strstr() keywords, then have a corresponding
parsers for each keyword.

A better approach is a container (struct) of keywords or arguments ( if 
it's not a keyword, it's an argument ) , an integer for the index of 
keywords in s string table, and an index for the position of the token 
within the command.

Extensibility should not be difficult.

-- 
Les Cargill

[toc] | [prev] | [next] | [standalone]

#18950

On 10/23/2014 11:30 PM, Les Cargill wrote:
> Don Y wrote:
>> Hi,
>>
>> [I probably should direct this at George...  :> ]
>>
>> I'm writing a command parser.  Some set of commands are "common".
>> But, other instances of the parser are augmented with additional
>> commands/syntax.
>>
>> [These additions are known at compile time, not "dynamic"]
>>
>> Ideally, I want a solution that allows folks developing those
>> "other" commands to just "bolt onto" what I have done.  E.g.,
>> creating a single "grammar definition" (lex/yacc) is probably
>> not a good way to go (IME, most small parsers tend to be ad hoc).
>>
>> [Note:  I can't even guarantee that the extensions will be
>> consistent or "harmonious" with the grammar that I implement]
>>
>> A naive approach (?) might be for my code to take a crack at
>> a particular "statement"/command and, in case of FAIL, invoke
>> some bolt-on parser to give it a chance to make sense out of
>> the input.  If *it* FAILs, then the input is invalid, etc.
>>
>> This sounds like an incredible kluge.  Any better suggestions?
>
>
> If all else fails, strstr() keywords, then have a corresponding
> parsers for each keyword.
>
> A better approach is a container (struct) of keywords or arguments ( if
> it's not a keyword, it's an argument ) , an integer for the index of
> keywords in s string table, and an index for the position of the token
> within the command.
>
> Extensibility should not be difficult.

That is starting to sound a lot like Forth.

-- 

Rick

[toc] | [prev] | [next] | [standalone]

#18951

rickman wrote:
> On 10/23/2014 11:30 PM, Les Cargill wrote:
>> Don Y wrote:
>>> Hi,
>>>
>>> [I probably should direct this at George...  :> ]
>>>
>>> I'm writing a command parser.  Some set of commands are "common".
>>> But, other instances of the parser are augmented with additional
>>> commands/syntax.
>>>
>>> [These additions are known at compile time, not "dynamic"]
>>>
>>> Ideally, I want a solution that allows folks developing those
>>> "other" commands to just "bolt onto" what I have done.  E.g.,
>>> creating a single "grammar definition" (lex/yacc) is probably
>>> not a good way to go (IME, most small parsers tend to be ad hoc).
>>>
>>> [Note:  I can't even guarantee that the extensions will be
>>> consistent or "harmonious" with the grammar that I implement]
>>>
>>> A naive approach (?) might be for my code to take a crack at
>>> a particular "statement"/command and, in case of FAIL, invoke
>>> some bolt-on parser to give it a chance to make sense out of
>>> the input.  If *it* FAILs, then the input is invalid, etc.
>>>
>>> This sounds like an incredible kluge.  Any better suggestions?
>>
>>
>> If all else fails, strstr() keywords, then have a corresponding
>> parsers for each keyword.
>>
>> A better approach is a container (struct) of keywords or arguments ( if
>> it's not a keyword, it's an argument ) , an integer for the index of
>> keywords in s string table, and an index for the position of the token
>> within the command.
>>
>> Extensibility should not be difficult.
>
> That is starting to sound a lot like Forth.
>

Forth could easily be a better choice. I need to spend
more time on Forth.

I tend towards Tcl because I have a large codebase of scripts
for it. It also excels at socket/serial port handling.

-- 
Les Cargill

[toc] | [prev] | [next] | [standalone]

#18952

On 10/25/2014 2:14 PM, Les Cargill wrote:
> rickman wrote:
>> On 10/23/2014 11:30 PM, Les Cargill wrote:
>>> Don Y wrote:
>>>> Hi,
>>>>
>>>> [I probably should direct this at George...  :> ]
>>>>
>>>> I'm writing a command parser.  Some set of commands are "common".
>>>> But, other instances of the parser are augmented with additional
>>>> commands/syntax.
>>>>
>>>> [These additions are known at compile time, not "dynamic"]
>>>>
>>>> Ideally, I want a solution that allows folks developing those
>>>> "other" commands to just "bolt onto" what I have done.  E.g.,
>>>> creating a single "grammar definition" (lex/yacc) is probably
>>>> not a good way to go (IME, most small parsers tend to be ad hoc).
>>>>
>>>> [Note:  I can't even guarantee that the extensions will be
>>>> consistent or "harmonious" with the grammar that I implement]
>>>>
>>>> A naive approach (?) might be for my code to take a crack at
>>>> a particular "statement"/command and, in case of FAIL, invoke
>>>> some bolt-on parser to give it a chance to make sense out of
>>>> the input.  If *it* FAILs, then the input is invalid, etc.
>>>>
>>>> This sounds like an incredible kluge.  Any better suggestions?
>>>
>>>
>>> If all else fails, strstr() keywords, then have a corresponding
>>> parsers for each keyword.
>>>
>>> A better approach is a container (struct) of keywords or arguments ( if
>>> it's not a keyword, it's an argument ) , an integer for the index of
>>> keywords in s string table, and an index for the position of the token
>>> within the command.
>>>
>>> Extensibility should not be difficult.
>>
>> That is starting to sound a lot like Forth.
>>
>
> Forth could easily be a better choice. I need to spend
> more time on Forth.
>
> I tend towards Tcl because I have a large codebase of scripts
> for it. It also excels at socket/serial port handling.

In many ways Forth is amazingly simple.  You define words (subroutines) 
that have actions.  Words are stored in the dictionary.  Forth has built 
in a "parser" that scans the input for words and numbers (in that 
order).  The dictionary is searched for words in the input stream and 
when found they are executed.  If no word is found Forth checks to see 
if the "word" is actually a number.  If it is a number it is pushed onto 
the stack.  Pretty simple, no?

The action of a word can make use of system words to further parse the 
input stream.  This is done if the input "grammar" is not RPN style with 
the values first (nouns) and the words (verbs) last.  This is done even 
for some Forth words like "TO" which is used to store a value in a 
variable, e.g. "99 TO BottlesOfBeer".

Most people find the use of the stack to be a problem for them while it 
is really no big deal.  It's just different.  Forth has other issues 
which relate to the fact that not so many people use it.  But it seems 
to be a very useful tool to me.

-- 

Rick

[toc] | [prev] | [next] | [standalone]

#18958

rickman wrote:
> On 10/25/2014 2:14 PM, Les Cargill wrote:
>> rickman wrote:
>>> On 10/23/2014 11:30 PM, Les Cargill wrote:
>>>> Don Y wrote:
>>>>> Hi,
>>>>>
>>>>> [I probably should direct this at George...  :> ]
>>>>>
>>>>> I'm writing a command parser.  Some set of commands are "common".
>>>>> But, other instances of the parser are augmented with additional
>>>>> commands/syntax.
>>>>>
>>>>> [These additions are known at compile time, not "dynamic"]
>>>>>
>>>>> Ideally, I want a solution that allows folks developing those
>>>>> "other" commands to just "bolt onto" what I have done.  E.g.,
>>>>> creating a single "grammar definition" (lex/yacc) is probably
>>>>> not a good way to go (IME, most small parsers tend to be ad hoc).
>>>>>
>>>>> [Note:  I can't even guarantee that the extensions will be
>>>>> consistent or "harmonious" with the grammar that I implement]
>>>>>
>>>>> A naive approach (?) might be for my code to take a crack at
>>>>> a particular "statement"/command and, in case of FAIL, invoke
>>>>> some bolt-on parser to give it a chance to make sense out of
>>>>> the input.  If *it* FAILs, then the input is invalid, etc.
>>>>>
>>>>> This sounds like an incredible kluge.  Any better suggestions?
>>>>
>>>>
>>>> If all else fails, strstr() keywords, then have a corresponding
>>>> parsers for each keyword.
>>>>
>>>> A better approach is a container (struct) of keywords or arguments ( if
>>>> it's not a keyword, it's an argument ) , an integer for the index of
>>>> keywords in s string table, and an index for the position of the token
>>>> within the command.
>>>>
>>>> Extensibility should not be difficult.
>>>
>>> That is starting to sound a lot like Forth.
>>>
>>
>> Forth could easily be a better choice. I need to spend
>> more time on Forth.
>>
>> I tend towards Tcl because I have a large codebase of scripts
>> for it. It also excels at socket/serial port handling.
>
> In many ways Forth is amazingly simple.  You define words (subroutines)
> that have actions.  Words are stored in the dictionary.  Forth has built
> in a "parser" that scans the input for words and numbers (in that
> order).  The dictionary is searched for words in the input stream and
> when found they are executed.  If no word is found Forth checks to see
> if the "word" is actually a number.  If it is a number it is pushed onto
> the stack.  Pretty simple, no?
>

Very.

> The action of a word can make use of system words to further parse the
> input stream.  This is done if the input "grammar" is not RPN style with
> the values first (nouns) and the words (verbs) last.  This is done even
> for some Forth words like "TO" which is used to store a value in a
> variable, e.g. "99 TO BottlesOfBeer".
>
> Most people find the use of the stack to be a problem for them while it
> is really no big deal.  It's just different.  Forth has other issues
> which relate to the fact that not so many people use it.  But it seems
> to be a very useful tool to me.
>

-- 
Les Cargill

[toc] | [prev] | [next] | [standalone]

#18970

In article <m2gps1$12c$1@dont-email.me>, rickman  <gnuarm@gmail.com> wrote:
>On 10/25/2014 2:14 PM, Les Cargill wrote:
>
>In many ways Forth is amazingly simple.  You define words (subroutines)
>that have actions.  Words are stored in the dictionary.  Forth has built
>in a "parser" that scans the input for words and numbers (in that
>order).  The dictionary is searched for words in the input stream and
>when found they are executed.  If no word is found Forth checks to see
>if the "word" is actually a number.  If it is a number it is pushed onto
>the stack.  Pretty simple, no?

This is one implementation.

Other systems just look up the word in the dictionary. If it matches, it is executed.
Numbers are handled by accepting an incomplete match, i.e. the word
"1" accepts all numbers starting with 1.

(I see two advantages. You can have strings as easy as numbers, and
the system is extensible.)

>
>The action of a word can make use of system words to further parse the
>input stream.  This is done if the input "grammar" is not RPN style with
>the values first (nouns) and the words (verbs) last.  This is done even
>for some Forth words like "TO" which is used to store a value in a
>variable, e.g. "99 TO BottlesOfBeer".

Although it is stated in the Standard that TO is required to parse,
it has been discovered that one can't write a standard program to
test this. So implementations of TO are possible that do not
parse at all and are standard compliant. That is not to say that
parsing words are not possible or useful. One of my favorites is
the defining word OS-IMPORT :
"/usr/bin/ee" OS-IMPORT my-favorite-editor

This defines a word my-favorite-editor with a path as given.

my-favorite-edit a b c

generates a string "/usr/bin/ee a b c" and passes it to SYSTEM.


>
>Most people find the use of the stack to be a problem for them while it
>is really no big deal.  It's just different.  Forth has other issues
>which relate to the fact that not so many people use it.  But it seems
>to be a very useful tool to me.

I'm inclined to say that a stack works purely sequential, and and
infix notation with high priority operators and brackets. give
much more of a visual clue about what is intended.
a*b+c*d (after training) groups things together.
For
    a b * c d * +
the question "what is the first term of the addition?" is indeed
slightly harder to answer. You have to think, the visual system
doesn't help you here.

>
>--
>
>Rick

Groetjes Albert.
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

[toc] | [prev] | [next] | [standalone]

#18971

On 10/26/2014 3:21 PM, Albert van der Horst wrote:
> In article <m2gps1$12c$1@dont-email.me>, rickman  <gnuarm@gmail.com> wrote:
>>
>> Most people find the use of the stack to be a problem for them while it
>> is really no big deal.  It's just different.  Forth has other issues
>> which relate to the fact that not so many people use it.  But it seems
>> to be a very useful tool to me.
>
> I'm inclined to say that a stack works purely sequential, and and
> infix notation with high priority operators and brackets. give
> much more of a visual clue about what is intended.
> a*b+c*d (after training) groups things together.
> For
>      a b * c d * +
> the question "what is the first term of the addition?" is indeed
> slightly harder to answer. You have to think, the visual system
> doesn't help you here.

Really?  You can't see the bias from your early education where you 
learned to use parentheses?  I can clearly pick out the terms from your 
example.  If you asked about a more complex equation, then yes, postfix 
would not be as clear.  But that is a very tiny part of Forth and I have 
yet to find it to be a problem.  When I have an equation much more 
complex than the example you give I usually break it down into simpler 
pieces anyway no matter the language.

-- 

Rick

[toc] | [prev] | [next] | [standalone]

#18931

Don Y wrote:

> Hi,
> 
> [I probably should direct this at George...  :> ]
> 
> I'm writing a command parser.  Some set of commands are "common".
> But, other instances of the parser are augmented with additional
> commands/syntax.
> 
> [These additions are known at compile time, not "dynamic"]
> 
> Ideally, I want a solution that allows folks developing those
> "other" commands to just "bolt onto" what I have done.  E.g.,
> creating a single "grammar definition" (lex/yacc) is probably
> not a good way to go (IME, most small parsers tend to be ad hoc).
> 
> [Note:  I can't even guarantee that the extensions will be
> consistent or "harmonious" with the grammar that I implement]
> 
> A naive approach (?) might be for my code to take a crack at
> a particular "statement"/command and, in case of FAIL, invoke
> some bolt-on parser to give it a chance to make sense out of
> the input.  If *it* FAILs, then the input is invalid, etc.
> 
> This sounds like an incredible kluge.  Any better suggestions?

Hi Don,

Have you looked at Forth? What you are describing sounds like the problem 
that Forth solved back in 1968. It is the sort of thing that Forth 
programmers have been doing for nearly 50 years and it works very 
successfully for us.

See <http://www.mpeforth.com/> and <http://www.forth.com/>

-- 
********************************************************************
Paul E. Bennett IEng MIET.....<email://Paul_E.Bennett@topmail.co.uk>
Forth based HIDECS Consultancy.............<http://www.hidecs.co.uk>
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-510979
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************

[toc] | [prev] | [next] | [standalone]

#18936

Hi Paul,

On 10/24/2014 1:56 AM, Paul E Bennett wrote:
> Don Y wrote:
>
>> I'm writing a command parser.  Some set of commands are "common".
>> But, other instances of the parser are augmented with additional
>> commands/syntax.
>>
>> [These additions are known at compile time, not "dynamic"]

> Have you looked at Forth? What you are describing sounds like the problem
> that Forth solved back in 1968. It is the sort of thing that Forth
> programmers have been doing for nearly 50 years and it works very
> successfully for us.

I am leary of taking things in a different direction from what
is (and has been) expected in the codebase.  IIRC, someone did
something like this as part of an early FreeBSD release (?).
And, I think SPARCs have a Forth-based "Open Firmware".  So,
it's not a "revolutionary" concept that I'd have to expend effort
defending...

Admittedly, it lets you (the user) do lots of things that you
wouldn't be able to even consider in, e.g., a conventional BIOS.
(And, would be far more "future safe" from my point of view).

I'll have to think real hard on that.  If the "typical" sorts
of things "look mildly familiar", then it might be possible
to slip it in under the radar (without religious issues biasing
the decision/acceptance).  OTOH, if things start to look too
funky, then it just gives people an excuse to dislike it.  :-/

Thanks!
--don

[toc] | [prev] | [next] | [standalone]

#18957

On 10/24/2014 7:46 AM, Don Y wrote:

[Forth]

> Admittedly, it lets you (the user) do lots of things that you
> wouldn't be able to even consider in, e.g., a conventional BIOS.
> (And, would be far more "future safe" from my point of view).
>
> I'll have to think real hard on that.  If the "typical" sorts
> of things "look mildly familiar", then it might be possible
> to slip it in under the radar (without religious issues biasing
> the decision/acceptance).  OTOH, if things start to look too
> funky, then it just gives people an excuse to dislike it.  :-/

Wow!  I was *amazed* at the intensity of the reactions to this
suggestion when I circulated it among colleagues!  In hindsight,
I wish I had presented the suggestion as "a generic scripting
language" instead of "Forth" as many of the replies seemed to
*choke* on the mention of Forth!  <frown>  Despite the fact
that several of us use Open Firmware.

[Still waiting for replies from two other folks, but...]

I guess I don't really understand the "disapproval" (for want of
a better word).  Granted, it's been almost 40 years since I used
Forth but I don't have a "bad taste" associated with the experience.

I tried to quickly shift the discussion away from Forth by offering
other "simple"/small languages as alternatives.  But, I think the idea
had already taken on a taint.  I even proposed a small C interpreter
for that role ("Why interpret C when you can COMPILE it??").  I have
an even simpler "language" I'll try proposing but fear it won't fly,
either!

(sigh)  I will have to wait for attitudes to settle back down before
looking for clarification of this "rejection".  And, perhaps reexamine
Forth to get a feel for what folks might be objecting to...

[toc] | [prev] | [next] | [standalone]

#18960

On 2014-10-26, Don Y <this@is.not.me.com> wrote:
> On 10/24/2014 7:46 AM, Don Y wrote:
>>
>> I'll have to think real hard on that.  If the "typical" sorts
>> of things "look mildly familiar", then it might be possible
>> to slip it in under the radar (without religious issues biasing
>> the decision/acceptance).  OTOH, if things start to look too
>> funky, then it just gives people an excuse to dislike it.  :-/
>
> Wow!  I was *amazed* at the intensity of the reactions to this
> suggestion when I circulated it among colleagues!  In hindsight,
> I wish I had presented the suggestion as "a generic scripting
> language" instead of "Forth" as many of the replies seemed to
> *choke* on the mention of Forth!  <frown>  Despite the fact
> that several of us use Open Firmware.
>

Do any of these people use RPN calculators ?

Simon.

-- 
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[toc] | [prev] | [next] | [standalone]

#18963

On 10/26/2014 5:15 AM, Simon Clubley wrote:
> On 2014-10-26, Don Y <this@is.not.me.com> wrote:
>> On 10/24/2014 7:46 AM, Don Y wrote:
>>>
>>> I'll have to think real hard on that.  If the "typical" sorts
>>> of things "look mildly familiar", then it might be possible
>>> to slip it in under the radar (without religious issues biasing
>>> the decision/acceptance).  OTOH, if things start to look too
>>> funky, then it just gives people an excuse to dislike it.  :-/
>>
>> Wow!  I was *amazed* at the intensity of the reactions to this
>> suggestion when I circulated it among colleagues!  In hindsight,
>> I wish I had presented the suggestion as "a generic scripting
>> language" instead of "Forth" as many of the replies seemed to
>> *choke* on the mention of Forth!  <frown>  Despite the fact
>> that several of us use Open Firmware.
>
> Do any of these people use RPN calculators ?

I'm sure they/we all have in the past (I don't use a calculator).

And, several of us "regularly" use Forth in configuring, for example,
SPARCstations, etc.  (OpenFirmware)!

But, it's not something that is done *often* so there's always
a relearning experience involved.  Forth is quirky -- remembering
which words serve which functionality, etc.  (what word displays
my IP address?  how do I remove an alias from NVRAMrc?  etc.)

By contrast, they/we have probably used a BASIC dialect even
LESS recently than OFW -- yet, I would imagine all of us could
craft a little BASIC "script" in a matter of seconds... and, be
assured it would work first time.

<shrug>  Dunno.  I'm only guessing as to the source of the
"resistance" (resentment?).  As I said, I tried to shift the
discussion away from Forth, per se, to see if the idea of a
procedural language was the issue ("We don't need that level
of functionality -- just parse COMMANDS!") or the proposed
language...

I'll revisit it, later (just for my own edification) but
won't chase that approach after this sort of "reception"  :-/

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.arch.embedded

csiph-web