Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #1840 > unrolled thread

Parsing partial sentences

Back to article view | Back to comp.compilers

Contents

  Parsing partial sentences Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2017-04-03 09:57 +0200
    Re: Parsing partial sentences Fernando <pronesto@gmail.com> - 2017-04-04 06:05 -0700
      Re: Parsing partial sentences Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2017-04-07 11:16 -0400
        Re: Parsing partial sentences George Neuner <gneuner2@comcast.net> - 2017-04-07 12:59 -0400
          Re: Parsing partial sentences Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2017-04-07 22:24 +0200
            Re: Parsing partial sentences George Neuner <gneuner2@comcast.net> - 2017-04-10 20:22 -0400
              Re: Parsing partial sentences Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2017-04-11 10:31 +0200
                Re: Parsing partial sentences Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2017-04-11 19:40 +0200
                Re: Parsing partial sentences George Neuner <gneuner2@comcast.net> - 2017-04-11 23:05 -0400
                  Re: Parsing partial sentences Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2017-04-12 13:45 +0200
            Re: Parsing partial sentences Martin Ward <martin@gkc.org.uk> - 2017-04-11 10:30 +0100
    Re: Parsing partial sentences Walter Banks <walter@bytecraft.com> - 2017-04-27 10:58 -0400
      Re: Parsing partial sentences Kaz Kylheku <686-678-9105@kylheku.com> - 2017-04-27 19:08 +0000
        Re: Parsing partial sentences rugxulo@gmail.com - 2017-04-28 16:32 -0700
        Re: Parsing partial sentences Marco van de Voort <marcov@toad.stack.nl> - 2017-04-29 14:57 +0000
          Re: Parsing partial sentences Kaz Kylheku <686-678-9105@kylheku.com> - 2017-04-30 16:31 +0000
      Re: Parsing partial sentences Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2017-04-28 02:48 +0200

#1840 — Parsing partial sentences

Is there an easy way to parse e.g. C #defines into constants, functions
or other non-terminals, which are not the goal of the entire grammar?

DoDi

[toc] | [next] | [standalone]

#1841

Em segunda-feira, 3 de abril de 2017 12:16:18 UTC-3, Hans-Peter Diettrich
escreveu:
> Is there an easy way to parse e.g. C #defines into constants, functions
> or other non-terminals, which are not the goal of the entire grammar?

Hi Hans,

    I am not sure I understood the question, but from the title of the e-mail,
I am assuming that you want to parse C code partially available? If that's so,
then we have a parser that handles that. It is open-source, and we have a web
interface that you can try: http://cuda.dcc.ufmg.br/psyche-c/. It resolves
ambiguities due to missing declarations, and infer the types that are missing
in the program. In the end, we produce enough code that lets you compile your
program.

Regards,

Fernando

[toc] | [prev] | [next] | [standalone]

#1842

Am 04.04.2017 um 15:05 schrieb Fernando:
> Em segunda-feira, 3 de abril de 2017 12:16:18 UTC-3, Hans-Peter Diettrich
> escreveu:
>> Is there an easy way to parse e.g. C #defines into constants, functions
>> or other non-terminals, which are not the goal of the entire grammar?

>     I am not sure I understood the question, but from the title of the e-mail,
> I am assuming that you want to parse C code partially available?

The purpose is just what I wrote: parse #defines into constructs which
can be translated into some other programming language. It shall become
part of my C-to-Pascal converter, written in Pascal (OPL).

DoDi

[toc] | [prev] | [next] | [standalone]

#1843

On Fri, 7 Apr 2017 11:16:18 -0400 (EDT), Hans-Peter Diettrich
<DrDiettrich1@netscape.net> wrote:

>Am 04.04.2017 um 15:05 schrieb Fernando:
>> Em segunda-feira, 3 de abril de 2017 12:16:18 UTC-3, Hans-Peter Diettrich
>> escreveu:
>>> Is there an easy way to parse e.g. C #defines into constants, functions
>>> or other non-terminals, which are not the goal of the entire grammar?
>
>>     I am not sure I understood the question, but from the title of the e-mail,
>> I am assuming that you want to parse C code partially available?
>
>The purpose is just what I wrote: parse #defines into constructs which
>can be translated into some other programming language. It shall become
>part of my C-to-Pascal converter, written in Pascal (OPL).

Unfortunately, #define is a textual substition ... the preprocessor
doesn't care whether the body is legal C, and the compiler itself
doesn't look at the raw #define bodies.

It might be easiest to just run the C preprocessor as a 1st pass and
then attempt to convert the result.


It's been a while since I used C regularly, but if my memory is
correct Lint was able to check and steer you to a faulty #define.
However, I *think* Lint used the preprocessor and required it to be
available.

There are some open-source alternatives to Lint. You might look into
what they are able to do.


Since a #define body is just text, it can be anything - people have
created whole DSLs using #define.  If you really need to figure them
out, I'm afraid you'll need (almost) the whole C language parser to do
it.

Sorry I can't help more.
George

[toc] | [prev] | [next] | [standalone]

#1845

Am 07.04.2017 um 18:59 schrieb George Neuner:
> On Fri, 7 Apr 2017 11:16:18 -0400 (EDT), Hans-Peter Diettrich
> <DrDiettrich1@netscape.net> wrote:

>> The purpose is just what I wrote: parse #defines into constructs which
>> can be translated into some other programming language. It shall become
>> part of my C-to-Pascal converter, written in Pascal (OPL).
>
> Unfortunately, #define is a textual substition ... the preprocessor
> doesn't care whether the body is legal C, and the compiler itself
> doesn't look at the raw #define bodies.

Right, but in a language translation I want to retain as much of the
original code as possible. Then a Pascal compiler can find bugs which
escape the C compiler.

> It might be easiest to just run the C preprocessor as a 1st pass and
> then attempt to convert the result.

That would obfuscate the source code widely :-(


> Since a #define body is just text, it can be anything - people have
> created whole DSLs using #define.  If you really need to figure them
> out, I'm afraid you'll need (almost) the whole C language parser to do
> it.

I already wrote the parser, but it's LL. For code snippets a LR parser
looks like the only solution?

DoDi

[Since preprocessor macros are text macros, there's no reason to
expect a macro's expansion can be parsed at all.  Consider this:

# define FOO 42 +
int a, b;
...
	a = FOO b;

A notorious example of preprocessor abuse was the Bourne shell, where
Steve Bourne wrote preprocessor macros to make C look like Algol 68.
-John]

[toc] | [prev] | [next] | [standalone]

#1846

On Fri, 7 Apr 2017 22:24:04 +0200, Hans-Peter Diettrich
<DrDiettrich1@netscape.net> wrote:

>Am 07.04.2017 um 18:59 schrieb George Neuner:
>
>> It might be easiest to just run the C preprocessor as a 1st pass and
>> then attempt to convert the result.
>
>That would obfuscate the source code widely :-(

That's true, but any C code you might encounter could, in itself, be
arbitrarily convoluted and hard to understand: e.g., deep pointer
chains, goto spaghetti, Duff devices, setjmp/longjmp, coroutines, etc.

I understand that you want to be able to preserve and relate the
original source to the translation, but I don't see any other real
choice other than to employ the C preprocessor.  As John said, a
#define body may not even be a complete expression - nevermind a legal
one.

Normally a parser would error on an illegal or incomplete expression.
Creating a parser that can return useful information about arbitrary
quasi-legal input I think would be extremely difficult.


>> Since a #define body is just text, it can be anything - people have
>> created whole DSLs using #define.  If you really need to figure them
>> out, I'm afraid you'll need (almost) the whole C language parser to do
>> it.
>
>I already wrote the parser, but it's LL. For code snippets a LR parser
>looks like the only solution?

You could do it in either LL or LR ... LR would be more runtime
efficient, but I think it would not be any easier to create the parser
in the first place.

To do what you want you'd need non-terminals not just for every legal
(sub)expression in C, but for every individual keyword, operator and
symbol, and also for any quasi-legal combination of them.  The parser
would be enormous.


Building on John's example, consider what you'd do with

# define FOO +
# define BAR + 42
# define BAZ + c   /* note 'c' is undefined */
int a, b;

    :
    a = a FOO b BAR BAZ;
    :


And then consider what you'd need to handle sh..stuff like this for
every keyword, operator, variable, etc.

George

[toc] | [prev] | [next] | [standalone]

#1847

Am 11.04.2017 um 02:22 schrieb George Neuner:
> On Fri, 7 Apr 2017 22:24:04 +0200, Hans-Peter Diettrich

>> I already wrote the parser, but it's LL. For code snippets a LR parser
>> looks like the only solution?
>
> You could do it in either LL or LR ... LR would be more runtime
> efficient, but I think it would not be any easier to create the parser
> in the first place.

Perhaps I should clarify: with LL I mean top-down, with LR a bottom-up
parser.

Such an LR parser will stop with one or more tokens, which then should
fit (reduce to) one of the expected non-terminals. This last step should
be automatic, if the goal is the set of all expected non-terminals. If
it's a statement, it can become a function. If it's an expression, it
can become a named constant or a function, depending on the operands.

> To do what you want you'd need non-terminals not just for every legal
> (sub)expression in C, but for every individual keyword, operator and
> symbol, and also for any quasi-legal combination of them.  The parser
> would be enormous.
>
>
> Building on John's example, consider what you'd do with
>
> # define FOO +
> # define BAR + 42
> # define BAZ + c   /* note 'c' is undefined */

All these can not be reduced into constants or functions. Eventually a
problem may arise for "+ 42", where the '+' could be interpreted as the
sign of a constant value. In this case the following code

> int a, b;
>
>     :
>     a = a FOO b BAR BAZ;

would raise a parser error "expecting operator between BAR and BAZ".

A more practical example were windows.h. I expect much more than 50%
named constants in it, which could be detected and converted easily.
Then this automated handling could reduce the manual inspection and
classification of many hundreds (thousands?) of #defines to a few
unhandled or not easily translatable macros.

> And then consider what you'd need to handle sh..stuff like this for
> every keyword, operator, variable, etc.

The identifier lists are already implemented, for both the preprocessor
and parser. The current implementation translates all MS (msvc), GNU
(gcc) and BCC style C source files. Further compilers can be added by
the user, who writes a header file containing all predefined symbols.
The output is Delphi (OPL) code, further back-ends could be added, like
C++, Java or whatever compatible language you prefer.

One use of such a translation into a strictly typed language are checks
for invalid type casts or wrong enumerated constants, as found e.g. in
the Windows SDK samples. Every inspected source file, compiled with BCC
after removal of excess type casts, contained more than 50 bugs. Even in
GNU sources a translation into OPL did reveal bugs, which had not been
found or fixed since many years.

DoDi

[I think you may be trying to solve the wrong problem.  Really, you
cannot preparse C #define statements unless you're prepared to handle
irregular parse tree fragments.  In most compilers, the lexer takes
vastly more time than the parser, so if you store tokens rather than
text, you'll get most of the performance.  Also, one of the reasons C
and C++ added const and inline is that they do most of what #define
does while being normal syntax.

Here's a thought that sometimes works: try parsing the #define text,
if it succeeds store the parsed version, otherwise store the text.
But be prepared for that to fail, e.g.:

#define FOO a + b

 d = FOO * c;

be sure that your preparsed FOO doesn't expand into (a+b)*c.  That
may well be what the programmer meant, but it's not what she said.
-John]

[toc] | [prev] | [next] | [standalone]

#1849

Am 11.04.2017 um 10:31 schrieb Hans-Peter Diettrich:

[John]
> Here's a thought that sometimes works: try parsing the #define text,
> if it succeeds store the parsed version, otherwise store the text.

How do you detect "succeeds", and what's the parser output in this case?

My LL parser fails with the first token, because it expects a C module,
so that there is no useful output available.

An LR parser instead could stop with something like
  goal-->expression-->addition-->(identifier="a", identifier="b").

Hmm, a similar result could be achieved with an LL top-down parser,
whose goal is composed of the expected (handled) non-terminals. This
would only require a modification to the parser for that goal, which
tries all alternatives in sequence and stops on the first successful
parse (NFA). Parsing has to be restarted at the begin of the text
snippet, after an alternative failed to parse. This part is covered in
LR bottom-up parsers by the NFA to DFA conversion.


> But be prepared for that to fail, e.g.:
>
> #define FOO a + b

When I'd try to convert that into a function body, the resulting
function could look like
   void FOO() { a + b; }
or
   int FOO() { return a + b; }
what are not valid C modules, so that FOO were flagged to be
not-a-function, and its further occurences have to be expanded as usual.

Only if the identifiers a and b have been declared or #defined before,
FOO() would become a valid function (above) or constant of a compatible
type, like
   const int FOO = a+b;

Instead
   #define FOO 0x12345
would parse into something like
   goal-->value-->literal="0x1235"
which could be translated into
   const int FOO=0x12345;

In this case some manual postprocessing could define an enum
   typedef enum SomeField = { FOO=0x1234, BAR=0, BAZ=42 };
which then could be used to assign a more specific type to strzct fields
or subroutine arguments.

DoDi

[toc] | [prev] | [next] | [standalone]

#1851

On Tue, 11 Apr 2017 10:31:30 +0200, Hans-Peter Diettrich
<DrDiettrich1@netscape.net> wrote:

>Am 11.04.2017 um 02:22 schrieb George Neuner:
>
>> You could do it in either LL or LR ... LR would be more runtime
>> efficient, but I think it would not be any easier to create the parser
>> in the first place.
>
>Perhaps I should clarify: with LL I mean top-down, with LR a bottom-up
>parser.

I know what you mean by LL and LR.  Apart from their Chomsky meanings,
they also refer to canonical implementations.

There may, in fact, be state machine LL and/or recursive descent LR
implementations - but if so, I have neither seen nor heard of them.


In terms of implementation, bottom-up [state machine] tends to be much
more efficient at handling languages that have less structure.

C's syntax is (structured but) far more permissive than Pascal's:
e.g., C allows declarations to be intermixed with other statements,
the only real requirements being that the declaration be in scope of
and preceed any use of the declared object.

A #define body is free form text.  There are no syntax rules for
#defines - all that is required (for compilation) is that the body
make sense /in context/ after being substituted into the code.


>> Building on John's example, consider what you'd do with
>>
>> # define FOO +
>> # define BAR + 42
>> # define BAZ + c   /* note 'c' is undefined */
>
>All these can not be reduced into constants or functions. Eventually a
>problem may arise for "+ 42", where the '+' could be interpreted as the
>sign of a constant value.

The point is that the preprocessor doesn't care about the meanings of
tokens in the #define body.  Modulo parameter substitution, the
preprocess simply inlines the #define body at the point of use.


>In this case the following code
>
>> int a, b;
>>
>>     :
>>     a = a FOO b BAR BAZ;
>
>would raise a parser error "expecting operator between BAR and BAZ".

After preprocessing, the C compiler will see

   a = a + b + 42 + c;

The compiler will complain about 'c' being undeclared, not about a
missing operator.  Try it.


>A more practical example were windows.h. I expect much more than 50%
>named constants in it, which could be detected and converted easily.
>Then this automated handling could reduce the manual inspection and
>classification of many hundreds (thousands?) of #defines to a few
>unhandled or not easily translatable macros.

In the case of Windows: 10s of thousands, counting all the nested
includes.


You are undoubtedly correct regarding the majority of headers ... as
John mentioned, constant definitions and real inline functions are
used more often now than "functional" #define macros.

But you still are likely to see weird stuff in older code, and system
and i/o headers in particular often define parameterized macros to
wrap intrinsics and low level library functions.

Macro substitution variables, if any, are local to the #define body -
they are not free references to C variables.  So if you see something
like

# define dumb(x) (x << y)

then you have a bound parameter 'x', a free variable 'y', and an
ambiguous expression of unknown types which must *not* cause an error
during parsing of the macro, but *must* cause an error if the fully
expanded and substituted expression is invalid AT THE POINT OF USE.

You rarely will have any clue regarding the types of variables that
appear in a macro.  In the example above, 'x' is a purely mathematical
variable, and the type of 'y' is unknown.  In context at the point of
substitution, when the types finally become known, the '<<' operator
may not even be valid.



>Our esteemed moderator wrote:
>
>[I think you may be trying to solve the wrong problem.  Really, you
>cannot preparse C #define statements unless you're prepared to handle
>irregular parse tree fragments.  In most compilers, the lexer takes
>vastly more time than the parser, so if you store tokens rather than
>text, you'll get most of the performance.  Also, one of the reasons C
>and C++ added const and inline is that they do most of what #define
>does while being normal syntax.
>
>Here's a thought that sometimes works: try parsing the #define text,
>if it succeeds store the parsed version, otherwise store the text.
>But be prepared for that to fail, e.g.:
>
>#define FOO a + b
>
> d = FOO * c;
>
>be sure that your preparsed FOO doesn't expand into (a+b)*c.  That
>may well be what the programmer meant, but it's not what she said.
>-John]

+1.

George

[toc] | [prev] | [next] | [standalone]

#1852

Am 12.04.2017 um 05:05 schrieb George Neuner:
> On Tue, 11 Apr 2017 10:31:30 +0200, Hans-Peter Diettrich
> <DrDiettrich1@netscape.net> wrote:

> In terms of implementation, bottom-up [state machine] tends to be much
> more efficient at handling languages that have less structure.

Provided that type names are recognized distinct from other identifiers,
the C language is almost LL(1). I only found one or two exceptions which
require LL(2). In so far my parser is very efficient.

C++ were much harder to parse, but its class model is so incompatible
with other languages, that a source-level translation is almost impossible.

> C's syntax is (structured but) far more permissive than Pascal's:
> e.g., C allows declarations to be intermixed with other statements,
> the only real requirements being that the declaration be in scope of
> and preceed any use of the declared object.

I've already solved this language difference. Some other constructs
require the use of objects and properties, to retain most of the C
"wording" in OPL code.


>>> Building on John's example, consider what you'd do with
>>>
>>> # define FOO +
>>> # define BAR + 42
>>> # define BAZ + c   /* note 'c' is undefined */
>>
>> All these can not be reduced into constants or functions. Eventually a
>> problem may arise for "+ 42", where the '+' could be interpreted as the
>> sign of a constant value.
>
> The point is that the preprocessor doesn't care about the meanings of
> tokens in the #define body.  Modulo parameter substitution, the
> preprocess simply inlines the #define body at the point of use.
>
>
>> In this case the following code
>>
>>> int a, b;
>>>
>>>     :
>>>     a = a FOO b BAR BAZ;
>>
>> would raise a parser error "expecting operator between BAR and BAZ".
>
> After preprocessing, the C compiler will see
>
>    a = a + b + 42 + c;
>
> The compiler will complain about 'c' being undeclared, not about a
> missing operator.  Try it.

I already mentioned that "+ 42" is ambiguous, and which error a compiler
would find with the attempted interpretation as a value "+42". At least
an error would occur later, when a usage of that macro is parsed, so
that BAR could be found out as being not-a-constant.

Actually I have no specific ideas yet, how unbound identifiers could be
handled in a transformation of macros into functions. Even bound
identifiers (macro arguments) can not be handled easily, not even in a
textual macro expansion.


>> A more practical example were windows.h. I expect much more than 50%
>> named constants in it, which could be detected and converted easily.
>> Then this automated handling could reduce the manual inspection and
>> classification of many hundreds (thousands?) of #defines to a few
>> unhandled or not easily translatable macros.
>
> In the case of Windows: 10s of thousands, counting all the nested
> includes.

This was the point when I gave up to improve ToPas, many years ago. Now
I'm considering another attempt to add more automatisms to it, for
handling at least literal constants and, possibly, constant expressions.
I'd be happy if this helps to reduce the number of unclassified #defines
in windows.h, to an amount that could be handled manually in reasonable
time. The same for the GNU (gcc) headers. I have to catch up with the
header evolution till now, and find out how translation-friendly these
are nowadays.


> Macro substitution variables, if any, are local to the #define body -
> they are not free references to C variables.  So if you see something
> like
>
> # define dumb(x) (x << y)
>
> then you have a bound parameter 'x', a free variable 'y', and an
> ambiguous expression of unknown types which must *not* cause an error
> during parsing of the macro, but *must* cause an error if the fully
> expanded and substituted expression is invalid AT THE POINT OF USE.
>
> You rarely will have any clue regarding the types of variables that
> appear in a macro.  In the example above, 'x' is a purely mathematical
> variable, and the type of 'y' is unknown.  In context at the point of
> substitution, when the types finally become known, the '<<' operator
> may not even be valid.

You are perfectly right. I don't think that code snippets with unbound
identifiers should ever be subject to a transformation into functions.

Eventually a special kind of function inlining could be implemented,
that allows to proof the correctness of assumptions about the location
of all identifiers, used in later macro expansion and calls of the
constructed functions. But that only in a far future...


>> Our esteemed moderator wrote:
[...]
>> #define FOO a + b
>>
>> d = FOO * c;
>>
>> be sure that your preparsed FOO doesn't expand into (a+b)*c.  That
>> may well be what the programmer meant, but it's not what she said.

Well, this might be a case where automatic source code analysis could
reveal possible coding bugs. Remember how many parentheses are (or are
not) added around macro arguments, just to prevent such unintended
compilation. This is another reason why I don't like C, to put it mildly
<BG>. My dream is an automated source code translation system, that
allows to safely translate and get rid of legacy C code, so that further
software development could be done in some better (safer, faster...)
programming language...

DoDi

[toc] | [prev] | [next] | [standalone]

#1848

On 07/04/17 21:24, Hans-Peter Diettrich wrote:
>> It might be easiest to just run the C preprocessor as a 1st pass and
>> then attempt to convert the result.
>
> That would obfuscate the source code widely

On 07/04/17 21:24, John wrote:
> Since preprocessor macros are text macros, there's no reason to
> expect a macro's expansion can be parsed at all.

This is true: handling the full generality of what is possible
with #define macros without first expanding all the macros
is (almost) impossible.

However, in practice it depends on the purpose of your C to Pascal
convertor. If it is to be a general purpose tool which has to be
able to handle any C source code that anyone throws at it,
then you are stuck. If, on the other hand, there is a certain
defined body of C code which you want to translate
then you only need to be able to handle the types of macro
definitions which are actually used in that body of code:
which is likely to be a small subset of the possibilities.
Most macro definitions in most code (other than the original
Bourne shell source code, and some similar examples)
are written such that the code using the macro looks
and parses like ordinary C code.

An intermediate approach might be to keep all the macros
which look straightforward and fall back on expanding macros
which do weird things with the syntax.

--
			Martin

Dr Martin Ward STRL Principal Lecturer & Reader in Software Engineering
martin@gkc.org.uk  http://www.cse.dmu.ac.uk/~mward/  Erdos number: 4
G.K.Chesterton web site: http://www.cse.dmu.ac.uk/~mward/gkc/
Mirrors:  http://www.gkc.org.uk  and  http://www.gkc.org.uk/gkc

[toc] | [prev] | [next] | [standalone]

#1862

On 2017-04-03 3:57 AM, Hans-Peter Diettrich wrote:
> Is there an easy way to parse e.g. C #defines into constants,
> functions or other non-terminals, which are not the goal of the
> entire grammar?

In a word NO. #defines are always strings even when they look like
constants, something I have found out the hard way. There have only been
two ways that I have successfully dealt with #defines: a preprocessor
pass or later and much faster pipeline the processing of C source and
add the defined definition processing into part of the source fetch
handling.

w..

[toc] | [prev] | [next] | [standalone]

#1863

On 2017-04-27, Walter Banks <walter@bytecraft.com> wrote:
> On 2017-04-03 3:57 AM, Hans-Peter Diettrich wrote:
>> Is there an easy way to parse e.g. C #defines into constants,
>> functions or other non-terminals, which are not the goal of the
>> entire grammar?
>
> In a word NO. #defines are always strings even when they look like
> constants, something I have found out the hard way. There have only been
> two ways that I have successfully dealt with #defines: a preprocessor
> pass or later and much faster pipeline the processing of C source and
> add the defined definition processing into part of the source fetch
> handling.

If we allow Pascal to be extended with a macro preprocessor,

I believe I could design a system for translating C to Pascal which
handles some macros, translating them to Pascal macros.  Even some
macros that "break" syntactic boundaries, such as "list_for_each (var,
list) { block }".

I don't believe such a project has any value beyond getting
a pat on the back from another developer; I wouldn't spend any
time on such a thing.  The end result might well be rejected by some
Pascal users, due to requiring the extended dialect, whether on
ideological grounds, or on practical issues with tooling (being able to
get the preprocessor running in a given Pascal development environment).

Here is a very high level sketch of the approach:

- We preprocess the C translation unit fully before parsing it.
  - However, we use our own specialized C preprocessor which is
    tightly integrated into our translator.
  - Our specialized C preprocessor carefully tracks, in detail
    the origin of every piece of syntax, to the macro which
    substituted that syntax, either as an argument or as body
    material.
  - As our parser is analyzing the code, it preserves this information
    in the abstract syntax tree: every tree fragment, if it
    was the result of a macro expansion, is tracked to the
    macro call. We also know that an entire node was the result
    of a macro call, and what that macro call looked like.
  - When we output the Pascal translation, if a tree node was the
    result of a macro call (a macro call that we we were successfully
    able to treat with our magic algoirthms) then rather than outputting
    the Pascal translation of that tree node, we output the macro
    call syntax (remembering that our Pascal dialect has a preprocessor
    to handle that).
  - We have a magic algorithm for reconstructing Pascal versions
    of macro bodies which works roughly like this:
    - When we translate a tree node from C to Pascal, if that tree node
      came from macrology, we keep track of which Pascal fragments
      correspond to C fragments.
    - We then reverse the macrology: we analyze the Pascal and see which
      fragments correspond to C material that came from a macro body,
      and which came from substitution of macro arguments and such.
      - From this we can reconstruct a Pascal macro body, using the
        corresponding Pascal fragments.
        - A Pascal fragment which coresponds to the insertion
          of some argument X, is just represented by the same X in the
          Pascal macro body.
        - A Pascal fragment which corresponds to the insertion of
          some body template material Y is replaced in the Pascal
          version of the macro by the corresponding Pascal piece.
    - Since a macro call often occurs in more than one place, we can
      somehow combine information from multiple sites to improve the
      translation, or at least validate that the same thing is
      happening.

  - The saving grace here what makes this even contemplatable is
    that C macros are dumb positional substitution: just pasting
    together of fragments. If C macros were like Lisp macros, doing
    arbitrary Turing computation, good luck with this approach, right?

  - I think, best forget C99 variadic macros and such cruft.

  - The system could have a mode whereby it gives up trying to translate
    a C macro to Pascal, but it at least reverses the syntax so that
    the overall Pascal fragment corresponding to the C code which came
    from that macro call is reversed back to a macro call. A diagnostic
    can be generated "missing macro needed", and the users themselves
    can use their human intelligence to finish the job (if possible).
    Find the C version of the macro and try to translate it.

[toc] | [prev] | [next] | [standalone]

#1865

Hi,

On Thursday, April 27, 2017 at 8:15:18 PM UTC-5, Kaz Kylheku wrote:
>
> If we allow Pascal to be extended with a macro preprocessor,
>
> I believe I could design a system for translating C to Pascal which
> handles some macros, translating them to Pascal macros.  Even some
> macros that "break" syntactic boundaries, such as "list_for_each (var,
> list) { block }".

Doug Comer wrote MAP (macro preprocessor) in 1978-9 for CDC Pascal.
(Pascal Users Group [PUG], Pascal News #17, March 1980, pages 30..40)

Free Pascal already has a preprocessor (and simple macros):
http://www.freepascal.org/docs-html/prog/progch2.html#progse4.html

> I don't believe such a project has any value beyond getting
> a pat on the back from another developer; I wouldn't spend any
> time on such a thing.

Well, in fairness, regarding your posts in comp.lang.misc three
years ago ("Wirth-ian languages"), you don't seem sympathetic
overall. I do think you underestimate their continued relevance.

> The end result might well be rejected by some
> Pascal users, due to requiring the extended dialect, whether on
> ideological grounds,

In fairness, with so many Pascal compilers and dialects and deviances
from the standards, I think most "Pascal" users are already
accustomed to various quirks.

> or on practical issues with tooling (being able to get the
> preprocessor running in a given Pascal development environment).

Keep in mind that several Pascal-to-C converters already exist (of
varying quality). It might be wiser to look at what's already been
done. I can find at least three. (Not quite the reverse but similar.)

Not sure what you mean about tooling, but FPC is quite portable, so
that isn't the main problem here.

I don't wish to discourage the OP. Certainly it's not an easy task.
Here's yet another document I stumbled upon years ago that may help:

http://www.knosof.co.uk/ctop.ps

[toc] | [prev] | [next] | [standalone]

#1866

On 2017-04-27, Kaz Kylheku <686-678-9105@kylheku.com> wrote:
>> In a word NO. #defines are always strings even when they look like
>> constants, something I have found out the hard way. There have only been
>> two ways that I have successfully dealt with #defines: a preprocessor
>> pass or later and much faster pipeline the processing of C source and
>> add the defined definition processing into part of the source fetch
>> handling.
>
> If we allow Pascal to be extended with a macro preprocessor,

I think most interest in C->Pascal header translators is for (semi)automated
translation of 3rd party library headers, not converting complete programs.

This means that it cannot be considered a given that a program exists that
uses all macros in the header.

Estabilishing rules or heuristics to judge if a macro could be a constant
would be useful though.

[toc] | [prev] | [next] | [standalone]

#1867

On 2017-04-29, Marco van de Voort <marcov@toad.stack.nl> wrote:
> On 2017-04-27, Kaz Kylheku <686-678-9105@kylheku.com> wrote:
>>> In a word NO. #defines are always strings even when they look like
>>> constants, something I have found out the hard way. There have only been
>>> two ways that I have successfully dealt with #defines: a preprocessor
>>> pass or later and much faster pipeline the processing of C source and
>>> add the defined definition processing into part of the source fetch
>>> handling.
>>
>> If we allow Pascal to be extended with a macro preprocessor,
>
> I think most interest in C->Pascal header translators is for (semi)automated
> translation of 3rd party library headers, not converting complete programs.

OP wrote that he's writing a C to Pascal translator. The face value
interpretation of that is that it handles programs.

Handling only headers would be the likely focus of a project for
automatically generating Pascal foreign function interface bindings
for C libraries: C to Pascal interface translation (C � la SWIG and
whatnot).

> This means that it cannot be considered a given that a program exists that
> uses all macros in the header.

A macro that is not used by a C program might be complete gibberish;
yet that program may still be well-formed.

   #define ABC { ( 42 -> * "foo" ]

A C to Pascal translator which tries to treat unused macros might
needlessly choke on such a thing.

It is reasonable, nay wise, for a whole-program C to Pascal translator
to ignore C macros that are unused by the C program.

[toc] | [prev] | [next] | [standalone]

#1864

Am 27.04.2017 um 16:58 schrieb Walter Banks:
> On 2017-04-03 3:57 AM, Hans-Peter Diettrich wrote:
>> Is there an easy way to parse e.g. C #defines into constants,
>> functions or other non-terminals, which are not the goal of the
>> entire grammar?
>
> In a word NO. #defines are always strings even when they look like
> constants, something I have found out the hard way.

You missed the "e.g." #defines. In fact everybody seems to have missed
it :-(

The focus of my question is on *partial* sentences of a language, that
are equivalent to a *single* non-terminal of the language.

Until now only the *use* of such partial sentences has been discussed,
not the *determination* of the resulting non-terminal.

DoDi

[toc] | [prev] | [standalone]

Back to top | Article view | comp.compilers

csiph-web