Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compilers > #1874 > unrolled thread

Designing a Domain-specific language

Back to article view | Back to comp.compilers

Contents

  Designing a Domain-specific language dror.openu@gmail.com - 2017-05-23 07:12 -0700
    Re: Designing a Domain-specific language bartc <bc@freeuk.com> - 2017-05-23 20:39 +0100
      Re: Designing a Domain-specific language dror.openu@gmail.com - 2017-05-25 12:51 -0700
        Re: Designing a Domain-specific language bartc <bc@freeuk.com> - 2017-05-27 16:00 +0100
    Re: Designing a Domain-specific language Hans-Peter Diettrich <DrDiettrich1@netscape.net> - 2017-05-24 06:42 +0200
    Re: Designing a Domain-specific language "lucretia9@lycos.co.uk" <laguest9000@googlemail.com> - 2017-05-24 04:13 -0700

#1874 — Designing a Domain-specific language

I'm implementing right now a "compiler" for a DSL that used at my work, and I'd like to get your advice for a dilemma I have.
One of the requirements at the begging was that the compiler should get an input only one file.

But, it's changed and I asked to add two more capabilities. and they are:
1. The author that uses the language should be able to include common files (std the comes with the compiler).
for example: include "file.common"
2. The author should be able to compile a directory and then split the project into many files. so the code will something like this:
state = include "./sibling"

The project status is:
1. I have a lexer/tokenizer that generates tokens and pass them to the parser.
2. The parser, which responsible for the syntax analysis, builds the parse tree.

My dilemma is: "How's the compile process should be when you have many files". I thought about this process for the second requirement, and I'd like to hear your opinion:
1. scan all the files (assume we have k files).
2. for each of the files:
2.1 run the lexer together with parser and build a parse tree.
3. now, I have k parse trees (but only one entry-point/main).
4. if there's a circular dependencies return an error
5. resolve the "include"s by merging the trees into one tree.

For handling the first requirement, I'll store the "common files" somewhere in the installation of the tool, and resolve them as a regular files.

What do you think about this solution?
How other compilers handle this kind of situation?

I'd like to get your feedback, and I'm really open for other suggestions.

Thank you guys.
[There's basically two approaches.  One is more or less what you're
doing, treat each file as a separate module, and combine them at tne
end.  The other is what C does, combine everything into one lexical
stream and scan and parse it as a unit.  Either way you can deal with
a recursive include either by treating it as an error, or by ignoring
includes of files you've seen before, depending on how the language
works.  The C approach is easier to implement, the separate module
approach has more flexibility.  For example, Python uses the separate
module approach, and caches parsed modules so on subsequent runs it
can just load the parsed version. -John]

[toc] | [next] | [standalone]

#1875

On 23/05/2017 15:12, dror.openu@gmail.com wrote:
> I'm implementing right now a "compiler" for a DSL that used at my work, and I'd like to get your advice for a dilemma I have.
> One of the requirements at the begging was that the compiler should get an input only one file.
>
> But, it's changed and I asked to add two more capabilities. and they are:
> [ some sort of multiple input file ]

This depends on lots of things. Are you taking something that would have
been one big file, and simply allowing it to be split?

Do the STD common parts, something that would have been part of one
file, now have to be part of each of the multiple files?

How does each file know about what's in the other files, when it's
something not in those STD common included files?

One easy solution (assuming having to compile all the sources is not an
issue) might be to just load all the files into memory and combine them
into one file. Then compile as a single file, dealing with possible
multiple includes, or out-of-order references, if that would be a problem.

But it really depends on how this language works.

(All the compiler projects I do now aim to compile all modules of a
project together. Some will have a hierarchical module system and that
can help: there is one symbol table, with the whole program as the root,
and (as I do it with modules sharing a flat namespace), each source file
or module at the next level.

Below that, whatever is defined within each module. This however is the
symbol table tree, not the parse tree. Individual parse trees exist per
function. This is why it depends a great deal on the language.)


--
bartc

[toc] | [prev] | [next] | [standalone]

#1880

On Thursday, May 25, 2017 at 5:58:28 PM UTC+3, bartc wrote:
> On 23/05/2017 15:12, dror.openu@gmail.com wrote:
> > I'm implementing right now a "compiler" for a DSL that used at my work, and I'd like to get your advice for a dilemma I have.
> > One of the requirements at the begging was that the compiler should get an input only one file.
> >
> > But, it's changed and I asked to add two more capabilities. and they are:
> > [ some sort of multiple input file ]
>
> This depends on lots of things. Are you taking something that would have
> been one big file, and simply allowing it to be split?

Yes. this is exactly the case.
The purpose of this language is to configure jobs we have in our system.
(the compiler will convert it to "byte code" that will be executed by the interpreter).
A good example is when you want to share configuration between group of jobs (or want to "inherit" configuration from a job).
You want to be able to create a "base" file, and use it somewhere in the project (I want to parse the "base" file only once, even if it imported multiple times).


> One easy solution (assuming having to compile all the sources is not an
> issue) might be to just load all the files into memory and combine them
> into one file. Then compile as a single file, dealing with possible
> multiple includes, or out-of-order references, if that would be a problem.
>

I prefer to parse each file separately, and only once.
it will also give a good, readable errors (file name + line number.)


I'm kind of newbie in the compilers world. my experience is only the university course.
So I really appreciate your help.

Thanks

[toc] | [prev] | [next] | [standalone]

#1881

On 25/05/2017 20:51, dror.openu@gmail.com wrote:
> On Thursday, May 25, 2017 at 5:58:28 PM UTC+3, bartc wrote:
>> On 23/05/2017 15:12, dror.openu@gmail.com wrote:
>>> I'm implementing right now a "compiler" for a DSL that used at my work, and I'd like to get your advice for a dilemma I have.
>>> One of the requirements at the begging was that the compiler should get an input only one file.
>>>
>>> But, it's changed and I asked to add two more capabilities. and they are:
>>> [ some sort of multiple input file ]

> The purpose of this language is to configure jobs we have in our system.
> (the compiler will convert it to "byte code" that will be executed by the interpreter).
> A good example is when you want to share configuration between group of jobs (or want to "inherit" configuration from a job).
> You want to be able to create a "base" file, and use it somewhere in the project (I want to parse the "base" file only once, even if it imported multiple times).

A lot more information is still needed.

Presumably you can't split a file at arbitrary points, as in the
middle of a token, or in the middle of an expression or statement.

The separate files must make sense by themselves. But, could they then
meaningfully be parsed as independent files? If file A refers to 'X',
but X is now declared in file B, how is that going to work? That is, if
knowing X is necessary to successfully processing A.


>> One easy solution (assuming having to compile all the sources is not an
>> issue) might be to just load all the files into memory and combine them
>> into one file. Then compile as a single file, dealing with possible
>> multiple includes, or out-of-order references, if that would be a problem.
>>
>
> I prefer to parse each file separately, and only once.
> it will also give a good, readable errors (file name + line number.)

You mean, only once during the processing of the whole program?

Is there any output resulting from processing each file? A traditional
compiler turns a source file into an object file. Then it won't need
compiling again until the source changes (or the compiler does).

Or is the output into memory? That sounds like all the files must be
processed, perhaps in a particular order.

Perhaps you can give a simple exercise of a short program in the source
language, and what it looks like split into two or three files.

--
bartc

[toc] | [prev] | [next] | [standalone]

#1876

Am 23.05.2017 um 16:12 schrieb dror.openu@gmail.com:

> My dilemma is: "How's the compile process should be when you have many files". I thought about this process for the second requirement, and I'd like to hear your opinion:
> 1. scan all the files (assume we have k files).
> 2. for each of the files:
> 2.1 run the lexer together with parser and build a parse tree.
> 3. now, I have k parse trees (but only one entry-point/main).
> 4. if there's a circular dependencies return an error
> 5. resolve the "include"s by merging the trees into one tree.

John already pointed out two commonly used methods, so let me only add
some keys.

A *preprocessor* can be implemented as a stand-alone tool, that resolves
the "include ..." directives, and produces a combined output file that
then can be compiled. Or it does the same as kind of a filter, sitting
between the lexer and parser, so that it can feed one sequential stream
of tokens to the parser. The preprocessor either switches its input
amongst files, in order to produce a combined file, or it switches
amongst lexers, each of which processes a single input file.

A *linker* resolves references between object modules, or AST's, so that
the result is either a single object or executable file, or a single
combined AST (parse tree). A clever linker can cache individual
definitions (variables, functions...), and load only the really required
definitions on demand. This technique allows to eliminate all unused
parts of the source files from the final program. Again a linker can be
implemented as a stand-alone tool, or as part of a compiler.

Caveat: If your "include" directive allows to insert *incomplete*
definitions, e.g. individual lines of code into a function, only a
preprocessor can produce valid input for the parser. Otherwise, if all
definitions can be parsed without occurences of embedded "include"
directives, all definitions (of identifiers) can be parsed individually,
and merged into a common list of definitions, as the top level of the
parse tree, or as a list of individual parse trees for each listed
definition.

DoDi

[toc] | [prev] | [next] | [standalone]

#1877

Don't do what C and C++ do and implement includes, it's a severely brain dead
approach. Implement a real module / unit / package system and when the
compiler reaches and import / use / with / whatever statement, the compiler
needs to read in the symbol table for that.

Do what Ada does and allow the driver to take many files and just compile
them.

Luke.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.compilers

csiph-web