Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #100994 > unrolled thread

'Amalgamated' source code

Back to article view | Back to comp.lang.c

Contents

  'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-03 15:14 +0000
    Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-03 08:47 -0800
      Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-03 19:47 +0000
        Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-03 12:48 -0800
          Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-03 21:36 +0000
            Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-03 16:01 -0800
              Re: 'Amalgamated' source code Keith Thompson <kst-u@mib.org> - 2017-01-03 16:26 -0800
                Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-03 18:00 -0800
                  Re: 'Amalgamated' source code Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-04 03:34 +0000
                    Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-04 03:31 -0800
                      Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-04 12:03 +0000
                        Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-04 05:00 -0800
                          Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-05 02:38 -0800
                            Re: 'Amalgamated' source code Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-05 14:28 +0000
                              Re: 'Amalgamated' source code supercat@casperkitty.com - 2017-01-05 08:20 -0800
                                Re: 'Amalgamated' source code Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-05 17:05 +0000
                                  Re: 'Amalgamated' source code supercat@casperkitty.com - 2017-01-05 10:11 -0800
                                    Re: 'Amalgamated' source code Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-05 19:46 +0000
                                      Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-05 12:15 -0800
                                        Re: 'Amalgamated' source code Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-05 21:54 +0000
                              Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-05 17:40 +0000
                                Re: 'Amalgamated' source code Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-05 19:33 +0000
              Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-04 10:53 +0000
            Re: 'Amalgamated' source code Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-04 01:51 +0000
          Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-04 11:54 +0000
            Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-04 05:57 -0800
              Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-04 14:20 +0000
              Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-04 15:07 +0000
                Re: 'Amalgamated' source code Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-04 15:34 +0000
                  Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-04 08:03 -0800
                    Re: 'Amalgamated' source code Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-04 17:09 +0000
                  Re: 'Amalgamated' source code bart4858@gmail.com - 2017-01-04 08:54 -0800
                    Re: 'Amalgamated' source code Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-04 17:22 +0000
                Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-04 07:55 -0800
                  Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-05 18:04 +0000
                    Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-01-05 11:50 -0800
    Re: 'Amalgamated' source code Jorgen Grahn <grahn+nntp@snipabacken.se> - 2017-01-04 16:29 +0000
      Re: 'Amalgamated' source code bart4858@gmail.com - 2017-01-04 09:15 -0800
        Re: 'Amalgamated' source code Ian Collins <ian-news@hotmail.com> - 2017-01-05 08:33 +1300
          Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-04 20:26 +0000
            Re: 'Amalgamated' source code supercat@casperkitty.com - 2017-01-04 13:09 -0800
            Re: 'Amalgamated' source code Ian Collins <ian-news@hotmail.com> - 2017-01-05 10:21 +1300
              Re: 'Amalgamated' source code supercat@casperkitty.com - 2017-01-04 13:51 -0800
              Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-04 22:02 +0000
                Re: 'Amalgamated' source code Ian Collins <ian-news@hotmail.com> - 2017-01-05 11:15 +1300
        Re: 'Amalgamated' source code Jorgen Grahn <grahn+nntp@snipabacken.se> - 2017-01-07 06:38 +0000
          Re: 'Amalgamated' source code BartC <bc@freeuk.com> - 2017-01-07 11:35 +0000
      Re: 'Amalgamated' source code supercat@casperkitty.com - 2017-01-04 09:53 -0800
      Re: 'Amalgamated' source code Malcolm McLean <malcolm.mclean5@btinternet.com> - 2017-01-04 11:50 -0800
      Re: 'Amalgamated' source code raltbos@xs4all.nl (Richard Bos) - 2017-01-10 01:34 +0000
        Re: 'Amalgamated' source code Malcolm McLean <malcolm.mclean5@btinternet.com> - 2017-01-10 01:59 -0800
        Re: 'Amalgamated' source code Les Cargill <lcargill99@comcast.com> - 2017-01-10 06:55 -0600
        Re: 'Amalgamated' source code Richard Heathfield <rjh@cpax.org.uk> - 2017-01-10 14:48 +0000
    Re: 'Amalgamated' source code Thiago Adams <thiago.adams@gmail.com> - 2017-04-26 07:17 -0700

Page 1 of 3  [1] 2 3  Next page →

#100994 — 'Amalgamated' source code

Interesting article here on converting large projects into just a few 
source files:

https://blog.forrestthewoods.com/improving-open-source-with-amalgamation-cf293592c5f4#.jzgq6ppvn

It's something I could do with because I always have trouble compiling 
open source projects. Especially now as I'm looking for test input to 
try out my tools on.

I do it myself when distributing some of projects, but it's a laborious 
set of manual steps.

(My new C processor ought to be able to do it automatically, but the 
starting point must be code that already compiles conventionally. Such 
projects are in short supply! Large projects anyway.

I've got a single file, 800Kloc version of gcc [for the 'cc1' module], 
but it doesn't compile under gcc! But then it has Linux dependencies, so 
maybe I can take care of those.)

-- 
Bartc

[toc] | [next] | [standalone]

#101010

On Tuesday, January 3, 2017 at 1:14:53 PM UTC-2, Bart wrote:
> Interesting article here on converting large projects into just a few 
> source files:
> 
> https://blog.forrestthewoods.com/improving-open-source-with-amalgamation-cf293592c5f4#.jzgq6ppvn
> 
> It's something I could do with because I always have trouble compiling 
> open source projects. Especially now as I'm looking for test input to 
> try out my tools on.
> 
> I do it myself when distributing some of projects, but it's a laborious 
> set of manual steps.
> 
> (My new C processor ought to be able to do it automatically, but the 
> starting point must be code that already compiles conventionally. Such 
> projects are in short supply! Large projects anyway.
> 
> I've got a single file, 800Kloc version of gcc [for the 'cc1' module], 
> but it doesn't compile under gcc! But then it has Linux dependencies, so 
> maybe I can take care of those.)
> 
> -- 

Hi, I like the concept of one-unit of compilation.
(I also have implemented an C preprocessor.)
I think there are two options to generate this amalgamation.

1) Mechanic form
 - Just merge files avoiding to include header files more than once.
 - Programmer needs to avoid name conflicts for the internal linkage functions. (let's say there are two static void F())
 - "system" or "external headers" are not included.
 - Programmer can create a macro PRIVATE that can be 
defined as static for the amalgamate and empty for normal 
development using multiple files.

2) " Transpilation" - generate C code
Read many C files and generates one file doing what is necessary to avoid internal linkage conflicts.

The generated code also could transform functions into static 
to make internal linkage if they are not "exported".

I was playing with this idea because I wanted to use my parser to extend the C language but generating one ANSI C file.

Let's say I want to implement the default initialization in a extended C language.

--file1.h--
#include <stdio.h>

struct X {
 int i = 1;
};

--file1.c--

#include "file1.h" 
void f()
{
  struct X x = {};
}

----------
Then the generated C file would be:

#include <stdio.h>

struct X {
 int i;
};

void f() {
  struct X x = {1};
}


Note that because I changed the declaration I cannot include file1.h but I still including system files <stdio.h>
For static function inside many files I would have to rename each function when generating one file

For instance:

--file1.c--

static void F() { }

void F2() {   F(); }

--file2.c--

static void F() {}

void F3() {  F(); }

--
merged file

static void _FILE1_F() { }
void F2() {   _FILE1_F(); }

static void _FILE2_F() { }

void F3() { _FILE2_F();}


---

More references:

https://www.sqlite.org/amalgamation.html

https://github.com/vinniefalco/Amalgamate

[toc] | [prev] | [next] | [standalone]

#101024

On 03/01/2017 16:47, Thiago Adams wrote:
> On Tuesday, January 3, 2017 at 1:14:53 PM UTC-2, Bart wrote:
>> Interesting article here on converting large projects into just a few
>> source files:
>>
>> https://blog.forrestthewoods.com/improving-open-source-with-amalgamation-cf293592c5f4#.jzgq6ppvn

> Hi, I like the concept of one-unit of compilation.
> (I also have implemented an C preprocessor.)

(I'm the middle of one at the minute. Most online refs on writing 
compilers tend to ignore this piffling little obstacle that you have to 
overcome first!)

> I think there are two options to generate this amalgamation.

> 2) " Transpilation" - generate C code
> Read many C files and generates one file doing what is necessary to avoid internal linkage conflicts.
>
> The generated code also could transform functions into static
> to make internal linkage if they are not "exported".
>
> I was playing with this idea because I wanted to use my parser to extend the C language but generating one ANSI C file.
>
> Let's say I want to implement the default initialization in a extended C language.
>
> --file1.h--
> #include <stdio.h>
...
> #include "file1.h"
...

>Then the generated C file would be:
>
> #include <stdio.h>
>
> struct X {
>  int i;
> };
>
> void f() {
>   struct X x = {1};
> }
>
>
> Note that because I changed the declaration I cannot include file1.h but I still including system files <stdio.h>
> For static function inside many files I would have to rename each function when generating one file

> --file1.c--
>
> static void F() { }
>
> void F2() {   F(); }
...
> merged file
>
> static void _FILE1_F() { }
> void F2() {   _FILE1_F(); }

This is like the approach I use at the minute, but part of a project 
where non-C source is converted to C, still a module at a time. A 
special compile option is used to change how declarations are generated 
and how global names are treated (by adding prefixing, but I'm also 
experimenting with no prefix, and manually avoiding clashes, as 
otherwise the output looks horrific. Or more horrifix!).

In this form, it can't be processed a C compiler as normal. A separate 
step is needed that scans all the modules, and collates the blocks of 
code, so that all type declarations, all variables etc are grouped 
together at the top of the file. (Example 
https://github.com/bartg/langs/blob/master/pcc.c)

It's a bit of a pain (and there are complications to do with the target 
OS where the right modules have to be picked). I'll be glad when my new 
C to C translator works then it'll be automatic.

 > https://www.sqlite.org/amalgamation.html

I wonder why other open source products don't take the same approach? 
Sometimes people just aren't interested in the details of how a project 
is structured, they just want to compile whatever source code is needed 
and end up with an executable, without any fuss, or having to install 
half of Unix.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#101032

On Tuesday, January 3, 2017 at 5:47:24 PM UTC-2, Bart wrote:
> On 03/01/2017 16:47, Thiago Adams wrote:
> > On Tuesday, January 3, 2017 at 1:14:53 PM UTC-2, Bart wrote:
> >> Interesting article here on converting large projects into just a few
> >> source files:
> >>
> >> https://blog.forrestthewoods.com/improving-open-source-with-amalgamation-cf293592c5f4#.jzgq6ppvn
> 
> > Hi, I like the concept of one-unit of compilation.
> > (I also have implemented an C preprocessor.)
> 
> (I'm the middle of one at the minute. Most online refs on writing 
> compilers tend to ignore this piffling little obstacle that you have to 
> overcome first!)

I didn't find a good references. I did it completely wrong at the beginning.
This is an sample from C standard that shows me my first mistakes.

#define hash_hash # ## #
#define mkstr(a) # a
#define in_between(a) mkstr(a)
#define join(c, d) in_between(c hash_hash d)
char p[] = join(x, y);
// equivalent to
// char p[] = "x ## y";

I saw much more mistakes, parsing real code. (windows.h)
It also requires expression evaluation for #if with the operator defined etc.

[...]


> OS where the right modules have to be picked). I'll be glad when my new 
> C to C translator works then it'll be automatic.

What is this "C to C translator"? 

I guess C Front, had these problems at the beginning when it use to generate C code.
C++ headers (class declaration) have different representation in C.
I think one possible solution could be generate an include with different declarations.

for instance:

--file1.h
class X { int i; };

--file1.cpp
#include "file1.h"

So to generate C:

--file1_modified.h

struct X { int i; };

--file1.c
#include "file1_modified.h"


The include still there but replaces by file1_modified.h
If the generated file is not for humans then the code could be all expanded with no includes.
But I think if the generated code looks natural then the adoption 
of some tool would be better because the programmers could take ownership of the code if they didn't like the tool.
This is another problem, how to keep macros at the generated code instead of expand it all.



> I wonder why other open source products don't take the same approach? 
> Sometimes people just aren't interested in the details of how a project 
> is structured, they just want to compile whatever source code is needed 
> and end up with an executable, without any fuss, or having to install 
> half of Unix.

I know more two projects using just one unit of compilation. One looks "merged" and the other manually created with in one file.

http://duktape.org/guide.html

[toc] | [prev] | [next] | [standalone]

#101040

On 03/01/2017 20:48, Thiago Adams wrote:
> On Tuesday, January 3, 2017 at 5:47:24 PM UTC-2, Bart wrote:
>> On 03/01/2017 16:47, Thiago Adams wrote:

>>> (I also have implemented an C preprocessor.)
>>
>> (I'm [in] the middle of one at the minute. Most online refs on writing
>> compilers tend to ignore this piffling little obstacle that you have to
>> overcome first!)
>
> I didn't find a good references. I did it completely wrong at the beginning.
> This is an sample from C standard that shows me my first mistakes.
>
> #define hash_hash # ## #
> #define mkstr(a) # a
> #define in_between(a) mkstr(a)
> #define join(c, d) in_between(c hash_hash d)
> char p[] = join(x, y);
> // equivalent to
> // char p[] = "x ## y";
>
> I saw much more mistakes, parsing real code. (windows.h)

I'll have to make a collection of tricky examples for my testing.

(But I've just seen one detail I hadn't allowed for, in gcc.c (from 
http://people.csail.mit.edu/smcc/projects/single-file-programs/). On 
line 733881 in a macro call:

DEF_FORMAT_ATTRIBUTE(PRINTF,1,1_0)

The token '1_0' appears. It starts as a number but presumably must be 
treated as alphanumeric.

I don't know if that puts paid to the idea of introducing underscores as 
numeric separators; from the bizarreness of how the CPP works, probably 
not!)

>> OS where the right modules have to be picked). I'll be glad when my new
>> C to C translator works then it'll be automatic.
>
> What is this "C to C translator"?

It was originally a compiler for another language with C source as a 
possible target (which builds entire projects). Then I thought it would 
be more fun to work on a front end for C source.

But retaining a C output option has several benefits, including 
converting multi-module apps to single-file C source code, and 
translating experimental language extensions to standard C.

> I guess C Front, had these problems at the beginning when it use to generate C code.
> C++ headers (class declaration) have different representation in C.
> I think one possible solution could be generate an include with different declarations.

C must be able to directly implement what is being expressed in the 
source language. I've had to drop (or avoid using) some features because 
of that restriction.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#101058

On Tuesday, January 3, 2017 at 7:36:53 PM UTC-2, Bart wrote:
> On 03/01/2017 20:48, Thiago Adams wrote:
> > On Tuesday, January 3, 2017 at 5:47:24 PM UTC-2, Bart wrote:
> >> On 03/01/2017 16:47, Thiago Adams wrote:
> 
> >>> (I also have implemented an C preprocessor.)
> >>
> >> (I'm [in] the middle of one at the minute. Most online refs on writing
> >> compilers tend to ignore this piffling little obstacle that you have to
> >> overcome first!)
> >
> > I didn't find a good references. I did it completely wrong at the beginning.
> > This is an sample from C standard that shows me my first mistakes.
> >
> > #define hash_hash # ## #
> > #define mkstr(a) # a
> > #define in_between(a) mkstr(a)
> > #define join(c, d) in_between(c hash_hash d)
> > char p[] = join(x, y);
> > // equivalent to
> > // char p[] = "x ## y";
> >
> > I saw much more mistakes, parsing real code. (windows.h)
> 
> I'll have to make a collection of tricky examples for my testing.
> 
> (But I've just seen one detail I hadn't allowed for, in gcc.c (from 
> http://people.csail.mit.edu/smcc/projects/single-file-programs/). On 
> line 733881 in a macro call:
> 
> DEF_FORMAT_ATTRIBUTE(PRINTF,1,1_0)
> 
> The token '1_0' appears. It starts as a number but presumably must be 
> treated as alphanumeric.

This did not work for me as well.

#define M(1_0)

M(1)

I have used the same tokens for preprocessor and parser. I just treat they differently.
I think my tokenizer will output [1:number] and [_0:identifier]. I guess my problem is simple i just count the wrong number of arguments for M. I will try to fix later.

 
> I don't know if that puts paid to the idea of introducing underscores as 
> numeric separators; from the bizarreness of how the CPP works, probably 
> not!)
> 
> >> OS where the right modules have to be picked). I'll be glad when my new
> >> C to C translator works then it'll be automatic.
> >
> > What is this "C to C translator"?
> 
> It was originally a compiler for another language with C source as a 
> possible target (which builds entire projects). Then I thought it would 
> be more fun to work on a front end for C source.
> 
> But retaining a C output option has several benefits, including 
> converting multi-module apps to single-file C source code, and 
> translating experimental language extensions to standard C.

If you do language extensions for instance I think the users would like to see one "natural" code.
This is almost like a code format, where you must keep comments and macro without expansion. 
For instance, if the programmer use #ifdef then the code extension should be always processed for each #ifdef option. If the generated C code were "natural" it could include these #ifdefs. Also for debbuging. And instead of having one big file you could keep the same file logic that user choose. As an option the compiler could generate the module one-file only. 

Today I can read one translation unit and generate the AST.  After that I can generate C code again.
If I do that, the generated code will contain all the definitions from system includes for instance, and no macros.
For each declaration I have the source file, so I can ignore declaration from extern code. Doing this the generated code is small but all the macros and #ifs disappears  (expanded).

I think I can keep comments inside the AST as something appended on each ast node.

for macros I dont have idea what to do.

[toc] | [prev] | [next] | [standalone]

#101063

Thiago Adams <thiago.adams@gmail.com> writes:
> On Tuesday, January 3, 2017 at 7:36:53 PM UTC-2, Bart wrote:
[...]
>> (But I've just seen one detail I hadn't allowed for, in gcc.c (from 
>> http://people.csail.mit.edu/smcc/projects/single-file-programs/). On 
>> line 733881 in a macro call:
>> 
>> DEF_FORMAT_ATTRIBUTE(PRINTF,1,1_0)
>> 
>> The token '1_0' appears. It starts as a number but presumably must be 
>> treated as alphanumeric.

1_0 is a pp-number, which will result in a syntax error if it survives
preprocessing (assuming you're trying to implement ISO C requirements).

> This did not work for me as well.
>
> #define M(1_0)
>
> M(1)

What did you expect that to do?  If the macro name in a #define is
followed by a '(' with no whitespace, the '( is followed by a list of
macro parameter names, which must be identifiers.  1_0 is not valid in
that context.  "gcc -E" complains:

    error: "1_0" may not appear in macro parameter list

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#101074

On Tuesday, January 3, 2017 at 10:25:20 PM UTC-2, Keith Thompson wrote:
> Thiago Adams  writes:
> > On Tuesday, January 3, 2017 at 7:36:53 PM UTC-2, Bart wrote:
> [...]
> >> (But I've just seen one detail I hadn't allowed for, in gcc.c (from 
> >> http://people.csail.mit.edu/smcc/projects/single-file-programs/). On 
> >> line 733881 in a macro call:
> >> 
> >> DEF_FORMAT_ATTRIBUTE(PRINTF,1,1_0)
> >> 
> >> The token '1_0' appears. It starts as a number but presumably must be 
> >> treated as alphanumeric.
> 
> 1_0 is a pp-number, which will result in a syntax error if it survives
> preprocessing (assuming you're trying to implement ISO C requirements).
> 
> > This did not work for me as well.
> >
> > #define M(1_0)
> >
> > M(1)
> 
> What did you expect that to do?  If the macro name in a #define is
> followed by a '(' with no whitespace, the '( is followed by a list of
> macro parameter names, which must be identifiers.  1_0 is not valid in
> that context.  "gcc -E" complains:
> 
>     error: "1_0" may not appear in macro parameter list

I have done an wrong sample: 
So the previous one , I just need to emit a
better error message. (I got an assert)

Let's try again:

#define M(a) X##a

M(1_0)

my preprocessor output is:

X1 _0 

the VC preprocessor output is :

X1_0

So my preprocessor still wrong.
Before to expand M , the argument 1_0 is expanded to 1 _0.
I insert spaces.

[toc] | [prev] | [next] | [standalone]

#101076

Thiago Adams <thiago.adams@gmail.com> writes:
<snip>
> Let's try again:
>
> #define M(a) X##a
>
> M(1_0)
>
> my preprocessor output is:
>
> X1 _0 
>
> the VC preprocessor output is :
>
> X1_0
>
> So my preprocessor still wrong.
> Before to expand M , the argument 1_0 is expanded to 1 _0.
> I insert spaces.

Yes, because you are treating 1_0 as two tokens.  You are doing this
because you've not followed the syntax for pp-token in the standard.
1_0 is a single token: a pp-number.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#101085

On Wednesday, January 4, 2017 at 1:34:51 AM UTC-2, Ben Bacarisse wrote:
> Thiago Adams  writes:
> <snip>
> > Let's try again:
> >
> > #define M(a) X##a
> >
> > M(1_0)
> >
> > my preprocessor output is:
> >
> > X1 _0 
> >
> > the VC preprocessor output is :
> >
> > X1_0
> >
> > So my preprocessor still wrong.
> > Before to expand M , the argument 1_0 is expanded to 1 _0.
> > I insert spaces.
> 
> Yes, because you are treating 1_0 as two tokens.  You are doing this
> because you've not followed the syntax for pp-token in the standard.
> 1_0 is a single token: a pp-number.
> 

Thanks.
I didn't see pp-number correctly yesterday.
Creating this single token it will work.

-
I am still adding extra spaces between tokens (except for ##)


#define M(a) a

M(   +++++   )

My preprocessor
++ ++ + 

VC preprocessor

+++++

But I think this will not create problems except more 
text to see.

[toc] | [prev] | [next] | [standalone]

#101087

On 04/01/2017 11:31, Thiago Adams wrote:

> I am still adding extra spaces between tokens (except for ##)
>
>
> #define M(a) a
>
> M(   +++++   )
>
> My preprocessor
> ++ ++ +
>
> VC preprocessor
>
> +++++
>
> But I think this will not create problems except more
> text to see.

What happens when you stringify the result? Eg:

#define M2(x) #x

M2(M(   +++++   ))



-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#101089

On Wednesday, January 4, 2017 at 10:03:30 AM UTC-2, Bart wrote:
> On 04/01/2017 11:31, Thiago Adams wrote:
> 
> > I am still adding extra spaces between tokens (except for ##)
> >
> >
> > #define M(a) a
> >
> > M(   +++++   )
> >
> > My preprocessor
> > ++ ++ +
> >
> > VC preprocessor
> >
> > +++++
> >
> > But I think this will not create problems except more
> > text to see.
> 
> What happens when you stringify the result? Eg:
> 
> #define M2(x) #x
> 
> M2(M(   +++++   ))
> 
> 
> 

#define M(a) a 
#define M2(x) #x 

M2(M(     +++++    ))

Visual C++ preprocessor and clang: (just one space added)

"M( +++++ )"

My preprocessor:
(I also need a unit test for this)

"++ ++ +" 

This was unexpected to me. 
Well, my preprocessor has a lot of errors. 
After to parser window.h I didn't check anymore if 
my preprocessor was right and just move to the parser.

I can try to create a github project with only the preprocessor.
Some parts of the parser are used also, because to evaluate #if expression I am using the same 'const-expression'  used in 
parser.

[toc] | [prev] | [next] | [standalone]

#101159

On Wednesday, January 4, 2017 at 11:00:30 AM UTC-2, Thiago Adams wrote:
> On Wednesday, January 4, 2017 at 10:03:30 AM UTC-2, Bart wrote:
> > On 04/01/2017 11:31, Thiago Adams wrote:
> > 
> > > I am still adding extra spaces between tokens (except for ##)
> > >
> > >
> > > #define M(a) a
> > >
> > > M(   +++++   )
> > >
> > > My preprocessor
> > > ++ ++ +
> > >
> > > VC preprocessor
> > >
> > > +++++
> > >
> > > But I think this will not create problems except more
> > > text to see.
> > 
> > What happens when you stringify the result? Eg:
> > 
> > #define M2(x) #x
> > 
> > M2(M(   +++++   ))
> > 
> > 
> > 
> 
> #define M(a) a 
> #define M2(x) #x 
> 
> M2(M(     +++++    ))
> 
> Visual C++ preprocessor and clang: (just one space added)
> 
> "M( +++++ )"
> 
> My preprocessor:
> (I also need a unit test for this)
> 
> "++ ++ +" 
> 
> This was unexpected to me. 

I did one more sample to see what happens:

#define M(a) X##a
#define M2(a, b) #a a b

M2(  M ( 1), M(1))

VC++ preprocessor output:

"M ( 1)" X1 X1

The same argument 'a' for macro can or not be expanded before 
the macro invocation.

Today I keep a map "parameter-name:a" - > "expanded:X1"

To fix my preprocessor I will keep both:

  "parameter-name:a" - > ["expanded:X1", "not expanded:M ( 1)"] 

and use the appropriated value depending on #a or a.

I will do the expansion (X1) before the macro call, I am 
not sure if is correct.

[toc] | [prev] | [next] | [standalone]

#101172

Thiago Adams <thiago.adams@gmail.com> writes:
<snip>
> I did one more sample to see what happens:
>
> #define M(a) X##a
> #define M2(a, b) #a a b
>
> M2(  M ( 1), M(1))
>
> VC++ preprocessor output:
>
> "M ( 1)" X1 X1
>
> The same argument 'a' for macro can or not be expanded before 
> the macro invocation.
>
> Today I keep a map "parameter-name:a" - > "expanded:X1"
>
> To fix my preprocessor I will keep both:
>
>   "parameter-name:a" - > ["expanded:X1", "not expanded:M ( 1)"] 
>
> and use the appropriated value depending on #a or a.

That seems like a very odd way to do it.  Have you read what the C
standard says about macro expansion?  It explains the algorithm quite
well, in my option.  If you expand macros at the right time, you won't
need to keep both expanded and un-expanded versions.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#101177

On Thursday, January 5, 2017 at 8:28:46 AM UTC-6, Ben Bacarisse wrote:
> That seems like a very odd way to do it.  Have you read what the C
> standard says about macro expansion?  It explains the algorithm quite
> well, in my option.  If you expand macros at the right time, you won't
> need to keep both expanded and un-expanded versions.

Given something like:

    #define join(a,b) a##b
    #define join2(a,b) join(a,b)
    #define plus() +
    #define incplusinc join2(+,plus()plus()++)
    #define incplusplusplus join2(+,plus()plus()plus()plus())

    #define asString2(x) #x
    #define asString(x) asString2(x)

    void useString(char const *st);
    void useInt(int x);
    volatile int xx,yy;
    void test(void)
    {
      useInt(xx incplusinc  yy);
      useString("#1:" asString(incplusinc));
      useInt(xx incplusplusplus yy);
      useString("#2:" asString(incplusplusplus));
    }

How could a compiler avoid having to either use different expansion logic
in cases where a macro is used as an operand to # versus cases where it
isn't, use some special delimiter for token boundaries that will evaporate
when string-expanded, or have the macro expansion logic produce two forms
of expanded macro based upon the intended usage?

Both incplusinc and incplusplusplus expand to the literal character sequence
+++++, but the first groups the tokens as ++ + ++ and the second as ++ + + +.
The plus() macro used on the right hand side of the token-pasting operator
must be expanded to + before the token-pasting operator is applied, and it
would seem that the other uses of plus() to the right of that operator would
get expanded at the same time.

What would you suggest as the process by which a compiler should handle the
above code?

[toc] | [prev] | [next] | [standalone]

#101181

supercat@casperkitty.com writes:
<snip>
> How could a compiler avoid having to either use different expansion logic
> in cases where a macro is used as an operand to # versus cases where it
> isn't,

The pre-processor should follow the specified algorithm.  The fact that
you refer to "a macro" bring an operand to # suggests you haven't
studied that algorithm.  The operand of # is a parameter name and the
corresponding argument sequence (a list of token) is never expanded
before being turned into a string.

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#101190

On Thursday, January 5, 2017 at 11:05:23 AM UTC-6, Ben Bacarisse wrote:
> supercat writes:
> <snip>
> > How could a compiler avoid having to either use different expansion logic
> > in cases where a macro is used as an operand to # versus cases where it
> > isn't,
> 
> The pre-processor should follow the specified algorithm.  The fact that
> you refer to "a macro" bring an operand to # suggests you haven't
> studied that algorithm.  The operand of # is a parameter name and the
> corresponding argument sequence (a list of token) is never expanded
> before being turned into a string.

Within the asString2 macro (the one that uses #), it acts upon the named
parameter x, but the purpose of that macro is to apply the operator to
the argument; the example code uses that macro twice, and in each case it
is applied to the expansion of another macro.  The expansions of both of
the macros to which it is applied have identical string representations
(i.e. +++++) but one behaves like ++ + ++ and the other behaves like
++ + + +.  Given the difference between the string representation that
would be needed for #, and the way in which the tokens would need to be
output for downstream processing, I would think keeping track of string
representations with and without the added spaces would be a reasonable
way to handle that (to get the string representation of two tokens that
appear without whitespace or are explicitly pasted, simply concatenate
their string representations; for the semantic form, either concatenate
the semantic forms with a whitespace between them (for consecutive
placement without a pasting operator) or with any whitespace between
them suppressed (when using the ## pasting operator).

While keeping track of both string forms would hardly be the only way to
handle things, it doesn't seem like an unreasonable one.  An alternative
would be to define an "invisible whitespace" character which will not get
included in strings generated by the # operator, but will get replaced
with whitespace in unquoted preprocessor output.

[toc] | [prev] | [next] | [standalone]

#101197

supercat@casperkitty.com writes:

> On Thursday, January 5, 2017 at 11:05:23 AM UTC-6, Ben Bacarisse wrote:
>> supercat writes:
>> <snip>
>> > How could a compiler avoid having to either use different expansion logic
>> > in cases where a macro is used as an operand to # versus cases where it
>> > isn't,
>> 
>> The pre-processor should follow the specified algorithm.  The fact that
>> you refer to "a macro" bring an operand to # suggests you haven't
>> studied that algorithm.  The operand of # is a parameter name and the
>> corresponding argument sequence (a list of token) is never expanded
>> before being turned into a string.
>
> Within the asString2 macro (the one that uses #), it acts upon the named
> parameter x, but the purpose of that macro is to apply the operator to
> the argument; the example code uses that macro twice, and in each case it
> is applied to the expansion of another macro.  The expansions of both of
> the macros to which it is applied have identical string representations
> (i.e. +++++) but one behaves like ++ + ++ and the other behaves like
> ++ + + +.  Given the difference between the string representation that
> would be needed for #, and the way in which the tokens would need to be
> output for downstream processing, I would think keeping track of string
> representations with and without the added spaces would be a reasonable
> way to handle that

I wouldn't.  I think it's much simpler to process token lists.  But I
haven't implemented it both way so I can't be 100% sure.  If your
intuition suggests multiple string representations are going to be a
good method, I can't talk you out of it, but my intuition (and
experience of having done it the other way) are saying that it's
bordering on bonkers.

Unfortunately it's too large a task to do just for the hell of finding
out, so let's agree to differ.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#101200

On Thursday, January 5, 2017 at 5:46:28 PM UTC-2, Ben Bacarisse wrote:
>  writes:
> 
> > On Thursday, January 5, 2017 at 11:05:23 AM UTC-6, Ben Bacarisse wrote:
> >> supercat writes:
> >> <snip>
> >> > How could a compiler avoid having to either use different expansion logic
> >> > in cases where a macro is used as an operand to # versus cases where it
> >> > isn't,
> >> 
> >> The pre-processor should follow the specified algorithm.  The fact that
> >> you refer to "a macro" bring an operand to # suggests you haven't
> >> studied that algorithm.  The operand of # is a parameter name and the
> >> corresponding argument sequence (a list of token) is never expanded
> >> before being turned into a string.
> >
> > Within the asString2 macro (the one that uses #), it acts upon the named
> > parameter x, but the purpose of that macro is to apply the operator to
> > the argument; the example code uses that macro twice, and in each case it
> > is applied to the expansion of another macro.  The expansions of both of
> > the macros to which it is applied have identical string representations
> > (i.e. +++++) but one behaves like ++ + ++ and the other behaves like
> > ++ + + +.  Given the difference between the string representation that
> > would be needed for #, and the way in which the tokens would need to be
> > output for downstream processing, I would think keeping track of string
> > representations with and without the added spaces would be a reasonable
> > way to handle that
> 
> I wouldn't.  I think it's much simpler to process token lists.  But I
> haven't implemented it both way so I can't be 100% sure.  If your
> intuition suggests multiple string representations are going to be a
> good method, I can't talk you out of it, but my intuition (and
> experience of having done it the other way) are saying that it's
> bordering on bonkers.
> 
> Unfortunately it's too large a task to do just for the hell of finding
> out, so let's agree to differ.
> 

My design for this preprocessor was to create an 
scanner where I could pull tokens already pre-processed.
I didn't want to preprocessor all files and generate 
temporaries before parsing. The dirty job must be done by
this "scanner" component.


To implement #idef #else for instance,
I had to create internal states because it's not using 
recursive functions. I have a stack for includes and 
macros being expanded. 

My idea to keep the "not-expand-string"  was to save the 
expansion code I have so far. But It can be completed replaced
by a new one and keep the other parts of my scanner.
That project I put on git is useful for that.

[toc] | [prev] | [next] | [standalone]

#101206

Thiago Adams <thiago.adams@gmail.com> writes:
<snip>
> My idea to keep the "not-expand-string"  was to save the 
> expansion code I have so far. But It can be completed replaced
> by a new one and keep the other parts of my scanner.
> That project I put on git is useful for that.

Sorry, I could not really follow this but it seems we may be talking
about different things.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.c

csiph-web