Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #101235 > unrolled thread

Macro Expansion

Back to article view | Back to comp.lang.c

Contents

  Macro Expansion BartC <bc@freeuk.com> - 2017-01-07 18:35 +0000
    Re: Macro Expansion Thiago Adams <thiago.adams@gmail.com> - 2017-01-07 11:23 -0800
      Re: Macro Expansion Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-07 21:28 +0000
    Re: Macro Expansion Siri Cruise <chine.bleu@yahoo.com> - 2017-01-07 11:45 -0800
      Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-07 20:16 +0000
        Re: Macro Expansion Siri Cruise <chine.bleu@yahoo.com> - 2017-01-07 12:23 -0800
          Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-07 20:31 +0000
            Re: Macro Expansion Siri Cruise <chine.bleu@yahoo.com> - 2017-01-07 12:42 -0800
            Re: Macro Expansion Ian Collins <ian-news@hotmail.com> - 2017-01-08 14:42 +1300
              Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-08 13:18 +0000
    Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-07 20:15 +0000
      Re: Macro Expansion Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-07 20:42 +0000
        Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-07 22:50 +0000
          Re: Macro Expansion Ian Collins <ian-news@hotmail.com> - 2017-01-08 11:59 +1300
            Re: Macro Expansion Malcolm McLean <malcolm.mclean5@btinternet.com> - 2017-01-07 15:10 -0800
          Re: Macro Expansion supercat@casperkitty.com - 2017-01-07 17:24 -0800
            Re: Macro Expansion Tim Rentsch <txr@alumni.caltech.edu> - 2017-01-15 21:55 -0800
              Re: Macro Expansion supercat@casperkitty.com - 2017-01-16 09:06 -0800
                Re: Macro Expansion Tim Rentsch <txr@alumni.caltech.edu> - 2017-01-31 08:51 -0800
                  Re: Macro Expansion supercat@casperkitty.com - 2017-01-31 13:06 -0800
                    Re: Macro Expansion Tim Rentsch <txr@alumni.caltech.edu> - 2017-02-03 07:49 -0800
                      Re: Macro Expansion supercat@casperkitty.com - 2017-02-03 12:03 -0800
                        Re: Macro Expansion Tim Rentsch <txr@alumni.caltech.edu> - 2017-02-09 09:00 -0800
          Re: Macro Expansion Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-08 01:37 +0000
            Re: Macro Expansion Jean-Marc Bourguet <jm@bourguet.org> - 2017-01-08 10:22 +0100
    Re: Macro Expansion Thiago Adams <thiago.adams@gmail.com> - 2017-01-07 12:18 -0800
      Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-07 20:28 +0000
        Re: Macro Expansion Thiago Adams <thiago.adams@gmail.com> - 2017-01-07 12:36 -0800
          Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-07 22:59 +0000
    Re: Macro Expansion Thiago Adams <thiago.adams@gmail.com> - 2017-01-08 00:20 -0800
      Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-08 13:30 +0000
        Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-10 14:00 +0000
          Re: Macro Expansion Thiago Adams <thiago.adams@gmail.com> - 2017-01-10 07:43 -0800
            Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-10 17:47 +0000
              Re: Macro Expansion Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-10 22:49 +0000
                Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-10 23:28 +0000
                Re: Macro Expansion "James R. Kuyper" <jameskuyper@verizon.net> - 2017-01-10 18:51 -0500
                  Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-11 00:06 +0000
            Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-10 20:30 +0000
              Re: Macro Expansion Thiago Adams <thiago.adams@gmail.com> - 2017-04-26 10:04 -0700
                Re: Macro Expansion Keith Thompson <kst-u@mib.org> - 2017-04-26 12:00 -0700
                  Re: Macro Expansion Thiago Adams <thiago.adams@gmail.com> - 2017-04-26 12:24 -0700
                Re: Macro Expansion bartc <bc@freeuk.com> - 2017-04-26 20:12 +0100
                  Re: Macro Expansion Thiago Adams <thiago.adams@gmail.com> - 2017-04-26 12:34 -0700
                  Re: Macro Expansion jameskuyper@verizon.net - 2017-04-26 13:46 -0700
    Re: Macro Expansion Jean-Marc Bourguet <jm@bourguet.org> - 2017-01-08 10:13 +0100
      Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-08 13:10 +0000
        Re: Macro Expansion Jean-Marc Bourguet <jm@bourguet.org> - 2017-01-08 15:51 +0100
          Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-08 17:17 +0000
            Re: Macro Expansion supercat@casperkitty.com - 2017-01-08 11:04 -0800
              Re: Macro Expansion Malcolm McLean <malcolm.mclean5@btinternet.com> - 2017-01-08 11:10 -0800
                Re: Macro Expansion supercat@casperkitty.com - 2017-01-08 14:08 -0800
              Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-08 19:49 +0000
                Re: Macro Expansion supercat@casperkitty.com - 2017-01-08 14:17 -0800
            Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-08 19:19 +0000
            Re: Macro Expansion Jean-Marc Bourguet <jm@bourguet.org> - 2017-01-08 21:43 +0100
        Re: Macro Expansion supercat@casperkitty.com - 2017-01-08 10:43 -0800
        Re: Macro Expansion Keith Thompson <kst-u@mib.org> - 2017-01-08 12:39 -0800
    Re: Macro Expansion antispam@math.uni.wroc.pl - 2017-01-08 22:54 +0000
      Re: Macro Expansion "James R. Kuyper" <jameskuyper@verizon.net> - 2017-01-08 18:36 -0500
        Re: Macro Expansion antispam@math.uni.wroc.pl - 2017-01-09 01:15 +0000
          Re: Macro Expansion "James R. Kuyper" <jameskuyper@verizon.net> - 2017-01-08 20:49 -0500
      Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-08 23:53 +0000
        Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-09 01:11 +0000
        Re: Macro Expansion antispam@math.uni.wroc.pl - 2017-01-09 01:31 +0000
          Re: Macro Expansion BartC <bc@freeuk.com> - 2017-01-10 11:28 +0000

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

#108558

Thiago Adams <thiago.adams@gmail.com> writes:
[...]
> One more test case, I saw this on windows headers.
>
>
> #define _VARIANT_BOOL    /##/
>
> struct {
>   _VARIANT_BOOL bool;
> }
>
> This will expand to :
>
> struct {
>  
> }

That's not valid.  Comments are replaced by spaces in translation phase 3.
Macros are expanded in phase 4.

gcc and clang both complain that pasting '/' and '/' does not give a
valid preprocessing token.  (And the empty struct would be a syntax
error anyway.)

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#108562

On Wednesday, April 26, 2017 at 4:00:08 PM UTC-3, Keith Thompson wrote:
> Thiago Adams  writes:
> [...]
> > One more test case, I saw this on windows headers.
> >
> >
> > #define _VARIANT_BOOL    /##/
> >
> > struct {
> >   _VARIANT_BOOL bool;
> > }
> >
> > This will expand to :
> >
> > struct {
> >  
> > }
> 
> That's not valid.  Comments are replaced by spaces in translation phase 3.
> Macros are expanded in phase 4.
> 
> gcc and clang both complain that pasting '/' and '/' does not give a
> valid preprocessing token.  (And the empty struct would be a syntax
> error anyway.)
> 

I found it on

C:\Program Files (x86)\Windows Kits\8.1\Include\shared\wtypes.h


#if !__STDC__ && (_MSC_VER <= 1000)
/* For backward compatibility */
typedef VARIANT_BOOL _VARIANT_BOOL;

#else
/* ANSI C/C++ reserve bool as keyword */
#define _VARIANT_BOOL    /##/
#endif

The real struct is not empty but it's too big to paste here.

The VC++ pre-processor gives the result I copied here using the sample struct.

[toc] | [prev] | [next] | [standalone]

#108561

On 26/04/2017 18:04, Thiago Adams wrote:

> One more test case, I saw this on windows headers.
>
>
> #define _VARIANT_BOOL    /##/
>
> struct {
>   _VARIANT_BOOL bool;
> }
>
> This will expand to :
>
> struct {
>
> }

I thought using macros to construct comments was not allowed?

Anyway, gcc gives a token pasting error.

Tiny C gives an error then gets stuck in a loop repeating it.

DMC gives an EOF error.

Lccwin presumably generates struct {bool;} as it reports an error about 
that. Pelles C I'm not sure. (MSVC I can't test on this machine.)

My program generates struct {bool;}.

But I'm not sure this is worth bothering with when every compiler does 
something different. And it's only in a header that is part of an 
implementation, so it's its headache to deal with.

However, antispam did mention a couple of other cases today which my 
program fails on, but other compilers deal with so I have to fix them. I 
think they are these two:

  #define define X
  #define A B

This one was easy. But this one:

  #define A(x) M(x)
  #define B A

  B(10)

(failing to expand a macro-call where one part is the result of 
expansion, so output is A(10) not M(10)) is going to be a nightmare, 
because of having to get back into the code.

-- 
bartc

[toc] | [prev] | [next] | [standalone]

#108564

On Wednesday, April 26, 2017 at 4:11:58 PM UTC-3, Bart wrote:
> On 26/04/2017 18:04, Thiago Adams wrote:
> 
> > One more test case, I saw this on windows headers.
> >
> >
> > #define _VARIANT_BOOL    /##/
> >
> > struct {
> >   _VARIANT_BOOL bool;
> > }
> >
> > This will expand to :
> >
> > struct {
> >
> > }
> 
> I thought using macros to construct comments was not allowed?
> 
> Anyway, gcc gives a token pasting error.
> 
> Tiny C gives an error then gets stuck in a loop repeating it.
> 
> DMC gives an EOF error.
> 
> Lccwin presumably generates struct {bool;} as it reports an error about 
> that. Pelles C I'm not sure. (MSVC I can't test on this machine.)
> 
> My program generates struct {bool;}.
> 
> But I'm not sure this is worth bothering with when every compiler does 
> something different. And it's only in a header that is part of an 
> implementation, so it's its headache to deal with.
> 
> However, antispam did mention a couple of other cases today which my 
> program fails on, but other compilers deal with so I have to fix them. I 
> think they are these two:
> 
>   #define define X
>   #define A B
> 
> This one was easy. But this one:
> 
>   #define A(x) M(x)
>   #define B A
> 
>   B(10)
> 
> (failing to expand a macro-call where one part is the result of 
> expansion, so output is A(10) not M(10)) is going to be a nightmare, 
> because of having to get back into the code.
> 
> -- 
> bartc

I found it because I am parsing the windows headers. 

(
My preprocessor is now based on this code:

https://github.com/dspinellis/cscout/blob/084d64dc7a0c5466dc2d505c1ca16fb303eb2bf1/src/macro.cpp

)

[toc] | [prev] | [next] | [standalone]

#108567

On Wednesday, April 26, 2017 at 3:11:58 PM UTC-4, Bart wrote:
> On 26/04/2017 18:04, Thiago Adams wrote:
> 
> > One more test case, I saw this on windows headers.
> >
> >
> > #define _VARIANT_BOOL    /##/
> >
> > struct {
> >   _VARIANT_BOOL bool;
> > }
> >
> > This will expand to :
> >
> > struct {
> >
> > }
> 
> I thought using macros to construct comments was not allowed?

The stringizing operator ## creates a single pre-processing token consisting of
//. "If the result is not a valid preprocessing token, the behavior is
undefined." (6.10.3.3p3). // is NOT a valid preprocessing token (6.4p1).

> Anyway, gcc gives a token pasting error.
> 
> Tiny C gives an error then gets stuck in a loop repeating it.
> 
> DMC gives an EOF error.
> 
> Lccwin presumably generates struct {bool;} as it reports an error about 
> that. Pelles C I'm not sure. (MSVC I can't test on this machine.)
> 
> My program generates struct {bool;}.
> 
> But I'm not sure this is worth bothering with when every compiler does 
> something different.

That's what "undefined behavior" allows. Therefore, you're free to deal with
the case in whichever way you prefer.

[toc] | [prev] | [next] | [standalone]

#101281

BartC <bc@freeuk.com> writes:

> I've knocked up a short script program to implement CPP macro expansion. It
> works with many of the examples I've tried, but I have little confidence
> that whatever I end up with will always work, as I'm guessing at the
> algorithms needed.
>
> The descriptions I've seen of the process seem to me ambiguous and
> difficult to understand (maybe they're striving to be as cryptic as the
> language they are describing!).

Have you looked at the latest standard?  I just looked at C90, C99 and C11
and it seems to be improving. For instance they make clear that given the
following macro definitions:

    #define f(a) a*g
    #define g(a) f(a)
    #define h(a) a*h
    
the invocation

    f(2)(9)

may expand to either

    2*f(9)

or

    2*9*g

although

    h(2)(9)

must be expanded to

    2*h(9)
    
> Here is what I've been able to understand of the process; maybe someone can
> tell me if it is on the right lines.
>
> Example:
>
>    #define XX 20
>    #define XXz 99
>    #define W W+W+XX+x
>    #define M(x,y)  x y #x y ## z M(a,b) W
>
>    M(10,XX)
>
> In the expansion of M(10,XX):
>
> * The token sequence '10' is used for argument x
>
> * The token sequence 'XX' is used for argument y.

There are two token sequences for each argument, one -- which is the one
you give -- is used for # and ##, the other, which is the previous token
sequence after replacement of all macros.  The macro M expansion is not
considered as started yet, thus M(M(foo),bar) is expanded as expected.

> * Start with the replacement token list as defined for M:
>
>     x y #x y ## z M(a,b) W
>
> * Replace x and y with their arguments, expanding, stringifying
>   or combining as needed:
>
>    10 20 "10" XXz M(a,b) W
>
>   Here, argument XX was separately macro expanded to 20 before replacing
>   the first y. The second y has ## so is not expanded.
>
> * After this substitution, the token list is scanned for macro calls to
>   expand. M is a macro call, but is the same name as the macro
>   currently being expanded, so maps to itself (and the (a,b) following
>   are just tokens).

It is not just map to itself, it is also marked in order to prevent
expansion further in the process.

> * That leaves XXz and W macros to expand, which now results in:
>
>    10 20 "10" 99 M(a,b) W+W+XX+x

Not really.  The macro XXz is fully expanded, during that process, M and
XXz are marked as not replacable as both are in the process of being
expanded.  Then XXz is replaced.  The same hold for W (but M and W are
marked here as not replacable).

10 20 "10" 99 M(a,b) W+W+20+x

> * The result is now scanned again looking for new macros to expand.
>   However, once any name (eg. M and W) has been expanded once, it is
>   never expanded again (in a subsequent pass). So only XX is changed
>   to result in:
>
>    10 20 "10" 99 M(a,b) W+W+20+x
>
>   (This agrees with what gcc -E gives, apart from an extra space before
>   +x)

The rescan here will do nothing.  (I think it does something only when the
expansion of a macro has a function macro name as the last token and after
the expansion it is inserted just before an open parenthesis, see the
f(2)(9) example)

> * Presumably that rule also applies to parameter names, so the new x at
>   end stays as x, it's not changed to 10.

Parameter names are handled specially only in the substitution phase, not
the rescan.

> * The re-scanning looking for new macros continues until there are no
>   macro expansions.
>
> * The # and ## processing, for # and ## present in the original
>   replacement list, is only done once. What's not clear what it will do
>   with # and ## introduced as the result of macro expansion (if that is
>   even possible).

There are three phases on macro expansion:
- parameter substitution (only for function like macro)
- handling of # and ## (# only for function like macro, both need to be in
the replacement list to be handled)
- repeated rescan

It is possible to have # and ## as the result of macro expansion.  The only
potential use (outside the use of the C processor for other languages) I
know is to apply # to that result to get a string.  The standard has this
example:

  #define hash_hash # ## #
  #define mkstr(a) # a
  #define in_between(a) mkstr(a)
  #define join(c, d) in_between(c hash_hash d)

  char p[] = join(x, y); // equivalent to char p[] = "x ## y";

(Note the very common trick of an intermediary macro to ensure an argument
of # or ## is expanded)

Yours,

-- 
Jean-Marc

[toc] | [prev] | [next] | [standalone]

#101289

On 08/01/2017 09:13, Jean-Marc Bourguet wrote:
> BartC <bc@freeuk.com> writes:
>
>> I've knocked up a short script program to implement CPP macro expansion. It
>> works with many of the examples I've tried, but I have little confidence
>> that whatever I end up with will always work, as I'm guessing at the
>> algorithms needed.
>>
>> The descriptions I've seen of the process seem to me ambiguous and
>> difficult to understand (maybe they're striving to be as cryptic as the
>> language they are describing!).
>
> Have you looked at the latest standard?

I use one for 201x dated 2009.

   I just looked at C90, C99 and C11
> and it seems to be improving. For instance they make clear that given the
> following macro definitions:
>
>     #define f(a) a*g
>     #define g(a) f(a)
>     #define h(a) a*h
>
> the invocation
>
>     f(2)(9)

I didn't even know you could do this! This is a simpler example:

    #define a b
    #define b(x) x x x
    a(10)

Result then needs to be 10 10 10.

Then is taken care of if 'a(10)' is part of a replacement token list 
being substituted then rescanned for macros.

But 'a(10)' can also be part of normal code, not inside a macro 
definition. Then the 'b' that is generated is in a different 
'token-space' from the "(10)" in the rest of the file (and likely to 
still exist as characters not tokens). Rescanning an entirely source 
file is hopelessly inefficient just for the off-chance someone might be 
using such macros.

That's going to mean some very untidy logic.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#101295

BartC <bc@freeuk.com> writes:

> But 'a(10)' can also be part of normal code, not inside a macro
> definition. Then the 'b' that is generated is in a different 'token-space'
> from the "(10)" in the rest of the file (and likely to still exist as
> characters not tokens). Rescanning an entirely source file is hopelessly
> inefficient just for the off-chance someone might be using such macros.

You don't rescan the whole file. The rescan of the expansion is made once
more than you though and in its context.

You don't know the steps people are ready to avoid using an external tool
while still using compact code. The technique is used in practice.  Look
for chaos pp and the boost preprocessor library for such kind of techniques
(the goal of those libraries is to encapsulate them in an more convenient
interface).

If your preprocessor is to be used with code that you don't control, you'll
meet such tricks.

Yours,

-- 
Jean-Marc

[toc] | [prev] | [next] | [standalone]

#101298

On 08/01/2017 14:51, Jean-Marc Bourguet wrote:
> BartC <bc@freeuk.com> writes:
>
>> But 'a(10)' can also be part of normal code, not inside a macro
>> definition. Then the 'b' that is generated is in a different 'token-space'
>> from the "(10)" in the rest of the file (and likely to still exist as
>> characters not tokens). Rescanning an entirely source file is hopelessly
>> inefficient just for the off-chance someone might be using such macros.
>
> You don't rescan the whole file. The rescan of the expansion is made once
> more than you though and in its context.
>
> You don't know the steps people are ready to avoid using an external tool
> while still using compact code. The technique is used in practice.  Look
> for chaos pp and the boost preprocessor library for such kind of techniques
> (the goal of those libraries is to encapsulate them in an more convenient
> interface).

I looked at chaos-pp. I didn't know the purpose of your comment and 
thought it was another pre-processor implementation to look at.

I just saw hundred and hundreds of files in dozens of folders: headers, 
xml, xsl and html. But I realised after a while that there were no .c 
files. Just 471 header files.

Presumably then (as the README was help, and the html, after I 
downloaded it all, had no obvious entry point), this is just an example 
of a massive library built solely with the C preprocessor.

And very daunting it looks. Even if I wasn't handicapped by having no 
idea what it's supposed to do, how does one even start testing their own 
pre-processor with something like this[1]? If it was normal C code 
(declarations, functions, all the familiar stuff), it would be very 
straightforward. But with this preprocessor language...

How did a no-nonsense language such as C end up with a dependency on a 
macro language so arcane that apparently few can fully understand it?

-------------------

([1] A testing method I'd envisioned was to write the preprocessed 
output as a flat text file, and do the same with something like gcc -E. 
Then compare the two visually, or compile them. Or feed my output 
through gcc -E and see if the result exactly matches the other.)

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#101301

On Sunday, January 8, 2017 at 11:18:03 AM UTC-6, Bart wrote:
> How did a no-nonsense language such as C end up with a dependency on a 
> macro language so arcane that apparently few can fully understand it?

A lot of things in computer languages are easier to implement than to
describe precisely in formal grammar.  Further, implementation can be
greatly eased by allowing for Unspecified behavior.  Unfortunately, the
C Standard tries to establish an unambiguous formal grammar to a language
that was not designed for that, and in so doing creates a language which
is harder to implement and yet inferior to many pre-existing implementations.

Further, because code which exploits intricacies of the rules was able
to do most of the things that could be done much more cleanly with a few
easy-to-implement extensions to the language, C never received any such
extensions (e.g. an operator which would evaluate the operand in the
way similar to an "if" and yield a constant of the longest integer type
and assign that to a macro name, thus allowing constructs like:

    #define COUNTER __MAC_EVAL_ASSIGN(count,count+1) count

Note that having the compiler evaluate the right-hand operand as an
integer would avoid issues of binding or nesting when evaluating the
newly-created macro, since it would simply be an integer).

[toc] | [prev] | [next] | [standalone]

#101303

On Sunday, January 8, 2017 at 7:04:30 PM UTC, supe...@casperkitty.com wrote:
> On Sunday, January 8, 2017 at 11:18:03 AM UTC-6, Bart wrote:
> > How did a no-nonsense language such as C end up with a dependency on a 
> > macro language so arcane that apparently few can fully understand it?
> 
> A lot of things in computer languages are easier to implement than to
> describe precisely in formal grammar.  
>
Exactly. You sometimes come across professional products with their 
own "enhanced" versions of some common standard language. You just
know that the real reason is not to provide enhancements, but because
it's easier to say that "the grammar of the language is defined by
the behaviour of the interpreter we happen to have written" rather 
than write an interpreter that conforms exactly to someone else's
standard.

[toc] | [prev] | [next] | [standalone]

#101313

On Sunday, January 8, 2017 at 1:10:56 PM UTC-6, Malcolm McLean wrote:
> On Sunday, January 8, 2017 at 7:04:30 PM UTC, supercat wrote:
> > A lot of things in computer languages are easier to implement than to
> > describe precisely in formal grammar.  
> >
> Exactly. You sometimes come across professional products with their 
> own "enhanced" versions of some common standard language. You just
> know that the real reason is not to provide enhancements, but because
> it's easier to say that "the grammar of the language is defined by
> the behaviour of the interpreter we happen to have written" rather 
> than write an interpreter that conforms exactly to someone else's
> standard.

Yup.  And that can sometimes be a good thing or a bad thing depending upon
whether such behaviors usefully extend the language in ways that end up
being widely implemented, or merely cause people to write in non-portable
fashion code which could just as well have been written in portable fashion.
I would suggest, though, that if programmers are frequently making use of
implementation details to do things which aren't provided for in the
documentation, that should be regarded as a sign that there is a need for
more documented functionality than exists.  Whether such need could better
be filled by promising to support the existing functionality, or providing
a new alternative and guaranteeing support for that, it is unhelpful to
deride as broken code which relies upon such behaviors *when there are no
practical documented alternatives*.

[toc] | [prev] | [next] | [standalone]

#101308

On 08/01/2017 19:04, supercat@casperkitty.com wrote:
> On Sunday, January 8, 2017 at 11:18:03 AM UTC-6, Bart wrote:
>> How did a no-nonsense language such as C end up with a dependency on a
>> macro language so arcane that apparently few can fully understand it?
>
> A lot of things in computer languages are easier to implement than to
> describe precisely in formal grammar.

The macro seems to be a counter example.

I keep getting told that everything I need to know is contained within a 
few paragraphs in the Standard. But the implementation, while not large, 
is considerably more difficult. And only works until the next example 
that reveals yet another aspect that hasn't been allowed for.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#101314

On Sunday, January 8, 2017 at 1:49:18 PM UTC-6, Bart wrote:
> On 08/01/2017 19:04, supercat wrote:
> > A lot of things in computer languages are easier to implement than to
> > describe precisely in formal grammar.
> 
> The macro seems to be a counter example.
> 
> I keep getting told that everything I need to know is contained within a 
> few paragraphs in the Standard. But the implementation, while not large, 
> is considerably more difficult. And only works until the next example 
> that reveals yet another aspect that hasn't been allowed for.

The Standard goes into all sorts of stuff about pp-numbers that end up
making the language quirkier than it would be if it simply said that
if a token which would match a decimal number with a trailing E is
followed by a decimal integer constant, or a sign and a decimal integer
constant, they'll be regarded together as a single token.  I haven't counted
up all the text describing the preprocessor, but it's more than a couple of
paragraphs.

[toc] | [prev] | [next] | [standalone]

#101304

On 08/01/2017 17:17, BartC wrote:
> On 08/01/2017 14:51, Jean-Marc Bourguet wrote:

>> Look
>> for chaos pp and the boost preprocessor library for such kind of
>> techniques

> I looked at chaos-pp.

 > how does one even start testing their own
> pre-processor with something like this?

Actually compiling it was not a problem.

(There no make and configure files, thank god, just 15 minutes of trial 
and error finding the right -I path option to gcc.)

Mainly it was all #defines; no actual uses of the macros, not outside of 
a macro definition, so there was virtually no output.

I found this example in one of the HTMLs, and decided to try it out 
(with a tweak so that it used a BYTE_... macro that was defined as the 
table was not filled in):

#include <chaos/preprocessor.h>
#include <chaos/preprocessor/algorithm/fold_left.h>
#include <chaos/preprocessor/arbitrary/add.h>
#include <chaos/preprocessor/arbitrary/demote.h>
#include <chaos/preprocessor/arbitrary/mul.h>
#include <chaos/preprocessor/config.h>
#include <chaos/preprocessor/control/while.h>
#include <chaos/preprocessor/detection/is_nullary.h>
#include <chaos/preprocessor/facilities/split.h>
#include <chaos/preprocessor/logical/compl.h>
#include <chaos/preprocessor/recursion/expr.h>
#include <chaos/preprocessor/string/core.h>

#if CHAOS_PP_VARIADICS

#define BINARY(x) BINARY_S(CHAOS_PP_STATE(), x)
#define BINARY_S(s, x) \
     CHAOS_PP_ARBITRARY_DEMOTE( \
         CHAOS_PP_EXPR_S(s)(CHAOS_PP_FOLD_LEFT_S( \
             s, ACC, \
             (CHAOS_PP_STRING) CHAOS_PP_SPLIT( \
                 0, \
                 CHAOS_PP_EXPR_S(s)(CHAOS_PP_WHILE_S( \
                     s, PRED, OP,, x \
                 )) \
             ), \
             (0) \
         )) \
     ) \
     /**/

#define ACC(s, bit, x) \
     CHAOS_PP_ARBITRARY_ADD( \
         CHAOS_PP_ARBITRARY_MUL(x, (2)), (bit) \
     ) \
     /**/

#define PRED(s, r, x) \
     CHAOS_PP_COMPL(CHAOS_PP_IS_NULLARY(x())) \
     /**/

#define OP(s, r, x) \
     r CHAOS_PP_SPLIT(0, BYTE_ ## x), CHAOS_PP_SPLIT(1, BYTE_ ## x) \
     /**/

#define BYTE_00000000 0 0 0 0 0 0 0 0,
#define BYTE_00000001 0 0 0 0 0 0 0 1,
#define BYTE_00000010 0 0 0 0 0 0 1 0,
#define BYTE_00000100 0 0 0 0 0 1 0 0,
#define BYTE_00000101 0 0 0 0 0 1 0 1,
//// ...
#define BYTE_11111111 1 1 1 1 1 1 1 1,
//
BINARY(00000101);

#endif


gcc gave me 247 lines of errors; see: http://pastebin.com/fzMhMLS0

The errors go away if I take out the BINARY(00000101) macro call.

That makes me feel better; if gcc can't handle it, what hope has any 
other compiler?

Or is this only supposed to be compilable under C++? (In which case why 
am I looking at it?)

Anyway, if someone wanted proof was to why such macro languages are evil...

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#101311

BartC <bc@freeuk.com> writes:

> On 08/01/2017 14:51, Jean-Marc Bourguet wrote:
>> BartC <bc@freeuk.com> writes:
>>
>>> But 'a(10)' can also be part of normal code, not inside a macro
>>> definition. Then the 'b' that is generated is in a different 'token-space'
>>> from the "(10)" in the rest of the file (and likely to still exist as
>>> characters not tokens). Rescanning an entirely source file is hopelessly
>>> inefficient just for the off-chance someone might be using such macros.
>>
>> You don't rescan the whole file. The rescan of the expansion is made once
>> more than you though and in its context.
>>
>> You don't know the steps people are ready to avoid using an external tool
>> while still using compact code. The technique is used in practice.  Look
>> for chaos pp and the boost preprocessor library for such kind of techniques
>> (the goal of those libraries is to encapsulate them in an more convenient
>> interface).
>
> I looked at chaos-pp. I didn't know the purpose of your comment and thought
> it was another pre-processor implementation to look at.

Again, it's a library providing a (relatively) high level interface for
intensive C preprocessor usage.  I've looked in the past at the boost
preprocessor library; I'm not really clear about the relationship with
chaos pp, they seem very similar and share an author, I've the vague
impression that chaos-pp both precedes and follows the boost component; at
least there is some documentation -- sadly more a reference than a user
guide -- which include examples.  See for instance
http://www.boost.org/doc/libs/1_63_0/libs/preprocessor/doc/ref/if.html

It is also the more limit-pushing use of the preprocessor that I'm aware
of.  (As limit cases, there is also the fact that some are using the C
preprocessor for other things than C)

> How did a no-nonsense language such as C end up with a dependency on a
> macro language so arcane that apparently few can fully understand it?

Macro language in general are arcane (see TeX for another example, even in
the case of Lisp dialects, macros, as hygienic they may be, are arcane).
That is partially but not uniquely due to the fact that it is another
paradigm than the one we are used to (it's funny to see how the same
techniques are used in TeX and with CPP).

Obviously, you also need to never forget that standardization is design by
comittee per nearly by definition (Ada succeeded in avoiding that for its
first two rounds -- I've not followed after -- but that's the only case I
know and it was quite special with DoD funding).  On those comittee
providers are overrepresented (and have a de facto veto when they agree)
and users which are present are more sophisticated than the common one.

In the case of C, there was the additional factor that it was a comittee
tasked with standardizing something which existed for a relatively long
time, for which the need of a standard appeared due to variation in
behavior between existing implementations (whose users and providers where
in the comittee).

Yours,

-- 
Jean-Marc

[toc] | [prev] | [next] | [standalone]

#101300

On Sunday, January 8, 2017 at 7:10:34 AM UTC-6, Bart wrote:
> But 'a(10)' can also be part of normal code, not inside a macro 
> definition. Then the 'b' that is generated is in a different 
> 'token-space' from the "(10)" in the rest of the file (and likely to 
> still exist as characters not tokens). Rescanning an entirely source 
> file is hopelessly inefficient just for the off-chance someone might be 
> using such macros.
> 
> That's going to mean some very untidy logic.

There are definite advantages to allowing rescanning.  Among other things,
it would allow for the use of preprocessing stages as pluggable modules.
For example, if one has a C89 compiler but wants to use some C99 features,
it would be possible to run the code through a C99 preprocessor and then
into the C89 compiler.  If the Standard were to leave unspecified the
question of how many times things might get expanded, provided that an
implementation limits recursion to a level that can handle constructs of
a certain complexity, that would make it practical to augment older
implementations with newer language features.

[toc] | [prev] | [next] | [standalone]

#101310

BartC <bc@freeuk.com> writes:
> On 08/01/2017 09:13, Jean-Marc Bourguet wrote:
[...]
>> Have you looked at the latest standard?
>
> I use one for 201x dated 2009.

It sounds like you're using a draft, not the actual standard.
There's no real problem with that, but you seem to be using an
older draft.

The latest publicly available draft of C11 is

    http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf

It was published shortly before the official C11 standard (which
costs money), and the differences between it and the official
standard are minor.  (I'm not aware of any differences relevant to
the preprocessor).  If you're interested in C11, there's no good
reason to use an earlier draft than that.

If you're implementing a C preprocessor, you should also be aware
that there's an error in both N1570 and C11, corrected later in
a Technical Corrigendum.  In section 6.10.8.1, the definitions of
__STDC_VERSION__ and __STDC_LIB_EXT1__ are shown with placeholder
values that were not updated before publication.  The correct
values are:

    __STDC_VERSION__  201112L
    __STDC_LIB_EXT1__ 201112L (if defined)

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#101317

BartC <bc@freeuk.com> wrote:
> I've knocked up a short script program to implement CPP macro expansion. 
> It works with many of the examples I've tried, but I have little 
> confidence that whatever I end up with will always work, as I'm guessing 
> at the algorithms needed.
> 
> The descriptions I've seen of the process seem to me ambiguous and 
> difficult to understand (maybe they're striving to be as cryptic as the 
> language they are describing!).

I looked at 2011 draft (n1570.pdf).  The description looks clear.
First, they say that different implementations (possibly giving
differing results are allowed).  So standard is unambigous:
the implementer of preprocessor has free hand for choosing
details.  OTOH C programmer must be careful so that program
uses only fully specified features.  Second, standard allows
naive very strigthforward implementation which should give
reasonable performance.  It also allow "one pass" implementation
which in principle may be faster.

> 
> Here is what I've been able to understand of the process; maybe someone 
> can tell me if it is on the right lines.
> 
> Example:
> 
>    #define XX 20
>    #define XXz 99
>    #define W W+W+XX+x
>    #define M(x,y)  x y #x y ## z M(a,b) W
> 
>    M(10,XX)
> 
> In the expansion of M(10,XX):
> 
> * The token sequence '10' is used for argument x
> 
> * The token sequence 'XX' is used for argument y.
> 
> * Start with the replacement token list as defined for M:
> 
>     x y #x y ## z M(a,b) W
> 
> * Replace x and y with their arguments, expanding, stringifying
>   or combining as needed:
> 
>    10 20 "10" XXz M(a,b) W
> 
>   Here, argument XX was separately macro expanded to 20 before replacing
>   the first y. The second y has ## so is not expanded.
> 
> * After this substitution, the token list is scanned for macro calls to
>   expand. M is a macro call, but is the same name as the macro
>   currently being expanded, so maps to itself (and the (a,b) following
>   are just tokens).
> 
> * That leaves XXz and W macros to expand, which now results in:
> 
>    10 20 "10" 99 M(a,b) W+W+XX+x
> 
> * The result is now scanned again looking for new macros to expand.
>   However, once any name (eg. M and W) has been expanded once, it is
>   never expanded again (in a subsequent pass). So only XX is changed
>   to result in:
> 
>    10 20 "10" 99 M(a,b) W+W+20+x

AFAICS this is your interetation, which is mostly correct but
in corner cases may produce wrong result and seem to be much
less efficient then what standard says:

: After all parameters in the replacement list have been substituted
: and # and ## processing has taken place, all placemarker preprocessing
: tokens are removed. The resulting preprocessing token sequence is
: then rescanned, along with all subsequent preprocessing tokens of
: the source file, for more macro names to replace.

So, after macro expanded parameters are substituted and # and ##
handling is done you simply prepend the result to source text
following macro call and start reading from the start of resulting
token list.  In other words there are _no_ repeated rescanning,
rescanning is done just once and during rescanning you may have
macro call which begins withing replacement list but ends in the
source text.

Note: most of the time repeated rescanning just wastes time.
But sometimes it gives wrong result (see examples at the end).

You can implement standard easily with stack of "source" buffers.
You read tokens from given buffer and at the end of buffer
you switch to previous buffer from the stack.  When you get
a macro call to do you first do # and ## processing and then
push current buffer on the stack and use result of # and ##
processing as the current source buffer.

Note that this process is kind of linear: number of times
that you look at a token is proportional to sum of sizes
of intermediate expansions.  There is no known way to
avoid generating intermediate expansions, so implementing
standard using stack of source buffers will give reasonable
performance.  You may try to optimize avoiding separate
pass for # and ## processing and substituting arguments
during rescan.  Saving on number of operations probably
is small, but you will save a lot on memory allocation:
only token lists that you need to remember are texts of
macro definition.

> 
>   (This agrees with what gcc -E gives, apart from an extra space before
>   +x)
> 
> * Presumably that rule also applies to parameter names, so the new x at
>   end stays as x, it's not changed to 10.

After parameters are substituted you are done with them: any new
token which appears during rescan is not treated as parameter even
if it is "the same" identifier.  In fact classic way to handle parameters
is to replace them by special markers during macro definition,
so that stored macro definition no longer contains parameter
names -- instead you have special marker tokens.  Thanks to
markers there is no need to explicitely produce repleacement
text with substituted arguments: you just take text of macro
defintion as you input buffer and when you see marker you
switch to buffer contaning argument (all using stack of input
buffer as before).

> * The re-scanning looking for new macros continues until there are no
>   macro expansions.

No, as explained.

> * The # and ## processing, for # and ## present in the original
>   replacement list, is only done once. What's not clear what it will do
>   with # and ## introduced as the result of macro expansion (if that is
>   even possible).

Rules say that in function like macro # must be followed by a
parameter and that it will be replaced by corresponding string.
So it will not survive expansion.  ## may be produced by
concatenation and will appear in the result of expansion.
Both may appear in result of object like macros.
Normally # and ## will probably appear in expansion of an
argument and will be used used to form a string.  Otherwise
they will survive (possibly leading to error in later stage).
Anyway, if one stage produces token that is illegal in next
stage, that is error in user program -- just report it at
proper stage.

> * (And, # used within a macro replacement list where there are no
>   parameters, is just a hash token. This seems to be implied in the
>   C standard.)
> 
> Summary:
> 
> * Macro arguments need only be expanded once (K&R2 A12.3 suggests this
>   could be done on each substitution, when a parameter occurs more than
>   once).
> 
> * Substitution of parameter names, and #/## processing, is done once.
>   Parameter names are not substituted again
> 
> * Repeated passes are now done to expand macro calls, until there are
>   no macro calls left to expand.

No.  There is single replacement pass.

> * A macro name A can be expanded several times in one pass, but then A
>   is not expanded again if re-encountered in subsequent passes.

Simple way to handle this is to use flags.  On each macro you have
'in_progess' flag which is set when you start rescan and reset
once you handled the whole replacement text.  If flag is set
then macro name is not replaced.  Moreover, you need flag
on tokens call it 'final'.  If name is not replaced due to
macro beeing in progress you set 'final' flag on the name.
If there is any rescan later (for example token appeared in
argument expansion) tokens with 'final' flag set are not
replaced.

Executive summary: rules in the standard are carefully written
to allow fast one pass implementation, but give freedom to
the implementer to choose other, less efficient way.

Few examples:

#define f a d d (x)
#define g a b (c)
#define h(x, y) x y
#define hash #
#define i j j
#define j i

// This form used to check for modern comment handling

// No rescanning when reading from source
a b (c)
a d (c)
// Expanded macro argument is rescanned
d(a b (c))
d(a d (c))
// expansion of f is rescanned only once
f
// placeholder protects first a from concatenation with the third
e(a,,a)
// only one token from second argument is concatented
e(a, c d, e)
// hash from argument expansion is preserved
h(hash, zz)
// Replacement stops when we get back i
i
// i from argument are no longer subject to replacement
d(i)

-- 
                              Waldek Hebisch

[toc] | [prev] | [next] | [standalone]

#101320

On 01/08/2017 05:54 PM, antispam@math.uni.wroc.pl wrote:
> BartC <bc@freeuk.com> wrote:
>> I've knocked up a short script program to implement CPP macro expansion.
>> It works with many of the examples I've tried, but I have little
>> confidence that whatever I end up with will always work, as I'm guessing
>> at the algorithms needed.
>>
>> The descriptions I've seen of the process seem to me ambiguous and
>> difficult to understand (maybe they're striving to be as cryptic as the
>> language they are describing!).
>
> I looked at 2011 draft (n1570.pdf).  The description looks clear.
> First, they say that different implementations (possibly giving
> differing results are allowed).  So standard is unambigous:
> the implementer of preprocessor has free hand for choosing
> details.

Could you please cite the exact text you're referring to? I'm not aware 
of anything that the C standard says about the preprocessor that matches 
that description. As with any other part of the C standard, when it 
imposes specific requirements for pre-processing, they must be met.

Producing differing results on different implementations is certainly 
possible, but only when the C standard gives permission to so so. When 
the behavior is unspecified, an implementation must choose one of the 
permitted possibilities. When the behavior is implementation-defined, 
the implementation must include documentation that specifies what choice 
the implementation has made. Only when the behavior is completely 
undefined does the standard give an implementation a completely "free hand".

According to Annex J, preprocessing involves the following portability 
issues:

The following are unspecified:
> The order in which # and ## operations are evaluated during macro substitution
> (6.10.3.2, 6.10.3.3).

The behavior is undefined in the following circumstances:
> — The token defined is generated during the expansion of a #if or #elif
> preprocessing directive, or the use of the defined unary operator does not match
> one of the two specified forms prior to macro replacement (6.10.1).
> — The #include preprocessing directive that results after expansion does not match
> one of the two header name forms (6.10.2).
> — The character sequence in an #include preprocessing directive does not start with a
> letter (6.10.2).
> — There are sequences of preprocessing tokens within the list of macro arguments that
> would otherwise act as preprocessing directives (6.10.3).
> — The result of the preprocessing operator # is not a valid character string literal
> (6.10.3.2).
> — The result of the preprocessing operator ## is not a valid preprocessing token
> (6.10.3.3).
> — The #line preprocessing directive that results after expansion does not match one of
> the two well-defined forms, or its digit sequence specifies zero or a number greater
> than 2147483647 (6.10.4).
> — A non-STDC #pragma preprocessing directive that is documented as causing
> translation failure or some other form of undefined behavior is encountered (6.10.6).
> — A #pragma STDC preprocessing directive does not match one of the well-defined
> forms (6.10.6).
> — The name of a predefined macro, or the identifier defined, is the subject of a
> #define or #undef preprocessing directive (6.10.8).

The following are implementation-defined:
> — The locations within #pragma directives where header name preprocessing tokens
> are recognized (6.4, 6.4.7).
> — How sequences in both forms of header names are mapped to headers or external
> source file names (6.4.7).
> — Whether the value of a character constant in a constant expression that controls
> conditional inclusion matches the value of the same character constant in the
> execution character set (6.10.1).
> — Whether the value of a single-character character constant in a constant expression
> that controls conditional inclusion may have a neg ative value (6.10.1).
> — The places that are searched for an included < > delimited header, and how the places
> are specified or the header is identified (6.10.2).
> — How the named source file is searched for in an included " " delimited header
> (6.10.2).
> — The method by which preprocessing tokens (possibly resulting from macro
> expansion) in a #include directive are combined into a header name (6.10.2).
> — The nesting limit for #include processing (6.10.2).
> — Whether the # operator inserts a \ character before the \ character that begins a
> universal character name in a character constant or string literal (6.10.3.2).
> — The behavior on each recognized non-STDC #pragma directive (6.10.6).
> — The definitions for _ _DATE_ _ and _ _TIME_ _ when respectively, the date and
> time of translation are not available (6.10.8.1).

That's a fair number of issues - but it's not enough to give an 
implementation a completely "free hand" - there's still lots of 
applicable restrictions on how preprocessing can be implemented.

[toc] | [prev] | [next] | [standalone]

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

Back to top | Article view | comp.lang.c

csiph-web