Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #10759 > unrolled thread

A very simple parser with scanf & C

Back to article view | Back to comp.lang.c

Contents

  A very simple parser with scanf & C pozz <pozzugno@gmail.com> - 2011-09-23 07:03 -0700
    Re: A very simple parser with scanf & C Nobody <nobody@nowhere.com> - 2011-09-23 18:28 +0100
    Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-23 18:29 +0100
      Re: A very simple parser with scanf & C pozz <pozzugno@gmail.com> - 2011-09-23 23:16 +0200
        Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-24 00:09 +0100
          Re: A very simple parser with scanf & C pozz <pozzugno@gmail.com> - 2011-09-24 09:32 +0200
    Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-23 21:24 +0100
      Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-23 22:00 +0100
        Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-24 00:04 +0100
          Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-24 00:34 +0100
            Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-24 01:00 +0100
              Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-24 02:42 +0100
                Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-24 12:13 +0100
                  Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-24 12:25 +0100
                    Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-24 15:05 +0100
                  Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-24 14:45 +0100
                    Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-24 17:45 +0100
                      Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-24 22:21 +0100
                        Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-25 00:52 +0100
                          Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-25 01:54 +0100
                            Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-25 02:51 +0100
                              Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-25 12:22 +0100
                          Re: A very simple parser with scanf & C Gene <gene.ressler@gmail.com> - 2011-09-25 01:35 -0700
                            Re: A very simple parser with scanf & C pete <pfiland@mindspring.com> - 2011-09-25 08:41 -0400
                              Re: A very simple parser with scanf & C Gene <gene.ressler@gmail.com> - 2011-09-25 11:23 -0700
                          Re: A very simple parser with scanf & C Phil Carmody <thefatphil_demunged@yahoo.co.uk> - 2011-09-25 19:41 +0300
                            Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-25 19:45 +0100
                              Re: A very simple parser with scanf & C Keith Thompson <kst-u@mib.org> - 2011-09-25 12:10 -0700
                                Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-25 20:43 +0100
                                  Re: A very simple parser with scanf & C James Kuyper <jameskuyper@verizon.net> - 2011-09-25 15:55 -0400
                                    Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-26 11:07 +0100
                                      Re: A very simple parser with scanf & C Richard Damon <news.x.richarddamon@xoxy.net> - 2011-09-26 11:40 -0400
                                        Re: A very simple parser with scanf & C Keith Thompson <kst-u@mib.org> - 2011-09-26 12:22 -0700
                                          Re: A very simple parser with scanf & C pacman@kosh.dhis.org (Alan Curry) - 2011-09-26 21:19 +0000
                                            Re: A very simple parser with scanf & C Keith Thompson <kst-u@mib.org> - 2011-09-26 16:11 -0700
                                        Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-26 23:26 +0100
                                      Re: A very simple parser with scanf & C Keith Thompson <kst-u@mib.org> - 2011-09-26 12:18 -0700
                                        Re: A very simple parser with scanf & C Phil Carmody <thefatphil_demunged@yahoo.co.uk> - 2011-09-27 02:27 +0300
                                          Re: A very simple parser with scanf & C Keith Thompson <kst-u@mib.org> - 2011-09-27 10:06 -0700
                                            Re: A very simple parser with scanf & C Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-09-27 21:10 +0100
                                      Re: A very simple parser with scanf & C Seebs <usenet-nospam@seebs.net> - 2011-09-26 22:59 +0000
                                      Re: A very simple parser with scanf & C Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-09-30 05:18 -0700
                                        Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-30 17:06 +0100
                                  Re: A very simple parser with scanf & C Richard Damon <news.x.richarddamon@xoxy.net> - 2011-09-25 18:05 -0400
                                  Re: A very simple parser with scanf & C Keith Thompson <kst-u@mib.org> - 2011-09-25 15:30 -0700
                                    Re: A very simple parser with scanf & C Tim Rentsch <txr@alumni.caltech.edu> - 2012-01-24 14:01 -0800
                                      Re: A very simple parser with scanf & C pete <pfiland@mindspring.com> - 2012-01-25 11:05 -0500
                                        Re: A very simple parser with scanf & C Tim Rentsch <txr@alumni.caltech.edu> - 2012-01-31 16:01 -0800
                                          Re: A very simple parser with scanf & C pete <pfiland@mindspring.com> - 2012-01-31 23:47 -0500
                                            Re: A very simple parser with scanf & C Tim Rentsch <txr@alumni.caltech.edu> - 2012-03-08 09:21 -0800
                                              Re: A very simple parser with scanf & C pete <pfiland@mindspring.com> - 2012-03-10 15:18 -0500
                                                Re: A very simple parser with scanf & C Tim Rentsch <txr@alumni.caltech.edu> - 2012-03-13 15:49 -0700
                                                  Re: A very simple parser with scanf & C pete <pfiland@mindspring.com> - 2012-03-14 18:56 -0500
                                                    Re: A very simple parser with scanf & C Tim Rentsch <txr@alumni.caltech.edu> - 2012-03-15 17:37 -0700
                                                      Re: A very simple parser with scanf & C pete <pfiland@mindspring.com> - 2012-03-17 00:24 -0500
                                                        Re: A very simple parser with scanf & C Tim Rentsch <txr@alumni.caltech.edu> - 2012-03-21 05:21 -0700
      Re: A very simple parser with scanf & C pozz <pozzugno@gmail.com> - 2011-09-23 23:23 +0200
        Re: A very simple parser with scanf & C "BartC" <bc@freeuk.com> - 2011-09-23 23:53 +0100
    Re: A very simple parser with scanf & C Rui Maciel <rui.maciel@gmail.com> - 2011-09-25 14:07 +0100

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#10897

"Ben Bacarisse" <ben.usenet@bsb.me.uk> wrote in message 
news:0.dc9f6dac9be14d60b0ee.20110925015426BST.871uv5sapp.fsf@bsb.me.uk...
> "BartC" <bc@freeuk.com> writes:

>> I posted the code below. I hope this does all the things I suggested!

> Upthread you said this approach was simpler and more efficient.  OK, the
> "more efficient" probably referred to using hashing to check keywords,
> but then there would be a lot more then 50 lines.  Is it really simpler?
> More extensible, yes, but not, I'd say, simpler.

Simplicity is not just about line-count. It's also knowing exactly what is 
being compared and being confident about the results.

Also, the actual syntax parsing *is* simpler, once the lexical stuff has 
been encapsulated.

The efficiency will come from opportunities to do more streamlined lookups. 
As it is, with an extra step to tokenise input before doing anything with it 
(and using all those isxxx() functions), it's about half the speed of the 
OP's original.

(Tokenising can also leave the token strings in-place, especially if the 
source string can be modified, but it's bit more awkward.)

-- 
Bartc 

[toc] | [prev] | [next] | [standalone]

#10911

"BartC" <bc@freeuk.com> writes:

> "Ben Bacarisse" <ben.usenet@bsb.me.uk> wrote in message
> news:0.dc9f6dac9be14d60b0ee.20110925015426BST.871uv5sapp.fsf@bsb.me.uk...
>> "BartC" <bc@freeuk.com> writes:
>
>>> I posted the code below. I hope this does all the things I suggested!
>
>> Upthread you said this approach was simpler and more efficient.  OK, the
>> "more efficient" probably referred to using hashing to check keywords,
>> but then there would be a lot more then 50 lines.  Is it really simpler?
>> More extensible, yes, but not, I'd say, simpler.
>
> Simplicity is not just about line-count. It's also knowing exactly
> what is being compared and being confident about the results.

That's true.  Part of the trouble was that I though the original
approach was clear enough to have confidence in the results.  Given
that, the actual amount of code needed does then measure complexity.

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#10907

On Sep 25, 1:52 am, "BartC" <b...@freeuk.com> wrote:
> "Ben Bacarisse" <ben.use...@bsb.me.uk> wrote in message
> >  How would you write the OP matcher?
>
> I posted the code below. I hope this does all the things I suggested!
>
> There is a function lex() which extracts the next token from the input in a
> general manner (and reports, to some extent, what type of token it is).
>
> And there is a function parsetest(), which takes a given string, and checks
> it corresponds to the OP's syntax spec.
>
> Here, the details of the character-level handling, and the syntax parsing,
> are largely separate.
>
> (I haven't bothered with a formal table lookup; lex() uses file globals for
> simplicity; and overflow of the lextoken[] array isn't checked. Some checks,
> eg. lex()!='A', could probably be dispensed with. And there's supposed to be
> something funny about using atoi(), but I can't remember what..)
>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <ctype.h>
>
> /* globals set by lex() */
> int lextype;         /* One of 'A' 'N' 'P' 'L' 'E' '?' means Alpha, Number,
>                         Punctuation, EOL, End of string, Error */
> int lexleadsp;       /* Set to 1 when leading whitespace seen (requirement
> of OP) */
> char lextoken[1000]; /* Contains token for 'A', 'N' and 'P' lexical types */
> char *lexptr;        /* Points to next character of input */
>
> int lex(void) {
>   char c;
>   int i;
>
>   lextoken[0]=0;
>   lexleadsp=0;
>
>   while (*lexptr!='\n' && isspace(*lexptr)) {
>     lexleadsp=1;
>     ++lexptr;
>   }
>
>   c=*lexptr;
>   i=0;
>
>   if (c==0)
>     lextype='E';
>   else if (isalpha(c)) {
>     do
>       lextoken[i++]=toupper(*lexptr++);
>     while (isalpha(*lexptr));
>     lextype='A';
>   }
>   else if (isdigit(c)) {
>     do
>       lextoken[i++]=toupper(*lexptr++);
>     while (isdigit(*lexptr));
>     lextype='N';
>   }
>   else if (c=='\n') {
>     lextype='L';
>     ++lexptr;
>   }
>   else if (ispunct(c)) {
>     lextoken[i++]=*lexptr++;
>     lextype='P';
>   }
>   else {
>     lextype='?';
>     ++lexptr;
>   }
>
>   lextoken[i]=0;
>   return lextype;
>
> }
>
> /* Make use of lex() function to test an input string in this syntax:
>   OPEN|CLOSE|LOCK DOOR [+]n where n must be 1 to 10, and without
>   leading or trailing white space.
>   Returns 1 (passed) or 0 (failed)
> */
> int parsetest(char *input) {
>   int n,t;
>
>   lexptr=input;
>
>   if (lex()!='A' || lexleadsp) return 0;
>   if (strcmp(lextoken,"OPEN")!=0 &&
>       strcmp(lextoken,"CLOSE")!=0 &&
>       strcmp(lextoken,"LOCK")!=0)
>     return 0;
>
>   if (lex()!='A' || strcmp(lextoken,"DOOR")!=0) return 0;
>
>   t=lex();
>   if (t=='P' && lextoken[0]=='+') t=lex();
>
>   if (t!='N') return 0;
>   n=atoi(lextoken);
>   if (n<1 || n>10) return 0;
>
>   switch (lex()) {
>   case 'E': case 'L':
>     if (lexleadsp) return 0;
>     break;
>   default:
>    return 0;
>   }
>  return 1;
>
> }
>
> int main(void) {
>   char* teststr = "Lock DOOR 10";
>   int status;
>
>   printf("Testing: \"%s\"\n",teststr);
>
>   status=parsetest(teststr);
>
>   printf("Result: %s\n",status?"Passed":"Failed");
>
> }

If there is a realistic chance the language will expand in the future
to a branching syntax, then tokenizing now is probably a good idea.
But if the language is such that the parser will always know what to
expect based on what it's already read (essentially the grammar is
LR(0)), then a separate tokenizing step is overkill.  It adds error
opportunities and slows things down.

What the OP did is fine, but I'll bet his tests didn't work the first
time.  Character-by-character approaches like this are fiddly.

A middle ground is to develop a set of prefix matchers that can be
reused.  I've tried many different APIs and idioms and settled on the
following over the years.  With this approach the code of the parser
can look just like the spec.  With a good inlining compiler, the
result will be as good or better than OP's code for performance.

typedef enum token_e {
  OPEN_TOKEN,
  CLOSE_TOKEN,
  LOCK_TOKEN,
  DOOR_TOKEN,
} TOKEN;

/*
 * Return true iff the given token is a proper directive.
 */
int is_directive(TOKEN token)
{
  return OPEN_TOKEN <= token && token <= LOCK_TOKEN;
}

/* Must be in alpha order, lower case */
KEYWORD_PAIR keywords[] = {
  { "close", CLOSE_TOKEN },
  { "door", DOOR_TOKEN },  // Could handle door specially....
  { "lock", LOCK_TOKEN },
  { "open", OPEN_TOKEN },
};

/*
 * Return true iff the given integer is a proper door number.
 */
int door_number_ok(unsigned n)
{
  return n <= 10;
}

/*
 * Parse a string with the following format:
 * - a directive token, case insensitive
 * - one or more whitespaces (spaces, tabs, newlines...)
 * - the word "door" (without quotes), case insensitive
 * - an optional number of whitespaces, even zero
 * - a door number in the range 0..10
 * - no other characters after the number
 * Return the directive as a token and the door number.
 * Function return is error code or 0 for success.
 */
int parse(char *s, TOKEN *directive, unsigned *door_number)
{
  int index;

  // Probably overkill to demand no whitespace.
  if (match_whitespace(s, &s))
    return -1;

  if (!match_keyword(s, &s,
                     keywords, TABLE_SIZE(keywords),
                     keyword_cmp_ignoring_case,
                     &index))
    return -2; // Missing token.

  *directive = (TOKEN)keywords[index].token;

  if (!is_directive(*directive))
    return -3; // Token not a directive.

  if (!match_token(s, &s,
                   keywords, TABLE_SIZE(keywords),
                   keyword_cmp_ignoring_case,
                   DOOR_TOKEN))
    return -4; // Door keyword missing or misspelled

  if (!match_unsigned(s, &s, door_number))
    return -5; // Door number missing

  if (!door_number_ok(*door_number))
    return -6; // Door number out of range

  if (*s)
    return -7; // Junk at end check.

  return 0; // 0 is more conventional for success
}

... and the reusable code looks essentially like this.  I've de-
parameterized some of my actual code to shorten it.

/*
 * Set *end one char past whitespace prefix of s.
 * Return true iff any characters matched.
 */
int match_whitespace(char *s, char **end)
{
  char *p = s;
  while (isspace(*p))
    ++p;
  *end = p;
  return p > s;
}

/*
 * Set *end one char past alpha prefix of s.
 * Return true iff any characters matched.
 */
int match_alpha(char *s, char **end)
{
  char *p = s;
  while (isalpha(*p))
    ++p;
  *end = p;
  return p > s;
}

/*
 * Skip whitespace and then scan as many decimal digits
 * as possible. Convert to an unsigned value with no
 * overflow check. Return true iff any digits were
 * matched.  Set *end one past the scanned chars.
 * FIXME: Add overflow check.
 */
int match_unsigned(char *s, char **end, unsigned *val_rtn)
{
  char *p;
  unsigned val;

  match_whitespace(s, &s);
  for (val = 0, p = s; isdigit(*p); p++)
    val = 10 * val + (*p - '0');
  *end = p;
  *val_rtn = val;
  return p > s;
}

/*
 * Convert a character to lower case.
 */
char lowercase(char c)
{
  return ('A' <= c && c <= 'Z') ? c - 'A' + 'a' : c;
}

/*
 * Key comparison function for keyword lookups.
 */
typedef int (*KEYWORD_CMP)(char *s, unsigned n, char *kw);

/*
 * Case insensitive keyword comparision.  This compares
 * a prefix of s of up to n characters with kw.
 */
int keyword_cmp_ignoring_case(char *s, size_t n, char *kw)
{
  while (n > 0 && *s && *kw && lowercase(*s) == *kw) {
    --n; ++s; ++kw;
  }
  return (n == 0) ? -*kw : lowercase(*s) - *kw;
}

/*
 * Keyword entries and tables. An empty key string in
 * a table produces undefined results.
 */
typedef struct keyword_pair_s {
  char *key;
  int token;
} KEYWORD_PAIR, KEYWORD_TABLE[];

#define TABLE_SIZE(A) (sizeof A / sizeof A[0])

/*
 * Match a maximal prefix of alphabetic characters of
 * string s to keywords in a table.  Binary search is
 * used, so the keys must be sorted. The keyword table
 * index is returned (not the token value).
 * On success return true and set *end one past the keyword.
 * If none found, set *end one past any whitespace and return 0
 * with index unchanged.
 */
int match_keyword(char *s, char **end,
                  KEYWORD_TABLE tbl, unsigned tbl_size,
                  KEYWORD_CMP cmp,
                  int *index)
{
  int diff, lo, hi, mid;

  match_whitespace(s, &s);
  if (match_alpha(s, end)) {
    lo = 0;
    hi = tbl_size - 1;
    while (lo <= hi) {
      mid = (lo + hi) / 2;
      diff = cmp(s, *end - s, tbl[mid].key);
      if (diff < 0)
        hi = mid - 1;
      else if (diff > 0)
        lo = mid + 1;
      else {
        *index = mid;
        return 1;
      }
    }
  }
  *end = s;
  return 0;
}

/*
 * Match a maximal prefix of string s to keywords in a table.
 * If found, verify its token matches the given value.
 * On success return true and set *end to character past
 * scanned keyword. On any failure, set *end one past any
 * leading whitespace.
 */
int match_token(char *s, char **end,
                KEYWORD_TABLE tbl, unsigned tbl_size,
                KEYWORD_CMP cmp,
                int token)
{
  int index;

  match_whitespace(s, &s);
  if (!match_keyword(s, end, tbl, tbl_size, cmp, &index) ||
      tbl[index].token != token)
    return 0;
  return 1;
}

[toc] | [prev] | [next] | [standalone]

#10915

Gene wrote:
 
> /*
>  * Convert a character to lower case.
>  */
> char lowercase(char c)
> {
>   return ('A' <= c && c <= 'Z') ? c - 'A' + 'a' : c;
> }

That's not 100% portable.
At least it won't work with EBCDIC.
http://en.wikipedia.org/wiki/Extended_Binary_Coded_Decimal_Interchange_Code

#include <limits.h>
#include <string.h>

#define UPPER   "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
#define LOWER   "abcdefghijklmnopqrstuvwxyz"

int to_lower(int c) 
{
    const char *lower;
    const char *const upper = UPPER;

    lower = CHAR_MAX >= c && c > '\0' ? strchr(upper, c) : NULL;
    return lower != NULL ? LOWER[lower - upper] : c;
}


-- 
pete

[toc] | [prev] | [next] | [standalone]

#10926

On Sep 25, 2:41 pm, pete <pfil...@mindspring.com> wrote:
> Gene wrote:
> > /*
> >  * Convert a character to lower case.
> >  */
> > char lowercase(char c)
> > {
> >   return ('A' <= c && c <= 'Z') ? c - 'A' + 'a' : c;
> > }
>
> That's not 100% portable.
> At least it won't work with EBCDIC.http://en.wikipedia.org/wiki/Extended_Binary_Coded_Decimal_Interchang...
>
> #include <limits.h>
> #include <string.h>
>
> #define UPPER   "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
> #define LOWER   "abcdefghijklmnopqrstuvwxyz"
>
> int to_lower(int c)
> {
>     const char *lower;
>     const char *const upper = UPPER;
>
>     lower = CHAR_MAX >= c && c > '\0' ? strchr(upper, c) : NULL;
>     return lower != NULL ? LOWER[lower - upper] : c;
>
> }

Got that. Cut my teeth on an IBM 1130.  In production code I'd rather
do something like

#if 'Z' - 'A' + 1 != 26 || 'z' - 'a' + 1 != 26
#error tolower() assumption violated
#endif

than replace 2 instructions with a few dozen executed once per input
character.  Could also use ctype.h's tolower, but probably give up
inlining.  And I've run into a couple of ctype.h's without it, though
not for a while.

[toc] | [prev] | [next] | [standalone]

#10925

"BartC" <bc@freeuk.com> writes:
> char *lexptr;        /* Points to next character of input */
...
>     while (isalpha(*lexptr));

Bang!

Phil
-- 
"Religion is what keeps the poor from murdering the rich."
-- Napoleon

[toc] | [prev] | [next] | [standalone]

#10927

"Phil Carmody" <thefatphil_demunged@yahoo.co.uk> wrote in message 
news:87d3eoy3p8.fsf@bazspaz.fatphil.org...
> "BartC" <bc@freeuk.com> writes:
>> char *lexptr;        /* Points to next character of input */
> ...
>>     while (isalpha(*lexptr));
>
> Bang!

What's the problem?

-- 
Bartc 

[toc] | [prev] | [next] | [standalone]

#10928

"BartC" <bc@freeuk.com> writes:
> "Phil Carmody" <thefatphil_demunged@yahoo.co.uk> wrote in message 
> news:87d3eoy3p8.fsf@bazspaz.fatphil.org...
>> "BartC" <bc@freeuk.com> writes:
>>> char *lexptr;        /* Points to next character of input */
>> ...
>>>     while (isalpha(*lexptr));
>>
>> Bang!
>
> What's the problem?

isalpha() expects an int argument, which either is representable
as an unsigned char or is equal to EOF.  If plain char is signed,
and *lexptr happens to be negative (and not equal to EOF), the
behavior is undefined.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#10930

"Keith Thompson" <kst-u@mib.org> wrote in message 
news:ln7h4w30c6.fsf@nuthaus.mib.org...
> "BartC" <bc@freeuk.com> writes:
>> "Phil Carmody" <thefatphil_demunged@yahoo.co.uk> wrote in message
>> news:87d3eoy3p8.fsf@bazspaz.fatphil.org...
>>> "BartC" <bc@freeuk.com> writes:
>>>> char *lexptr;        /* Points to next character of input */
>>> ...
>>>>     while (isalpha(*lexptr));
>>>
>>> Bang!
>>
>> What's the problem?
>
> isalpha() expects an int argument, which either is representable
> as an unsigned char or is equal to EOF.  If plain char is signed,
> and *lexptr happens to be negative (and not equal to EOF), the
> behavior is undefined.

I still don't get it.

Are you saying that while an implementation's char type can be signed or 
unsigned, it's isalpha() function assumes unsigned?

Apart from being ludicrous, how are you supposed to use it then? With an 
(unsigned char) cast? Suppose the char value really is EOF?

(Quite a few threads in the past about the pointlessness of having signed 
char at all, now I know it is totally crazy, when one part of the same 
implementation can't agree with another about whether chars are supposed to 
be signed or signed!)

Anyway I only used isalpha() because with my usual >=A and <=Z tests, 
someone would have called me out on that too!

(BTW if I was implementing isalpha(), after checking for EOF, any negative 
char values would just be converted to unsigned ones; end of story.)

-- 
Bartc 

[toc] | [prev] | [next] | [standalone]

#10931

On 09/25/2011 03:43 PM, BartC wrote:
> 
> 
> "Keith Thompson" <kst-u@mib.org> wrote in message 
> news:ln7h4w30c6.fsf@nuthaus.mib.org...
...
>> isalpha() expects an int argument, which either is representable
>> as an unsigned char or is equal to EOF.  If plain char is signed,
>> and *lexptr happens to be negative (and not equal to EOF), the
>> behavior is undefined.
> 
> I still don't get it.
> 
> Are you saying that while an implementation's char type can be signed or 
> unsigned, it's isalpha() function assumes unsigned?

7.4p1 describes the character classification functions: "In all cases
the argument is an int, the value of which shall be representable as an
unsigned char or shall equal the value of the macro EOF. If the argument
has any other value, the behavior is undefined."

> Apart from being ludicrous, how are you supposed to use it then? With an 
> (unsigned char) cast? Suppose the char value really is EOF?

That's covered - see above.
Keep in mind that the standard specifies that  fgetc() "obtains that
character as an unsigned char converted to an int", unless it returns
EOF to indicate an error. Those specifications make fgetc() and the
character classification macros good matches for each other.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#10942

"James Kuyper" <jameskuyper@verizon.net> wrote in message
news:j5o0vd$jqr$1@dont-email.me...
> On 09/25/2011 03:43 PM, BartC wrote:

>> Are you saying that while an implementation's char type can be signed or
>> unsigned, it's isalpha() function assumes unsigned?

> Keep in mind that the standard specifies that  fgetc() "obtains that
> character as an unsigned char converted to an int", unless it returns
> EOF to indicate an error. Those specifications make fgetc() and the
> character classification macros good matches for each other.

OK, so everywhere you look, it's clear that having unsigned char types is a
good idea.

So, even if signed chars are not eliminated completely, why are they so
popular, given that it is (presumably) the implementation's free choice to
choose between signed and unsigned? (At least, the three compilers I have,
all have char as signed.)

Then signed chars would only ever be used by someone needing a small signed 
integer.

-- 
Bartc 

[toc] | [prev] | [next] | [standalone]

#10953

On 9/26/11 6:07 AM, BartC wrote:
> "James Kuyper" <jameskuyper@verizon.net> wrote in message
> news:j5o0vd$jqr$1@dont-email.me...
>> On 09/25/2011 03:43 PM, BartC wrote:
>
>>> Are you saying that while an implementation's char type can be signed or
>>> unsigned, it's isalpha() function assumes unsigned?
>
>> Keep in mind that the standard specifies that fgetc() "obtains that
>> character as an unsigned char converted to an int", unless it returns
>> EOF to indicate an error. Those specifications make fgetc() and the
>> character classification macros good matches for each other.
>
> OK, so everywhere you look, it's clear that having unsigned char types is a
> good idea.
>
> So, even if signed chars are not eliminated completely, why are they so
> popular, given that it is (presumably) the implementation's free choice to
> choose between signed and unsigned? (At least, the three compilers I have,
> all have char as signed.)
>
> Then signed chars would only ever be used by someone needing a small
> signed integer.
>

My guess is that there must be a body of code that expects (incorrectly) 
that "plain" char will be signed, and implementations are choosing chars 
to be signed to meet customers need.

Those who know better, add the cast to unsigned, those that don't 
probably more expect it to be signed.

Note that the isxxx functions work properly with signed characters as 
long as the character set uses just base ASCII. Where it starts to fail 
is when you get into extended character sets, and you need to do more 
than just make char unsigned to handle those, so other parts of the 
program will break as well, and the "undefined behavior" that comes out 
of putting negative values into them is normally "just" wrong answers, 
not "crashes", and if you don't know enough to set the locale, the 
answers are likely unspecified anyway. If you know enough to set the 
locale, you probably know enough to add the cast, or to change the 
setting on the compiler to make char unsigned for your program.

[toc] | [prev] | [next] | [standalone]

#10960

Richard Damon <news.x.richarddamon@xoxy.net> writes:
> On 9/26/11 6:07 AM, BartC wrote:
>> "James Kuyper" <jameskuyper@verizon.net> wrote in message
>> news:j5o0vd$jqr$1@dont-email.me...
>>> On 09/25/2011 03:43 PM, BartC wrote:
>>
>>>> Are you saying that while an implementation's char type can be signed or
>>>> unsigned, it's isalpha() function assumes unsigned?
>>
>>> Keep in mind that the standard specifies that fgetc() "obtains that
>>> character as an unsigned char converted to an int", unless it returns
>>> EOF to indicate an error. Those specifications make fgetc() and the
>>> character classification macros good matches for each other.
>>
>> OK, so everywhere you look, it's clear that having unsigned char types is a
>> good idea.
>>
>> So, even if signed chars are not eliminated completely, why are they so
>> popular, given that it is (presumably) the implementation's free choice to
>> choose between signed and unsigned? (At least, the three compilers I have,
>> all have char as signed.)
>>
>> Then signed chars would only ever be used by someone needing a small
>> signed integer.
>>
>
> My guess is that there must be a body of code that expects (incorrectly) 
> that "plain" char will be signed, and implementations are choosing chars 
> to be signed to meet customers need.
>
> Those who know better, add the cast to unsigned, those that don't 
> probably more expect it to be signed.

That's a good guess -- but I don't think it's correct.  I've worked on
systems that weren't particularly exotic that had plain char unsigned:
IBM AIX (a Unix system) on PowerPC, and SGI IRIX (another Unix system)
on MIPS.  I'm not aware of any code that had problems because of this.

As I said in my other followup, I think it's more a matter of code
efficiency.

[...]

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#10966

In article <lnty7z1545.fsf@nuthaus.mib.org>,
Keith Thompson  <kst-u@mib.org> wrote:
>Richard Damon <news.x.richarddamon@xoxy.net> writes:
>> My guess is that there must be a body of code that expects (incorrectly) 
>> that "plain" char will be signed, and implementations are choosing chars 
>> to be signed to meet customers need.
 
>That's a good guess -- but I don't think it's correct.  I've worked on
>systems that weren't particularly exotic that had plain char unsigned:
>IBM AIX (a Unix system) on PowerPC, and SGI IRIX (another Unix system)
>on MIPS.  I'm not aware of any code that had problems because of this.

You were lucky. I've seen those problems. Two that I remember:

  char c;
  while((c=getopt(...))!=-1)
    ...

(The bug report on that one is http://bugs.debian.org/327327 and it was
fixed pretty quickly after I reported it.)

The other was shortly after the DOOM source code release I tried to set
up a multiplayer game where one player was on a PowerPC and the others
were i386's. The game aborted as soon as anybody moved, because player
movements were being sent over the network as small signed integers,
stored in a plain char.

This was a game that was praised for having all of its core logic in
portable C with the system-dependent parts neatly isolated, and its
portable core assumed that plain char was signed. On the bright side,
the use of chars rather than shorts prevented any possible endianness
mismatches.

Oh, and a third example: the Jacob Navia "containers" library had a
similar kind of failure when I first ran it on a plain-char-is-unsigned
machine. I posted about that and he fixed it, in a thread here a few
months ago.

If any compiler changes from "plain char is signed" to "plain char is
unsigned", its users will experience bugs of this type. Some of them
will be hard to track down, and the users will grumble about the
compiler vendor breaking code that's always worked before.

That's enough to keep signed chars around forever.

-- 
Alan Curry

[toc] | [prev] | [next] | [standalone]

#10977

pacman@kosh.dhis.org (Alan Curry) writes:
> In article <lnty7z1545.fsf@nuthaus.mib.org>,
> Keith Thompson  <kst-u@mib.org> wrote:
>>Richard Damon <news.x.richarddamon@xoxy.net> writes:
>>> My guess is that there must be a body of code that expects (incorrectly) 
>>> that "plain" char will be signed, and implementations are choosing chars 
>>> to be signed to meet customers need.
>  
>>That's a good guess -- but I don't think it's correct.  I've worked on
>>systems that weren't particularly exotic that had plain char unsigned:
>>IBM AIX (a Unix system) on PowerPC, and SGI IRIX (another Unix system)
>>on MIPS.  I'm not aware of any code that had problems because of this.
>
> You were lucky. I've seen those problems. Two that I remember:
>
>   char c;
>   while((c=getopt(...))!=-1)
>     ...
>
> (The bug report on that one is http://bugs.debian.org/327327 and it was
> fixed pretty quickly after I reported it.)

That doesn't just assume that plain char is signed (though it might
happen to work if it is); it assumes, incorrectly, that getopt() returns
a char (in fact it returns an int).  Even if plain char is unsigned, it
will misbehave if getopt() returns an actual 0xff character.

[...]

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#10968

Richard Damon <news.x.richarddamon@xoxy.net> writes:

> On 9/26/11 6:07 AM, BartC wrote:
>> "James Kuyper" <jameskuyper@verizon.net> wrote in message
>> news:j5o0vd$jqr$1@dont-email.me...
>>> On 09/25/2011 03:43 PM, BartC wrote:
>>
>>>> Are you saying that while an implementation's char type can be signed or
>>>> unsigned, it's isalpha() function assumes unsigned?
>>
>>> Keep in mind that the standard specifies that fgetc() "obtains that
>>> character as an unsigned char converted to an int", unless it returns
>>> EOF to indicate an error. Those specifications make fgetc() and the
>>> character classification macros good matches for each other.
>>
>> OK, so everywhere you look, it's clear that having unsigned char types is a
>> good idea.
>>
>> So, even if signed chars are not eliminated completely, why are they so
>> popular, given that it is (presumably) the implementation's free choice to
>> choose between signed and unsigned? (At least, the three compilers I have,
>> all have char as signed.)
>>
>> Then signed chars would only ever be used by someone needing a small
>> signed integer.
>>
>
> My guess is that there must be a body of code that expects
> (incorrectly) that "plain" char will be signed, and implementations
> are choosing chars to be signed to meet customers need.

Just a historical note.  Of the four machines implementing C that are
documented in K&R, only one has signed chars: the PDP-11.  It was
perhaps the most influential, so that might be part of the reason for
the common assumption that chars are signed.

K&R C also had no signed keyword, so if you had unsigned char, you could
not get a signed one.  Also, the "unsigned" adjective was not allowed to
qualify char[1], so of you had signed chars you could not get unsigned
ones.  This is significant, because the cast to unsigned char is
required in modern C.  I don't know the sequence, but there may have
been a time when the isxxxx functions existed with no way to fix the
problem.

[1] I'd also forgotten that unsigned was only allowed to used with int.
unsigned long and unsigned short were not permitted.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#10959

"BartC" <bc@freeuk.com> writes:
> "James Kuyper" <jameskuyper@verizon.net> wrote in message
> news:j5o0vd$jqr$1@dont-email.me...
>> On 09/25/2011 03:43 PM, BartC wrote:
>
>>> Are you saying that while an implementation's char type can be signed or
>>> unsigned, it's isalpha() function assumes unsigned?
>
>> Keep in mind that the standard specifies that  fgetc() "obtains that
>> character as an unsigned char converted to an int", unless it returns
>> EOF to indicate an error. Those specifications make fgetc() and the
>> character classification macros good matches for each other.
>
> OK, so everywhere you look, it's clear that having unsigned char types is a
> good idea.
>
> So, even if signed chars are not eliminated completely, why are they so
> popular, given that it is (presumably) the implementation's free choice to
> choose between signed and unsigned? (At least, the three compilers I have,
> all have char as signed.)
>
> Then signed chars would only ever be used by someone needing a small signed 
> integer.

Expressions of type char are promoted to int in most contexts.  If plain
char is signed, this promotion can be done with a sign extension, which
is typically quite cheap.  If plain char is unsigned, it might be more
expensive.  (I don't have more specific information, but it's what I was
told when I asked a similar question some time ago.)

I agree that making plain char unsigned would be conceptually simpler.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#10978

Keith Thompson <kst-u@mib.org> writes:
> "BartC" <bc@freeuk.com> writes:
> > "James Kuyper" <jameskuyper@verizon.net> wrote in message
> > news:j5o0vd$jqr$1@dont-email.me...
> >> On 09/25/2011 03:43 PM, BartC wrote:
> >
> >>> Are you saying that while an implementation's char type can be signed or
> >>> unsigned, it's isalpha() function assumes unsigned?
> >
> >> Keep in mind that the standard specifies that  fgetc() "obtains that
> >> character as an unsigned char converted to an int", unless it returns
> >> EOF to indicate an error. Those specifications make fgetc() and the
> >> character classification macros good matches for each other.
> >
> > OK, so everywhere you look, it's clear that having unsigned char types is a
> > good idea.
> >
> > So, even if signed chars are not eliminated completely, why are they so
> > popular, given that it is (presumably) the implementation's free choice to
> > choose between signed and unsigned? (At least, the three compilers I have,
> > all have char as signed.)
> >
> > Then signed chars would only ever be used by someone needing a small signed 
> > integer.
> 
> Expressions of type char are promoted to int in most contexts.  If plain
> char is signed, this promotion can be done with a sign extension, which
> is typically quite cheap.  If plain char is unsigned, it might be more
> expensive.  (I don't have more specific information, but it's what I was
> told when I asked a similar question some time ago.)

I suspect those architectures may well live only in Bernsteinian la-la
land.  Quite why the other 100% of the world should care one flying
ferret about expensiveness of operations on such architectures, I'm
not entirely sure. (And I say that as someone who has not used a
mainstream architecture as his primary machine for over a decade. (7
happy years on alpha, 4 happy years on POWER. And most development in
that time done for ARM.))

> I agree that making plain char unsigned would be conceptually simpler.

I'm pretty sure you disagreed with me when I made that suggestion
a while back.

Phil
-- 
"Religion is what keeps the poor from murdering the rich."
-- Napoleon

[toc] | [prev] | [next] | [standalone]

#11016

Phil Carmody <thefatphil_demunged@yahoo.co.uk> writes:
> Keith Thompson <kst-u@mib.org> writes:
[...]
>> Expressions of type char are promoted to int in most contexts.  If plain
>> char is signed, this promotion can be done with a sign extension, which
>> is typically quite cheap.  If plain char is unsigned, it might be more
>> expensive.  (I don't have more specific information, but it's what I was
>> told when I asked a similar question some time ago.)
>
> I suspect those architectures may well live only in Bernsteinian la-la
> land.  Quite why the other 100% of the world should care one flying
> ferret about expensiveness of operations on such architectures, I'm
> not entirely sure. (And I say that as someone who has not used a
> mainstream architecture as his primary machine for over a decade. (7
> happy years on alpha, 4 happy years on POWER. And most development in
> that time done for ARM.))

Ok.  If anyone else has solid information on this, I'd be interested in
hearing about it.

>> I agree that making plain char unsigned would be conceptually simpler.
>
> I'm pretty sure you disagreed with me when I made that suggestion
> a while back.

Hmm.  I may have disagreed that requiring plain char to be unsigned
would be a good idea for the performance reasons stated above.
I don't recall arguing that it wouldn't be *conceptually* simpler.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#11023

Keith Thompson <kst-u@mib.org> writes:

> Phil Carmody <thefatphil_demunged@yahoo.co.uk> writes:
>> Keith Thompson <kst-u@mib.org> writes:
> [...]
>>> Expressions of type char are promoted to int in most contexts.  If plain
>>> char is signed, this promotion can be done with a sign extension, which
>>> is typically quite cheap.  If plain char is unsigned, it might be more
>>> expensive.  (I don't have more specific information, but it's what I was
>>> told when I asked a similar question some time ago.)
>>
>> I suspect those architectures may well live only in Bernsteinian la-la
>> land.  Quite why the other 100% of the world should care one flying
>> ferret about expensiveness of operations on such architectures, I'm
>> not entirely sure. (And I say that as someone who has not used a
>> mainstream architecture as his primary machine for over a decade. (7
>> happy years on alpha, 4 happy years on POWER. And most development in
>> that time done for ARM.))
>
> Ok.  If anyone else has solid information on this, I'd be interested in
> hearing about it.

K&R say this about the signed/unsigned char issue: (sorry, there will
probably be typos since this is not cut and paste)

  "There is one subtle point about the conversion of characters to
  integers.  The language does not specify whether variables of type
  char are signed or unsigned quantities.  When a char is converted to
  int, can it ever produce a /negative/ integer?  Unfortunately, this
  varies from machine to machine, reflecting differences in
  architecture.  On some machine (the PDP-11, for instance), a char
  whose leftmost bit is 1 will be converted to a negative integer ("sign
  extension").  On others, a char is promoted to an int by adding zeros
  at the left end, and is thus always positive."
                                                      K&R 1st ed.

They don't explicitly say that not sign extending a char on the PDP-11
was considered too expensive but it seems to be implied by the "that's
what machines do so that's what C does" attitude.  The PDP-11 had only
the one MOVB instruction, and that did sign extension.

I wonder how many later designed had this restriction.  Not many, I'd
wager, but I look forward to hearing about them.

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.c

csiph-web