Re: when is typecasting (unsigned char) to (char) dangerous?

From	Keith Thompson <kst-u@mib.org>
Newsgroups	comp.lang.c
Subject	Re: when is typecasting (unsigned char) to (char) dangerous?
Date	2011-11-17 13:16 -0800
Organization	None to speak of
Message-ID	<lnehx6xxev.fsf@nuthaus.mib.org> (permalink)
References	<j9ugaj$q30$4@speranza.aioe.org> <ja3m5c$coa$1@oravannahka.helsinki.fi>

Show all headers | View raw

Lauri Alanko <la@iki.fi> writes:
> I actually stumbled into this problem just now.
>
> I need to represent UTF-8 strings in C, but although the common wisdom
> is that UTF-8 is nicely compatible with legacy C code, it seems that
> this isn't strictly true: we cannot portably cast an (unsigned char*)
> buffer containing a UTF-8-encoded, 0-terminated string to (char*) and
> operate on it with standard C functions. A UTF-8 encoded buffer may
> contain the byte value 0x80, which, when casted to char, might be a
> trap representation on platforms where CHAR_MIN is -127.
>
> This is awful shame, since there are byte values that never occur in
> well-formed UTF-8: 0xc0, 0xc1 and 0xf5-0xff. If one of those had been
> 0x80, to my understanding there wouldn't have been a problem.
>
> But if char might not be able to represent all the possible bytes of
> UTF-8, how can C1X have UTF-8 encoded string literals? Maybe I'll
> write a separate post about this to comp.std.c.

C does seem to be a bit inconsistent about whether unsigned chars can
safely be aliased as plain chars.  Plain char *can* have trap
representations, and if it does, a lot of common C idioms can break.

Practically speaking, implementations don't do this.

I think you can be reasonably safe if you do something like:

    #include <limits.h>
    #include <assert.h>
    ...
    assert(CHAR_BIT == 8 &&
           ((CHAR_MIN == -128 && CHAR_MAX == +127) ||
            (CHAR_MIN ==    0 && CHAR_MAX == +255)));

There are tricks you can use to get the effect of a compile-time
assertion as well.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

Thread

when is typecasting (unsigned char*) to (char*) dangerous? tim <nospam@nospam.com> - 2011-11-15 19:59 +0000
  Re: when is typecasting (unsigned char*) to (char*) dangerous? Kaz Kylheku <kaz@kylheku.com> - 2011-11-15 20:51 +0000
    Re: when is typecasting (unsigned char*) to (char*) dangerous? tim <nospam@nospam.com> - 2011-11-16 16:55 +0000
      Re: when is typecasting (unsigned char*) to (char*) dangerous? Kaz Kylheku <kaz@kylheku.com> - 2011-11-16 17:54 +0000
        Re: when is typecasting (unsigned char*) to (char*) dangerous? Keith Thompson <kst-u@mib.org> - 2011-11-16 11:27 -0800
          Re: when is typecasting (unsigned char*) to (char*) dangerous? Kaz Kylheku <kaz@kylheku.com> - 2011-11-16 19:49 +0000
            Re: when is typecasting (unsigned char*) to (char*) dangerous? Keith Thompson <kst-u@mib.org> - 2011-11-16 11:52 -0800
              Re: when is typecasting (unsigned char*) to (char*) dangerous? Kaz Kylheku <kaz@kylheku.com> - 2011-11-16 20:06 +0000
                Re: when is typecasting (unsigned char*) to (char*) dangerous? Keith Thompson <kst-u@mib.org> - 2011-11-16 14:55 -0800
                Re: when is typecasting (unsigned char*) to (char*) dangerous? Tim Rentsch <txr@alumni.caltech.edu> - 2012-01-25 04:42 -0800
              Re: when is typecasting (unsigned char*) to (char*) dangerous? James Kuyper <jameskuyper@verizon.net> - 2011-11-16 15:15 -0500
          Re: when is typecasting (unsigned char*) to (char*) dangerous? James Kuyper <jameskuyper@verizon.net> - 2011-11-16 15:24 -0500
            Re: when is typecasting (unsigned char*) to (char*) dangerous? Harald van Dĳk <truedfx@gmail.com> - 2011-11-16 12:41 -0800
              Re: when is typecasting (unsigned char*) to (char*) dangerous? James Kuyper <jameskuyper@verizon.net> - 2011-11-16 16:02 -0500
                Re: when is typecasting (unsigned char*) to (char*) dangerous? Harald van Dĳk <truedfx@gmail.com> - 2011-11-16 13:23 -0800
                Re: when is typecasting (unsigned char*) to (char*) dangerous? Keith Thompson <kst-u@mib.org> - 2011-11-16 15:32 -0800
                Re: when is typecasting (unsigned char*) to (char*) dangerous? Harald van Dĳk <truedfx@gmail.com> - 2011-11-16 17:02 -0800
                Re: when is typecasting (unsigned char*) to (char*) dangerous? pete <pfiland@mindspring.com> - 2011-11-16 21:38 -0500
                Re: when is typecasting (unsigned char*) to (char*) dangerous? Keith Thompson <kst-u@mib.org> - 2011-11-16 19:22 -0800
                Re: when is typecasting (unsigned char*) to (char*) dangerous? Harald van Dĳk <truedfx@gmail.com> - 2011-11-16 23:08 -0800
                Re: when is typecasting (unsigned char*) to (char*) dangerous? Keith Thompson <kst-u@mib.org> - 2011-11-16 23:46 -0800
                Re: when is typecasting (unsigned char*) to (char*) dangerous? Harald van Dĳk <truedfx@gmail.com> - 2011-11-17 09:15 -0800
                Re: when is typecasting (unsigned char*) to (char*) dangerous? Tim Rentsch <txr@alumni.caltech.edu> - 2012-01-25 04:54 -0800
          Re: when is typecasting (unsigned char*) to (char*) dangerous? Vincenzo Mercuri <vincenzo.mercuri@yahoo.it> - 2011-11-16 21:44 +0100
      Re: when is typecasting (unsigned char*) to (char*) dangerous? Kenneth Brody <kenbrody@spamcop.net> - 2011-11-16 13:17 -0500
  Re: when is typecasting (unsigned char*) to (char*) dangerous? Nick Keighley <nick_keighley_nospam@hotmail.com> - 2011-11-16 00:58 -0800
  Re: when is typecasting (unsigned char*) to (char*) dangerous? Malcolm McLean <malcolm.mclean5@btinternet.com> - 2011-11-16 01:36 -0800
    Re: when is typecasting (unsigned char*) to (char*) dangerous? Markus Wichmann <nullplan@gmx.net> - 2011-11-16 13:28 +0100
    Re: when is typecasting (unsigned char*) to (char*) dangerous? nroberts <roberts.noah@gmail.com> - 2011-11-16 08:14 -0800
      Re: when is typecasting (unsigned char*) to (char*) dangerous? BGB <cr88192@hotmail.com> - 2011-11-16 12:03 -0700
  Re: when is typecasting (unsigned char*) to (char*) dangerous? blp@cs.stanford.edu (Ben Pfaff) - 2011-11-16 08:58 -0800
    Re: when is typecasting (unsigned char*) to (char*) dangerous? Vincenzo Mercuri <vincenzo.mercuri@yahoo.it> - 2011-11-16 19:04 +0100
      Re: when is typecasting (unsigned char*) to (char*) dangerous? Ben Pfaff <blp@cs.stanford.edu> - 2011-11-16 10:05 -0800
      Re: when is typecasting (unsigned char*) to (char*) dangerous? Keith Thompson <kst-u@mib.org> - 2011-11-16 11:19 -0800
        Re: when is typecasting (unsigned char*) to (char*) dangerous? Vincenzo Mercuri <vincenzo.mercuri@yahoo.it> - 2011-11-16 20:58 +0100
          Re: when is typecasting (unsigned char*) to (char*) dangerous? Phil Carmody <thefatphil_demunged@yahoo.co.uk> - 2011-11-18 15:37 +0200
            Re: when is typecasting (unsigned char*) to (char*) dangerous? Vincenzo Mercuri <vincenzo.mercuri@yahoo.it> - 2011-11-18 19:41 +0100
              Re: when is typecasting (unsigned char*) to (char*) dangerous? Vincenzo Mercuri <vincenzo.mercuri@yahoo.it> - 2011-11-18 19:43 +0100
  Re: when is typecasting (unsigned char*) to (char*) dangerous? Lauri Alanko <la@iki.fi> - 2011-11-17 19:09 +0000
    Re: when is typecasting (unsigned char*) to (char*) dangerous? Harald van Dĳk <truedfx@gmail.com> - 2011-11-17 11:44 -0800
    Re: when is typecasting (unsigned char*) to (char*) dangerous? Keith Thompson <kst-u@mib.org> - 2011-11-17 13:16 -0800
  Re: when is casting (unsigned char*) to (signed char*) dangerous? Shao Miller <sha0.miller@gmail.com> - 2012-01-22 02:22 -0500
    Re: when is casting (unsigned char*) to (signed char*) dangerous? Harald van Dijk <haraldvdijk@gmail.com> - 2012-01-22 12:14 -0800
    Re: when is casting (unsigned char*) to (signed char*) dangerous? Keith Thompson <kst-u@mib.org> - 2012-01-23 00:37 -0800
      Re: when is casting (unsigned char*) to (signed char*) dangerous? Jens Gustedt <jens.gustedt@loria.fr> - 2012-01-24 15:10 +0100
      Re: when is casting (unsigned char*) to (signed char*) dangerous? Tim Rentsch <txr@alumni.caltech.edu> - 2012-01-26 08:55 -0800
        Re: when is casting (unsigned char*) to (signed char*) dangerous? Keith Thompson <kst-u@mib.org> - 2012-01-26 11:32 -0800
          Re: when is casting (unsigned char*) to (signed char*) dangerous? Tim Rentsch <txr@alumni.caltech.edu> - 2012-01-31 21:57 -0800
            Re: when is casting (unsigned char*) to (signed char*) dangerous? Shao Miller <sha0.miller@gmail.com> - 2012-02-01 01:12 -0500
    Re: when is casting (unsigned char*) to (signed char*) dangerous? Tim Rentsch <txr@alumni.caltech.edu> - 2012-01-26 08:48 -0800

csiph-web