Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #97616 > unrolled thread

UTF-8 with C, simply

Back to article view | Back to comp.lang.c

Contents

  UTF-8 with C, simply luser droog <luser.droog@gmail.com> - 2016-11-27 18:09 -0800
    Re: UTF-8 with C, simply James Kuyper <jameskuyper@verizon.net> - 2016-11-27 22:05 -0500
    Re: UTF-8 with C, simply Lorinczy Zsigmond <zsiga@nospam.for.me> - 2016-11-28 09:42 +0100
      Re: UTF-8 with C, simply James Kuyper <jameskuyper@verizon.net> - 2016-11-28 09:10 -0500
        Re: UTF-8 with C, simply Philipp Klaus Krause <pkk@spth.de> - 2016-11-28 21:55 +0100
          Re: UTF-8 with C, simply "James R. Kuyper" <jameskuyper@verizon.net> - 2016-11-28 16:18 -0500
    Re: UTF-8 with C, simply Siri Cruise <chine.bleu@yahoo.com> - 2016-11-28 07:00 -0800
      Re: UTF-8 with C, simply Philipp Klaus Krause <pkk@spth.de> - 2016-11-28 21:53 +0100
      Re: UTF-8 with C, simply Keith Thompson <kst-u@mib.org> - 2016-11-28 21:38 -0800
    Re: UTF-8 with C, simply Nobody <nobody@nowhere.invalid> - 2016-12-01 01:52 +0000
      Re: UTF-8 with C, simply luser droog <luser.droog@gmail.com> - 2016-11-30 18:20 -0800
        Re: UTF-8 with C, simply luser droog <luser.droog@gmail.com> - 2016-11-30 18:25 -0800
          Re: UTF-8 with C, simply luser droog <luser.droog@gmail.com> - 2016-11-30 18:28 -0800

#97616 — UTF-8 with C, simply

I knew nothing of the mbtowc and friends except that they
existed but didn't seem to describe anything useful to me. 
I never got the connection that the wc could be the ucs2 or
ucs4 code and the mb could be the corresponding utf8 code.
But James's recent post about the file mode and fgetwc lead
me to this setup:

call setlocale(LC_CTYPE, "UTF-8");
then use fgetwc() and fputwc() to read and write ucs2 or ucs4
depending on the implementation's definition of wchar_t, right?
And it's all automagic?

[toc] | [next] | [standalone]

#97617

On 11/27/2016 09:09 PM, luser droog wrote:
> I knew nothing of the mbtowc and friends except that they
> existed but didn't seem to describe anything useful to me. 
> I never got the connection that the wc could be the ucs2 or
> ucs4 code and the mb could be the corresponding utf8 code.
> But James's recent post about the file mode and fgetwc lead
> me to this setup:
> 
> call setlocale(LC_CTYPE, "UTF-8");
> then use fgetwc() and fputwc() to read and write ucs2 or ucs4
> depending on the implementation's definition of wchar_t, right?
> And it's all automagic?

There's no guarantee that wchar_t is ucs2 or ucs4 - but considering what
wchar_t is supposed to do, I think those are the most likely encodings
(an 8-bit encoding is also possible, in which case wchar_t and char
would be essentially equivalent). That judgment is NOT based on any
actual experience, just theory, so treat it with no more respect than it
deserves.

[toc] | [prev] | [next] | [standalone]

#97621

Well, you should now that neither ucs2, ucs4 or utf16 isn't equal to utf8.
under newer C-standards you can use data-types char16_t and char32_t 
(from header uchar.h),
they are meant to be utf16 and utf32 respectively.

[toc] | [prev] | [next] | [standalone]

#97636

On 11/28/2016 03:42 AM, Lorinczy Zsigmond wrote:
> Well, you should now that neither ucs2, ucs4 or utf16 isn't equal to utf8.

Well, that seems consistent with what he said. You wouldn't need to call
special routines to  cause a conversion from utf8 to ucs2 or ucs4 if
utf8 were, in fact, equal to one of them. What made you feel the need to
point that out? Normally, unless you have a specific reason for thinking
otherwise, the simple fact that they have different names would imply
that they're probably not equal.

> under newer C-standards you can use data-types char16_t and char32_t 
> (from header uchar.h),
> they are meant to be utf16 and utf32 respectively

That's only true on implementations that choose to pre#define
__STDC_UTF_16__ or __STDC_UTF_32__, respectively. Otherwise, the
encoding associated with those typedefs is implementation-defined
(6.10.8.2p1). Of course, this is a significant improvement over wchar_t,
for which no such macros are available.

[toc] | [prev] | [next] | [standalone]

#97684

>> under newer C-standards you can use data-types char16_t and char32_t 
>> (from header uchar.h),
>> they are meant to be utf16 and utf32 respectively
> 
> That's only true on implementations that choose to pre#define
> __STDC_UTF_16__ or __STDC_UTF_32__, respectively. Otherwise, the
> encoding associated with those typedefs is implementation-defined
> (6.10.8.2p1). Of course, this is a significant improvement over wchar_t,
> for which no such macros are available.
> 

How about __STDC_ISO_10646__?

Philipp

[toc] | [prev] | [next] | [standalone]

#97686

On 11/28/2016 03:55 PM, Philipp Klaus Krause wrote:
>>> under newer C-standards you can use data-types char16_t and char32_t
>>> (from header uchar.h),
>>> they are meant to be utf16 and utf32 respectively
>>
>> That's only true on implementations that choose to pre#define
>> __STDC_UTF_16__ or __STDC_UTF_32__, respectively. Otherwise, the
>> encoding associated with those typedefs is implementation-defined
>> (6.10.8.2p1). Of course, this is a significant improvement over wchar_t,
>> for which no such macros are available.
>>
>
> How about __STDC_ISO_10646__?

You're right - I missed that. It was just two paragraphs above the ones 
I did look at - you'd think I might have noticed it.
When I say that something is there, I'm pretty reliable. When I say that 
something isn't there - less so.

[toc] | [prev] | [next] | [standalone]

#97638

In article <6a3e7ab3-eb0d-4794-befb-adae3729c746@googlegroups.com>,
 luser droog <luser.droog@gmail.com> wrote:

> I knew nothing of the mbtowc and friends except that they
> existed but didn't seem to describe anything useful to me. 
> I never got the connection that the wc could be the ucs2 or
> ucs4 code and the mb could be the corresponding utf8 code.
> But James's recent post about the file mode and fgetwc lead
> me to this setup:

I found the documentation too indecisive, making it sound like I would have to 
run test programs to figure out what's going on.

The conversion between unicode and UTF-8 is very simple intentionnally. So I use 
a library like Tcl which explicitly states it is exactly UTF-8 and unicode or I 
just write my own. For any other conversions I use iconv with char strings and 
explicitly given character encoding names.

-- 
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted.
'I desire mercy, not sacrifice.'
Free the Amos Yee one.
Yeah, too bad about your so-called life. Ha-ha.

[toc] | [prev] | [next] | [standalone]

#97683

On 28.11.2016 16:00, Siri Cruise wrote:
> In article <6a3e7ab3-eb0d-4794-befb-adae3729c746@googlegroups.com>,
>  luser droog <luser.droog@gmail.com> wrote:
> 
>> I knew nothing of the mbtowc and friends except that they
>> existed but didn't seem to describe anything useful to me. 
>> I never got the connection that the wc could be the ucs2 or
>> ucs4 code and the mb could be the corresponding utf8 code.
>> But James's recent post about the file mode and fgetwc lead
>> me to this setup:
> 
> I found the documentation too indecisive, making it sound like I would have to 
> run test programs to figure out what's going on.
> 
> The conversion between unicode and UTF-8 is very simple intentionnally. So I use 
> a library like Tcl which explicitly states it is exactly UTF-8 and unicode or I 
> just write my own. For any other conversions I use iconv with char strings and 
> explicitly given character encoding names.
> 

These days most sane implementations (and probably and implementation
that matters to you) will use Unicode for char16_t, char32_t and wchar_t.
I wouldn't bother with runtime checks, and instead just check for the
relevant macros (__STDC_UTF_16__, __STDC_UTF_32__, __STDC_ISO_10646__)
at compile time, just to give a proper #error in case someone compiles
my code on something where it wouldn't work.
Unfortunately, for UTF-8 the situation is more complicated.

Philipp

[toc] | [prev] | [next] | [standalone]

#97707

Siri Cruise <chine.bleu@yahoo.com> writes:
[...]
> The conversion between unicode and UTF-8 is very simple
> intentionnally. So I use a library like Tcl which explicitly states it
> is exactly UTF-8 and unicode or I just write my own. For any other
> conversions I use iconv with char strings and explicitly given
> character encoding names.

What do you mean by "conversion between unicode and UTF-8"?  Unicode is
not a representation.  UTF-8 is one of several representations *of*
Unicode.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#97804

On Sun, 27 Nov 2016 18:09:53 -0800, luser droog wrote:

> call setlocale(LC_CTYPE, "UTF-8");
> then use fgetwc() and fputwc() to read and write ucs2 or ucs4 depending on
> the implementation's definition of wchar_t, right? And it's all automagic?

"UTF-8" probably isn't a valid locale.

On Linux, you'd typically use something like "en_US.UTF-8" to select the
en_US locale with utf-8 as the character encoding. wchar_t will always be
Unicode (UCS-4).

On Windows, your life will be somewhat harder. Windows barely tolerates
the ISO C APIs, and doesn't use UTF-8 for any of its locales. If you want
Unicode, you're supposed to use UTF-16 and wchar_t* for everything, which
means using non-standard functions such as

FILE *_wfopen(const wchar_t *filename, const wchar_t *mode);

[toc] | [prev] | [next] | [standalone]

#97806

On Wednesday, November 30, 2016 at 7:52:25 PM UTC-6, Nobody wrote:
> On Sun, 27 Nov 2016 18:09:53 -0800, luser droog wrote:
> 
> > call setlocale(LC_CTYPE, "UTF-8");
> > then use fgetwc() and fputwc() to read and write ucs2 or ucs4 depending on
> > the implementation's definition of wchar_t, right? And it's all automagic?
> 
> "UTF-8" probably isn't a valid locale.
> 
> On Linux, you'd typically use something like "en_US.UTF-8" to select the
> en_US locale with utf-8 as the character encoding. wchar_t will always be
> Unicode (UCS-4).
> 
> On Windows, your life will be somewhat harder. Windows barely tolerates
> the ISO C APIs, and doesn't use UTF-8 for any of its locales. If you want
> Unicode, you're supposed to use UTF-16 and wchar_t* for everything, which
> means using non-standard functions such as
> 
> FILE *_wfopen(const wchar_t *filename, const wchar_t *mode);

Yuck. Thankfully, I'm not worried about filenames just yet.
But for my own purposes I need extensibility as well as portability.
So the best solution seems to be to roll-my-own.

My current conversion functions operate on a single code-point.
Prototype for C99 + POSIX systems reads one keyboard char from stdin,
and prints the bytes received and the Unicode code-point and
converts back to utf-8 for output. (expand_shortcut is a stub
for my app-specific extension.)

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <math.h> // log2

typedef struct {
    int n;
    unsigned char b[4];
} utfcp;
uint32_t to_ucs4(utfcp c);
utfcp to_utf8(uint32_t u);

#define REPLACEMENT 0xFFFD

uint32_t expand_shortcut(unsigned char b){
    return REPLACEMENT;
}

uint32_t to_ucs4(utfcp c){
    int prefix = leading1s(c.b[0]);
    int n = prefix? prefix: 1;
    uint32_t u;
    //printf("prefix:%d\n",n);
    //if (n != c.n)
    switch(prefix){
    case 0: u = c.b[0]; break;
    case 1: return u = expand_shortcut(c.b[0]);
    case 2: u = c.b[0] & 0x1f; break;
    case 3: u = c.b[0] & 0x0f; break;
    case 4: u = c.b[0] & 0x07; break;
    }
    //printf("%04x\n", u);
    for(int i=1; i<n; ++i){
        u = (u << 6) | (c.b[i] & 0x3f);
        //printf("%04x\n", u);
    }

    if (u < ((int[]){0,0,0x80,0x800,0x10000})[prefix]) {
        //error |= over_length_encoding;
        u=REPLACEMENT;
    }
    return u;
}

utfcp to_utf8(uint32_t u){
    if (u<0x80) return (utfcp){1,u};
    if (u<0x800) return (utfcp){2,0xC0|(u>>6),
                     0x80|(u&0x3f)};
    if (u<0x10000) return (utfcp){3,0xE0|(u>>12),
                       0x80|((u>>6)&0x3f),0x80|(u&0x3f)};
    if (u<0x110000) return (utfcp){4,0xF0|(u>>18),
                        0x80|((u>>12)&0x3f),0x80|((u>>6)&0x3f),0x80|(u&0x3f)};
    //(else) error RANGE
    return (utfcp){0,0};
}

#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <termios.h>
#include <unistd.h>

#include "io.h"

struct termios saved_settings;

void restore_terminal(void){
    tcsetattr(0, TCSANOW, &saved_settings);
}

void init_terminal(void){
    tcgetattr(0, &saved_settings);
    atexit(restore_terminal);

    struct termios raw_mode = saved_settings;

    raw_mode.c_iflag |= IGNPAR; //ignore parity errors
    raw_mode.c_iflag &=  //non-canon, no echo, no kill
        ~(IGNBRK | PARMRK | ISTRIP | ICRNL | IXON | IXANY | IXOFF);

    raw_mode.c_lflag &=
        ~(ECHO | ECHOE | ECHOK | ECHONL | ICANON);

    raw_mode.c_cflag &= ~(CSIZE | PARENB);
    raw_mode.c_cflag |= CS8;

    raw_mode.c_oflag |= OPOST;  //special output processing

    raw_mode.c_cc[VMIN] = 4;  //min chars to read
    raw_mode.c_cc[VTIME] = 1;  //timeout

    if (tcsetattr(0, TCSANOW, &raw_mode) == -1)
        perror("init_terminal");
}

unsigned read_character(void){
    int len;
    char buf[4];
    do {
        len = read(fileno(stdin), buf, 4);
    } while(len == -1 && errno == EAGAIN);
    printf("%d:", len);
    for (int i=0; i<len; ++i)
        printf(" %02x", (unsigned)(unsigned char)buf[i]);
    puts("");

    return len==0
        ? EOF
        : to_ucs4((utfcp){len, buf[0], buf[1], buf[2], buf[3]});
}

int main(void){
    unsigned c;
    init_terminal();

    c = read_character();
    printf("U%04x\n", c);
    utfcp u = to_utf8(c);
    printf("%*s", u.n, u.b);
}

[toc] | [prev] | [next] | [standalone]

#97807

On Wednesday, November 30, 2016 at 8:20:44 PM UTC-6, luser droog wrote:

> #include "io.h"

Ignore that. The needed definitions and functions are inlined above
this part.

[toc] | [prev] | [next] | [standalone]

#97808

On Wednesday, November 30, 2016 at 8:25:56 PM UTC-6, luser droog wrote:
> On Wednesday, November 30, 2016 at 8:20:44 PM UTC-6, luser droog wrote:
> 
> > #include "io.h"
> 
> Ignore that. The needed definitions and functions are inlined above
> this part.

Except this other part. sigh.

//additional excerpt from io.c need for code in previous post,
//which also explains the mysterious need for math.h, maybe, sortof.

/* number of leading zeros of byte-sized value */
static int leading0s(uint_least32_t x){ return 7 - (x? floor(log2(x)): -1); }

/* number of leading ones of byte-sized value */
#define leading1s(x) leading0s(0xFF^(x))

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web