Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #5655

Re: fgetwc doesn't read Unicode

Show all headers | View raw

felix.leg <felix.leg@vp.pl> wrote:
> I wrote a simple code to test unicode reading in C:

> [code]
> #include <stdlib.h>
> #include <stdio.h>
> #include <wctype.h>

> int main (int argc, char *argv[])
> {
>         FILE* f;
>         wint_t chrk;
>                 
>         f = fopen("characters","r");
>         
>         while( (chrk = fgetwc(f)) != WEOF ) {
>                 wprintf(L"%C ,", chrk );
>         }
>         printf("\b.\n"); //#1
>         fclose(f);
>         return 0;
> }
> [/code]

> The file "characters" contains a list of characters from different 
> Unicode blocks, each divided by a space. The first block contains 
> characters vaild in both ASCII and Unicode.

> But the output is just:

> a ,b ,c ,d ,e ,f ,: ,; ,' ," ,[ ,] ,{ ,} ,  ,
> (it looks like the #1 line wasn't executed too)

> It stops after fgetwc reach the first non-ASCII character from file.
> I thought this function was *supposed to* read them...WTF?

Note: I am not an Unicode expert, but since you haven't got an
answer yet I give it a try.

I guess that what's in your file is in UTF8. And UTF8 is not wide
char - a wide char has a fixed width, depending on the system your
trying this on, of 16, 32 or even 64 bits, while UTF8 is a diffe-
rent encoding systen where the chars differ in length (and only
with UTF8 the ASCII characters still work). One thing you could
try to figure out if this might be the problem is to check the
value of errno after the read failed (as indicated by ferror()),
chances are that it is EILSEQ, indicating that the what fgetwc()
tried to read was not recognized as a proper wide char.

I have no idea if this is possible on all systems but on mine
(Linux) you can coax fgetwc() into dealing with UTF8 characters
as expected (i.e. automatically converting them to wide chars)
by setting the LC_CTYPE category of the locale to something
with UTF8. I.e. if I have an input file with UTF8 characters
and add to the start of your program

  setlocale( LC_CTYPE, "en_US.UTF8" );

(after including <locale.h>) it starts to work in that fgetwc()
stops refusing to read in the characters. And if I check the
numerical values of the resulting wide chars they definitely
aren't the UTF8 characters from the input file anymore.

                            Regards, Jens
-- 
  \   Jens Thoms Toerring  ___      jt@toerring.de
   \__________________________      http://toerring.de

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

fgetwc doesn't read Unicode "felix.leg" <felix.leg@vp.pl> - 2011-06-08 13:41 +0200
  Re: fgetwc doesn't read Unicode jt@toerring.de (Jens Thoms Toerring) - 2011-06-08 20:18 +0000
  Re: fgetwc doesn't read Unicode Nobody <nobody@nowhere.com> - 2011-06-09 02:39 +0100
    Re: fgetwc doesn't read Unicode "felix.leg" <felix.leg@vp.pl> - 2011-06-12 11:30 +0200
      Re: fgetwc doesn't read Unicode Nobody <nobody@nowhere.com> - 2011-06-13 02:09 +0100
      Re: fgetwc doesn't read Unicode jt@toerring.de (Jens Thoms Toerring) - 2011-06-13 21:38 +0000
  Re: fgetwc doesn't read Unicode Dirk Krause <devnull@fh-schmalkalden.de> - 2011-06-09 13:19 +0000

csiph-web