Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #161323 > unrolled thread

Re: Locale-related questions

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Locale-related questions Alex Vong <alexvong1995@gmail.com> - 2015-11-29 14:40 +0100
    Re: Locale-related questions "Morten W. Petersen" <morphex@gmail.com> - 2015-11-29 16:00 +0100
      Re: Locale-related questions Nicolas George <george@nsup.org> - 2015-11-29 16:10 +0100
        Re: Locale-related questions "Morten W. Petersen" <morphex@gmail.com> - 2015-11-29 17:40 +0100
          Re: Locale-related questions "Thomas Schmitt" <scdbackup@gmx.net> - 2015-11-29 19:50 +0100
          Re: Locale-related questions Nicolas George <george@nsup.org> - 2015-11-29 20:30 +0100
            Re: Locale-related questions "Thomas Schmitt" <scdbackup@gmx.net> - 2015-11-29 20:50 +0100
              Re: Locale-related questions Nicolas George <george@nsup.org> - 2015-11-29 21:00 +0100
                Re: Locale-related questions Nicolas George <george@nsup.org> - 2015-11-29 21:50 +0100
                Re: Locale-related questions "Thomas Schmitt" <scdbackup@gmx.net> - 2015-11-29 21:50 +0100
        Re: Locale-related questions "Morten W. Petersen" <morphex@gmail.com> - 2015-12-10 08:40 +0100
          Re: Locale-related questions Nicolas George <george@nsup.org> - 2015-12-10 09:00 +0100
          Re: Locale-related questions "Thomas Schmitt" <scdbackup@gmx.net> - 2015-12-10 10:40 +0100

#161323 — Re: Locale-related questions

Hi Morten,

I have CCed this mail to <debian-user@lists.debian.org>. I think it is
a good place to talk about your question.

Regarding the question, could you please give more information? What
problem are you facing? Are you trying to convert encoding from/to
UTF-32 ? If yes, perhaps libiconv
<https://www.gnu.org/software/libiconv/> could help you.

Cheers,
Alex

On 29/11/2015, Morten W. Petersen <morphex@gmail.com> wrote:
> Hi there.
>
> I have some questions about building/enabling a UTF-32 locale in Debian or
> Linux in general, who could I talk to about that?
>
> Thanks,
>
> Morten
>
> --
> Twittering at http://twitter.com/blogologue
> Blogging at http://blogologue.com
> Playing music at https://soundcloud.com/morten-w-petersen
> Also playing music and podcasting here:
> http://www.mixcloud.com/morten-w-petersen/
> On Google+ here https://plus.google.com/107781930037068750156
> On Instagram at https://instagram.com/morphexx/
>

[toc] | [next] | [standalone]

#161326

[Multipart message — attachments visible in raw view] — view raw

Hi Alex,

thanks for that.

I was looking for a locale that would enable me to putwchar a 32-bit
Unicode character to stdout and have things handled correctly,
automatically.  Without any re-encoding to UTF-8 and so on.

-Morten

On Sun, Nov 29, 2015 at 2:21 PM, Alex Vong <alexvong1995@gmail.com> wrote:

> Hi Morten,
>
> I have CCed this mail to <debian-user@lists.debian.org>. I think it is
> a good place to talk about your question.
>
> Regarding the question, could you please give more information? What
> problem are you facing? Are you trying to convert encoding from/to
> UTF-32 ? If yes, perhaps libiconv
> <https://www.gnu.org/software/libiconv/> could help you.
>
> Cheers,
> Alex
>
> On 29/11/2015, Morten W. Petersen <morphex@gmail.com> wrote:
> > Hi there.
> >
> > I have some questions about building/enabling a UTF-32 locale in Debian
> or
> > Linux in general, who could I talk to about that?
> >
> > Thanks,
> >
> > Morten
> >
> > --
> > Twittering at http://twitter.com/blogologue
> > Blogging at http://blogologue.com
> > Playing music at https://soundcloud.com/morten-w-petersen
> > Also playing music and podcasting here:
> > http://www.mixcloud.com/morten-w-petersen/
> > On Google+ here https://plus.google.com/107781930037068750156
> > On Instagram at https://instagram.com/morphexx/
> >
>



-- 
Twittering at http://twitter.com/blogologue
Blogging at http://blogologue.com
Playing music at https://soundcloud.com/morten-w-petersen
Also playing music and podcasting here:
http://www.mixcloud.com/morten-w-petersen/
On Google+ here https://plus.google.com/107781930037068750156
On Instagram at https://instagram.com/morphexx/

[toc] | [prev] | [next] | [standalone]

#161327

[Multipart message — attachments visible in raw view] — view raw

Le nonidi 9 frimaire, an CCXXIV, Morten W. Petersen a écrit :
> I was looking for a locale that would enable me to putwchar a 32-bit
> Unicode character to stdout and have things handled correctly,
> automatically.  Without any re-encoding to UTF-8 and so on.

I do not know what your program is about, but what you are asking seems to
me like a very bad idea.

You should use putwchar() if you WANT automatic recoding to the current
locale, transparently and for any locale, as long as the character is
possible.

On the other hand, if as you say you do not want automatic recoding, then
you should use octet-based output functions: serialize your Unicode code
point to little or big endian as you prefer and use putchar() or fwrite() to
send it.

Note that wchar_t are Unicode code point under GNU but that is neither
guaranteed nor portable. Personally, I recommend not to use wchar_t
entirely.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#161332

[Multipart message — attachments visible in raw view] — view raw

Nicolas,

OK.  I don't know exactly what I'm talking about here as you can see. :)

I'm writing an XML parser/writer/simple DOM, which will input and output
primarily in UTF-32.

This program outputs some information to stdout in the testing process,
and this is also UTF-32 (internally a character is an integer >= 32 bits).

I tried now to use putwchar:

    putwchar((wchar_t) buffer[index]);

And the output is the same as if had used

     printf("%c", (char) buffer[index]);

That is, non-ASCII characters are garbled.

All the locale settings except LC_ALL are en_US.UTF-8, LC_ALL is empty.

What I'm looking for is a cross-platform way to output some data, to aid
in the testing process.  Reading and writing from files will probably be
binary and handled internally in the program.

-Morten

On Sun, Nov 29, 2015 at 4:04 PM, Nicolas George <george@nsup.org> wrote:

> Le nonidi 9 frimaire, an CCXXIV, Morten W. Petersen a écrit :
> > I was looking for a locale that would enable me to putwchar a 32-bit
> > Unicode character to stdout and have things handled correctly,
> > automatically.  Without any re-encoding to UTF-8 and so on.
>
> I do not know what your program is about, but what you are asking seems to
> me like a very bad idea.
>
> You should use putwchar() if you WANT automatic recoding to the current
> locale, transparently and for any locale, as long as the character is
> possible.
>
> On the other hand, if as you say you do not want automatic recoding, then
> you should use octet-based output functions: serialize your Unicode code
> point to little or big endian as you prefer and use putchar() or fwrite()
> to
> send it.
>
> Note that wchar_t are Unicode code point under GNU but that is neither
> guaranteed nor portable. Personally, I recommend not to use wchar_t
> entirely.
>
> Regards,
>
> --
>   Nicolas George
>



-- 
Twittering at http://twitter.com/blogologue
Blogging at http://blogologue.com
Playing music at https://soundcloud.com/morten-w-petersen
Also playing music and podcasting here:
http://www.mixcloud.com/morten-w-petersen/
On Google+ here https://plus.google.com/107781930037068750156
On Instagram at https://instagram.com/morphexx/

[toc] | [prev] | [next] | [standalone]

#161335

Hi,

Morten W. Petersen wrote:
> This program outputs some information to stdout in the testing process,
> and this is also UTF-32

As long as they are in stdout, the UTF-32 characters are byte
sequences of no special meaning.
The meaning as characters is attributed to them by the display
program (e.g. an xterm).


> I tried now to use putwchar:
>    putwchar((wchar_t) buffer[index]);
> And the output is the same as if had used
>    printf("%c", (char) buffer[index]);

It depends on the byte-order of 4-byte words whether the
output from this is equivalent. putwchar() might swap bytes
relative to the internal byte representation.

(Besides the waste of bytes, the byte order is another
 disadvantage of UTF-32 in comparison to UTF-8.)


> That is, non-ASCII characters are garbled.

Probably because your terminal (or other text display program)
expects UTF-8.

I googled for a terminal type which supports UTF-32, but failed.
So the only proposal i can make is to use iconv(1) in order
to convert the UTF-32 output to UTF-8 before it gets displayed.

  producer_process | iconv -f UTF-32LE -t UTF-8 | display_process

or

  producer_process | iconv -f UTF-32BE -t UTF-8 | display_process

depending on which byte order of UTF-32 your program puts out.
If you guessed the wrong order, you will probably get iconv errors
like
  iconv: illegal input sequence at position 0


Of course you should not convert to UTF-8 if the consumer process
expects UTF-32. In this case you only have to make sure that
producer and consumer work by the same byte order.
If "cross-platform" is meant seriously, then you will have to control
the byte order at output and input by own means. I.e. convert your
integers to bytes (and vice versa) by shifting in 8-bit steps.

E.g. integer to little endian (UTF-32LE):
  byte[0] = (integer >> 0) & 0xff;
  byte[1] = (integer >> 8) & 0xff;
  byte[2] = (integer >> 16) & 0xff;
  byte[3] = (integer >> 24) & 0xff;


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#161336

[Multipart message — attachments visible in raw view] — view raw

Thomas already gave an answer with relevant points.

Le nonidi 9 frimaire, an CCXXIV, Morten W. Petersen a écrit :
> I'm writing an XML parser/writer/simple DOM, which will input and output
> primarily in UTF-32.

Is there a good or unavoidable reason to use UTF-32? This is really a bad
choice of format for external representation. Nowadays, I would say that
UTF-8 should always be the preferred choice (for external representation;
for internal representation, using integers for code points may be better
depending on the use case).

> What I'm looking for is a cross-platform way to output some data, to aid
> in the testing process.  Reading and writing from files will probably be
> binary and handled internally in the program.

If you want cross-platform, stay away from wchar_t. It allows you to do
SIMPLE tings in a cross-platform way, such as printing an error message, but
no more. If you need control over the encoding, then you can not do it with
wchar_t portably. For starters, the i4s at microsoft decided that 64k
characters should be enough for everyone, so if your cross-platform includes
microsoftisms, you can not use wchar_t to represent an Unicode code point.
The i4s at sun had other interesting ideas on how to make the coding for
wchar_t itself depend on the locale.

If you really need to write UTF-32, writing the corresponding function takes
about half a minute:

void put_utf32be(FILE *f, unsigned c)
{
    putc(f, (c >> 24) & 0xFF);
    putc(f, (c >> 16) & 0xFF);
    putc(f, (c >>  8) & 0xFF);
    putc(f, (c >>  0) & 0xFF);
}

Note: I hereby place this code under the terms of the GNU GPL. And correctly
handling errors is left as an exercise to the reader.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#161337

Hi,

Nicolas George wrote:
> void put_utf32be(FILE *f, unsigned c)
>   ...
> Note: I hereby place this code under the terms of the GNU GPL.

Are you sure this function qualifies for copyright at all ? ~:o)


To be nitpicking ...

The man page of putc(3) looks suspicious:
       fputc() writes the character c, cast to an unsigned char, to stream.
       ...
       putc() is equivalent to fputc() except that it may be implemented as
       a macro which evaluates stream more than once.

sizeof(unsigned char) is not guarateed to be 1.
(Although it would break lots of software if not.)

So probably one should allocate an array of uint8_t, fill it
by shifted snippets of the integer, and write it by fwrite(3)
or write(2).


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#161338

[Multipart message — attachments visible in raw view] — view raw

Le nonidi 9 frimaire, an CCXXIV, Thomas Schmitt a écrit :
> Are you sure this function qualifies for copyright at all ? ~:o)

This would be for a judge to decide, if I were to sue.

> The man page of putc(3) looks suspicious:
>        fputc() writes the character c, cast to an unsigned char, to stream.
>        ...
>        putc() is equivalent to fputc() except that it may be implemented as
>        a macro which evaluates stream more than once.
> 
> sizeof(unsigned char) is not guarateed to be 1.

You are mistaken. Quoting the C standard:

# 6.5.3.4  The sizeof operator
#
# [#3] When applied to an operand that has type char, unsigned char, or
# signed char, (or a qualified version thereof) the result is 1.

Furthermore, C does not, but POSIX does mandate that chars are exactly
octets (C only mandates they are at least 8 bits).

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#161340

[Multipart message — attachments visible in raw view] — view raw

Le nonidi 9 frimaire, an CCXXIV, Thomas Schmitt a écrit :
> Well, in this case a byte does not necessarily consist of 8 bits,
> which would make data exchange somewhat difficult.

As I said, POSIX mandates that bytes are octets. C without POSIX does not
allow reliable binary I/O.

> But i confess that this is nearly a moot point.
> Important is to explicitely control the byte sequence on both sides
> - writer and reader.

Indeed.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#161341

Hi,

Nicolas George wrote:
> # [#3] When applied to an operand that has type char, unsigned char, or
> # signed char, (or a qualified version thereof) the result is 1.

Well, in this case a byte does not necessarily consist of 8 bits,
which would make data exchange somewhat difficult.

But i confess that this is nearly a moot point.
Important is to explicitely control the byte sequence on both sides
- writer and reader.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#161668

[Multipart message — attachments visible in raw view] — view raw

On Sun, Nov 29, 2015 at 4:04 PM, Nicolas George <george@nsup.org> wrote:

> Le nonidi 9 frimaire, an CCXXIV, Morten W. Petersen a écrit :
> > I was looking for a locale that would enable me to putwchar a 32-bit
> > Unicode character to stdout and have things handled correctly,
> > automatically.  Without any re-encoding to UTF-8 and so on.
>
> I do not know what your program is about, but what you are asking seems to
> me like a very bad idea.
>
> You should use putwchar() if you WANT automatic recoding to the current
> locale, transparently and for any locale, as long as the character is
> possible.
>
>
>
Hm.  But how would I go about setting up a locale that is in fact UTF-32?

I guess I need to build locale files that contain translations and so on in
UTF-32, and also need to hack a bit on terminal code and so on to get
that going?

-Morten

-- 
Twittering at http://twitter.com/blogologue
Blogging at http://blogologue.com
Playing music at https://soundcloud.com/morten-w-petersen
Also playing music and podcasting here:
http://www.mixcloud.com/morten-w-petersen/
On Google+ here https://plus.google.com/107781930037068750156
On Instagram at https://instagram.com/morphexx/

[toc] | [prev] | [next] | [standalone]

#161669

[Multipart message — attachments visible in raw view] — view raw

Le decadi 20 frimaire, an CCXXIV, Morten W. Petersen a écrit :
> Hm.  But how would I go about setting up a locale that is in fact UTF-32?

You do not.

Locales are a bad API, do not use it except for the most basic tasks, and
this is not a basic case.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#161672

Hi,

Morten W. Petersen :
> need to hack a bit on terminal code

This is not only about terminals but also about any other
program or module which shall process your UTF-32 text.
You would have to set up a complete UTF-32 capable system
while everything around it expects UTF-8.

For example, in C language the byte value 0 indicates the
end of a string. UTF-32 characters out of the ASCII range
contain 75 percent of 0-bytes.
If your terminal's code is in C or uses C oriented libraries
for processing text, then you have to check and modify all
this stuff.

Compare this with the effort needed to create an interface to
the UTF-8 world by which your UTF-32 programs can communicate
with normal programs.


Have a nice day :)

Thomas

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web