Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #73706 > unrolled thread

How to get a character consisting of multiple bytes into a string literal?

Back to article view | Back to comp.lang.c

Contents

  How to get a character consisting of multiple bytes into a string literal? Philipp Klaus Krause <pkk@spth.de> - 2015-10-24 19:18 +0200
    Re: How to get a character consisting of multiple bytes into a string literal? James Kuyper <jameskuyper@verizon.net> - 2015-10-24 14:06 -0400
      Re: How to get a character consisting of multiple bytes into a string literal? Philipp Klaus Krause <pkk@spth.de> - 2015-10-24 20:39 +0200
        Re: How to get a character consisting of multiple bytes into a string literal? James Kuyper <jameskuyper@verizon.net> - 2015-10-24 15:05 -0400
    Re: How to get a character consisting of multiple bytes into a string literal? Richard Damon <Richard@Damon-Family.org> - 2015-10-24 14:07 -0400
      Re: How to get a character consisting of multiple bytes into a string literal? James Kuyper <jameskuyper@verizon.net> - 2015-10-24 14:33 -0400
      Re: How to get a character consisting of multiple bytes into a string literal? Philipp Klaus Krause <pkk@spth.de> - 2015-10-24 21:30 +0200
    Re: How to get a character consisting of multiple bytes into a string literal? Malcolm McLean <malcolm.mclean5@btinternet.com> - 2015-10-24 11:11 -0700
      Re: How to get a character consisting of multiple bytes into a string literal? Keith Thompson <kst-u@mib.org> - 2015-10-24 13:44 -0700
    Re: How to get a character consisting of multiple bytes into a string literal? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2015-10-24 19:18 +0100
    Re: How to get a character consisting of multiple bytes into a string literal? Tim Rentsch <txr@alumni.caltech.edu> - 2015-10-24 11:57 -0700
      Re: How to get a character consisting of multiple bytes into a string literal? Philipp Klaus Krause <pkk@spth.de> - 2015-10-24 21:32 +0200
        Re: How to get a character consisting of multiple bytes into a string literal? James Kuyper <jameskuyper@verizon.net> - 2015-10-24 15:54 -0400
        Re: How to get a character consisting of multiple bytes into a string literal? Tim Rentsch <txr@alumni.caltech.edu> - 2015-10-24 14:43 -0700
          Re: How to get a character consisting of multiple bytes into a string literal? Philipp Klaus Krause <pkk@spth.de> - 2015-10-25 10:38 +0100
            Re: How to get a character consisting of multiple bytes into a string literal? Tim Rentsch <txr@alumni.caltech.edu> - 2015-10-25 10:05 -0700

#73706 — How to get a character consisting of multiple bytes into a string literal?

Assuming my implementation supports the basic source character set, how
can I create a string literal that contains multibyte character
consisting of multiple bytes?

I can't just write the character directly, since it is not in the basic
source character set.

And I don't think I can use an escape sequence: Verse 4 of section 6.4.5
of the C11 standard states "The same considerations apply to each
element of the sequence in a string literal as if it were in an integer
character constant […]". I guess that means that 6.4.4.4, verse 10
applies, and thus the value of the character constant has to be
something representable in a char.

So is it possible to create a string literal containing a multibyte
character of more than one byte when the implementation supports only
the basic source character set as source character set?

Philipp

[toc] | [next] | [standalone]

#73709

On 10/24/2015 01:18 PM, Philipp Klaus Krause wrote:
> Assuming my implementation supports the basic source character set, how

Every conforming implementation is required to support the basic source
character set. It's also required to support an extended source
character set - but it gets to define what the extensions are - there
need not be any difference between the basic and extended source
character sets.

Note, however, that the minimum number of different char values is 255
(if CHAR_BIT==8, and char is a signed type using 1's complement or
sign-magnitude representation, so that CHAR_MAX == 127, CHAR_MIN==-127).
That's much larger than the size of the basic character set, so there
must be char values that don't represent any member of the basic
character set.

> can I create a string literal that contains multibyte character
> consisting of multiple bytes?

"The source character set may contain multibyte characters, used to
represent members of the extended character set. The execution character
set may also contain multibyte characters, ..." (5.2.1.2p1). Note the
use of the word "may" - a conforming implementation need not have any
such characters, in either the source or execution character sets.

> I can't just write the character directly, since it is not in the basic
> source character set.
> 
> And I don't think I can use an escape sequence: Verse 4 of section 6.4.5
> of the C11 standard states "The same considerations apply to each
> element of the sequence in a string literal as if it were in an integer
> character constant […]". I guess that means that 6.4.4.4, verse 10
> applies, and thus the value of the character constant has to be
> something representable in a char.
> 
> So is it possible to create a string literal containing a multibyte
> character of more than one byte when the implementation supports only
> the basic source character set as source character set?

No.

[toc] | [prev] | [next] | [standalone]

#73715

On 24.10.2015 20:06, James Kuyper wrote:

> "The source character set may contain multibyte characters, used to
> represent members of the extended character set. The execution character
> set may also contain multibyte characters, ..." (5.2.1.2p1). Note the
> use of the word "may" - a conforming implementation need not have any
> such characters, in either the source or execution character sets.

That verse from the C11 standard seems like a mistake to me. Elsewhere
in the standard, anything is a multibyte character, with the character
being exactly one byte being just a special case of a multibyte character.

Philipp

[toc] | [prev] | [next] | [standalone]

#73720

On 10/24/2015 02:39 PM, Philipp Klaus Krause wrote:
> On 24.10.2015 20:06, James Kuyper wrote:
> 
>> "The source character set may contain multibyte characters, used to
>> represent members of the extended character set. The execution character
>> set may also contain multibyte characters, ..." (5.2.1.2p1). Note the
>> use of the word "may" - a conforming implementation need not have any
>> such characters, in either the source or execution character sets.
> 
> That verse from the C11 standard seems like a mistake to me. Elsewhere
> in the standard, anything is a multibyte character, with the character
> being exactly one byte being just a special case of a multibyte character.

You're correct, 3.7.2 defines "multibyte character" as including
characters that can be represented using only one byte. However, using
that interpretation in this context only makes 5.2.1.2p1 support my
conclusion even more strongly: it means that the source character set
need not even have single-byte characters capable of representing
members of the extended character set - something that seems obvious
simply by comparing the definitions of the two character sets.

Personally, I believe that 5.2.1.2p1 is using "multibyte character" to
referring specifically to characters that require more than one byte to
represent them. However, I won't insist on that interpretation.

[toc] | [prev] | [next] | [standalone]

#73710

On 10/24/15 1:18 PM, Philipp Klaus Krause wrote:
> Assuming my implementation supports the basic source character set, how
> can I create a string literal that contains multibyte character
> consisting of multiple bytes?
>
> I can't just write the character directly, since it is not in the basic
> source character set.
>
> And I don't think I can use an escape sequence: Verse 4 of section 6.4.5
> of the C11 standard states "The same considerations apply to each
> element of the sequence in a string literal as if it were in an integer
> character constant […]". I guess that means that 6.4.4.4, verse 10
> applies, and thus the value of the character constant has to be
> something representable in a char.
>
> So is it possible to create a string literal containing a multibyte
> character of more than one byte when the implementation supports only
> the basic source character set as source character set?
>
> Philipp
>

You represent the multi-byte character as the sequence of bytes in the 
multi-byte representation.

Note that 'representable in a char' means in the value range allowed by 
the type, not that there is an allowable 'character' that is represented 
by that value.

[toc] | [prev] | [next] | [standalone]

#73714

On 10/24/2015 02:07 PM, Richard Damon wrote:
> On 10/24/15 1:18 PM, Philipp Klaus Krause wrote:
...
>> So is it possible to create a string literal containing a multibyte
>> character of more than one byte when the implementation supports only
>> the basic source character set as source character set?
>>
>> Philipp
>>
> 
> You represent the multi-byte character as the sequence of bytes in the 
> multi-byte representation.

In one sense, what you're saying is correct - but if so, it's a
multibyte representation that the C implementation itself knows nothing
about. The printf() family, the scanf() family, and every function with
an "mb" in its name assumes that one or more of the char* pointers that
it takes as an argument points at a multibyte character string. On an
implementation which doesn't support any multibyte characters, don't
expect the construct you describe to be recognized as one by any of
those functions.

[toc] | [prev] | [next] | [standalone]

#73726

On 24.10.2015 20:07, Richard Damon wrote:
> On 10/24/15 1:18 PM, Philipp Klaus Krause wrote:
>> Assuming my implementation supports the basic source character set, how
>> can I create a string literal that contains multibyte character
>> consisting of multiple bytes?
>>
>> I can't just write the character directly, since it is not in the basic
>> source character set.
>>
>> And I don't think I can use an escape sequence: Verse 4 of section 6.4.5
>> of the C11 standard states "The same considerations apply to each
>> element of the sequence in a string literal as if it were in an integer
>> character constant […]". I guess that means that 6.4.4.4, verse 10
>> applies, and thus the value of the character constant has to be
>> something representable in a char.
>>
>> So is it possible to create a string literal containing a multibyte
>> character of more than one byte when the implementation supports only
>> the basic source character set as source character set?
>>
>> Philipp
>>
> 
> You represent the multi-byte character as the sequence of bytes in the
> multi-byte representation.

OK, works for string literals prefixed by u8, since those are UTF-8. But
for other string literals the encoding is not fixed by the standard.

Philipp

[toc] | [prev] | [next] | [standalone]

#73711

On Saturday, October 24, 2015 at 6:18:56 PM UTC+1, Philipp Klaus Krause wrote:
>
> Assuming my implementation supports the basic source character set, how
> can I create a string literal that contains multibyte character
> consisting of multiple bytes?
> 
Depends how you want to pass the data about.

You can use the L prefix L"Wide chars"; to create a 16-bit string
literal. Or you use use UTF-8 and write char *str = "utf-8 is ascii
with extended characters like aleph (\xe2\x84\xb5)";

The second method is OK for a few embedded non-English characters,
but not human-maintainable for flowing text, so realistically you'll
need a tool to generate the literals.

[toc] | [prev] | [next] | [standalone]

#73737

Malcolm McLean <malcolm.mclean5@btinternet.com> writes:
> On Saturday, October 24, 2015 at 6:18:56 PM UTC+1, Philipp Klaus Krause wrote:
>> Assuming my implementation supports the basic source character set, how
>> can I create a string literal that contains multibyte character
>> consisting of multiple bytes?
>> 
> Depends how you want to pass the data about.
>
> You can use the L prefix L"Wide chars"; to create a 16-bit string
> literal.

A string literal prefixed with L represent a sequence of wchar_t
elements.  wchar_t is typically 16 bits on Windows, but 32 bits on
Unix-like systems.

>          Or you use use UTF-8 and write char *str = "utf-8 is ascii
> with extended characters like aleph (\xe2\x84\xb5)";

Or, if your compiler supports it, u8"..." is a UTF-8 string literal,
u"..." represents a sequence of char16_t, and U"..." represents a
sequence of char32_t.  These are new in C11.

> The second method is OK for a few embedded non-English characters,
> but not human-maintainable for flowing text, so realistically you'll
> need a tool to generate the literals.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#73713

Philipp Klaus Krause <pkk@spth.de> writes:

> Assuming my implementation supports the basic source character set, how
> can I create a string literal that contains multibyte character
> consisting of multiple bytes?
>
> I can't just write the character directly, since it is not in the basic
> source character set.
>
> And I don't think I can use an escape sequence: Verse 4 of section 6.4.5
> of the C11 standard states "The same considerations apply to each
> element of the sequence in a string literal as if it were in an integer
> character constant […]". I guess that means that 6.4.4.4, verse 10
> applies, and thus the value of the character constant has to be
> something representable in a char.

Sure, but each of the characters that make up the multibyte character
can be represented in an unsigned char (see 6.4.4.4 p9 and below).

> So is it possible to create a string literal containing a multibyte
> character of more than one byte when the implementation supports only
> the basic source character set as source character set?

u8"\xC2\xA3" (meaning "£") is fine from that point of view because
there's nothing in the source that is outside of the source character
set.  What's more, the values 0xC2 and 0xA3 can represented as values of
the type unsigned char as required by 6.4.4.4 p9.  (This requirement
applies even if char is signed.  I.e. you can write '\xC2' even when
0xC2 is not in the range of char itself.)

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#73717

Philipp Klaus Krause <pkk@spth.de> writes:

> So is it possible to create a string literal containing a multibyte
> character of more than one byte when the implementation supports only
> the basic source character set as source character set?

Yes, it has been since C99:

    #include <stdio.h>

    int
    main(){
        char foo[] = "\u1234";
        printf( "sizeof foo = %zu\n", sizeof foo );
        return  0;
    }

Running this on my system prints 'sizeof foo = 4'.

[toc] | [prev] | [next] | [standalone]

#73727

On 24.10.2015 20:57, Tim Rentsch wrote:
> Philipp Klaus Krause <pkk@spth.de> writes:
> 
>> So is it possible to create a string literal containing a multibyte
>> character of more than one byte when the implementation supports only
>> the basic source character set as source character set?
> 
> Yes, it has been since C99:
> 
>     #include <stdio.h>
> 
>     int
>     main(){
>         char foo[] = "\u1234";
>         printf( "sizeof foo = %zu\n", sizeof foo );
>         return  0;
>     }
> 
> Running this on my system prints 'sizeof foo = 4'.
> 

This looks like implementation-defined behaviour by verse 10 of section
6.4.4.4 of the C11 standard to me.
What compiler did you use?

Philipp

[toc] | [prev] | [next] | [standalone]

#73730

On 10/24/2015 03:32 PM, Philipp Klaus Krause wrote:
> On 24.10.2015 20:57, Tim Rentsch wrote:
>> Philipp Klaus Krause <pkk@spth.de> writes:
>>
>>> So is it possible to create a string literal containing a multibyte
>>> character of more than one byte when the implementation supports only
>>> the basic source character set as source character set?
>>
>> Yes, it has been since C99:
>>
>>     #include <stdio.h>
>>
>>     int
>>     main(){
>>         char foo[] = "\u1234";
>>         printf( "sizeof foo = %zu\n", sizeof foo );
>>         return  0;
>>     }
>>
>> Running this on my system prints 'sizeof foo = 4'.

Does the compiler on your system "support only the basic character set",
as specified in the question? While universal character names (UCNs) can
be expressed without using any characters outside of that set, they're
not required to be meaningfully implemented.

> This looks like implementation-defined behaviour by verse 10 of section
> 6.4.4.4 of the C11 standard to me.
> What compiler did you use?

The "value ... is implementation-defined", not the behavior.
Note that the implementation-defined value need not be different for
UCNs. Since the target implementation is specified to be one that
"supports only the basic source character set", this is likely to be the
case - there's no reason for such an implementation to have CHAR_BIT >
8, no requirement that it support characters that are more than one byte
long, and there aren't enough distinct single-byte values to encode all
UCNs uniquely.

[toc] | [prev] | [next] | [standalone]

#73742

Philipp Klaus Krause <pkk@spth.de> writes:

> On 24.10.2015 20:57, Tim Rentsch wrote:
>> Philipp Klaus Krause <pkk@spth.de> writes:
>>
>>> So is it possible to create a string literal containing a multibyte
>>> character of more than one byte when the implementation supports only
>>> the basic source character set as source character set?
>>
>> Yes, it has been since C99:
>>
>>     #include <stdio.h>
>>
>>     int
>>     main(){
>>         char foo[] = "\u1234";
>>         printf( "sizeof foo = %zu\n", sizeof foo );
>>         return  0;
>>     }
>>
>> Running this on my system prints 'sizeof foo = 4'.
>
> This looks like implementation-defined behaviour by verse 10 of
> section 6.4.4.4 of the C11 standard to me.

Certainly I would agree that the value of a /character constant/
with a multibyte character is implementation-defined, but your
question is about string literals, not character constants.  Note
the last sentence of 6.4.5 p6, which says:

    The value of a string literal containing a multibyte
    character or escape sequence not represented in the
    execution character set is implementation-defined.

What this says to me is that a multibyte charater that /is/ in
the execution character set is represented (in a string literal)
in a defined way.  To be sure, what that is depends on some
values that are locale-specific, and may not be the encoding you
want.  But I think it has to work with the standard library
functions for dealing with multibyte characters, assuming of
course that the character is in the execution character set
(and presumably under the default locale).

Disclaimer:  I haven't researched this as thoroughly as is my
usual custom, and may very well have missed something.  My sense
though is that universal character names (\u and \U) will give
you what you need, if that is not too exotic or esoteric.

> What compiler did you use?

Two different versions of gcc, one x86, one x86-64.  The encoding
in both cases turned out to be UTF-8.

[toc] | [prev] | [next] | [standalone]

#73757

On 24.10.2015 23:43, Tim Rentsch wrote:
> Philipp Klaus Krause <pkk@spth.de> writes:
> 
>> On 24.10.2015 20:57, Tim Rentsch wrote:
>>> Philipp Klaus Krause <pkk@spth.de> writes:
>>>
>>>> So is it possible to create a string literal containing a multibyte
>>>> character of more than one byte when the implementation supports only
>>>> the basic source character set as source character set?
>>>
>>> Yes, it has been since C99:
>>>
>>>     #include <stdio.h>
>>>
>>>     int
>>>     main(){
>>>         char foo[] = "\u1234";
>>>         printf( "sizeof foo = %zu\n", sizeof foo );
>>>         return  0;
>>>     }
>>>
>>> Running this on my system prints 'sizeof foo = 4'.
>>
>> This looks like implementation-defined behaviour by verse 10 of
>> section 6.4.4.4 of the C11 standard to me.
> 
> Certainly I would agree that the value of a /character constant/
> with a multibyte character is implementation-defined, but your
> question is about string literals, not character constants.

But we also have C11 6.4.5 verse 4: "The same considerations apply to
each element of the sequence in a string literal as if it were in an
integer character constant […]"

Philipp

[toc] | [prev] | [next] | [standalone]

#73771

Philipp Klaus Krause <pkk@spth.de> writes:

> On 24.10.2015 23:43, Tim Rentsch wrote:
>> Philipp Klaus Krause <pkk@spth.de> writes:
>>
>>> On 24.10.2015 20:57, Tim Rentsch wrote:
>>>> Philipp Klaus Krause <pkk@spth.de> writes:
>>>>
>>>>> So is it possible to create a string literal containing a multibyte
>>>>> character of more than one byte when the implementation supports only
>>>>> the basic source character set as source character set?
>>>>
>>>> Yes, it has been since C99:
>>>>
>>>>     #include <stdio.h>
>>>>
>>>>     int
>>>>     main(){
>>>>         char foo[] = "\u1234";
>>>>         printf( "sizeof foo = %zu\n", sizeof foo );
>>>>         return  0;
>>>>     }
>>>>
>>>> Running this on my system prints 'sizeof foo = 4'.
>>>
>>> This looks like implementation-defined behaviour by verse 10 of
>>> section 6.4.4.4 of the C11 standard to me.
>>
>> Certainly I would agree that the value of a /character constant/
>> with a multibyte character is implementation-defined, but your
>> question is about string literals, not character constants.
>
> But we also have C11 6.4.5 verse 4:  "The same considerations
> apply to each element of the sequence in a string literal as if it
> were in an integer character constant [?]"

That's true, but notice an important detail:  "The same
considerations apply /to each element/ of the sequence in a
string literal as if /it/ were in an integer character
constant [my emphasis]".  The considerations being referred to
are just the element-level aspects of 6.4.4.4.  The statement
of implementation-defined behavior (in p10) pertains to the
character constant as a whole, not to individual elements;  it
is not included in the set of considerations referenced in
6.4.5 p4, just as the other "Semantics" aspects of 6.4.4.4 are
not applicable to 6.4.5 either.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web