Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.php > #17850 > unrolled thread

Twice UTF-8 byte-order mark (BOM)

Back to article view | Back to comp.lang.php

Contents

  Twice UTF-8 byte-order mark (BOM) vfhhutt45hghjjdewrghb@gmail.com - 2018-12-24 08:39 -0800
    Re: Twice UTF-8 byte-order mark (BOM) Tim Streater <timstreater@greenbee.net> - 2018-12-24 16:49 +0000
      Re: Twice UTF-8 byte-order mark (BOM) Luuk <luuk@invalid.lan> - 2018-12-24 19:07 +0100
        Re: Twice UTF-8 byte-order mark (BOM) "J.O. Aho" <user@example.net> - 2018-12-24 20:54 +0100
          Re: Twice UTF-8 byte-order mark (BOM) Luuk <luuk@invalid.lan> - 2018-12-24 21:15 +0100
        Re: Twice UTF-8 byte-order mark (BOM) Arno Welzel <usenet@arnowelzel.de> - 2018-12-26 09:13 +0100
          Re: Twice UTF-8 byte-order mark (BOM) Luuk <luuk@invalid.lan> - 2018-12-26 12:41 +0100
            Re: Twice UTF-8 byte-order mark (BOM) Arno Welzel <usenet@arnowelzel.de> - 2018-12-26 15:32 +0100
          Re: Twice UTF-8 byte-order mark (BOM) "J.O. Aho" <user@example.net> - 2018-12-26 15:41 +0100
            Re: Twice UTF-8 byte-order mark (BOM) vfhhutt45hghjjdewrghb@gmail.com - 2019-01-04 08:29 -0800
              Re: Twice UTF-8 byte-order mark (BOM) Richard Damon <Richard@Damon-Family.org> - 2019-01-04 21:08 -0500
              Re: Twice UTF-8 byte-order mark (BOM) "J.O. Aho" <user@example.net> - 2019-01-05 11:04 +0100
                Re: Twice UTF-8 byte-order mark (BOM) Richard Damon <Richard@Damon-Family.org> - 2019-01-05 11:26 -0500
                  Re: Twice UTF-8 byte-order mark (BOM) Tim Streater <timstreater@greenbee.net> - 2019-01-05 16:33 +0000
                    Re: Twice UTF-8 byte-order mark (BOM) Luuk <luuk@invalid.lan> - 2019-01-05 19:34 +0100
                      Re: Twice UTF-8 byte-order mark (BOM) Tim Streater <timstreater@greenbee.net> - 2019-01-05 19:08 +0000
                        Re: Twice UTF-8 byte-order mark (BOM) Luuk <luuk@invalid.lan> - 2019-01-06 20:25 +0100

#17850 — Twice UTF-8 byte-order mark (BOM)

I'm not sure if it is a bug, feature or what ever and if this is the correct group. Just for the archive:

In my script I create a file and offer it for downloading. But in the downloaded file was the  UTF-8 byte-order mark (BOM) twice at the beginning:
> EF BB BF EF BB BF ...

The code was:
<?php
include 'dbconfig.php';
//... no output etc.
header('Content-type: text/xml; charset=utf-8');		
header('Content-Disposition: attachment; filename="' . $titel . '.gpx"');
print "Hello";
exit();
?>

The dbconfig.php did not create any blanks etc. only:
<?php
$db_server = "xx";
$db_user = "xx";
?>

I figured out the problem: The dbconfig.php was saved as UTF-8 with 1 BOM at beginning. It seems that this BOM was added with include. After I delete the 3 first Bytes from the dbconfig.php everything works fine and only 1 BOM is in the downloaded file.

[toc] | [next] | [standalone]

#17851

In article <82a7cd23-82be-4611-a691-e0edcf8c30d6@googlegroups.com>,
<vfhhutt45hghjjdewrghb@gmail.com> wrote:

>I'm not sure if it is a bug, feature or what ever and if this is the correct
>group. Just for the archive:
>
>In my script I create a file and offer it for downloading. But in the
>downloaded file was the  UTF-8 byte-order mark (BOM) twice at the beginning:
>> EF BB BF EF BB BF ...
>
>The code was:
><?php
>include 'dbconfig.php';
>//... no output etc.
>header('Content-type: text/xml; charset=utf-8');      
>header('Content-Disposition: attachment; filename="' . $titel . '.gpx"');
>print "Hello";
>exit();
>?>
>
>The dbconfig.php did not create any blanks etc. only:
><?php
>$db_server = "xx";
>$db_user = "xx";
>?>
>
>I figured out the problem: The dbconfig.php was saved as UTF-8 with 1 BOM at
>beginning. It seems that this BOM was added with include. After I delete the 3
>first Bytes from the dbconfig.php everything works fine and only 1 BOM is in
>the downloaded file.

If it's UTF-8 you don't need *any* BOM in the file.

-- 
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf

[toc] | [prev] | [next] | [standalone]

#17852

On 24-12-2018 17:49, Tim Streater wrote:
> In article <82a7cd23-82be-4611-a691-e0edcf8c30d6@googlegroups.com>,
> <vfhhutt45hghjjdewrghb@gmail.com> wrote:
> 
>> I'm not sure if it is a bug, feature or what ever and if this is the 
>> correct
>> group. Just for the archive:
>>
>> In my script I create a file and offer it for downloading. But in the
>> downloaded file was the  UTF-8 byte-order mark (BOM) twice at the 
>> beginning:
>>> EF BB BF EF BB BF ...
>>
>> The code was:
>> <?php
>> include 'dbconfig.php';
>> //... no output etc.
>> header('Content-type: text/xml; charset=utf-8'); 
>> header('Content-Disposition: attachment; filename="' . $titel . '.gpx"');
>> print "Hello";
>> exit();
>> ?>
>>
>> The dbconfig.php did not create any blanks etc. only:
>> <?php
>> $db_server = "xx";
>> $db_user = "xx";
>> ?>
>>
>> I figured out the problem: The dbconfig.php was saved as UTF-8 with 1 
>> BOM at
>> beginning. It seems that this BOM was added with include. After I 
>> delete the 3
>> first Bytes from the dbconfig.php everything works fine and only 1 BOM 
>> is in
>> the downloaded file.
> 
> If it's UTF-8 you don't need *any* BOM in the file.
> 

Also specifying two-BOM's should not make the file invalid, unless 
there's some CRC code to check if its valid (probably only in executable 
files?)

Back to the subject:
I do not think this is a bug in PHP, but a 'bug' in your application.
Everythin PHP has done, is done accourindg to definition.

BTW: Then only thing PHP can do is add this to their docs (if its not 
already in the docs): YOU should make sure not any of your PHP-files 
starts with a BOM

[toc] | [prev] | [next] | [standalone]

#17853

On 12/24/18 7:07 PM, Luuk wrote:
> On 24-12-2018 17:49, Tim Streater wrote:
>> In article <82a7cd23-82be-4611-a691-e0edcf8c30d6@googlegroups.com>,
>> <vfhhutt45hghjjdewrghb@gmail.com> wrote:
>>
>>> I'm not sure if it is a bug, feature or what ever and if this is the
>>> correct
>>> group. Just for the archive:
>>>
>>> In my script I create a file and offer it for downloading. But in the
>>> downloaded file was the  UTF-8 byte-order mark (BOM) twice at the
>>> beginning:
>>>> EF BB BF EF BB BF ...
>>>
>>> The code was:
>>> <?php
>>> include 'dbconfig.php';
>>> //... no output etc.
>>> header('Content-type: text/xml; charset=utf-8');
>>> header('Content-Disposition: attachment; filename="' . $titel .
>>> '.gpx"');
>>> print "Hello";
>>> exit();
>>> ?>
>>>
>>> The dbconfig.php did not create any blanks etc. only:
>>> <?php
>>> $db_server = "xx";
>>> $db_user = "xx";
>>> ?>
>>>
>>> I figured out the problem: The dbconfig.php was saved as UTF-8 with 1
>>> BOM at
>>> beginning. It seems that this BOM was added with include. After I
>>> delete the 3
>>> first Bytes from the dbconfig.php everything works fine and only 1
>>> BOM is in
>>> the downloaded file.
>>
>> If it's UTF-8 you don't need *any* BOM in the file.
>>
> 
> Also specifying two-BOM's should not make the file invalid, unless
> there's some CRC code to check if its valid (probably only in executable
> files?)
> 
> Back to the subject:
> I do not think this is a bug in PHP, but a 'bug' in your application.
> Everythin PHP has done, is done accourindg to definition.
> 
> BTW: Then only thing PHP can do is add this to their docs (if its not
> already in the docs): YOU should make sure not any of your PHP-files
> starts with a BOM
> 
I would suggest that the BOM is the bug and shouldn't have been accepted
as part of the UTF-8 standard.

-- 

 //Aho

[toc] | [prev] | [next] | [standalone]

#17854

On 24-12-2018 20:54, J.O. Aho wrote:
> On 12/24/18 7:07 PM, Luuk wrote:
>> On 24-12-2018 17:49, Tim Streater wrote:
>>> In article <82a7cd23-82be-4611-a691-e0edcf8c30d6@googlegroups.com>,
>>> <vfhhutt45hghjjdewrghb@gmail.com> wrote:
>>>
>>>> I'm not sure if it is a bug, feature or what ever and if this is the
>>>> correct
>>>> group. Just for the archive:
>>>>
>>>> In my script I create a file and offer it for downloading. But in the
>>>> downloaded file was the  UTF-8 byte-order mark (BOM) twice at the
>>>> beginning:
>>>>> EF BB BF EF BB BF ...
>>>>
>>>> The code was:
>>>> <?php
>>>> include 'dbconfig.php';
>>>> //... no output etc.
>>>> header('Content-type: text/xml; charset=utf-8');
>>>> header('Content-Disposition: attachment; filename="' . $titel .
>>>> '.gpx"');
>>>> print "Hello";
>>>> exit();
>>>> ?>
>>>>
>>>> The dbconfig.php did not create any blanks etc. only:
>>>> <?php
>>>> $db_server = "xx";
>>>> $db_user = "xx";
>>>> ?>
>>>>
>>>> I figured out the problem: The dbconfig.php was saved as UTF-8 with 1
>>>> BOM at
>>>> beginning. It seems that this BOM was added with include. After I
>>>> delete the 3
>>>> first Bytes from the dbconfig.php everything works fine and only 1
>>>> BOM is in
>>>> the downloaded file.
>>>
>>> If it's UTF-8 you don't need *any* BOM in the file.
>>>
>>
>> Also specifying two-BOM's should not make the file invalid, unless
>> there's some CRC code to check if its valid (probably only in executable
>> files?)
>>
>> Back to the subject:
>> I do not think this is a bug in PHP, but a 'bug' in your application.
>> Everythin PHP has done, is done accourindg to definition.
>>
>> BTW: Then only thing PHP can do is add this to their docs (if its not
>> already in the docs): YOU should make sure not any of your PHP-files
>> starts with a BOM
>>
> I would suggest that the BOM is the bug and shouldn't have been accepted
> as part of the UTF-8 standard.
> 

But even than it's still not a bug in PHP,
just a nasty side-effect

[toc] | [prev] | [next] | [standalone]

#17855

Luuk:

[...]
> Back to the subject:
> I do not think this is a bug in PHP, but a 'bug' in your application.
> Everythin PHP has done, is done accourindg to definition.

I think it's a bug.

PHP should NOT output anything unless told so. If there are two files
which only contain "<?php" as the very first content, there should not
be any output at all. And even an echo statement should not output any
BOM, just because the file itself contains it.


-- 
Arno Welzel
https://arnowelzel.de

[toc] | [prev] | [next] | [standalone]

#17856

On 26-12-2018 09:13, Arno Welzel wrote:
> Luuk:
> 
> [...]
>> Back to the subject:
>> I do not think this is a bug in PHP, but a 'bug' in your application.
>> Everythin PHP has done, is done accourindg to definition.
> 
> I think it's a bug.
> 
> PHP should NOT output anything unless told so. If there are two files
> which only contain "<?php" as the very first content, there should not
> be any output at all. And even an echo statement should not output any
> BOM, just because the file itself contains it.
> 
> 

PHP is not outputing anything...

Its because you can combine HTML and PHP in one file like:
<?php
echo "2";
?>
+3
<?php
echo "=5";
?>

which outputs:
2+3
=5


If you create an empty php-file with these contents:
<?php

Than noting will be outputted, unless....
There's a BOM before the '<?php'


You can test this using this:
$ echo -e '\xef\xbb\xbf<?php' >bom.php
$ file bom.php
bom.php: PHP script, UTF-8 Unicode (with BOM) text
$ php bom.php | hexdump -C
00000000  ef bb bf                                          |...|
00000003

again, it's not PHP

also read: 'Potential issues with the UTF-8 BOM' on this page:
https://www.w3.org/International/questions/qa-byte-order-mark.en

wich states: "You should ensure that the included files do not start 
with a BOM."

[toc] | [prev] | [next] | [standalone]

#17857

Luuk:

[...]
> If you create an empty php-file with these contents:
> <?php
> 
> Than noting will be outputted, unless....
> There's a BOM before the '<?php'
[...]
> also read: 'Potential issues with the UTF-8 BOM' on this page:
> https://www.w3.org/International/questions/qa-byte-order-mark.en
> 
> wich states: "You should ensure that the included files do not start 
> with a BOM."

Thanks for the clarification.


-- 
Arno Welzel
https://arnowelzel.de

[toc] | [prev] | [next] | [standalone]

#17858

On 12/26/18 9:13 AM, Arno Welzel wrote:
> Luuk:
> 
> [...]
>> Back to the subject:
>> I do not think this is a bug in PHP, but a 'bug' in your application.
>> Everythin PHP has done, is done accourindg to definition.
> 
> I think it's a bug.
> 
> PHP should NOT output anything unless told so. If there are two files
> which only contain "<?php" as the very first content, there should not
> be any output at all. And even an echo statement should not output any
> BOM, just because the file itself contains it.

As long as it begins with <?php it will not output anything, but the BOM
is put before the <?php and so you will have output which is outside the
php engines control.

-- 

 //Aho

[toc] | [prev] | [next] | [standalone]

#17859

Am Mittwoch, 26. Dezember 2018 15:41:06 UTC+1 schrieb J.O. Aho:
> As long as it begins with <?php it will not output anything, but the BOM
> is put before the <?php and so you will have output which is outside the
> php engines control.
> 

I understand the modus operandi but I think it's not a good way. UTF-8 should be used as a norm. And the BOM is part of a UTF-8 file. In other cases (without BOM) it is a ISO 8859-1 file or anything else. 

I'm wondering that not everybody have my problems. To include the DB conneting data is a normal procedure for security reasons and for usability.

Maybe php should filter the BOM if "include" is used?!

[toc] | [prev] | [next] | [standalone]

#17860

On 1/4/19 11:29 AM, vfhhutt45hghjjdewrghb@gmail.com wrote:
> Am Mittwoch, 26. Dezember 2018 15:41:06 UTC+1 schrieb J.O. Aho:
>> As long as it begins with <?php it will not output anything, but the BOM
>> is put before the <?php and so you will have output which is outside the
>> php engines control.
>>
> 
> I understand the modus operandi but I think it's not a good way. UTF-8 should be used as a norm. And the BOM is part of a UTF-8 file. In other cases (without BOM) it is a ISO 8859-1 file or anything else. 
> 
> I'm wondering that not everybody have my problems. To include the DB conneting data is a normal procedure for security reasons and for usability.
> 
> Maybe php should filter the BOM if "include" is used?!
> 

The BOM is NOT 'part of a UTF-8 File' in the sense that it is needed to
say that the file is encoded in UTF-8. The Unicode Standard permits the
BOM at the beginning of the file, but does not encourage it.

PHP, by its definition, says that the contents of a file, outside the
<?php and ?> is immediately output to the output stream, so that
includes any whitespace like the BOM.

This is one reason some coding guidelines say to not place the closing
?> on the file if it is all PHP code so you can't have any invisible
white space (like a new line) after it that might get output before the
script gets done with the things that have to be done before anything is
output.

[toc] | [prev] | [next] | [standalone]

#17861

On 1/4/19 5:29 PM, vfhhutt45hghjjdewrghb@gmail.com wrote:
> Am Mittwoch, 26. Dezember 2018 15:41:06 UTC+1 schrieb J.O. Aho:
>> As long as it begins with <?php it will not output anything, but the BOM
>> is put before the <?php and so you will have output which is outside the
>> php engines control.
>>
> 
> I understand the modus operandi but I think it's not a good way. UTF-8 should be used as a norm. And the BOM is part of a UTF-8 file. In other cases (without BOM) it is a ISO 8859-1 file or anything else.

The BOM was just an addition for issues in some lesser operating 
systems, in newer versions like UTF-16 BOM has been removed from the 
standard.


> I'm wondering that not everybody have my problems. To include the DB conneting data is a normal procedure for security reasons and for usability.

Not everyone uses a lesser OS and have BOM enabled in text editors.


> Maybe php should filter the BOM if "include" is used?!

PHP do not know what your intention is with the file, say you are 
generating some data inside a UTF-8 BOM document, then you don't want it 
to be stripped.

Just do the sensible thing and say no to BOM as there are other ways to 
identify your file as UTF-8, and if you don't use UTF-8 characters in 
your text, then it's just plain ASCII and has no use of BOM in anyway.


-- 

  //Aho

[toc] | [prev] | [next] | [standalone]

#17862

On 1/5/19 5:04 AM, J.O. Aho wrote:
> On 1/4/19 5:29 PM, vfhhutt45hghjjdewrghb@gmail.com wrote:
>> Am Mittwoch, 26. Dezember 2018 15:41:06 UTC+1 schrieb J.O. Aho:
>>> As long as it begins with <?php it will not output anything, but the BOM
>>> is put before the <?php and so you will have output which is outside the
>>> php engines control.
>>>
>>
>> I understand the modus operandi but I think it's not a good way. UTF-8
>> should be used as a norm. And the BOM is part of a UTF-8 file. In
>> other cases (without BOM) it is a ISO 8859-1 file or anything else.
> 
> The BOM was just an addition for issues in some lesser operating
> systems, in newer versions like UTF-16 BOM has been removed from the
> standard.

To my knowledge, the Unicode Standard still defines the BOM, and it is
actually very important for formats like UTF-16 (but not needed for
UTF-16LE or UTF-16BE, where the endianness is provided by the encoding
definition).

[toc] | [prev] | [next] | [standalone]

#17863

In article <2V4YD.191363$lx7.100121@fx44.iad>, Richard Damon
<Richard@Damon-Family.org> wrote:

>On 1/5/19 5:04 AM, J.O. Aho wrote:
>> On 1/4/19 5:29 PM, vfhhutt45hghjjdewrghb@gmail.com wrote:
>>> Am Mittwoch, 26. Dezember 2018 15:41:06 UTC+1 schrieb J.O. Aho:
>>>> As long as it begins with <?php it will not output anything, but the BOM
>>>> is put before the <?php and so you will have output which is outside the
>>>> php engines control.
>>>>
>>>
>>> I understand the modus operandi but I think it's not a good way. UTF-8
>>> should be used as a norm. And the BOM is part of a UTF-8 file. In
>>> other cases (without BOM) it is a ISO 8859-1 file or anything else.
>> 
>> The BOM was just an addition for issues in some lesser operating
>> systems, in newer versions like UTF-16 BOM has been removed from the
>> standard.
>
>To my knowledge, the Unicode Standard still defines the BOM, and it is
>actually very important for formats like UTF-16 (but not needed for
>UTF-16LE or UTF-16BE, where the endianness is provided by the encoding
>definition).

Or UTF-8.

-- 
What power have you got?
Where did you get it from?
In whose interests do you use it?
To whom are you accountable?
How do we get rid of you?

Tony Benn

[toc] | [prev] | [next] | [standalone]

#17864

On 5-1-2019 17:33, Tim Streater wrote:
> In article <2V4YD.191363$lx7.100121@fx44.iad>, Richard Damon
> <Richard@Damon-Family.org> wrote:
> 
>> On 1/5/19 5:04 AM, J.O. Aho wrote:
>>> On 1/4/19 5:29 PM, vfhhutt45hghjjdewrghb@gmail.com wrote:
>>>> Am Mittwoch, 26. Dezember 2018 15:41:06 UTC+1 schrieb J.O. Aho:
>>>>> As long as it begins with <?php it will not output anything, but 
>>>>> the BOM
>>>>> is put before the <?php and so you will have output which is 
>>>>> outside the
>>>>> php engines control.
>>>>>
>>>>
>>>> I understand the modus operandi but I think it's not a good way. UTF-8
>>>> should be used as a norm. And the BOM is part of a UTF-8 file. In
>>>> other cases (without BOM) it is a ISO 8859-1 file or anything else.
>>>
>>> The BOM was just an addition for issues in some lesser operating
>>> systems, in newer versions like UTF-16 BOM has been removed from the
>>> standard.
>>
>> To my knowledge, the Unicode Standard still defines the BOM, and it is
>> actually very important for formats like UTF-16 (but not needed for
>> UTF-16LE or UTF-16BE, where the endianness is provided by the encoding
>> definition).
> 

Wikipedia is pretty clear about this:

Wikipedia:
In UTF-16, a BOM (U+FEFF) may be placed as the first character of a file 
or character stream to indicate the endianness (byte order) of all the 
16-bit code units of the file or stream.

> Or UTF-8.
> 


Wikipedia:
The Unicode Standard permits the BOM in UTF-8,[3] but does not require 
or recommend its use.[4] Byte order has no meaning in UTF-8,[5] so its 
only use in UTF-8 is to signal at the start that the text stream is 
encoded in UTF-8, or that it was converted to UTF-8 from a stream that 
contained an optional BOM. The standard also does not recommend removing 
a BOM when it is there, so that round-tripping between encodings does 
not lose information, and so that code that relies on it continues to 
work.[6][7] The IETF recommends that if a protocol either (a) always 
uses UTF-8, or (b) has some other way to indicate what encoding is being 
used, then it "SHOULD forbid use of U+FEFF as a signature."[8]


( https://en.wikipedia.org/wiki/Byte_order_mark )

[toc] | [prev] | [next] | [standalone]

#17865

In article <5c30f891$0$22360$e4fe514c@news.xs4all.nl>, Luuk
<luuk@invalid.lan> wrote:

>On 5-1-2019 17:33, Tim Streater wrote:
>> In article <2V4YD.191363$lx7.100121@fx44.iad>, Richard Damon
>> <Richard@Damon-Family.org> wrote:
>> 
>>> On 1/5/19 5:04 AM, J.O. Aho wrote:
>>>> On 1/4/19 5:29 PM, vfhhutt45hghjjdewrghb@gmail.com wrote:
>>>>> Am Mittwoch, 26. Dezember 2018 15:41:06 UTC+1 schrieb J.O. Aho:
>>>>>> As long as it begins with <?php it will not output anything, but 
>>>>>> the BOM
>>>>>> is put before the <?php and so you will have output which is 
>>>>>> outside the
>>>>>> php engines control.
>>>>>>
>>>>>
>>>>> I understand the modus operandi but I think it's not a good way. UTF-8
>>>>> should be used as a norm. And the BOM is part of a UTF-8 file. In
>>>>> other cases (without BOM) it is a ISO 8859-1 file or anything else.
>>>>
>>>> The BOM was just an addition for issues in some lesser operating
>>>> systems, in newer versions like UTF-16 BOM has been removed from the
>>>> standard.
>>>
>>> To my knowledge, the Unicode Standard still defines the BOM, and it is
>>> actually very important for formats like UTF-16 (but not needed for
>>> UTF-16LE or UTF-16BE, where the endianness is provided by the encoding
>>> definition).

>Wikipedia is pretty clear about this:
>
>Wikipedia:
>In UTF-16, a BOM (U+FEFF) may be placed as the first character of a file 
>or character stream to indicate the endianness (byte order) of all the 
>16-bit code units of the file or stream.

A good enough reason not to use UTF-16.

>> Or UTF-8.

>Wikipedia:
>The Unicode Standard permits the BOM in UTF-8,[3] but does not require 
>or recommend its use.[4] Byte order has no meaning in UTF-8,[5] so its 
>only use in UTF-8 is to signal at the start that the text stream is 
>encoded in UTF-8, or that it was converted to UTF-8 from a stream that 
>contained an optional BOM. The standard also does not recommend removing 
>a BOM when it is there, so that round-tripping between encodings does 
>not lose information, and so that code that relies on it continues to 
>work.[6][7] The IETF recommends that if a protocol either (a) always 
>uses UTF-8, or (b) has some other way to indicate what encoding is being 
>used, then it "SHOULD forbid use of U+FEFF as a signature."[8]

The best way to indicate the encoding is out-of-band, such as the
charset attribute of the Content-Type header.

Even if the BOM is there, the sender could be lying about the actual
content. The BOM does not rescue you from this problem, it just adds
another as this thread shows.

-- 
When I saw how the European Union was developing, it was very obvious what they
had in mind was not democratic. In Britain you vote for a government so the
government has to listen to you, and if you don't like it you can change it.

Tony Benn

[toc] | [prev] | [next] | [standalone]

#17866

On 5-1-2019 20:08, Tim Streater wrote:
> In article <5c30f891$0$22360$e4fe514c@news.xs4all.nl>, Luuk
> <luuk@invalid.lan> wrote:
> 
>> On 5-1-2019 17:33, Tim Streater wrote:
>>> In article <2V4YD.191363$lx7.100121@fx44.iad>, Richard Damon
>>> <Richard@Damon-Family.org> wrote:
>>>
>>>> On 1/5/19 5:04 AM, J.O. Aho wrote:
>>>>> On 1/4/19 5:29 PM, vfhhutt45hghjjdewrghb@gmail.com wrote:
>>>>>> Am Mittwoch, 26. Dezember 2018 15:41:06 UTC+1 schrieb J.O. Aho:
>>>>>>> As long as it begins with <?php it will not output anything, but 
>>>>>>> the BOM
>>>>>>> is put before the <?php and so you will have output which is 
>>>>>>> outside the
>>>>>>> php engines control.
>>>>>>>
>>>>>>
>>>>>> I understand the modus operandi but I think it's not a good way. 
>>>>>> UTF-8
>>>>>> should be used as a norm. And the BOM is part of a UTF-8 file. In
>>>>>> other cases (without BOM) it is a ISO 8859-1 file or anything else.
>>>>>
>>>>> The BOM was just an addition for issues in some lesser operating
>>>>> systems, in newer versions like UTF-16 BOM has been removed from the
>>>>> standard.
>>>>
>>>> To my knowledge, the Unicode Standard still defines the BOM, and it is
>>>> actually very important for formats like UTF-16 (but not needed for
>>>> UTF-16LE or UTF-16BE, where the endianness is provided by the encoding
>>>> definition).
> 
>> Wikipedia is pretty clear about this:
>>
>> Wikipedia:
>> In UTF-16, a BOM (U+FEFF) may be placed as the first character of a 
>> file or character stream to indicate the endianness (byte order) of 
>> all the 16-bit code units of the file or stream.
> 
> A good enough reason not to use UTF-16.

There is the word 'may' in that sentence, which indicates that the use 
of BOM is not mandatory.

There is software, i.e. Microsoft SQL Server, which stores date 
internally as UTF-16
( 
https://stackoverflow.com/questions/5182164/sql-server-default-character-encoding 
)

> 
>>> Or UTF-8.
> 
>> Wikipedia:
>> The Unicode Standard permits the BOM in UTF-8,[3] but does not require 
>> or recommend its use.[4] Byte order has no meaning in UTF-8,[5] so its 
>> only use in UTF-8 is to signal at the start that the text stream is 
>> encoded in UTF-8, or that it was converted to UTF-8 from a stream that 
>> contained an optional BOM. The standard also does not recommend 
>> removing a BOM when it is there, so that round-tripping between 
>> encodings does not lose information, and so that code that relies on 
>> it continues to work.[6][7] The IETF recommends that if a protocol 
>> either (a) always uses UTF-8, or (b) has some other way to indicate 
>> what encoding is being used, then it "SHOULD forbid use of U+FEFF as a 
>> signature."[8]
> 
> The best way to indicate the encoding is out-of-band, such as the
> charset attribute of the Content-Type header.
> 
> Even if the BOM is there, the sender could be lying about the actual
> content. The BOM does not rescue you from this problem, it just adds
> another as this thread shows.
> 

The last part seems to be, or is (!), the important part:
.. if a protocol either (a) always uses UTF-8, or (b) has some other way 
to indicate what encoding is being used, then it "SHOULD forbid use of 
U+FEFF as a signature."[8]

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.php

csiph-web