Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!ecngs!feeder2.ecngs.de!217.188.199.168.MISMATCH!takemy.news.telefonica.de!telefonica.de!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <5578f1be$0$12979$c3e8da3$5496439d@news.astraweb.com>
References: <20150610082812.2ce887c3@bigbox.christie.dr> <mailman.344.1433946513.13271.python-list@python.org> <55786fd5$0$13003$c3e8da3$5496439d@news.astraweb.com> <mailman.370.1433981374.13271.python-list@python.org> <5578f1be$0$12979$c3e8da3$5496439d@news.astraweb.com>
Date: Thu, 11 Jun 2015 13:05:35 +1000
Subject: Re: Python NBSP DWIM
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.374.1433991937.13271.python-list@python.org>
Lines: 55
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:92443

On Thu, Jun 11, 2015 at 12:26 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> No, despite the name, that is not a space character, it is a formatting
> character. Due to Unicode's stability policy, the name is stuck forever,
> but it should not be treated as a space character:
>
> py> unicodedata.category(' ')
> 'Zs'
> py> unicodedata.category('\u00A0')  # NBSP
> 'Zs'
> py> unicodedata.category('\uFEFF')  # ZWNBSP
> 'Cf'
>
>
> Ideally, outside of the BOM, you should never come across a ZWNBSP. You
> should use U+2060 WORD JOINER instead. But if you do come across one
> outside of the BOM, it should be treated as a legitimate non-space
> character:
>
> http://www.unicode.org/faq/utf_bom.html#bom6
>
> Although ZWNBSP is a "default ignorable" code point, I believe that the font
> is well within its rights to show it with a visible glyph:
>
>     "Fonts can contain glyphs intended for visible display of
>     default ignorable code points that would otherwise be
>     rendered invisibly when not supported."
>
> http://www.unicode.org/faq/unsup_char.html

Huh. Okay, my bad. I was under the impression that it was supposed to
take up no width, as the name implies, but stability trumps logic
sometimes. Learn something new every day.

>> notable because it's also used as
>> the byte-order mark (as its counterpart, U+FFFE, is unallocated). I've
>> been fighting with VLC Media Player over the font it uses for subtitles;
>> for some bizarre reason, that font represents U+FEFF not with zero pixels
>> of emptiness, but with a box containing the letters "ZWN" "BSP" on two
>> lines. Yeah, because that totally takes up zero width and looks like blank
>> space.
>
> Why do the subtitles contain ZWNBSP in the first place? Surely they're not
> English subtitles?

No, they're not :) The character comes up in the Cantonese and
Japanese subs for Once Upon A December.

http://youtu.be/CEpcUeWP0bg
http://youtu.be/WFZAaHrHens

Possibly some others in the series as well. It may well be a fault in
the subtitles, but most programs I've seen don't show U+FEFF as a big
fat box.

ChrisA