Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Date: Tue, 07 Jan 2014 18:22:28 +0000
From: MRAB <python@mrabarnett.plus.com>
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0
MIME-Version: 1.0
To: python-list@python.org
CC: python-list@python.org
Subject: Re: [Python-ideas] RFC: bytestring as a str representation [was: a new bytestring type?]
References: <52C9B39F.6060205@stoneleaf.us> <CADbA=FXFah=pJq1jHME5P6E2_YV8Xr_i-YHavXEXg+38LYYdFg@mail.gmail.com> <87fvp1idip.fsf@uwakimon.sk.tsukuba.ac.jp> <CADbA=FUjeHzF46-fsBQxRNKzXF+pzTxyF-LHpueVmP8FVZVsqQ@mail.gmail.com> <1389009153.31778.YahooMailNeo@web181001.mail.ne1.yahoo.com> <87bnzphs7j.fsf@uwakimon.sk.tsukuba.ac.jp> <20140107154401.GK29356@ando> <FAE8F24A-C18A-4CC7-BE9D-06804BC99A3A@yahoo.com>
In-Reply-To: <FAE8F24A-C18A-4CC7-BE9D-06804BC99A3A@yahoo.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.5147.1389119135.18130.python-list@python.org>
Lines: 90
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:63445

On 2014-01-07 17:46, Andrew Barnert wrote:
 > I think Stephen's name "7-bit" is confusing people. If you try to
 > interpret the name sensibly, you get Steven's broken interpretation.
 > But if you read it as a nonsense word and work through the logic, it
 > all makes sense.
 >
 > On Jan 7, 2014, at 7:44, Steven D'Aprano <steve@pearwood.info> wrote:
 >
I was thinking about Ethan's suggestion of introducing a new bytestring
class and a lot of these suggestions are what I thought the bytestring
class could do.

[snip]
 >>
 >> Suppose we take a pure-ASCII byte-string and decode it:
 >>
 >>    b'abcd'.decode('ascii-compatible')
 >>
That would be:

     bytestring(b'abcd')

or even:

     bytestring('abcd')

[snip]
 >
 >> Suppose we take a byte-string with a non-ASCII byte:
 >>
 >>    b'abc\xFF'.decode('ascii-compatible')
 >>
That would be:

     bytestring(b'abc\xFF')

Bytes outside the ASCII range would be mapped to Unicode low
surrogates:

     bytestring(b'abc\xFF') == bytestring('abc\uDCFF')

[snip]
 >> Presumably they will compare equal, yes?
 >
 > I would hope not. One of them has the Unicode character U+FF, the
 > other has smuggled byte 0xFF, so they'd better not compare equal.
 >
 > However, the latter should compare equal to 'abc\uDCFF'. That's the
 > entire key here: the new representation is nothing but a more compact
 > way to represent strings that contain nothing but ASCII and surrogate
 > escapes.
 >
[snip]
 >>
 >> A concrete example:
 >>
 >>    s = b'abcd'.decode('ascii-compatible')
 >>    t = 'x'  # ASCII-compatible
 >>    s + t
 >>    => returns 'abcdx', with the "7-bit repr" flag cleared.

     s = bytestring(b'abcd')
     t = 'x'  # ASCII-compatible
     s + t
     => returns 'abcdx'
 >
 > Right. Here both s and t are normal 8-bit strings reprs in the first
 > place, so the new logic doesn't even get invoked. So yes, that's what
 > it returns.
 >
 >>    s = b'abcd'.decode('ascii-compatible')
 >>    t = 'ÿ'  # U+00FF, non-ASCII.
 >>
 >>    s + t
 >>    => returns 'abcd\uDCFF', with the "7-bit repr" flag set

     s = bytestring(b'abcd')
     t = 'ÿ'  # U+00FF, non-ASCII.
     s + t
     => returns 'abcd\xFF'

[snip]
There were also some other equivalences I was considering:

     bytestring(b'abc') == b'abc'
     bytestring(b'abc') == 'abc'

The problem there is that it wouldn't be transitive because:

     b'abc' != 'abc'