Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!goblin3!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Peter Otten <__peter__@web.de>
Subject: Re: Coding challenge: Optimise a custom string encoding
Date: Tue, 19 Aug 2014 01:35:08 +0200
Organization: None
References: <6e869040-98e9-437b-b024-4ffe7abc3054@googlegroups.com> <mailman.13113.1408393206.18130.python-list@python.org> <ca7d388f-fd15-47bb-a500-b3aa10b707c6@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8Bit
User-Agent: KNode/4.13.2
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.13122.1408404927.18130.python-list@python.org>
Lines: 85
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:76524

Alex Willmer wrote:

> On Monday, 18 August 2014 21:16:26 UTC+1, Terry Reedy  wrote:
>> On 8/18/2014 3:16 PM, Alex Willmer wrote:
>> > A challenge, just for fun. Can you speed up this function?
>> 
>> You should give a specification here, with examples. You should perhaps
> 
> Sorry, the (informal) spec was further down.
> 
>> > a custom encoding to store unicode usernames in a config file that only
>> > allowed mixed case ascii, digits, underscore, dash, at-sign and plus
>> > sign. We also wanted to keeping the encoded usernames somewhat human
>> > readable.
> 
>> > My design was utf-8 and a variant of %-escaping, using the plus symbol.
>> > So u'alic EURO 123' would be encoded as b'alic+e2+82+ac123'.
> 
> Other examples:
>>>> plus_encode(u'alice')
> 'alice'
>>>> plus_encode(u'Bacon & eggs only $19.95')
> 'Bacon+20+26+20eggs+20only+20+2419+2e95'
>>>> plus_encode(u'üｎïｃｏԁë')
> '+c3+bc+ef+bd+8e+c3+af+ef+bd+83+ef+bd+8f+d4+81+c3+ab'
> 
>> You should perhaps be using .maketrans and .translate.
> 
> That wouldn't work, maketrans() can only map single bytes to other single
> bytes. To encode 256 possible source bytes with 66 possible symbols
> requires a multi-symbol expansion of some or all source bytes.

You can do the translation in unicode, but you have to cope with a big 
translation table and the speed-up doesn't seem to be worthwhile:

$ cat plus_encode.py
# -*- coding: utf-8 -*-
import string

charset = set(string.ascii_letters + string.digits + '@_-')
byteseq = [chr(i) for i in xrange(256)]
bytemap = {byte: byte if byte in charset else '+' + byte.encode('hex')
           for byte in byteseq}

def plus_encode(s):
    """Encode a unicode string with only ascii letters, digits, _, -, @, +
    """
    bytemap_ = bytemap
    s_utf8 = s.encode('utf-8')
    return ''.join([bytemap[byte] for byte in s_utf8])


import sys
from itertools import imap as map

MAXUNICODE = 9000 #should be sys.maxunicode
ucharset = set(c.decode("ascii") for c in charset)
xmap = [u if u in ucharset else 
        u"".join("+" + c.encode("hex") for c in u.encode("utf-8"))
        for u in map(unichr, xrange(MAXUNICODE))]

def plus_encode2(s):
    return s.translate(xmap).encode("ascii")

if __name__ == "__main__":
    sample = u"".join(map(unichr, range(MAXUNICODE))) + u"€"
    assert plus_encode(sample) == plus_encode2(sample)
$ python plus_encode.py
$ python -m timeit -s 'import plus_encode' 'plus_encode.plus_encode(u"""qwertyuiop1234567890!"£$%^&*()\n{EURO-SIGN}""")'
100000 loops, best of 3: 10.6 usec per loop
$ python -m timeit -s 'import plus_encode' 'plus_encode.plus_encode2(u"""qwertyuiop1234567890!"£$%^&*()\n{EURO-SIGN}""")'
100000 loops, best of 3: 3.74 usec per loop

A smaller table is possible, but costs time:

ymap = [u if u in ucharset else
        u"+" + u.encode("latin1").encode("hex").decode("ascii")
        for u in map(unichr, xrange(256))]
def plus_encode3(s):
    return s.encode("utf-8").decode("latin1").translate(ymap).encode("ascii")

$ python -m timeit -s 'import plus_encode' 'plus_encode.plus_encode3(u"""qwertyuiop1234567890!"£$%^&*()\n{EURO-SIGN}""")'
100000 loops, best of 3: 5.91 usec per loop