Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #8191

Re: Unicode codepoints

Show all headers | View raw

2011/6/22 Saul Spatz <saul.spatz@gmail.com>:
> Hi,
>
> I'm just starting to learn a bit about Unicode. I want to be able to read a utf-8 encoded file, and print out the codepoints it encodes.  After many false starts, here's a script that seems to work, but it strikes me as awfully awkward and unpythonic.  Have you a better way?
>
> def codePoints(s):
>    ''' return a list of the Unicode codepoints in the string s '''
>    answer = []
>    skip = False
>    for k, c in enumerate(s):
>        if skip:
>            skip = False
>            answer.append(ord(s[k-1:k+1]))
>            continue
>        if not 0xd800 <= ord(c) <= 0xdfff:
>            answer.append(ord(c))
>        else:
>            skip = True
>    return answer
>
> if __name__ == '__main__':
>    s = open('test.txt', encoding = 'utf8', errors = 'replace').read()
>    code = codePoints(s)
>    for c in code:
>        print('U+'+hex(c)[2:])
>
> Thanks for any help you can give me.
>
> Saul
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>

Hi,
what functionality should codePoints(...) add over just iterating
through the characters in the unicode string directly (besides
filtering out the surrogates)?

It seems, that you can just use

    s = open(r'C:\install\filter-utf-8.txt', encoding = 'utf8', errors
= 'replace').read()
    for c in s:
        print('U+'+hex(ord(c))[2:])

or eventually add the condition before the print:
    if not 0xd800 <= ord(c) <= 0xdfff:

you can also use string formatting to do the hex conversion and a more
usual zero padding; the print(...) calls would be:

"older style formatting"
        print("U+%04x"%(ord(c),))

or the newer, potentially more powerful way using format(...)
        print("U+{:04x}".format(ord(c)))

hth,
   vbr

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Unicode codepoints Saul Spatz <saul.spatz@gmail.com> - 2011-06-21 20:37 -0700
  Re: Unicode codepoints Chris Angelico <rosuav@gmail.com> - 2011-06-22 14:00 +1000
  Re: Unicode codepoints Vlastimil Brom <vlastimil.brom@gmail.com> - 2011-06-22 10:42 +0200
  Re: Unicode codepoints Peter Otten <__peter__@web.de> - 2011-06-22 11:00 +0200
    Re: Unicode codepoints jmfauth <wxjmfauth@gmail.com> - 2011-06-22 03:00 -0700

csiph-web