Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #8193

Re: Unicode codepoints

Followups directed to: comp.lang.python

Show all headers | View raw

Saul Spatz wrote:

> Hi,
> 
> I'm just starting to learn a bit about Unicode. I want to be able to read
> a utf-8 encoded file, and print out the codepoints it encodes.  After many
> false starts, here's a script that seems to work, but it strikes me as
> awfully awkward and unpythonic.  Have you a better way?
> 
> def codePoints(s):
>     ''' return a list of the Unicode codepoints in the string s '''
>     answer = []
>     skip = False
>     for k, c in enumerate(s):
>         if skip:
>             skip = False
>             answer.append(ord(s[k-1:k+1]))
>             continue
>         if not 0xd800 <= ord(c) <= 0xdfff:
>             answer.append(ord(c))
>         else:
>             skip = True
>     return answer
>             
> if __name__ == '__main__':
>     s = open('test.txt', encoding = 'utf8', errors = 'replace').read()
>     code = codePoints(s)
>     for c in code:
>         print('U+'+hex(c)[2:])
> 
> Thanks for any help you can give me.
> 
> Saul

Here's an alternative implementation that follows Chris' suggestion to use a 
generator:

def codepoints(s):
    s = iter(s)
    for c in s:
        if 0xd800 <= ord(c) <= 0xdfff:
            c += next(s, "")
        yield ord(c)

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Unicode codepoints Saul Spatz <saul.spatz@gmail.com> - 2011-06-21 20:37 -0700
  Re: Unicode codepoints Chris Angelico <rosuav@gmail.com> - 2011-06-22 14:00 +1000
  Re: Unicode codepoints Vlastimil Brom <vlastimil.brom@gmail.com> - 2011-06-22 10:42 +0200
  Re: Unicode codepoints Peter Otten <__peter__@web.de> - 2011-06-22 11:00 +0200
    Re: Unicode codepoints jmfauth <wxjmfauth@gmail.com> - 2011-06-22 03:00 -0700

csiph-web