Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #13234

Re: How do I automate the removal of all non-ascii characters from my code?

Show key headers only | View raw

Hmm, nothing mentioned so far works for me...

Here's a very small test case:

>>> python -u "Convert to Creole.py"
  File "Convert to Creole.py", line 1
SyntaxError: Non-ASCII character '\xe2' in file Convert to Creole.py
on line 1, but no encoding declared; see
http://www.python.org/peps/pep-0263.html for details
>>> Exit Code: 1

Line 1: a=u'''≤'''.encode("ascii", "ignore").decode("ascii")

On Tue, Sep 13, 2011 at 11:33 PM, Vlastimil Brom
<vlastimil.brom@gmail.com> wrote:
> 2011/9/13 ron <vacorama@gmail.com>:
>>
>> Depending on the load, you can do something like:
>>
>> "".join([x for x in string if ord(x) < 128])
>>
>> It's worked great for me in cleaning input on webapps where there's a
>> lot of copy/paste from varied sources.
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>>
> Well, for this kind of dirty "data cleaning" you may as well use e.g.
>
>>>> u"äteöxt ÛÜÝ wiÉÊËÌthÞßà áânoûüýþn ASɔɕɖCɗɘəɚɛIɗɘəɚɛIεζ iηθιn жзbetийклweeჟრსn .ტუ..ფ".encode("ascii", "ignore").decode("ascii")
> u'text  with non ASCII in between ...'
>>>>
>
> vbr
> --
> http://mail.python.org/mailman/listinfo/python-list
>

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: How do I automate the removal of all non-ascii characters from my code? Stefan Behnel <stefan_ml@behnel.de> - 2011-09-12 10:43 +0200
  Re: How do I automate the removal of all non-ascii characters from my code? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-09-12 18:49 +1000
    Re: How do I automate the removal of all non-ascii characters from my code? Dave Angel <davea@ieee.org> - 2011-09-12 08:09 -0400
    Re: How do I automate the removal of all non-ascii characters from my code? jmfauth <wxjmfauth@gmail.com> - 2011-09-12 07:47 -0700
      Re: How do I automate the removal of all non-ascii characters from my code? "Rhodri James" <rhodri@wildebst.demon.co.uk> - 2011-09-12 22:39 +0100
        Re: How do I automate the removal of all non-ascii characters from my code? jmfauth <wxjmfauth@gmail.com> - 2011-09-13 00:49 -0700
          Re: How do I automate the removal of all non-ascii characters from my code? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-09-13 18:15 +1000
            Re: How do I automate the removal of all non-ascii characters from my code? jmfauth <wxjmfauth@gmail.com> - 2011-09-13 02:04 -0700
    Re: How do I automate the removal of all non-ascii characters from my code? ron <vacorama@gmail.com> - 2011-09-13 05:31 -0700
      Re: How do I automate the removal of all non-ascii characters from my code? Vlastimil Brom <vlastimil.brom@gmail.com> - 2011-09-13 15:33 +0200
      Re: How do I automate the removal of all non-ascii characters from my code? Alec Taylor <alec.taylor6@gmail.com> - 2011-09-14 01:02 +1000
        Re: How do I automate the removal of all non-ascii characters from my code? Jussi Piitulainen <jpiitula@ling.helsinki.fi> - 2011-09-13 18:29 +0300
      Re: How do I automate the removal of all non-ascii characters from my code? Vlastimil Brom <vlastimil.brom@gmail.com> - 2011-09-13 20:13 +0200

csiph-web