Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90158

Re: Stripping unencodable characters from a string

Show key headers only | View raw

On 05.05.15 21:19, Paul Moore wrote:
> I want to write a string to an already-open file (sys.stdout, typically). However, I *don't* want encoding errors, and the string could be arbitrary Unicode (in theory). The best way I've found is
>
>      data = data.encode(file.encoding, errors='replace').decode(file.encoding)
>      file.write(data)
>
> (I'd probably use backslashreplace rather than replace, but that's a minor point).
>
> Is that the best way? The multiple re-encoding dance seems a bit clumsy, but it was the best I could think of.

There are flaws in this approach.

1) file.encoding can be None (StringIO) or absent (general file-like 
object, that implements only write()).

2) When the encoding is UTF-16, UTF-32, UTF-8-SIG, the output will 
contain superfluous byte order marks.

This is not easy problem and there is no simple solution. In particular 
cases you can create TextIOWrapper(file.buffer, 'w', 
encoding=file.encoding, errors='replace', newline=file.newlines, 
write_through=True) and write to it, but be aware of limitations.

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Stripping unencodable characters from a string Paul  Moore <p.f.moore@gmail.com> - 2015-05-05 11:19 -0700
  Re: Stripping unencodable characters from a string Dave Angel <davea@davea.name> - 2015-05-05 15:00 -0400
    Re: Stripping unencodable characters from a string Paul  Moore <p.f.moore@gmail.com> - 2015-05-05 12:24 -0700
      Re: Stripping unencodable characters from a string Marko Rauhamaa <marko@pacujo.net> - 2015-05-05 22:55 +0300
  Re: Stripping unencodable characters from a string Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-05-05 19:33 +0000
  Re: Stripping unencodable characters from a string Chris Angelico <rosuav@gmail.com> - 2015-05-06 10:02 +1000
  Re: Stripping unencodable characters from a string Serhiy Storchaka <storchaka@gmail.com> - 2015-05-08 15:28 +0300

csiph-web