Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #26069

Re: catch UnicodeDecodeError

Show all headers | View raw

On Jul 25, 8:50 pm, Dave Angel <d...@davea.name> wrote:
> On 07/25/2012 08:09 AM, jaroslav.dob...@gmail.com wrote:
>
>
>
>
>
>
>
>
>
> > On Wednesday, July 25, 2012 1:35:09 PM UTC+2, Philipp Hagemeister wrote:
> >> Hi Jaroslav,
>
> >> you can catch a UnicodeDecodeError just like any other exception. Can
> >> you provide a full example program that shows your problem?
>
> >> This works fine on my system:
>
> >> import sys
> >> open(&#39;tmp&#39;, &#39;wb&#39;).write(b&#39;\xff\xff&#39;)
> >> try:
> >>     buf = open(&#39;tmp&#39;, &#39;rb&#39;).read()
> >>     buf.decode(&#39;utf-8&#39;)
> >> except UnicodeDecodeError as ude:
> >>     sys.exit(&quot;Found a bad char in file &quot; + &quot;tmp&quot;)
>
> > Thank you. I got it. What I need to do is explicitly decode text.
>
> > But I think trial and error with moving files around will in most cases be faster. Usually, such a problem occurs with some (usually complex) program that I wrote quite a long time ago. I don't like editing old and complex programs that work under all normal circumstances.
>
> > What I am missing (especially for Python3) is something like:
>
> > try:
> >     for line in sys.stdin:
> > except UnicodeDecodeError:
> >     sys.exit("Encoding problem in line " + str(line_number))
>
> > I got the point that there is no such thing as encoding-independent lines. But if no line ending can be found, then the file simply has one single line.
>
> i can't understand your question.  if the problem is that the system
> doesn't magically produce a variable called line_number, then generate
> it yourself, by counting
> in the loop.


That was just a very incomplete and general example.

My problem is solved. What I need to do is explicitly decode text when
reading it. Then I can catch exceptions. I might do this in future
programs.

I dislike about this solution that it complicates most programs
unnecessarily. In programs that open, read and process many files I
don't want to explicitly decode and encode characters all the time. I
just want to write:

for line in f:

or something like that. Yet, writing this means to *implicitly* decode
text. And, because the decoding is implicit, you cannot say

try:
    for line in f: # here text is decoded implicitly
       do_something()
except UnicodeDecodeError():
    do_something_different()

This isn't possible for syntactic reasons.

The problem is that vast majority of the thousands of files that I
process are correctly encoded. But then, suddenly, there is a bad
character in a new file. (This is so because most files today are
generated by people who don't know that there is such a thing as
encodings.) And then I need to rewrite my very complex program just
because of one single character in one single file.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

catch UnicodeDecodeError jaroslav.dobrek@gmail.com - 2012-07-25 04:05 -0700
  Re: catch UnicodeDecodeError Andrew Berg <bahamutzero8825@gmail.com> - 2012-07-25 06:34 -0500
  Re: catch UnicodeDecodeError Philipp Hagemeister <phihag@phihag.de> - 2012-07-25 13:35 +0200
    Re: catch UnicodeDecodeError jaroslav.dobrek@gmail.com - 2012-07-25 05:09 -0700
    Re: catch UnicodeDecodeError jaroslav.dobrek@gmail.com - 2012-07-25 05:09 -0700
      Re: catch UnicodeDecodeError Dave Angel <d@davea.name> - 2012-07-25 14:50 -0400
        Re: catch UnicodeDecodeError Jaroslav Dobrek <jaroslav.dobrek@gmail.com> - 2012-07-26 00:46 -0700
          Re: catch UnicodeDecodeError Stefan Behnel <stefan_ml@behnel.de> - 2012-07-26 10:28 +0200
            Re: catch UnicodeDecodeError Jaroslav Dobrek <jaroslav.dobrek@gmail.com> - 2012-07-26 03:51 -0700
              Re: catch UnicodeDecodeError Stefan Behnel <stefan_ml@behnel.de> - 2012-07-26 13:15 +0200
                Re: catch UnicodeDecodeError jaroslav.dobrek@gmail.com - 2012-07-26 04:58 -0700
                Re: catch UnicodeDecodeError jaroslav.dobrek@gmail.com - 2012-07-26 04:58 -0700
              Re: catch UnicodeDecodeError Philipp Hagemeister <phihag@phihag.de> - 2012-07-26 14:17 +0200
              Re: catch UnicodeDecodeError Stefan Behnel <stefan_ml@behnel.de> - 2012-07-26 14:24 +0200
          Re: catch UnicodeDecodeError Chris Angelico <rosuav@gmail.com> - 2012-07-26 19:46 +1000
          Re: catch UnicodeDecodeError wxjmfauth@gmail.com - 2012-07-26 03:19 -0700
      Re: catch UnicodeDecodeError Philipp Hagemeister <phihag@phihag.de> - 2012-07-26 14:43 +0200

csiph-web