Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #88948 > unrolled thread

using DictReader() with .decode('utf-8', 'ignore')

Back to article view | Back to comp.lang.python

Contents

  using DictReader() with .decode('utf-8', 'ignore') Vincent Davis <vincent@vincentdavis.net> - 2015-04-14 06:54 -0600
    Re: using DictReader() with .decode('utf-8', 'ignore') Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-04-14 23:23 +1000
      Re: using DictReader() with .decode('utf-8', 'ignore') Vincent Davis <vincent@vincentdavis.net> - 2015-04-14 07:37 -0600
        Re: using DictReader() with .decode('utf-8', 'ignore') Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-04-14 23:48 +1000
          Re: using DictReader() with .decode('utf-8', 'ignore') Vincent Davis <vincent@vincentdavis.net> - 2015-04-14 08:51 -0600

#88948 — using DictReader() with .decode('utf-8', 'ignore')

[Multipart message — attachments visible in raw view] — view raw

I had been reading in a file like so. (python 3)
with open(dfile, 'rb') as f:
    for line in f:

​line
 = line.decode('utf-8', 'ignore').split(',')

​How can I ​do accomplish decode('utf-8', 'ignore') when reading with
 DictReader()


Vincent Davis
720-301-3003

[toc] | [next] | [standalone]

#88953

On Tue, 14 Apr 2015 10:54 pm, Vincent Davis wrote:

> I had been reading in a file like so. (python 3)
> with open(dfile, 'rb') as f:
>     for line in f:
> 
> line
>  = line.decode('utf-8', 'ignore').split(',')
> 
> How can I ​do accomplish decode('utf-8', 'ignore') when reading with
>  DictReader()


Which DictReader? Do you mean the one in the csv module? I will assume so.

I haven't tried it, but I think something like this will work:


# untested
with open(dfile, 'r', encoding='utf-8', errors='ignore', newline='') as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row['fieldname'])



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#88956

[Multipart message — attachments visible in raw view] — view raw

> Which DictReader? Do you mean the one in the csv module? I will assume so.
>
​yes.​


>
> # untested
> with open(dfile, 'r', encoding='utf-8', errors='ignore', newline='') as f:
>     reader = csv.DictReader(f)
>     for row in reader:
>         print(row['fieldname'])
>

What you have seems to work, now I need to go find my strange symbols that
are not ​'utf-8' and see what happens
I was thought, that I had to open with 'rb' to use ​encoding?


Vincent Davis

[toc] | [prev] | [next] | [standalone]

#88959

On Tue, 14 Apr 2015 11:37 pm, Vincent Davis wrote:

>> Which DictReader? Do you mean the one in the csv module? I will assume
>> so.
>>
> yes.
> 
> 
>>
>> # untested
>> with open(dfile, 'r', encoding='utf-8', errors='ignore', newline='') as
>> f:
>>     reader = csv.DictReader(f)
>>     for row in reader:
>>         print(row['fieldname'])
>>
> 
> What you have seems to work, now I need to go find my strange symbols that
> are not ​'utf-8' and see what happens
> I was thought, that I had to open with 'rb' to use ​encoding?

No, in Python 3 the rules are:

'rb' reads in binary mode, returns raw bytes without doing any decoding;

'r' reads in text mode, returns Unicode text, using the codec/encoding
specified. By default, if no encoding is specified, I think UTF-8 is used,
but it may depend on the platform.


If you are getting decoding errors when reading the file, it is possible
that the file isn't actually UTF-8. One test you can do:

with open(dfile, 'rb') as f:
    for line in f:
        try:
            s = line.decode('utf-8', 'strict')
        except UnicodeDecodeError as err:
            print(err)

If you need help deciphering the errors, please copy and paste them here and
we'll see what we can do.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#88966

[Multipart message — attachments visible in raw view] — view raw

On Tue, Apr 14, 2015 at 7:48 AM, Steven D'Aprano <
steve+comp.lang.python@pearwood.info> wrote:

> with open(dfile, 'rb') as f:
>     for line in f:
>         try:
>             s = line.decode('utf-8', 'strict')
>         except UnicodeDecodeError as err:
>             print(err)
>
> If you need help deciphering the errors, please copy and paste them here
> and
> we'll see what we can do.


Below are the errors. I knew about these and I think the correct encoding
is windows-1252. I will paste some code and output at the end of this email
that prints the offending column in the line. These are very likely errors,
and so I what to remove them. I am reading this csv into django sqlite3 db.
What is strange to me is that using
​"​
with open(dfile, 'r', encoding='utf-8', errors='ignore', newline='')
​"​
 does not seem to remove these
​, it seems to correctly save them to the db which I don't understand.​
​


'utf-8' codec can't decode byte 0xa6 in position 368: invalid start byte
'utf-8' codec can't decode byte 0xac in position 223: invalid start byte
'utf-8' codec can't decode byte 0xa6 in position 1203: invalid start byte
'utf-8' codec can't decode byte 0xa2 in position 44: invalid start byte
'utf-8' codec can't decode byte 0xac in position 396: invalid start byte

import chardet
with open("DATA/ATSDTA_ATSP600.csv", 'rb') as f:
    for line in f:
        code = chardet.detect(line)
        #if code == {'confidence': 0.5, 'encoding': 'windows-1252'}:
        if code != {'encoding': 'ascii', 'confidence': 1.0}:
            print(code)
        win = line.decode('windows-1252').split(',') #windows-1252
        norm = line.decode('utf-8', 'ignore').split(',')
        ascii = line.decode('ascii', "ignore").split(',')
        ascii2 = line.decode('ISO-8859-1').split(',')

        for w, n, a, a2 in zip(win, norm, ascii, ascii2):
            if w != n:
                print(w
​)
​             print(
n
​)
​
a, a2)
                print(win[0])

​## Output​

{'encoding': 'windows-1252', 'confidence': 0.5}
"¦   " "   " "   " "¦   "
"040543"
{'encoding': 'windows-1252', 'confidence': 0.5}
"LEASE GREGPRU D ¬ETERSPM                 " "LEASE GREGPRU D ETERSPM
              " "LEASE GREGPRU D ETERSPM                 " "LEASE
GREGPRU D ¬ETERSPM                 "
"979643"
{'encoding': 'windows-1252', 'confidence': 0.5}
"¦   " "   " "   " "¦   "
"986979"
{'encoding': 'windows-1252', 'confidence': 0.5}
"WELLS FARGO &¢ COMPANY                   " "WELLS FARGO & COMPANY
              " "WELLS FARGO & COMPANY                   " "WELLS
FARGO &¢ COMPANY                   "
"994946"
{'encoding': 'windows-1252', 'confidence': 0.5}
OSSOSSO¬¬O         " OSSOSSOO         " OSSOSSOO         " OSSOSSO¬¬O         "
"996535"



Vincent Davis
720-301-3003

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web