Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #74004 > unrolled thread

python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position, to print out the file contents to stdout,

Back to article view | Back to comp.lang.python

Contents

  python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position,  to print out the file contents to stdout, gintare <g.statkute@gmail.com> - 2014-07-06 01:45 -0700
    Re: python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position,  to print out the file contents to stdout, gintare <g.statkute@gmail.com> - 2014-07-06 02:05 -0700
      Re: python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position,  to print out the file contents to stdout, Dave Angel <davea@davea.name> - 2014-07-06 08:57 -0400
      Re: python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position,  to print out the file contents to stdout, Rick Johnson <rantingrickjohnson@gmail.com> - 2014-07-06 07:52 -0700
        Re: python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position,  to print out the file contents to stdout, Terry Reedy <tjreedy@udel.edu> - 2014-07-06 12:14 -0400
          Re: python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position,  to print out the file contents to stdout, Rick Johnson <rantingrickjohnson@gmail.com> - 2014-07-06 09:53 -0700
            Re: python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position,  to print out the file contents to stdout, wxjmfauth@gmail.com - 2014-07-06 11:14 -0700
              Re: python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position,  to print out the file contents to stdout, Rick Johnson <rantingrickjohnson@gmail.com> - 2014-07-06 12:37 -0700
                Re: python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position,  to print out the file contents to stdout, wxjmfauth@gmail.com - 2014-07-06 23:58 -0700

#74004 — python33, windows, UnicodeEncodeError: 'charmap' codec can't encode characters in position, to print out the file contents to stdout,

I run python33 scrupt SVtxt.py from Windows command line
I want to print out the file contents to stdout. 

File contents: "Vi kan inte längre underordna den vinster..."

Python script sentences i have tries to use:
f=open('C:\Python33\Scripts\lang\langu\svtxt.txt','r')
linef=f.readlines()
f.close()
print(linef)#gives error
print(linef).decode("utf-8","ignore")#gives error

ERROR:
c:\Python33\Scripts\lang\langu>python SVtxt.py
Traceback (most recent call last):
  File "SVtxt.py", line 8, in <module>
    print(linef).decode("utf-8","ignore")
  File "C:\Python33\lib\encodings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 91-92:

[toc] | [next] | [standalone]

#74005

The answer is on page:https://docs.python.org/3.3/howto/unicode.html#reading-and-writing-unicode-data

The correct code:
f=open('C:\Python33\Scripts\lang\langu\svtxt.txt','r', encoding='utf-8')
linef=f.readlines()
print(repr(linef))

[toc] | [prev] | [next] | [standalone]

#74010

gintare <g.statkute@gmail.com> Wrote in message:
> The answer is on page:https://docs.python.org/3.3/howto/unicode.html#reading-and-writing-unicode-data
> 
> The correct code:
> f=open('C:\Python33\Scripts\lang\langu\svtxt.txt','r', encoding='utf-8')
> linef=f.readlines()
> print(repr(linef))
> 

But naturally the path should be fixed as well. Either use forward
 slashes,  escape the backslashes,  or use a raw string.
 

f=open( r'C:\Python33\Scripts\lang\langu\svtxt.txt','r',
 encoding='utf-8')

-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#74020

On Sunday, July 6, 2014 4:05:10 AM UTC-5, gintare wrote:
> The correct code:
> f=open(<snip-path>,'r', encoding='utf-8')
> linef=f.readlines()
> print(repr(linef))

Yes but do you understand why? And even if you DO understand
why, you should explain the details because the neophytes
are always watching!

    >_>
    
    <_<
    
    o_O? ...what, me worry?

Now don't misunderstand me, i am happy that you took the
time to report to us that you found the answer! Yes, nothing
is more aggravating than writing a long, thoughtful, and
elegant answer, and then have the OP come back and say:

    "Oh i found the answer, but thanks anyway"
    
But really, kudos to you for at least making the effort to
report. Because the only thing worse than a report without
context is no report at all! Indeed, then we end up with
unresolved threads and our community mojo suffers.... but i
digress!


Now, we must explain WHY this new code works, and WHY the
old code failed.

    SIMPLE!
    
"file.readlines()" returns a list, with each value of the list
representing a line of the file. Nothing wrong with using
"readlines()" of course, UNTIL you try to pass the value of
"readlines" into the "print()" function, ah-aH-AH! 

So the direct reason for failure is due to the fact that the
"print()" function ONLY handles strings, not list objects.

So in light of this problem there are a few solutions:

  1. Cast the list into a string, which you did using the
  "repr()" function, although you can also use the "str()"
  function, or use "string formatting" if formatting makes
  sense (in this case formatting does not make sense).
  
  2. Use a different method of the file object which returns
  a string instead of a "list-of-lines". Your code seems to
  make no use of "line-ified-data-structures", so why
  burn cycles creating one?

PS: We could use a few more members of the female persuasion
in this community, so thanks for dropping by and hope to see
you posting in the future! We desperately need something
"fair" to offset the reeking of smelly socks, stinky
armpits, and the stomach churning *stench* of illogical
naming conventions!

[toc] | [prev] | [next] | [standalone]

#74030

On 7/6/2014 10:52 AM, Rick Johnson wrote:

> So the direct reason for failure is due to the fact that the
> "print()" function ONLY handles strings, not list objects.

 >>> print(object())
<object object at 0x00000000020D2140>
 >>> print(['abc', 'cdf'])
['abc', 'cdf']

Since the original poster did not copy the traceback from the print 
error, it is unclear what the error was.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#74035

On Sunday, July 6, 2014 11:14:26 AM UTC-5, Terry Reedy wrote:
> On 7/6/2014 10:52 AM, Rick Johnson wrote:
> > So the direct reason for failure is due to the fact that the
> > "print()" function ONLY handles strings, not list objects.
> >>> print(object())
> <object object at 0x00000000020D2140>  
> >>> print(['abc', 'cdf'])
> ['abc', 'cdf']

Hmm, you are correct Terry, i must have been confusing
file#write() and print() when i wrote all that nonsense.
Thanks for pointing out the error.

> Since the original poster did not copy the traceback from
> the print error, it is unclear what the error was.

I'm wondering what she is expecting this line to do: 

    "print(linef).decode("utf-8","ignore")

My understanding is that print is a one way destination to a
stream, and the only return value would be the default None.
Urm, last time i got myself into trouble by not testing,
not letting that happen again! O:-)

============================================================
 INTERACTIVE SESSION: Python 2.x
============================================================
py> from __future__ import print_function
py> print([1,2,3])
[1, 2, 3]
py> print(print([1,2,3]))
[1, 2, 3]
None
py> returnvalue = print([1,2,3])
[1, 2, 3]
py> repr(returnvalue)
'None'

Seems like she'd better do the decoding before printing, or
am i wrong again?

[toc] | [prev] | [next] | [standalone]

#74039

Le dimanche 6 juillet 2014 18:53:34 UTC+2, Rick Johnson a écrit :

[...]
> 
> 
> Seems like she'd better do the decoding before printing

No

> [...], or
> 
> am i wrong again?

Yes


>>> s = 'abc需'
>>> sys.stdout.encoding
'<unicode>'
>>> print(s)
abc需
>>> sys.stdout.encoding = 'cp437'
>>> sys.stdout.encoding
'cp437'
>>> print(s)
Traceback (most recent call last):
  File "<eta last command>", line 1, in <module>
  File "D:\jm\jmpy\eta\eta40beta2\etastdio.py", line 158, in write
    s = s.encode(self.pencoding).decode('cp1252')
  File "C:\Python32\lib\encodings\cp437.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 4-5: 
character maps to <undefined>
>>> print(s.encode(sys.stdout.encoding, 'replace'))
'abcé??'
>>> sys.stdout.encoding = 'cp850'
>>> sys.stdout.encoding
'cp850'
>>> print(s)
Traceback (most recent call last):
  File "<eta last command>", line 1, in <module>
  File "D:\jm\jmpy\eta\eta40beta2\etastdio.py", line 158, in write
    s = s.encode(self.pencoding).decode('cp1252')
  File "C:\Python32\lib\encodings\cp850.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 4-5: 
character maps to <undefined>
>>> print(s.encode(sys.stdout.encoding, 'replace'))
'abcé??'
>>> # and so on
>>> sys.stdout.encoding = 'cp1252')
  File "<eta last command>", line 1
    sys.stdout.encoding = 'cp1252')
                                  ^
SyntaxError: invalid syntax
>>> # oops
>>> sys.stdout.encoding = 'cp1252'
>>> print(s)
abc需
>>> sys.stdout.encoding = 'mac-roman'
>>> print(s)
abcŽÏÛ
>>> print(s.encode(sys.stdout.encoding, 'replace'))
'abc需'
>>> sys.stdout.encoding = 'utf-8'
>>> print(s.encode(sys.stdout.encoding, 'replace'))
'abc需'
>>> sys.stdout.encoding = 'utf-16-le'
>>> print(s.encode(sys.stdout.encoding, 'replace'))
'abc需'
>>> 
>>> sys.stdout.encoding = 'utf-32-be'
>>> print(s.encode(sys.stdout.encoding, 'replace'))
'abc需'
>>>

jmf

[toc] | [prev] | [next] | [standalone]

#74055

On Sunday, July 6, 2014 1:14:38 PM UTC-5, wxjm...@gmail.com wrote:
> Le dimanche 6 juillet 2014 18:53:34 UTC+2, Rick Johnson a écrit :
> [...]
> 
> > Seems like she'd better do the decoding before printing
> No
> 
> > or am i wrong again?
> Yes
> 
> >>> s = 'abc需'
> >>> sys.stdout.encoding
> '<unicode>'
> >>> print(s)
> abc需
> >>> sys.stdout.encoding = 'cp437'
> >>> sys.stdout.encoding
> 'cp437'
> >>> print(s)
> Traceback (most recent call last):
>   File "<eta last command>", line 1, in <module>   File "D:\jm\jmpy\eta\eta40beta2\etastdio.py", line 158, in write
>     s = s.encode(self.pencoding).decode('cp1252')
>   File "C:\Python32\lib\encodings\cp437.py", line 12, in encode
>     return codecs.charmap_encode(input,errors,encoding_map)
> UnicodeEncodeError: 'charmap' codec can't encode characters in position 4-5: 
> character maps to <undefined> >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 'abcé??'
> >>> sys.stdout.encoding = 'cp850'
> >>> sys.stdout.encoding
> 'cp850'
> >>> print(s)
> Traceback (most recent call last):
>   File "<eta last command>", line 1, in <module>   File "D:\jm\jmpy\eta\eta40beta2\etastdio.py", line 158, in write
>     s = s.encode(self.pencoding).decode('cp1252')
>   File "C:\Python32\lib\encodings\cp850.py", line 12, in encode
>     return codecs.charmap_encode(input,errors,encoding_map)
> UnicodeEncodeError: 'charmap' codec can't encode characters in position 4-5: 
> character maps to <undefined> >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 'abcé??'
> >>> # and so on
> >>> sys.stdout.encoding = 'cp1252')
>   File "<eta last command>", line 1
>     sys.stdout.encoding = 'cp1252')
>                                   ^
> SyntaxError: invalid syntax
> >>> # oops
> >>> sys.stdout.encoding = 'cp1252'
> >>> print(s)
> abc需
> >>> sys.stdout.encoding = 'mac-roman'
> >>> print(s)
> abcŽÏÛ
> >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 'abc需'
> >>> sys.stdout.encoding = 'utf-8'
> >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 'abc需'
> >>> sys.stdout.encoding = 'utf-16-le'
> >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 'abc需'
> >>> sys.stdout.encoding = 'utf-32-be'
> >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 'abc需'
> jmf

Oh my, all that code just so you can handle glyphs with
squiggly little accent marks? Are you so afraid you'll
forget how to pronounce the words without them? I wonder how
those simpleton Americans are able to pronounce words without
a tutorial? Boggles the mind really, BOGGLES THE MIND!

For the remainder of your bloated Unicode char set that
defines symbols, and cute little miniature fractions, and
snowmen, and all sorts of ridiculous scrawl... what a waste
of time!

You know, instead of bending over backwards to "include"
every selfish char man in his complete and utter stupidity
can muster, when are we going to realize that keyboards can
only contain a very *finite* number of keys before they
become unusable. If you want to draw shapes and scrawl to a
screen, USE THE CORRECT DAMN TOOL!

In this day and age of "globalism", when are we going to
unite under a single form of written communication. And if
not for the sake of programmers, i can think of an even more
important reason,,, for the sake of miscommunications and
animosity. How many fights and wars have been started simply
on the grounds of a miscommunication?

    "THE MINDS OF LITTLE MEN ARE CONSUMED WITH 
    EMOTIONAL IDENTITIES AND THE PURSUIT OF CREATING 
    SYMBOLS THAT DEFINE THOSE IDENTITIES"
    
The true free man does not belong to any "imaginary" group,
he does not pay allegiance to any one country, or any one
religion, or any one sports team (if at all), no, he is free
and belongs to group defined by a single word, a group from
which he does NOT choose, but a group from which he is
*BOUND*:

    THAT GROUP BE HUMANITY!

My keyboard has every char i could ever need to express
myself sufficiently across a medium that requires redundant
"pecking" of keys on a keyboard. This is the form of
communication we have achieved thus far, so until we can
evolve past this "pecking olymics", we would be wise to keep
our pecking to a minimum and employ a ubiquitous written
language that is elegant and simplistic, and optimized for
typing.

I'm sorry but i guess i'm just too practical for all this
nonsense. To me Unicode is just like automobiles. Everyone
has their own color and brand, and this one has a butt
warmer and that one has a vanity mirror, when in fact all
automobiles serve the same purpose of transportation.

Although, unlike automobiles where i can choose to drive
ONLY my car, with Unicode i'm forced to interface with your
selfish idea of what a car should be. I just want to get
from point A to point B without being pulled over and
arrested because you were "partying" with three prostitutes
who forgot their crack pipe under the seat!

But don't bother listening to me anyway, and go on with your
selfish pursuits, continue dividing people instead of
uniting them, continue creating a world that is superfluously
complex by your own hand -- but don't be surprised when this
monstrosity comes crashing down on top of you!

    THERE IS PRIDE BEFORE THE FALL

[toc] | [prev] | [next] | [standalone]

#74079

Le dimanche 6 juillet 2014 21:37:36 UTC+2, Rick Johnson a écrit :
> On Sunday, July 6, 2014 1:14:38 PM UTC-5, wxjm...@gmail.com wrote:
> 
> > Le dimanche 6 juillet 2014 18:53:34 UTC+2, Rick Johnson a écrit :
> 
> > [...]
> 
> > 
> 
> > > Seems like she'd better do the decoding before printing
> 
> > No
> 
> > 
> 
> > > or am i wrong again?
> 
> > Yes
> 
> > 
> 
> > >>> s = 'abc需'
> 
> > >>> sys.stdout.encoding
> 
> > '<unicode>'
> 
> > >>> print(s)
> 
> > abc需
> 
> > >>> sys.stdout.encoding = 'cp437'
> 
> > >>> sys.stdout.encoding
> 
> > 'cp437'
> 
> > >>> print(s)
> 
> > Traceback (most recent call last):
> 
> >   File "<eta last command>", line 1, in <module>   File "D:\jm\jmpy\eta\eta40beta2\etastdio.py", line 158, in write
> 
> >     s = s.encode(self.pencoding).decode('cp1252')
> 
> >   File "C:\Python32\lib\encodings\cp437.py", line 12, in encode
> 
> >     return codecs.charmap_encode(input,errors,encoding_map)
> 
> > UnicodeEncodeError: 'charmap' codec can't encode characters in position 4-5: 
> 
> > character maps to <undefined> >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 
> > 'abcé??'
> 
> > >>> sys.stdout.encoding = 'cp850'
> 
> > >>> sys.stdout.encoding
> 
> > 'cp850'
> 
> > >>> print(s)
> 
> > Traceback (most recent call last):
> 
> >   File "<eta last command>", line 1, in <module>   File "D:\jm\jmpy\eta\eta40beta2\etastdio.py", line 158, in write
> 
> >     s = s.encode(self.pencoding).decode('cp1252')
> 
> >   File "C:\Python32\lib\encodings\cp850.py", line 12, in encode
> 
> >     return codecs.charmap_encode(input,errors,encoding_map)
> 
> > UnicodeEncodeError: 'charmap' codec can't encode characters in position 4-5: 
> 
> > character maps to <undefined> >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 
> > 'abcé??'
> 
> > >>> # and so on
> 
> > >>> sys.stdout.encoding = 'cp1252')
> 
> >   File "<eta last command>", line 1
> 
> >     sys.stdout.encoding = 'cp1252')
> 
> >                                   ^
> 
> > SyntaxError: invalid syntax
> 
> > >>> # oops
> 
> > >>> sys.stdout.encoding = 'cp1252'
> 
> > >>> print(s)
> 
> > abc需
> 
> > >>> sys.stdout.encoding = 'mac-roman'
> 
> > >>> print(s)
> 
> > abcŽÏÛ
> 
> > >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 
> > 'abc需'
> 
> > >>> sys.stdout.encoding = 'utf-8'
> 
> > >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 
> > 'abc需'
> 
> > >>> sys.stdout.encoding = 'utf-16-le'
> 
> > >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 
> > 'abc需'
> 
> > >>> sys.stdout.encoding = 'utf-32-be'
> 
> > >>> print(s.encode(sys.stdout.encoding, 'replace'))
> 
> > 'abc需'
> 
> > jmf
> 
> 
> 
> Oh my, all that code just so you can handle glyphs with
> 
> squiggly little accent marks? Are you so afraid you'll
> 
> forget how to pronounce the words without them? I wonder how
> 
> those simpleton Americans are able to pronounce words without
> 
> a tutorial? Boggles the mind really, BOGGLES THE MIND!
> 
> 
> 
> For the remainder of your bloated Unicode char set that
> 
> defines symbols, and cute little miniature fractions, and
> 
> snowmen, and all sorts of ridiculous scrawl... what a waste
> 
> of time!
> 
> 
> 
> You know, instead of bending over backwards to "include"
> 
> every selfish char man in his complete and utter stupidity
> 
> can muster, when are we going to realize that keyboards can
> 
> only contain a very *finite* number of keys before they
> 
> become unusable. If you want to draw shapes and scrawl to a
> 
> screen, USE THE CORRECT DAMN TOOL!
> 
> 
> 
> In this day and age of "globalism", when are we going to
> 
> unite under a single form of written communication. And if
> 
> not for the sake of programmers, i can think of an even more
> 
> important reason,,, for the sake of miscommunications and
> 
> animosity. How many fights and wars have been started simply
> 
> on the grounds of a miscommunication?
> 
> 
> 
>     "THE MINDS OF LITTLE MEN ARE CONSUMED WITH 
> 
>     EMOTIONAL IDENTITIES AND THE PURSUIT OF CREATING 
> 
>     SYMBOLS THAT DEFINE THOSE IDENTITIES"
> 
>     
> 
> The true free man does not belong to any "imaginary" group,
> 
> he does not pay allegiance to any one country, or any one
> 
> religion, or any one sports team (if at all), no, he is free
> 
> and belongs to group defined by a single word, a group from
> 
> which he does NOT choose, but a group from which he is
> 
> *BOUND*:
> 
> 
> 
>     THAT GROUP BE HUMANITY!
> 
> 
> 
> My keyboard has every char i could ever need to express
> 
> myself sufficiently across a medium that requires redundant
> 
> "pecking" of keys on a keyboard. This is the form of
> 
> communication we have achieved thus far, so until we can
> 
> evolve past this "pecking olymics", we would be wise to keep
> 
> our pecking to a minimum and employ a ubiquitous written
> 
> language that is elegant and simplistic, and optimized for
> 
> typing.
> 
> 
> 
> I'm sorry but i guess i'm just too practical for all this
> 
> nonsense. To me Unicode is just like automobiles. Everyone
> 
> has their own color and brand, and this one has a butt
> 
> warmer and that one has a vanity mirror, when in fact all
> 
> automobiles serve the same purpose of transportation.
> 
> 
> 
> Although, unlike automobiles where i can choose to drive
> 
> ONLY my car, with Unicode i'm forced to interface with your
> 
> selfish idea of what a car should be. I just want to get
> 
> from point A to point B without being pulled over and
> 
> arrested because you were "partying" with three prostitutes
> 
> who forgot their crack pipe under the seat!
> 
> 
> 
> But don't bother listening to me anyway, and go on with your
> 
> selfish pursuits, continue dividing people instead of
> 
> uniting them, continue creating a world that is superfluously
> 
> complex by your own hand -- but don't be surprised when this
> 
> monstrosity comes crashing down on top of you!
> 
> 
> 
>     THERE IS PRIDE BEFORE THE FALL

--------

I forgot to mention, the example I gave came from
one of my interactive Python interpreter (a gui
app).

You have certainly noticed, it has the faculty
to modify on the fly the coding of the "device" that
will host a unicode. It mimics, in fact, any device
which may receive a unicode (terminal, file, db,
gui text widget, ...) and that a unicode has
to be encoded.

The characters I'm using are just characters very
representative of unicode and I deliberatly choose
them with care.
Glyphs have here nothing to do. But, if you wish
examples with others chars/fonts like
junicode (medieval scripts), stix (math symbols),
polytonic Greek or why not Urdu script (nastaliq,
naskh [*]), ...

https://medium.com/@eteraz/the-death-of-the-urdu-script-9ce935435d90

[*] not yet tested.

jmf

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web