Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #43230 > unrolled thread

Unicode issue with Python v3.3

Back to article view | Back to comp.lang.python

Contents

  Unicode issue with Python v3.3 Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-09 14:10 -0700
    Re: Unicode issue with Python v3.3 Ian Kelly <ian.g.kelly@gmail.com> - 2013-04-09 15:34 -0600
      Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-09 20:16 -0700
        Re: Unicode issue with Python v3.3 Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-10 04:25 +0000
          Re: Unicode issue with Python v3.3 Chris Angelico <rosuav@gmail.com> - 2013-04-10 14:46 +1000
            Re: Unicode issue with Python v3.3 rusi <rustompmody@gmail.com> - 2013-04-09 22:06 -0700
              Re: Unicode issue with Python v3.3 rusi <rustompmody@gmail.com> - 2013-04-09 23:04 -0700
                Re: Unicode issue with Python v3.3 Antoine Pitrou <solipsis@pitrou.net> - 2013-04-10 07:04 +0000
                Re: Unicode issue with Python v3.3 Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-10 08:28 +0000
                  People in the python community [was Re: Unicode issue with Python v3.3] Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-10 08:34 +0000
                    Re: People in the python community [was Re: Unicode issue with Python v3.3] Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-04-10 10:31 +0100
                      Re: People in the python community [was Re: Unicode issue with Python v3.3] Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-10 03:50 -0700
                      Re: People in the python community [was Re: Unicode issue with Python v3.3] Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-10 03:50 -0700
                        Re: People in the python community [was Re: Unicode issue with Python v3.3] Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-10 03:53 -0700
                        Re: People in the python community [was Re: Unicode issue with Python v3.3] Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-10 03:53 -0700
                    Re: People in the python community [was Re: Unicode issue with Python v3.3] Peter Otten <__peter__@web.de> - 2013-04-10 13:11 +0200
                    Re: People in the python community [was Re: Unicode issue with Python v3.3] Peter Otten <__peter__@web.de> - 2013-04-10 13:13 +0200
                      Re: People in the python community [was Re: Unicode issue with Python v3.3] Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-10 07:43 -0700
                        Re: People in the python community [was Re: Unicode issue with Python v3.3] Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-04-10 16:15 +0100
                          Re: People in the python community [was Re: Unicode issue with Python v3.3] Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-10 09:50 -0700
                            Re: People in the python community [was Re: Unicode issue with Python v3.3] Michael Torrie <torriem@gmail.com> - 2013-04-11 21:20 -0600
                          Re: People in the python community [was Re: Unicode issue with Python v3.3] Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-10 09:50 -0700
                        Re: People in the python community [was Re: Unicode issue with Python v3.3] Chris Angelico <rosuav@gmail.com> - 2013-04-11 01:19 +1000
                      Re: People in the python community [was Re: Unicode issue with Python v3.3] Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-10 07:43 -0700
                  Re: Unicode issue with Python v3.3 Arnaud Delobelle <arnodel@gmail.com> - 2013-04-10 23:56 +0100
          Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-10 00:23 -0700
            Re: Unicode issue with Python v3.3 Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-10 01:06 -0700
              Re: Unicode issue with Python v3.3 Cameron Simpson <cs@zip.com.au> - 2013-04-11 09:17 +1000
                Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-10 21:50 -0700
                  Re: Unicode issue with Python v3.3 Cameron Simpson <cs@zip.com.au> - 2013-04-11 20:45 +1000
                    Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-11 03:54 -0700
                    Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-11 03:54 -0700
                    Re: Unicode issue with Python v3.3 Nikos <nagia.retsina@gmail.com> - 2013-04-11 09:55 -0700
                      Re: Unicode issue with Python v3.3 Cameron Simpson <cs@zip.com.au> - 2013-04-13 11:41 +1000
                        Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-12 21:50 -0700
                          Re: Unicode issue with Python v3.3 Cameron Simpson <cs@zip.com.au> - 2013-04-13 20:28 +1000
                            Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-13 07:16 -0700
                              Re: Unicode issue with Python v3.3 Chris Angelico <rosuav@gmail.com> - 2013-04-14 01:45 +1000
                              Re: Unicode issue with Python v3.3 Cameron Simpson <cs@zip.com.au> - 2013-04-14 10:01 +1000
                            Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-13 07:16 -0700
                        Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-12 21:50 -0700
                    Re: Unicode issue with Python v3.3 Nikos <nagia.retsina@gmail.com> - 2013-04-11 09:55 -0700
                Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-10 21:50 -0700
                  Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-11 00:13 -0700
                  Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-11 00:13 -0700
                    Re: Unicode issue with Python v3.3 Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-11 07:50 +0000
                      Re: Unicode issue with Python v3.3 Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-11 08:20 +0000
                        Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-11 03:07 -0700
                          Re: Unicode issue with Python v3.3 Lele Gaifax <lele@metapensiero.it> - 2013-04-11 12:45 +0200
            Re: Unicode issue with Python v3.3 Nobody <nobody@nowhere.com> - 2013-04-10 19:08 +0100
              Re: Unicode issue with Python v3.3 Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-10 11:25 -0700
                Re: Unicode issue with Python v3.3 Ian Kelly <ian.g.kelly@gmail.com> - 2013-04-10 13:50 -0600
      Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-09 20:16 -0700
    Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-11 21:36 -0700
      Re: Unicode issue with Python v3.3 alex23 <wuwei23@gmail.com> - 2013-04-11 22:06 -0700
        Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-11 22:42 -0700
          Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-12 05:50 -0700
            Re: Unicode issue with Python v3.3 Chris Angelico <rosuav@gmail.com> - 2013-04-12 23:14 +1000
              Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-12 06:18 -0700
                Re: Unicode issue with Python v3.3 Chris Angelico <rosuav@gmail.com> - 2013-04-12 23:21 +1000
              Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-12 06:18 -0700
                Re: Unicode issue with Python v3.3 rusi <rustompmody@gmail.com> - 2013-04-12 06:29 -0700
                  Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-12 07:36 -0700
                    Re: Unicode issue with Python v3.3 Ian Kelly <ian.g.kelly@gmail.com> - 2013-04-12 12:37 -0600
                      Re: Unicode issue with Python v3.3 Roy Smith <roy@panix.com> - 2013-04-12 14:49 -0400
                      Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-12 13:48 -0700
                      Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-12 13:48 -0700
    Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-13 23:00 -0700
      Re: Unicode issue with Python v3.3 Cameron Simpson <cs@zip.com.au> - 2013-04-14 19:28 +1000
        Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-14 04:22 -0700
          Re: Unicode issue with Python v3.3 Cameron Simpson <cs@zip.com.au> - 2013-04-18 09:00 +1000
            Re: Unicode issue with Python v3.3 Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-17 20:37 -0700
            Re: Unicode issue with Python v3.3 Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-17 20:37 -0700
            Re: Unicode issue with Python v3.3 Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-19 12:16 -0700
        Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-14 04:22 -0700
        Re: Unicode issue with Python v3.3 Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-15 11:42 -0700
        Re: Unicode issue with Python v3.3 Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-04-15 11:42 -0700
        Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-16 23:56 -0700
          Re: Unicode issue with Python v3.3 Chris Angelico <rosuav@gmail.com> - 2013-04-17 17:01 +1000
          Re: Unicode issue with Python v3.3 Chris Angelico <rosuav@gmail.com> - 2013-04-17 17:32 +1000
        Re: Unicode issue with Python v3.3 nagia.retsina@gmail.com - 2013-04-16 23:56 -0700

Page 2 of 5 — ← Prev page 1 [2] 3 4 5  Next page →

#43416 — Re: People in the python community [was Re: Unicode issue with Python v3.3]

On 04/10/2013 10:50 AM, Νίκος Γκρ33κ wrote:
> I'am not sure i follow you. How did my topic changed?! Is this
> possible?

This is a mailing list/nntp newsgroup.  The subject line can be changed
arbitrarily by anyone replying to another message.  Normally this is
done to indicate a natural progression of the conversation in a new
direction.  In this case, Steven D'Aprano wrote a reply that did not
answer your pleas, but instead made some observations, and so he changed
the subject line to reflect that.

If you read your messages using a threaded message display, this will
make more sense to you.  But if you use Gmail's (or Google's) broken
conversation view, then this information about who is responding to whom
does get lost--actually in conversation view a lot of information about
the message flow is lost; it really is unfortunate that this way of
communicating has become so widespread.

> How about the oce i posted at patebin.com. Did anyone by any chnace
> had a look into?
>
> It's only a single thing iam missing for the encoding and the the
> script will load properly with python 3.3

I'm truly sorry, but I simply do not have the time to do so.

[toc] | [prev] | [next] | [standalone]

#43285 — Re: People in the python community [was Re: Unicode issue with Python v3.3]

I'am not sure i follow you.
How did my topic changed?! Is this possible?

How about the oce i posted at patebin.com.
Did anyone by any chnace had a look into?

It's only a single thing iam missing for the encoding and the the script will load properly with python 3.3

[toc] | [prev] | [next] | [standalone]

#43279 — Re: People in the python community [was Re: Unicode issue with Python v3.3]

On Thu, Apr 11, 2013 at 1:15 AM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
> On 10/04/2013 15:43, Νίκος Γκρ33κ wrote:
>>
>> Anyone please?
>>
>
> I have already shown my support for Peter Otten on this thread.  Are you
> asking for more people to do so?

Sure, I can! He's one of the people who keeps this list/ng productive
and helpful. People can come here with Python problems and get Python
solutions.

(I wouldn't normally "me too" a thread, but hey, with that opening!)

ChrisA

[toc] | [prev] | [next] | [standalone]

#43274 — Re: People in the python community [was Re: Unicode issue with Python v3.3]

Anyone please?

[toc] | [prev] | [next] | [standalone]

#43300

On 10 April 2013 09:28, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Tue, 09 Apr 2013 23:04:35 -0700, rusi wrote:
[...]
> I think it is quite unfair of you to mischaracterise the entire community
> response in this way. One person made a light-hearted, silly, unhelpful
> response. (As sarcasm, I'm afraid it missed the target.) Two people made
> good, sensible responses -- and you were not either of them.

Enough already with the thought police.

It was me who made the silly reply to the guy who was ranting about
everything being broken, giving us nothing to help in on, ending his
message in an edifying and in my judgement, largely rhetorical
"Suggestions?".  So I gave him some silly suggestions (*not* intended
to be sarcasm), and I'm not apologising for it.  At least I'm not
presuming to take the moral high ground at every half-opportunity.

Recently I gave a very quick reply to someone who was wondering why he
couldn't get the docstring from his descriptor - I didn't have the
time to expand because two of my kids had jumped on my knees almost as
soon as I'd got on the computer.  I decided to post the reply anyway
as I thought it would give the OP something to get started on and
nobody else seemed to have replied so far - but I got remonstrated for
not being complete enough in my reply!  What is that about?

AFAIK, this is not Python Customer Service, but a place for people who
are interested in Python to discuss problems and *freely* exchange
thoughts about the language and its ecosystem.  Over the year I've
posted the occasional silly message but I think my record is
overwhelmingly that I've tried to be helpful, and when I've needed
some help myself, I've got some great advice.  My first question on
this list was answered by Alex Martelli and nowadays I get most
excellent and concise tips from Peter Otten - thanks, Peter! If
there's one person on this list I don't want to offend, it's you!

So here's to lots more good and bad humour on this list, and the
occasional slightly un-pc remark even!

Cheers,

-- 
Arnaud

[toc] | [prev] | [next] | [standalone]

#43252

Τη Τετάρτη, 10 Απριλίου 2013 7:25:21 π.μ. UTC+3, ο χρήστης Steven D'Aprano έγραψε:

> What does os.environ['REMOTE_ADDR'] give? Until you answer that question, 
> you won't make any progress.

I insists stevv.

Look at what 'python3 metrites.py' gives me

<!-- The above is a description of an error in a Python program, formatted
     for a Web browser because the 'cgitb' module was enabled.  In case you
     are not reading this in a Web browser, here is the original traceback:

Traceback (most recent call last):
  File "metrites.py", line 34, in &lt;module&gt;
    userinfo = os.environ['HTTP_USER_AGENT']
  File "/root/.local/lib/python2.7/lib/python3.3/os.py", line 669, in __getitem__
    value = self._data[self.encodekey(key)]
KeyError: b'HTTP_USER_AGENT'

-->

[toc] | [prev] | [next] | [standalone]

#43253

Here is the whole code for metrites.py in case someone wants to take allok.

Everything is correct after altering it to meet python 3.3, everythign aprt from the weird unicode error thing.

http://pastebin.com/5Mpjx5Fd

please take a look.
Thank you. 

[toc] | [prev] | [next] | [standalone]

#43302

On 10Apr2013 01:06, Νίκος Γκρ33κ <nikos.gr33k@gmail.com> wrote:
| Here is the whole code for metrites.py in case someone wants to take allok.
| 
| Everything is correct after altering it to meet python 3.3,
| everythign aprt from the weird unicode error thing.
| 
| http://pastebin.com/5Mpjx5Fd
| 
| please take a look.

From looking at the HTML source of the page:

  http://superhost.gr/

I see near the start:

  b'<!DOCTYPE html

I'd say you have a bytes object that you've fed to print().
In python2, str is effectively bytes.
In python3, str is a sequence of Unicode code points, and bytes are
arrays of small integers.
If you feed a bytes object to print it will print a strig represenation
of it, starting with "b'...".

The question is: where did the bytes object come from? A cursory
glance through your pastebin code doesn't show me anthing very
obvious.

I'd start by asking: where does the string "<!DOCTYPE" come from?
Wherever that is, it seems to be bytes rather than str.
Start with that.

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

You don't have to live on the edge, but you have to know where it is.
        - Scott Lilliott, csl@swl.msd.ray.com

[toc] | [prev] | [next] | [standalone]

#43314

Firtly thank uou for taking a look into the code.

the doctype is coming form the attempt of script metrites.py to open and read the 'index.html' file.

But i don't know how to try to open it as a byte file instead of an tetxt file.

[toc] | [prev] | [next] | [standalone]

#43348

On 10Apr2013 21:50, nagia.retsina@gmail.com <nagia.retsina@gmail.com> wrote:
| Firtly thank uou for taking a look into the code.
| the doctype is coming form the attempt of script metrites.py to open and read the 'index.html' file.
| But i don't know how to try to open it as a byte file instead of an tetxt file.

I think you've got it backwards. It looks like metrites.py has
opened the file as bytes instead of as text (probably utf8, but
that remains to be seen). Because it has opened it in binary mode
you're getting bytes when you read from the file.

Can you show the relevant code that opens the files and reads from
it, and the print statement that is putting it back out?

You probably need to ensure that metrites.py is opening it as text,
with the correct encoding.  Note that the encoding is nothing to
do with your _output_. It is the encoding of the data in the file
you are reading, and that is dictated by the editor used to make
the file.

Anyway, code first. What does it look like?

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

Six trillion RFID tags is four orders of magnitude bigger than any electronic item ever made.
- overhead by WIRED at the Intelligent Printing conference Oct2006

[toc] | [prev] | [next] | [standalone]

#43350

Of course here is how it look like:

	if page.endswith('.html'):
		f = open( "/home/nikos/www/" + page, encoding="utf-8" )
		htmldata = f.read()
		htmldata = htmldata % (quote, music)
		
		counter = ''' <center>
					  <a href="mailto:support@superhost.gr"> <img src="/data/images/mail.png"></a>
					  <table border=2 cellpadding=2 bgcolor=black>
							<td><font color=lime>Αριθμός Επισκεπτών</td>
							<td><a href="http://superhost.gr/?show=log&page=%s"><font color=yellow> %d </td>
					  </table><br>
			  ''' % (page, data[0])
			  
		template = htmldata + counter
		print( template )

[toc] | [prev] | [next] | [standalone]

#43351

Of course here is how it look like:

	if page.endswith('.html'):
		f = open( "/home/nikos/www/" + page, encoding="utf-8" )
		htmldata = f.read()
		htmldata = htmldata % (quote, music)
		
		counter = ''' <center>
					  <a href="mailto:support@superhost.gr"> <img src="/data/images/mail.png"></a>
					  <table border=2 cellpadding=2 bgcolor=black>
							<td><font color=lime>Αριθμός Επισκεπτών</td>
							<td><a href="http://superhost.gr/?show=log&page=%s"><font color=yellow> %d </td>
					  </table><br>
			  ''' % (page, data[0])
			  
		template = htmldata + counter
		print( template )

[toc] | [prev] | [next] | [standalone]

#43389

Τη Πέμπτη, 11 Απριλίου 2013 1:45:22 μ.μ. UTC+3, ο χρήστης Cameron Simpson έγραψε:
> On 10Apr2013 21:50, nagia.retsina@gmail.com <nagia.retsina@gmail.com> wrote:
> 
> | Firtly thank uou for taking a look into the code.
> 
> | the doctype is coming form the attempt of script metrites.py to open and read the 'index.html' file.
> 
> | But i don't know how to try to open it as a byte file instead of an tetxt file.
> 
> 
> 
> I think you've got it backwards. It looks like metrites.py has
> 
> opened the file as bytes instead of as text (probably utf8, but
> 
> that remains to be seen). Because it has opened it in binary mode
> 
> you're getting bytes when you read from the file.
> 
> 
> 
> Can you show the relevant code that opens the files and reads from
> 
> it, and the print statement that is putting it back out?
> 
> 
> 
> You probably need to ensure that metrites.py is opening it as text,
> 
> with the correct encoding.  Note that the encoding is nothing to
> 
> do with your _output_. It is the encoding of the data in the file
> 
> you are reading, and that is dictated by the editor used to make
> 
> the file.

>
> Webhost && Weblog
This works in the shell, but doesn't work on my website:

$ cat utf8.txt
υλικό!Πρόκειται γ
$ python3
Python 3.2.3 (default, Oct 19 2012, 20:10:41)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> data = open('utf8.txt').read()
>>> print(data)
υλικό!Πρόκειται γ

>>> print(data.encode('utf-8'))
b'\xcf\x85\xce\xbb\xce\xb9\xce\xba\xcf\x8c!\xce\xa0\xcf\x81\xcf\x8c\xce\xba\xce\xb5\xce\xb9\xcf\x84\xce\xb1\xce\xb9 \xce\xb3\n'

See, the last line is what i'am getting on my website. If i remove the encode('utf-8') part in metrites.py, the webpage will not show anything at all...

[toc] | [prev] | [next] | [standalone]

#43495

On 11Apr2013 09:55, Nikos <nagia.retsina@gmail.com> wrote:
| Τη Πέμπτη, 11 Απριλίου 2013 1:45:22 μ.μ. UTC+3, ο χρήστης Cameron Simpson έγραψε:
| > On 10Apr2013 21:50, nagia.retsina@gmail.com <nagia.retsina@gmail.com> wrote:
| > | the doctype is coming form the attempt of script metrites.py to open and read the 'index.html' file.
| > | But i don't know how to try to open it as a byte file instead of an tetxt file.

Lele Gaifax showed one way:

    from codecs import open
    with open('index.html', encoding='utf-8') as f:
        content = f.read()

But a plain open() should also do:

    with open('index.html') as f:
        content = f.read()

if you're not taking tight control of the file encoding.

The point here is to get _text_ (i.e. str) data from the file, not bytes.

If the text turns out to be incorrectly decoded (i.e. incorrectly
reading the file bytes and assembling them into text strings) because
the default encoding is wrong, then you may need to read for Lele's
more verbose open() example to select the correct encoding.

But first ignore that and get text (str) instead of bytes.
If you're already getting text from the file, something later is
making bytes and handing it to print().

Another approach to try is to use
  sys.stdout.write()
instead of
  print()

The print() function will take _anything_ and write text of some form.
The write() function will throw an exception if it gets the wrong type of data.

If sys.stdout is opened in binary mode then write() will require
bytes as data; strings will need to be explicitly turned into bytes
via .encode() in order to not raise an exception.

If sys.stdout is open in text mode, write() will require str data.
The sys.stdout file itself will transcribe to bytes for you.

If you take that route, at least you will not have confusion about
str versus bytes.

For an HTML output page I would advocate arranging that sys.stdout
is in text mode; that way you can do the natural thing and .write()
str data and lovely UTF-8 bytes will come out the other end.

If the above test (using .write() instead of print()) shows it to
be in binary mode we can fix that. But you need to find out.

You will want access to the error messages from the CGI environment;
do you have access to the web servers error_log? You can tail that
in a terminal while you reload the page to see what's going on.

| This works in the shell, but doesn't work on my website:
| 
| $ cat utf8.txt
| υλικό!Πρόκειται γ

Ok, so your terminal is using UTF-8 as its output coding. (And so
is your mail posting program, since we see it unmangled on my screen
here.)

| $ python3
| Python 3.2.3 (default, Oct 19 2012, 20:10:41)
| [GCC 4.6.3] on linux2
| Type "help", "copyright", "credits" or "license" for more information.
| >>> data = open('utf8.txt').read()
| >>> print(data)
| υλικό!Πρόκειται γ

Likewise.

However, in an exciting twist, I seem to recall that Python invoked
interactively with aterminal as output will have the default terminal
encoding in place on sys.stdout. Producing what you expect. _However_,
python invoked in a batch environment where stdout is not a terminal
(such as in the CGI environment producing your web page), that is
_not_ necessarily the case.

| >>> print(data.encode('utf-8'))
| b'\xcf\x85\xce\xbb\xce\xb9\xce\xba\xcf\x8c!\xce\xa0\xcf\x81\xcf\x8c\xce\xba\xce\xb5\xce\xb9\xcf\x84\xce\xb1\xce\xb9 \xce\xb3\n'
| 
| See, the last line is what i'am getting on my website.

The above line takes your Unicode text in "data" and transcribed
it to bytes using UTF-8 as the encoding. And print() is then receiving
that bytes object and printing its str() representation as "b'....'".
That str is itself unicode, and when print passes it to sys.stdout,
_that_ transcribed the unicode "b'...'" string as bytes to your
terminal. Using UTF-8 based on the previous examples above, but
since all those characters are in the bottom 127 code range the
byte sequence will be the same if it uses ASCII or ISO8859-1 or
almost anything else:-)

As you can see, there's a lot of encoding/decoding going on behind
the scenes even in this superficially simple example.

| If i remove
| the encode('utf-8') part in metrites.py, the webpage will not show
| anything at all...

Ah, but data will be being output. The print() function _will_ be
writing "data" out in some form.  I suggest you remove the .encode()
and then examine the _source_ text of the web page, not its visible
form.

So: remove .encode(), reload the web page, "view page source"
(depends on your browser, it is ctrl-U in Firefox ((Cmd-U in firefox
on a Mac))).

I think a lot of the issue you have in this thread is that your
page is too complex. Make another page to do the same thing, and
start with nothing. Add stuff to it a single item at a time until
the page behaves incorrectly. Then you will know the exact item of
code that introduced the issue. And then that single item can be
examined in detail for the decode/encode issues.

The other issue in the thread is that people losing patience get
snarky. Respond only to the technical content. If a message is only
snarky, _ignore_ it. People like the last word; let them have it
and you won't get sidetracked into arguments.

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

PCs are like a submarine, it will work fine till you open Windows. - zollie101

[toc] | [prev] | [next] | [standalone]

#43500

Τη Σάββατο, 13 Απριλίου 2013 4:41:57 π.μ. UTC+3, ο χρήστης Cameron Simpson έγραψε:
> On 11Apr2013 09:55, Nikos <nagia.retsina@gmail.com> wrote:
> 
> | Τη Πέμπτη, 11 Απριλίου 2013 1:45:22 μ.μ. UTC+3, ο χρήστης Cameron Simpson έγραψε:
> 
> | > On 10Apr2013 21:50, nagia.retsina@gmail.com <nagia.retsina@gmail.com> wrote:
> 
> | > | the doctype is coming form the attempt of script metrites.py to open and read the 'index.html' file.
> 
> | > | But i don't know how to try to open it as a byte file instead of an tetxt file.
> 
> 
> 
> Lele Gaifax showed one way:
> 
> 
> 
>     from codecs import open
> 
>     with open('index.html', encoding='utf-8') as f:
> 
>         content = f.read()
> 
> 
> 
> But a plain open() should also do:
> 
> 
> 
>     with open('index.html') as f:
> 
>         content = f.read()
> 
> 
> 
> if you're not taking tight control of the file encoding.
> 
> 
> 
> The point here is to get _text_ (i.e. str) data from the file, not bytes.
> 
> 
> 
> If the text turns out to be incorrectly decoded (i.e. incorrectly
> 
> reading the file bytes and assembling them into text strings) because
> 
> the default encoding is wrong, then you may need to read for Lele's
> 
> more verbose open() example to select the correct encoding.
> 
> 
> 
> But first ignore that and get text (str) instead of bytes.
> 
> If you're already getting text from the file, something later is
> 
> making bytes and handing it to print().
> 
> 
> 
> Another approach to try is to use
> 
>   sys.stdout.write()
> 
> instead of
> 
>   print()
> 
> 
> 
> The print() function will take _anything_ and write text of some form.
> 
> The write() function will throw an exception if it gets the wrong type of data.
> 
> 
> 
> If sys.stdout is opened in binary mode then write() will require
> 
> bytes as data; strings will need to be explicitly turned into bytes
> 
> via .encode() in order to not raise an exception.
> 
> 
> 
> If sys.stdout is open in text mode, write() will require str data.
> 
> The sys.stdout file itself will transcribe to bytes for you.
> 
> 
> 
> If you take that route, at least you will not have confusion about
> 
> str versus bytes.
> 
> 
> 
> For an HTML output page I would advocate arranging that sys.stdout
> 
> is in text mode; that way you can do the natural thing and .write()
> 
> str data and lovely UTF-8 bytes will come out the other end.
> 
> 
> 
> If the above test (using .write() instead of print()) shows it to
> 
> be in binary mode we can fix that. But you need to find out.
> 
> 
> 
> You will want access to the error messages from the CGI environment;
> 
> do you have access to the web servers error_log? You can tail that
> 
> in a terminal while you reload the page to see what's going on.
> 
> 
> 
> | This works in the shell, but doesn't work on my website:
> 
> | 
> 
> | $ cat utf8.txt
> 
> | υλικό!Πρόκειται γ
> 
> 
> 
> Ok, so your terminal is using UTF-8 as its output coding. (And so
> 
> is your mail posting program, since we see it unmangled on my screen
> 
> here.)
> 
> 
> 
> | $ python3
> 
> | Python 3.2.3 (default, Oct 19 2012, 20:10:41)
> 
> | [GCC 4.6.3] on linux2
> 
> | Type "help", "copyright", "credits" or "license" for more information.
> 
> | >>> data = open('utf8.txt').read()
> 
> | >>> print(data)
> 
> | υλικό!Πρόκειται γ
> 
> 
> 
> Likewise.
> 
> 
> 
> However, in an exciting twist, I seem to recall that Python invoked
> 
> interactively with aterminal as output will have the default terminal
> 
> encoding in place on sys.stdout. Producing what you expect. _However_,
> 
> python invoked in a batch environment where stdout is not a terminal
> 
> (such as in the CGI environment producing your web page), that is
> 
> _not_ necessarily the case.
> 
> 
> 
> | >>> print(data.encode('utf-8'))
> 
> | b'\xcf\x85\xce\xbb\xce\xb9\xce\xba\xcf\x8c!\xce\xa0\xcf\x81\xcf\x8c\xce\xba\xce\xb5\xce\xb9\xcf\x84\xce\xb1\xce\xb9 \xce\xb3\n'
> 
> | 
> 
> | See, the last line is what i'am getting on my website.
> 
> 
> 
> The above line takes your Unicode text in "data" and transcribed
> 
> it to bytes using UTF-8 as the encoding. And print() is then receiving
> 
> that bytes object and printing its str() representation as "b'....'".
> 
> That str is itself unicode, and when print passes it to sys.stdout,
> 
> _that_ transcribed the unicode "b'...'" string as bytes to your
> 
> terminal. Using UTF-8 based on the previous examples above, but
> 
> since all those characters are in the bottom 127 code range the
> 
> byte sequence will be the same if it uses ASCII or ISO8859-1 or
> 
> almost anything else:-)
> 
> 
> 
> As you can see, there's a lot of encoding/decoding going on behind
> 
> the scenes even in this superficially simple example.
> 
> 
> 
> | If i remove
> 
> | the encode('utf-8') part in metrites.py, the webpage will not show
> 
> | anything at all...
> 
> 
> 
> Ah, but data will be being output. The print() function _will_ be
> 
> writing "data" out in some form.  I suggest you remove the .encode()
> 
> and then examine the _source_ text of the web page, not its visible
> 
> form.
> 
> 
> 
> So: remove .encode(), reload the web page, "view page source"
> 
> (depends on your browser, it is ctrl-U in Firefox ((Cmd-U in firefox
> 
> on a Mac))).
> 
> 
> 
> I think a lot of the issue you have in this thread is that your
> 
> page is too complex. Make another page to do the same thing, and
> 
> start with nothing. Add stuff to it a single item at a time until
> 
> the page behaves incorrectly. Then you will know the exact item of
> 
> code that introduced the issue. And then that single item can be
> 
> examined in detail for the decode/encode issues.
> 
> 
> 
> The other issue in the thread is that people losing patience get
> 
> snarky. Respond only to the technical content. If a message is only
> 
> snarky, _ignore_ it. People like the last word; let them have it
> 
> and you won't get sidetracked into arguments.
> 
> 
> 
> Cheers,
> 
> -- 
> 
> Cameron Simpson <cs@zip.com.au>
> 
> 
> 
> PCs are like a submarine, it will work fine till you open Windows. - zollie101

First of all thank you very much Cameron for your detailed help and effort to write to me:

It seems another issue had happened without my knowledge, i was uploading stuff at /root/public_html/cgi-bin instead of /home/nikos/public_html/cgi-bin.

I realized that when i deliberately made error to metrites.py scropt and i got still the same page.

Ookey after that is corrected, i then tried the plain solution and i got this response back form the shell:

Traceback (most recent call last):
  File "metrites.py", line 213, in &lt;module&gt;
    htmldata = f.read()
  File "/root/.local/lib/python2.7/lib/python3.3/encodings/iso8859_7.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0xae in position 47: character maps to &lt;undefined&gt;

then i switched to:

		with open('/home/nikos/www/' + page, encoding='utf-8') as f:
			htmldata = f.read()

and i got no error at all, just pure run *from the shell*!
But i get internal server error when i try to run the webpage from the browser(Chrome).

So, can you tell me please where can i find the apache error log so to display here please?

Apcher error_log is always better than running 'python3 metrites.py' because even if the python script has no error apache will also display more web related things?

Thank you Cameron.

[toc] | [prev] | [next] | [standalone]

#43507

On 12Apr2013 21:50, nagia.retsina@gmail.com <nagia.retsina@gmail.com> wrote:
| Ookey after that is corrected, i then tried the plain solution and i got this response back form the shell:
| 
| Traceback (most recent call last):
|   File "metrites.py", line 213, in &lt;module&gt;
|     htmldata = f.read()
|   File "/root/.local/lib/python2.7/lib/python3.3/encodings/iso8859_7.py", line 23, in decode
|     return codecs.charmap_decode(input,self.errors,decoding_table)[0]
| UnicodeDecodeError: 'charmap' codec can't decode byte 0xae in position 47: character maps to &lt;undefined&gt;
| 
| then i switched to:
| 
| 		with open('/home/nikos/www/' + page, encoding='utf-8') as f:
| 			htmldata = f.read()
| 
| and i got no error at all, just pure run *from the shell*!

Ok, so you need to specify utf-8 to decode the file. Good.

| But i get internal server error when i try to run the webpage from the browser(Chrome).

That is standard for a CGI script that does not complete successfully.

| So, can you tell me please where can i find the apache error log so to display here please?

That depends on the install. Have a look in /var/log/apache or similar.
Otherwise you need to find the httpd.conf for the apache and look
for its log coniguration settings.

| Apcher error_log is always better than running 'python3 metrites.py' because even if the python script has no error apache will also display more web related things?

The error log is where error messages from CGI scripts go. And other error messages.
It is very useful when testing CGI scripts.

Of course, it's best to work out as much as possible from the command
line first; you have much more direct control and access to errors
there. That only gets you so far though; the environment the CGI
script runs in is not the same as your command line, and some
different behaviour can come from this.

BTW, are you sure python3 is running your CGI script?
Also, the CGI script may not be running as you, but as the apache user.
In that case, it may fail if it does not has permission to access various
files owned by you.

Anyway, you need to see the error messages to work this out.

If you can't find the error log you can divert stderr at the
start of your python program:

  sys.stderr = open('/home/nikos/cgi.err.out', 'a')

and watch that in a shell:

  tail -f cgi.err.out

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

If you 'aint falling off, you ar'nt going hard enough.  - Fred Gassit

[toc] | [prev] | [next] | [standalone]

#43513

Τη Σάββατο, 13 Απριλίου 2013 1:28:07 μ.μ. UTC+3, ο χρήστης Cameron Simpson έγραψε:
> On 12Apr2013 21:50, nagia.retsina@gmail.com <nagia.retsina@gmail.com> wrote:
> 
> | Ookey after that is corrected, i then tried the plain solution and i got this response back form the shell:
> 
> | 
> 
> | Traceback (most recent call last):
> 
> |   File "metrites.py", line 213, in &lt;module&gt;
> 
> |     htmldata = f.read()
> 
> |   File "/root/.local/lib/python2.7/lib/python3.3/encodings/iso8859_7.py", line 23, in decode
> 
> |     return codecs.charmap_decode(input,self.errors,decoding_table)[0]
> 
> | UnicodeDecodeError: 'charmap' codec can't decode byte 0xae in position 47: character maps to &lt;undefined&gt;
> 
> | 
> 
> | then i switched to:
> 
> | 
> 
> | 		with open('/home/nikos/www/' + page, encoding='utf-8') as f:
> 
> | 			htmldata = f.read()
> 
> | 
> 
> | and i got no error at all, just pure run *from the shell*!
> 
> 
> 
> Ok, so you need to specify utf-8 to decode the file. Good.
> 
> 
> 
> | But i get internal server error when i try to run the webpage from the browser(Chrome).
> 
> 
> 
> That is standard for a CGI script that does not complete successfully.
> 
> 
> 
> | So, can you tell me please where can i find the apache error log so to display here please?
> 
> 
> 
> That depends on the install. Have a look in /var/log/apache or similar.
> 
> Otherwise you need to find the httpd.conf for the apache and look
> 
> for its log coniguration settings.
> 
> 
> 
> | Apcher error_log is always better than running 'python3 metrites.py' because even if the python script has no error apache will also display more web related things?
> 
> 
> 
> The error log is where error messages from CGI scripts go. And other error messages.
> 
> It is very useful when testing CGI scripts.
> 
> 
> 
> Of course, it's best to work out as much as possible from the command
> 
> line first; you have much more direct control and access to errors
> 
> there. That only gets you so far though; the environment the CGI
> 
> script runs in is not the same as your command line, and some
> 
> different behaviour can come from this.
> 
> 
> 
> BTW, are you sure python3 is running your CGI script?
> 
> Also, the CGI script may not be running as you, but as the apache user.
> 
> In that case, it may fail if it does not has permission to access various
> 
> files owned by you.
> 
> 
> 
> Anyway, you need to see the error messages to work this out.
> 
> 
> 
> If you can't find the error log you can divert stderr at the
> 
> start of your python program:
> 
> 
> 
>   sys.stderr = open('/home/nikos/cgi.err.out', 'a')
> 
> 
> 
> and watch that in a shell:
> 
> 
> 
>   tail -f cgi.err.out
> 
> 
> 
> Cheers,
> 
> -- 
> 
> Cameron Simpson <cs@zip.com.au>
> 
> 
> 
> If you 'aint falling off, you ar'nt going hard enough.  - Fred Gassit

root@macgyver [/home/nikos/public_html/cgi-bin]# ls ../cgi.err.out 
../cgi.err.out
root@macgyver [/home/nikos/public_html/cgi-bin]# cat ../cgi.err.out 
root@macgyver [/home/nikos/public_html/cgi-bin]# 

Also i have foudn the error log and i tried to view it but it was empty and then i removed it and then run the script both from shell and broswer but it didnt reappeared.

root@macgyver [/home/nikos/public_html/cgi-bin]# cat /var/log/httpd/error_log
cat: /var/log/httpd/error_log: No such file or directory
root@macgyver [/home/nikos/public_html/cgi-bin]# 

Maybe its somehtign wron with my enviroment?
Should we check the Apache and CGI enviroment somehow and also make sure as you say that *I* run the CGI scripts and not user 'Apache' ?

Tell me what commands i should issues please and i will display the output to you.

Thank you Cameron, for helpimg me. Somehow the script doesnt seem to be the issue in  my VPS.

[toc] | [prev] | [next] | [standalone]

#43519

On Sun, Apr 14, 2013 at 12:16 AM,  <nagia.retsina@gmail.com> wrote:
> Also i have foudn the error log and i tried to view it but it was empty and then i removed it and then run the script both from shell and broswer but it didnt reappeared.
>
> root@macgyver [/home/nikos/public_html/cgi-bin]# cat /var/log/httpd/error_log
> cat: /var/log/httpd/error_log: No such file or directory
> root@macgyver [/home/nikos/public_html/cgi-bin]#

https://www.google.com/search?q=log+file+rotation

ChrisA

[toc] | [prev] | [next] | [standalone]

#43556

On 13Apr2013 07:16, nagia.retsina@gmail.com <nagia.retsina@gmail.com> wrote:
| root@macgyver [/home/nikos/public_html/cgi-bin]# ls ../cgi.err.out 
| ../cgi.err.out

I prefer "ls -ld" myself.

| root@macgyver [/home/nikos/public_html/cgi-bin]# cat ../cgi.err.out 
| 
| Also i have foudn the error log and i tried to view it but it was
| empty and then i removed it and then run the script both from shell
| and broswer but it didnt reappeared.

Never remove it. It is only created by the web server at startup or log rotation time. So now you need to restart the apache to get it back.

Just open a spare terminal and run:

  tail -f /var/log/httpd/error_log

| Should we check the Apache and CGI enviroment somehow and also
| make sure as you say that *I* run the CGI scripts and not user
| 'Apache' ?

Well, it is helpful to know. if the CGI script tries to write any data to files,
if it runs as a different user it will need different permissions on the files.

| Tell me what commands i should issues please and i will display the output to you.

I would be tempter to wrap the CGI script in a shell script.
Suppose your script is named foo.py. 

You can move the script to foo-py and make a shell script called "foo.py" looking like this:

  #!/bin/sh
  exec 2>>/home/nikos/cgi.err.out
  echo "$0 $*" >&2
  id >&2
  env | sort >&2
  set -x
  exec /full/path/to/foo-py ${1+"$@"}

and make sure it, like the original, is readable and executable:

  chmod a+rx foo.py foo-py

Make sure cgi.err.out is publicly writable (in case the apache is
not running the CGIs are you):

  chmod a+w cgi.err.out

Then:

  tail -f cgi.err.out

in a spare window.

Then try the script.

It should transcribe information about the script's user and
environment and also catch errors.

This should help in debugging.

Cheers,
-- 

I die. I have a terrible fever in my head and it gets hotter and hotter until
my head is a fire, a forge, a star. I set the world on fire and all die. O the
embarrassment.  - Joe Haldeman, _A !Tangled Web_

[toc] | [prev] | [next] | [standalone]

#43514

Τη Σάββατο, 13 Απριλίου 2013 1:28:07 μ.μ. UTC+3, ο χρήστης Cameron Simpson έγραψε:
> On 12Apr2013 21:50, nagia.retsina@gmail.com <nagia.retsina@gmail.com> wrote:
> 
> | Ookey after that is corrected, i then tried the plain solution and i got this response back form the shell:
> 
> | 
> 
> | Traceback (most recent call last):
> 
> |   File "metrites.py", line 213, in &lt;module&gt;
> 
> |     htmldata = f.read()
> 
> |   File "/root/.local/lib/python2.7/lib/python3.3/encodings/iso8859_7.py", line 23, in decode
> 
> |     return codecs.charmap_decode(input,self.errors,decoding_table)[0]
> 
> | UnicodeDecodeError: 'charmap' codec can't decode byte 0xae in position 47: character maps to &lt;undefined&gt;
> 
> | 
> 
> | then i switched to:
> 
> | 
> 
> | 		with open('/home/nikos/www/' + page, encoding='utf-8') as f:
> 
> | 			htmldata = f.read()
> 
> | 
> 
> | and i got no error at all, just pure run *from the shell*!
> 
> 
> 
> Ok, so you need to specify utf-8 to decode the file. Good.
> 
> 
> 
> | But i get internal server error when i try to run the webpage from the browser(Chrome).
> 
> 
> 
> That is standard for a CGI script that does not complete successfully.
> 
> 
> 
> | So, can you tell me please where can i find the apache error log so to display here please?
> 
> 
> 
> That depends on the install. Have a look in /var/log/apache or similar.
> 
> Otherwise you need to find the httpd.conf for the apache and look
> 
> for its log coniguration settings.
> 
> 
> 
> | Apcher error_log is always better than running 'python3 metrites.py' because even if the python script has no error apache will also display more web related things?
> 
> 
> 
> The error log is where error messages from CGI scripts go. And other error messages.
> 
> It is very useful when testing CGI scripts.
> 
> 
> 
> Of course, it's best to work out as much as possible from the command
> 
> line first; you have much more direct control and access to errors
> 
> there. That only gets you so far though; the environment the CGI
> 
> script runs in is not the same as your command line, and some
> 
> different behaviour can come from this.
> 
> 
> 
> BTW, are you sure python3 is running your CGI script?
> 
> Also, the CGI script may not be running as you, but as the apache user.
> 
> In that case, it may fail if it does not has permission to access various
> 
> files owned by you.
> 
> 
> 
> Anyway, you need to see the error messages to work this out.
> 
> 
> 
> If you can't find the error log you can divert stderr at the
> 
> start of your python program:
> 
> 
> 
>   sys.stderr = open('/home/nikos/cgi.err.out', 'a')
> 
> 
> 
> and watch that in a shell:
> 
> 
> 
>   tail -f cgi.err.out
> 
> 
> 
> Cheers,
> 
> -- 
> 
> Cameron Simpson <cs@zip.com.au>
> 
> 
> 
> If you 'aint falling off, you ar'nt going hard enough.  - Fred Gassit

root@macgyver [/home/nikos/public_html/cgi-bin]# ls ../cgi.err.out 
../cgi.err.out
root@macgyver [/home/nikos/public_html/cgi-bin]# cat ../cgi.err.out 
root@macgyver [/home/nikos/public_html/cgi-bin]# 

Also i have foudn the error log and i tried to view it but it was empty and then i removed it and then run the script both from shell and broswer but it didnt reappeared.

root@macgyver [/home/nikos/public_html/cgi-bin]# cat /var/log/httpd/error_log
cat: /var/log/httpd/error_log: No such file or directory
root@macgyver [/home/nikos/public_html/cgi-bin]# 

Maybe its somehtign wron with my enviroment?
Should we check the Apache and CGI enviroment somehow and also make sure as you say that *I* run the CGI scripts and not user 'Apache' ?

Tell me what commands i should issues please and i will display the output to you.

Thank you Cameron, for helpimg me. Somehow the script doesnt seem to be the issue in  my VPS.

[toc] | [prev] | [next] | [standalone]

Page 2 of 5 — ← Prev page 1 [2] 3 4 5  Next page →

Back to top | Article view | comp.lang.python

csiph-web