Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #85070 > unrolled thread

Re: How to write a non blocking SimpleHTTPRequestHandler ?

Started byAmirouche Boubekki <amirouche.boubekki@gmail.com>
First post2015-02-02 10:07 +0000
Last post2015-02-03 12:10 +0000
Articles 20 on this page of 21 — 7 participants

Back to article view | Back to comp.lang.python


Contents

  Re: How to write a non blocking SimpleHTTPRequestHandler ? Amirouche Boubekki <amirouche.boubekki@gmail.com> - 2015-02-02 10:07 +0000
    Re: How to write a non blocking SimpleHTTPRequestHandler ? Yassine Chaouche <yacinechaouche@yahoo.com> - 2015-02-03 01:08 -0800
      Re: How to write a non blocking SimpleHTTPRequestHandler ? Marko Rauhamaa <marko@pacujo.net> - 2015-02-03 11:27 +0200
        Re: How to write a non blocking SimpleHTTPRequestHandler ? Yassine Chaouche <yacinechaouche@yahoo.com> - 2015-02-03 02:47 -0800
          Re: How to write a non blocking SimpleHTTPRequestHandler ? Chris Angelico <rosuav@gmail.com> - 2015-02-03 21:54 +1100
            Re: How to write a non blocking SimpleHTTPRequestHandler ? Yassine Chaouche <yacinechaouche@yahoo.com> - 2015-02-03 03:04 -0800
              Re: How to write a non blocking SimpleHTTPRequestHandler ? Chris Angelico <rosuav@gmail.com> - 2015-02-03 22:07 +1100
                Re: How to write a non blocking SimpleHTTPRequestHandler ? Yassine Chaouche <yacinechaouche@yahoo.com> - 2015-02-03 03:23 -0800
                Re: How to write a non blocking SimpleHTTPRequestHandler ? Marko Rauhamaa <marko@pacujo.net> - 2015-02-03 13:35 +0200
                  Re: How to write a non blocking SimpleHTTPRequestHandler ? Yassine Chaouche <yacinechaouche@yahoo.com> - 2015-02-03 04:56 -0800
                    Re: How to write a non blocking SimpleHTTPRequestHandler ? Marko Rauhamaa <marko@pacujo.net> - 2015-02-03 15:45 +0200
                      Re: How to write a non blocking SimpleHTTPRequestHandler ? Chris Angelico <rosuav@gmail.com> - 2015-02-04 01:03 +1100
                        Re: How to write a non blocking SimpleHTTPRequestHandler ? Marko Rauhamaa <marko@pacujo.net> - 2015-02-03 16:20 +0200
                      Re: How to write a non blocking SimpleHTTPRequestHandler ? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2015-02-03 20:25 -0500
                    Re: How to write a non blocking SimpleHTTPRequestHandler ? Chris Angelico <rosuav@gmail.com> - 2015-02-04 00:21 +1100
                    Re: How to write a non blocking SimpleHTTPRequestHandler ? Amirouche Boubekki <amirouche.boubekki@gmail.com> - 2015-02-03 14:09 +0000
                      Re: How to write a non blocking SimpleHTTPRequestHandler ? Yassine Chaouche <yacinechaouche@yahoo.com> - 2015-02-03 06:50 -0800
                        Re: How to write a non blocking SimpleHTTPRequestHandler ? Filadelfo Fiamma <philosganga@gmail.com> - 2015-02-03 16:03 +0100
                        Re: How to write a non blocking SimpleHTTPRequestHandler ? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-02-03 16:39 +0000
                      Re: How to write a non blocking SimpleHTTPRequestHandler ? Yassine Chaouche <yacinechaouche@yahoo.com> - 2015-02-08 06:13 -0800
              Re: How to write a non blocking SimpleHTTPRequestHandler ? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-02-03 12:10 +0000

Page 1 of 2  [1] 2  Next page →


#85070 — Re: How to write a non blocking SimpleHTTPRequestHandler ?

FromAmirouche Boubekki <amirouche.boubekki@gmail.com>
Date2015-02-02 10:07 +0000
SubjectRe: How to write a non blocking SimpleHTTPRequestHandler ?
Message-ID<mailman.18397.1422871634.18130.python-list@python.org>

[Multipart message — attachments visible in raw view] — view raw

On Mon Feb 02 2015 at 10:55:26 AM <yacinechaouche@yahoo.com.dmarc.invalid>
wrote:

> I wrote a little script that acts like a proxy, you just give it a URL and
> it will fetch the content and display it back to you.
>
> For some reason, this proxy blocks sometimes and refuses to serve any new
> queries. The script still runs, but it seems like it's stuck somewhere.
>
> When I strace it to see what it's doing, I find it hanging on this
> instruction :
> root@backup[10.10.10.21] ~/SCRIPTS/INFOMANIAK # strace -fp 6918
> Process 6918 attached - interrupt to quit
> recvfrom(6,
> ^CProcess 6918 detached
> root@backup[10.10.10.21] ~/SCRIPTS/INFOMANIAK #
>
> I read in the SimpleHTTPServer source code that one can inherit from the
> SocketServer.TrheadingMixIn mixin to enable a threaded server to handle
> multiple requests at a time instead of just one (thinking maybe that's what
> was blocking it). However, it seems like it has nothing to do with my
> problem. What I need to do is not only handle multiple requests at a time,
> but more importantly to make the request handler non-blocking.
>
> Any ideas ? here's come code :
>
> import SimpleHTTPServer
> import BaseHTTPServer
> import SocketServer
> import requests
>
> class Handler(SocketServer.ThreadingMixIn,SimpleHTTPServer.SimpleH
> TTPRequestHandler):
>     def do_GET(self):
>         self.send_response(200)
>         self.send_header('Content-Type', 'text/html')
>         self.end_headers()
>         # self.path will contain a URL to be fetched by my proxy
>         self.wfile.write(getFlux(self.path.lstrip("/")))
>
> session = requests.Session()
> IP,PORT = "MY_IP_HERE",8080
>
> def getFlux(url):
>     response  = session.get(url)
>     s = response.text
>     return s
>
> server = BaseHTTPServer.HTTPServer((IP,PORT),Handler)
> server.serve_forever()
>


Your code seem perfectly fine. I had some trouble with py3's http.server
with IE10 (in a virtualbox...), I put together a small server script
similar to http.server that doesn't hang up on microsoft. It works with
ayncio. It's not ready to serve big files, but hopefully you can fix that.


HTH



> Thank you.
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [next] | [standalone]


#85124

FromYassine Chaouche <yacinechaouche@yahoo.com>
Date2015-02-03 01:08 -0800
Message-ID<47031e69-e94d-4257-8c7d-e7c00a543634@googlegroups.com>
In reply to#85070
Thank you Amirouch. I was hoping to use something very simple and already provided by the standard library. If I can fix the script, the better. If the script can't be fixed, then I'll switch to another library (I already have one in mind).

[toc] | [prev] | [next] | [standalone]


#85127

FromMarko Rauhamaa <marko@pacujo.net>
Date2015-02-03 11:27 +0200
Message-ID<874mr3ibve.fsf@elektro.pacujo.net>
In reply to#85124
Yassine Chaouche <yacinechaouche@yahoo.com>:

> I was hoping to use something very simple and already provided by the
> standard library.

The standard library and nonblocking can't be used in the same sentence.

That is, unless and until you go asyncio.


Marko

[toc] | [prev] | [next] | [standalone]


#85130

FromYassine Chaouche <yacinechaouche@yahoo.com>
Date2015-02-03 02:47 -0800
Message-ID<c0e17d52-4978-48fd-8848-30426ea42e14@googlegroups.com>
In reply to#85127
On Tuesday, February 3, 2015 at 10:27:11 AM UTC+1, Marko Rauhamaa wrote:
> The standard library and nonblocking can't be used in the same sentence.

Thanks Marko. It's a lost cause then. I am thinking about switching to one of the following :
 - CherryPy
 - Bottle
 - circuits
 - Quixote
 - Weblayer.

If anybody has a pointer to an already made comparison I'm a taker.

Thanks for your help !

[toc] | [prev] | [next] | [standalone]


#85132

FromChris Angelico <rosuav@gmail.com>
Date2015-02-03 21:54 +1100
Message-ID<mailman.18420.1422960897.18130.python-list@python.org>
In reply to#85130
On Tue, Feb 3, 2015 at 9:47 PM, Yassine Chaouche
<yacinechaouche@yahoo.com.dmarc.invalid> wrote:
> On Tuesday, February 3, 2015 at 10:27:11 AM UTC+1, Marko Rauhamaa wrote:
>> The standard library and nonblocking can't be used in the same sentence.
>
> Thanks Marko. It's a lost cause then.

You trimmed out the part where he mentioned asyncio. :)

ChrisA

[toc] | [prev] | [next] | [standalone]


#85133

FromYassine Chaouche <yacinechaouche@yahoo.com>
Date2015-02-03 03:04 -0800
Message-ID<b6cb0fcb-9934-41b6-a8f6-902d26f2588c@googlegroups.com>
In reply to#85132
> > Thanks Marko. It's a lost cause then.
> 
> You trimmed out the part where he mentioned asyncio. :)
> 
> ChrisA

IIRC asyncio is python 3 only and I'm not ready yet to make the leap.

[toc] | [prev] | [next] | [standalone]


#85134

FromChris Angelico <rosuav@gmail.com>
Date2015-02-03 22:07 +1100
Message-ID<mailman.18421.1422961678.18130.python-list@python.org>
In reply to#85133
On Tue, Feb 3, 2015 at 10:04 PM, Yassine Chaouche
<yacinechaouche@yahoo.com.dmarc.invalid> wrote:
>> > Thanks Marko. It's a lost cause then.
>>
>> You trimmed out the part where he mentioned asyncio. :)
>>
>> ChrisA
>
> IIRC asyncio is python 3 only and I'm not ready yet to make the leap.

Then you're stuck with whatever you have, because the Py2 standard
library isn't being expanded any. Why not make the leap? Py3 has a lot
of advantages over Py2.

ChrisA

[toc] | [prev] | [next] | [standalone]


#85136

FromYassine Chaouche <yacinechaouche@yahoo.com>
Date2015-02-03 03:23 -0800
Message-ID<2b73e0ba-c50d-4abe-af36-ff586e643450@googlegroups.com>
In reply to#85134
Thanks Chris, it's only a matter of time, I'll eventually make the transition to python3 when I'll learn it well enough.

[toc] | [prev] | [next] | [standalone]


#85139

FromMarko Rauhamaa <marko@pacujo.net>
Date2015-02-03 13:35 +0200
Message-ID<87r3u7grd3.fsf@elektro.pacujo.net>
In reply to#85134
Chris Angelico <rosuav@gmail.com>:

> On Tue, Feb 3, 2015 at 10:04 PM, Yassine Chaouche
>> IIRC asyncio is python 3 only and I'm not ready yet to make the leap.
>
> Then you're stuck with whatever you have, because the Py2 standard
> library isn't being expanded any. Why not make the leap? Py3 has a lot
> of advantages over Py2.

I'm all for Py3, but I'm not ready to conclude asyncio is the way to go.
The coroutines haven't won me over. The programming model is quite messy
and simply weird.

So far I've been happy with select.epoll(), socket.socket() and ten
fingers.


Marko

[toc] | [prev] | [next] | [standalone]


#85144

FromYassine Chaouche <yacinechaouche@yahoo.com>
Date2015-02-03 04:56 -0800
Message-ID<e6d081df-1db0-41a3-905c-e3c2845d5dda@googlegroups.com>
In reply to#85139
On Tuesday, February 3, 2015 at 12:35:32 PM UTC+1, Marko Rauhamaa wrote:

> So far I've been happy with select.epoll(), socket.socket() and ten
> fingers.
> Marko

There's already software written to take care of much of the HTTP stuff protocol stuff, the headers etc. I wouldn't rewrite it. I prefer to monkey patch parts of existing code rather then rewrite all of it.

But your comment is interesting because, as I understand it, a non-blocking web server is simply a matter of setting timeouts on sockets, catch the exceptions and move on. I don't know why wouldn't that be possible with python stdlib ?


> The standard library and nonblocking can't be used in the same sentence. 

?

[toc] | [prev] | [next] | [standalone]


#85145

FromMarko Rauhamaa <marko@pacujo.net>
Date2015-02-03 15:45 +0200
Message-ID<87iofjglbg.fsf@elektro.pacujo.net>
In reply to#85144
Yassine Chaouche <yacinechaouche@yahoo.com>:

> On Tuesday, February 3, 2015 at 12:35:32 PM UTC+1, Marko Rauhamaa wrote:
>> So far I've been happy with select.epoll(), socket.socket() and ten
>> fingers.
>
> [...]
>
> But your comment is interesting because, as I understand it, a
> non-blocking web server is simply a matter of setting timeouts on
> sockets, catch the exceptions and move on.

Now I think you might have some misconceptions about nonblocking
networking I/O. Nonblocking I/O is done using asynchronous, or
event-driven, programming. Your code reacts to external stimuli, never
blocking, mostly just sleeping. The reactions are defined in callback
routings, aka listeners, aka event handlers.

> I don't know why wouldn't that be possible with python stdlib ?

It is possible using the low-level facilities. However, the traditional
high-level facilities are built on multithreading, which (as a rule) is
based on blocking I/O.


Marko

[toc] | [prev] | [next] | [standalone]


#85147

FromChris Angelico <rosuav@gmail.com>
Date2015-02-04 01:03 +1100
Message-ID<mailman.18424.1422972220.18130.python-list@python.org>
In reply to#85145
On Wed, Feb 4, 2015 at 12:45 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> But your comment is interesting because, as I understand it, a
>> non-blocking web server is simply a matter of setting timeouts on
>> sockets, catch the exceptions and move on.
>
> Now I think you might have some misconceptions about nonblocking
> networking I/O. Nonblocking I/O is done using asynchronous, or
> event-driven, programming. Your code reacts to external stimuli, never
> blocking, mostly just sleeping. The reactions are defined in callback
> routings, aka listeners, aka event handlers.

Not strictly true - that's just one convenient way of doing things. A
callback/event-handler structure lets you write a bunch of listeners
that coexist effortlessly, but it's not the only way to do
non-blocking I/O, and it's certainly not an intrinsic part of the
concept.

That said, though, it is a VERY convenient way to lay things out in
the code. The Pike system I offered, and most of the older
multiplexed-I/O systems I've used, did work that way. It just isn't
something that non-blocking necessarily implies.

>> I don't know why wouldn't that be possible with python stdlib ?
>
> It is possible using the low-level facilities. However, the traditional
> high-level facilities are built on multithreading, which (as a rule) is
> based on blocking I/O.

Multithreading is another way to cope with the same problem of wanting
to deal with different sockets on a single CPU, but I don't think it's
inherently a part of any of Python's own high-level facilities - not
that I can think of, at least? However, in terms of common programming
models, yes, multithreading+blocking I/O is an effective way to write
code, and will therefore be commonly used.

I wish more people had grown up on OS/2 instead of (or as well as)
Windows or Unix. Threading is not such a bugbear as a lot of people
seem to think. Yes, some platforms have traditionally had poor
implementations, and to be sure, you don't want to mix threading and
forking without a *lot* of care, but threads aren't inherently bad.
They're a useful tool in the toolbox. Sometimes non-blocking I/O is
the right thing to do; sometimes threads suit the problem better;
other times, something else again.

ChrisA

[toc] | [prev] | [next] | [standalone]


#85151

FromMarko Rauhamaa <marko@pacujo.net>
Date2015-02-03 16:20 +0200
Message-ID<87a90vgjpj.fsf@elektro.pacujo.net>
In reply to#85147
Chris Angelico <rosuav@gmail.com>:

> Threading is not such a bugbear as a lot of people
> seem to think. Yes, some platforms have traditionally had poor
> implementations

Java has excellent multithreading facilities. Still, I have seen
seasoned Java developers commit atrocious crimes against thread-safety
that are impossible to troubleshoot and repair afterwards.

There are ways to manage the complications of multithreading, but no
universal practices are agreed upon so you can't count on other/legacy
software to obey your policies or your software obey theirs. And even
under the best guidelines there are seemingly intractable cornercases
where you feel helplessly lost.


Marko

[toc] | [prev] | [next] | [standalone]


#85188

FromDennis Lee Bieber <wlfraed@ix.netcom.com>
Date2015-02-03 20:25 -0500
Message-ID<mailman.18452.1423013153.18130.python-list@python.org>
In reply to#85145
On Tue, 03 Feb 2015 15:45:55 +0200, Marko Rauhamaa <marko@pacujo.net>
declaimed the following:

>Now I think you might have some misconceptions about nonblocking
>networking I/O. Nonblocking I/O is done using asynchronous, or
>event-driven, programming. Your code reacts to external stimuli, never
>blocking, mostly just sleeping. The reactions are defined in callback
>routings, aka listeners, aka event handlers.
>
	In my world, "sleep" is a blocking operation -- the process is blocked
from execution until the sleep expires or is otherwise canceled.

	So a socket.select() with timeout is a blocking operation -- just that
it is released when any of the multiple items (read sockets, write sockets,
timeout) is satisfied.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]


#85146

FromChris Angelico <rosuav@gmail.com>
Date2015-02-04 00:21 +1100
Message-ID<mailman.18423.1422971344.18130.python-list@python.org>
In reply to#85144
On Tue, Feb 3, 2015 at 11:56 PM, Yassine Chaouche
<yacinechaouche@yahoo.com.dmarc.invalid> wrote:
> But your comment is interesting because, as I understand it, a non-blocking web server is simply a matter of setting timeouts on sockets, catch the exceptions and move on. I don't know why wouldn't that be possible with python stdlib ?
>

Not really. You could, in theory, set very low timeouts and then poll
everything, but it's not efficient. What you want to do is say to the
system "Hey, see all these sockets? Let me know when *any one of them*
has stuff for me", where "stuff" would be a new connected client if
it's a listening socket, or some data written if it's a connected
socket; and you might need to check if there's room to write more
data, too, which you can do with the same syscall.

The key here is that you have a long timeout on the meta-event "any
one of these being ready". That's not simply a matter of setting
socket timeouts; you need a way to handle the meta-event, and that's
something along the lines of select():

http://linux.die.net/man/2/select

Other languages have inbuilt asynchronous I/O handlers; eg Pike
handles this fairly well, and I've made some use of it with a generic
networking system:

https://github.com/Rosuav/Hogan

Basically, you spin up a server with any number of listening sockets,
each of which can talk to any number of connected clients, and all of
those sockets get smoothly multiplexed on a single thread. Lots of
other languages have similar facilities. Python 2.x doesn't have
anything of that nature; Python 3's asyncio is exactly that.

ChrisA

[toc] | [prev] | [next] | [standalone]


#85150

FromAmirouche Boubekki <amirouche.boubekki@gmail.com>
Date2015-02-03 14:09 +0000
Message-ID<mailman.18427.1422973045.18130.python-list@python.org>
In reply to#85144

[Multipart message — attachments visible in raw view] — view raw

> The standard library and nonblocking can't be used in the same sentence.

python 2.x stdlib has no high level support of *async* code. There is
trollius library that ports asyncio to py2 though.

I was a bit quick in my first anwser. What you want is to prevent the
socket to wait indefinetly for data (based on strace output) which is done
with socket.setblocking/settimeout [1]. asynchronous (asyncio) is something
else, and you would still need to handle blocking I think.

There is contentbrowser [2] which is somekind of web proxy.

IMO, python 2 -> python 3 is not a big leap. Some things are better in
python 3.


[1] https://docs.python.org/2/library/socket.html#socket.socket.setblocking
[2] https://bitbucket.org/david/contentbrowser/src





On Tue Feb 03 2015 at 2:00:27 PM Yassine Chaouche
<yacinechaouche@yahoo.com.dmarc.invalid> wrote:

> On Tuesday, February 3, 2015 at 12:35:32 PM UTC+1, Marko Rauhamaa wrote:
>
> > So far I've been happy with select.epoll(), socket.socket() and ten
> > fingers.
> > Marko
>
> There's already software written to take care of much of the HTTP stuff
> protocol stuff, the headers etc. I wouldn't rewrite it. I prefer to monkey
> patch parts of existing code rather then rewrite all of it.
>
> But your comment is interesting because, as I understand it, a
> non-blocking web server is simply a matter of setting timeouts on sockets,
> catch the exceptions and move on. I don't know why wouldn't that be
> possible with python stdlib ?
>
>
> > The standard library and nonblocking can't be used in the same sentence.
>
> ?
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]


#85155

FromYassine Chaouche <yacinechaouche@yahoo.com>
Date2015-02-03 06:50 -0800
Message-ID<3b558129-df61-4a3d-9cca-17e5340c33a3@googlegroups.com>
In reply to#85150
On Tuesday, February 3, 2015 at 3:17:37 PM UTC+1, Amirouche Boubekki wrote:
> What you want is to prevent the socket to wait indefinetly for data (based on strace output) which is done with socket.setblocking/settimeout [1]

Exactly ! thanks for taking time to reading my original post a second time. Maybe I didn't express my problem well enough and the discussion has maybe drifted to an XYProblem type discussion :)

I was about to say that most webframeworks for python already take care of the problem I'm facing, so I certainly confused people with improper formulation.

> There is contentbrowser [2] which is somekind of web proxy.

It's using werkzeug and it seems to be a full web application. I just want something that is right above sockets and selects. All that my script is doing is grabbing some json data from an external url provided by the browser and return a formatted page showing the json data in human-readable HTML pages. Nothing fancy.

[toc] | [prev] | [next] | [standalone]


#85156

FromFiladelfo Fiamma <philosganga@gmail.com>
Date2015-02-03 16:03 +0100
Message-ID<mailman.18431.1422975829.18130.python-list@python.org>
In reply to#85155

[Multipart message — attachments visible in raw view] — view raw

I think You can try the asyncore lib:
https://docs.python.org/2/library/asyncore.html


2015-02-03 15:50 GMT+01:00 Yassine Chaouche <
yacinechaouche@yahoo.com.dmarc.invalid>:

> On Tuesday, February 3, 2015 at 3:17:37 PM UTC+1, Amirouche Boubekki wrote:
> > What you want is to prevent the socket to wait indefinetly for data
> (based on strace output) which is done with socket.setblocking/settimeout
> [1]
>
> Exactly ! thanks for taking time to reading my original post a second
> time. Maybe I didn't express my problem well enough and the discussion has
> maybe drifted to an XYProblem type discussion :)
>
> I was about to say that most webframeworks for python already take care of
> the problem I'm facing, so I certainly confused people with improper
> formulation.
>
> > There is contentbrowser [2] which is somekind of web proxy.
>
> It's using werkzeug and it seems to be a full web application. I just want
> something that is right above sockets and selects. All that my script is
> doing is grabbing some json data from an external url provided by the
> browser and return a formatted page showing the json data in human-readable
> HTML pages. Nothing fancy.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Filadelfo Fiamma
mail: philosganga@gmail.com

[toc] | [prev] | [next] | [standalone]


#85162

FromMark Lawrence <breamoreboy@yahoo.co.uk>
Date2015-02-03 16:39 +0000
Message-ID<mailman.18434.1422981552.18130.python-list@python.org>
In reply to#85155
On 03/02/2015 15:03, Filadelfo Fiamma wrote:
> I think You can try the asyncore lib:
> https://docs.python.org/2/library/asyncore.html
>

People can try it but it's effectively deprecated, with its partner 
asynchat, in favour of asyncio.

Also please don't top post here, thank you.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]


#85359

FromYassine Chaouche <yacinechaouche@yahoo.com>
Date2015-02-08 06:13 -0800
Message-ID<d2980225-49f0-4cee-a03e-885029e5f422@googlegroups.com>
In reply to#85150
On Tuesday, February 3, 2015 at 3:17:37 PM UTC+1, Amirouche Boubekki wrote:
> What you want is to prevent the socket to wait indefinetly for data (based on strace output) which is done with socket.setblocking/settimeout [1]. asynchronous (asyncio) is something else, and you would still need to handle blocking I think.

I have installed Faulthandler, a beautiful tool written by Victor "Haypo" Stinner, and thanks to it I could determine precisely where the program hangs. It is in ssl.py::SSLSocket::read


    def read(self, len=1024):

        """Read up to LEN bytes and return them.                                                                                                                                                                                 
        Return zero-length string on EOF."""

        try:
>            return self._sslobj.read(len) >*< python hangs on this line 
        except SSLError, x:
            if x.args[0] == SSL_ERROR_EOF and self.suppress_ragged_eofs:
                return ''
            else:
                raise

From the traceback given by faulthandler it seems to me that the problem isn't from my webserver trying to receive connections from clients, but from my server (acting as a client) trying a request on a distant server (https URL -> use of ssl.py). Here's the traceback : 

     1	Current thread 0x00007fb9cb41f700 (most recent call first):
     2	File "/usr/lib/python2.7/ssl.py", line 160 in read
     3	File "/usr/lib/python2.7/ssl.py", line 241 in recv
     4	File "/usr/lib/python2.7/socket.py", line 447 in readline
     5	File "/usr/lib/python2.7/httplib.py", line 365 in _read_status
     6	File "/usr/lib/python2.7/httplib.py", line 407 in begin
     7	File "/usr/lib/python2.7/httplib.py", line 1034 in getresponse
     8	File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 353 in _make_request
     9	File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 518 in urlopen
    10	File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 370 in send
    11	File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 573 in send
    12	File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 461 in request
    13	File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 473 in get
    14	File "/usr/local/lib/python2.7/dist-packages/infomaniak/infomaniak.py", line 29 in getFlux
    15	File "/usr/local/lib/python2.7/dist-packages/infomaniak/server.py", line 52 in do_GET
    16	File "/usr/lib/python2.7/BaseHTTPServer.py", line 328 in handle_one_request
    17	File "/usr/lib/python2.7/BaseHTTPServer.py", line 340 in handle
    18	File "/usr/lib/python2.7/SocketServer.py", line 649 in __init__
    19	File "/usr/lib/python2.7/SocketServer.py", line 334 in finish_request
    20	File "/usr/lib/python2.7/SocketServer.py", line 321 in process_request
    21	File "/usr/lib/python2.7/SocketServer.py", line 295 in _handle_request_noblock
    22	File "/usr/lib/python2.7/SocketServer.py", line 238 in serve_forever
    23	File "/usr/local/lib/python2.7/dist-packages/infomaniak/server.py", line 71 in <module>
    24	File "/usr/bin/infomaniak", line 2 in <module>
    25	 

If you look at line 13, it shows that the server is actually doing a get to an external URL via the requests library, which is itself relying on urllib3, which in turn is using httplib.py

Below is the code of the last three functions to have been called, in chronoligical order : 

In socket.py::_fileobject::readline
           [...]
            while True:
                try:
                    data = self._sock.recv(self._rbufsize) #<------------
                except error, e:
                    if e.args[0] == EINTR:
                        continue
                    raise
                if not data:
                    break
                nl = data.find('\n')
                if nl >= 0:
                    nl += 1
                    buf.write(data[:nl])
                    self._rbuf.write(data[nl:])
                    del data
                    break
                buf.write(data)
            return buf.getvalue()
           [...]

In ssl.py::SSLSocket::recv 

    def recv(self, buflen=1024, flags=0):
        if self._sslobj:
            if flags != 0:
                raise ValueError(
                    "non-zero flags not allowed in calls to recv() on %s" %
                    self.__class__)
            return self.read(buflen)                                  #<-------
        else:
            return self._sock.recv(buflen, flags)


In ssl.py::SSLSocket::read

    def read(self, len=1024):

        """Read up to LEN bytes and return them.                                                                                                                                                                                 
        Return zero-length string on EOF."""

        try:
            return self._sslobj.read(len) # >*< python hangs on this line 
        except SSLError, x:
            if x.args[0] == SSL_ERROR_EOF and self.suppress_ragged_eofs:
                return ''
            else:
                raise

I can't go any further because the _sslobj is create via the _ssl.so library, it is very likely C code.

Do you have any idea about how I can investigate this any further ?

[toc] | [prev] | [next] | [standalone]


Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python


csiph-web