gevent + urllib2 blocking

Date	2013-02-22 23:40 +0800
Subject	gevent + urllib2 blocking
From	月忧茗 <yueyoum@gmail.com>
Newsgroups	comp.lang.python
Message-ID	<mailman.2277.1361547632.2939.python-list@python.org> (permalink)

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

Hi,

One problem, thanks for help.


import gevent.monkey
gevent.monkey.match_all()

from lxml import etree
# I using xpath parse the html


def _get(p):
    url = BUILD_URL(p)
    html = urllib2.urlopen(url)

    # RUN AT HERE AND BLOCKING

    # ver1
    tree = etree.parse(html, parse)

    # ver2
    # tree = etree.fromstring(html.read(), parse)

    ....


workers  = [gevent.spawn(_get, i) for i in xrange(1, 100)]
gevent.joinall(workers)



When using ver1,  etree.parse(html, parse)
I noticed there are only two greenlets runs at "RUN AT HERE AND BLOCKING"

others are not running!


But, when I using ver2,  this code will running normally as expected.


Why this ?

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

gevent + urllib2 blocking 月忧茗 <yueyoum@gmail.com> - 2013-02-22 23:40 +0800

csiph-web