Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #5003 > unrolled thread

can't get urllib2 or httplib to work with tor & privoxy

Back to article view | Back to comp.lang.python

Contents

  can't get urllib2 or httplib to work with tor & privoxy Bob Fnord <bob@example.com> - 2011-05-09 20:20 +0200
    Re: can't get urllib2 or httplib to work with tor & privoxy Chris Angelico <rosuav@gmail.com> - 2011-05-10 20:33 +1000
      Re: can't get urllib2 or httplib to work with tor & privoxy Bob Fnord <bob@example.com> - 2011-05-12 17:51 +0200

#5003 — can't get urllib2 or httplib to work with tor & privoxy

Here's my python code:

import httplib, urllib2

proxy_handler = {'http' : 'localhost:8118',
                 'https' : 'localhost:8118'}

def connect_u2(url = 'http://ipid.shat.net/iponly/'):,
    proxied = urllib2.ProxyHandler(proxy_handler)
    opnr = urllib2.build_opener(proxied)
    opnr.addheaders = [('User-agent', agent)]
    rsp = opnr.open(url)
    page = rsp.read()
    return page

def connect_h(url = 'http://ipid.shat.net/iponly/'):
    cnn = httplib.HTTPConnection('127.0.0.1', 8118)
    cnn.connect()
    cnn.request('GET', url)
    rsp = cnn.getresponse()
    stderr.write('%u %s\n' % (rsp.status, rsp.reason))
    page = rsp.read(500)
    cnn.close()
    return page

Both methods give me a 503 error, even though this

export http_proxy='http://localhost:8118/'
lynx -dump 'http://ipid.shat.net/iponly/'

works in bash and prints out the IP of a tor exit.

What am I doing wrong in python?  (I tried both 'localhost:8118' and
'http://localhost:8118/' in the proxy_handler dict.)

[toc] | [next] | [standalone]

#5064

On Tue, May 10, 2011 at 4:20 AM, Bob Fnord <bob@example.com> wrote:
> Both methods give me a 503 error...

As a networking geek, my first thought would be to fire up a tiny
little "snoop server" and see what, exactly, the two methods are
doing. (Ignore the HTTPS options as they're more complicated to snoop
on; you're almost certainly going to see exactly the same on the
HTTP.)

A snoop server is simply an HTTP server that dumps its requests to
stdout or a file, utterly raw. You can write one in Python fairly
easily, or use another program; if I'm working on Windows, I'll
generally use my own RosMud MUD client, but you'll be able to whip up
a cross-platform one in half a page of code in any decent high level
language. I recommend you work at the sockets level, rather than using
an HTTP library, unless you have one that can emit the entire request,
exactly as it came from the client.

Once you've found the difference(s) between Lynx and your script, you
can see what's causing the 503 (Service Unavailable) error; it may be
that you need to authenticate with the proxy.

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#5243

Chris Angelico <rosuav@gmail.com> wrote:

> On Tue, May 10, 2011 at 4:20 AM, Bob Fnord <bob@example.com> wrote:
> > Both methods give me a 503 error...
> 
> As a networking geek, my first thought would be to fire up a tiny
> little "snoop server" and see what, exactly, the two methods are
> doing. (Ignore the HTTPS options as they're more complicated to snoop
> on; you're almost certainly going to see exactly the same on the
> HTTP.)
> 
> A snoop server is simply an HTTP server that dumps its requests to
> stdout or a file, utterly raw. You can write one in Python fairly
> easily, or use another program; if I'm working on Windows, I'll
> generally use my own RosMud MUD client, but you'll be able to whip up
> a cross-platform one in half a page of code in any decent high level
> language. I recommend you work at the sockets level, rather than using
> an HTTP library, unless you have one that can emit the entire request,
> exactly as it came from the client.
> 
> Once you've found the difference(s) between Lynx and your script, you
> can see what's causing the 503 (Service Unavailable) error; it may be
> that you need to authenticate with the proxy.

Duh, I was doing something else stupid. 
Thanks but never mind!

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web