Re: Python simulate browser activity

Path	csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!newsreader4.netcologne.de!news.netcologne.de!feed.xsnews.nl!border-1.ams.xsnews.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<rosuav@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.045
X-Spam-Evidence	'H': 0.91; 'S': 0.00; 'socket': 0.04; 'javascript,': 0.05; 'subject:Python': 0.05; 'python': 0.08; 'ajax,': 0.09; 'figuring': 0.09; 'server,': 0.12; '16,': 0.15; 'cookies': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'helps!': 0.16; 'packet': 0.16; 'python;': 0.16; 'simulate': 0.16; "that'll": 0.16; 'wrote:': 0.18; 'received:209.85.210.174': 0.21; 'received:mail- iy0-f174.google.com': 0.21; 'header:In-Reply-To:1': 0.22; 'basis,': 0.23; 'libraries': 0.24; 'load': 0.26; 'monitor': 0.28; 'script': 0.28; 'message-id:@mail.gmail.com': 0.29; 'made.': 0.29; 'pm,': 0.29; 'menu,': 0.30; 'headers': 0.32; 'actual': 0.32; 'there': 0.33; 'fri,': 0.34; 'running': 0.34; 'body.': 0.34; 'to:addr:python-list': 0.35; 'none': 0.35; '...': 0.35; 'be.': 0.35; 'form,': 0.35; 'connection': 0.36; 'post': 0.36; 'ajax': 0.37; 'http': 0.37; 'query': 0.37; 'page': 0.37; 'but': 0.37; 'received:google.com': 0.37; 'using': 0.37; 'another': 0.37; 'received:209.85': 0.38; 'some': 0.38; 'think': 0.38; 'easier': 0.38; 'difficult': 0.39; 'tasks': 0.39; 'received:209': 0.39; 'application': 0.40; 'user': 0.40; 'to:addr:python.org': 0.40; 'simple': 0.61; 'your': 0.61; 'hope': 0.61; 'browser': 0.62; 'here': 0.64; 'request.': 0.65; 'website': 0.65; 'visit': 0.67; 'connection,': 0.73; 'have.': 0.77; 'browser.': 0.80; 'application:': 0.84; 'replay': 0.84; 'sniffer': 0.84; 'tricky': 0.84; 'plugins': 0.91; 'redirect': 0.91; 'subject:activity': 0.91
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=X/w5PxZiiw5m9P22IEINvVyMQ43TmP5/g1OUELVGxSU=; b=UA42UIm9l4iQSY+1eFayKJJZLqnfYTpDLc4iej7pS2nl3CfjwyWCp46NhZmlJVR7Gx 5GCcN+PkaL+plPfbl0pMTsIhNdZFJVYgC3HJLVFO4sehe4gNaF20hjFGfDcckibKIGYm Ja0m0718sR8iYYnkCcaWO9InrlfjQaF5CIWcv7K2EZF9lG/R+uP+uR719Qz9QmeQ3vRv 5E7et8+Mi+kaD7rq/CnSm1NAkwfQlkLqOiwxWtOpDlC6G8qM8TvO2O/mrlQaXnLsro2M gCQUO3VaP13/Yv42r3g+XrNf8/CHaZWFdnXTBqFp+2IpN6lIOwZ6kcDcdKGBik+NVL2W /rqw==
MIME-Version	1.0
In-Reply-To	<214c4c0c-f8ec-4030-946b-8becc8e1aa9c@ur9g2000pbc.googlegroups.com>
References	<214c4c0c-f8ec-4030-946b-8becc8e1aa9c@ur9g2000pbc.googlegroups.com>
Date	Sat, 17 Mar 2012 06:20:17 +1100
Subject	Re: Python simulate browser activity
From	Chris Angelico <rosuav@gmail.com>
To	python-list@python.org
Content-Type	text/plain; charset=ISO-8859-1
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.731.1331925620.3037.python-list@python.org> (permalink)
Lines	30
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1331925620 news.xs4all.nl 6909 [2001:888:2000:d::a6]:59123
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:21771

Show key headers only | View raw

On Fri, Mar 16, 2012 at 1:23 PM, choi2k <rex.0510@gmail.com> wrote:
> The application aims to simulate user activity including visit a
> website and perform some interactive actions (click on the menu,
> submit a form, redirect to another pages...etc)
> I have found some libraries / plugins which aims to simulate browser
> activity but ... none of them support AJAX request and/or running
> javascript.
>
> Here is one of the simple tasks which I would like to do using the
> application:
> 4. post ajax request using javascript and if the page has been
> redirected, load the new page content

To the web server, everything is a request. Don't think in terms of
Javascript; there is no Javascript, there is no AJAX, there is not
even a web browser. There's just a TCP socket connection and an HTTP
request. That's all you have.

On that basis, your Python script most definitely can simulate the
request. It needs simply to make the same HTTP query that the
Javascript would have made. The only tricky part is figuring out what
that request would be. If you can use a non-SSL connection, that'll
make your job easier - just get a proxy or packet sniffer to monitor
an actual web browser, then save the request headers and body. You can
then replay that using Python; it's not difficult to handle cookies
and such, too.

Hope that helps!

ChrisA

Thread

Python simulate browser activity choi2k <rex.0510@gmail.com> - 2012-03-15 19:23 -0700
  Re: Python simulate browser activity Chris Rebert <clp2@rebertia.com> - 2012-03-15 19:54 -0700
  Re: Python simulate browser activity Jerry Hill <malaclypse2@gmail.com> - 2012-03-15 23:44 -0400
  Re: Python simulate browser activity Roy Smith <roy@panix.com> - 2012-03-16 08:57 -0400
  Re: Python simulate browser activity Chris Angelico <rosuav@gmail.com> - 2012-03-17 06:20 +1100
  Re: Python simulate browser activity NA <NA> - 2012-03-16 21:51 -0400

csiph-web