Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #21732 > unrolled thread

Python simulate browser activity

Back to article view | Back to comp.lang.python

Contents

  Python simulate browser activity choi2k <rex.0510@gmail.com> - 2012-03-15 19:23 -0700
    Re: Python simulate browser activity Chris Rebert <clp2@rebertia.com> - 2012-03-15 19:54 -0700
    Re: Python simulate browser activity Jerry Hill <malaclypse2@gmail.com> - 2012-03-15 23:44 -0400
    Re: Python simulate browser activity Roy Smith <roy@panix.com> - 2012-03-16 08:57 -0400
    Re: Python simulate browser activity Chris Angelico <rosuav@gmail.com> - 2012-03-17 06:20 +1100
    Re: Python simulate browser activity NA <NA> - 2012-03-16 21:51 -0400

#21732 — Python simulate browser activity

Hi, everyone

I am trying to write a small application using python but I am not
sure whether it is possible to do so..
The application aims to simulate user activity including visit a
website and perform some interactive actions (click on the menu,
submit a form, redirect to another pages...etc)
I have found some libraries / plugins which aims to simulate browser
activity but ... none of them support AJAX request and/or running
javascript.

Here is one of the simple tasks which I would like to do using the
application:
1. simulate the user activity, visit the website ("https://
www.abc.com")
2. find out the target element by id ("submitBtn") and simulate mouse
click on the item.
3. perform some javascript functions which invoked by click event from
step 2. ( the function is bind to the item by jquery)
4. post ajax request using javascript and if the page has been
redirected, load the new page content

Is it possible to do so?

Thank you in advance.

[toc] | [next] | [standalone]

#21733

On Thu, Mar 15, 2012 at 7:23 PM, choi2k <rex.0510@gmail.com> wrote:
> Hi, everyone
>
> I am trying to write a small application using python but I am not
> sure whether it is possible to do so..
> The application aims to simulate user activity including visit a
> website and perform some interactive actions (click on the menu,
> submit a form, redirect to another pages...etc)
> I have found some libraries / plugins which aims to simulate browser
> activity but ... none of them support AJAX request and/or running
> javascript.

Did you look at Selenium?
http://seleniumhq.org/

Cheers,
Chris

[toc] | [prev] | [next] | [standalone]

#21736

On Thu, Mar 15, 2012 at 10:54 PM, Chris Rebert <clp2@rebertia.com> wrote:
> On Thu, Mar 15, 2012 at 7:23 PM, choi2k <rex.0510@gmail.com> wrote:
>> The application aims to simulate user activity including visit a
>> website and perform some interactive actions (click on the menu,
>> submit a form, redirect to another pages...etc)
>
> Did you look at Selenium?
> http://seleniumhq.org/

You might also be interested in Sikuli (http://sikuli.org/), which
uses Jython to write scripts to automate other programs using
screenshots.  It's pretty neat if you need to automate a GUI that is
otherwise difficult to deal with.

-- 
Jerry

[toc] | [prev] | [next] | [standalone]

#21748

In article 
<214c4c0c-f8ec-4030-946b-8becc8e1aa9c@ur9g2000pbc.googlegroups.com>,
 choi2k <rex.0510@gmail.com> wrote:

> Hi, everyone
> 
> I am trying to write a small application using python but I am not
> sure whether it is possible to do so..
> The application aims to simulate user activity including visit a
> website and perform some interactive actions (click on the menu,
> submit a form, redirect to another pages...etc)
> I have found some libraries / plugins which aims to simulate browser
> activity but ... none of them support AJAX request and/or running
> javascript.
> 
> Here is one of the simple tasks which I would like to do using the
> application:
> 1. simulate the user activity, visit the website ("https://
> www.abc.com")
> 2. find out the target element by id ("submitBtn") and simulate mouse
> click on the item.
> 3. perform some javascript functions which invoked by click event from
> step 2. ( the function is bind to the item by jquery)
> 4. post ajax request using javascript and if the page has been
> redirected, load the new page content
> 
> Is it possible to do so?
> 
> Thank you in advance.

Depending on exactly what you're trying to test, this may or may not be 
possible.

#1 is easy.  Just get the URL with urllib (or, even better, Kenneth 
Reitz's  very cool requests library (http://docs.python-requests.org/).

#2 gets a little more interesting, but still not that big a deal.  Parse 
the HTML you get back with something like lxml (http://lxml.de/).  
Navigate the DOM with a CSSSelector to find the element you're looking 
for.  Use urllib/requests again to get the href.

#3 is where things get fun.  What, exactly, are you trying to test?  Are 
you trying to test that the js works, or are you really trying to test 
your server and you're just using the embedded js as a means to generate 
the next HTTP request?

If the former, then you really want to be exploring something like 
selenium (http://seleniumhq.org/).  If the latter, then skip all the js 
crap and just go a GET or POST (back to urllib/requests) on the 
appropriate url.

#4 falls into the same bucket as #3.

Once you have figured all this out, you will get a greater appreciation 
for the need to cleanly separate your presentation (HTML, CSS, 
Javascript) from your business logic and data layers.  If there is a 
clean interface between them, it becomes easy to test one in isolation 
from the other.  If not, then you end up playing games with screen 
scraping via lxml and selenium just to test your database queries.  
Which will quickly lead to you running screaming into the night.

[toc] | [prev] | [next] | [standalone]

#21771

On Fri, Mar 16, 2012 at 1:23 PM, choi2k <rex.0510@gmail.com> wrote:
> The application aims to simulate user activity including visit a
> website and perform some interactive actions (click on the menu,
> submit a form, redirect to another pages...etc)
> I have found some libraries / plugins which aims to simulate browser
> activity but ... none of them support AJAX request and/or running
> javascript.
>
> Here is one of the simple tasks which I would like to do using the
> application:
> 4. post ajax request using javascript and if the page has been
> redirected, load the new page content

To the web server, everything is a request. Don't think in terms of
Javascript; there is no Javascript, there is no AJAX, there is not
even a web browser. There's just a TCP socket connection and an HTTP
request. That's all you have.

On that basis, your Python script most definitely can simulate the
request. It needs simply to make the same HTTP query that the
Javascript would have made. The only tricky part is figuring out what
that request would be. If you can use a non-SSL connection, that'll
make your job easier - just get a proxy or packet sniffer to monitor
an actual web browser, then save the request headers and body. You can
then replay that using Python; it's not difficult to handle cookies
and such, too.

Hope that helps!

ChrisA

[toc] | [prev] | [next] | [standalone]

#21792

selenium is the best bet.

http://github.com/antlong/selenium

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web