Automatically driving a browser

From	Theo Markettos <theom+news@chiark.greenend.org.uk>
Newsgroups	comp.lang.javascript
Subject	Automatically driving a browser
Date	2013-02-27 14:48 +0000
Organization	University of Cambridge, England
Message-ID	<pBE*2C4su@news.chiark.greenend.org.uk> (permalink)

Show all headers | View raw

Is there any means of programmatically operating a browser - for example,
inspecting the DOM, click on links it finds, waiting for the next page to
load, enter some text into a form, submit it, etc etc?  By this I mean the
browser is operating as normal (with a UI) not as a perl script that's doing
HTTP itself.

An example of the sort of thing: say you wanted to copy your Facebook
profile to Google Plus.  First you have to deal with the login process on
each, which might involve sending you an SMS, making you recognise cat
pictures, or whatever.  The human can do this, the program can't.  Then,
when we're in, the program has to inspect the Facebook profile page and
drive the relevant form on Google+ to fill in the boxes.  So 'likes: pizza,
music' might need both conversion of the strings (a regex or similar) and a
means of operating DOM elements (type 'piz' into box, select 'pizza' from
dropdown, click 'submit', wait for the page's Javascript handler to process
it and maybe load a new page, repeat).

Being able to export the full cookie state from a running GUI browser to a
perl script /might/ work, but I suspect perl etc would have difficulty
driving websites that are heavy in Javascript.

I've seen things like Google Scraper which poke about in the HTML hierarchy,
but do tools/scripts for actually operating a browser as well exist?  I'm
guessing Javascript is the most likely starting point, since it has access
to the DOM already, though any other language is fine too.  Am happy to be
tied to a specific browser, it doesn't need to be generic.

Thanks
Theo

Back to comp.lang.javascript | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Automatically driving a browser Theo Markettos <theom+news@chiark.greenend.org.uk> - 2013-02-27 14:48 +0000
  Re: Automatically driving a browser Christoph Becker <cmbecker69@gmx.de> - 2013-02-27 15:53 +0100
    Re: Automatically driving a browser Theo Markettos <theom+news@chiark.greenend.org.uk> - 2013-02-28 00:34 +0000
      Re: Automatically driving a browser Stefan Weiss <krewecherl@gmail.com> - 2013-02-28 17:33 +0100
  Re: Automatically driving a browser Bjoern Hoehrmann <bjoern@hoehrmann.de> - 2013-02-27 15:57 +0100
    Re: Automatically driving a browser Theo Markettos <theom+news@chiark.greenend.org.uk> - 2013-02-27 15:14 +0000
      Re: Automatically driving a browser Eric Bednarz <bednarz@fahr-zur-hoelle.org> - 2013-02-27 22:38 +0100
        Re: Automatically driving a browser Theo Markettos <theom+news@chiark.greenend.org.uk> - 2013-02-28 00:43 +0000
  Re: Automatically driving a browser Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-02-27 16:45 +0100

csiph-web