Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.javascript > #18647 > unrolled thread

Automatically driving a browser

Back to article view | Back to comp.lang.javascript

Contents

  Automatically driving a browser Theo Markettos <theom+news@chiark.greenend.org.uk> - 2013-02-27 14:48 +0000
    Re: Automatically driving a browser Christoph Becker <cmbecker69@gmx.de> - 2013-02-27 15:53 +0100
      Re: Automatically driving a browser Theo Markettos <theom+news@chiark.greenend.org.uk> - 2013-02-28 00:34 +0000
        Re: Automatically driving a browser Stefan Weiss <krewecherl@gmail.com> - 2013-02-28 17:33 +0100
    Re: Automatically driving a browser Bjoern Hoehrmann <bjoern@hoehrmann.de> - 2013-02-27 15:57 +0100
      Re: Automatically driving a browser Theo Markettos <theom+news@chiark.greenend.org.uk> - 2013-02-27 15:14 +0000
        Re: Automatically driving a browser Eric Bednarz <bednarz@fahr-zur-hoelle.org> - 2013-02-27 22:38 +0100
          Re: Automatically driving a browser Theo Markettos <theom+news@chiark.greenend.org.uk> - 2013-02-28 00:43 +0000
    Re: Automatically driving a browser Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-02-27 16:45 +0100

#18647 — Automatically driving a browser

Is there any means of programmatically operating a browser - for example,
inspecting the DOM, click on links it finds, waiting for the next page to
load, enter some text into a form, submit it, etc etc?  By this I mean the
browser is operating as normal (with a UI) not as a perl script that's doing
HTTP itself.

An example of the sort of thing: say you wanted to copy your Facebook
profile to Google Plus.  First you have to deal with the login process on
each, which might involve sending you an SMS, making you recognise cat
pictures, or whatever.  The human can do this, the program can't.  Then,
when we're in, the program has to inspect the Facebook profile page and
drive the relevant form on Google+ to fill in the boxes.  So 'likes: pizza,
music' might need both conversion of the strings (a regex or similar) and a
means of operating DOM elements (type 'piz' into box, select 'pizza' from
dropdown, click 'submit', wait for the page's Javascript handler to process
it and maybe load a new page, repeat).

Being able to export the full cookie state from a running GUI browser to a
perl script /might/ work, but I suspect perl etc would have difficulty
driving websites that are heavy in Javascript.

I've seen things like Google Scraper which poke about in the HTML hierarchy,
but do tools/scripts for actually operating a browser as well exist?  I'm
guessing Javascript is the most likely starting point, since it has access
to the DOM already, though any other language is fine too.  Am happy to be
tied to a specific browser, it doesn't need to be generic.

Thanks
Theo

[toc] | [next] | [standalone]

#18648

Theo Markettos wrote:

> Is there any means of programmatically operating a browser - for example,
> inspecting the DOM, click on links it finds, waiting for the next page to
> load, enter some text into a form, submit it, etc etc?  By this I mean the
> browser is operating as normal (with a UI) not as a perl script that's doing
> HTTP itself.

The iMacros Add-On for Firefox may serve your needs:
<https://addons.mozilla.org/en/firefox/addon/imacros-for-firefox/>

-- 
Christoph M. Becker

[toc] | [prev] | [next] | [standalone]

#18658

Christoph Becker <cmbecker69@gmx.de> wrote:
> The iMacros Add-On for Firefox may serve your needs:
> <https://addons.mozilla.org/en/firefox/addon/imacros-for-firefox/>

Interesting.  Looks easy to use, but I'm not sure it's scriptable enough.
For example, things like:

Please enter these characters from your password
**[]****[]**[]

Count the number of table rows and click Delete that number of times

Read the price of a product, calculate how many you can buy for $1000, and
Add To Basket that number of items, compensating for shipping costs which
increase as you add more items so that the total never exceeds $1000.

This kind of interaction is tricky...

Theo

[toc] | [prev] | [next] | [standalone]

#18662

On 2013-02-28 01:34, Theo Markettos wrote:
> Christoph Becker <cmbecker69@gmx.de> wrote:
>> The iMacros Add-On for Firefox may serve your needs:
>> <https://addons.mozilla.org/en/firefox/addon/imacros-for-firefox/>
> 
> Interesting.  Looks easy to use, but I'm not sure it's scriptable enough.

Maybe this could work for you:
http://docs.seleniumhq.org/


- stefan

[toc] | [prev] | [next] | [standalone]

#18649

* Theo Markettos wrote in comp.lang.javascript:
>Is there any means of programmatically operating a browser - for example,
>inspecting the DOM, click on links it finds, waiting for the next page to
>load, enter some text into a form, submit it, etc etc?  By this I mean the
>browser is operating as normal (with a UI) not as a perl script that's doing
>HTTP itself.

http://phantomjs.org/
-- 
Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 

[toc] | [prev] | [next] | [standalone]

#18650

Bjoern Hoehrmann <bjoern@hoehrmann.de> wrote:
> http://phantomjs.org/

Looks useful, but I suppose the distinction here is I still need a UI for
the login phase.  It supports jQuery which might do some of the automation
stuff though, if the login isn't too complex.

Theo

[toc] | [prev] | [next] | [standalone]

#18655

Theo Markettos <theom+news@chiark.greenend.org.uk> writes:

> Bjoern Hoehrmann <bjoern@hoehrmann.de> wrote:
>> http://phantomjs.org/
>
> Looks useful, but I suppose the distinction here is I still need a UI for
> the login phase.

http://casperjs.org/

[toc] | [prev] | [next] | [standalone]

#18659

Eric Bednarz <bednarz@fahr-zur-hoelle.org> wrote:
> > Bjoern Hoehrmann <bjoern@hoehrmann.de> wrote:
> >> http://phantomjs.org/
> http://casperjs.org/

Hmm... looks nice.  I wonder if I can automate enough to avoid having an
actual GUI for login, because otherwise that looks like it might do what I
want.

Theo

[toc] | [prev] | [next] | [standalone]

#18651

Theo Markettos wrote:

> Is there any means of programmatically operating a browser - for example,
> inspecting the DOM, click on links it finds, waiting for the next page to
> load, enter some text into a form, submit it, etc etc?  By this I mean the
> browser is operating as normal (with a UI) not as a perl script that's
> doing HTTP itself.

In theory, there is the possibility that you can access documents in other 
frames, you can call access properties of element objects, and you can 
create events and propagate them to elements.  In practice, that possibility 
is limited by the Same Origin Policy (SOP).

However, it is possible to use a non-standard host environment, such as a 
user profile for testing (and only for that) with sufficiently low (but not 
lower) security settings.  I suspect that it also could be done with a 
browser extension since that is running in a privileged context.

Or you could perhaps run all foreign traffic through a transparent HTTP 
proxy so that the SOP is not an issue despite strong security settings.

The simplest thing to do is, of course, to use the script console on the 
site that you are testing.  I have not yet tried what you could call “cross-
site driving”, though.
 
> I've seen things like Google Scraper which poke about in the HTML
> hierarchy, but do tools/scripts for actually operating a browser as well
> exist?

Probably yes.  Have you searched for them?

> I'm guessing Javascript is the most likely starting point,

There is no Javascript.  <http://PointedEars.de/es-matrix>

> since it has access to the DOM already, though any other language is fine
> too.  Am happy to be tied to a specific browser, it doesn't need to be
> generic.

In that case Firefox or in general Gecko-based environments are the most 
likely candidates.

-- 
PointedEars

Twitter: @PointedEars2
Please do not Cc: me. / Bitte keine Kopien per E-Mail.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.javascript

csiph-web