Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > perl.beginners > #19282

reading data from a web site

Show all headers | View raw

Hi,

how can I read data from a web site which is using multiple frames and some 
javascript?

When using a web browser, I need to log in into the web site and follow a 
couple links to finally get to the page I want.  This page I need to get data 
from has a frame with a select list and another frame displaying a table.  By 
default, the table contains data corresponding to the first entry in the 
select list.  Selecting an entry from the select list reloads the table in the 
other frame once I click on the entry.

I need to automatically pick all the entries from the select list one after 
another so that the table is being updated.  Once I've read the new table, the 
next entry in the select list needs to be picked to get the table updated, 
until there are no more entries.  The order in which the entries are being 
picked can be random.

Once all the available tables are read, only the first entry in the select 
needs to be read once per day.  "Reading the table" means that I need to put 
the data in the table into a database.  It would help if could save all the 
tables to files and convert the files later; the problem is getting the 
tables.

Is this even possible?  There doesn't seem be any useful support for 
javascript with WWW::Mechanize, and even frames seem to be an issue.

I can only see that some java (or javascript or whatever it is) function is 
being called when clicking on an entry in the select list the only purpose of 
which seems to be to display a banner showing "Loading ..." with the dots 
moving while the table is being loaded.  Yet the table is being updated.  I 
can't see how and if there are GET or POST requests being sent by the web 
browser.  The only way to update seems to be to somehow fake a click on an 
entry in the select list.

Any ideas?

Back to perl.beginners | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

reading data from a web site hw@gc-24.de (hw) - 2019-11-19 19:20 +0100
  Re: reading data from a web site Olivier.Nicole@cs.ait.ac.th (Olivier) - 2019-11-20 09:29 +0700
    Re: reading data from a web site hw@gc-24.de (hw) - 2019-11-20 19:52 +0100
      Re: reading data from a web site temp4@mflan.com (Mike) - 2019-11-20 17:18 -0600
      Re: reading data from a web site shlomif@shlomifish.org (Shlomi Fish) - 2019-11-21 08:57 +0200
        Re: reading data from a web site Olivier.Nicole@cs.ait.ac.th (Olivier) - 2019-11-21 14:34 +0700
          Re: reading data from a web site hw@gc-24.de (hw) - 2019-11-21 13:27 +0100
        Re: reading data from a web site hw@gc-24.de (hw) - 2019-11-21 12:17 +0100

csiph-web