Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.javascript > #19585 > unrolled thread

OT (Probably ??) -- on Parsing a complex web page

Back to article view | Back to comp.lang.javascript

Contents

  OT (Probably ??) -- on Parsing a complex web page "Mel Smith" <med_cutout_syntel@aol.com> - 2013-05-02 09:54 -0600
    Re: OT (Probably ??) -- on Parsing a complex web page "Evertjan." <exxjxw.hannivoort@inter.nl.net> - 2013-05-02 18:19 +0200
      Re: OT (Probably ??) -- on Parsing a complex web page "Mel Smith" <med_cutout_syntel@aol.com> - 2013-05-02 13:39 -0600
    Re: OT (Probably ??) -- on Parsing a complex web page Scott Sauyet <scott.sauyet@gmail.com> - 2013-05-02 10:41 -0700
    Re: OT (Probably ??) -- on Parsing a complex web page Denis McMahon <denismfmcmahon@gmail.com> - 2013-05-02 19:05 +0000
      Re: OT (Probably ??) -- on Parsing a complex web page "Mel Smith" <med_cutout_syntel@aol.com> - 2013-05-02 13:41 -0600

#19585 — OT (Probably ??) -- on Parsing a complex web page

Hi:

    Soon, I'll be using the CURL package (in particular, the libcurl 
library) to download approx 20,000 pages with the same structure but 
different values. THe pages are each approx 64K bytes

    These values are mostly in the <tr> and then <td> elements of the second 
table in this complex page.

    Here is an example of the id of a particular <td> element:

*******************************************************************************************
id="ctl00_ContentPlaceHolder1_bindingPanel_xxxxcard1_RadGridCard_ctl00_ctl05_GECBtnExpandColumn"
*******************************************************************************************

    and I have about 20 of these <td> elements in each of these some 20K 
pages to pick out and save away in a database.

The Question:

    Is there any server-end package that can easily 'get into' the 
downloaded page and pick out the values for the <td> elements that I need ??

    Can server-side JavaScript look at these pages and pick out <td> values 
??  Altho I use JS daily building my apps, I ve never used JS on my *server* 
(I don't even no *how* to !)  -- just my own Harbour language (C-based) 
tools.

    I'm ready to start 'parsing', but its real brain-burner to get into 
doing this with my own language --- even with its strong string functions.

Thanks.

-- 
Mel Smith

[toc] | [next] | [standalone]

#19586

Mel Smith wrote on 02 mei 2013 in comp.lang.javascript:

> Hi:
> 
>     Soon, I'll be using the CURL package (in particular, the libcurl 
> library) to download approx 20,000 pages with the same structure but 
> different values. THe pages are each approx 64K bytes
> 
>     These values are mostly in the <tr> and then <td> elements of the
>     second 
> table in this complex page.
> 
>     Here is an example of the id of a particular <td> element:
> 
> ***********************************************************************
> ******************** 
> id="ctl00_ContentPlaceHolder1_bindingPanel_xxxxcard1_RadGridCard_ctl00_
> ctl05_GECBtnExpandColumn" 
> ***********************************************************************
> ******************** 
> 
>     and I have about 20 of these <td> elements in each of these some
>     20K 
> pages to pick out and save away in a database.
> 
> The Question:
> 
>     Is there any server-end package that can easily 'get into' the 
> downloaded page and pick out the values for the <td> elements that I
> need ?? 

What is a "downloaded page" when thinking serverside???
 
>     Can server-side JavaScript look at these pages 

Yes, like any other serverside language.

> and pick out <td> values ??
  
If you mean the contend delimited by <td>...</td>'s.

Get the file and make a string of the content.
Use regex to delimit <td>...</td>? the strings you need.
Perhaps using match() or execute() to have those strings in an array.
Do your worst with that array.
Or even better, put them into a serverside database, and sql-peruse that..

> Altho I use JS daily building my apps, I ve never used JS on my
> *server* (I don't even no *how* to !) 

No better moment than this to start learning.

> -- just my own Harbour language (C-based) tools.
> 
>     I'm ready to start 'parsing', but its real brain-burner to get
>     into 
> doing this with my own language --- even with its strong string
> functions. 

Regex will help you. 

Old fashioned parsing is probably to slow, and in any case less fun.

-- 
Evertjan.
The Netherlands.
(Please change the x'es to dots in my emailaddress)

[toc] | [prev] | [next] | [standalone]

#19601

Evertjan said:


>
> What is a "downloaded page" when thinking serverside???
>

Hi Evertjan:;

    I really meant that my 'server' turns around and becomes a client (with 
Curl) and downloads the pages from a public web site

Thanks for all the further info you provided.

-Mel

[toc] | [prev] | [next] | [standalone]

#19589

Mel Smith wrote:
>     Soon, I'll be using the CURL package (in particular, the libcurl
> library) to download approx 20,000 pages with the same structure but
> different values. THe pages are each approx 64K bytes
> [ ... ]
> The Question:
>
>     Is there any server-end package that can easily 'get into' the
> downloaded page and pick out the values for the <td> elements that I need ??

There are at least two well-known DOM parsers for Node.js:  htmlparser
[1] and jsdom [2].  I haven't used either, but either one might do.
(A quick glance at their documentation suggests to me that the latter
is quite tied to jQuery, if that affects your decision.)  But you
might want to consider another solution that doesn't involve spinning
up a server, or at least not to collect the data.  PhantomJS [3] is a
headless Webkit browser which I've used for a somewhat similar
purpose, albeit with less than a quarter the number of pages.  You
might find it as useful as I did.

It took me about ten minutes to write a quick program that looped over
the pages, extracted my data from each and wrote a formatted version
of it to the console.  When it failed (in under two minutes) on file
#857, I added an extra log that kept track of those files that I might
have to look over manually.  It turned out there were only six of
them.  In the end, I responded to the request from my potential client
("Can you find a quicker way to do this?  The other vendor's estimate
of three weeks and $6000 seems pretty high.") in under an hour, with
"Yes, that estimate is way to high.  Here's the data."  I got the
gig.  :-)

Phantom is easy to use, and seems now to be well-documented.  For me
it was extremely quick.  I haven't used it in close to two years.  I
can only imagine that it has improved still since then.

Good luck,

  -- Scott

[1] https://npmjs.org/package/htmlparser
[2] https://npmjs.org/package/jsdom
[3] http://phantomjs.org/

[toc] | [prev] | [next] | [standalone]

#19599

On Thu, 02 May 2013 09:54:41 -0600, Mel Smith wrote:

>     Soon, I'll be using the CURL package (in particular, the libcurl
> library) to download approx 20,000 pages with the same structure but
> different values. THe pages are each approx 64K bytes

Assuming this is the sort of job that you can leave going overnight, what 
you could do is grab the page, load it into a dom parser, and then use dom 
manipulation to pull out the data you want.

Quick & dirty (and untested) example in php, assuming $urlList is an 
array of url strings, and you have some way to tell if a td's id matches 
your requirements:

<?php
$doc = new DOMDocument();
foreach ( $urlList as $url ) {
  $doc->loadHTMLFile( $url );
  $cells = $doc->getElementsByTagName( "td" );
  for ( $i = 0; $i < $cells->length; $i++ ) {
    $cellId = $cells->item( $i )->attributes->getNamedItem( "id" );
    if ( <some condition involving $cellId, eg preg_match> ) {
      $cellValue = $cells->item( $i )->textContent;
      // do something with $cellValue here!
    }
  }
}

If you have a way of deriving the specific ids you want from each 
document url, then instead of looping through the cells and testing every 
id, you could use something like:

<?php

function lookupIds( $url ) {
  return arrayOfIdsForUrl;
}

$doc = new DOMDocument();
foreach ( $urlList as $url ) {
  $doc->loadHTMLFile( $url );
  $idsInThisDoc = lookupIds( $url );
  foreach ( $idsInThisDoc as $nextId ) {
    $cell = $doc->getElementById( $nextId );
    $cellValue = $cell->textContent;
    // do something with $cellValue here!
  }
}

and yeah, this is drifting rather off topic here.

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#19602

Denis said:

>
> Assuming this is the sort of job that you can leave going overnight, what
> you could do is grab the page, load it into a dom parser, and then use dom
> manipulation to pull out the data you want.
>
> Quick & dirty (and untested) example in php, assuming $urlList is an
> array of url strings, and you have some way to tell if a td's id matches
> your requirements:
>
> <?php
> $doc = new DOMDocument();
> foreach ( $urlList as $url ) {
>  $doc->loadHTMLFile( $url );
>  $cells = $doc->getElementsByTagName( "td" );
>  for ( $i = 0; $i < $cells->length; $i++ ) {
>    $cellId = $cells->item( $i )->attributes->getNamedItem( "id" );
>    if ( <some condition involving $cellId, eg preg_match> ) {
>      $cellValue = $cells->item( $i )->textContent;
>      // do something with $cellValue here!
>    }
>  }
> }
>
> If you have a way of deriving the specific ids you want from each
> document url, then instead of looping through the cells and testing every
> id, you could use something like:
>
> <?php
>
> function lookupIds( $url ) {
>  return arrayOfIdsForUrl;
> }
>
> $doc = new DOMDocument();
> foreach ( $urlList as $url ) {
>  $doc->loadHTMLFile( $url );
>  $idsInThisDoc = lookupIds( $url );
>  foreach ( $idsInThisDoc as $nextId ) {
>    $cell = $doc->getElementById( $nextId );
>    $cellValue = $cell->textContent;
>    // do something with $cellValue here!
>  }
> }
>


Than ks to you Denis and Scott and Evertjan for the info provided.

Now I'll have to put my head down and stasrt 'scribbling' !

-Mel

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.javascript

csiph-web