Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.javascript > #19599

Re: OT (Probably ??) -- on Parsing a complex web page

Show all headers | View raw

On Thu, 02 May 2013 09:54:41 -0600, Mel Smith wrote:

>     Soon, I'll be using the CURL package (in particular, the libcurl
> library) to download approx 20,000 pages with the same structure but
> different values. THe pages are each approx 64K bytes

Assuming this is the sort of job that you can leave going overnight, what 
you could do is grab the page, load it into a dom parser, and then use dom 
manipulation to pull out the data you want.

Quick & dirty (and untested) example in php, assuming $urlList is an 
array of url strings, and you have some way to tell if a td's id matches 
your requirements:

<?php
$doc = new DOMDocument();
foreach ( $urlList as $url ) {
  $doc->loadHTMLFile( $url );
  $cells = $doc->getElementsByTagName( "td" );
  for ( $i = 0; $i < $cells->length; $i++ ) {
    $cellId = $cells->item( $i )->attributes->getNamedItem( "id" );
    if ( <some condition involving $cellId, eg preg_match> ) {
      $cellValue = $cells->item( $i )->textContent;
      // do something with $cellValue here!
    }
  }
}

If you have a way of deriving the specific ids you want from each 
document url, then instead of looping through the cells and testing every 
id, you could use something like:

<?php

function lookupIds( $url ) {
  return arrayOfIdsForUrl;
}

$doc = new DOMDocument();
foreach ( $urlList as $url ) {
  $doc->loadHTMLFile( $url );
  $idsInThisDoc = lookupIds( $url );
  foreach ( $idsInThisDoc as $nextId ) {
    $cell = $doc->getElementById( $nextId );
    $cellValue = $cell->textContent;
    // do something with $cellValue here!
  }
}

and yeah, this is drifting rather off topic here.

-- 
Denis McMahon, denismfmcmahon@gmail.com

Back to comp.lang.javascript | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

OT (Probably ??) -- on Parsing a complex web page "Mel Smith" <med_cutout_syntel@aol.com> - 2013-05-02 09:54 -0600
  Re: OT (Probably ??) -- on Parsing a complex web page "Evertjan." <exxjxw.hannivoort@inter.nl.net> - 2013-05-02 18:19 +0200
    Re: OT (Probably ??) -- on Parsing a complex web page "Mel Smith" <med_cutout_syntel@aol.com> - 2013-05-02 13:39 -0600
  Re: OT (Probably ??) -- on Parsing a complex web page Scott Sauyet <scott.sauyet@gmail.com> - 2013-05-02 10:41 -0700
  Re: OT (Probably ??) -- on Parsing a complex web page Denis McMahon <denismfmcmahon@gmail.com> - 2013-05-02 19:05 +0000
    Re: OT (Probably ??) -- on Parsing a complex web page "Mel Smith" <med_cutout_syntel@aol.com> - 2013-05-02 13:41 -0600

csiph-web