Path: csiph.com!news.uzoreto.com!feeder.usenetexpress.com!tr2.eu1.usenetexpress.com!81.171.65.13.MISMATCH!peer01.ams4!peer.am4.highwinds-media.com!news.highwinds-media.com!fx31.ams4.POSTED!not-for-mail
From: Spiros Bousbouras <spibou@gmail.com>
Newsgroups: comp.programming
Subject: Re: Advice needed on parsing a youtube page
Message-ID: <e4FtbBG32jNcr7@bongo-ra.co>
References: <Rafaello@bongo-ra.co> <Rafeallo@bongo-ra.co> <C5x_H.21182$gG4.9701@fx13.iad> <rafaello@bongo-ra.co> <87sg5hp4ra.fsf@bsb.me.uk> <wing-it@bongo-ra.co> <87zgznoihk.fsf@bsb.me.uk>
In-Reply-To: <87zgznoihk.fsf@bsb.me.uk>
X-Organisation: Weyland-Yutani
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
Lines: 158
X-Complaints-To: http://netreport.virginmedia.com
NNTP-Posting-Date: Tue, 02 Mar 2021 18:51:40 UTC
Organization: virginmedia.com
Date: Tue, 02 Mar 2021 18:51:40 GMT
X-Received-Bytes: 6586
Xref: csiph.com comp.programming:12293

On Sun, 28 Feb 2021 23:55:19 +0000
Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
> Spiros Bousbouras <spibou@gmail.com> writes:
[...]

> >                                            When I wrote "find the right tags"
> > I meant something different. Here is an example from the video I mentioned
> > earlier (www.youtube.com/watch?v=w2itwFJCgFQ) :
> > [I have added newlines for readability]
> >
> > === QUOTE
> > "videoDetails":{"videoId":"w2itwFJCgFQ",
> > "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea",
> > "lengthSeconds":"968",
> > "keywords":["TEDTalk",
> > "TEDTalks",
> > "TED Talk",
> > "TED Talks",
> > "TEDGlobal",
> > "\\Raffaello D'Andrea\\",
> > "robots",
> > "technology"],
> > "channelId":"UCAuUUnT6oDeKwE6v1NGQxug",
> > "isOwnerViewing":false,
> > "shortDescription":"In a robot lab at TEDGlobal,
> >     SNIP
> > https://www.youtube.com/TED",
> > "isCrawlable":true,
> >     SNIP
> > "averageRating":4.9200745,
> > "allowRatings":true,
> > "viewCount":"9755173",
> > "author":"TED",
> > "isPrivate":false,
> > "isUnpluggedCorpus":false,
> > "isLiveContent":false},
> > === ENDQUOTE
> >
> > What I have been calling a tag is a string (like "videoDetails") followed by
> > a colon (possibly with intervening whitespace). Lets call those
> > JS-tags.
> 
> Ah, I see.  So not HTML tags at all.
> 
> > Also
> > , I have been referring to the segment above as JSON. I take it it's not JSON
> > but Javascript ? Lets call it JSON-like .I'm only interested in the JSON-like
> > parts of a page.
> 
> Yes, this is just an object initialiser in JS.  The things you call tags
> are property names.
> 
> > Some of the JS-tags are followed by a string , some by a  [ , some by a  { ,
> > some by a number and some by  true  or  false (not within quotes).
> 
> The property name is followed by an expression that denotes the value of
> that property.  []s denote array values and {}s denote objects.

Ok , it's good to know the correct terminology.

> > So I'm
> > interested in finding some of those JS-tags and printing the "object" which
> > follows. If the JS-tag is followed by a  [  or  {  then I consider as the
> > object everything until the matching  ]  or  }  respectively whereas if it is
> > followed by a string or number or boolean then I want the string or number or
> > boolean.
> 
> That's going to work, but only id you start at the "top".

Where else would I start ?

> > One of the JS-tags I'm interested in is "title" but there are various
> > appearances of "title" on the page and the object which follows is not always
> > the same. I'm interested in the appearance of "title" which is followed by
> > the video title. So for this video
> >
> >     "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea"
> >
> > Through studying several video pages , the pattern I have noticed is the
> > following
> >
> >     "videoDetails":{
> >         ......
> >         "title":"<video title>"
> >         .....
> >     }
> >
> > So you have a pattern where a JS-tag "videoDetails" appears somewhere followed
> > by a  {  and between the opening brace and the matching  }  you have another
> > JS-tag  "title":"<video title>" .In all the videos I have examined so far such
> > a pattern appears only once. So to get the video title I need to locate the
> > JS-tag "videoDetails" , do bracket matching (in general you may have nested
> > {} and [] ) and between the matching  {  and  } , search for the JS-tag
> > "title" .Another JS-tag I'm interested in is  "averageRating"  but this appears
> > only once on each page so finding it is simpler.
> 
> You could find the top-level object (the one that has a property named
> "videoDetails") and pass it to a JS processor line node:

Are you referring to  en.wikipedia.org/wiki/Node.js  or something more general ?

> $ cat t.js
> var obj = {
>     // ...
>     "videoDetails":{
>         // ...
>         "title":"<video title>"
>         // ...
>     }
>     // ...
> };
> console.log(obj.videoDetails.title);
> $ node t.js
> <video title>
> 

If I have managed to isolate the
      "videoDetails":{
          // ...
          "title":"<video title>"
          // ...
      }

part , my current code already works. So you are suggesting a much more
complicated solution (run a Javascript engine , if I understand correctly) to
achieve something which I already handle. The issue is to recognise the
appearance of (for example) "videoDetails" which means that I also recognise
strings which appear earlier whether they are part of HTML tags or Javascript
properties.

> But then you might get away with just printing the first line with "title" that
> follows a line with "videoDetails".

[...]

> > So is there an algorithm to skip all the HTML tags and parse further the
> > JSON-like part ?
> 
> The proper way is to parse the HTML and the parse the JS to find the
> place where the object in question is initialised.  It's a lot of work.
> I'd go for a quick and dirty search.

Up to now , the parsing of Javascript I have been doing has been working well
enough. So I'm looking for a quick and dirty way to skip the non Javascript
part.

> I'd search for '"videoDetails"' and then for '"title":' and take the
> string that follows the ':'.

So not take at all into account which property is part of which object.
Perhaps even more simplicity than what I have now is the way to go , I'll
think about it. Thanks.

-- 
This was certainly an objection heard in the West where the glissandi of the
trombone which accompanied Sergei's movements during a love scene earned the
opera the term 'pornophony'.
  http://www.quartets.de/articles/Lady_Macbeth.html