Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #12293

Re: Advice needed on parsing a youtube page

Show all headers | View raw

On Sun, 28 Feb 2021 23:55:19 +0000
Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
> Spiros Bousbouras <spibou@gmail.com> writes:
[...]

> >                                            When I wrote "find the right tags"
> > I meant something different. Here is an example from the video I mentioned
> > earlier (www.youtube.com/watch?v=w2itwFJCgFQ) :
> > [I have added newlines for readability]
> >
> > === QUOTE
> > "videoDetails":{"videoId":"w2itwFJCgFQ",
> > "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea",
> > "lengthSeconds":"968",
> > "keywords":["TEDTalk",
> > "TEDTalks",
> > "TED Talk",
> > "TED Talks",
> > "TEDGlobal",
> > "\\Raffaello D'Andrea\\",
> > "robots",
> > "technology"],
> > "channelId":"UCAuUUnT6oDeKwE6v1NGQxug",
> > "isOwnerViewing":false,
> > "shortDescription":"In a robot lab at TEDGlobal,
> >     SNIP
> > https://www.youtube.com/TED",
> > "isCrawlable":true,
> >     SNIP
> > "averageRating":4.9200745,
> > "allowRatings":true,
> > "viewCount":"9755173",
> > "author":"TED",
> > "isPrivate":false,
> > "isUnpluggedCorpus":false,
> > "isLiveContent":false},
> > === ENDQUOTE
> >
> > What I have been calling a tag is a string (like "videoDetails") followed by
> > a colon (possibly with intervening whitespace). Lets call those
> > JS-tags.
> 
> Ah, I see.  So not HTML tags at all.
> 
> > Also
> > , I have been referring to the segment above as JSON. I take it it's not JSON
> > but Javascript ? Lets call it JSON-like .I'm only interested in the JSON-like
> > parts of a page.
> 
> Yes, this is just an object initialiser in JS.  The things you call tags
> are property names.
> 
> > Some of the JS-tags are followed by a string , some by a  [ , some by a  { ,
> > some by a number and some by  true  or  false (not within quotes).
> 
> The property name is followed by an expression that denotes the value of
> that property.  []s denote array values and {}s denote objects.

Ok , it's good to know the correct terminology.

> > So I'm
> > interested in finding some of those JS-tags and printing the "object" which
> > follows. If the JS-tag is followed by a  [  or  {  then I consider as the
> > object everything until the matching  ]  or  }  respectively whereas if it is
> > followed by a string or number or boolean then I want the string or number or
> > boolean.
> 
> That's going to work, but only id you start at the "top".

Where else would I start ?

> > One of the JS-tags I'm interested in is "title" but there are various
> > appearances of "title" on the page and the object which follows is not always
> > the same. I'm interested in the appearance of "title" which is followed by
> > the video title. So for this video
> >
> >     "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea"
> >
> > Through studying several video pages , the pattern I have noticed is the
> > following
> >
> >     "videoDetails":{
> >         ......
> >         "title":"<video title>"
> >         .....
> >     }
> >
> > So you have a pattern where a JS-tag "videoDetails" appears somewhere followed
> > by a  {  and between the opening brace and the matching  }  you have another
> > JS-tag  "title":"<video title>" .In all the videos I have examined so far such
> > a pattern appears only once. So to get the video title I need to locate the
> > JS-tag "videoDetails" , do bracket matching (in general you may have nested
> > {} and [] ) and between the matching  {  and  } , search for the JS-tag
> > "title" .Another JS-tag I'm interested in is  "averageRating"  but this appears
> > only once on each page so finding it is simpler.
> 
> You could find the top-level object (the one that has a property named
> "videoDetails") and pass it to a JS processor line node:

Are you referring to  en.wikipedia.org/wiki/Node.js  or something more general ?

> $ cat t.js
> var obj = {
>     // ...
>     "videoDetails":{
>         // ...
>         "title":"<video title>"
>         // ...
>     }
>     // ...
> };
> console.log(obj.videoDetails.title);
> $ node t.js
> <video title>
> 

If I have managed to isolate the
      "videoDetails":{
          // ...
          "title":"<video title>"
          // ...
      }

part , my current code already works. So you are suggesting a much more
complicated solution (run a Javascript engine , if I understand correctly) to
achieve something which I already handle. The issue is to recognise the
appearance of (for example) "videoDetails" which means that I also recognise
strings which appear earlier whether they are part of HTML tags or Javascript
properties.

> But then you might get away with just printing the first line with "title" that
> follows a line with "videoDetails".

[...]

> > So is there an algorithm to skip all the HTML tags and parse further the
> > JSON-like part ?
> 
> The proper way is to parse the HTML and the parse the JS to find the
> place where the object in question is initialised.  It's a lot of work.
> I'd go for a quick and dirty search.

Up to now , the parsing of Javascript I have been doing has been working well
enough. So I'm looking for a quick and dirty way to skip the non Javascript
part.

> I'd search for '"videoDetails"' and then for '"title":' and take the
> string that follows the ':'.

So not take at all into account which property is part of which object.
Perhaps even more simplicity than what I have now is the way to go , I'll
think about it. Thanks.

-- 
This was certainly an objection heard in the West where the glissandi of the
trombone which accompanied Sergei's movements during a love scene earned the
opera the term 'pornophony'.
  http://www.quartets.de/articles/Lady_Macbeth.html

Back to comp.programming | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 14:43 +0000
  Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 16:43 +0000
    Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 14:30 -0500
      Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 20:30 +0000
        Re: Advice needed on parsing a youtube page Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-02-27 21:42 +0000
          Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-28 21:02 +0000
            Re: Advice needed on parsing a youtube page Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-02-28 23:55 +0000
              Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-03-02 18:51 +0000
                Re: Advice needed on parsing a youtube page Kristjan Robam <he12457@hotmail.com> - 2021-03-02 17:50 -0800
  Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 14:09 -0500
    Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 20:12 +0000
      Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 22:18 -0500
        Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-03-02 17:30 +0000

csiph-web