Path: csiph.com!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: Ben Bacarisse <ben.usenet@bsb.me.uk>
Newsgroups: comp.programming
Subject: Re: Advice needed on parsing a youtube page
Date: Sun, 28 Feb 2021 23:55:19 +0000
Organization: A noiseless patient Spider
Lines: 176
Message-ID: <87zgznoihk.fsf@bsb.me.uk>
References: <Rafaello@bongo-ra.co> <Rafeallo@bongo-ra.co> <C5x_H.21182$gG4.9701@fx13.iad> <rafaello@bongo-ra.co> <87sg5hp4ra.fsf@bsb.me.uk> <wing-it@bongo-ra.co>
Mime-Version: 1.0
Content-Type: text/plain
Injection-Info: reader02.eternal-september.org; posting-host="b632a0d5348845cc287f5c1e43d1bb39"; logging-data="25874"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX1/aTem9cevEN+ibEieROX1O9umTbW8/zL4="
Cancel-Lock: sha1:0S2rDWDLXq4bgh4YvyFf3KpbWHA= sha1:qG0raijdu5FUOckbXhGp/LpLVZY=
X-BSB-Auth: 1.d988a2b3704e15b61eb3.20210228235519GMT.87zgznoihk.fsf@bsb.me.uk
Xref: csiph.com comp.programming:12284

Spiros Bousbouras <spibou@gmail.com> writes:

> On Sat, 27 Feb 2021 21:42:01 +0000
> Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
>> Spiros Bousbouras <spibou@gmail.com> writes:
>> 
>> > Otherwise , in your last 2 paragraphs you seem to only restate my question.
>> > In order to find the right tags , I need to find the JSON strings ; and in
>> > order to find those I need to locate correctly where each string (whether
>> > it is part of JSON or part of an HTML tag) starts and ends ; and in order
>> > to do that I need to know whether a string is part of JSON or part of an
>> > HTML tag so that I can apply the correct rule to locate the closing double
>> > quote.
>> 
>> I know it's annoying, but I have to correct some of your terms (or at
>> least clarify them) so we can all be on the same page.  An HTML tag
>> looks like this: <p>.  That's an opening tag.  You can also have closing
>> tags like </p>.  Some tags denote elements (like <hr>) all by
>> themselves, but often an element is constructed form the opening tag,
>> the content and the (sometimes implicit) closing tag.
>
> No , it's not annoying and it's necessary. When I wrote "find the right tags"
> I meant something different. Here is an example from the video I mentioned
> earlier (www.youtube.com/watch?v=w2itwFJCgFQ) :
> [I have added newlines for readability]
>
> === QUOTE
> "videoDetails":{"videoId":"w2itwFJCgFQ",
> "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea",
> "lengthSeconds":"968",
> "keywords":["TEDTalk",
> "TEDTalks",
> "TED Talk",
> "TED Talks",
> "TEDGlobal",
> "\\Raffaello D'Andrea\\",
> "robots",
> "technology"],
> "channelId":"UCAuUUnT6oDeKwE6v1NGQxug",
> "isOwnerViewing":false,
> "shortDescription":"In a robot lab at TEDGlobal,
>     SNIP
> https://www.youtube.com/TED",
> "isCrawlable":true,
>     SNIP
> "averageRating":4.9200745,
> "allowRatings":true,
> "viewCount":"9755173",
> "author":"TED",
> "isPrivate":false,
> "isUnpluggedCorpus":false,
> "isLiveContent":false},
> === ENDQUOTE
>
> What I have been calling a tag is a string (like "videoDetails") followed by
> a colon (possibly with intervening whitespace). Lets call those
> JS-tags.

Ah, I see.  So not HTML tags at all.

> Also
> , I have been referring to the segment above as JSON. I take it it's not JSON
> but Javascript ? Lets call it JSON-like .I'm only interested in the JSON-like
> parts of a page.

Yes, this is just an object initialiser in JS.  The things you call tags
are property names.

> Some of the JS-tags are followed by a string , some by a  [ , some by a  { ,
> some by a number and some by  true  or  false (not within quotes).

The property name is followed by an expression that denotes the value of
that property.  []s denote array values and {}s denote objects.

> So I'm
> interested in finding some of those JS-tags and printing the "object" which
> follows. If the JS-tag is followed by a  [  or  {  then I consider as the
> object everything until the matching  ]  or  }  respectively whereas if it is
> followed by a string or number or boolean then I want the string or number or
> boolean.

That's going to work, but only id you start at the "top".

> One of the JS-tags I'm interested in is "title" but there are various
> appearances of "title" on the page and the object which follows is not always
> the same. I'm interested in the appearance of "title" which is followed by
> the video title. So for this video
>
>     "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea"
>
> Through studying several video pages , the pattern I have noticed is the
> following
>
>     "videoDetails":{
>         ......
>         "title":"<video title>"
>         .....
>     }
>
> So you have a pattern where a JS-tag "videoDetails" appears somewhere followed
> by a  {  and between the opening brace and the matching  }  you have another
> JS-tag  "title":"<video title>" .In all the videos I have examined so far such
> a pattern appears only once. So to get the video title I need to locate the
> JS-tag "videoDetails" , do bracket matching (in general you may have nested
> {} and [] ) and between the matching  {  and  } , search for the JS-tag
> "title" .Another JS-tag I'm interested in is  "averageRating"  but this appears
> only once on each page so finding it is simpler.

You could find the top-level object (the one that has a property named
"videoDetails") and pass it to a JS processor line node:

$ cat t.js
var obj = {
    // ...
    "videoDetails":{
        // ...
        "title":"<video title>"
        // ...
    }
    // ...
};
console.log(obj.videoDetails.title);
$ node t.js
<video title>

But then you might get away with just printing the first line with "title" that
follows a line with "videoDetails".

> Such and similar are of course only empirical observations so even if I had a
> full parser for HTML , Javvascript , etc. and even if the pages faithfully
> follow the relevant grammars , I would still need to rely on such heuristics
> to find the information I want.
>
>> The point of this is just to say the you won't usually find JSON in a
>> tag (in fact, you won't find it "native" in HTML at all).  You do
>> sometimes find JSON as an ECMAScript (AKA JavaScript) string.  If you
>> do, it will be in the content of a <script> element.
>> 
>> The specific string that was causing trouble (content="\Raffaello
>> D&#39;Andrea\") /is/ inside a tag.  It's the value of the 'content'
>> attribute in a 'mata' tag.  The slashes are just part of the value of
>> the attribute.  They don't escape anything.
>
> So is there an algorithm to skip all the HTML tags and parse further the
> JSON-like part ?

The proper way is to parse the HTML and the parse the JS to find the
place where the object in question is initialised.  It's a lot of work.
I'd go for a quick and dirty search.

>> I could not find any JSON in the document you cited, but I confess I did
>> not look all that hard.  I did find list of scripting that has parts
>> that look like JSON, but that's not quite the same thing.
>> 
>> To do what you want in the general case involves parsing HTML.  The
>> trouble with doing that is that real-world documents often have
>> ungrammatical constructs, so browsers have parsers that apply a lot of
>> heuristics to guess what was meant.  Fortunately, you can find HTML
>> parsers for many programming languages.
>> 
>> If you don't go that route you have to decide how much you are prepared
>> simply to "wing it".  Can you point out where the parts of the document
>> are that you want to scrape?  Maybe there is some simple work-around
>> that does not involve much parsing at all.
>
> My code is in Common Lisp. Since the code has worked correctly with dozens of
> videos , I'd rather not start from scratch even if there is a Common Lisp
> library for parsing HTML. And yes , I'm willing to "wing it". As I explain
> above , using some heuristics would be inevitable even if I did have
> parsers.

I'd search for '"videoDetails"' and then for '"title":' and take the
string that follows the ':'.

-- 
Ben.