Path: csiph.com!news.dns-netz.com!news.freedyn.net!newsreader4.netcologne.de!news.netcologne.de!peer03.ams1!peer.ams1.xlned.com!news.xlned.com!peer01.ams4!peer.am4.highwinds-media.com!news.highwinds-media.com!fx46.ams4.POSTED!not-for-mail
From: Spiros Bousbouras <spibou@gmail.com>
Newsgroups: comp.programming
Subject: Re: Advice needed on parsing a youtube page
Message-ID: <wing-it@bongo-ra.co>
References: <Rafaello@bongo-ra.co> <Rafeallo@bongo-ra.co> <C5x_H.21182$gG4.9701@fx13.iad> <rafaello@bongo-ra.co> <87sg5hp4ra.fsf@bsb.me.uk>
In-Reply-To: <87sg5hp4ra.fsf@bsb.me.uk>
X-Organisation: Weyland-Yutani
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
Lines: 134
X-Complaints-To: http://netreport.virginmedia.com
NNTP-Posting-Date: Sun, 28 Feb 2021 21:02:52 UTC
Organization: virginmedia.com
Date: Sun, 28 Feb 2021 21:02:52 GMT
X-Received-Bytes: 7026
Xref: csiph.com comp.programming:12283

On Sat, 27 Feb 2021 21:42:01 +0000
Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
> Spiros Bousbouras <spibou@gmail.com> writes:
> 
> > Otherwise , in your last 2 paragraphs you seem to only restate my question.
> > In order to find the right tags , I need to find the JSON strings ; and in
> > order to find those I need to locate correctly where each string (whether
> > it is part of JSON or part of an HTML tag) starts and ends ; and in order
> > to do that I need to know whether a string is part of JSON or part of an
> > HTML tag so that I can apply the correct rule to locate the closing double
> > quote.
> 
> I know it's annoying, but I have to correct some of your terms (or at
> least clarify them) so we can all be on the same page.  An HTML tag
> looks like this: <p>.  That's an opening tag.  You can also have closing
> tags like </p>.  Some tags denote elements (like <hr>) all by
> themselves, but often an element is constructed form the opening tag,
> the content and the (sometimes implicit) closing tag.

No , it's not annoying and it's necessary. When I wrote "find the right tags"
I meant something different. Here is an example from the video I mentioned
earlier (www.youtube.com/watch?v=w2itwFJCgFQ) :
[I have added newlines for readability]

=== QUOTE
"videoDetails":{"videoId":"w2itwFJCgFQ",
"title":"The astounding athletic power of quadcopters | Raffaello D'Andrea",
"lengthSeconds":"968",
"keywords":["TEDTalk",
"TEDTalks",
"TED Talk",
"TED Talks",
"TEDGlobal",
"\\Raffaello D'Andrea\\",
"robots",
"technology"],
"channelId":"UCAuUUnT6oDeKwE6v1NGQxug",
"isOwnerViewing":false,
"shortDescription":"In a robot lab at TEDGlobal,
    SNIP
https://www.youtube.com/TED",
"isCrawlable":true,
    SNIP
"averageRating":4.9200745,
"allowRatings":true,
"viewCount":"9755173",
"author":"TED",
"isPrivate":false,
"isUnpluggedCorpus":false,
"isLiveContent":false},
=== ENDQUOTE

What I have been calling a tag is a string (like "videoDetails") followed by
a colon (possibly with intervening whitespace). Lets call those JS-tags .Also
, I have been referring to the segment above as JSON. I take it it's not JSON
but Javascript ? Lets call it JSON-like .I'm only interested in the JSON-like
parts of a page.

Some of the JS-tags are followed by a string , some by a  [ , some by a  { ,
some by a number and some by  true  or  false (not within quotes). So I'm
interested in finding some of those JS-tags and printing the "object" which
follows. If the JS-tag is followed by a  [  or  {  then I consider as the
object everything until the matching  ]  or  }  respectively whereas if it is
followed by a string or number or boolean then I want the string or number or
boolean.

One of the JS-tags I'm interested in is "title" but there are various
appearances of "title" on the page and the object which follows is not always
the same. I'm interested in the appearance of "title" which is followed by
the video title. So for this video

    "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea"

Through studying several video pages , the pattern I have noticed is the
following

    "videoDetails":{
        ......
        "title":"<video title>"
        .....
    }

So you have a pattern where a JS-tag "videoDetails" appears somewhere followed
by a  {  and between the opening brace and the matching  }  you have another
JS-tag  "title":"<video title>" .In all the videos I have examined so far such
a pattern appears only once. So to get the video title I need to locate the
JS-tag "videoDetails" , do bracket matching (in general you may have nested
{} and [] ) and between the matching  {  and  } , search for the JS-tag
"title" .Another JS-tag I'm interested in is  "averageRating"  but this appears
only once on each page so finding it is simpler.

Such and similar are of course only empirical observations so even if I had a
full parser for HTML , Javvascript , etc. and even if the pages faithfully
follow the relevant grammars , I would still need to rely on such heuristics
to find the information I want.

> The point of this is just to say the you won't usually find JSON in a
> tag (in fact, you won't find it "native" in HTML at all).  You do
> sometimes find JSON as an ECMAScript (AKA JavaScript) string.  If you
> do, it will be in the content of a <script> element.
> 
> The specific string that was causing trouble (content="\Raffaello
> D&#39;Andrea\") /is/ inside a tag.  It's the value of the 'content'
> attribute in a 'mata' tag.  The slashes are just part of the value of
> the attribute.  They don't escape anything.

So is there an algorithm to skip all the HTML tags and parse further the
JSON-like part ?

> I could not find any JSON in the document you cited, but I confess I did
> not look all that hard.  I did find list of scripting that has parts
> that look like JSON, but that's not quite the same thing.
> 
> To do what you want in the general case involves parsing HTML.  The
> trouble with doing that is that real-world documents often have
> ungrammatical constructs, so browsers have parsers that apply a lot of
> heuristics to guess what was meant.  Fortunately, you can find HTML
> parsers for many programming languages.
> 
> If you don't go that route you have to decide how much you are prepared
> simply to "wing it".  Can you point out where the parts of the document
> are that you want to scrape?  Maybe there is some simple work-around
> that does not involve much parsing at all.

My code is in Common Lisp. Since the code has worked correctly with dozens of
videos , I'd rather not start from scratch even if there is a Common Lisp
library for parsing HTML. And yes , I'm willing to "wing it". As I explain
above , using some heuristics would be inevitable even if I did have parsers.

-- 
A reader identifies favorite stories in serialization, then buys and saves the
paperback collections, weaving the stories into her own sense of identity. In this
way, manga become part of a reader's autobiography.
  https://web.archive.org/web/20160303170443/https://www.matt-thorn.com/shoujo_manga/japan_quarterly/index.html