Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #12273 > unrolled thread

Advice needed on parsing a youtube page

Back to article view | Back to comp.programming

Contents

  Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 14:43 +0000
    Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 16:43 +0000
      Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 14:30 -0500
        Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 20:30 +0000
          Re: Advice needed on parsing a youtube page Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-02-27 21:42 +0000
            Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-28 21:02 +0000
              Re: Advice needed on parsing a youtube page Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-02-28 23:55 +0000
                Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-03-02 18:51 +0000
                  Re: Advice needed on parsing a youtube page Kristjan Robam <he12457@hotmail.com> - 2021-03-02 17:50 -0800
    Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 14:09 -0500
      Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 20:12 +0000
        Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 22:18 -0500
          Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-03-02 17:30 +0000

#12273 — Advice needed on parsing a youtube page

I have some code which parses the source of pages for youtube videos and
prints some information. My code has been working fine for months but
recently it choked on  https://www.youtube.com/watch?v=w2itwFJCgFQ .After
some investigations I found the part of the page which causes the problem :
there is a single appearance of

    content="\Raffaello D'Andrea\"

.My code assumed that the closing double quote is escaped so it did not count
it as a closing quote and so parsed incorrectly the rest of the page. So I
have 2 questions :

1. Is the above segment valid according to whatever the relevant specification
is ?

2. My code only needs to parse the JSON content of pages and extract the
information I want. Is there a simple algorithm which strips all the non JSON
content of a page ? Or alternatively , is there an algorithm which parses
correctly constructs like the above , perhaps taking into account surrounding
tags ?


I should warn you that the source of the page has some very long lines , the
longest being 201416 bytes. So examining the content with some text editors or
less  will not work smoothly. Filtering the page through something like
sed 's/,/,\n/g'   makes things a lot more manageable.

-- 
You're on the right path when you understand what Luke Skywalker, Obi-Wan Kenobi,
and Darth Vader did not: It's not the force, it's the field!
  http://www-personal.umich.edu/~williams/emquotes.html
  David N. Williams

[toc] | [next] | [standalone]

#12275

On Sat, 27 Feb 2021 14:43:35 GMT
Spiros Bousbouras <spibou@gmail.com> wrote:
> I have some code which parses the source of pages for youtube videos and
> prints some information. My code has been working fine for months but
> recently it choked on  https://www.youtube.com/watch?v=w2itwFJCgFQ .After
> some investigations I found the part of the page which causes the problem :
> there is a single appearance of
> 
>     content="\Raffaello D&#39;Andrea\"
> 
> .My code assumed that the closing double quote is escaped so it did not count
> it as a closing quote and so parsed incorrectly the rest of the page.

I note that if I remove the 2nd backslash , my code can parse the page
correctly. Also , there are other appearances of the 2 character sequence  \"
on the page and for those the  "  is indeed escaped. So a different way to
approach my problem is to come up with an algorithm which decides when  \"
denotes an escaped double quote and when it doesn't. Ideally , I'm looking
for a solution which does not involve writing a full parser for HTML ,
Javascript , JSON and who knows what else is on such pages.

> So I have 2 questions :
> 
> 1. Is the above segment valid according to whatever the relevant specification
> is ?
> 
> 2. My code only needs to parse the JSON content of pages and extract the
> information I want. Is there a simple algorithm which strips all the non JSON
> content of a page ? Or alternatively , is there an algorithm which parses
> correctly constructs like the above , perhaps taking into account surrounding
> tags ?

-- 
As always, supporting opinionated programming is one of Forth's 
features, even if it becomes an obstacle to building a critical 
mass around specific programming strategies.
  Bruce R. McFarling
  <c8cbc925.0407232157.6b733a8f@posting.google.com>

[toc] | [prev] | [next] | [standalone]

#12277

On 2/27/21 11:43 AM, Spiros Bousbouras wrote:
> On Sat, 27 Feb 2021 14:43:35 GMT
> Spiros Bousbouras <spibou@gmail.com> wrote:
>> I have some code which parses the source of pages for youtube videos and
>> prints some information. My code has been working fine for months but
>> recently it choked on  https://www.youtube.com/watch?v=w2itwFJCgFQ .After
>> some investigations I found the part of the page which causes the problem :
>> there is a single appearance of
>>
>>     content="\Raffaello D&#39;Andrea\"
>>
>> .My code assumed that the closing double quote is escaped so it did not count
>> it as a closing quote and so parsed incorrectly the rest of the page.
> 
> I note that if I remove the 2nd backslash , my code can parse the page
> correctly. Also , there are other appearances of the 2 character sequence  \"
> on the page and for those the  "  is indeed escaped. So a different way to
> approach my problem is to come up with an algorithm which decides when  \"
> denotes an escaped double quote and when it doesn't. Ideally , I'm looking
> for a solution which does not involve writing a full parser for HTML ,
> Javascript , JSON and who knows what else is on such pages.

My guess is that the spots you see \" as an escaped quote it is a piece
of JavaScript on the page, the " isn't part of an HTML tag, so isn't
processed as part of one, but the \" is just passed through to the
JavaScript processor. The JavaScript processor DOES use \" as an escaped
quote inside its strings.
> 
>> So I have 2 questions :
>>
>> 1. Is the above segment valid according to whatever the relevant specification
>> is ?
>>
>> 2. My code only needs to parse the JSON content of pages and extract the
>> information I want. Is there a simple algorithm which strips all the non JSON
>> content of a page ? Or alternatively , is there an algorithm which parses
>> correctly constructs like the above , perhaps taking into account surrounding
>> tags ?
> 

One issue with scraping pages is that you might want to parse the final
page that will get rendered. If you do, you really need to run a full
parsing and rendering engine to get the data (and some pages
intentionally make you do this to fight scraping).

Otherwise, you need to be able to parse enough of the page to extract
what you want. Normally JSON is processed by JavaScript on the page, so
you will need to parse the page well enough to find it.

Your problem seems to be that you ARE trying to parse the whole page
(but not using a smart enough parser), unless the page is being
deceptive (maybe unintentionally) it may be better to just NOT try to
parse most of the page, just that part within the tags marking the data
you are looking for.

[toc] | [prev] | [next] | [standalone]

#12279

On Sat, 27 Feb 2021 14:30:42 -0500
Richard Damon <Richard@Damon-Family.org> wrote:
> On 2/27/21 11:43 AM, Spiros Bousbouras wrote:
> > On Sat, 27 Feb 2021 14:43:35 GMT
> > Spiros Bousbouras <spibou@gmail.com> wrote:
> >> I have some code which parses the source of pages for youtube videos and
> >> prints some information. My code has been working fine for months but
> >> recently it choked on  https://www.youtube.com/watch?v=w2itwFJCgFQ .After
> >> some investigations I found the part of the page which causes the problem :
> >> there is a single appearance of
> >>
> >>     content="\Raffaello D&#39;Andrea\"
> >>
> >> .My code assumed that the closing double quote is escaped so it did not count
> >> it as a closing quote and so parsed incorrectly the rest of the page.
> > 
> > I note that if I remove the 2nd backslash , my code can parse the page
> > correctly. Also , there are other appearances of the 2 character sequence  \"
> > on the page and for those the  "  is indeed escaped. So a different way to
> > approach my problem is to come up with an algorithm which decides when  \"
> > denotes an escaped double quote and when it doesn't. Ideally , I'm looking
> > for a solution which does not involve writing a full parser for HTML ,
> > Javascript , JSON and who knows what else is on such pages.
> 
> My guess is that the spots you see \" as an escaped quote it is a piece
> of JavaScript on the page, the " isn't part of an HTML tag, so isn't
> processed as part of one, but the \" is just passed through to the
> JavaScript processor. The JavaScript processor DOES use \" as an escaped
> quote inside its strings.

Actually it's part of JSON , I don't need to execute any Javascript which is
way beyond what my code supports (or I'm hoping to support).

[...]

> One issue with scraping pages is that you might want to parse the final
> page that will get rendered. If you do, you really need to run a full
> parsing and rendering engine to get the data (and some pages
> intentionally make you do this to fight scraping).

That's not the case here. Pages for youtube videos only require funky stuff
in order to get the URL for the actual video and also for the user comments
on each page. I'm not trying to get any of that.

> Otherwise, you need to be able to parse enough of the page to extract
> what you want. Normally JSON is processed by JavaScript on the page, so
> you will need to parse the page well enough to find it.
> 
> Your problem seems to be that you ARE trying to parse the whole page
> (but not using a smart enough parser), unless the page is being
> deceptive (maybe unintentionally) it may be better to just NOT try to
> parse most of the page, just that part within the tags marking the data
> you are looking for.

The page is not being deceptive intentionally otherwise I would have had
problems much earlier. I mean youtube has no reason to make the page for this
particular video deceptive so I would have had problems with other pages. For
some reason the name of the speaker in the video was entered with some
backslashes , likely by the uploader ; that was probably unintentional.

Otherwise , in your last 2 paragraphs you seem to only restate my question.
In order to find the right tags , I need to find the JSON strings ; and in
order to find those I need to locate correctly where each string (whether
it is part of JSON or part of an HTML tag) starts and ends ; and in order
to do that I need to know whether a string is part of JSON or part of an
HTML tag so that I can apply the correct rule to locate the closing double
quote.

-- 
I don't know what you were expecting ... an assassin during the Tang Dynasty
to burst out into song about her inner anguish and emotional turmoil?
  https://www.imdb.com/review/rw3290355

[toc] | [prev] | [next] | [standalone]

#12280

Spiros Bousbouras <spibou@gmail.com> writes:

> Otherwise , in your last 2 paragraphs you seem to only restate my question.
> In order to find the right tags , I need to find the JSON strings ; and in
> order to find those I need to locate correctly where each string (whether
> it is part of JSON or part of an HTML tag) starts and ends ; and in order
> to do that I need to know whether a string is part of JSON or part of an
> HTML tag so that I can apply the correct rule to locate the closing double
> quote.

I know it's annoying, but I have to correct some of your terms (or at
least clarify them) so we can all be on the same page.  An HTML tag
looks like this: <p>.  That's an opening tag.  You can also have closing
tags like </p>.  Some tags denote elements (like <hr>) all by
themselves, but often an element is constructed form the opening tag,
the content and the (sometimes implicit) closing tag.

The point of this is just to say the you won't usually find JSON in a
tag (in fact, you won't find it "native" in HTML at all).  You do
sometimes find JSON as an ECMAScript (AKA JavaScript) string.  If you
do, it will be in the content of a <script> element.

The specific string that was causing trouble (content="\Raffaello
D&#39;Andrea\") /is/ inside a tag.  It's the value of the 'content'
attribute in a 'mata' tag.  The slashes are just part of the value of
the attribute.  They don't escape anything.

I could not find any JSON in the document you cited, but I confess I did
not look all that hard.  I did find list of scripting that has parts
that look like JSON, but that's not quite the same thing.

To do what you want in the general case involves parsing HTML.  The
trouble with doing that is that real-world documents often have
ungrammatical constructs, so browsers have parsers that apply a lot of
heuristics to guess what was meant.  Fortunately, you can find HTML
parsers for many programming languages.

If you don't go that route you have to decide how much you are prepared
simply to "wing it".  Can you point out where the parts of the document
are that you want to scrape?  Maybe there is some simple work-around
that does not involve much parsing at all.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#12283

On Sat, 27 Feb 2021 21:42:01 +0000
Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
> Spiros Bousbouras <spibou@gmail.com> writes:
> 
> > Otherwise , in your last 2 paragraphs you seem to only restate my question.
> > In order to find the right tags , I need to find the JSON strings ; and in
> > order to find those I need to locate correctly where each string (whether
> > it is part of JSON or part of an HTML tag) starts and ends ; and in order
> > to do that I need to know whether a string is part of JSON or part of an
> > HTML tag so that I can apply the correct rule to locate the closing double
> > quote.
> 
> I know it's annoying, but I have to correct some of your terms (or at
> least clarify them) so we can all be on the same page.  An HTML tag
> looks like this: <p>.  That's an opening tag.  You can also have closing
> tags like </p>.  Some tags denote elements (like <hr>) all by
> themselves, but often an element is constructed form the opening tag,
> the content and the (sometimes implicit) closing tag.

No , it's not annoying and it's necessary. When I wrote "find the right tags"
I meant something different. Here is an example from the video I mentioned
earlier (www.youtube.com/watch?v=w2itwFJCgFQ) :
[I have added newlines for readability]

=== QUOTE
"videoDetails":{"videoId":"w2itwFJCgFQ",
"title":"The astounding athletic power of quadcopters | Raffaello D'Andrea",
"lengthSeconds":"968",
"keywords":["TEDTalk",
"TEDTalks",
"TED Talk",
"TED Talks",
"TEDGlobal",
"\\Raffaello D'Andrea\\",
"robots",
"technology"],
"channelId":"UCAuUUnT6oDeKwE6v1NGQxug",
"isOwnerViewing":false,
"shortDescription":"In a robot lab at TEDGlobal,
    SNIP
https://www.youtube.com/TED",
"isCrawlable":true,
    SNIP
"averageRating":4.9200745,
"allowRatings":true,
"viewCount":"9755173",
"author":"TED",
"isPrivate":false,
"isUnpluggedCorpus":false,
"isLiveContent":false},
=== ENDQUOTE

What I have been calling a tag is a string (like "videoDetails") followed by
a colon (possibly with intervening whitespace). Lets call those JS-tags .Also
, I have been referring to the segment above as JSON. I take it it's not JSON
but Javascript ? Lets call it JSON-like .I'm only interested in the JSON-like
parts of a page.

Some of the JS-tags are followed by a string , some by a  [ , some by a  { ,
some by a number and some by  true  or  false (not within quotes). So I'm
interested in finding some of those JS-tags and printing the "object" which
follows. If the JS-tag is followed by a  [  or  {  then I consider as the
object everything until the matching  ]  or  }  respectively whereas if it is
followed by a string or number or boolean then I want the string or number or
boolean.

One of the JS-tags I'm interested in is "title" but there are various
appearances of "title" on the page and the object which follows is not always
the same. I'm interested in the appearance of "title" which is followed by
the video title. So for this video

    "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea"

Through studying several video pages , the pattern I have noticed is the
following

    "videoDetails":{
        ......
        "title":"<video title>"
        .....
    }

So you have a pattern where a JS-tag "videoDetails" appears somewhere followed
by a  {  and between the opening brace and the matching  }  you have another
JS-tag  "title":"<video title>" .In all the videos I have examined so far such
a pattern appears only once. So to get the video title I need to locate the
JS-tag "videoDetails" , do bracket matching (in general you may have nested
{} and [] ) and between the matching  {  and  } , search for the JS-tag
"title" .Another JS-tag I'm interested in is  "averageRating"  but this appears
only once on each page so finding it is simpler.

Such and similar are of course only empirical observations so even if I had a
full parser for HTML , Javvascript , etc. and even if the pages faithfully
follow the relevant grammars , I would still need to rely on such heuristics
to find the information I want.

> The point of this is just to say the you won't usually find JSON in a
> tag (in fact, you won't find it "native" in HTML at all).  You do
> sometimes find JSON as an ECMAScript (AKA JavaScript) string.  If you
> do, it will be in the content of a <script> element.
> 
> The specific string that was causing trouble (content="\Raffaello
> D&#39;Andrea\") /is/ inside a tag.  It's the value of the 'content'
> attribute in a 'mata' tag.  The slashes are just part of the value of
> the attribute.  They don't escape anything.

So is there an algorithm to skip all the HTML tags and parse further the
JSON-like part ?

> I could not find any JSON in the document you cited, but I confess I did
> not look all that hard.  I did find list of scripting that has parts
> that look like JSON, but that's not quite the same thing.
> 
> To do what you want in the general case involves parsing HTML.  The
> trouble with doing that is that real-world documents often have
> ungrammatical constructs, so browsers have parsers that apply a lot of
> heuristics to guess what was meant.  Fortunately, you can find HTML
> parsers for many programming languages.
> 
> If you don't go that route you have to decide how much you are prepared
> simply to "wing it".  Can you point out where the parts of the document
> are that you want to scrape?  Maybe there is some simple work-around
> that does not involve much parsing at all.

My code is in Common Lisp. Since the code has worked correctly with dozens of
videos , I'd rather not start from scratch even if there is a Common Lisp
library for parsing HTML. And yes , I'm willing to "wing it". As I explain
above , using some heuristics would be inevitable even if I did have parsers.

-- 
A reader identifies favorite stories in serialization, then buys and saves the
paperback collections, weaving the stories into her own sense of identity. In this
way, manga become part of a reader's autobiography.
  https://web.archive.org/web/20160303170443/https://www.matt-thorn.com/shoujo_manga/japan_quarterly/index.html

[toc] | [prev] | [next] | [standalone]

#12284

Spiros Bousbouras <spibou@gmail.com> writes:

> On Sat, 27 Feb 2021 21:42:01 +0000
> Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
>> Spiros Bousbouras <spibou@gmail.com> writes:
>> 
>> > Otherwise , in your last 2 paragraphs you seem to only restate my question.
>> > In order to find the right tags , I need to find the JSON strings ; and in
>> > order to find those I need to locate correctly where each string (whether
>> > it is part of JSON or part of an HTML tag) starts and ends ; and in order
>> > to do that I need to know whether a string is part of JSON or part of an
>> > HTML tag so that I can apply the correct rule to locate the closing double
>> > quote.
>> 
>> I know it's annoying, but I have to correct some of your terms (or at
>> least clarify them) so we can all be on the same page.  An HTML tag
>> looks like this: <p>.  That's an opening tag.  You can also have closing
>> tags like </p>.  Some tags denote elements (like <hr>) all by
>> themselves, but often an element is constructed form the opening tag,
>> the content and the (sometimes implicit) closing tag.
>
> No , it's not annoying and it's necessary. When I wrote "find the right tags"
> I meant something different. Here is an example from the video I mentioned
> earlier (www.youtube.com/watch?v=w2itwFJCgFQ) :
> [I have added newlines for readability]
>
> === QUOTE
> "videoDetails":{"videoId":"w2itwFJCgFQ",
> "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea",
> "lengthSeconds":"968",
> "keywords":["TEDTalk",
> "TEDTalks",
> "TED Talk",
> "TED Talks",
> "TEDGlobal",
> "\\Raffaello D'Andrea\\",
> "robots",
> "technology"],
> "channelId":"UCAuUUnT6oDeKwE6v1NGQxug",
> "isOwnerViewing":false,
> "shortDescription":"In a robot lab at TEDGlobal,
>     SNIP
> https://www.youtube.com/TED",
> "isCrawlable":true,
>     SNIP
> "averageRating":4.9200745,
> "allowRatings":true,
> "viewCount":"9755173",
> "author":"TED",
> "isPrivate":false,
> "isUnpluggedCorpus":false,
> "isLiveContent":false},
> === ENDQUOTE
>
> What I have been calling a tag is a string (like "videoDetails") followed by
> a colon (possibly with intervening whitespace). Lets call those
> JS-tags.

Ah, I see.  So not HTML tags at all.

> Also
> , I have been referring to the segment above as JSON. I take it it's not JSON
> but Javascript ? Lets call it JSON-like .I'm only interested in the JSON-like
> parts of a page.

Yes, this is just an object initialiser in JS.  The things you call tags
are property names.

> Some of the JS-tags are followed by a string , some by a  [ , some by a  { ,
> some by a number and some by  true  or  false (not within quotes).

The property name is followed by an expression that denotes the value of
that property.  []s denote array values and {}s denote objects.

> So I'm
> interested in finding some of those JS-tags and printing the "object" which
> follows. If the JS-tag is followed by a  [  or  {  then I consider as the
> object everything until the matching  ]  or  }  respectively whereas if it is
> followed by a string or number or boolean then I want the string or number or
> boolean.

That's going to work, but only id you start at the "top".

> One of the JS-tags I'm interested in is "title" but there are various
> appearances of "title" on the page and the object which follows is not always
> the same. I'm interested in the appearance of "title" which is followed by
> the video title. So for this video
>
>     "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea"
>
> Through studying several video pages , the pattern I have noticed is the
> following
>
>     "videoDetails":{
>         ......
>         "title":"<video title>"
>         .....
>     }
>
> So you have a pattern where a JS-tag "videoDetails" appears somewhere followed
> by a  {  and between the opening brace and the matching  }  you have another
> JS-tag  "title":"<video title>" .In all the videos I have examined so far such
> a pattern appears only once. So to get the video title I need to locate the
> JS-tag "videoDetails" , do bracket matching (in general you may have nested
> {} and [] ) and between the matching  {  and  } , search for the JS-tag
> "title" .Another JS-tag I'm interested in is  "averageRating"  but this appears
> only once on each page so finding it is simpler.

You could find the top-level object (the one that has a property named
"videoDetails") and pass it to a JS processor line node:

$ cat t.js
var obj = {
    // ...
    "videoDetails":{
        // ...
        "title":"<video title>"
        // ...
    }
    // ...
};
console.log(obj.videoDetails.title);
$ node t.js
<video title>

But then you might get away with just printing the first line with "title" that
follows a line with "videoDetails".

> Such and similar are of course only empirical observations so even if I had a
> full parser for HTML , Javvascript , etc. and even if the pages faithfully
> follow the relevant grammars , I would still need to rely on such heuristics
> to find the information I want.
>
>> The point of this is just to say the you won't usually find JSON in a
>> tag (in fact, you won't find it "native" in HTML at all).  You do
>> sometimes find JSON as an ECMAScript (AKA JavaScript) string.  If you
>> do, it will be in the content of a <script> element.
>> 
>> The specific string that was causing trouble (content="\Raffaello
>> D&#39;Andrea\") /is/ inside a tag.  It's the value of the 'content'
>> attribute in a 'mata' tag.  The slashes are just part of the value of
>> the attribute.  They don't escape anything.
>
> So is there an algorithm to skip all the HTML tags and parse further the
> JSON-like part ?

The proper way is to parse the HTML and the parse the JS to find the
place where the object in question is initialised.  It's a lot of work.
I'd go for a quick and dirty search.

>> I could not find any JSON in the document you cited, but I confess I did
>> not look all that hard.  I did find list of scripting that has parts
>> that look like JSON, but that's not quite the same thing.
>> 
>> To do what you want in the general case involves parsing HTML.  The
>> trouble with doing that is that real-world documents often have
>> ungrammatical constructs, so browsers have parsers that apply a lot of
>> heuristics to guess what was meant.  Fortunately, you can find HTML
>> parsers for many programming languages.
>> 
>> If you don't go that route you have to decide how much you are prepared
>> simply to "wing it".  Can you point out where the parts of the document
>> are that you want to scrape?  Maybe there is some simple work-around
>> that does not involve much parsing at all.
>
> My code is in Common Lisp. Since the code has worked correctly with dozens of
> videos , I'd rather not start from scratch even if there is a Common Lisp
> library for parsing HTML. And yes , I'm willing to "wing it". As I explain
> above , using some heuristics would be inevitable even if I did have
> parsers.

I'd search for '"videoDetails"' and then for '"title":' and take the
string that follows the ':'.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#12293

On Sun, 28 Feb 2021 23:55:19 +0000
Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
> Spiros Bousbouras <spibou@gmail.com> writes:
[...]

> >                                            When I wrote "find the right tags"
> > I meant something different. Here is an example from the video I mentioned
> > earlier (www.youtube.com/watch?v=w2itwFJCgFQ) :
> > [I have added newlines for readability]
> >
> > === QUOTE
> > "videoDetails":{"videoId":"w2itwFJCgFQ",
> > "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea",
> > "lengthSeconds":"968",
> > "keywords":["TEDTalk",
> > "TEDTalks",
> > "TED Talk",
> > "TED Talks",
> > "TEDGlobal",
> > "\\Raffaello D'Andrea\\",
> > "robots",
> > "technology"],
> > "channelId":"UCAuUUnT6oDeKwE6v1NGQxug",
> > "isOwnerViewing":false,
> > "shortDescription":"In a robot lab at TEDGlobal,
> >     SNIP
> > https://www.youtube.com/TED",
> > "isCrawlable":true,
> >     SNIP
> > "averageRating":4.9200745,
> > "allowRatings":true,
> > "viewCount":"9755173",
> > "author":"TED",
> > "isPrivate":false,
> > "isUnpluggedCorpus":false,
> > "isLiveContent":false},
> > === ENDQUOTE
> >
> > What I have been calling a tag is a string (like "videoDetails") followed by
> > a colon (possibly with intervening whitespace). Lets call those
> > JS-tags.
> 
> Ah, I see.  So not HTML tags at all.
> 
> > Also
> > , I have been referring to the segment above as JSON. I take it it's not JSON
> > but Javascript ? Lets call it JSON-like .I'm only interested in the JSON-like
> > parts of a page.
> 
> Yes, this is just an object initialiser in JS.  The things you call tags
> are property names.
> 
> > Some of the JS-tags are followed by a string , some by a  [ , some by a  { ,
> > some by a number and some by  true  or  false (not within quotes).
> 
> The property name is followed by an expression that denotes the value of
> that property.  []s denote array values and {}s denote objects.

Ok , it's good to know the correct terminology.

> > So I'm
> > interested in finding some of those JS-tags and printing the "object" which
> > follows. If the JS-tag is followed by a  [  or  {  then I consider as the
> > object everything until the matching  ]  or  }  respectively whereas if it is
> > followed by a string or number or boolean then I want the string or number or
> > boolean.
> 
> That's going to work, but only id you start at the "top".

Where else would I start ?

> > One of the JS-tags I'm interested in is "title" but there are various
> > appearances of "title" on the page and the object which follows is not always
> > the same. I'm interested in the appearance of "title" which is followed by
> > the video title. So for this video
> >
> >     "title":"The astounding athletic power of quadcopters | Raffaello D'Andrea"
> >
> > Through studying several video pages , the pattern I have noticed is the
> > following
> >
> >     "videoDetails":{
> >         ......
> >         "title":"<video title>"
> >         .....
> >     }
> >
> > So you have a pattern where a JS-tag "videoDetails" appears somewhere followed
> > by a  {  and between the opening brace and the matching  }  you have another
> > JS-tag  "title":"<video title>" .In all the videos I have examined so far such
> > a pattern appears only once. So to get the video title I need to locate the
> > JS-tag "videoDetails" , do bracket matching (in general you may have nested
> > {} and [] ) and between the matching  {  and  } , search for the JS-tag
> > "title" .Another JS-tag I'm interested in is  "averageRating"  but this appears
> > only once on each page so finding it is simpler.
> 
> You could find the top-level object (the one that has a property named
> "videoDetails") and pass it to a JS processor line node:

Are you referring to  en.wikipedia.org/wiki/Node.js  or something more general ?

> $ cat t.js
> var obj = {
>     // ...
>     "videoDetails":{
>         // ...
>         "title":"<video title>"
>         // ...
>     }
>     // ...
> };
> console.log(obj.videoDetails.title);
> $ node t.js
> <video title>
> 

If I have managed to isolate the
      "videoDetails":{
          // ...
          "title":"<video title>"
          // ...
      }

part , my current code already works. So you are suggesting a much more
complicated solution (run a Javascript engine , if I understand correctly) to
achieve something which I already handle. The issue is to recognise the
appearance of (for example) "videoDetails" which means that I also recognise
strings which appear earlier whether they are part of HTML tags or Javascript
properties.

> But then you might get away with just printing the first line with "title" that
> follows a line with "videoDetails".

[...]

> > So is there an algorithm to skip all the HTML tags and parse further the
> > JSON-like part ?
> 
> The proper way is to parse the HTML and the parse the JS to find the
> place where the object in question is initialised.  It's a lot of work.
> I'd go for a quick and dirty search.

Up to now , the parsing of Javascript I have been doing has been working well
enough. So I'm looking for a quick and dirty way to skip the non Javascript
part.

> I'd search for '"videoDetails"' and then for '"title":' and take the
> string that follows the ':'.

So not take at all into account which property is part of which object.
Perhaps even more simplicity than what I have now is the way to go , I'll
think about it. Thanks.

-- 
This was certainly an objection heard in the West where the glissandi of the
trombone which accompanied Sergei's movements during a love scene earned the
opera the term 'pornophony'.
  http://www.quartets.de/articles/Lady_Macbeth.html

[toc] | [prev] | [next] | [standalone]

#12294

Shouldn't it be not dependent on the page you are scanning ?





Kristjan Robam

[toc] | [prev] | [next] | [standalone]

#12276

On 2/27/21 9:43 AM, Spiros Bousbouras wrote:
> I have some code which parses the source of pages for youtube videos and
> prints some information. My code has been working fine for months but
> recently it choked on  https://www.youtube.com/watch?v=w2itwFJCgFQ .After
> some investigations I found the part of the page which causes the problem :
> there is a single appearance of
> 
>     content="\Raffaello D'Andrea\"
> 
> .My code assumed that the closing double quote is escaped so it did not count
> it as a closing quote and so parsed incorrectly the rest of the page. So I
> have 2 questions :
> 
> 1. Is the above segment valid according to whatever the relevant specification
> is ?
> 
> 2. My code only needs to parse the JSON content of pages and extract the
> information I want. Is there a simple algorithm which strips all the non JSON
> content of a page ? Or alternatively , is there an algorithm which parses
> correctly constructs like the above , perhaps taking into account surrounding
> tags ?
> 
> 
> I should warn you that the source of the page has some very long lines , the
> longest being 201416 bytes. So examining the content with some text editors or
> less  will not work smoothly. Filtering the page through something like
> sed 's/,/,\n/g'   makes things a lot more manageable.
> 

I think the issue is that \" is NOT an escaped quote in HTML, but the
character \ at the end of the string.

If you want to insert a " into a string, you use @quot; or @#34;

Note the leading \R, this is not some special R character, and the '
which is an encoded ' character.

You are parsing the page with the wrong grammar (for a different
language that DOES use \" as the way to escape a quote)

[toc] | [prev] | [next] | [standalone]

#12278

On Sat, 27 Feb 2021 14:09:16 -0500
Richard Damon <Richard@Damon-Family.org> wrote:
> On 2/27/21 9:43 AM, Spiros Bousbouras wrote:
> > I have some code which parses the source of pages for youtube videos and
> > prints some information. My code has been working fine for months but
> > recently it choked on  https://www.youtube.com/watch?v=w2itwFJCgFQ .After
> > some investigations I found the part of the page which causes the problem :
> > there is a single appearance of
> > 
> >     content="\Raffaello D&#39;Andrea\"
> > 
> > .My code assumed that the closing double quote is escaped so it did not count
> > it as a closing quote and so parsed incorrectly the rest of the page. So I
> > have 2 questions :

[...]

> I think the issue is that \" is NOT an escaped quote in HTML, but the
> character \ at the end of the string.
> 
> If you want to insert a " into a string, you use @quot; or @#34;
> 
> Note the leading \R, this is not some special R character, and the &#39;
> which is an encoded ' character.
> 
> You are parsing the page with the wrong grammar (for a different
> language that DOES use \" as the way to escape a quote)

Yes part of the question is how to determine which parts of the page follow
which grammar. My grammar seems to work correctly for the JSON part of the page
and that's what I'm interested in.

[toc] | [prev] | [next] | [standalone]

#12281

On 2/27/21 3:12 PM, Spiros Bousbouras wrote:
> On Sat, 27 Feb 2021 14:09:16 -0500
> Richard Damon <Richard@Damon-Family.org> wrote:
>> On 2/27/21 9:43 AM, Spiros Bousbouras wrote:
>>> I have some code which parses the source of pages for youtube videos and
>>> prints some information. My code has been working fine for months but
>>> recently it choked on  https://www.youtube.com/watch?v=w2itwFJCgFQ .After
>>> some investigations I found the part of the page which causes the problem :
>>> there is a single appearance of
>>>
>>>     content="\Raffaello D&#39;Andrea\"
>>>
>>> .My code assumed that the closing double quote is escaped so it did not count
>>> it as a closing quote and so parsed incorrectly the rest of the page. So I
>>> have 2 questions :
> 
> [...]
> 
>> I think the issue is that \" is NOT an escaped quote in HTML, but the
>> character \ at the end of the string.
>>
>> If you want to insert a " into a string, you use @quot; or @#34;
>>
>> Note the leading \R, this is not some special R character, and the &#39;
>> which is an encoded ' character.
>>
>> You are parsing the page with the wrong grammar (for a different
>> language that DOES use \" as the way to escape a quote)
> 
> Yes part of the question is how to determine which parts of the page follow
> which grammar. My grammar seems to work correctly for the JSON part of the page
> and that's what I'm interested in.
> 

As Ben said, on an HTML page that was sent out to what it thinks is a
browser, the only place you will find 'active' JSON data is contained in
data inside a script, so withing an area denoted by a <script> element.

HOW to find that JSON will be a matter of understanding how the page is
built, and what 'JavaScript' code indicates the location of the data you
are looking for.

Unless you are concerned that some HTML tag is going to have a parameter
that looks like value="<script>", you can probably just ignore all the
stuff outside the script elements, and just parse what is inside them,
as that sounds to be the only stuff you are concerned with (unless you
need to look for something out there to determine which script elements
to process).

It should be pointed out that this data you are finding isn't really
JSON itself, but probably some string value in the script with JSON data
inside (so it will need to be parsed as a 'JavaScript' string to get to
the real JSON data. 'Real' JSON isn't embedded in an HTML page, but is
typically sent as a response on its own, and in that case, it doesn't
need to first be parsed as a 'string' to get to the JSON data.

[toc] | [prev] | [next] | [standalone]

#12292

On Sat, 27 Feb 2021 22:18:12 -0500
Richard Damon <Richard@Damon-Family.org> wrote:
> On 2/27/21 3:12 PM, Spiros Bousbouras wrote:
[...]
> > Yes part of the question is how to determine which parts of the page follow
> > which grammar. My grammar seems to work correctly for the JSON part of the page
> > and that's what I'm interested in.
> > 
> 
> As Ben said, on an HTML page that was sent out to what it thinks is a
> browser, the only place you will find 'active' JSON data is contained in
> data inside a script, so withing an area denoted by a <script> element.
> 
> HOW to find that JSON will be a matter of understanding how the page is
> built, and what 'JavaScript' code indicates the location of the data you
> are looking for.
> 
> Unless you are concerned that some HTML tag is going to have a parameter
> that looks like value="<script>", you can probably just ignore all the
> stuff outside the script elements, and just parse what is inside them,
> as that sounds to be the only stuff you are concerned with (unless you
> need to look for something out there to determine which script elements
> to process).

So one solution to pick out the Javascript part (which I have erroneously
been calling JSON) would be to pick out the stuff between  <script  ....>  and
</script ...> , yes ?

> It should be pointed out that this data you are finding isn't really
> JSON itself, but probably some string value in the script with JSON data
> inside (so it will need to be parsed as a 'JavaScript' string to get to
> the real JSON data. 'Real' JSON isn't embedded in an HTML page, but is
> typically sent as a response on its own, and in that case, it doesn't
> need to first be parsed as a 'string' to get to the JSON data.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.programming

csiph-web