Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #12279

Re: Advice needed on parsing a youtube page

Show all headers | View raw

On Sat, 27 Feb 2021 14:30:42 -0500
Richard Damon <Richard@Damon-Family.org> wrote:
> On 2/27/21 11:43 AM, Spiros Bousbouras wrote:
> > On Sat, 27 Feb 2021 14:43:35 GMT
> > Spiros Bousbouras <spibou@gmail.com> wrote:
> >> I have some code which parses the source of pages for youtube videos and
> >> prints some information. My code has been working fine for months but
> >> recently it choked on  https://www.youtube.com/watch?v=w2itwFJCgFQ .After
> >> some investigations I found the part of the page which causes the problem :
> >> there is a single appearance of
> >>
> >>     content="\Raffaello D&#39;Andrea\"
> >>
> >> .My code assumed that the closing double quote is escaped so it did not count
> >> it as a closing quote and so parsed incorrectly the rest of the page.
> > 
> > I note that if I remove the 2nd backslash , my code can parse the page
> > correctly. Also , there are other appearances of the 2 character sequence  \"
> > on the page and for those the  "  is indeed escaped. So a different way to
> > approach my problem is to come up with an algorithm which decides when  \"
> > denotes an escaped double quote and when it doesn't. Ideally , I'm looking
> > for a solution which does not involve writing a full parser for HTML ,
> > Javascript , JSON and who knows what else is on such pages.
> 
> My guess is that the spots you see \" as an escaped quote it is a piece
> of JavaScript on the page, the " isn't part of an HTML tag, so isn't
> processed as part of one, but the \" is just passed through to the
> JavaScript processor. The JavaScript processor DOES use \" as an escaped
> quote inside its strings.

Actually it's part of JSON , I don't need to execute any Javascript which is
way beyond what my code supports (or I'm hoping to support).

[...]

> One issue with scraping pages is that you might want to parse the final
> page that will get rendered. If you do, you really need to run a full
> parsing and rendering engine to get the data (and some pages
> intentionally make you do this to fight scraping).

That's not the case here. Pages for youtube videos only require funky stuff
in order to get the URL for the actual video and also for the user comments
on each page. I'm not trying to get any of that.

> Otherwise, you need to be able to parse enough of the page to extract
> what you want. Normally JSON is processed by JavaScript on the page, so
> you will need to parse the page well enough to find it.
> 
> Your problem seems to be that you ARE trying to parse the whole page
> (but not using a smart enough parser), unless the page is being
> deceptive (maybe unintentionally) it may be better to just NOT try to
> parse most of the page, just that part within the tags marking the data
> you are looking for.

The page is not being deceptive intentionally otherwise I would have had
problems much earlier. I mean youtube has no reason to make the page for this
particular video deceptive so I would have had problems with other pages. For
some reason the name of the speaker in the video was entered with some
backslashes , likely by the uploader ; that was probably unintentional.

Otherwise , in your last 2 paragraphs you seem to only restate my question.
In order to find the right tags , I need to find the JSON strings ; and in
order to find those I need to locate correctly where each string (whether
it is part of JSON or part of an HTML tag) starts and ends ; and in order
to do that I need to know whether a string is part of JSON or part of an
HTML tag so that I can apply the correct rule to locate the closing double
quote.

-- 
I don't know what you were expecting ... an assassin during the Tang Dynasty
to burst out into song about her inner anguish and emotional turmoil?
  https://www.imdb.com/review/rw3290355

Back to comp.programming | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 14:43 +0000
  Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 16:43 +0000
    Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 14:30 -0500
      Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 20:30 +0000
        Re: Advice needed on parsing a youtube page Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-02-27 21:42 +0000
          Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-28 21:02 +0000
            Re: Advice needed on parsing a youtube page Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-02-28 23:55 +0000
              Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-03-02 18:51 +0000
                Re: Advice needed on parsing a youtube page Kristjan Robam <he12457@hotmail.com> - 2021-03-02 17:50 -0800
  Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 14:09 -0500
    Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 20:12 +0000
      Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 22:18 -0500
        Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-03-02 17:30 +0000

csiph-web