Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #12277

Re: Advice needed on parsing a youtube page

Show all headers | View raw

On 2/27/21 11:43 AM, Spiros Bousbouras wrote:
> On Sat, 27 Feb 2021 14:43:35 GMT
> Spiros Bousbouras <spibou@gmail.com> wrote:
>> I have some code which parses the source of pages for youtube videos and
>> prints some information. My code has been working fine for months but
>> recently it choked on  https://www.youtube.com/watch?v=w2itwFJCgFQ .After
>> some investigations I found the part of the page which causes the problem :
>> there is a single appearance of
>>
>>     content="\Raffaello D&#39;Andrea\"
>>
>> .My code assumed that the closing double quote is escaped so it did not count
>> it as a closing quote and so parsed incorrectly the rest of the page.
> 
> I note that if I remove the 2nd backslash , my code can parse the page
> correctly. Also , there are other appearances of the 2 character sequence  \"
> on the page and for those the  "  is indeed escaped. So a different way to
> approach my problem is to come up with an algorithm which decides when  \"
> denotes an escaped double quote and when it doesn't. Ideally , I'm looking
> for a solution which does not involve writing a full parser for HTML ,
> Javascript , JSON and who knows what else is on such pages.

My guess is that the spots you see \" as an escaped quote it is a piece
of JavaScript on the page, the " isn't part of an HTML tag, so isn't
processed as part of one, but the \" is just passed through to the
JavaScript processor. The JavaScript processor DOES use \" as an escaped
quote inside its strings.
> 
>> So I have 2 questions :
>>
>> 1. Is the above segment valid according to whatever the relevant specification
>> is ?
>>
>> 2. My code only needs to parse the JSON content of pages and extract the
>> information I want. Is there a simple algorithm which strips all the non JSON
>> content of a page ? Or alternatively , is there an algorithm which parses
>> correctly constructs like the above , perhaps taking into account surrounding
>> tags ?
> 

One issue with scraping pages is that you might want to parse the final
page that will get rendered. If you do, you really need to run a full
parsing and rendering engine to get the data (and some pages
intentionally make you do this to fight scraping).

Otherwise, you need to be able to parse enough of the page to extract
what you want. Normally JSON is processed by JavaScript on the page, so
you will need to parse the page well enough to find it.

Your problem seems to be that you ARE trying to parse the whole page
(but not using a smart enough parser), unless the page is being
deceptive (maybe unintentionally) it may be better to just NOT try to
parse most of the page, just that part within the tags marking the data
you are looking for.

Back to comp.programming | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 14:43 +0000
  Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 16:43 +0000
    Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 14:30 -0500
      Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 20:30 +0000
        Re: Advice needed on parsing a youtube page Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-02-27 21:42 +0000
          Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-28 21:02 +0000
            Re: Advice needed on parsing a youtube page Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-02-28 23:55 +0000
              Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-03-02 18:51 +0000
                Re: Advice needed on parsing a youtube page Kristjan Robam <he12457@hotmail.com> - 2021-03-02 17:50 -0800
  Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 14:09 -0500
    Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 20:12 +0000
      Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 22:18 -0500
        Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-03-02 17:30 +0000

csiph-web