Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #12281

Re: Advice needed on parsing a youtube page

Show all headers | View raw

On 2/27/21 3:12 PM, Spiros Bousbouras wrote:
> On Sat, 27 Feb 2021 14:09:16 -0500
> Richard Damon <Richard@Damon-Family.org> wrote:
>> On 2/27/21 9:43 AM, Spiros Bousbouras wrote:
>>> I have some code which parses the source of pages for youtube videos and
>>> prints some information. My code has been working fine for months but
>>> recently it choked on  https://www.youtube.com/watch?v=w2itwFJCgFQ .After
>>> some investigations I found the part of the page which causes the problem :
>>> there is a single appearance of
>>>
>>>     content="\Raffaello D&#39;Andrea\"
>>>
>>> .My code assumed that the closing double quote is escaped so it did not count
>>> it as a closing quote and so parsed incorrectly the rest of the page. So I
>>> have 2 questions :
> 
> [...]
> 
>> I think the issue is that \" is NOT an escaped quote in HTML, but the
>> character \ at the end of the string.
>>
>> If you want to insert a " into a string, you use @quot; or @#34;
>>
>> Note the leading \R, this is not some special R character, and the &#39;
>> which is an encoded ' character.
>>
>> You are parsing the page with the wrong grammar (for a different
>> language that DOES use \" as the way to escape a quote)
> 
> Yes part of the question is how to determine which parts of the page follow
> which grammar. My grammar seems to work correctly for the JSON part of the page
> and that's what I'm interested in.
> 

As Ben said, on an HTML page that was sent out to what it thinks is a
browser, the only place you will find 'active' JSON data is contained in
data inside a script, so withing an area denoted by a <script> element.

HOW to find that JSON will be a matter of understanding how the page is
built, and what 'JavaScript' code indicates the location of the data you
are looking for.

Unless you are concerned that some HTML tag is going to have a parameter
that looks like value="<script>", you can probably just ignore all the
stuff outside the script elements, and just parse what is inside them,
as that sounds to be the only stuff you are concerned with (unless you
need to look for something out there to determine which script elements
to process).

It should be pointed out that this data you are finding isn't really
JSON itself, but probably some string value in the script with JSON data
inside (so it will need to be parsed as a 'JavaScript' string to get to
the real JSON data. 'Real' JSON isn't embedded in an HTML page, but is
typically sent as a response on its own, and in that case, it doesn't
need to first be parsed as a 'string' to get to the JSON data.

Back to comp.programming | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 14:43 +0000
  Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 16:43 +0000
    Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 14:30 -0500
      Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 20:30 +0000
        Re: Advice needed on parsing a youtube page Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-02-27 21:42 +0000
          Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-28 21:02 +0000
            Re: Advice needed on parsing a youtube page Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-02-28 23:55 +0000
              Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-03-02 18:51 +0000
                Re: Advice needed on parsing a youtube page Kristjan Robam <he12457@hotmail.com> - 2021-03-02 17:50 -0800
  Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 14:09 -0500
    Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-02-27 20:12 +0000
      Re: Advice needed on parsing a youtube page Richard Damon <Richard@Damon-Family.org> - 2021-02-27 22:18 -0500
        Re: Advice needed on parsing a youtube page Spiros Bousbouras <spibou@gmail.com> - 2021-03-02 17:30 +0000

csiph-web