Path: csiph.com!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: Ben Bacarisse <ben.usenet@bsb.me.uk>
Newsgroups: comp.programming
Subject: Re: Advice needed on parsing a youtube page
Date: Sat, 27 Feb 2021 21:42:01 +0000
Organization: A noiseless patient Spider
Lines: 44
Message-ID: <87sg5hp4ra.fsf@bsb.me.uk>
References: <Rafaello@bongo-ra.co> <Rafeallo@bongo-ra.co> <C5x_H.21182$gG4.9701@fx13.iad> <rafaello@bongo-ra.co>
Mime-Version: 1.0
Content-Type: text/plain
Injection-Info: reader02.eternal-september.org; posting-host="dc3708ef38deaea37a4d3cec20cfd632"; logging-data="3289"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX1+9WHJgG3aslVLkHvhEMeQhdLaclL9qHhg="
Cancel-Lock: sha1:X4BUk6TlgImGoHjyrHF2TloBXts= sha1:RE5H0HJmVATUelDzqYlRHl3qZ/o=
X-BSB-Auth: 1.c14a24a6ce84e4c2ee1d.20210227214201GMT.87sg5hp4ra.fsf@bsb.me.uk
Xref: csiph.com comp.programming:12280

Spiros Bousbouras <spibou@gmail.com> writes:

> Otherwise , in your last 2 paragraphs you seem to only restate my question.
> In order to find the right tags , I need to find the JSON strings ; and in
> order to find those I need to locate correctly where each string (whether
> it is part of JSON or part of an HTML tag) starts and ends ; and in order
> to do that I need to know whether a string is part of JSON or part of an
> HTML tag so that I can apply the correct rule to locate the closing double
> quote.

I know it's annoying, but I have to correct some of your terms (or at
least clarify them) so we can all be on the same page.  An HTML tag
looks like this: <p>.  That's an opening tag.  You can also have closing
tags like </p>.  Some tags denote elements (like <hr>) all by
themselves, but often an element is constructed form the opening tag,
the content and the (sometimes implicit) closing tag.

The point of this is just to say the you won't usually find JSON in a
tag (in fact, you won't find it "native" in HTML at all).  You do
sometimes find JSON as an ECMAScript (AKA JavaScript) string.  If you
do, it will be in the content of a <script> element.

The specific string that was causing trouble (content="\Raffaello
D&#39;Andrea\") /is/ inside a tag.  It's the value of the 'content'
attribute in a 'mata' tag.  The slashes are just part of the value of
the attribute.  They don't escape anything.

I could not find any JSON in the document you cited, but I confess I did
not look all that hard.  I did find list of scripting that has parts
that look like JSON, but that's not quite the same thing.

To do what you want in the general case involves parsing HTML.  The
trouble with doing that is that real-world documents often have
ungrammatical constructs, so browsers have parsers that apply a lot of
heuristics to guess what was meant.  Fortunately, you can find HTML
parsers for many programming languages.

If you don't go that route you have to decide how much you are prepared
simply to "wing it".  Can you point out where the parts of the document
are that you want to scrape?  Maybe there is some simple work-around
that does not involve much parsing at all.

-- 
Ben.