X-Received: by 2002:a37:4589:: with SMTP id s131mr8340869qka.269.1614457857963; Sat, 27 Feb 2021 12:30:57 -0800 (PST) MIME-Version: 1.0 Path: csiph.com!xmission!news.snarked.org!border2.nntp.dca1.giganews.com!nntp.giganews.com!news-out.google.com!nntp.google.com!peer02.iad!feed-me.highwinds-media.com!peer04.ams1!peer.ams1.xlned.com!news.xlned.com!peer03.ams4!peer.am4.highwinds-media.com!news.highwinds-media.com!fx23.ams4.POSTED!not-for-mail From: Spiros Bousbouras Newsgroups: comp.programming Subject: Re: Advice needed on parsing a youtube page Message-ID: References: In-Reply-To: X-Organisation: Weyland-Yutani Lines: 72 X-Complaints-To: http://netreport.virginmedia.com NNTP-Posting-Date: Sat, 27 Feb 2021 20:30:55 UTC Organization: virginmedia.com Date: Sat, 27 Feb 2021 20:30:55 GMT X-Received-Bytes: 4637 X-Received-Body-CRC: 3476754736 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Original-Bytes: 4448 Xref: csiph.com comp.programming:12279 On Sat, 27 Feb 2021 14:30:42 -0500 Richard Damon wrote: > On 2/27/21 11:43 AM, Spiros Bousbouras wrote: > > On Sat, 27 Feb 2021 14:43:35 GMT > > Spiros Bousbouras wrote: > >> I have some code which parses the source of pages for youtube videos and > >> prints some information. My code has been working fine for months but > >> recently it choked on https://www.youtube.com/watch?v=w2itwFJCgFQ .After > >> some investigations I found the part of the page which causes the problem : > >> there is a single appearance of > >> > >> content="\Raffaello D'Andrea\" > >> > >> .My code assumed that the closing double quote is escaped so it did not count > >> it as a closing quote and so parsed incorrectly the rest of the page. > > > > I note that if I remove the 2nd backslash , my code can parse the page > > correctly. Also , there are other appearances of the 2 character sequence \" > > on the page and for those the " is indeed escaped. So a different way to > > approach my problem is to come up with an algorithm which decides when \" > > denotes an escaped double quote and when it doesn't. Ideally , I'm looking > > for a solution which does not involve writing a full parser for HTML , > > Javascript , JSON and who knows what else is on such pages. > > My guess is that the spots you see \" as an escaped quote it is a piece > of JavaScript on the page, the " isn't part of an HTML tag, so isn't > processed as part of one, but the \" is just passed through to the > JavaScript processor. The JavaScript processor DOES use \" as an escaped > quote inside its strings. Actually it's part of JSON , I don't need to execute any Javascript which is way beyond what my code supports (or I'm hoping to support). [...] > One issue with scraping pages is that you might want to parse the final > page that will get rendered. If you do, you really need to run a full > parsing and rendering engine to get the data (and some pages > intentionally make you do this to fight scraping). That's not the case here. Pages for youtube videos only require funky stuff in order to get the URL for the actual video and also for the user comments on each page. I'm not trying to get any of that. > Otherwise, you need to be able to parse enough of the page to extract > what you want. Normally JSON is processed by JavaScript on the page, so > you will need to parse the page well enough to find it. > > Your problem seems to be that you ARE trying to parse the whole page > (but not using a smart enough parser), unless the page is being > deceptive (maybe unintentionally) it may be better to just NOT try to > parse most of the page, just that part within the tags marking the data > you are looking for. The page is not being deceptive intentionally otherwise I would have had problems much earlier. I mean youtube has no reason to make the page for this particular video deceptive so I would have had problems with other pages. For some reason the name of the speaker in the video was entered with some backslashes , likely by the uploader ; that was probably unintentional. Otherwise , in your last 2 paragraphs you seem to only restate my question. In order to find the right tags , I need to find the JSON strings ; and in order to find those I need to locate correctly where each string (whether it is part of JSON or part of an HTML tag) starts and ends ; and in order to do that I need to know whether a string is part of JSON or part of an HTML tag so that I can apply the correct rule to locate the closing double quote. -- I don't know what you were expecting ... an assassin during the Tang Dynasty to burst out into song about her inner anguish and emotional turmoil? https://www.imdb.com/review/rw3290355