Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail From: Rob Hills Newsgroups: comp.lang.python Subject: Re: Find relative url in mixed text/html Date: Sun, 29 Nov 2015 00:25:07 +0800 Lines: 51 Message-ID: References: <8737vqyag1.fsf@jester.gateway.pace.com> Reply-To: rhills@medimorphosis.com.au Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Trace: news.uni-berlin.de bQAnrbQ4pgJy4DLzhUgsHwDaf8AQFnpvqDIpRdjaq9Tw== Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.013 X-Spam-Evidence: '*H*': 0.97; '*S*': 0.00; 'subject:text': 0.04; 'badly': 0.07; 'rest,': 0.07; 'versions.': 0.07; 'php,': 0.09; 'separating': 0.09; 'python': 0.10; 'itself.': 0.11; 'suggest': 0.15; 'instead.': 0.15; '(relative': 0.16; 'crude': 0.16; 'investigate': 0.16; 'python-based': 0.16; 'received:74.55.86': 0.16; 'received:74.55.86.74': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'received:smtp.webfaction.com': 0.16; 'received:webfaction.com': 0.16; 'soup': 0.16; 'wrote:': 0.16; 'looked': 0.16; 'project,': 0.18; '(on': 0.22; 'parsing': 0.22; 'text,': 0.22; 'cheers,': 0.22; 'seems': 0.23; 'header:In-Reply- To:1': 0.24; 'paul': 0.24; 'testing': 0.25; 'hosting': 0.25; "i've": 0.25; 'header:User-Agent:1': 0.26; "doesn't": 0.26; 'earlier': 0.27; 'coding': 0.27; 'least': 0.27; 'converting': 0.27; 'said,': 0.27; 'formed': 0.29; 'there.': 0.30; "i'm": 0.30; 'subject:/': 0.30; 'that.': 0.30; 'task': 0.30; "i'd": 0.31; 'seconds': 0.31; 'probably': 0.31; 'another': 0.32; 'especially': 0.32; 'embedded': 0.32; 'skills.': 0.32; 'maybe': 0.33; 'options': 0.33; 'useful': 0.33; 'grateful': 0.33; 'hopefully': 0.33; 'running': 0.34; 'info': 0.34; 'gets': 0.35; 'gives': 0.35; 'could': 0.35; 'stopped': 0.35; 'but': 0.36; 'there': 0.36; '(and': 0.36; 'beginning': 0.36; 'heard': 0.36; 'urls': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'really': 0.37; 'agree': 0.37; 'wrong': 0.38; 'several': 0.38; 'data': 0.39; 'takes': 0.39; 'received:192': 0.39; 'to:addr:python.org': 0.40; 'still': 0.40; 'australia': 0.61; 'default': 0.61; 'charset:windows-1252': 0.62; 'relatively': 0.63; 'more': 0.63; 'different': 0.63; 'within': 0.64; 'our': 0.64; 'believe': 0.66; 'beautiful': 0.66; 'laptop': 0.67; 'header:Reply-To:1': 0.67; 'therefore': 0.67; 'choose': 0.68; 'reply-to:no real name:2**0': 0.71; 'opinions': 0.71; 'loose': 0.84; 'phpbb': 0.84; 'rap': 0.84; 'western': 0.89; 'belief': 0.91; 'have.': 0.93; 'hills': 0.93 X-Enigmail-Draft-Status: N1110 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 In-Reply-To: <8737vqyag1.fsf@jester.gateway.pace.com> X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.20+ Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Xref: csiph.com comp.lang.python:99667 Hi Paul, On 28/11/15 13:11, Paul Rubin wrote: > Rob Hills writes: >> Note, in the beginning of this project, I looked at using "Beautiful >> Soup" but my reading and limited testing lead me to believe that it is= >> designed for well-formed HTML/XML and therefore was unsuitable for the= >> text/html soup I have. If that belief is incorrect, I'd be grateful f= or >> general tips about using Beautiful Soup in this scenario... > Beautiful Soup can deal with badly formed HTML pretty well, or at least= > it could in earlier versions. It gives you several different parsing > options to choose from now. I think the default is lxml which is fast > but maybe more strict. Check what the others are and see if a loose > slow one is still there. It really is pretty slow so plan on a big > computation task if you're converting a large forum. I've had another look at Beautiful Soup and while it doesn't really help me much with urls (relative or absolute) embedded within text, it seems to do a good job of separating out links from the rest, so that could be useful in itself. WRT time, I'm converting about 65MB of data which currently takes 14 seconds (on a 3yo laptop with a SSD running Ubuntu), which I reckon is pretty amazing performance for Python3, especially given my relatively crude coding skills. It'll be interesting to see if using Beautiful Soup adds significantly to that. > phpBB gets a bad rap that's maybe well-deserved but I don't know what t= o > suggest instead. I did start to investigate Python-based alternatives; I've not heard much good said about php, but I probably move in the wrong circles.=20 However, our hosting service doesn't support Python so I stopped hunting. Plus there is a significant group of forum members who hold very strong opinions about the functionality they want and it took a lot of work to get them to agree on something! All that said, I'd be interested to see specific (and hopefully unbiased) info about phpBB's failings... Cheers, --=20 Rob Hills Waikiki, Western Australia