Re: Find relative url in mixed text/html

Path	csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From	Rob Hills <rhills@medimorphosis.com.au>
Newsgroups	comp.lang.python
Subject	Re: Find relative url in mixed text/html
Date	Sun, 29 Nov 2015 00:25:07 +0800
Lines	51
Message-ID	<mailman.188.1448727914.20593.python-list@python.org> (permalink)
References	<mailman.182.1448678122.20593.python-list@python.org> <8737vqyag1.fsf@jester.gateway.pace.com>
Reply-To	rhills@medimorphosis.com.au
Mime-Version	1.0
Content-Type	text/plain; charset=windows-1252
Content-Transfer-Encoding	quoted-printable
X-Trace	news.uni-berlin.de bQAnrbQ4pgJy4DLzhUgsHwDaf8AQFnpvqDIpRdjaq9Tw==
Return-Path	<rhills@medimorphosis.com.au>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.013
X-Spam-Evidence	'H': 0.97; 'S': 0.00; 'subject:text': 0.04; 'badly': 0.07; 'rest,': 0.07; 'versions.': 0.07; 'php,': 0.09; 'separating': 0.09; 'python': 0.10; 'itself.': 0.11; 'suggest': 0.15; 'instead.': 0.15; '(relative': 0.16; 'crude': 0.16; 'investigate': 0.16; 'python-based': 0.16; 'received:74.55.86': 0.16; 'received:74.55.86.74': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'received:smtp.webfaction.com': 0.16; 'received:webfaction.com': 0.16; 'soup': 0.16; 'wrote:': 0.16; 'looked': 0.16; 'project,': 0.18; '(on': 0.22; 'parsing': 0.22; 'text,': 0.22; 'cheers,': 0.22; 'seems': 0.23; 'header:In-Reply- To:1': 0.24; 'paul': 0.24; 'testing': 0.25; 'hosting': 0.25; "i've": 0.25; 'header:User-Agent:1': 0.26; "doesn't": 0.26; 'earlier': 0.27; 'coding': 0.27; 'least': 0.27; 'converting': 0.27; 'said,': 0.27; 'formed': 0.29; 'there.': 0.30; "i'm": 0.30; 'subject:/': 0.30; 'that.': 0.30; 'task': 0.30; "i'd": 0.31; 'seconds': 0.31; 'probably': 0.31; 'another': 0.32; 'especially': 0.32; 'embedded': 0.32; 'skills.': 0.32; 'maybe': 0.33; 'options': 0.33; 'useful': 0.33; 'grateful': 0.33; 'hopefully': 0.33; 'running': 0.34; 'info': 0.34; 'gets': 0.35; 'gives': 0.35; 'could': 0.35; 'stopped': 0.35; 'but': 0.36; 'there': 0.36; '(and': 0.36; 'beginning': 0.36; 'heard': 0.36; 'urls': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'really': 0.37; 'agree': 0.37; 'wrong': 0.38; 'several': 0.38; 'data': 0.39; 'takes': 0.39; 'received:192': 0.39; 'to:addr:python.org': 0.40; 'still': 0.40; 'australia': 0.61; 'default': 0.61; 'charset:windows-1252': 0.62; 'relatively': 0.63; 'more': 0.63; 'different': 0.63; 'within': 0.64; 'our': 0.64; 'believe': 0.66; 'beautiful': 0.66; 'laptop': 0.67; 'header:Reply-To:1': 0.67; 'therefore': 0.67; 'choose': 0.68; 'reply-to:no real name:2**0': 0.71; 'opinions': 0.71; 'loose': 0.84; 'phpbb': 0.84; 'rap': 0.84; 'western': 0.89; 'belief': 0.91; 'have.': 0.93; 'hills': 0.93
X-Enigmail-Draft-Status	N1110
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0
In-Reply-To	<8737vqyag1.fsf@jester.gateway.pace.com>
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.20+
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref	csiph.com comp.lang.python:99667

Show key headers only | View raw

Hi Paul,

On 28/11/15 13:11, Paul Rubin wrote:
> Rob Hills <rhills@medimorphosis.com.au> writes:
>> Note, in the beginning of this project, I looked at using "Beautiful
>> Soup" but my reading and limited testing lead me to believe that it is
>> designed for well-formed HTML/XML and therefore was unsuitable for the
>> text/html soup I have.  If that belief is incorrect, I'd be grateful for
>> general tips about using Beautiful Soup in this scenario...
> Beautiful Soup can deal with badly formed HTML pretty well, or at least
> it could in earlier versions.  It gives you several different parsing
> options to choose from now.  I think the default is lxml which is fast
> but maybe more strict.  Check what the others are and see if a loose
> slow one is still there.  It really is pretty slow so plan on a big
> computation task if you're converting a large forum.

I've had another look at Beautiful Soup and while it doesn't really help
me much with urls (relative or absolute) embedded within text, it seems
to do a good job of separating out links from the rest, so that could be
useful in itself.

WRT time, I'm converting about 65MB of data which currently takes 14
seconds (on a 3yo laptop with a SSD running Ubuntu), which I reckon is
pretty amazing performance for Python3, especially given my relatively
crude coding skills.  It'll be interesting to see if using Beautiful
Soup adds significantly to that.

> phpBB gets a bad rap that's maybe well-deserved but I don't know what to
> suggest instead.

I did start to investigate Python-based alternatives; I've not heard
much good said about php, but I probably move in the wrong circles. 
However, our hosting service doesn't support Python so I stopped
hunting.  Plus there is a significant group of forum members who hold
very strong opinions about the functionality they want and it took a lot
of work to get them to agree on something!

All that said, I'd be interested to see specific (and hopefully
unbiased) info about phpBB's failings...

Cheers,

-- 
Rob Hills
Waikiki, Western Australia

Thread

Find relative url in mixed text/html Rob Hills <rhills@medimorphosis.com.au> - 2015-11-28 10:35 +0800
  Re: Find relative url in mixed text/html Paul Rubin <no.email@nospam.invalid> - 2015-11-27 21:11 -0800
    Re: Find relative url in mixed text/html Rob Hills <rhills@medimorphosis.com.au> - 2015-11-29 00:25 +0800
    Re: Find relative url in mixed text/html Laura Creighton <lac@openend.se> - 2015-11-28 18:04 +0100
    Re: Find relative url in mixed text/html Rob Hills <rhills@medimorphosis.com.au> - 2015-11-29 01:40 +0800
      Re: Find relative url in mixed text/html Paul Rubin <no.email@nospam.invalid> - 2015-11-28 10:10 -0800
  Re: Find relative url in mixed text/html Grobu <snailcoder@retrosite.invalid> - 2015-11-28 08:07 +0100
    Re: Find relative url in mixed text/html Rob Hills <rhills@medimorphosis.com.au> - 2015-11-29 01:44 +0800

csiph-web