Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Rob Hills <rhills@medimorphosis.com.au>
Newsgroups: comp.lang.python
Subject: Find relative url in mixed text/html
Date: Sat, 28 Nov 2015 10:35:16 +0800
Lines: 67
Message-ID: <mailman.182.1448678122.20593.python-list@python.org>
Reply-To: rhills@medimorphosis.com.au
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de PY2L0ys7TO/tBiODcH6W1gOGZDESLscK4Rykk91rsmPg==
Return-Path: <rhills@medimorphosis.com.au>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.128
X-Spam-Level: *
X-Spam-Evidence: '*H*': 0.75; '*S*': 0.00; 'subject:text': 0.04; 'converts': 0.07; 'tia,': 0.09; 'python': 0.10; 'volunteer': 0.11; 'prototype': 0.15; 'googling': 0.16; 'received:74.55.86': 0.16; 'received:74.55.86.74': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'received:smtp.webfaction.com': 0.16; 'received:webfaction.com': 0.16; 'sins': 0.16; 'soup': 0.16; 'looked': 0.16; 'url:au': 0.18; 'project,': 0.18; 'mixed': 0.22; 'text,': 0.22; 'trying': 0.22; 'examples': 0.24; 'plain': 0.24; 'testing': 0.25; 'header:User-Agent:1': 0.26; 'question': 0.27; 'converting': 0.27; 'regular': 0.29; 'skip:/ 40': 0.29; 'convert': 0.29; "i'm": 0.30; 'subject:/': 0.30; 'code': 0.30; 'posts': 0.30; 'relative': 0.30; 'skip:[ 10': 0.31; "i'd": 0.31; 'skip:s 30': 0.31; 'post': 0.31; 'another': 0.32; 'are:': 0.32; 'extract': 0.33; 'grateful': 0.33; 'skip:h 40': 0.33; 'list': 0.34; 'so,': 0.35; 'ones': 0.35; 'done': 0.35; 'sometimes': 0.35; 'but': 0.36; 'beginning': 0.36; 'to:addr:python-list': 0.36; 'starting': 0.37; 'wrong': 0.38; 'skip:p 20': 0.38; 'hi,': 0.38; 'received:192': 0.39; 'to:addr:python.org': 0.40; 'some': 0.40; 'australia': 0.61; 'identify': 0.61; 'our': 0.64; 'management': 0.64; 'believe': 0.66; 'beautiful': 0.66; 'talking': 0.67; 'header:Reply-To:1': 0.67; 'therefore': 0.67; 'reply-to:no real name:2**0': 0.71; 'walk': 0.72; 'url:asp': 0.80; '&lt;br': 0.84; '95%': 0.84; 'battery': 0.84; 'skip:/ 30': 0.84; 'url:forums': 0.84; 'url:tid': 0.84; 'western': 0.89; '&lt;a': 0.91; 'belief': 0.91; 'comprise': 0.91; 'fragment': 0.91; 'migrating': 0.91; 'have.': 0.93; 'hills': 0.93
X-Enigmail-Draft-Status: N1110
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0
X-Content-Filtered-By: Mailman/MimeDel 2.1.20+
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.20+
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref: csiph.com comp.lang.python:99651

Hi,

For my sins I am migrating a volunteer association forum from one
platform (WebWiz) to another (phpBB).  I am (I hope) 95% of the way
through the process.

Posts to our original forum comprise a soup of plain text, HTML and
BBCodes.  A post */may/* include links done as either standard HTML
links ( <a href=3D... ), BBCode links ( [url]http://... [/url] ) or
sometimes just text: ( http://blah.blah.com.au or even just
www.blah.blah.com.au ).

In my conversion process, I am trying to identify cross-links (links
from one post on the forum to another) so I can convert them to links
that will work in the new forum.

My current code uses a Regular Expression (yes, I read the recent posts
on this forum about regex and HTML!) to pull out "absolute" links (
starting with http:// ) and then I use Python to identify and convert
the specific links I am interested in.  However, the forum also contains
"cross-links" done using relative links and I'm unsure how best to
proceed with that one.  Googling so far has not been helpful, but that
might be me using the wrong search terms.=20

Some examples of what I am talking about are:

    Post fragment containing an "Absolute" cross-link:

    <br />ive made a new thread:
    <br />http://www.aeva.asn.au/forums/forum_posts.asp?TID=3D316&PID=3D1=
958#1958
    <br />

    converts to:

    <br />
    <br />ive made a new thread:
    <br />/viewtopic.php?t=3D316&p=3D1958#1958

    Post fragment containing a "Relative" cross-link:

    <font size=3D"3"><u>Battery Management System</u></font><br /><a href=
=3D"/forum_posts.asp?TID=3D980&PID=3D15479#15479" target=3D"_blank" rel=3D=
"nofollow">Veroboard prototype</a><br />

    Needs converting to:

    <font size=3D"3"><u>Battery Management System</u></font><br /><a href=
=3D"/viewtopic.php?p=3D15479&t=3D980#15479" target=3D"_blank" rel=3D"nofo=
llow">Veroboard prototype</a><br />

So, my question is:  What is the best way to extract a list of "relative
links" from mixed text/html that I can then walk through to identify the
specific ones I want to convert?

Note, in the beginning of this project, I looked at using "Beautiful
Soup" but my reading and limited testing lead me to believe that it is
designed for well-formed HTML/XML and therefore was unsuitable for the
text/html soup I have.  If that belief is incorrect, I'd be grateful for
general tips about using Beautiful Soup in this scenario...

TIA,

--=20
Rob Hills
Waikiki, Western Australia