Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Peter Otten <__peter__@web.de>
Newsgroups: comp.lang.python
Subject: Re: Question about how to do something in BeautifulSoup?
Date: Fri, 22 Jan 2016 16:22:54 +0100
Organization: None
Lines: 58
Message-ID: <mailman.170.1453476191.15297.python-list@python.org>
References: <CAPeSgWorAAUz7Enkbrkfiuiu5MF9U4dQo0EBCvgvzCcYFru7Zw@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
X-Trace: news.uni-berlin.de tUmQ/SUDm0knYJ87jPIRTAQ6lBG19G3/LatxCbPeTiDg==
Return-Path: <python-python-list@m.gmane.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'yet.': 0.03; 'none:': 0.05; 'subject:Question': 0.05; 'name)': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'def': 0.13; 'appropriate': 0.14; "hasn't": 0.15; '"is': 0.16; 'elem': 0.16; 'path:': 0.16; 'received:80.91.229.3': 0.16; 'received:dip0.t-ipconnect.de': 0.16; 'received:io': 0.16; 'received:plane.gmane.org': 0.16; 'received:psf.io': 0.16; 'received:t-ipconnect.de': 0.16; 'soup': 0.16; 'started:': 0.16; 'wrote:': 0.16; 'nested': 0.18; 'code,': 0.23; 'defined': 0.23; '(or': 0.23; 'third-party': 0.23; "haven't": 0.24; 'plain': 0.24; "i've": 0.25; 'header:User- Agent:1': 0.26; 'header:X-Complaints-To:1': 0.26; 'skip:# 10': 0.27; 'yield': 0.27; 'that.': 0.30; 'noticed': 0.32; 'posting': 0.32; 'extract': 0.33; 'list': 0.34; 'text': 0.35; 'done': 0.35; 'path': 0.35; 'sometimes': 0.35; 'but': 0.36; 'there': 0.36; 'modules': 0.36; 'to:addr:python-list': 0.36; 'subject:?': 0.36; 'subject:: ': 0.37; 'two': 0.37; 'say': 0.37; 'thanks': 0.37; 'received:org': 0.37; 'skip:s 40': 0.38; 'anything': 0.38; 'skip:p 20': 0.38; 'mailing': 0.38; 'google': 0.39; 'rather': 0.39; 'to:addr:python.org': 0.40; 'received:de': 0.40; 'hope': 0.61; 'skip:n 10': 0.62; 'within': 0.64; 'places': 0.64; 'today': 0.65; 'python-list': 0.66; 'subject:skip:B 10': 0.66; 'here': 0.66; 'color': 0.67; 'day': 0.67; 'levels': 0.70
X-Injected-Via-Gmane: http://gmane.org/
X-Gmane-NNTP-Posting-Host: p57bd8bb3.dip0.t-ipconnect.de
User-Agent: KNode/4.13.3
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.20+
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref: csiph.com comp.lang.python:102019

inhahe wrote:

> I hope this is an appropriate mailing list for BeautifulSoup questions,
> it's been a long time since I've used python-list and I don't remember if
> third-party modules are on topic. I did try posting to the BeautifulSoup
> mailing list on Google groups, but I've waited a day or two and my message
> hasn't been approved yet.
> 
> Say I have the following HTML (I hope this shows up as plain text here
> rather than formatting):
> 
> <div style="font-size: 20pt;"><span style="color:
> #000000;"><em><strong>"Is today the day?"</strong></em></span></div>
> 
> And I want to extract the "Is today the day?" part. There are other places
> in the document with <em> and <strong>, but this is the only place that
> uses color #000000, so I want to extract anything that's within a color
> #000000 style, even if it's nested multiple levels deep within that.
> 
> - Sometimes the color is defined as RGB(0, 0, 0) and sometimes it's
> defined as #000000
> - Sometimes the <strong> is within the <em> and sometimes the <em> is
> within the <strong>.
> - There may be other discrepancies I haven't noticed yet
> 
> How can I do this in BeautifulSoup (or is this better done in lxml.html)?
> Thanks

I don't see how to do this with a lot of glue code, but it may get you 
started:

def recursive_attr(elem, path):
    path = path.split("/")
    for name in path:
        if elem is None:
            break
        elem = getattr(elem, name)
    return elem

def find(soup):
    for outer in soup.find_all(
            "span",
            style=re.compile(r"color:\s*(RGB\(0,\s*0,\s* 0\)|#000000)")):
        for inner in [
                recursive_attr(outer, "strong/em"),
                recursive_attr(outer, "em/strong"),]:
            if inner is not None:
                yield inner.string

def normalize_ws(s):
    return " ".join(s.split())

html = ...
soup = bs4.BeautifulSoup(html)
for match in find(soup):
    print(normalize_ws(match))