Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Peter Otten <__peter__@web.de>
Newsgroups: comp.lang.python
Subject: Re: non printable (moving away from Perl)
Date: Fri, 11 Mar 2016 16:22:16 +0100
Organization: None
Lines: 62
Message-ID: <mailman.24.1457709748.26429.python-list@python.org>
References: <nbt27u$fe7$1@gioia.aioe.org> <mailman.17.1457698399.26429.python-list@python.org> <nbukcd$gs2$1@gioia.aioe.org>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
X-Trace: news.uni-berlin.de GyDjxhFDZSLAgwFDXaxwbgK5d7WiqJ5j2hKhMN52s8IA==
Return-Path: <python-python-list@m.gmane.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'else:': 0.03; 'lines,': 0.05; 'python3': 0.05; 'sys': 0.05; 'newline': 0.07; 'lookup': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'python': 0.10; 'def': 0.13; 'compares': 0.16; 'received:80.91.229.3': 0.16; 'received:dip0.t-ipconnect.de': 0.16; 'received:io': 0.16; 'received:plane.gmane.org': 0.16; 'received:psf.io': 0.16; 'received:t-ipconnect.de': 0.16; 'subject:non': 0.16; 'wrote:': 0.16; '>>>': 0.20; 'am,': 0.23; 'defined': 0.23; 'seems': 0.23; 'performing': 0.23; 'import': 0.24; 'script': 0.25; 'header:User- Agent:1': 0.26; 'header:X-Complaints-To:1': 0.26; 'function': 0.28; 'idea': 0.28; 'skip:u 20': 0.28; 'cat': 0.29; 'perl': 0.29; 'skip:_ 10': 0.32; 'class': 0.33; 'file': 0.34; 'false': 0.35; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'received:org': 0.37; 'things': 0.38; 'subject:from': 0.39; 'to:addr:python.org': 0.40; 'received:de': 0.40; 'below:': 0.71
X-Injected-Via-Gmane: http://gmane.org/
X-Gmane-NNTP-Posting-Host: p57bd9af8.dip0.t-ipconnect.de
User-Agent: KNode/4.13.3
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref: csiph.com comp.lang.python:104621

Fillmore wrote:

> On 03/11/2016 07:13 AM, Wolfgang Maier wrote:
>> One lesson for Perl regex users is that in Python many things can be
>> solved without regexes. How about defining:
>>
>> printable = {chr(n) for n in range(32, 127)}
>>
>> then using:
>>
>> if (set(my_string) - set(printable)):
>>      break
> 
> seems computationally heavy. I have a file with about 70k lines, of which
> only 20 contain "funny" chars.
> 
> ANy idea on how I can create a script that compares Perl speed vs. Python
> speed in performing the cleaning operation?

Try 

for line in ...:
    if has_nonprint(line):
        continue
    ...

with the has_nonprint() function as defined below:

$ cat isprint.py
import sys
import unicodedata


class Lookup(dict):
    def __missing__(self, n):
        c = chr(n)
        cat = unicodedata.category(c)
        if cat in {'Cs', 'Cn', 'Zl', 'Cc', 'Zp'}:
            self[n] = c
            return c
        else:
            self[n] = None
            return None


lookup = Lookup()
lookup[10] = None # allow newline

def has_nonprint(s):
    return bool(s.translate(lookup))

$ python3 -i isprint.py
>>> has_nonprint("foo")
False
>>> has_nonprint("foo\n")
False
>>> has_nonprint("foo\t")
True
>>> has_nonprint("\0foo")
True