Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #92225

Re: Find in ipython3

Show key headers only | View raw

Cecil Westerhof wrote:

> On Saturday  6 Jun 2015 13:07 CEST, Laura Creighton wrote:
> 
>> The !find version is C code optimised to do one thing, find files in
>> your directory structure, which happens to be what you want to do.
>> General regular expression matching is harder.
>>
>> Carl Friedrich Bolz investigated regular expression algorithms and
>> their implementation to see if this is the sort of task that a JIT
>> can improve. He blogged about it in 2 posts (part1 and part2). There
>> are benchmarks for part2. Benchmarks in part2.
>>
>> see:
>> http://morepypy.blogspot.se/2010/05/efficient-and-elegant-regular.html
>> http://morepypy.blogspot.se/2010/06/jit-for-regular-expression-matching.html
>>
>> You may get faster results if you use Matthew Barnett's replacement
>> for re here: https://pypi.python.org/pypi/regex
>>
>> You will get faster results if you build your IPython shell to use
>> PyPy, but I would still be very surprised if it beat the C program
>> find.
> 
> I have to look into that. But I prefer to write a version that can be
> used by ‘everyone’.
> 
> It is of-course not a very big program. The difference is significant,
> but I do not use find that much. And if it is significant I still can
> use the shell version.
> 
> There is no gain to get in standard Python? By switching from fnmatch
> to re I got almost a speed gain of two. So I was wondering if I could
> do more.

Just wait for Python 3.5. The switch from os.listdir() to the (new) 
os.scandir() in the implementation of os.walk() is likely to improve the 
situation: 

$ cat findfiles.py                    
import fnmatch
import os
import re
import subprocess
import time


def find_re(root, pattern, ignore_case=False):
    match = re.compile(
        fnmatch.translate(pattern),
        re.IGNORECASE if ignore_case else 0).match

    results = []
    for path, _folders, files in os.walk(root):
        for filename in files:
            if match(filename):
                results.append(os.path.join(path, filename))
    return results


def find_sp(
        root, pattern, ignore_case=False,
        encoding="utf-8", errors="surrogateescape"):

    name_opt = "-iname" if ignore_case else "-name"
    matches = subprocess.Popen(
        ["find", root, name_opt, pattern, "-print0"], stdout=subprocess.PIPE
    ).communicate()[0].decode(encoding, errors=errors).split("\0")
    assert len(matches[-1]) == 0
    del matches[-1]
    return matches


def measure(f, *args):
    start = time.time()
    try:
        return f(*args)
    finally:
        end = time.time()
        print("{}{}".format(f.__name__, args), end - start)


def main():
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument("root")
    parser.add_argument("pattern")
    parser.add_argument("-i", "--ignore-case", action="store_true")
    args = parser.parse_args()

    a = measure(find_re, args.root, args.pattern, args.ignore_case)
    b = measure(find_sp, args.root, args.pattern, args.ignore_case)
    measure(find_re, args.root, args.pattern, args.ignore_case)
    assert sorted(a) == sorted(b)
    print(len(a), "matches")


if __name__ == "__main__":
    main()

$ python3.4 findfiles.py . '*a*.PY' -i
find_re('.', '*a*.PY', True) 0.14614605903625488
find_sp('.', '*a*.PY', True) 0.043445587158203125
find_re('.', '*a*.PY', True) 0.16485309600830078
1454 matches

$ python3.5 findfiles.py . '*a*.PY' -i
find_re('.', '*a*.PY', True) 0.07263660430908203
find_sp('.', '*a*.PY', True) 0.04418468475341797
find_re('.', '*a*.PY', True) 0.07320952415466309
1454 matches

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Find in ipython3 Cecil Westerhof <Cecil@decebal.nl> - 2015-06-02 18:13 +0200
  Re: Find in ipython3 Cameron Simpson <cs@zip.com.au> - 2015-06-04 12:54 +1000
    Re: Find in ipython3 Cecil Westerhof <Cecil@decebal.nl> - 2015-06-04 07:09 +0200
      Re: Find in ipython3 Cameron Simpson <cs@zip.com.au> - 2015-06-04 15:43 +1000
      Re: Find in ipython3 Grant Edwards <invalid@invalid.invalid> - 2015-06-04 14:27 +0000
        Re: Find in ipython3 Cecil Westerhof <Cecil@decebal.nl> - 2015-06-04 17:12 +0200
          Re: Find in ipython3 Michael Torrie <torriem@gmail.com> - 2015-06-04 13:11 -0600
  Re: Find in ipython3 Michael Torrie <torriem@gmail.com> - 2015-06-04 13:09 -0600
  Re: Find in ipython3 Tim Chase <python.list@tim.thechases.com> - 2015-06-04 14:17 -0500
  Re: Find in ipython3 random832@fastmail.us - 2015-06-04 16:13 -0400
    Re: Find in ipython3 Cecil Westerhof <Cecil@decebal.nl> - 2015-06-05 09:17 +0200
      Re: Find in ipython3 Cecil Westerhof <Cecil@decebal.nl> - 2015-06-06 11:57 +0200
        Re: Find in ipython3 Laura Creighton <lac@openend.se> - 2015-06-06 13:07 +0200
          Re: Find in ipython3 Cecil Westerhof <Cecil@decebal.nl> - 2015-06-07 08:20 +0200
            Re: Find in ipython3 Cameron Simpson <cs@zip.com.au> - 2015-06-07 17:38 +1000
            Re: Find in ipython3 Laura Creighton <lac@openend.se> - 2015-06-07 11:33 +0200
              Re: Find in ipython3 Steven D'Aprano <steve@pearwood.info> - 2015-06-07 23:16 +1000
            Re: Find in ipython3 Peter Otten <__peter__@web.de> - 2015-06-07 12:27 +0200
            Re: Find in ipython3 Laura Creighton <lac@openend.se> - 2015-06-07 15:01 +0200
            Re: Find in ipython3 Chris Angelico <rosuav@gmail.com> - 2015-06-07 22:13 +1000

csiph-web