Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!news.unit0.net!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.002 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; '"""': 0.05; 'filename': 0.07; 'filenames': 0.07; 'works.': 0.07; 'api': 0.09; 'docs.': 0.09; 'iterate': 0.09; 'subject:files': 0.09; 'sure)': 0.09; 'url:github': 0.09; 'library': 0.15; 'gilles': 0.16; 'googled': 0.16; 'i.e': 0.16; 'iterates': 0.16; 'iterator': 0.16; 'reasonably': 0.16; 'to:name:python list': 0.16; 'wrote:': 0.17; 'directory.': 0.17; 'memory': 0.18; 'discussion': 0.20; '(on': 0.22; 'to:2**1': 0.23; 'external': 0.24; 'header:In-Reply-To:1': 0.25; 'creating': 0.26; 'in.': 0.27; 'skip:e 30': 0.27; 'message- id:@mail.gmail.com': 0.27; 'efficiently': 0.29; 'giant': 0.29; 'yields': 0.29; "i'm": 0.29; 'feedback': 0.30; 'code': 0.31; 'december': 0.32; 'problem.': 0.32; 'you?': 0.32; 'file': 0.32; 'asked': 0.33; 'says': 0.33; 'to:addr:python-list': 0.33; 'version': 0.34; 'received:google.com': 0.34; 'list': 0.35; 'adds': 0.35; 'faster': 0.35; 'returning': 0.35; 'so,': 0.35; 'received:209.85': 0.35; 'there': 0.35; 'list.': 0.35; 'but': 0.36; 'author': 0.37; 'does': 0.37; 'received:209': 0.37; 'far': 0.37; 'subject:: ': 0.38; 'files': 0.38; 'skip:o 20': 0.38; 'some': 0.38; 'instead': 0.39; 'to:addr:python.org': 0.39; 'header:Received:5': 0.40; 'easy': 0.60; 'customer': 0.61; 'more': 0.63; 'potentially': 0.66; 'oscar': 0.84; 'python-dev': 0.84; 'subject:over': 0.84; 'imagine': 0.96 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=QP+oGbAfP67OfC7iZf2hoM1WtrDr78d9yhWcHstX1S0=; b=Brmo29L5t6Eb4WlMc75aBTPi1EAD4LuK8d54w2I3UijkDPCaWnbZ08tggBJIefyv2H f/LfRPpYywhsW9GlRRvufpdnmnZQ4/uSR1+E5AG5LkDwnW8V4VEVOzrt/6rjoNKGs1rY c5UGwIyEa5qpnU0mvqcaFHCZ6S/rKQf5OXZPsDF9YJfFAezMqQPcMfhbWpTeYTa2I0RL js8MUnhV1atBVCAXLe76m/+uwCn5FYrGIQ0wKKreP6+KlmyMGNss5XZGZsqmU+k4GQhq 9VgVcCAWoG20aPHJVrwQCbh9Dhgz+/zCaX249GEDNkRy13i8W5jZAaaZF2uviHQj49C4 eqwA== MIME-Version: 1.0 In-Reply-To: References: Date: Mon, 17 Dec 2012 15:52:19 +0000 Subject: Re: Iterating over files of a huge directory From: Oscar Benjamin To: Gilles Lenfant , Python List Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 34 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1355759541 news.xs4all.nl 6931 [2001:888:2000:d::a6]:34888 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:34983 On 17 December 2012 15:28, Gilles Lenfant wrote: > I have googled but did not find an efficient solution to my problem. My customer provides a directory with a huuuuge list of files (flat, potentially 100000+) and I cannot reasonably use os.listdir(this_path) unless creating a big memory footprint. > > So I'm looking for an iterator that yields the file names of a directory and does not make a giant list of what's in. > > i.e : > > for filename in enumerate_files(some_directory): > # My cooking... In the last couple of months there has been a lot of discussion (on python-list or python-dev - not sure) about creating a library to more efficiently iterate over the files in a directory. The result so far is this library on github: https://github.com/benhoyt/betterwalk It says there that """ Somewhat relatedly, many people have also asked for a version of os.listdir() that yields filenames as it iterates instead of returning them as one big list. So as well as a faster walk(), BetterWalk adds iterdir_stat() and iterdir(). They're pretty easy to use, but see below for the full API docs. """ Does that code work for you? If so, I imagine the author would be interested to get some feedback on how well it works. Alternatively, perhaps consider calling an external utility. Oscar