Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #35016

Re: Iterating over files of a huge directory

Show all headers | View raw

On 12/17/2012 10:28 AM, Gilles Lenfant wrote:
> Hi,
>
> I have googled but did not find an efficient solution to my problem.
> My customer provides a directory with a huuuuge list of files (flat,
> potentially 100000+) and I cannot reasonably use
> os.listdir(this_path) unless creating a big memory footprint.

Is is really big enough to be a real problem? See below.

> So I'm looking for an iterator that yields the file names of a
> directory and does not make a giant list of what's in.
>
> i.e :
>
> for filename in enumerate_files(some_directory): # My cooking...

See http://bugs.python.org/issue11406
As I said there, I personally think (and still do) that listdir should 
have been changed in 3.0 to return an iterator rather than a list. 
Developers who count more than me disagree on the basis that no 
application has the millions of directory entries needed to make space a 
real issue. They also claim that time is a wash either way.

As for space, 100000 entries x 100 bytes/entry (generous guess at 
average) = 10,000,000 bytes, no big deal with gigabyte memories. So the 
logic goes. A smaller example from my machine with 3.3.

from sys import getsizeof

def seqsize(seq):
     "Get size of flat sequence and contents"
     return sum((getsizeof(item) for item in seq), getsizeof(seq))

import os
d = os.listdir()
print(seqsize([1,2,3]), len(d), seqsize(d))
#
172 45 3128

The size per entry is relatively short because the two-level directory 
prefix for each path is only about 15 bytes. By using 3.3 rather than 
3.0-3.2, the all-ascii-char unicode paths only take 1 byte per char 
rather than 2 or 4.

If you disagree with the responses on the issue, after reading them, 
post one yourself with real numbers.

-- 
Terry Jan Reedy

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Iterating over files of a huge directory Gilles Lenfant <gilles.lenfant@gmail.com> - 2012-12-17 07:28 -0800
  Re: Iterating over files of a huge directory Chris Angelico <rosuav@gmail.com> - 2012-12-18 02:41 +1100
    Re: Iterating over files of a huge directory Paul Rudin <paul.nospam@rudin.co.uk> - 2012-12-17 17:27 +0000
      Re: Iterating over files of a huge directory MRAB <python@mrabarnett.plus.com> - 2012-12-17 18:29 +0000
      Re: Iterating over files of a huge directory Chris Angelico <rosuav@gmail.com> - 2012-12-18 08:10 +1100
  Re: Iterating over files of a huge directory Tim Golden <mail@timgolden.me.uk> - 2012-12-17 15:48 +0000
  Re: Iterating over files of a huge directory Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-12-17 15:52 +0000
    Re: Iterating over files of a huge directory Gilles Lenfant <gilles.lenfant@gmail.com> - 2012-12-17 08:06 -0800
    Re: Iterating over files of a huge directory Gilles Lenfant <gilles.lenfant@gmail.com> - 2012-12-17 08:06 -0800
  Re: Iterating over files of a huge directory marduk <marduk@python.net> - 2012-12-17 10:50 -0500
  Re: Re: Iterating over files of a huge directory Evan Driscoll <driscoll@cs.wisc.edu> - 2012-12-17 12:40 -0600
  Re: Re: Iterating over files of a huge directory Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-12-17 19:50 +0000
  Re: Iterating over files of a huge directory Evan Driscoll <driscoll@cs.wisc.edu> - 2012-12-17 14:09 -0600
  Re: Iterating over files of a huge directory Terry Reedy <tjreedy@udel.edu> - 2012-12-17 16:27 -0500

csiph-web