Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #34980
| References | <c2b15410-12e0-4645-a77f-9944bfd674a8@googlegroups.com> |
|---|---|
| Date | 2012-12-18 02:41 +1100 |
| Subject | Re: Iterating over files of a huge directory |
| From | Chris Angelico <rosuav@gmail.com> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.961.1355758912.29569.python-list@python.org> (permalink) |
On Tue, Dec 18, 2012 at 2:28 AM, Gilles Lenfant <gilles.lenfant@gmail.com> wrote: > Hi, > > I have googled but did not find an efficient solution to my problem. My customer provides a directory with a huuuuge list of files (flat, potentially 100000+) and I cannot reasonably use os.listdir(this_path) unless creating a big memory footprint. > > So I'm looking for an iterator that yields the file names of a directory and does not make a giant list of what's in. Sounds like you want os.walk. But... a hundred thousand files? I know the Zen of Python says that flat is better than nested, but surely there's some kind of directory structure that would make this marginally manageable? http://docs.python.org/3.3/library/os.html#os.walk ChrisA
Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Iterating over files of a huge directory Gilles Lenfant <gilles.lenfant@gmail.com> - 2012-12-17 07:28 -0800
Re: Iterating over files of a huge directory Chris Angelico <rosuav@gmail.com> - 2012-12-18 02:41 +1100
Re: Iterating over files of a huge directory Paul Rudin <paul.nospam@rudin.co.uk> - 2012-12-17 17:27 +0000
Re: Iterating over files of a huge directory MRAB <python@mrabarnett.plus.com> - 2012-12-17 18:29 +0000
Re: Iterating over files of a huge directory Chris Angelico <rosuav@gmail.com> - 2012-12-18 08:10 +1100
Re: Iterating over files of a huge directory Tim Golden <mail@timgolden.me.uk> - 2012-12-17 15:48 +0000
Re: Iterating over files of a huge directory Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-12-17 15:52 +0000
Re: Iterating over files of a huge directory Gilles Lenfant <gilles.lenfant@gmail.com> - 2012-12-17 08:06 -0800
Re: Iterating over files of a huge directory Gilles Lenfant <gilles.lenfant@gmail.com> - 2012-12-17 08:06 -0800
Re: Iterating over files of a huge directory marduk <marduk@python.net> - 2012-12-17 10:50 -0500
Re: Re: Iterating over files of a huge directory Evan Driscoll <driscoll@cs.wisc.edu> - 2012-12-17 12:40 -0600
Re: Re: Iterating over files of a huge directory Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-12-17 19:50 +0000
Re: Iterating over files of a huge directory Evan Driscoll <driscoll@cs.wisc.edu> - 2012-12-17 14:09 -0600
Re: Iterating over files of a huge directory Terry Reedy <tjreedy@udel.edu> - 2012-12-17 16:27 -0500
csiph-web