Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #108636 > unrolled thread

How to get a directory list sorted by date?

Back to article view | Back to comp.lang.python

Contents

  How to get a directory list sorted by date? cl@isbd.net - 2016-05-15 09:47 +0100
    Re: How to get a directory list sorted by date? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-05-15 20:48 +1000
    Re: How to get a directory list sorted by date? Michael Selik <michael.selik@gmail.com> - 2016-05-15 15:50 +0000
    Re: How to get a directory list sorted by date? Grant Edwards <grant.b.edwards@gmail.com> - 2016-05-15 16:00 +0000
    Re: How to get a directory list sorted by date? Tim Chase <python.list@tim.thechases.com> - 2016-05-15 13:12 -0500
      Re: How to get a directory list sorted by date? cl@isbd.net - 2016-05-15 21:23 +0100
    Re: How to get a directory list sorted by date? Chris Angelico <rosuav@gmail.com> - 2016-05-16 07:52 +1000
    Re: How to get a directory list sorted by date? Random832 <random832@fastmail.com> - 2016-05-16 12:40 -0400

#108636 — How to get a directory list sorted by date?

I have a little Python program I wrote myself which copies images from
a camera (well, any mounted directory) to my picture archive.  The
picture archive is simply a directory hierarchy of dates with years at
the top, then months, then days.

My Python program simply extracts the date from the image (put there
by the camera) and copies the image to the appropriate place in the
picture archive.

There is one small bit of speeding up done by my program, it checks if
the file is already in the archive and doesn't copy if it's already
there.  I don't generally clear pictures off the camera memory cards
so it's often the case that most of the pcitures are already in the
archive and all I actually want to do is copy the last couple of weeks
of new pictures.

As camera memory card sizes get bigger (and images get more detailed)
this is beginning to take rather a long time.

So, to the question, how can I get a list of the files on the memory
card (i.e. files in a directory) in date/time order, latest first.  I
can then copy them until the first one I find which is already in the
archive.  The file date may not *exactly* match the date/time in the
image file but it will produce the right order which is what I need.

What I want is a list in the order produced by:-
    ls --sort=time

I suppose I could use a system call but that seems a little inelegant.

-- 
Chris Green
·

[toc] | [next] | [standalone]

#108637

On Sunday 15 May 2016 18:47, cl@isbd.net wrote:

> What I want is a list in the order produced by:-
>     ls --sort=time

I'm not sure if this is the best way to do it, but what I would do is sort the 
directory list by the file's time metadata, which you can access using os.stat.

To sort the file names, I use a key function which takes the filename, joins it 
to the parent directory, grabs its stat data, and extracts the time field. This 
is done only once per file name.


import os
import functools

def by_date(where, fname):
    return os.stat(os.path.join(where, fname)).st_mtime

location = '/tmp/'
files = sorted(os.listdir(location), key=functools.partial(by_date, location))


The mysterious call to functools.partial creates a new function which is 
exactly the same as by_date except the "where" argument is already filled in. 
If functools.partial scares you, you can replace it with:

key=lambda fname: by_date(location, fname)

and if lambda scares you, you can create a top-level function:

def keyfunc(fname):
    return by_date(location, fname)



Also, remember that most operating systems provide (at least) three different 
times. I'm not sure which one you want, but if I had to guess, I would probably 
guess mtime. If I remember correctly:

atime is usually the last access time;
mtime is usually the last modification time;
ctime is the creation time, but only on OS-X;
ctime on Linux and Windows is, um, something else...

To use one of the other two, change "st_mtime" to "st_ctime" or "st_atime".


-- 
Steve

[toc] | [prev] | [next] | [standalone]

#108640

On Sun, May 15, 2016, 10:37 AM Grant Edwards <grant.b.edwards@gmail.com>
wrote:

> On 2016-05-15, Tim Chase <python.list@tim.thechases.com> wrote:
> > On 2016-05-15 11:46, Peter Otten wrote:
> >> def sorted_dir(folder):
> >>     def getmtime(name):
> >>         path = os.path.join(folder, name)
> >>         return os.path.getmtime(path)
> >>
> >>     return sorted(os.listdir(folder), key=getmtime, reverse=True)
> >>
> >> The same idea will work with pathlib and os.scandir():
> >>
> >> def _getmtime(entry):
> >>     return entry.stat().st_mtime
> >>
> >> def sd_sorted_dir(folder):
> >>     return sorted(os.scandir(folder), key=_getmtime, reverse=True)
> >
> > unless sorted() returns a lazy sorter,
>
> What's a lazy sorter?
>

One that doesn't calculate the next item in the sequence until you ask for
it. It's impossible unless you don't mind an approximation rather than
correct sort.

>

[toc] | [prev] | [next] | [standalone]

#108641

On 2016-05-15, Michael Selik <michael.selik@gmail.com> wrote:
> On Sun, May 15, 2016, 10:37 AM Grant Edwards <grant.b.edwards@gmail.com> wrote:
>> On 2016-05-15, Tim Chase <python.list@tim.thechases.com> wrote:
>>>
>>> unless sorted() returns a lazy sorter,
>>
>> What's a lazy sorter?
>
> One that doesn't calculate the next item in the sequence until you
> ask for it. It's impossible

Why?  As long as the function has access to the entire sequence, it
should be trivial.  Just find the min (or max) item in the sequence,
remove it, then return it.  It's horribly inefficient, but...

> unless you don't mind an approximation rather than correct sort.

I have a feeling that I've missed the joke somewhere.

-- 
Grant

[toc] | [prev] | [next] | [standalone]

#108643

On 2016-05-15 14:36, Grant Edwards wrote:
> On 2016-05-15, Tim Chase <python.list@tim.thechases.com> wrote:
> > unless sorted() returns a lazy sorter,
> 
> What's a lazy sorter?

A hypothetical algorithm that can spool out a sorted sequence without
holding the entire sequence in memory at the same time. 

Though I typed that initial reply a little quickly, as I hadn't known
at the time that scandir()'s results also provide stat() results with
reduced overhead (whereas listdir() just returns the filenames, so
you have to stat() each one).  Thanks, Peter, for highlighting this
feature.

So in light of my newfound knowledge, I humbly retract my snark and
agree that it's better to exploit scandir()'s inclusion of stat()
details without additional calls.

-tkc

[toc] | [prev] | [next] | [standalone]

#108645

Tim Chase <python.list@tim.thechases.com> wrote:
> On 2016-05-15 14:36, Grant Edwards wrote:
> > On 2016-05-15, Tim Chase <python.list@tim.thechases.com> wrote:
> > > unless sorted() returns a lazy sorter,
> > 
> > What's a lazy sorter?
> 
> A hypothetical algorithm that can spool out a sorted sequence without
> holding the entire sequence in memory at the same time. 
> 
> Though I typed that initial reply a little quickly, as I hadn't known
> at the time that scandir()'s results also provide stat() results with
> reduced overhead (whereas listdir() just returns the filenames, so
> you have to stat() each one).  Thanks, Peter, for highlighting this
> feature.
> 
> So in light of my newfound knowledge, I humbly retract my snark and
> agree that it's better to exploit scandir()'s inclusion of stat()
> details without additional calls.
> 
Thanks everyone for the ideas etc.

-- 
Chris Green
·

[toc] | [prev] | [next] | [standalone]

#108646

On Mon, May 16, 2016 at 2:00 AM, Grant Edwards
<grant.b.edwards@gmail.com> wrote:
> On 2016-05-15, Michael Selik <michael.selik@gmail.com> wrote:
>> On Sun, May 15, 2016, 10:37 AM Grant Edwards <grant.b.edwards@gmail.com> wrote:
>>> On 2016-05-15, Tim Chase <python.list@tim.thechases.com> wrote:
>>>>
>>>> unless sorted() returns a lazy sorter,
>>>
>>> What's a lazy sorter?
>>
>> One that doesn't calculate the next item in the sequence until you
>> ask for it. It's impossible
>
> Why?  As long as the function has access to the entire sequence, it
> should be trivial.  Just find the min (or max) item in the sequence,
> remove it, then return it.  It's horribly inefficient, but...
>
>> unless you don't mind an approximation rather than correct sort.
>
> I have a feeling that I've missed the joke somewhere.

Sure, it's not impossible to implement, but generally the point of
lazy generation is that it's more efficient. Going from O(N log N)
with O(N) memory to O(N*N) with O(1) memory is not usually what you
want!

ChrisA

[toc] | [prev] | [next] | [standalone]

#108671

On Sun, May 15, 2016, at 17:52, Chris Angelico wrote:
> On Mon, May 16, 2016 at 2:00 AM, Grant Edwards
> <grant.b.edwards@gmail.com> wrote:
> > On 2016-05-15, Michael Selik <michael.selik@gmail.com> wrote:
> >> On Sun, May 15, 2016, 10:37 AM Grant Edwards <grant.b.edwards@gmail.com> wrote:
> >>> On 2016-05-15, Tim Chase <python.list@tim.thechases.com> wrote:
> >>>>
> >>>> unless sorted() returns a lazy sorter,
> >>>
> >>> What's a lazy sorter?
> >>
> >> One that doesn't calculate the next item in the sequence until you
> >> ask for it. It's impossible
> >
> > Why?  As long as the function has access to the entire sequence, it
> > should be trivial.  Just find the min (or max) item in the sequence,
> > remove it, then return it.  It's horribly inefficient, but...
> >
> >> unless you don't mind an approximation rather than correct sort.
> >
> > I have a feeling that I've missed the joke somewhere.
> 
> Sure, it's not impossible to implement, but generally the point of
> lazy generation is that it's more efficient. Going from O(N log N)
> with O(N) memory to O(N*N) with O(1) memory is not usually what you
> want!

It could be "lazy" in the sense that it doesn't consume the source
iterator or do any sorting at all until you start iterating it. And if
you perform some operation (such as, for example, sorting again by a
different key) that is destructive of the original order it could
dispense with the sorting process (or compose the two key functions, if
the sorting algorithm is meant to be stable)

Or, for example, if you filter the sorted collection, a "lazy"
filter/sorter could reorder the operations so that the unsorted list is
filtered before being sorted, reducing N for the sort operation.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web