Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #51475 > unrolled thread

timing issue: shutil.rmtree and os.makedirs

Back to article view | Back to comp.lang.python

Contents

  timing issue: shutil.rmtree and os.makedirs Tim <jtim.arnold@gmail.com> - 2013-07-29 12:16 -0700
    Re: timing issue: shutil.rmtree and os.makedirs Chris Angelico <rosuav@gmail.com> - 2013-07-30 00:52 +0100
      Re: timing issue: shutil.rmtree and os.makedirs Tim <jtim.arnold@gmail.com> - 2013-07-30 06:10 -0700
        Re: timing issue: shutil.rmtree and os.makedirs Chris Angelico <rosuav@gmail.com> - 2013-07-30 14:27 +0100
          Re: timing issue: shutil.rmtree and os.makedirs Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-07-30 14:07 +0000
            Re: timing issue: shutil.rmtree and os.makedirs Chris Angelico <rosuav@gmail.com> - 2013-07-30 15:47 +0100
          Re: timing issue: shutil.rmtree and os.makedirs Tim <jtim.arnold@gmail.com> - 2013-07-30 08:37 -0700
            Re: timing issue: shutil.rmtree and os.makedirs Chris Angelico <rosuav@gmail.com> - 2013-07-30 17:03 +0100
        Re: timing issue: shutil.rmtree and os.makedirs Göktuğ Kayaalp <goktug.kayaalp@gmail.com> - 2013-07-30 20:09 +0300

#51475 — timing issue: shutil.rmtree and os.makedirs

I have the following function (Python2.7 on FreeBSD) that results in an OSError.

My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:
 
def make_clean_dir(directory):
    if os.path.exists(directory):
        if os.path.isdir(directory):
            shutil.rmtree(directory)
        else:
            os.unlink(directory)
    os.makedirs(directory)

The last bit of the traceback is:
File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
    os.makedirs(directory)
  File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'

The directory 'testing_html' existed when I executed the function;
So I suppose the directory wasn't finished being removed by the time os.makedirs was invoked. How can avoid this? (A context manager maybe?).

thanks,
--Tim

[toc] | [next] | [standalone]

#51527

On Mon, Jul 29, 2013 at 8:16 PM, Tim <jtim.arnold@gmail.com> wrote:
> My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:
>
> def make_clean_dir(directory):
>     if os.path.exists(directory):
>         if os.path.isdir(directory):
>             shutil.rmtree(directory)
>         else:
>             os.unlink(directory)
>     os.makedirs(directory)
>
> The last bit of the traceback is:
> File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
>     os.makedirs(directory)
>   File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
>     mkdir(name, mode)
> OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'
>
> The directory 'testing_html' existed when I executed the function;

First thing I'd check is: Did rmtree succeed? Try removing the
makedirs and test it again; then, when your process has completely
finished, see if the directory is there. If it is, the problem is in
rmtree - for instance:

* You might not have permission to remove everything
* There might be a messed-up object in the file system
* If the directory is a remote share mount point, the other end might
have lied about the removal
* Something might have been created inside the directory during the removal
* Myriad other possibilities

As I understand rmtree's docs, any errors *that it detects* will be
raised as exceptions (since you haven't told it to suppress or handle
them), but possibly there's an error that it isn't able to detect.
Worth a test, anyhow.

ChrisA

[toc] | [prev] | [next] | [standalone]

#51555

On Monday, July 29, 2013 7:52:36 PM UTC-4, Chris Angelico wrote:
> On Mon, Jul 29, 2013 at 8:16 PM, Tim wrote:
> > My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:

> > def make_clean_dir(directory):
> >     if os.path.exists(directory):
> >         if os.path.isdir(directory):
> >             shutil.rmtree(directory)
> >         else:
> >             os.unlink(directory)
> >     os.makedirs(directory)
> >
> > The last bit of the traceback is:
> > File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
> >     os.makedirs(directory)
> >   File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
> >     mkdir(name, mode)
> > OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'
> >
> > The directory 'testing_html' existed when I executed the function;
>  
> First thing I'd check is: Did rmtree succeed? Try removing the
> makedirs and test it again; then, when your process has completely
> finished, see if the directory is there. If it is, the problem is in
> rmtree - for instance:

> * You might not have permission to remove everything
> * There might be a messed-up object in the file system
> * If the directory is a remote share mount point, the other end might
> have lied about the removal
> * Something might have been created inside the directory during the removal
> * Myriad other possibilities
> As I understand rmtree's docs, any errors *that it detects* will be
> raised as exceptions (since you haven't told it to suppress or handle
> them), but possibly there's an error that it isn't able to detect.
> Worth a test, anyhow.
> 
> ChrisA

Thanks Chris, but the directory was actually removed on the first run in spite of the traceback; when I run it a second time (immediately after the first time), it runs fine. That's why I thought it was a timing issue. I thought about just putting a sleep in there, but that made me feel dirty. 

hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote.

But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.

thanks,
--Tim

[toc] | [prev] | [next] | [standalone]

#51556

On Tue, Jul 30, 2013 at 2:10 PM, Tim <jtim.arnold@gmail.com> wrote:
> hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote.
>
> But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.

Hmm. What system is used for the file system sharing? I know quite a
few of them lie about whether something's been completely done or not.

Can you use inotify to tell you when the directory's been deleted?
Seems stupid though.

Worst case, all you need is a quick loop at the bottom, eg:

for delay in 100,300,600,1000,3000,5000,10000:
  if not os.path.exists(directory): break
  sleep(delay)

That'll sleep a maximum of 20 seconds, tune as required. Of course, if
there's a way to tune the FS to guarantee that the removal blocks
correctly, that would be way better than sleep()!

ChrisA

[toc] | [prev] | [next] | [standalone]

#51560

On Tue, 30 Jul 2013 14:27:10 +0100, Chris Angelico wrote:

> for delay in 100,300,600,1000,3000,5000,10000:
>   if not os.path.exists(directory): break
>   sleep(delay)
> 
> That'll sleep a maximum of 20 seconds, tune as required.

Actually, that will sleep a maximum of 5.55 hours, and a minimum of 1.7 
minutes (assuming the directory doesn't get deleted instantaneously).

time.sleep() takes an argument in seconds.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#51564

On Tue, Jul 30, 2013 at 3:07 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Tue, 30 Jul 2013 14:27:10 +0100, Chris Angelico wrote:
>
>> for delay in 100,300,600,1000,3000,5000,10000:
>>   if not os.path.exists(directory): break
>>   sleep(delay)
>>
>> That'll sleep a maximum of 20 seconds, tune as required.
>
> Actually, that will sleep a maximum of 5.55 hours, and a minimum of 1.7
> minutes (assuming the directory doesn't get deleted instantaneously).
>
> time.sleep() takes an argument in seconds.

LOL! Whoops. That's what I get for not checking my docs. This is why
we have public responses, my errors can be caught by someone else.

ChrisA

[toc] | [prev] | [next] | [standalone]

#51567

On Tuesday, July 30, 2013 9:27:10 AM UTC-4, Chris Angelico wrote:
> On Tue, Jul 30, 2013 at 2:10 PM, Tim wrote:
> > hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote.
> >
> > But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.
> 
> Hmm. What system is used for the file system sharing? I know quite a
> few of them lie about whether something's been completely done or not.  
> Can you use inotify to tell you when the directory's been deleted? 
> Seems stupid though.   
> Worst case, all you need is a quick loop at the bottom, eg:
> 
<<snip, thanks for the code and correction>>  
> ChrisA

Argg, this isn't the first time I've had troubles with the file system. This is FreeBSD and NFS. I will code up a progressive delay as you mentioned (with Steve's correction).

thanks much!
--Tim

[toc] | [prev] | [next] | [standalone]

#51575

On Tue, Jul 30, 2013 at 4:37 PM, Tim <jtim.arnold@gmail.com> wrote:
> Argg, this isn't the first time I've had troubles with the file system. This is FreeBSD and NFS. I will code up a progressive delay as you mentioned (with Steve's correction).

I've used several different networked file systems, including
NetBIOS/NetBEUI/SMB/Samba/etc, sshfs/cifs, and nfs. Not one of them
"feels" as clean as I'd like, though sshfs comes closest. There always
seem to be hacks around.

ChrisA

[toc] | [prev] | [next] | [standalone]

#51585

[Multipart message — attachments visible in raw view] — view raw

On Jul 30, 2013 3:29 PM, "Chris Angelico" <rosuav@gmail.com> wrote:
>
> On Tue, Jul 30, 2013 at 2:10 PM, Tim <jtim.arnold@gmail.com> wrote:
> > hmm, now that you mention it, this is executing on a remote box with
access to the same file system my local calling program is on. That is,
there is a local call to an intermediate script that connects to a socket
on the remote where the above program actually runs, but the file system is
the same place for both local and remote.
> >
> > But even so, since the script that does the rmtree and mkdir is running
on the same machine (even though it's remote), I would think the mkdir
couldn't execute until the rmtree was completely finished.
>
> Hmm. What system is used for the file system sharing? I know quite a
> few of them lie about whether something's been completely done or not.
>
> Can you use inotify to tell you when the directory's been deleted?
> Seems stupid though.

Inotify is a linux thing, but there is kqueue for Free?BSD.  OP can run the
deletion procedure, wait for a NOTE_DELETE event, which would block, and
create the fresh directory afterwards.  It may require some C hacking
though, in case Python lacks a kqueue wrapper.  I think that this kind of
approach would be more sound than a check-wait-loop approach.

(I would elaborate more with pointers to appropriate documentation, but I'm
on a silly tablet, please excuse me for that.)

-gk

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web