Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #94243 > unrolled thread

linux os.rename() not an actual rename?

Back to article view | Back to comp.lang.python

Contents

  linux os.rename() not an actual rename? "Jason H" <jhihn@gmx.com> - 2015-07-20 20:02 +0200
    Re: linux os.rename() not an actual rename? Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2015-07-20 18:43 +0000
    Re: linux os.rename() not an actual rename? Marko Rauhamaa <marko@pacujo.net> - 2015-07-20 21:50 +0300
      Re: linux os.rename() not an actual rename? Christian Heimes <christian@python.org> - 2015-07-20 20:58 +0200
      Re: linux os.rename() not an actual rename? "Jason H" <jhihn@gmx.com> - 2015-07-20 21:40 +0200

#94243 — linux os.rename() not an actual rename?

I have a server process that looks (watches via inotify)  for files to be moved (renamed) into a particular directory from elsewhere on the same filesystem. We do this because it is an atomic operation, and our server process can see the modify events of the file being written before it is closed. The rename functions as a 'completed' event.  We have a python script that attempts to perform this behavior - to os.rename() a file into the watched directory after it is done being written. However unlike other tools, we don't see a proper 'rename' event. Instead we just see a 'changed' event. I've changed the implementation of the script to os.system('mv ...') and we get the expected 'rename' event. 

Is this known issue? Should I be seeing a proper rename event? The only mention in the docs about the rename behavior is that it is atomic, as required by POSIX. 

[toc] | [next] | [standalone]

#94244

On 2015-07-20, Jason H <jhihn@gmx.com> wrote:
> I have a server process that looks (watches via inotify)  for files
> to be moved (renamed) into a particular directory from elsewhere on
> the same filesystem. We do this because it is an atomic operation,
> and our server process can see the modify events of the file being
> written before it is closed. The rename functions as a 'completed'
> event.  We have a python script that attempts to perform this
> behavior - to os.rename() a file into the watched directory after it
> is done being written. However unlike other tools, we don't see a
> proper 'rename' event. Instead we just see a 'changed' event. I've
> changed the implementation of the script to os.system('mv ...') and
> we get the expected 'rename' event. 
>
> Is this known issue? Should I be seeing a proper rename event? The
> only mention in the docs about the rename behavior is that it is
> atomic, as required by POSIX. 

os.rename() should just be calling the operating system rename(2)
function. I think you must be doing something wrong.

[toc] | [prev] | [next] | [standalone]

#94245

"Jason H" <jhihn@gmx.com>:

> I have a server process that looks (watches via inotify) for files to
> be moved (renamed) into a particular directory from elsewhere on the
> same filesystem. We do this because it is an atomic operation, and our
> server process can see the modify events of the file being written
> before it is closed. The rename functions as a 'completed' event. We
> have a python script that attempts to perform this behavior - to
> os.rename() a file into the watched directory after it is done being
> written. However unlike other tools, we don't see a proper 'rename'
> event. Instead we just see a 'changed' event. I've changed the
> implementation of the script to os.system('mv ...') and we get the
> expected 'rename' event.

Don't know about inotify(). However, strace reveals that python3's
os.rename() performs a regular rename(2) system call.


Marko

[toc] | [prev] | [next] | [standalone]

#94246

On 2015-07-20 20:50, Marko Rauhamaa wrote:
> "Jason H" <jhihn@gmx.com>:
> 
>> I have a server process that looks (watches via inotify) for files to
>> be moved (renamed) into a particular directory from elsewhere on the
>> same filesystem. We do this because it is an atomic operation, and our
>> server process can see the modify events of the file being written
>> before it is closed. The rename functions as a 'completed' event. We
>> have a python script that attempts to perform this behavior - to
>> os.rename() a file into the watched directory after it is done being
>> written. However unlike other tools, we don't see a proper 'rename'
>> event. Instead we just see a 'changed' event. I've changed the
>> implementation of the script to os.system('mv ...') and we get the
>> expected 'rename' event.
> 
> Don't know about inotify(). However, strace reveals that python3's
> os.rename() performs a regular rename(2) system call.

So does Python 2.7:

$ touch test
$ strace -e trace=file -- python -c 'import os; os.rename("test", "test2")'
execve("/bin/python", ["python", "-c", "import os; os.rename(\"test\",
\"te"...], [/* 76 vars */]) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
...
rename("test", "test2") = 0
+++ exited with 0 +++

[toc] | [prev] | [next] | [standalone]

#94247

> From: "Christian Heimes" <christian@python.org>
> On 2015-07-20 20:50, Marko Rauhamaa wrote:
> > "Jason H" <jhihn@gmx.com>:
> > 
> >> I have a server process that looks (watches via inotify) for files to
> >> be moved (renamed) into a particular directory from elsewhere on the
> >> same filesystem. We do this because it is an atomic operation, and our
> >> server process can see the modify events of the file being written
> >> before it is closed. The rename functions as a 'completed' event. We
> >> have a python script that attempts to perform this behavior - to
> >> os.rename() a file into the watched directory after it is done being
> >> written. However unlike other tools, we don't see a proper 'rename'
> >> event. Instead we just see a 'changed' event. I've changed the
> >> implementation of the script to os.system('mv ...') and we get the
> >> expected 'rename' event.
> > 
> > Don't know about inotify(). However, strace reveals that python3's
> > os.rename() performs a regular rename(2) system call.
> 
> So does Python 2.7:
> 
> $ touch test
> $ strace -e trace=file -- python -c 'import os; os.rename("test", "test2")'
> execve("/bin/python", ["python", "-c", "import os; os.rename(\"test\",
> \"te"...], [/* 76 vars */]) = 0
> access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
> open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
> ...
> rename("test", "test2") = 0
> +++ exited with 0 +++

Hrm, provably, you're right. But I was seeing 'rename', then two 'changed' events on the dest name, but the last thing the process did was rename before it exited. 

I'll look into it some more now that I know python should be using the OS implementation of rename. 

Thanks everyone.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web