Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #25043 > unrolled thread

Re: How to safely maintain a status file

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: How to safely maintain a status file Christian Heimes <lists@cheimes.de> - 2012-07-08 13:53 +0200
    Re: How to safely maintain a status file Plumo <richardbp@gmail.com> - 2012-07-08 22:50 -0700
      Re: How to safely maintain a status file Christian Heimes <lists@cheimes.de> - 2012-07-09 10:17 +0200
      Re: How to safely maintain a status file Laszlo Nagy <gandalf@shopzeus.com> - 2012-07-12 19:46 +0200

#25043 — Re: How to safely maintain a status file

Am 08.07.2012 13:29, schrieb Richard Baron Penman:
> My initial solution was a thread that writes status to a tmp file
> first and then renames:
> 
> open(tmp_file, 'w').write(status)
> os.rename(tmp_file, status_file)

You algorithm may not write and flush all data to disk. You need to do
additional work. You must also store the tmpfile on the same partition
(better: same directory) as the status file

with open(tmp_file, "w") as f:
    f.write(status)
    # flush buffer and write data/metadata to disk
    f.flush()
    os.fsync(f.fileno())

# now rename the file
os.rename(tmp_file, status_file)

# finally flush metadata of directory to disk
dirfd = os.open(os.path.dirname(status_file), os.O_RDONLY)
try:
    os.fsync(dirfd)
finally:
    os.close(dirfd)


> This works well on Linux but Windows raises an error when status_file
> already exists.
> http://docs.python.org/library/os.html#os.rename

Windows doesn't suppport atomic renames if the right side exists.  I
suggest that you implement two code paths:

if os.name == "posix":
    rename = os.rename
else:
    def rename(a, b):
        try:
            os.rename(a, b)
        except OSError, e:
            if e.errno != 183:
                raise
            os.unlink(b)
            os.rename(a, b)

Christian

[toc] | [next] | [standalone]

#25067

> Windows doesn't suppport atomic renames if the right side exists.  I
> suggest that you implement two code paths:
>
> if os.name == "posix":
>     rename = os.rename
> else:
>     def rename(a, b):
>         try:
>             os.rename(a, b)
>         except OSError, e:
>             if e.errno != 183:
>                 raise
>             os.unlink(b)
>             os.rename(a, b)


Problem is if the process is stopped between unlink and rename there
would no status file.

[toc] | [prev] | [next] | [standalone]

#25071

Am 09.07.2012 07:50, schrieb Plumo:
>> Windows doesn't suppport atomic renames if the right side exists.  I
>> suggest that you implement two code paths:
>
> Problem is if the process is stopped between unlink and rename there
> would no status file.

Yeah, you have to suffer all of Windows' design flaws. You could add a
backup status file or use a completely different approach.

[toc] | [prev] | [next] | [standalone]

#25229

>> Windows doesn't suppport atomic renames if the right side exists.  I
>> suggest that you implement two code paths:
>>
>> if os.name == "posix":
>>      rename = os.rename
>> else:
>>      def rename(a, b):
>>          try:
>>              os.rename(a, b)
>>          except OSError, e:
>>              if e.errno != 183:
>>                  raise
>>              os.unlink(b)
>>              os.rename(a, b)
>
> Problem is if the process is stopped between unlink and rename there
> would no status file.
Yes, and actually it does not need to be an abnormal termination. It is 
enough if the OS scheduler puts this process on hold for some time...

But using a lock file, the problem can be solved. However in that case, 
reading a status file can be a blocking operation.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web