Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #80009 > unrolled thread

Re: Saving a file "in the background" -- How?

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Saving a file "in the background" -- How? Chris Angelico <rosuav@gmail.com> - 2014-10-31 23:19 +1100
    Re: Saving a file "in the background" -- How? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-01 01:06 +1100
      Re: Saving a file "in the background" -- How? Chris Angelico <rosuav@gmail.com> - 2014-11-01 01:29 +1100

#80009 — Re: Saving a file "in the background" -- How?

On Fri, Oct 31, 2014 at 11:07 PM, Akira Li <4kir4.1i@gmail.com> wrote:
> where atomic_open() [1] tries to overcome multiple issues with saving
> data reliably:
>
> - write to a temporary file so that the old data is always available
> - rename the file when all new data is written, handle cases such as:
>   * "antivirus opens old file thus preventing me from replacing it"
>
> either the operation succeeds and 'backup' contains new data or it fails
> and 'backup' contains untouched ready-to-restore old data -- nothing in
> between.

Sounds like a lot of hassle, and a lot of things that could be done
wrongly. Personally, if I need that level of reliability and
atomicity, I'd rather push the whole question down to a lower level:
maybe commit something to a git repository and push it to a remote
server, or use a PostgreSQL database, or something of that sort. Let
someone else have the headaches about "what if AV opens the file". Let
someone else worry about how to cope with power failures at arbitrary
points in the code. (Though, to be fair, using git for this doesn't
fully automate failure handling; what it does is allow you to detect
issues on startup, and either roll back ("git checkout -f") or apply
("git commit -a") if it looks okay.)

ChrisA

[toc] | [next] | [standalone]

#80014

Chris Angelico wrote:

> Sounds like a lot of hassle, and a lot of things that could be done
> wrongly. Personally, if I need that level of reliability and
> atomicity, I'd rather push the whole question down to a lower level:
> maybe commit something to a git repository and push it to a remote
> server, 

How is pushing something to a remote server more reliable than writing to a
local drive?

> or use a PostgreSQL database, or something of that sort. Let 
> someone else have the headaches about "what if AV opens the file". Let
> someone else worry about how to cope with power failures at arbitrary
> points in the code. 

That "someone else" is still you though. And "use a PostgreSQL database" is
no solution to the problem "how do I reliably write a 250MB video file?".

Or for that matter, a 2KB text file. Databases are great as databases, but
they aren't files.

> (Though, to be fair, using git for this doesn't 
> fully automate failure handling; what it does is allow you to detect
> issues on startup, and either roll back ("git checkout -f") or apply
> ("git commit -a") if it looks okay.)

I don't see how this sort of manual handling is less trouble than using a
proper, reliable save routine. To start with, what do you mean "detect
issues on startup" -- startup of what?


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#80019

On Sat, Nov 1, 2014 at 1:06 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Chris Angelico wrote:
>
>> Sounds like a lot of hassle, and a lot of things that could be done
>> wrongly. Personally, if I need that level of reliability and
>> atomicity, I'd rather push the whole question down to a lower level:
>> maybe commit something to a git repository and push it to a remote
>> server,
>
> How is pushing something to a remote server more reliable than writing to a
> local drive?

It's not, as far as the program's concerned, but when I need
reliability I usually also need replication/backups.

>> or use a PostgreSQL database, or something of that sort. Let
>> someone else have the headaches about "what if AV opens the file". Let
>> someone else worry about how to cope with power failures at arbitrary
>> points in the code.
>
> That "someone else" is still you though. And "use a PostgreSQL database" is
> no solution to the problem "how do I reliably write a 250MB video file?".

If the commit happens, it's saved, and everything else isn't my
problem. And yes, Postgres isn't ideal for a 250MB file, but then,
neither is any sort of autosave good for that. I'm assuming the
content isn't that big.

> Or for that matter, a 2KB text file. Databases are great as databases, but
> they aren't files.

A database can handle a 2KB text file, no problem. That's how most CMSes work.

>> (Though, to be fair, using git for this doesn't
>> fully automate failure handling; what it does is allow you to detect
>> issues on startup, and either roll back ("git checkout -f") or apply
>> ("git commit -a") if it looks okay.)
>
> I don't see how this sort of manual handling is less trouble than using a
> proper, reliable save routine. To start with, what do you mean "detect
> issues on startup" -- startup of what?

Because fundamentally it's impossible to write a reliable save
routine. The best you can do is detect, on subsequent restart of your
program, that there was an incomplete save in a previous session, and
either roll it back or complete it. In some cases that's trivial - if
there's a temporary file, you have the partial save, so you delete the
temporary (rolling back) - or you just ignore it and overwrite, and
hope that it's not going to get stuck anywhere. But it's still the
same broad technique, whichever way you do it.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web