Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #679

Re: Robust file I/O

Show all headers | View raw

Chris Uppal wrote:
>Hallvard B Furuseth wrote:
>> How hard must a program work to read/write file records as
>> uncorrupted as SQL databases do?  I've read this:
>>    Atomic Commit In SQLite: 2.0 Hardware assumptions
>>    http://www.sqlite.org/draft/atomiccommit.html#hardware
>> and I'm left thinking that my data would be safer in an SQL database
>> than in a regular file, even when I have no use for SQL itself.  Except
>> SQL ACID probably dislikes remote filesystems, which I want to support.
>
> Maybe.  Depends what failure scenarios you want to avoid.  For example
> (this is a real example) at my day job we have recently discovered
> that several dozen multi-gigabyte SQL databases are all more-or-less
> corrupt, after a creeping disk/motherboard problem has injected small
> amounts of garbage into critical disk structures (not to mention the
> data itself).  Fortunately none of those databases held anything of
> critical importance (just test data and the like), but still...

I expect I'd be content to catch that with a checksum and tell the
user to restore from backup.  But I know little of what kind of
hardware problems to be concerned with nowadays.

> I mean, do you really /need/ disk IO that's more reliable than that
> used by the editor you wrote it in, of the OS that is running it ?
> ;-)

Depends on what you mean.  Assuming OS and disk work perfectly, I'm
concerned with the inconsistent data the file contains during an
update.  Which can be exposed by ^C between two write()s, bugs in my
program, sync issues, disk full, power failure, etc.
For example, the SQLite URL above says:

  SQLite does _not_ assume that a sector write is atomic.  However, it
  does assume that a sector write is linear. (...)  If a power failure
  occurs in the middle of a sector write it might be that part of the
  sector was modified and another part was left unchanged.  The key
  assumption by SQLite is that if any part of the sector gets changed,
  then either the first or the last bytes will be changed.  So the
  hardware will never start writing a sector in the middle and work
  towards the ends.  We do not know if this assumption is always true
  but it seems reasonable.

Maybe I should goto comp.databases and ask there for that kind of thing.

> (...)
> You might be interested in log-structured storage, which is nicely
> expounded on Nick Johnson's blog, at:
>  http://blog.notdot.net/2009/12/Damn-Cool-Algorithms-Log-structured-storage

Thanks!  That's also a nice key phrase to Google for.

>> If I write some data and the write fails halfway through, can data in
>> the _neighbouring_ record(s) be corrupted in today's filesystems?
>
> As I mentioned above, data in /any/ record can be corrupted at any time in 
> today's filesystems.  So, you'll need some sort of checksumming.

Ok, rephrasing:  I don't care about "at any time", only the presumably
higher risk during a write.  Which you actally answered already:

  "Since IO happens in pages, pages are going to be your "unit" of
  correctness."

How about if there is no disk problem, only a power outage?  

>> If I write two records, I want another reader process to see them hit
>> the disk in the correct order, despite all the buffering which happens
>> nowadays, and even if something crashes in the middle.  So I must drop
>> to OS-level fdatasync() @ unix and whatever() @ Windows.
>
> Don't forget that there's caching below the level of the OS in the disk 
> controller hardware.  I doubt (without being an expert) that there is any 
> generic and totally reliable method of ensuring what has been flushed to the 
> actual disk.  There may be exceptions, but not generic.

Yes, that seems iffy.  Probably have to go for "mostly reliable",
but it may be that people have worked out some tricks that often work.

>> I expect I should stick to stdio to avoid OS-specific quirks with Unix
>> write() or the Windows equivalent, in particular in case write() is a
>> Unix write() emulation on Windows.
>
> But stdio on Windows is only an emulation of the Unix stdio.

Weird.  The C language's stdio definition tries hard to be
OS-independent.  Oh well.

> (...)
>> Then there is attempting file locking which works even across NFS, IIRC
>> some special trick are needed for that.
>
> As far as I know/remember, all the serverless disk-based databases (such as 
> SQLite, BerkleyDB, etc) advise against putting the data on a networked 
> filesystem.  Shared, fast, reliable, access to a disk over a network is just 
> Too Hard (tm).

I did't ask for fast:-)  I was thinking this could be easier if I stick
to append-only, no seeking before a write.

-- 
Hallvard

Back to comp.programming | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-13 14:59 +0200
  Re: Robust file I/O Zeljko Vrba <mordor.nospam@fly.srk.fer.hr> - 2011-07-13 18:15 +0000
    Re: Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-15 14:26 +0200
      Re: Robust file I/O Ian Collins <ian-news@hotmail.com> - 2011-07-16 13:21 +1200
        Re: Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-19 15:51 +0200
  Re: Robust file I/O "Chris Uppal" <chris.uppal@metagnostic.REMOVE-THIS.org> - 2011-07-16 14:19 +0100
    Re: Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-19 15:44 +0200

csiph-web