Robust file I/O

From	Hallvard B Furuseth <h.b.furuseth@usit.uio.no>
Newsgroups	comp.programming
Subject	Robust file I/O
Date	2011-07-13 14:59 +0200
Organization	University of Oslo, Norway
Message-ID	<hbf.20110713f8qr@bombur.uio.no> (permalink)

Show all headers | View raw

How hard must a program work to read/write file records as
uncorrupted as SQL databases do?  I've read this:
    Atomic Commit In SQLite: 2.0 Hardware assumptions
    http://www.sqlite.org/draft/atomiccommit.html#hardware
and I'm left thinking that my data would be safer in an SQL database
than in a regular file, even when I have no use for SQL itself.  Except
SQL ACID probably dislikes remote filesystems, which I want to support.

Simple case:  The file structure is just a set of variable-length
records.  No indexing needed:  The program reads all record headers
at startup and maintains the file.  No concurrent writers.

The user should not need to maintain or configure anything, I just want
a simple file or directory with a file.

Is there a portable library which can be used for that, without dragging
in something huge like SQL?  And how would it work, anyway?  I've been
living in the lofty land of high-level programming so long I don't even
know where to start reading...


I can think of/reinvent some issues and solutions.  For example:

Don't not overwrite an active record since a write failure would break
it.  Instead records can have format {record ID, record version, data}
and I'd obsolete the old record by writing another record with the same
ID and a higher version.  Then sometimes garbage-collect to a new file.
That also allows backup by simple file copy, only the last record in a
backup could be incomplete and might need to be ignored.

If I write some data and the write fails halfway through, can data in
the _neighbouring_ record(s) be corrupted in today's filesystems?

If I write two records, I want another reader process to see them hit
the disk in the correct order, despite all the buffering which happens
nowadays, and even if something crashes in the middle.  So I must drop
to OS-level fdatasync() @ unix and whatever() @ Windows.

I'll want a checksum (CRC?) to catch error states like half-written
records, and a way to deal with broken records.

I expect I should stick to stdio to avoid OS-specific quirks with Unix
write() or the Windows equivalent, in particular in case write() is a
Unix write() emulation on Windows.

Is there a reason other than performance to align on the file system's
page size?

If I ever overwrite old records, the file format would be inconsistent
at the time when I'm the middle of overwriting the record length.  So
I've just reinvented overflow pages: I want fixed-size record headers
with references to variable-length data records in overflow pages.

Then there is attempting file locking which works even across NFS, IIRC
some special trick are needed for that.

Anything else?

-- 
Hallvard

Back to comp.programming | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-13 14:59 +0200
  Re: Robust file I/O Zeljko Vrba <mordor.nospam@fly.srk.fer.hr> - 2011-07-13 18:15 +0000
    Re: Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-15 14:26 +0200
      Re: Robust file I/O Ian Collins <ian-news@hotmail.com> - 2011-07-16 13:21 +1200
        Re: Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-19 15:51 +0200
  Re: Robust file I/O "Chris Uppal" <chris.uppal@metagnostic.REMOVE-THIS.org> - 2011-07-16 14:19 +0100
    Re: Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-19 15:44 +0200

csiph-web