Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #647 > unrolled thread

Robust file I/O

Back to article view | Back to comp.programming

Contents

  Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-13 14:59 +0200
    Re: Robust file I/O Zeljko Vrba <mordor.nospam@fly.srk.fer.hr> - 2011-07-13 18:15 +0000
      Re: Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-15 14:26 +0200
        Re: Robust file I/O Ian Collins <ian-news@hotmail.com> - 2011-07-16 13:21 +1200
          Re: Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-19 15:51 +0200
    Re: Robust file I/O "Chris Uppal" <chris.uppal@metagnostic.REMOVE-THIS.org> - 2011-07-16 14:19 +0100
      Re: Robust file I/O Hallvard B Furuseth <h.b.furuseth@usit.uio.no> - 2011-07-19 15:44 +0200

#647 — Robust file I/O

How hard must a program work to read/write file records as
uncorrupted as SQL databases do?  I've read this:
    Atomic Commit In SQLite: 2.0 Hardware assumptions
    http://www.sqlite.org/draft/atomiccommit.html#hardware
and I'm left thinking that my data would be safer in an SQL database
than in a regular file, even when I have no use for SQL itself.  Except
SQL ACID probably dislikes remote filesystems, which I want to support.

Simple case:  The file structure is just a set of variable-length
records.  No indexing needed:  The program reads all record headers
at startup and maintains the file.  No concurrent writers.

The user should not need to maintain or configure anything, I just want
a simple file or directory with a file.

Is there a portable library which can be used for that, without dragging
in something huge like SQL?  And how would it work, anyway?  I've been
living in the lofty land of high-level programming so long I don't even
know where to start reading...


I can think of/reinvent some issues and solutions.  For example:

Don't not overwrite an active record since a write failure would break
it.  Instead records can have format {record ID, record version, data}
and I'd obsolete the old record by writing another record with the same
ID and a higher version.  Then sometimes garbage-collect to a new file.
That also allows backup by simple file copy, only the last record in a
backup could be incomplete and might need to be ignored.

If I write some data and the write fails halfway through, can data in
the _neighbouring_ record(s) be corrupted in today's filesystems?

If I write two records, I want another reader process to see them hit
the disk in the correct order, despite all the buffering which happens
nowadays, and even if something crashes in the middle.  So I must drop
to OS-level fdatasync() @ unix and whatever() @ Windows.

I'll want a checksum (CRC?) to catch error states like half-written
records, and a way to deal with broken records.

I expect I should stick to stdio to avoid OS-specific quirks with Unix
write() or the Windows equivalent, in particular in case write() is a
Unix write() emulation on Windows.

Is there a reason other than performance to align on the file system's
page size?

If I ever overwrite old records, the file format would be inconsistent
at the time when I'm the middle of overwriting the record length.  So
I've just reinvented overflow pages: I want fixed-size record headers
with references to variable-length data records in overflow pages.

Then there is attempting file locking which works even across NFS, IIRC
some special trick are needed for that.

Anything else?

-- 
Hallvard

[toc] | [next] | [standalone]

#649

On 2011-07-13, Hallvard B Furuseth <h.b.furuseth@usit.uio.no> wrote:
>
> Is there a portable library which can be used for that, without dragging
> in something huge like SQL?  And how would it work, anyway?  I've been
>
Maybe something like this: http://fallabs.com/tokyocabinet/
(tokyo cabinet is LGPL, the newer Kyoto cabinet is GPL).

[toc] | [prev] | [next] | [standalone]

#652

Zeljko Vrba writes:
> On 2011-07-13, Hallvard B Furuseth <h.b.furuseth@usit.uio.no> wrote:
>>
>> Is there a portable library which can be used for that, without dragging
>> in something huge like SQL?  And how would it work, anyway?  I've been
>>
> Maybe something like this: http://fallabs.com/tokyocabinet/
> (tokyo cabinet is LGPL, the newer Kyoto cabinet is GPL).

Thanks, that looks good.  Though it's actually more high-level than I
had in mind.

And I still wonder how one programs such robust I/O from scratch, if
anyone has pointers for reading about that.

-- 
Hallvard

[toc] | [prev] | [next] | [standalone]

#653

On 07/16/11 12:26 AM, Hallvard B Furuseth wrote:
> Zeljko Vrba writes:
>> On 2011-07-13, Hallvard B Furuseth<h.b.furuseth@usit.uio.no>  wrote:
>>>
>>> Is there a portable library which can be used for that, without dragging
>>> in something huge like SQL?  And how would it work, anyway?  I've been
>>>
>> Maybe something like this: http://fallabs.com/tokyocabinet/
>> (tokyo cabinet is LGPL, the newer Kyoto cabinet is GPL).
>
> Thanks, that looks good.  Though it's actually more high-level than I
> had in mind.
>
> And I still wonder how one programs such robust I/O from scratch, if
> anyone has pointers for reading about that.

SQLite?

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#680

Ian Collins writes:
>On 07/16/11 12:26 AM, Hallvard B Furuseth wrote:
>>>On 2011-07-13, Hallvard B Furuseth<h.b.furuseth@usit.uio.no>  wrote:
>>>>
>>>> Is there a portable library which can be used for that, without dragging
>>>> in something huge like SQL?  And how would it work, anyway?  I've been
>> (...)
>> And I still wonder how one programs such robust I/O from scratch, if
>> anyone has pointers for reading about that.
>
> SQLite?

That's where I started.  Unless you are referring to some design
documents there which I've missed.

-- 
Hallvard

[toc] | [prev] | [next] | [standalone]

#663

Hallvard B Furuseth wrote:
> How hard must a program work to read/write file records as
> uncorrupted as SQL databases do?  I've read this:
>    Atomic Commit In SQLite: 2.0 Hardware assumptions
>    http://www.sqlite.org/draft/atomiccommit.html#hardware
> and I'm left thinking that my data would be safer in an SQL database
> than in a regular file, even when I have no use for SQL itself.  Except
> SQL ACID probably dislikes remote filesystems, which I want to support.

Maybe.  Depends what failure scenarios you want to avoid.  For example (this is 
a real example) at my day job we have recently discovered that several dozen 
multi-gigabyte SQL databases are all more-or-less corrupt, after a creeping 
disk/motherboard problem has injected small amounts of garbage into critical 
disk structures (not to mention the data itself).  Fortunately none of those 
databases held anything of critical importance (just test data and the like), 
but still...

I mean, do you really /need/ disk IO that's more reliable than that used by the 
editor you wrote it in, of the OS that is running it ?  ;-)

Anyway...

> Don't not overwrite an active record since a write failure would break
> it.  Instead records can have format {record ID, record version, data}
> and I'd obsolete the old record by writing another record with the same
> ID and a higher version.  Then sometimes garbage-collect to a new file.
> That also allows backup by simple file copy, only the last record in a
> backup could be incomplete and might need to be ignored.

You might be interested in log-structured storage, which is nicely expounded on 
Nick Johnson's blog, at:
    http://blog.notdot.net/2009/12/Damn-Cool-Algorithms-Log-structured-storage


> If I write some data and the write fails halfway through, can data in
> the _neighbouring_ record(s) be corrupted in today's filesystems?

As I mentioned above, data in /any/ record can be corrupted at any time in 
today's filesystems.  So, you'll need some sort of checksumming.


> If I write two records, I want another reader process to see them hit
> the disk in the correct order, despite all the buffering which happens
> nowadays, and even if something crashes in the middle.  So I must drop
> to OS-level fdatasync() @ unix and whatever() @ Windows.

Don't forget that there's caching below the level of the OS in the disk 
controller hardware.  I doubt (without being an expert) that there is any 
generic and totally reliable method of ensuring what has been flushed to the 
actual disk.  There may be exceptions, but not generic.


> I'll want a checksum (CRC?) to catch error states like half-written
> records, and a way to deal with broken records.
>
> I expect I should stick to stdio to avoid OS-specific quirks with Unix
> write() or the Windows equivalent, in particular in case write() is a
> Unix write() emulation on Windows.

But stdio on Windows is only an emulation of the Unix stdio.  If you don't go 
to the bottom of the software stack, then you'll have no hope at all of 
controlling how the OS is buffering your data.

Alternatively, if you can work out a way of doing this that is /independent/ of 
the buffering strategy, then maybe stdio would be your friend, since it won't 
tempt you to depend on apparent guarantees about behaviour which may not 
actually exist, or may not be fully honoured by the OS.


> Is there a reason other than performance to align on the file system's
> page size?

Since IO happens in pages, pages are going to be your "unit" of correctness.


> Then there is attempting file locking which works even across NFS, IIRC
> some special trick are needed for that.

As far as I know/remember, all the serverless disk-based databases (such as 
SQLite, BerkleyDB, etc) advise against putting the data on a networked 
filesystem.  Shared, fast, reliable, access to a disk over a network is just 
Too Hard (tm).

    -- chris 

[toc] | [prev] | [next] | [standalone]

#679

Chris Uppal wrote:
>Hallvard B Furuseth wrote:
>> How hard must a program work to read/write file records as
>> uncorrupted as SQL databases do?  I've read this:
>>    Atomic Commit In SQLite: 2.0 Hardware assumptions
>>    http://www.sqlite.org/draft/atomiccommit.html#hardware
>> and I'm left thinking that my data would be safer in an SQL database
>> than in a regular file, even when I have no use for SQL itself.  Except
>> SQL ACID probably dislikes remote filesystems, which I want to support.
>
> Maybe.  Depends what failure scenarios you want to avoid.  For example
> (this is a real example) at my day job we have recently discovered
> that several dozen multi-gigabyte SQL databases are all more-or-less
> corrupt, after a creeping disk/motherboard problem has injected small
> amounts of garbage into critical disk structures (not to mention the
> data itself).  Fortunately none of those databases held anything of
> critical importance (just test data and the like), but still...

I expect I'd be content to catch that with a checksum and tell the
user to restore from backup.  But I know little of what kind of
hardware problems to be concerned with nowadays.

> I mean, do you really /need/ disk IO that's more reliable than that
> used by the editor you wrote it in, of the OS that is running it ?
> ;-)

Depends on what you mean.  Assuming OS and disk work perfectly, I'm
concerned with the inconsistent data the file contains during an
update.  Which can be exposed by ^C between two write()s, bugs in my
program, sync issues, disk full, power failure, etc.
For example, the SQLite URL above says:

  SQLite does _not_ assume that a sector write is atomic.  However, it
  does assume that a sector write is linear. (...)  If a power failure
  occurs in the middle of a sector write it might be that part of the
  sector was modified and another part was left unchanged.  The key
  assumption by SQLite is that if any part of the sector gets changed,
  then either the first or the last bytes will be changed.  So the
  hardware will never start writing a sector in the middle and work
  towards the ends.  We do not know if this assumption is always true
  but it seems reasonable.

Maybe I should goto comp.databases and ask there for that kind of thing.

> (...)
> You might be interested in log-structured storage, which is nicely
> expounded on Nick Johnson's blog, at:
>  http://blog.notdot.net/2009/12/Damn-Cool-Algorithms-Log-structured-storage

Thanks!  That's also a nice key phrase to Google for.

>> If I write some data and the write fails halfway through, can data in
>> the _neighbouring_ record(s) be corrupted in today's filesystems?
>
> As I mentioned above, data in /any/ record can be corrupted at any time in 
> today's filesystems.  So, you'll need some sort of checksumming.

Ok, rephrasing:  I don't care about "at any time", only the presumably
higher risk during a write.  Which you actally answered already:

  "Since IO happens in pages, pages are going to be your "unit" of
  correctness."

How about if there is no disk problem, only a power outage?  

>> If I write two records, I want another reader process to see them hit
>> the disk in the correct order, despite all the buffering which happens
>> nowadays, and even if something crashes in the middle.  So I must drop
>> to OS-level fdatasync() @ unix and whatever() @ Windows.
>
> Don't forget that there's caching below the level of the OS in the disk 
> controller hardware.  I doubt (without being an expert) that there is any 
> generic and totally reliable method of ensuring what has been flushed to the 
> actual disk.  There may be exceptions, but not generic.

Yes, that seems iffy.  Probably have to go for "mostly reliable",
but it may be that people have worked out some tricks that often work.

>> I expect I should stick to stdio to avoid OS-specific quirks with Unix
>> write() or the Windows equivalent, in particular in case write() is a
>> Unix write() emulation on Windows.
>
> But stdio on Windows is only an emulation of the Unix stdio.

Weird.  The C language's stdio definition tries hard to be
OS-independent.  Oh well.

> (...)
>> Then there is attempting file locking which works even across NFS, IIRC
>> some special trick are needed for that.
>
> As far as I know/remember, all the serverless disk-based databases (such as 
> SQLite, BerkleyDB, etc) advise against putting the data on a networked 
> filesystem.  Shared, fast, reliable, access to a disk over a network is just 
> Too Hard (tm).

I did't ask for fast:-)  I was thinking this could be easier if I stick
to append-only, no seeking before a write.

-- 
Hallvard

[toc] | [prev] | [standalone]

Back to top | Article view | comp.programming

csiph-web