Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.development.system > #661

Re: shred or scrub

Show all headers | View raw

On 04/27/2014 01:41 PM, David Brown wrote:
> On 27/04/14 13:59, crankypuss wrote:
>> On 04/26/2014 08:30 AM, David Brown wrote:
>>
>>> I think you are failing to understand that there are multiple layers
>>> here.
>>
>>> If the media does not support erasing or overwriting, or if it makes it
>>> hard to do so reliably, then /no/ filesystem can help.
>>
>>> The device is not lying - it is just that no one claims that harddrives
>>> (or other media) work the way you apparently think they do.  Once you
>>> realise that your idea of harddisks as a simple array of directly
>>> addressable sectors is oversimplifying, and completely wrong for flash
>>> devices, then you will understand the issues here.
>>
>>> Nonsense.  You are imagining rules about what /you/ think device
>>> manufacturers should do - but they are your rules, and neither device
>>> manufacturers nor other users play by those rules.  There are no hard
>>> and fast dividing lines here.
>>
>>> Reliability has no remote connection with your desire for 100%
>>> erasability.
>>
>>> See earlier in this post for the critical issue of what a filesystem
>>> actually /is/, and how it is independent of the media.
>>>
>>> A combination of filesystem and media may give you close to the 100%
>>> data erasability that you are looking for, although it still assumes
>>> there are no failures in hardware and software along the way.  Old hard
>>> disks did not have relocatable sectors, and thus no risk of storing data
>>> in places that were not addressable by the filesystem - such disks could
>>> be used with a filesystem that supports scrubbing.
>>
>> It seems to me that we are talking at cross-purposes here; we are
>> largely stating the same meanings but expressing them in different ways,
>> and the syntax of our expressions is making their semantics seem more
>> different than it is.
>>
>> You seem to think I'm saying that hardware makers should conform to my
>> views of how hardware should work, but what I'm saying is something
>> different, I'm saying that the software on the client system needs to
>> ascertain the nature of the subsystem that serves it files and behave
>> accordingly.  Maybe that will be ascertained by looking up the device
>> identification presented by the device (they do sometimes fib about what
>> they are when new hardware is introduced to support existing interface
>> models) then finding its characteristics in some device information
>> database, or it might be ascertained by monitoring the behavior of the
>> device; eventually, we are likely to reach the point where only
>> behavior-monitoring is useful, bestcase.
>>
>
> This would ruin the flexibility, testability, and modularisation of
> filesystems which are dependent on a separation of layers.  A filesystem
> should /not/ know about the block device under it - it should know only
> about the set of operations provided by that device (such as to read or
> write sectors of data, and flush queues of writes or otherwise report
> back that a write has "hit the metal").  There are a few other things
> that filesystems ask about in order to better optimise accesses, such as
> whether it is an SSD-class or harddisk-class device (to know about head
> movements), and stripe sizes (if it is a raid device).
>
> Trying to have something like a database of devices leads to broken
> systems - it makes testing infeasible (as you need to test every
> filesystem with every device type), it leads to bugs and security holes
> (as everything is needlessly complicated, and cannot be tested), and
> endless user frustration (as they are unable to use the filesystem of
> their choice on the device of their choice, as that combination is not
> in the database).
>
> There are good reasons why operating systems are built in layers.

Talking across one another some more, implementation layers versus 
abstraction layers, so it goes.

>> However.  One thing I am saying is that if an operating system claims to
>> support file-scrubbing on a "filesystem" that does not support
>> file-scrubbing, then that operating system is broken.
>
> I don't disagree with that.  But since no operating system (that I know
> of) and no filesystem (that I know of) claim to support file scrubbing,
> it is a non-issue.

Whatever, I've written a filesystem that would support scrubbing, but 
it's been obsolete for longer than most posters here have been alive.

Over the years we learn things, and sometimes we see that what we 
thought were "abstractions" weren't really "the" abstraction we thought 
they were because of the things we've learned.

Some people grasp a new level of abstraction and others don't, and 
whether it's because they aren't capable of further abstraction, or 
because they have vested interests in existing ones, isn't much relevant.

>> If it allows a
>> file that needs to be placed only on a "scrub-able" filesystem to be
>> placed on any other type of filesystem, it is broken.  And if it does
>> not provide interfaces that allow the programmer to identify the file as
>> "scrub-able" versus "plain-old-data" when it is being created, then it
>> is not exactly broken, but it definitely lame.
>
> /True/ scrubbing is not possible, and is not supported by any system (as
> I have explained).
>
> You can get /reasonable/ scrubbing on some filesystems, if the
> underlying media is a harddisk.  Such scrubs are often good enough.

I suppose that whether one considers the "involate" attribute to mean 
one thing, or another, can make a difference.  Back to "levels" again, 
oh my.

>> So, disclaimers that shred/scrub are only useful on this or that kind of
>> filesystem are inherently faulty; at a minimum the shred/scrub
>> operations should be returning a blatant failure indication when issued
>> against a file residing on an unsupported filesystem (and never having
>> used them, much less read their code, I don't know if that is the case
>> or not, but as I recall the OP didn't mention their failing, only the
>> warning not to trust them on this or that filesystem).
>
> Note that the original poster is a half-wit troll (just look him up in
> the Usenet archives if you are not familiar with him), so don't pay any
> attention to what he has said.

Even trolls occasionally stumble across an interesting point.

> He accidentally triggered an interesting
> discussion, but we would do best to ignore his actual post.
>
> The "shred" manual page says fairly clearly what it does - it writes
> over a file one or more times "in order to make it harder for even very
> expensive hardware probing to recover the data".  And it makes clear the
> critical assumption - that the filesystem overwrites data in place.  It
> does not claim to completely delete all the data - it just claims to
> make it harder to recover, and only if the filesystem works in a
> particular way.
>
> And even these claims are being made by a /program/ - not the operating
> system or the filesystem.

I'm going to carefully restrain myself here so as not to open the can of 
worms associated with the question of where an "operating system" begins 
and ends, whether the "operating system" is comprised of the 
kernel-and-below, or all-provided-interfaces (which includes commands as 
well as libs, and which pipeline-lovers could argue includes 
command-output as well).

>> Now, granted there are not a preponderance of devices available today
>> that support the concept of scrubbing, but that does not mean that
>> encryption as we know it today is the answer.  An interim answer, yes, I
>> suppose, but not a real answer.  Passphrase-oriented protection
>> mechanisms are only useful if the passphrase cannot be guessed (through
>> cleverness or a brute-force attack) and if the owner of the data cannot
>> be dragged away to some cellar where he is thumped-on or waterboarded or
>> whatever until he gives up the passphrase.
>
> There /is/ a way to handle this, and it is done today - either in
> software or in hardware (you can get disk drives with this feature built
> in).  You need two passwords - one is known to the user, and one is
> generated randomly on creation of the filesystem (or when the harddisk
> is first locked).  The user password is used to encrypt the random
> password, and this random password is stored in the disk's controller or
> in some other special way.  All data on the disk is encrypted using the
> random password.  So to be able to read something off the disk, the user
> must first present the user password in order to decode the real
> encryption password.

It isn't entirely clear whether what you are describing is dual-key 
cryptography like PGP uses, or something simpler and not truly 
multiplicative.

Whatever level of encryption suits you is fine with me, it's a whole 
bunch easier to encrypt something than it is to decrypt it, there are 
not-very-difficult techniques that must make the NSA crazy.

However, my point was that instead of using sloppy access protocols that 
make encryption essential, perhaps we should put more attention into the 
access protocol and leave data encryption as a private matter.

I'm more interested in moving my own codebase forward another tiny step 
than I am in winning the perpetual debate; it's a more useful activity 
for all concerned imo.

Back to comp.os.linux.development.system | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-16 18:17 -0400
  Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-17 04:19 -0600
    Re: shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-18 22:30 -0400
      Re: shred or scrub Jasen Betts <jasen@xnet.co.nz> - 2014-04-19 07:42 +0000
        Re: shred or scrub Richard Kettlewell <rjk@greenend.org.uk> - 2014-04-19 10:04 +0100
      Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-19 02:15 -0600
      Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-19 23:05 +0100
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-20 02:47 -0600
          Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-20 07:56 -0500
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-21 03:51 -0600
              Re: shred or scrub Jasen Betts <jasen@xnet.co.nz> - 2014-04-21 11:50 +0000
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-21 06:14 -0600
            Re: shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-21 18:44 -0400
          Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-21 13:24 +0100
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-22 04:10 -0600
              Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-22 14:39 +0100
  Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-17 13:15 +0000
    Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-17 09:40 -0500
      Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-18 14:40 +0000
    Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 02:12 -0600
      Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-18 11:49 +0200
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 09:59 -0600
          Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-21 16:14 +0200
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-22 04:22 -0600
              Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-23 00:06 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-23 05:50 -0600
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-24 22:46 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-25 03:57 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-25 19:14 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-26 04:02 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-27 21:26 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 03:27 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 12:17 +0100
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 13:01 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-29 02:50 -0600
                UNIX(*)/ Linux history & system design (was: shred or scrub) Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-05 21:31 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-05 16:02 -0600
                Re: UNIX(*)/ Linux history & system design David Brown <david.brown@hesbynett.no> - 2014-05-06 01:17 +0200
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-06 01:46 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-06 15:09 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-06 23:47 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-07 16:23 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 10:51 -0600
                Re: UNIX(*)/ Linux history & system design Jerry Peters <jerry@example.invalid> - 2014-05-07 20:25 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-08 03:50 -0600
                Re: UNIX(*)/ Linux history & system design Jerry Peters <jerry@example.invalid> - 2014-05-08 20:24 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-09 02:23 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-09 18:36 +0100
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-09 21:24 +0100
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-07 22:01 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-08 03:37 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-08 14:02 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-09 02:56 -0600
                Re: UNIX(*)/ Linux history & system design David Brown <david.brown@hesbynett.no> - 2014-05-07 00:15 +0200
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 00:32 -0600
                Re: UNIX(*)/ Linux history & system design Jorgen Grahn <grahn+nntp@snipabacken.se> - 2014-05-07 08:47 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 10:59 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-06 14:35 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-26 16:30 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-27 05:59 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-27 20:15 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 03:29 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 12:06 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-27 21:41 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 04:03 -0600
                Re: shred or scrub Richard Kettlewell <rjk@greenend.org.uk> - 2014-04-28 16:44 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-28 23:39 +0200
      Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-18 07:37 -0500
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 10:16 -0600
          Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-18 12:01 -0500
      Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-18 14:42 +0000
  Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-17 16:41 +0200

csiph-web