Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.development.system > #654

Re: shred or scrub

Show all headers | View raw

On 25/04/14 11:57, crankypuss wrote:
> On 04/24/2014 02:46 PM, David Brown wrote:
>> On 23/04/14 13:50, crankypuss wrote:
>>> On 04/22/2014 04:06 PM, David Brown wrote:
>>
>> My point is that it is impossible for a /filesystem/ to guarantee
>> erasure of old data.
>
> If you're talking specific linux filesystems, or linux filesystems in
> general, that may be the case; if you're talking filesystems in the
> wholly general sense, it is not the case.

No, it is the case for /all/ filesystems.  For some combinations of 
particular filesystems and particular media, you can be sure of erasing 
old data (or at least, sure that you can erase old data assuming you 
don't have a hardware error that prevents the change).

I think you are failing to understand that there are multiple layers 
here.  A filesystem sits on top of a block device of some sort.  There 
can be multiple layers of block devices (such as raid, encryption, lvm, 
etc.).  Then there is the media, which will normally have some sort of 
translation or abstraction layer to make it look like an array of 
sectors (for flash devices, this layer handles logical-to-physical 
translation, garbage collection, etc., while for hard disks it handles 
sector relocations, error correction, etc.).

If the media does not support erasing or overwriting, or if it makes it 
hard to do so reliably, then /no/ filesystem can help.

For example, if we take your mythical "erase on demand" filesystem and 
use it on a raid1 disk pair, it might seem to be able to erase deleted 
files completely.  But if we take out one of the disks, the raid1 array 
will still work fine in degraded mode - you can use it to read, write 
and erase files.  As far as the filesystem is concerned, everything is 
working as it should.  If you ask the filesystem to securely erase a 
file, it will appear to do so - and the filesystem thinks the file is 
erased.  But in reality a copy of the file is still there on the other 
disk that was removed.


>
>> In many cases, you could get very close - on a harddisk, if you
>> overwrite an existing sector then old data on the sector is gone for
>> ever (the idea of having to make multiple passes of random data, ones,
>> zeros, etc., is a baseless myth).
>
> Assuming that you verify that the data has actually been overwritten,
> yes; however, read/verify-after-write is costly, so it becomes a
> performance-vs-reliability issue unless there are specific interfaces to
> specify the various operations.

And what does this verify tell you?  It tells you only that the system 
believes the data has been written to the media.  It says nothing about 
what might have been overwritten or what might still exist.

>
>> However, harddisks occasionally
>> relocate sectors - you can't control this, and you cannot access the old
>> data or erase it without opening the disk physically.  It's a rare
>> occurrence - but for the paranoid, less than 100% is not good enough.
>
> If the device lies about what it is doing, all one can do is put it on a
> blacklist and refuse to support the device entirely, or establish it as
> a device to be used for non-critical data only.

The device is not lying - it is just that no one claims that harddrives 
(or other media) work the way you apparently think they do.  Once you 
realise that your idea of harddisks as a simple array of directly 
addressable sectors is oversimplifying, and completely wrong for flash 
devices, then you will understand the issues here.

In reality, of course, you do not need 100% guarantees that deleted data 
is gone - normally, it is sufficient to say that it will /probably/ be 
gone, and it will certainly be hard to recover anything that happens to 
be left over.  Hard disks and SSDs will give you that, as will most 
filesystems - even those that have a policy of not overwriting old data. 
  Even though in theory there may be some or all of the data on the 
disk, it is extremely hard to get at it.

>
> When "device" behavior reaches a certain point of independence, it has
> actually become a server and needs to be given no more trust than a
> server accessed over a trusted connection.  It is (imo, at least)
> necessary to be cognizant of whether the hardware designers have stuck
> their nose into your business, and treat those "devices" with
> significant skepticism.

Nonsense.  You are imagining rules about what /you/ think device 
manufacturers should do - but they are your rules, and neither device 
manufacturers nor other users play by those rules.  There are no hard 
and fast dividing lines here.

>
>> In other cases, particularly SSDs, then overwriting existing logical
>> sectors does not affect the old data directly, and it could be
>> recovered.  Other forms of data storage, such as cloud storage, are
>> getting more common - and you have even less control there.
>
> If there is no device reliability, the device should not be used for
> critical data.  You wouldn't expect a large business to send its daily
> cash intake to the bank by using some homeless person grabbed off the
> street, and using cloud storage is similar; some may be reliable but
> others not.

Reliability has no remote connection with your desire for 100% erasability.

>
>> So no filesystem can actually guarantee that it can erase old data - and
>> filesystems therefore normally provide no functions to erase old data.
>
> If a courier service routinely grabs homeless people off the street to
> deliver large amounts of cash, one might wisely choose not to use them.
>
> We still disagree about what "no" filesystem can do.  If you mean to
> limit the discussion to "existing linux filesystems" then I might have
> no choice but to concede your point; if you want to talk about what can
> be done, what is actually possible, then we will probably continue to
> disagree on this subject.

See earlier in this post for the critical issue of what a filesystem 
actually /is/, and how it is independent of the media.

A combination of filesystem and media may give you close to the 100% 
data erasability that you are looking for, although it still assumes 
there are no failures in hardware and software along the way.  Old hard 
disks did not have relocatable sectors, and thus no risk of storing data 
in places that were not addressable by the filesystem - such disks could 
be used with a filesystem that supports scrubbing.

Other than that, there is an extremely simple way to get 100% 
erasability on whatever media you want, and with whatever filesystem you 
want - use encryption.  At the point of encryption (either on the block 
layer, or in the filesystem, or on particular files), you can get full 
erasure.

There is also physical destruction of the media itself, if that appeals.

>
>> Whether they re-use deleted sectors quickly or slowly is a matter of the
>> structure of the file-system, and what is most efficient, and it may
>> vary depending on the media (placement to minimise fragmentation and
>> head movement is usually best on harddisks, re-using deleted logical
>> sectors is most efficient on SSD's because it becomes an automatic TRIM
>> - even though the physical sectors will not be reused quickly).
>>
>>>
>>> While encryption does provide the advantage of helping to protect data
>>> stored on a device that becomes physically available to the malicious,
>>> the advent of "key managers" leads one to consider the possibility that
>>> encrypted filesystems may amount to much ado about nothing, and that
>>> other avenues of protection may be more effective overall.
>>
>> That is certainly true - your data is only as safe as your key.  But the
>> usual procedure for encrypted partitions is to enter your key (or
>> passphrase) at the keyboard when mounting.
>>
>
> Yes, and once you have done so, the mounted partition is no more secure
> than a non-encrypted partition; data security has, through your
> supplying the key, been degraded from whatever level of security is
> provided by the encryption algorithm used, to the simple "security"
> provided by the linux protection modes.  To quote you, "for the
> paranoid, less than 100% is not good enough".

It does not take a genius to understand that the key is needed to access 
the data.  If you want to minimise the time that the partition or data 
is unlocked, which is understandable for some uses, then simply unmount 
the encrypted partition when it is not in use.  You can also easily 
encrypt individual files rather than the whole partition, if it makes 
things easier for you.

>
> I am of the opinion that the linux protection modes inherited from Unix
> are not very "adequate", and the idea of a "superuser" is inherently
> crippled "security".  The whole user/group/world concept along with the
> ability to open files for non-exclusive access is ridiculously
> antiquated and stems from the pre-1970 era when bisync was
> state-of-the-art and the concept of servers as separate asset owners was
> little more than a foggy idea in the minds of a few.  The architecture
> survives to the present basically because it is politically advantageous
> to maintain posix compatability, but personally I would prefer something
> a bit more modern even at the cost of losing compatability.

This is again a completely different issue, and bears no connection with 
the discussion so far.  But since you have brought it up, I would say 
that I find the user/group/world arrangement extremely useful, and 
flexible enough to cover almost all security situations I have come 
across (I run the company network and servers, for about 80 users) with 
very little maintenance and effort.  In our pre-Linux days when we had a 
Windows NT 4.0 server, using Windows ACL's was a great pain and a huge 
waste of time - and usually problems with access were "fixed" by giving 
access to everyone, or the administrator having to first claim ownership 
of directory trees before being able to grant appropriate access.  In 
theory it offered very fine grain control - in practice, it was a mess.

Having said that, there /are/ occasions when finer control than posix 
permissions can be useful.  This is why Linux has had a number of 
additional security systems, including Access Control Lists, selinux, 
capabilities, etc., for many years.  Most people (including me) don't 
use them because we don't need them, but they certainly exist for those 
that have special requirements.

>
> Sorry for ranting, I'll stop before beginning to spew about whether
> encryption is really useful. <g>

That's fine - I'm ranting too, and it's best to get it out in the open!

Back to comp.os.linux.development.system | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-16 18:17 -0400
  Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-17 04:19 -0600
    Re: shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-18 22:30 -0400
      Re: shred or scrub Jasen Betts <jasen@xnet.co.nz> - 2014-04-19 07:42 +0000
        Re: shred or scrub Richard Kettlewell <rjk@greenend.org.uk> - 2014-04-19 10:04 +0100
      Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-19 02:15 -0600
      Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-19 23:05 +0100
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-20 02:47 -0600
          Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-20 07:56 -0500
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-21 03:51 -0600
              Re: shred or scrub Jasen Betts <jasen@xnet.co.nz> - 2014-04-21 11:50 +0000
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-21 06:14 -0600
            Re: shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-21 18:44 -0400
          Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-21 13:24 +0100
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-22 04:10 -0600
              Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-22 14:39 +0100
  Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-17 13:15 +0000
    Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-17 09:40 -0500
      Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-18 14:40 +0000
    Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 02:12 -0600
      Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-18 11:49 +0200
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 09:59 -0600
          Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-21 16:14 +0200
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-22 04:22 -0600
              Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-23 00:06 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-23 05:50 -0600
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-24 22:46 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-25 03:57 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-25 19:14 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-26 04:02 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-27 21:26 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 03:27 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 12:17 +0100
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 13:01 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-29 02:50 -0600
                UNIX(*)/ Linux history & system design (was: shred or scrub) Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-05 21:31 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-05 16:02 -0600
                Re: UNIX(*)/ Linux history & system design David Brown <david.brown@hesbynett.no> - 2014-05-06 01:17 +0200
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-06 01:46 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-06 15:09 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-06 23:47 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-07 16:23 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 10:51 -0600
                Re: UNIX(*)/ Linux history & system design Jerry Peters <jerry@example.invalid> - 2014-05-07 20:25 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-08 03:50 -0600
                Re: UNIX(*)/ Linux history & system design Jerry Peters <jerry@example.invalid> - 2014-05-08 20:24 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-09 02:23 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-09 18:36 +0100
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-09 21:24 +0100
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-07 22:01 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-08 03:37 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-08 14:02 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-09 02:56 -0600
                Re: UNIX(*)/ Linux history & system design David Brown <david.brown@hesbynett.no> - 2014-05-07 00:15 +0200
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 00:32 -0600
                Re: UNIX(*)/ Linux history & system design Jorgen Grahn <grahn+nntp@snipabacken.se> - 2014-05-07 08:47 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 10:59 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-06 14:35 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-26 16:30 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-27 05:59 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-27 20:15 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 03:29 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 12:06 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-27 21:41 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 04:03 -0600
                Re: shred or scrub Richard Kettlewell <rjk@greenend.org.uk> - 2014-04-28 16:44 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-28 23:39 +0200
      Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-18 07:37 -0500
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 10:16 -0600
          Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-18 12:01 -0500
      Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-18 14:42 +0000
  Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-17 16:41 +0200

csiph-web