Groups | Search | Server Info | Keyboard shortcuts | Login | Register

Groups > comp.os.linux.development.system > #650

Re: shred or scrub

Show all headers | View raw

On 04/24/2014 02:46 PM, David Brown wrote:
> On 23/04/14 13:50, crankypuss wrote:
>> On 04/22/2014 04:06 PM, David Brown wrote:
>>> On 22/04/14 12:22, crankypuss wrote:
>>>> On 04/21/2014 08:14 AM, David Brown wrote:
>>>>> On 18/04/14 17:59, crankypuss wrote:
>>>>>> On 04/18/2014 03:49 AM, David Brown wrote:
>>>>>>> On 18/04/14 10:12, crankypuss wrote:
>>>>>>>> On 04/17/2014 07:15 AM, Kristof Provost wrote:
>>>>>>>>> On 2014-04-16, Bill Cunningham <nospam@nspam.invalid> wrote:
>>>>>>>>>>      I am using ext4 on my linux. I'm not quite sure of the
>>>>>>>>>> difference in it
>>>>>>>>>> and ext3 but anyway; the shred man page says with the ext3
>>>>>>>>>> filesystem
>>>>>>>>>> shred
>>>>>>>>>> cannot be guaranteed to  work.
>>>>>>>>>>
>>>>>>>>> That's because there's no way to guarantee that the file system
>>>>>>>>> will
>>>>>>>>> write the new data over the same block as the old data. In
>>>>>>>>> fact, in
>>>>>>>>> log-structured file systems (like ZFS, but not ext3/4) the file
>>>>>>>>> system
>>>>>>>>> will deliberately not do this.
>>>>>>>>
>>>>>>>> That seems very messed up.
>>>>>>>
>>>>>>> Some filesystems work this way for particular reasons, such as wear
>>>>>>> leveling (for SSDs), better distribution of data across the disk or
>>>>>>> disks, minimal head movement (for HDs), minimising overwrites in
>>>>>>> flash
>>>>>>> (when combined with background garbage collection), less
>>>>>>> fragmentation
>>>>>>> on some times of access patterns, better re-use of data with
>>>>>>> copy-on-write, better safety on power failures or unexpected breaks
>>>>>>> (such as with USB flash sticks), cheap snapshots and rollbacks, etc.
>>>>>>>
>>>>>>> There are many different strategies for how to put data onto disks
>>>>>>> - no
>>>>>>> one size fits all usage.
>>>>>>
>>>>>> Understood, however it does seem to leave a security exposure.
>>>>>
>>>>> No, it is not a security hole.
>>>>>
>>>>> Quite simply, you can never rely 100% on old data being overwritten by
>>>>> new data, regardless of the filesystem and of the disk type.  For some
>>>>> filesystems it is more likely that data will be overwritten than for
>>>>> others, and for some disk types it is more likely that data will be
>>>>> overwritten.  But you never have any guarantees.
>>>>>
>>>>> So if you really want to make sure that your data cannot be recovered,
>>>>> there is only one way - don't write the data to the disk in the first
>>>>> place.  Use an encrypted filesystem (with a good password/passphrase,
>>>>> obviously) and your data is safe.
>>>>>
>>>>> Encrypted data partitions is so simple in Linux that there is no
>>>>> excuse
>>>>> for not using them if you want safe data.  If you just want to encrypt
>>>>> some few vital files, then make a suitably sized empty file, turn it
>>>>> into a loopback block device, encrypt it, put a filesystem on top
>>>>> of it
>>>>> and mount it at something like /home/<user>/secret.  Anything you save
>>>>> there is safe.  Anything you save elsewhere, you have to assume it can
>>>>> be recovered unless you physically destroy the drive.
>>>>>
>>>>
>>>> I disagree on so many counts that a detailed reply is pointless.
>>>
>>> OK, I suppose.  I can't see that any of my points are controversial or
>>> subjective, but if you don't want to ask for more information or learn
>>> more about how storage systems work, then that's your choice.
>>>
>>
>> Thank you, but if I need more information about how this linux
>> filesystem or that linux filesystem performs certain functions then it
>> will probably be necessary for me to run some tests and/or read the
>> actual code.
>>
>> As for the general theory of storage systems, it is too vast a topic for
>> a usenet discussion.  File deallocation is a complex multi-step process
>> and filesystem implementations that support only a "make file go away"
>> interface will be forced to choose between performance and reliability.
>>   Choosing toward reliability is too potentially costly in terms of
>> performance; as a result, the available interfaces are largely
>> determinative.
>
> My point is that it is impossible for a /filesystem/ to guarantee
> erasure of old data.

If you're talking specific linux filesystems, or linux filesystems in 
general, that may be the case; if you're talking filesystems in the 
wholly general sense, it is not the case.

> In many cases, you could get very close - on a harddisk, if you
> overwrite an existing sector then old data on the sector is gone for
> ever (the idea of having to make multiple passes of random data, ones,
> zeros, etc., is a baseless myth).

Assuming that you verify that the data has actually been overwritten, 
yes; however, read/verify-after-write is costly, so it becomes a 
performance-vs-reliability issue unless there are specific interfaces to 
specify the various operations.

> However, harddisks occasionally
> relocate sectors - you can't control this, and you cannot access the old
> data or erase it without opening the disk physically.  It's a rare
> occurrence - but for the paranoid, less than 100% is not good enough.

If the device lies about what it is doing, all one can do is put it on a 
blacklist and refuse to support the device entirely, or establish it as 
a device to be used for non-critical data only.

When "device" behavior reaches a certain point of independence, it has 
actually become a server and needs to be given no more trust than a 
server accessed over a trusted connection.  It is (imo, at least) 
necessary to be cognizant of whether the hardware designers have stuck 
their nose into your business, and treat those "devices" with 
significant skepticism.

> In other cases, particularly SSDs, then overwriting existing logical
> sectors does not affect the old data directly, and it could be
> recovered.  Other forms of data storage, such as cloud storage, are
> getting more common - and you have even less control there.

If there is no device reliability, the device should not be used for 
critical data.  You wouldn't expect a large business to send its daily 
cash intake to the bank by using some homeless person grabbed off the 
street, and using cloud storage is similar; some may be reliable but 
others not.

> So no filesystem can actually guarantee that it can erase old data - and
> filesystems therefore normally provide no functions to erase old data.

If a courier service routinely grabs homeless people off the street to 
deliver large amounts of cash, one might wisely choose not to use them.

We still disagree about what "no" filesystem can do.  If you mean to 
limit the discussion to "existing linux filesystems" then I might have 
no choice but to concede your point; if you want to talk about what can 
be done, what is actually possible, then we will probably continue to 
disagree on this subject.

> Whether they re-use deleted sectors quickly or slowly is a matter of the
> structure of the file-system, and what is most efficient, and it may
> vary depending on the media (placement to minimise fragmentation and
> head movement is usually best on harddisks, re-using deleted logical
> sectors is most efficient on SSD's because it becomes an automatic TRIM
> - even though the physical sectors will not be reused quickly).
>
>>
>> While encryption does provide the advantage of helping to protect data
>> stored on a device that becomes physically available to the malicious,
>> the advent of "key managers" leads one to consider the possibility that
>> encrypted filesystems may amount to much ado about nothing, and that
>> other avenues of protection may be more effective overall.
>
> That is certainly true - your data is only as safe as your key.  But the
> usual procedure for encrypted partitions is to enter your key (or
> passphrase) at the keyboard when mounting.
>

Yes, and once you have done so, the mounted partition is no more secure 
than a non-encrypted partition; data security has, through your 
supplying the key, been degraded from whatever level of security is 
provided by the encryption algorithm used, to the simple "security" 
provided by the linux protection modes.  To quote you, "for the 
paranoid, less than 100% is not good enough".

I am of the opinion that the linux protection modes inherited from Unix 
are not very "adequate", and the idea of a "superuser" is inherently 
crippled "security".  The whole user/group/world concept along with the 
ability to open files for non-exclusive access is ridiculously 
antiquated and stems from the pre-1970 era when bisync was 
state-of-the-art and the concept of servers as separate asset owners was 
little more than a foggy idea in the minds of a few.  The architecture 
survives to the present basically because it is politically advantageous 
to maintain posix compatability, but personally I would prefer something 
a bit more modern even at the cost of losing compatability.

Sorry for ranting, I'll stop before beginning to spew about whether 
encryption is really useful. <g>

Back to comp.os.linux.development.system | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-16 18:17 -0400
  Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-17 04:19 -0600
    Re: shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-18 22:30 -0400
      Re: shred or scrub Jasen Betts <jasen@xnet.co.nz> - 2014-04-19 07:42 +0000
        Re: shred or scrub Richard Kettlewell <rjk@greenend.org.uk> - 2014-04-19 10:04 +0100
      Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-19 02:15 -0600
      Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-19 23:05 +0100
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-20 02:47 -0600
          Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-20 07:56 -0500
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-21 03:51 -0600
              Re: shred or scrub Jasen Betts <jasen@xnet.co.nz> - 2014-04-21 11:50 +0000
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-21 06:14 -0600
            Re: shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-21 18:44 -0400
          Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-21 13:24 +0100
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-22 04:10 -0600
              Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-22 14:39 +0100
  Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-17 13:15 +0000
    Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-17 09:40 -0500
      Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-18 14:40 +0000
    Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 02:12 -0600
      Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-18 11:49 +0200
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 09:59 -0600
          Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-21 16:14 +0200
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-22 04:22 -0600
              Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-23 00:06 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-23 05:50 -0600
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-24 22:46 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-25 03:57 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-25 19:14 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-26 04:02 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-27 21:26 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 03:27 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 12:17 +0100
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 13:01 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-29 02:50 -0600
                UNIX(*)/ Linux history & system design (was: shred or scrub) Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-05 21:31 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-05 16:02 -0600
                Re: UNIX(*)/ Linux history & system design David Brown <david.brown@hesbynett.no> - 2014-05-06 01:17 +0200
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-06 01:46 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-06 15:09 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-06 23:47 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-07 16:23 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 10:51 -0600
                Re: UNIX(*)/ Linux history & system design Jerry Peters <jerry@example.invalid> - 2014-05-07 20:25 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-08 03:50 -0600
                Re: UNIX(*)/ Linux history & system design Jerry Peters <jerry@example.invalid> - 2014-05-08 20:24 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-09 02:23 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-09 18:36 +0100
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-09 21:24 +0100
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-07 22:01 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-08 03:37 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-08 14:02 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-09 02:56 -0600
                Re: UNIX(*)/ Linux history & system design David Brown <david.brown@hesbynett.no> - 2014-05-07 00:15 +0200
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 00:32 -0600
                Re: UNIX(*)/ Linux history & system design Jorgen Grahn <grahn+nntp@snipabacken.se> - 2014-05-07 08:47 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 10:59 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-06 14:35 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-26 16:30 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-27 05:59 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-27 20:15 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 03:29 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 12:06 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-27 21:41 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 04:03 -0600
                Re: shred or scrub Richard Kettlewell <rjk@greenend.org.uk> - 2014-04-28 16:44 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-28 23:39 +0200
      Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-18 07:37 -0500
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 10:16 -0600
          Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-18 12:01 -0500
      Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-18 14:42 +0000
  Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-17 16:41 +0200

csiph-web