Groups > comp.os.linux.development.system > #649

Re: shred or scrub

From	David Brown <david.brown@hesbynett.no>
Newsgroups	comp.os.linux.development.system
Subject	Re: shred or scrub
Date	2014-04-24 22:46 +0200
Organization	A noiseless patient Spider
Message-ID	<ljbt7r$5bc$1@dont-email.me> (permalink)
References	(4 earlier) <liri6m$tk9$1@dont-email.me> <lj393j$iha$1@dont-email.me> <lj5fs8$49k$1@dont-email.me> <lj6p5h$m0g$1@dont-email.me> <lj89en$q2a$1@dont-email.me>

Show all headers | View raw

On 23/04/14 13:50, crankypuss wrote:
> On 04/22/2014 04:06 PM, David Brown wrote:
>> On 22/04/14 12:22, crankypuss wrote:
>>> On 04/21/2014 08:14 AM, David Brown wrote:
>>>> On 18/04/14 17:59, crankypuss wrote:
>>>>> On 04/18/2014 03:49 AM, David Brown wrote:
>>>>>> On 18/04/14 10:12, crankypuss wrote:
>>>>>>> On 04/17/2014 07:15 AM, Kristof Provost wrote:
>>>>>>>> On 2014-04-16, Bill Cunningham <nospam@nspam.invalid> wrote:
>>>>>>>>>      I am using ext4 on my linux. I'm not quite sure of the
>>>>>>>>> difference in it
>>>>>>>>> and ext3 but anyway; the shred man page says with the ext3
>>>>>>>>> filesystem
>>>>>>>>> shred
>>>>>>>>> cannot be guaranteed to  work.
>>>>>>>>>
>>>>>>>> That's because there's no way to guarantee that the file system
>>>>>>>> will
>>>>>>>> write the new data over the same block as the old data. In fact, in
>>>>>>>> log-structured file systems (like ZFS, but not ext3/4) the file
>>>>>>>> system
>>>>>>>> will deliberately not do this.
>>>>>>>
>>>>>>> That seems very messed up.
>>>>>>
>>>>>> Some filesystems work this way for particular reasons, such as wear
>>>>>> leveling (for SSDs), better distribution of data across the disk or
>>>>>> disks, minimal head movement (for HDs), minimising overwrites in
>>>>>> flash
>>>>>> (when combined with background garbage collection), less
>>>>>> fragmentation
>>>>>> on some times of access patterns, better re-use of data with
>>>>>> copy-on-write, better safety on power failures or unexpected breaks
>>>>>> (such as with USB flash sticks), cheap snapshots and rollbacks, etc.
>>>>>>
>>>>>> There are many different strategies for how to put data onto disks
>>>>>> - no
>>>>>> one size fits all usage.
>>>>>
>>>>> Understood, however it does seem to leave a security exposure.
>>>>
>>>> No, it is not a security hole.
>>>>
>>>> Quite simply, you can never rely 100% on old data being overwritten by
>>>> new data, regardless of the filesystem and of the disk type.  For some
>>>> filesystems it is more likely that data will be overwritten than for
>>>> others, and for some disk types it is more likely that data will be
>>>> overwritten.  But you never have any guarantees.
>>>>
>>>> So if you really want to make sure that your data cannot be recovered,
>>>> there is only one way - don't write the data to the disk in the first
>>>> place.  Use an encrypted filesystem (with a good password/passphrase,
>>>> obviously) and your data is safe.
>>>>
>>>> Encrypted data partitions is so simple in Linux that there is no excuse
>>>> for not using them if you want safe data.  If you just want to encrypt
>>>> some few vital files, then make a suitably sized empty file, turn it
>>>> into a loopback block device, encrypt it, put a filesystem on top of it
>>>> and mount it at something like /home/<user>/secret.  Anything you save
>>>> there is safe.  Anything you save elsewhere, you have to assume it can
>>>> be recovered unless you physically destroy the drive.
>>>>
>>>
>>> I disagree on so many counts that a detailed reply is pointless.
>>
>> OK, I suppose.  I can't see that any of my points are controversial or
>> subjective, but if you don't want to ask for more information or learn
>> more about how storage systems work, then that's your choice.
>>
>
> Thank you, but if I need more information about how this linux
> filesystem or that linux filesystem performs certain functions then it
> will probably be necessary for me to run some tests and/or read the
> actual code.
>
> As for the general theory of storage systems, it is too vast a topic for
> a usenet discussion.  File deallocation is a complex multi-step process
> and filesystem implementations that support only a "make file go away"
> interface will be forced to choose between performance and reliability.
>   Choosing toward reliability is too potentially costly in terms of
> performance; as a result, the available interfaces are largely
> determinative.

My point is that it is impossible for a /filesystem/ to guarantee 
erasure of old data.

In many cases, you could get very close - on a harddisk, if you 
overwrite an existing sector then old data on the sector is gone for 
ever (the idea of having to make multiple passes of random data, ones, 
zeros, etc., is a baseless myth).  However, harddisks occasionally 
relocate sectors - you can't control this, and you cannot access the old 
data or erase it without opening the disk physically.  It's a rare 
occurrence - but for the paranoid, less than 100% is not good enough.

In other cases, particularly SSDs, then overwriting existing logical 
sectors does not affect the old data directly, and it could be 
recovered.  Other forms of data storage, such as cloud storage, are 
getting more common - and you have even less control there.

So no filesystem can actually guarantee that it can erase old data - and 
filesystems therefore normally provide no functions to erase old data. 
Whether they re-use deleted sectors quickly or slowly is a matter of the 
structure of the file-system, and what is most efficient, and it may 
vary depending on the media (placement to minimise fragmentation and 
head movement is usually best on harddisks, re-using deleted logical 
sectors is most efficient on SSD's because it becomes an automatic TRIM 
- even though the physical sectors will not be reused quickly).

>
> While encryption does provide the advantage of helping to protect data
> stored on a device that becomes physically available to the malicious,
> the advent of "key managers" leads one to consider the possibility that
> encrypted filesystems may amount to much ado about nothing, and that
> other avenues of protection may be more effective overall.

That is certainly true - your data is only as safe as your key.  But the 
usual procedure for encrypted partitions is to enter your key (or 
passphrase) at the keyboard when mounting.

Back to comp.os.linux.development.system | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-16 18:17 -0400
  Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-17 04:19 -0600
    Re: shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-18 22:30 -0400
      Re: shred or scrub Jasen Betts <jasen@xnet.co.nz> - 2014-04-19 07:42 +0000
        Re: shred or scrub Richard Kettlewell <rjk@greenend.org.uk> - 2014-04-19 10:04 +0100
      Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-19 02:15 -0600
      Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-19 23:05 +0100
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-20 02:47 -0600
          Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-20 07:56 -0500
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-21 03:51 -0600
              Re: shred or scrub Jasen Betts <jasen@xnet.co.nz> - 2014-04-21 11:50 +0000
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-21 06:14 -0600
            Re: shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-21 18:44 -0400
          Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-21 13:24 +0100
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-22 04:10 -0600
              Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-22 14:39 +0100
  Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-17 13:15 +0000
    Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-17 09:40 -0500
      Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-18 14:40 +0000
    Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 02:12 -0600
      Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-18 11:49 +0200
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 09:59 -0600
          Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-21 16:14 +0200
            Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-22 04:22 -0600
              Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-23 00:06 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-23 05:50 -0600
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-24 22:46 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-25 03:57 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-25 19:14 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-26 04:02 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-27 21:26 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 03:27 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 12:17 +0100
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 13:01 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-29 02:50 -0600
                UNIX(*)/ Linux history & system design (was: shred or scrub) Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-05 21:31 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-05 16:02 -0600
                Re: UNIX(*)/ Linux history & system design David Brown <david.brown@hesbynett.no> - 2014-05-06 01:17 +0200
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-06 01:46 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-06 15:09 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-06 23:47 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-07 16:23 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 10:51 -0600
                Re: UNIX(*)/ Linux history & system design Jerry Peters <jerry@example.invalid> - 2014-05-07 20:25 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-08 03:50 -0600
                Re: UNIX(*)/ Linux history & system design Jerry Peters <jerry@example.invalid> - 2014-05-08 20:24 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-09 02:23 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-09 18:36 +0100
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-09 21:24 +0100
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-07 22:01 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-08 03:37 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-08 14:02 +0100
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-09 02:56 -0600
                Re: UNIX(*)/ Linux history & system design David Brown <david.brown@hesbynett.no> - 2014-05-07 00:15 +0200
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 00:32 -0600
                Re: UNIX(*)/ Linux history & system design Jorgen Grahn <grahn+nntp@snipabacken.se> - 2014-05-07 08:47 +0000
                Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 10:59 -0600
                Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-06 14:35 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-26 16:30 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-27 05:59 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-27 20:15 +0100
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 03:29 -0600
                Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 12:06 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-27 21:41 +0200
                Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 04:03 -0600
                Re: shred or scrub Richard Kettlewell <rjk@greenend.org.uk> - 2014-04-28 16:44 +0100
                Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-28 23:39 +0200
      Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-18 07:37 -0500
        Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 10:16 -0600
          Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-18 12:01 -0500
      Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-18 14:42 +0000
  Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-17 16:41 +0200

csiph-web