Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.os.linux.development.system > #618 > unrolled thread
| Started by | "Bill Cunningham" <nospam@nspam.invalid> |
|---|---|
| First post | 2014-04-16 18:17 -0400 |
| Last post | 2014-04-17 16:41 +0200 |
| Articles | 20 on this page of 72 — 10 participants |
Back to article view | Back to comp.os.linux.development.system
shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-16 18:17 -0400
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-17 04:19 -0600
Re: shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-18 22:30 -0400
Re: shred or scrub Jasen Betts <jasen@xnet.co.nz> - 2014-04-19 07:42 +0000
Re: shred or scrub Richard Kettlewell <rjk@greenend.org.uk> - 2014-04-19 10:04 +0100
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-19 02:15 -0600
Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-19 23:05 +0100
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-20 02:47 -0600
Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-20 07:56 -0500
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-21 03:51 -0600
Re: shred or scrub Jasen Betts <jasen@xnet.co.nz> - 2014-04-21 11:50 +0000
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-21 06:14 -0600
Re: shred or scrub "Bill Cunningham" <nospam@nspam.invalid> - 2014-04-21 18:44 -0400
Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-21 13:24 +0100
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-22 04:10 -0600
Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-22 14:39 +0100
Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-17 13:15 +0000
Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-17 09:40 -0500
Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-18 14:40 +0000
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 02:12 -0600
Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-18 11:49 +0200
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 09:59 -0600
Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-21 16:14 +0200
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-22 04:22 -0600
Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-23 00:06 +0200
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-23 05:50 -0600
Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-24 22:46 +0200
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-25 03:57 -0600
Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-25 19:14 +0100
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-26 04:02 -0600
Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-27 21:26 +0100
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 03:27 -0600
Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 12:17 +0100
Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 13:01 +0100
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-29 02:50 -0600
UNIX(*)/ Linux history & system design (was: shred or scrub) Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-05 21:31 +0100
Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-05 16:02 -0600
Re: UNIX(*)/ Linux history & system design David Brown <david.brown@hesbynett.no> - 2014-05-06 01:17 +0200
Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-06 01:46 -0600
Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-06 15:09 +0100
Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-06 23:47 -0600
Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-07 16:23 +0100
Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 10:51 -0600
Re: UNIX(*)/ Linux history & system design Jerry Peters <jerry@example.invalid> - 2014-05-07 20:25 +0000
Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-08 03:50 -0600
Re: UNIX(*)/ Linux history & system design Jerry Peters <jerry@example.invalid> - 2014-05-08 20:24 +0000
Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-09 02:23 -0600
Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-09 18:36 +0100
Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-09 21:24 +0100
Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-07 22:01 +0100
Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-08 03:37 -0600
Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-08 14:02 +0100
Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-09 02:56 -0600
Re: UNIX(*)/ Linux history & system design David Brown <david.brown@hesbynett.no> - 2014-05-07 00:15 +0200
Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 00:32 -0600
Re: UNIX(*)/ Linux history & system design Jorgen Grahn <grahn+nntp@snipabacken.se> - 2014-05-07 08:47 +0000
Re: UNIX(*)/ Linux history & system design crankypuss <crankypuss@nomail.invalid> - 2014-05-07 10:59 -0600
Re: UNIX(*)/ Linux history & system design Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-05-06 14:35 +0100
Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-26 16:30 +0200
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-27 05:59 -0600
Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-27 20:15 +0100
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 03:29 -0600
Re: shred or scrub Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2014-04-28 12:06 +0100
Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-27 21:41 +0200
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-28 04:03 -0600
Re: shred or scrub Richard Kettlewell <rjk@greenend.org.uk> - 2014-04-28 16:44 +0100
Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-28 23:39 +0200
Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-18 07:37 -0500
Re: shred or scrub crankypuss <crankypuss@nomail.invalid> - 2014-04-18 10:16 -0600
Re: shred or scrub John Hasler <jhasler@newsguy.com> - 2014-04-18 12:01 -0500
Re: shred or scrub Kristof Provost <kristof@codepro.be> - 2014-04-18 14:42 +0000
Re: shred or scrub David Brown <david.brown@hesbynett.no> - 2014-04-17 16:41 +0200
Page 2 of 4 — ← Prev page 1 [2] 3 4 Next page →
| From | David Brown <david.brown@hesbynett.no> |
|---|---|
| Date | 2014-04-18 11:49 +0200 |
| Message-ID | <liqsf7$6sg$1@dont-email.me> |
| In reply to | #623 |
On 18/04/14 10:12, crankypuss wrote: > On 04/17/2014 07:15 AM, Kristof Provost wrote: >> On 2014-04-16, Bill Cunningham <nospam@nspam.invalid> wrote: >>> I am using ext4 on my linux. I'm not quite sure of the >>> difference in it >>> and ext3 but anyway; the shred man page says with the ext3 filesystem >>> shred >>> cannot be guaranteed to work. >>> >> That's because there's no way to guarantee that the file system will >> write the new data over the same block as the old data. In fact, in >> log-structured file systems (like ZFS, but not ext3/4) the file system >> will deliberately not do this. > > That seems very messed up. Some filesystems work this way for particular reasons, such as wear leveling (for SSDs), better distribution of data across the disk or disks, minimal head movement (for HDs), minimising overwrites in flash (when combined with background garbage collection), less fragmentation on some times of access patterns, better re-use of data with copy-on-write, better safety on power failures or unexpected breaks (such as with USB flash sticks), cheap snapshots and rollbacks, etc. There are many different strategies for how to put data onto disks - no one size fits all usage.
[toc] | [prev] | [next] | [standalone]
| From | crankypuss <crankypuss@nomail.invalid> |
|---|---|
| Date | 2014-04-18 09:59 -0600 |
| Message-ID | <liri6m$tk9$1@dont-email.me> |
| In reply to | #624 |
On 04/18/2014 03:49 AM, David Brown wrote: > On 18/04/14 10:12, crankypuss wrote: >> On 04/17/2014 07:15 AM, Kristof Provost wrote: >>> On 2014-04-16, Bill Cunningham <nospam@nspam.invalid> wrote: >>>> I am using ext4 on my linux. I'm not quite sure of the >>>> difference in it >>>> and ext3 but anyway; the shred man page says with the ext3 filesystem >>>> shred >>>> cannot be guaranteed to work. >>>> >>> That's because there's no way to guarantee that the file system will >>> write the new data over the same block as the old data. In fact, in >>> log-structured file systems (like ZFS, but not ext3/4) the file system >>> will deliberately not do this. >> >> That seems very messed up. > > Some filesystems work this way for particular reasons, such as wear > leveling (for SSDs), better distribution of data across the disk or > disks, minimal head movement (for HDs), minimising overwrites in flash > (when combined with background garbage collection), less fragmentation > on some times of access patterns, better re-use of data with > copy-on-write, better safety on power failures or unexpected breaks > (such as with USB flash sticks), cheap snapshots and rollbacks, etc. > > There are many different strategies for how to put data onto disks - no > one size fits all usage. Understood, however it does seem to leave a security exposure.
[toc] | [prev] | [next] | [standalone]
| From | David Brown <david.brown@hesbynett.no> |
|---|---|
| Date | 2014-04-21 16:14 +0200 |
| Message-ID | <lj393j$iha$1@dont-email.me> |
| In reply to | #628 |
On 18/04/14 17:59, crankypuss wrote: > On 04/18/2014 03:49 AM, David Brown wrote: >> On 18/04/14 10:12, crankypuss wrote: >>> On 04/17/2014 07:15 AM, Kristof Provost wrote: >>>> On 2014-04-16, Bill Cunningham <nospam@nspam.invalid> wrote: >>>>> I am using ext4 on my linux. I'm not quite sure of the >>>>> difference in it >>>>> and ext3 but anyway; the shred man page says with the ext3 filesystem >>>>> shred >>>>> cannot be guaranteed to work. >>>>> >>>> That's because there's no way to guarantee that the file system will >>>> write the new data over the same block as the old data. In fact, in >>>> log-structured file systems (like ZFS, but not ext3/4) the file system >>>> will deliberately not do this. >>> >>> That seems very messed up. >> >> Some filesystems work this way for particular reasons, such as wear >> leveling (for SSDs), better distribution of data across the disk or >> disks, minimal head movement (for HDs), minimising overwrites in flash >> (when combined with background garbage collection), less fragmentation >> on some times of access patterns, better re-use of data with >> copy-on-write, better safety on power failures or unexpected breaks >> (such as with USB flash sticks), cheap snapshots and rollbacks, etc. >> >> There are many different strategies for how to put data onto disks - no >> one size fits all usage. > > Understood, however it does seem to leave a security exposure. No, it is not a security hole. Quite simply, you can never rely 100% on old data being overwritten by new data, regardless of the filesystem and of the disk type. For some filesystems it is more likely that data will be overwritten than for others, and for some disk types it is more likely that data will be overwritten. But you never have any guarantees. So if you really want to make sure that your data cannot be recovered, there is only one way - don't write the data to the disk in the first place. Use an encrypted filesystem (with a good password/passphrase, obviously) and your data is safe. Encrypted data partitions is so simple in Linux that there is no excuse for not using them if you want safe data. If you just want to encrypt some few vital files, then make a suitably sized empty file, turn it into a loopback block device, encrypt it, put a filesystem on top of it and mount it at something like /home/<user>/secret. Anything you save there is safe. Anything you save elsewhere, you have to assume it can be recovered unless you physically destroy the drive.
[toc] | [prev] | [next] | [standalone]
| From | crankypuss <crankypuss@nomail.invalid> |
|---|---|
| Date | 2014-04-22 04:22 -0600 |
| Message-ID | <lj5fs8$49k$1@dont-email.me> |
| In reply to | #642 |
On 04/21/2014 08:14 AM, David Brown wrote: > On 18/04/14 17:59, crankypuss wrote: >> On 04/18/2014 03:49 AM, David Brown wrote: >>> On 18/04/14 10:12, crankypuss wrote: >>>> On 04/17/2014 07:15 AM, Kristof Provost wrote: >>>>> On 2014-04-16, Bill Cunningham <nospam@nspam.invalid> wrote: >>>>>> I am using ext4 on my linux. I'm not quite sure of the >>>>>> difference in it >>>>>> and ext3 but anyway; the shred man page says with the ext3 filesystem >>>>>> shred >>>>>> cannot be guaranteed to work. >>>>>> >>>>> That's because there's no way to guarantee that the file system will >>>>> write the new data over the same block as the old data. In fact, in >>>>> log-structured file systems (like ZFS, but not ext3/4) the file system >>>>> will deliberately not do this. >>>> >>>> That seems very messed up. >>> >>> Some filesystems work this way for particular reasons, such as wear >>> leveling (for SSDs), better distribution of data across the disk or >>> disks, minimal head movement (for HDs), minimising overwrites in flash >>> (when combined with background garbage collection), less fragmentation >>> on some times of access patterns, better re-use of data with >>> copy-on-write, better safety on power failures or unexpected breaks >>> (such as with USB flash sticks), cheap snapshots and rollbacks, etc. >>> >>> There are many different strategies for how to put data onto disks - no >>> one size fits all usage. >> >> Understood, however it does seem to leave a security exposure. > > No, it is not a security hole. > > Quite simply, you can never rely 100% on old data being overwritten by > new data, regardless of the filesystem and of the disk type. For some > filesystems it is more likely that data will be overwritten than for > others, and for some disk types it is more likely that data will be > overwritten. But you never have any guarantees. > > So if you really want to make sure that your data cannot be recovered, > there is only one way - don't write the data to the disk in the first > place. Use an encrypted filesystem (with a good password/passphrase, > obviously) and your data is safe. > > Encrypted data partitions is so simple in Linux that there is no excuse > for not using them if you want safe data. If you just want to encrypt > some few vital files, then make a suitably sized empty file, turn it > into a loopback block device, encrypt it, put a filesystem on top of it > and mount it at something like /home/<user>/secret. Anything you save > there is safe. Anything you save elsewhere, you have to assume it can > be recovered unless you physically destroy the drive. > I disagree on so many counts that a detailed reply is pointless.
[toc] | [prev] | [next] | [standalone]
| From | David Brown <david.brown@hesbynett.no> |
|---|---|
| Date | 2014-04-23 00:06 +0200 |
| Message-ID | <lj6p5h$m0g$1@dont-email.me> |
| In reply to | #645 |
On 22/04/14 12:22, crankypuss wrote: > On 04/21/2014 08:14 AM, David Brown wrote: >> On 18/04/14 17:59, crankypuss wrote: >>> On 04/18/2014 03:49 AM, David Brown wrote: >>>> On 18/04/14 10:12, crankypuss wrote: >>>>> On 04/17/2014 07:15 AM, Kristof Provost wrote: >>>>>> On 2014-04-16, Bill Cunningham <nospam@nspam.invalid> wrote: >>>>>>> I am using ext4 on my linux. I'm not quite sure of the >>>>>>> difference in it >>>>>>> and ext3 but anyway; the shred man page says with the ext3 >>>>>>> filesystem >>>>>>> shred >>>>>>> cannot be guaranteed to work. >>>>>>> >>>>>> That's because there's no way to guarantee that the file system will >>>>>> write the new data over the same block as the old data. In fact, in >>>>>> log-structured file systems (like ZFS, but not ext3/4) the file >>>>>> system >>>>>> will deliberately not do this. >>>>> >>>>> That seems very messed up. >>>> >>>> Some filesystems work this way for particular reasons, such as wear >>>> leveling (for SSDs), better distribution of data across the disk or >>>> disks, minimal head movement (for HDs), minimising overwrites in flash >>>> (when combined with background garbage collection), less fragmentation >>>> on some times of access patterns, better re-use of data with >>>> copy-on-write, better safety on power failures or unexpected breaks >>>> (such as with USB flash sticks), cheap snapshots and rollbacks, etc. >>>> >>>> There are many different strategies for how to put data onto disks - no >>>> one size fits all usage. >>> >>> Understood, however it does seem to leave a security exposure. >> >> No, it is not a security hole. >> >> Quite simply, you can never rely 100% on old data being overwritten by >> new data, regardless of the filesystem and of the disk type. For some >> filesystems it is more likely that data will be overwritten than for >> others, and for some disk types it is more likely that data will be >> overwritten. But you never have any guarantees. >> >> So if you really want to make sure that your data cannot be recovered, >> there is only one way - don't write the data to the disk in the first >> place. Use an encrypted filesystem (with a good password/passphrase, >> obviously) and your data is safe. >> >> Encrypted data partitions is so simple in Linux that there is no excuse >> for not using them if you want safe data. If you just want to encrypt >> some few vital files, then make a suitably sized empty file, turn it >> into a loopback block device, encrypt it, put a filesystem on top of it >> and mount it at something like /home/<user>/secret. Anything you save >> there is safe. Anything you save elsewhere, you have to assume it can >> be recovered unless you physically destroy the drive. >> > > I disagree on so many counts that a detailed reply is pointless. OK, I suppose. I can't see that any of my points are controversial or subjective, but if you don't want to ask for more information or learn more about how storage systems work, then that's your choice.
[toc] | [prev] | [next] | [standalone]
| From | crankypuss <crankypuss@nomail.invalid> |
|---|---|
| Date | 2014-04-23 05:50 -0600 |
| Message-ID | <lj89en$q2a$1@dont-email.me> |
| In reply to | #647 |
On 04/22/2014 04:06 PM, David Brown wrote: > On 22/04/14 12:22, crankypuss wrote: >> On 04/21/2014 08:14 AM, David Brown wrote: >>> On 18/04/14 17:59, crankypuss wrote: >>>> On 04/18/2014 03:49 AM, David Brown wrote: >>>>> On 18/04/14 10:12, crankypuss wrote: >>>>>> On 04/17/2014 07:15 AM, Kristof Provost wrote: >>>>>>> On 2014-04-16, Bill Cunningham <nospam@nspam.invalid> wrote: >>>>>>>> I am using ext4 on my linux. I'm not quite sure of the >>>>>>>> difference in it >>>>>>>> and ext3 but anyway; the shred man page says with the ext3 >>>>>>>> filesystem >>>>>>>> shred >>>>>>>> cannot be guaranteed to work. >>>>>>>> >>>>>>> That's because there's no way to guarantee that the file system will >>>>>>> write the new data over the same block as the old data. In fact, in >>>>>>> log-structured file systems (like ZFS, but not ext3/4) the file >>>>>>> system >>>>>>> will deliberately not do this. >>>>>> >>>>>> That seems very messed up. >>>>> >>>>> Some filesystems work this way for particular reasons, such as wear >>>>> leveling (for SSDs), better distribution of data across the disk or >>>>> disks, minimal head movement (for HDs), minimising overwrites in flash >>>>> (when combined with background garbage collection), less fragmentation >>>>> on some times of access patterns, better re-use of data with >>>>> copy-on-write, better safety on power failures or unexpected breaks >>>>> (such as with USB flash sticks), cheap snapshots and rollbacks, etc. >>>>> >>>>> There are many different strategies for how to put data onto disks >>>>> - no >>>>> one size fits all usage. >>>> >>>> Understood, however it does seem to leave a security exposure. >>> >>> No, it is not a security hole. >>> >>> Quite simply, you can never rely 100% on old data being overwritten by >>> new data, regardless of the filesystem and of the disk type. For some >>> filesystems it is more likely that data will be overwritten than for >>> others, and for some disk types it is more likely that data will be >>> overwritten. But you never have any guarantees. >>> >>> So if you really want to make sure that your data cannot be recovered, >>> there is only one way - don't write the data to the disk in the first >>> place. Use an encrypted filesystem (with a good password/passphrase, >>> obviously) and your data is safe. >>> >>> Encrypted data partitions is so simple in Linux that there is no excuse >>> for not using them if you want safe data. If you just want to encrypt >>> some few vital files, then make a suitably sized empty file, turn it >>> into a loopback block device, encrypt it, put a filesystem on top of it >>> and mount it at something like /home/<user>/secret. Anything you save >>> there is safe. Anything you save elsewhere, you have to assume it can >>> be recovered unless you physically destroy the drive. >>> >> >> I disagree on so many counts that a detailed reply is pointless. > > OK, I suppose. I can't see that any of my points are controversial or > subjective, but if you don't want to ask for more information or learn > more about how storage systems work, then that's your choice. > Thank you, but if I need more information about how this linux filesystem or that linux filesystem performs certain functions then it will probably be necessary for me to run some tests and/or read the actual code. As for the general theory of storage systems, it is too vast a topic for a usenet discussion. File deallocation is a complex multi-step process and filesystem implementations that support only a "make file go away" interface will be forced to choose between performance and reliability. Choosing toward reliability is too potentially costly in terms of performance; as a result, the available interfaces are largely determinative. While encryption does provide the advantage of helping to protect data stored on a device that becomes physically available to the malicious, the advent of "key managers" leads one to consider the possibility that encrypted filesystems may amount to much ado about nothing, and that other avenues of protection may be more effective overall.
[toc] | [prev] | [next] | [standalone]
| From | David Brown <david.brown@hesbynett.no> |
|---|---|
| Date | 2014-04-24 22:46 +0200 |
| Message-ID | <ljbt7r$5bc$1@dont-email.me> |
| In reply to | #648 |
On 23/04/14 13:50, crankypuss wrote: > On 04/22/2014 04:06 PM, David Brown wrote: >> On 22/04/14 12:22, crankypuss wrote: >>> On 04/21/2014 08:14 AM, David Brown wrote: >>>> On 18/04/14 17:59, crankypuss wrote: >>>>> On 04/18/2014 03:49 AM, David Brown wrote: >>>>>> On 18/04/14 10:12, crankypuss wrote: >>>>>>> On 04/17/2014 07:15 AM, Kristof Provost wrote: >>>>>>>> On 2014-04-16, Bill Cunningham <nospam@nspam.invalid> wrote: >>>>>>>>> I am using ext4 on my linux. I'm not quite sure of the >>>>>>>>> difference in it >>>>>>>>> and ext3 but anyway; the shred man page says with the ext3 >>>>>>>>> filesystem >>>>>>>>> shred >>>>>>>>> cannot be guaranteed to work. >>>>>>>>> >>>>>>>> That's because there's no way to guarantee that the file system >>>>>>>> will >>>>>>>> write the new data over the same block as the old data. In fact, in >>>>>>>> log-structured file systems (like ZFS, but not ext3/4) the file >>>>>>>> system >>>>>>>> will deliberately not do this. >>>>>>> >>>>>>> That seems very messed up. >>>>>> >>>>>> Some filesystems work this way for particular reasons, such as wear >>>>>> leveling (for SSDs), better distribution of data across the disk or >>>>>> disks, minimal head movement (for HDs), minimising overwrites in >>>>>> flash >>>>>> (when combined with background garbage collection), less >>>>>> fragmentation >>>>>> on some times of access patterns, better re-use of data with >>>>>> copy-on-write, better safety on power failures or unexpected breaks >>>>>> (such as with USB flash sticks), cheap snapshots and rollbacks, etc. >>>>>> >>>>>> There are many different strategies for how to put data onto disks >>>>>> - no >>>>>> one size fits all usage. >>>>> >>>>> Understood, however it does seem to leave a security exposure. >>>> >>>> No, it is not a security hole. >>>> >>>> Quite simply, you can never rely 100% on old data being overwritten by >>>> new data, regardless of the filesystem and of the disk type. For some >>>> filesystems it is more likely that data will be overwritten than for >>>> others, and for some disk types it is more likely that data will be >>>> overwritten. But you never have any guarantees. >>>> >>>> So if you really want to make sure that your data cannot be recovered, >>>> there is only one way - don't write the data to the disk in the first >>>> place. Use an encrypted filesystem (with a good password/passphrase, >>>> obviously) and your data is safe. >>>> >>>> Encrypted data partitions is so simple in Linux that there is no excuse >>>> for not using them if you want safe data. If you just want to encrypt >>>> some few vital files, then make a suitably sized empty file, turn it >>>> into a loopback block device, encrypt it, put a filesystem on top of it >>>> and mount it at something like /home/<user>/secret. Anything you save >>>> there is safe. Anything you save elsewhere, you have to assume it can >>>> be recovered unless you physically destroy the drive. >>>> >>> >>> I disagree on so many counts that a detailed reply is pointless. >> >> OK, I suppose. I can't see that any of my points are controversial or >> subjective, but if you don't want to ask for more information or learn >> more about how storage systems work, then that's your choice. >> > > Thank you, but if I need more information about how this linux > filesystem or that linux filesystem performs certain functions then it > will probably be necessary for me to run some tests and/or read the > actual code. > > As for the general theory of storage systems, it is too vast a topic for > a usenet discussion. File deallocation is a complex multi-step process > and filesystem implementations that support only a "make file go away" > interface will be forced to choose between performance and reliability. > Choosing toward reliability is too potentially costly in terms of > performance; as a result, the available interfaces are largely > determinative. My point is that it is impossible for a /filesystem/ to guarantee erasure of old data. In many cases, you could get very close - on a harddisk, if you overwrite an existing sector then old data on the sector is gone for ever (the idea of having to make multiple passes of random data, ones, zeros, etc., is a baseless myth). However, harddisks occasionally relocate sectors - you can't control this, and you cannot access the old data or erase it without opening the disk physically. It's a rare occurrence - but for the paranoid, less than 100% is not good enough. In other cases, particularly SSDs, then overwriting existing logical sectors does not affect the old data directly, and it could be recovered. Other forms of data storage, such as cloud storage, are getting more common - and you have even less control there. So no filesystem can actually guarantee that it can erase old data - and filesystems therefore normally provide no functions to erase old data. Whether they re-use deleted sectors quickly or slowly is a matter of the structure of the file-system, and what is most efficient, and it may vary depending on the media (placement to minimise fragmentation and head movement is usually best on harddisks, re-using deleted logical sectors is most efficient on SSD's because it becomes an automatic TRIM - even though the physical sectors will not be reused quickly). > > While encryption does provide the advantage of helping to protect data > stored on a device that becomes physically available to the malicious, > the advent of "key managers" leads one to consider the possibility that > encrypted filesystems may amount to much ado about nothing, and that > other avenues of protection may be more effective overall. That is certainly true - your data is only as safe as your key. But the usual procedure for encrypted partitions is to enter your key (or passphrase) at the keyboard when mounting.
[toc] | [prev] | [next] | [standalone]
| From | crankypuss <crankypuss@nomail.invalid> |
|---|---|
| Date | 2014-04-25 03:57 -0600 |
| Message-ID | <ljdbhe$ol8$1@dont-email.me> |
| In reply to | #649 |
On 04/24/2014 02:46 PM, David Brown wrote: > On 23/04/14 13:50, crankypuss wrote: >> On 04/22/2014 04:06 PM, David Brown wrote: >>> On 22/04/14 12:22, crankypuss wrote: >>>> On 04/21/2014 08:14 AM, David Brown wrote: >>>>> On 18/04/14 17:59, crankypuss wrote: >>>>>> On 04/18/2014 03:49 AM, David Brown wrote: >>>>>>> On 18/04/14 10:12, crankypuss wrote: >>>>>>>> On 04/17/2014 07:15 AM, Kristof Provost wrote: >>>>>>>>> On 2014-04-16, Bill Cunningham <nospam@nspam.invalid> wrote: >>>>>>>>>> I am using ext4 on my linux. I'm not quite sure of the >>>>>>>>>> difference in it >>>>>>>>>> and ext3 but anyway; the shred man page says with the ext3 >>>>>>>>>> filesystem >>>>>>>>>> shred >>>>>>>>>> cannot be guaranteed to work. >>>>>>>>>> >>>>>>>>> That's because there's no way to guarantee that the file system >>>>>>>>> will >>>>>>>>> write the new data over the same block as the old data. In >>>>>>>>> fact, in >>>>>>>>> log-structured file systems (like ZFS, but not ext3/4) the file >>>>>>>>> system >>>>>>>>> will deliberately not do this. >>>>>>>> >>>>>>>> That seems very messed up. >>>>>>> >>>>>>> Some filesystems work this way for particular reasons, such as wear >>>>>>> leveling (for SSDs), better distribution of data across the disk or >>>>>>> disks, minimal head movement (for HDs), minimising overwrites in >>>>>>> flash >>>>>>> (when combined with background garbage collection), less >>>>>>> fragmentation >>>>>>> on some times of access patterns, better re-use of data with >>>>>>> copy-on-write, better safety on power failures or unexpected breaks >>>>>>> (such as with USB flash sticks), cheap snapshots and rollbacks, etc. >>>>>>> >>>>>>> There are many different strategies for how to put data onto disks >>>>>>> - no >>>>>>> one size fits all usage. >>>>>> >>>>>> Understood, however it does seem to leave a security exposure. >>>>> >>>>> No, it is not a security hole. >>>>> >>>>> Quite simply, you can never rely 100% on old data being overwritten by >>>>> new data, regardless of the filesystem and of the disk type. For some >>>>> filesystems it is more likely that data will be overwritten than for >>>>> others, and for some disk types it is more likely that data will be >>>>> overwritten. But you never have any guarantees. >>>>> >>>>> So if you really want to make sure that your data cannot be recovered, >>>>> there is only one way - don't write the data to the disk in the first >>>>> place. Use an encrypted filesystem (with a good password/passphrase, >>>>> obviously) and your data is safe. >>>>> >>>>> Encrypted data partitions is so simple in Linux that there is no >>>>> excuse >>>>> for not using them if you want safe data. If you just want to encrypt >>>>> some few vital files, then make a suitably sized empty file, turn it >>>>> into a loopback block device, encrypt it, put a filesystem on top >>>>> of it >>>>> and mount it at something like /home/<user>/secret. Anything you save >>>>> there is safe. Anything you save elsewhere, you have to assume it can >>>>> be recovered unless you physically destroy the drive. >>>>> >>>> >>>> I disagree on so many counts that a detailed reply is pointless. >>> >>> OK, I suppose. I can't see that any of my points are controversial or >>> subjective, but if you don't want to ask for more information or learn >>> more about how storage systems work, then that's your choice. >>> >> >> Thank you, but if I need more information about how this linux >> filesystem or that linux filesystem performs certain functions then it >> will probably be necessary for me to run some tests and/or read the >> actual code. >> >> As for the general theory of storage systems, it is too vast a topic for >> a usenet discussion. File deallocation is a complex multi-step process >> and filesystem implementations that support only a "make file go away" >> interface will be forced to choose between performance and reliability. >> Choosing toward reliability is too potentially costly in terms of >> performance; as a result, the available interfaces are largely >> determinative. > > My point is that it is impossible for a /filesystem/ to guarantee > erasure of old data. If you're talking specific linux filesystems, or linux filesystems in general, that may be the case; if you're talking filesystems in the wholly general sense, it is not the case. > In many cases, you could get very close - on a harddisk, if you > overwrite an existing sector then old data on the sector is gone for > ever (the idea of having to make multiple passes of random data, ones, > zeros, etc., is a baseless myth). Assuming that you verify that the data has actually been overwritten, yes; however, read/verify-after-write is costly, so it becomes a performance-vs-reliability issue unless there are specific interfaces to specify the various operations. > However, harddisks occasionally > relocate sectors - you can't control this, and you cannot access the old > data or erase it without opening the disk physically. It's a rare > occurrence - but for the paranoid, less than 100% is not good enough. If the device lies about what it is doing, all one can do is put it on a blacklist and refuse to support the device entirely, or establish it as a device to be used for non-critical data only. When "device" behavior reaches a certain point of independence, it has actually become a server and needs to be given no more trust than a server accessed over a trusted connection. It is (imo, at least) necessary to be cognizant of whether the hardware designers have stuck their nose into your business, and treat those "devices" with significant skepticism. > In other cases, particularly SSDs, then overwriting existing logical > sectors does not affect the old data directly, and it could be > recovered. Other forms of data storage, such as cloud storage, are > getting more common - and you have even less control there. If there is no device reliability, the device should not be used for critical data. You wouldn't expect a large business to send its daily cash intake to the bank by using some homeless person grabbed off the street, and using cloud storage is similar; some may be reliable but others not. > So no filesystem can actually guarantee that it can erase old data - and > filesystems therefore normally provide no functions to erase old data. If a courier service routinely grabs homeless people off the street to deliver large amounts of cash, one might wisely choose not to use them. We still disagree about what "no" filesystem can do. If you mean to limit the discussion to "existing linux filesystems" then I might have no choice but to concede your point; if you want to talk about what can be done, what is actually possible, then we will probably continue to disagree on this subject. > Whether they re-use deleted sectors quickly or slowly is a matter of the > structure of the file-system, and what is most efficient, and it may > vary depending on the media (placement to minimise fragmentation and > head movement is usually best on harddisks, re-using deleted logical > sectors is most efficient on SSD's because it becomes an automatic TRIM > - even though the physical sectors will not be reused quickly). > >> >> While encryption does provide the advantage of helping to protect data >> stored on a device that becomes physically available to the malicious, >> the advent of "key managers" leads one to consider the possibility that >> encrypted filesystems may amount to much ado about nothing, and that >> other avenues of protection may be more effective overall. > > That is certainly true - your data is only as safe as your key. But the > usual procedure for encrypted partitions is to enter your key (or > passphrase) at the keyboard when mounting. > Yes, and once you have done so, the mounted partition is no more secure than a non-encrypted partition; data security has, through your supplying the key, been degraded from whatever level of security is provided by the encryption algorithm used, to the simple "security" provided by the linux protection modes. To quote you, "for the paranoid, less than 100% is not good enough". I am of the opinion that the linux protection modes inherited from Unix are not very "adequate", and the idea of a "superuser" is inherently crippled "security". The whole user/group/world concept along with the ability to open files for non-exclusive access is ridiculously antiquated and stems from the pre-1970 era when bisync was state-of-the-art and the concept of servers as separate asset owners was little more than a foggy idea in the minds of a few. The architecture survives to the present basically because it is politically advantageous to maintain posix compatability, but personally I would prefer something a bit more modern even at the cost of losing compatability. Sorry for ranting, I'll stop before beginning to spew about whether encryption is really useful. <g>
[toc] | [prev] | [next] | [standalone]
| From | Rainer Weikusat <rweikusat@mobileactivedefense.com> |
|---|---|
| Date | 2014-04-25 19:14 +0100 |
| Message-ID | <87k3adxomn.fsf@sable.mobileactivedefense.com> |
| In reply to | #650 |
crankypuss <crankypuss@nomail.invalid> writes:
> On 04/24/2014 02:46 PM, David Brown wrote:
[...]
>> My point is that it is impossible for a /filesystem/ to guarantee
>> erasure of old data.
>
> If you're talking specific linux filesystems, or linux filesystems in
> general, that may be the case; if you're talking filesystems in the
> wholly general sense, it is not the case.
>
>> In many cases, you could get very close - on a harddisk, if you
>> overwrite an existing sector then old data on the sector is gone for
>> ever (the idea of having to make multiple passes of random data, ones,
>> zeros, etc., is a baseless myth).
>
> Assuming that you verify that the data has actually been overwritten,
> yes; however, read/verify-after-write is costly, so it becomes a
> performance-vs-reliability issue unless there are specific interfaces
> to specify the various operations.
[...]
>> In other cases, particularly SSDs, then overwriting existing logical
>> sectors does not affect the old data directly, and it could be
>> recovered. Other forms of data storage, such as cloud storage, are
>> getting more common - and you have even less control there.
>
> If there is no device reliability, the device should not be used for
> critical data.
If the device is 'a SSD', that is, a NAND flash ROM with and onboard
computer supposed to provide the illusion of an IDE/ SATA device and to
hide the deficiencies of the 'storage technology' as hard as possible
(ie, at least one hardware manufacturer's manual states that the
recommended procedure for dealing with sectors refusing to be programmed
is to keep erasing and re-programming them until write errors can't be
detected immediately anymore[*]) it shouldn't - technically - be used for
any data. These devices also usually employ so-called 'wear-leveling',
meaning they try to distribute writes evenly among all sectors so that
they all wear out at the some speed and (likely, since nobody really
knows what the onboard computer is actually doing) also relocated data
to 'still working sectors' to cope with 'not working anymore' sectors.
[*] This doesn't necessarily mean that the next read will still return
the data, especially considering that 'NAND flash reads' can destroy
the data 'spatially close other data' as a side effect.
(OTOH, they're cheap and fast (to read, the write speed is seriously
grotty but nobody wants to notice that) and only some orders of
magnitude less reliable than DRAM without error detection and/or
correction ---
> You wouldn't expect a large business to send its daily
> cash intake to the bank by using some homeless person grabbed off the
> street, and using cloud storage is similar; some may be reliable but
> others not.
--- and 'large businesses' routinely use this technology for dealing
with other people's money: ATMs/ cash machines usually contain PCs
runnig some ancient version of Windows or (reportedly) OS/2.)
>
>> So no filesystem can actually guarantee that it can erase old data - and
>> filesystems therefore normally provide no functions to erase old data.
>
> If a courier service routinely grabs homeless people off the street to
> deliver large amounts of cash, one might wisely choose not to use
> them.
>
> We still disagree about what "no" filesystem can do. If you mean to
> limit the discussion to "existing linux filesystems" then I might have
> no choice but to concede your point; if you want to talk about what
> can be done, what is actually possible, then we will probably continue
> to disagree on this subject.
Every filesystem is at the mercy of the storage devices it is used on.
[...]
>> That is certainly true - your data is only as safe as your key. But the
>> usual procedure for encrypted partitions is to enter your key (or
>> passphrase) at the keyboard when mounting.
>>
>
> Yes, and once you have done so, the mounted partition is no more
> secure than a non-encrypted partition; data security has, through your
> supplying the key, been degraded from whatever level of security is
> provided by the encryption algorithm used,
It is actually worse because the key must be retained in memory which
implies that it is possible to get it out of that again, even in ways
people usually don't want to think about, cf
https://freedom-to-tinker.com/blog/felten/new-research-result-cold-boot-attacks-disk-encryption/
> to the simple "security"
> provided by the linux protection modes. To quote you, "for the
> paranoid, less than 100% is not good enough".
It supports stopping "unauthorized personnel" from "accessing files
they're not supposed to access". That's enough 'security'.
> I am of the opinion that the linux protection modes inherited from
> Unix are not very "adequate", and the idea of a "superuser" is
> inherently crippled "security". The whole user/group/world concept
> along with the ability to open files for non-exclusive access is
> ridiculously antiquated and stems from the pre-1970 era when bisync
> was state-of-the-art and the concept of servers as separate asset
> owners was little more than a foggy idea in the minds of a few.
There are at least three different concepts in this paragraph:
1. The kind of access policies which can or cannot be configured
via 'traditional UNIX(*) access permissions'
2. The 'privilege separation model' use to determine which
applications can execute which kind of system calls.
3. Implicit mandatory file locking (I wonder how that landed in
here)
ad 1) 'common Linux filesystems' support access control lists, too. I've
so far (in about 20 years) encountered exactly one 'real-world problem'
which couldn't be solved without them.
ad 2) 'Linux system call execution privileges' are actually based on a
more fine-grained 'capabilities model' and a 'UNIX(*) superuser' is
emulated by assigning all capabilities to 'UID 0 processes' by
default. AFAIK, nobody uses this except "dexstop devsloppers' and even
in this case, I strongly suspect that they're rather suffering from an
"OMG! So 1970s!"(BTNHMDI!![*]) knee-jerk reaction than trying to solve real
problems.
[*] "But That's Not How Microsoft Does It"
ad 3) BTNHMDI? Cooperating applications have the necessary facilities
for coordinating accesses to shared files in case this is
necessary. Which it isn't always.
[toc] | [prev] | [next] | [standalone]
| From | crankypuss <crankypuss@nomail.invalid> |
|---|---|
| Date | 2014-04-26 04:02 -0600 |
| Message-ID | <ljg06q$f5s$1@dont-email.me> |
| In reply to | #652 |
On 04/25/2014 12:14 PM, Rainer Weikusat wrote: > crankypuss <crankypuss@nomail.invalid> writes: >> On 04/24/2014 02:46 PM, David Brown wrote: <lots of snippage, apologies if results have become muddled> from RW: > Every filesystem is at the mercy of the storage devices it is used on. Every filesystem is *limited* by the storage devices it uses, which is not identical to "at the mercy of". Device behavior can be monitored and its capabilities can become restricted, *if* the filesystem is sufficiently sophisticated. Without digging into code, just being aware of the multi-threaded nature of what is required for this sort of thing, I am doubtful that such is often employed at this stage of the game. The more our technology moves toward cloud computing and servers-as-devices, the more we have devices-acting-as-servers, the more important this will become. I think we are already running somewhat "behind the curve" and need to consider some catch-up. DB: >>> [...] - your data is only as safe as your key. But the >>> usual procedure for encrypted partitions is to enter your key (or >>> passphrase) at the keyboard when mounting. C: >> Yes, and once you have done so, the mounted partition is no more >> secure than a non-encrypted partition; data security has, through your >> supplying the key, been degraded from whatever level of security is >> provided by the encryption algorithm used, RW: > It is actually worse because the key must be retained in memory which > implies that it is possible to get it out of that again, even in ways > people usually don't want to think about, cf > > https://freedom-to-tinker.com/blog/felten/new-research-result-cold-boot-attacks-disk-encryption/ [...] C: >> to the simple "security" >> provided by the linux protection modes. To quote you [DB], "for the >> paranoid, less than 100% is not good enough". RW: > It supports stopping "unauthorized personnel" from "accessing files > they're not supposed to access". That's enough 'security'. It isn't just about personnel. The model that relies on 'root' as superuser has come askew imo. There are no "superuser" people, some know more than others but none of us know it all; making the split between superuser/other is inherently flawed because it does not represent reality. Give someone the root password, or add someone to the "sudoers" list, and you've opened the gates. Worstcase is when someone wants a swell new application and needs to install it as root; that opens a hole between what the person believes about the application, and what is true about it. The root password is required for too many things, too many people have access to it. >> I am of the opinion that the linux protection modes inherited from >> Unix are not very "adequate", and the idea of a "superuser" is >> inherently crippled "security". The whole user/group/world concept >> along with the ability to open files for non-exclusive access is >> ridiculously antiquated and stems from the pre-1970 era when bisync >> was state-of-the-art and the concept of servers as separate asset >> owners was little more than a foggy idea in the minds of a few. RW: > There are at least three different concepts in this paragraph: > > 1. The kind of access policies which can or cannot be configured > via 'traditional UNIX(*) access permissions' > > 2. The 'privilege separation model' use to determine which > applications can execute which kind of system calls. > > 3. Implicit mandatory file locking (I wonder how that landed in > here) It landed here because it is a related problem that is imo not being fully handled. The shared-file model originated back in the pre-1970 era when few people were even aware of the concept of a "server". It is imo an obsolete model; there should only be one user reading or writing any given file, and that should be the "file server"; the file server model handles all cases, where shared file access does not. Implicit mandatory locking is a quick-and-dirty implementation of the file-server model, and anything less reliable than mandatory locking is too sloppy to touch with a pole imo. RW: > ad 1) 'common Linux filesystems' support access control lists, too. I've > so far (in about 20 years) encountered exactly one 'real-world problem' > which couldn't be solved without them. ACL's should not be an optional add-on imo; the only way to properly test file access is to have one-method-for-all. And if you're consistently using ACL's then the ogw-model becomes obsolete. In fact the whole "group" concept is imo hopelessly obsolete already, its only real value is toward file-sharing-sans-proper-locking. > ad 2) 'Linux system call execution privileges' are actually based on a > more fine-grained 'capabilities model' and a 'UNIX(*) superuser' is > emulated by assigning all capabilities to 'UID 0 processes' by > default. It doesn't matter how good the underlying mechanisms are if the top-level interfaces offer support based on potentially incorrect assumptions. > AFAIK, nobody uses this except "dexstop devsloppers' and even > in this case, I strongly suspect that they're rather suffering from an > "OMG! So 1970s!"(BTNHMDI!![*]) knee-jerk reaction than trying to solve real > problems. > > [*] "But That's Not How Microsoft Does It" > > ad 3) BTNHMDI? Cooperating applications have the necessary facilities > for coordinating accesses to shared files in case this is > necessary. Which it isn't always. > Perhaps it should always be presumed necessary, that's the way it seems to me anyhow, the number of programs I've written since 1969 which assumed less than the most rigorous set of requirements and did not later need to be updated or (more usually) rewritten to accede to the fact that the initial implementation was too naive, amounts of almost nil.
[toc] | [prev] | [next] | [standalone]
| From | Rainer Weikusat <rweikusat@mobileactivedefense.com> |
|---|---|
| Date | 2014-04-27 21:26 +0100 |
| Message-ID | <87wqear01o.fsf@sable.mobileactivedefense.com> |
| In reply to | #653 |
crankypuss <crankypuss@nomail.invalid> writes:
> On 04/25/2014 12:14 PM, Rainer Weikusat wrote:
>> crankypuss <crankypuss@nomail.invalid> writes:
[...]
> C:
>>> to the simple "security"
>>> provided by the linux protection modes. To quote you [DB], "for the
>>> paranoid, less than 100% is not good enough".
>
> RW:
>> It supports stopping "unauthorized personnel" from "accessing files
>> they're not supposed to access". That's enough 'security'.
>
> It isn't just about personnel. The model that relies on 'root' as
> superuser has come askew imo. There are no "superuser" people, some
> know more than others but none of us know it all; making the split
> between superuser/other is inherently flawed because it does not
> represent reality.
[...]
This is an entirely different topic. So-called 'UNIX(*) file system
permissions' enable defining access control policies for 'filesystem
objects'. The 'security' offered by the kernel is that these access
rights are enforced. 'Superuser' is a technical term and refers to a
system account whose access to the system is not limited like that
of an 'ordinary' user account. Hence, it has 'relative superpowers'
which implies exactly nothing a about the (real) people who might be
able to use these 'superpowers'.
>>> I am of the opinion that the linux protection modes inherited from
>>> Unix are not very "adequate", and the idea of a "superuser" is
>>> inherently crippled "security". The whole user/group/world concept
>>> along with the ability to open files for non-exclusive access is
>>> ridiculously antiquated and stems from the pre-1970 era when bisync
>>> was state-of-the-art and the concept of servers as separate asset
>>> owners was little more than a foggy idea in the minds of a few.
>
> RW:
>> There are at least three different concepts in this paragraph:
>>
>> 1. The kind of access policies which can or cannot be configured
>> via 'traditional UNIX(*) access permissions'
>>
>> 2. The 'privilege separation model' use to determine which
>> applications can execute which kind of system calls.
>>
>> 3. Implicit mandatory file locking (I wonder how that landed in
>> here)
>
> It landed here because it is a related problem that is imo not being
> fully handled. The shared-file model originated back in the pre-1970
> era when few people were even aware of the concept of a "server". It
> is imo an obsolete model; there should only be one user reading or
> writing any given file, and that should be the "file server"; the file
> server model handles all cases, where shared file access does not.
> Implicit mandatory locking is a quick-and-dirty implementation of the
> file-server model, and anything less reliable than mandatory locking
> is too sloppy to touch with a pole imo.
Ehh ... you do understand that programs, shared libraries and stuff like
'documentation' are ultimatively all files accessible via the filesystem
namespace? Concurrent read accesses don't conflict with each other and
concurrent write access to files may or may not: It is up to a set of
cooperating applications to implement whatever 'access coordination
policy' might be necessary for a given case. Eg, one example I'm dealing
with is a IKE-server ('Internet Key Exhange' [protocol], used for
creating dynamic IPsec-based VPNs) which writes a record (in append
mode) to a certain file whenever a user has 'logged in' and an IP was
assigned to it and a second program responsible for doing all kinds of
'user-specific, automatic reconfigurations', eg, in order to support a
notion of 'firewall/ packet filter rules' supposed to affect users or
groups of users' reads these records as they are appended in order to
update its idea of which IPs currently represent which users.
> RW:
>> ad 1) 'common Linux filesystems' support access control lists, too. I've
>> so far (in about 20 years) encountered exactly one 'real-world problem'
>> which couldn't be solved without them.
>
> ACL's should not be an optional add-on imo; the only way to properly
> test file access is to have one-method-for-all. And if you're
> consistently using ACL's then the ogw-model becomes obsolete.
... and if I'm "consistently" walking, all other means of land
transportation 'become obsolete' because I can walk to any place
someone could reach by car or train but the reverse is not
true. Nevertheless, I prefer using the train when I want to visit
someone living 30 miles distant because while this is an inflexible and
very limited way of reaching 'a destination', it's a lot faster where
applicable.
NB: This is example is IMHO less stupid than it may sound.
> In fact the whole "group" concept is imo hopelessly obsolete already,
> its only real value is toward file-sharing-sans-proper-locking.
As already mentioned above, locking isn't needed for concurrent read
accesses. Also, group access rights have other uses. A common one would
be 'allowing all members of a group read or read/write access to some
filesystem objects or filesystem objects but deny them the ability to
change the permissions'. This will usually mean a file owned by root in
a directory owned by root whose 'group access permissions' permit the
intended access, eg, a file storing a database access password needed by
some processes.
[toc] | [prev] | [next] | [standalone]
| From | crankypuss <crankypuss@nomail.invalid> |
|---|---|
| Date | 2014-04-28 03:27 -0600 |
| Message-ID | <ljl6uh$rbi$1@dont-email.me> |
| In reply to | #658 |
On 04/27/2014 02:26 PM, Rainer Weikusat wrote:
> crankypuss <crankypuss@nomail.invalid> writes:
>> On 04/25/2014 12:14 PM, Rainer Weikusat wrote:
>>> crankypuss <crankypuss@nomail.invalid> writes:
>
> [...]
>
>> C:
>>>> to the simple "security"
>>>> provided by the linux protection modes. To quote you [DB], "for the
>>>> paranoid, less than 100% is not good enough".
>>
>> RW:
>>> It supports stopping "unauthorized personnel" from "accessing files
>>> they're not supposed to access". That's enough 'security'.
>>
>> It isn't just about personnel. The model that relies on 'root' as
>> superuser has come askew imo. There are no "superuser" people, some
>> know more than others but none of us know it all; making the split
>> between superuser/other is inherently flawed because it does not
>> represent reality.
>
> [...]
>
> This is an entirely different topic. So-called 'UNIX(*) file system
> permissions' enable defining access control policies for 'filesystem
> objects'. The 'security' offered by the kernel is that these access
> rights are enforced. 'Superuser' is a technical term and refers to a
> system account whose access to the system is not limited like that
> of an 'ordinary' user account. Hence, it has 'relative superpowers'
> which implies exactly nothing a about the (real) people who might be
> able to use these 'superpowers'.
>
>>>> I am of the opinion that the linux protection modes inherited from
>>>> Unix are not very "adequate", and the idea of a "superuser" is
>>>> inherently crippled "security". The whole user/group/world concept
>>>> along with the ability to open files for non-exclusive access is
>>>> ridiculously antiquated and stems from the pre-1970 era when bisync
>>>> was state-of-the-art and the concept of servers as separate asset
>>>> owners was little more than a foggy idea in the minds of a few.
>>
>> RW:
>>> There are at least three different concepts in this paragraph:
>>>
>>> 1. The kind of access policies which can or cannot be configured
>>> via 'traditional UNIX(*) access permissions'
>>>
>>> 2. The 'privilege separation model' use to determine which
>>> applications can execute which kind of system calls.
>>>
>>> 3. Implicit mandatory file locking (I wonder how that landed in
>>> here)
>>
>> It landed here because it is a related problem that is imo not being
>> fully handled. The shared-file model originated back in the pre-1970
>> era when few people were even aware of the concept of a "server". It
>> is imo an obsolete model; there should only be one user reading or
>> writing any given file, and that should be the "file server"; the file
>> server model handles all cases, where shared file access does not.
>> Implicit mandatory locking is a quick-and-dirty implementation of the
>> file-server model, and anything less reliable than mandatory locking
>> is too sloppy to touch with a pole imo.
>
> Ehh ... you do understand that programs, shared libraries and stuff like
> 'documentation' are ultimatively all files accessible via the filesystem
> namespace?
I have understood that I've been working with lame software since 1969
and at this point I'm sick of converting code to meet yet-another-set of
system deficiencies with more workarounds; feel free to justify the set
of deficiencies you prefer, just don't expect me to care much that you
don't recognize them as deficiencies.
> Concurrent read accesses don't conflict with each other and
> concurrent write access to files may or may not: It is up to a set of
> cooperating applications to implement whatever 'access coordination
> policy' might be necessary for a given case. Eg, one example I'm dealing
> with is a IKE-server ('Internet Key Exhange' [protocol], used for
> creating dynamic IPsec-based VPNs) which writes a record (in append
> mode) to a certain file whenever a user has 'logged in' and an IP was
> assigned to it and a second program responsible for doing all kinds of
> 'user-specific, automatic reconfigurations', eg, in order to support a
> notion of 'firewall/ packet filter rules' supposed to affect users or
> groups of users' reads these records as they are appended in order to
> update its idea of which IPs currently represent which users.
>
>> RW:
>>> ad 1) 'common Linux filesystems' support access control lists, too. I've
>>> so far (in about 20 years) encountered exactly one 'real-world problem'
>>> which couldn't be solved without them.
>>
>> ACL's should not be an optional add-on imo; the only way to properly
>> test file access is to have one-method-for-all. And if you're
>> consistently using ACL's then the ogw-model becomes obsolete.
>
> ... and if I'm "consistently" walking, all other means of land
> transportation 'become obsolete' because I can walk to any place
> someone could reach by car or train but the reverse is not
> true. Nevertheless, I prefer using the train when I want to visit
> someone living 30 miles distant because while this is an inflexible and
> very limited way of reaching 'a destination', it's a lot faster where
> applicable.
>
> NB: This is example is IMHO less stupid than it may sound.
>
>> In fact the whole "group" concept is imo hopelessly obsolete already,
>> its only real value is toward file-sharing-sans-proper-locking.
>
> As already mentioned above, locking isn't needed for concurrent read
> accesses. Also, group access rights have other uses. A common one would
> be 'allowing all members of a group read or read/write access to some
> filesystem objects or filesystem objects but deny them the ability to
> change the permissions'. This will usually mean a file owned by root in
> a directory owned by root whose 'group access permissions' permit the
> intended access, eg, a file storing a database access password needed by
> some processes.
>
If you to need to continue arguing toward winning a debate, you'll have
to do that without my participation. I'm not interested in winning
debates, I'm interested in better software, which imo includes moving
past the archaic furniture that linux inherited from unix.
[toc] | [prev] | [next] | [standalone]
| From | Rainer Weikusat <rweikusat@mobileactivedefense.com> |
|---|---|
| Date | 2014-04-28 12:17 +0100 |
| Message-ID | <874n1dwvo0.fsf@sable.mobileactivedefense.com> |
| In reply to | #659 |
crankypuss <crankypuss@nomail.invalid> writes: > On 04/27/2014 02:26 PM, Rainer Weikusat wrote: [...] >> As already mentioned above, locking isn't needed for concurrent read >> accesses. Also, group access rights have other uses. A common one would >> be 'allowing all members of a group read or read/write access to some >> filesystem objects or filesystem objects but deny them the ability to >> change the permissions'. This will usually mean a file owned by root in >> a directory owned by root whose 'group access permissions' permit the >> intended access, eg, a file storing a database access password needed by >> some processes. > > If you to need to continue arguing toward winning a debate, you'll > have to do that without my participation. I'm not interested in > winning debates, I'm interested in better software, which imo includes > moving past the archaic furniture that linux inherited from unix. As I already wrote in the other post, I'm not really argueing, more pointing out facts which are (or - to me - at least seam to be) at odds with what you've been writing. If you were interested in something other than "winning the debate", you'd either admit that you were wrong or explain what I didn't understand and thus, got wrong, instead of asserting that "whoever mentions something which doesn't find into your agenda is Surely Evil[tm]/ interested in bad stuff for nefarious purposes".
[toc] | [prev] | [next] | [standalone]
| From | Rainer Weikusat <rweikusat@mobileactivedefense.com> |
|---|---|
| Date | 2014-04-28 13:01 +0100 |
| Message-ID | <87zjj5vf1k.fsf@sable.mobileactivedefense.com> |
| In reply to | #663 |
Rainer Weikusat <rweikusat@mobileactivedefense.com> writes: > crankypuss <crankypuss@nomail.invalid> writes: >> On 04/27/2014 02:26 PM, Rainer Weikusat wrote: > > [...] > >>> As already mentioned above, locking isn't needed for concurrent read >>> accesses. Also, group access rights have other uses. A common one would >>> be 'allowing all members of a group read or read/write access to some >>> filesystem objects or filesystem objects but deny them the ability to >>> change the permissions'. This will usually mean a file owned by root in >>> a directory owned by root whose 'group access permissions' permit the >>> intended access, eg, a file storing a database access password needed by >>> some processes. >> >> If you to need to continue arguing toward winning a debate, you'll >> have to do that without my participation. I'm not interested in >> winning debates, I'm interested in better software, which imo includes >> moving past the archaic furniture that linux inherited from unix. > > As I already wrote in the other post, I'm not really argueing, more > pointing out facts which are (or - to me - at least seam to be) at odds > with what you've been writing. If you were interested in something other > than "winning the debate", you'd either admit that you were wrong or > explain what I didn't understand and thus, got wrong, instead of > asserting that "whoever mentions something which doesn't find into your > agenda is Surely Evil[tm]/ interested in bad stuff for nefarious > purposes". As a particularly striking example: Assuming that a filesystem namespace shared among multiple processes is not supposed to exist, how is the system supposed to provide access to 'applications' (since they cannot be files accessible via the filesystem namespace anymore)?
[toc] | [prev] | [next] | [standalone]
| From | crankypuss <crankypuss@nomail.invalid> |
|---|---|
| Date | 2014-04-29 02:50 -0600 |
| Message-ID | <ljnp3t$ni7$1@dont-email.me> |
| In reply to | #664 |
On 04/28/2014 06:01 AM, Rainer Weikusat wrote: > Rainer Weikusat <rweikusat@mobileactivedefense.com> writes: >> crankypuss <crankypuss@nomail.invalid> writes: >>> On 04/27/2014 02:26 PM, Rainer Weikusat wrote: >> >> [...] >> >>>> As already mentioned above, locking isn't needed for concurrent read >>>> accesses. Also, group access rights have other uses. A common one would >>>> be 'allowing all members of a group read or read/write access to some >>>> filesystem objects or filesystem objects but deny them the ability to >>>> change the permissions'. This will usually mean a file owned by root in >>>> a directory owned by root whose 'group access permissions' permit the >>>> intended access, eg, a file storing a database access password needed by >>>> some processes. >>> >>> If you to need to continue arguing toward winning a debate, you'll >>> have to do that without my participation. I'm not interested in >>> winning debates, I'm interested in better software, which imo includes >>> moving past the archaic furniture that linux inherited from unix. >> >> As I already wrote in the other post, I'm not really argueing, more >> pointing out facts which are (or - to me - at least seam to be) at odds >> with what you've been writing. If you were interested in something other >> than "winning the debate", you'd either admit that you were wrong or >> explain what I didn't understand and thus, got wrong, instead of >> asserting that "whoever mentions something which doesn't find into your >> agenda is Surely Evil[tm]/ interested in bad stuff for nefarious >> purposes". > > As a particularly striking example: Assuming that a filesystem namespace > shared among multiple processes is not supposed to exist, how is the > system supposed to provide access to 'applications' (since they cannot > be files accessible via the filesystem namespace anymore)? > We are attempting to communicate across a large gap in viewpoint, and that makes things more difficult than they might otherwise be; if it is to work, some patience is required of both parties. Linux stems from Unix, which was developed during what one might call "mainframe days". During the late 1960s when Unix was developed, computers were not yet what they are now. Cost and features drove a specific architecture to the fore. It was necessary for many users to share the resources of a single machine because there simply were not that many machines in existence as compared to the number of users. Machine-to-machine communication technology was rudimentary. What constituted a "filesystem" then was not what we might think of as a "filesystem" today... the term "filesystem" has, in a sense, outlasted its earliest semantics. In those days, the whole client/server model had yet to emerge, it wasn't until systems communicating with other systems became commonplace that the client/server model became obvious. Under the mainframe model, sharing files between multiple users of a single "mainframe" was an obvious "necessity" if much actual work was to get done. The world has changed, hardware has evolved, new technology and new viewpoints have emerged. You wish to "educate" me so that "you do understand that programs, shared libraries and stuff like 'documentation' are ultimatively all files accessible via the filesystem namespace?" The answer is no, what I understand is perhaps different from what you think I should understand. Files are irrelevant, it is the storage and retrieval of named data that is relevant; "files" are a conceptual artifact held over from historic storage technology, a special case of the more general case "named data". You seem to like the term "namespace" and yes, it has been all the rage for the past decade or so to recognize the concept of a "namespace" and realize that a given name has one meaning in one namespace, another possibly quite different meaning in another namespace. However the "namespace" concept is really nothing new, it was embodied in IBM's OS/360 as introduced in 1964 which used hierarchical dot-qualification to separate "namespaces", essentially the same thing as a linux/Unix "directory tree" but using periods instead of slashes. The idea of "namespaces" is little more than a technique for question-begging and unintentional obfuscation, since an individual element is only uniquely addressed by its fully-qualified name, regardless of how many "namespaces" are considered to exist. The whole thing is about interfaces, their semantics, and the syntax through which the two are connected. The interfaces provided by Unix, and hence by linux, for working with "files", no longer match the reality existent when those interfaces were initially defined. In those days things were simpler, "files" were stored on devices that either worked or not; there were fewer problems in that world, if the device didn't respond "immediately" it was never going to respond because it had malfunctioned. It wasn't a case of a response being delayed because the file-server was undergoing maintenance and was temporarily unavailable, or because there was a DNS update and just where a given domain was happened to be a bit wiggly, there were fewer cases and the interfaces provided in those early days only provided what was necessary, they didn't account for cases that wouldn't come into existence until years later when communication technology had emerged in a world where "machines" were cheap and plentiful. Linux seems still locked into those "ancient" interfaces, and what I say is that it's time to set aside "Unix compatability" and build something more modern so that the increasingly complex world does not ambush us. Some seem to disagree with that view, but then some people blindly parse command output with sed and use that to drive system-critical functionality; go figure.
[toc] | [prev] | [next] | [standalone]
| From | Rainer Weikusat <rweikusat@mobileactivedefense.com> |
|---|---|
| Date | 2014-05-05 21:31 +0100 |
| Subject | UNIX(*)/ Linux history & system design (was: shred or scrub) |
| Message-ID | <87k3a0q867.fsf@sable.mobileactivedefense.com> |
| In reply to | #668 |
crankypuss <crankypuss@nomail.invalid> writes: > On 04/28/2014 06:01 AM, Rainer Weikusat wrote: >> Rainer Weikusat <rweikusat@mobileactivedefense.com> writes: >>> crankypuss <crankypuss@nomail.invalid> writes: >>>> On 04/27/2014 02:26 PM, Rainer Weikusat wrote: [...] > Linux stems from Unix, which was developed during what one might call > "mainframe days". During the late 1960s when Unix was developed, > computers were not yet what they are now. Cost and features drove a > specific architecture to the fore. It was necessary for many users to > share the resources of a single machine because there simply were not > that many machines in existence as compared to the number of > users. Machine-to-machine communication technology was rudimentary. Linux does not 'stem from UNIX(*)', it is (among other things) an uncertified 'more-or-less' implementation of the POSIX/ UNIX(*) API. UNIX(*) wasn't developed 'in the late 1960s'. Some of the concepts came to life in embryonic form after Bell Labs pulled out of the Multics-project in 1969 but something which actually resembled 'current UNIX(*)' didn't exist until 1973 (first PDP-11 UNIX(*) with a kernel written in C) and many things people tend to associate with UNIX(*), eg the Bourne shell or 'stdio & malloc' saw the light of the day with the 7th edition released in 1979. By that time, it was a system for 16-bit mini-computers (not mainframes) and it would soon become one for 32-bit mini-computers, in particular, VAXen. Since then, development has continued both on 'mini-computer style machines', nowadays usually referred to as 'servers', and single-user workstations, the latter being eventually replaced by 'Linux on PC hardware' around 'the mid 2010s'. 'Networking support' was roughly added with 4.2BSD in 1983 which means that the (ongoing) lifetime of 'the system with networking' is more than twice as long as the lifetime of the 'system without networking' (UUCP was also part of the 7th edition). (Admittedly, the idea of 'a hierarchically organized system for accessing data located on unspecified secondary storage devices' seems to have originated with Multics (but see below) and Multics was a 'mainframe time-sharing OS developed in the 2nd half of the 1960') > What constituted a "filesystem" then was not what we might think of as > a "filesystem" today... the term "filesystem" has, in a sense, > outlasted its earliest semantics. > > In those days, the whole client/server model had yet to emerge, it > wasn't until systems communicating with other systems became > commonplace that the client/server model became obvious. Under the > mainframe model, sharing files between multiple users of a single > "mainframe" was an obvious "necessity" if much actual work was to get > done. The world has changed, hardware has evolved, new technology and > new viewpoints have emerged. Even on a time-shared mini-computer (or mainframe, fwiw), files are not really shared 'between users' but between different processes which happen to run on behalf of different 'physical users'. I'm the sole 'actual user' on the computer I'm using to write this, however, it nevertheless runs 121 different processes using 'the filesystem namespace' for addressing shared resources, eg, libraries or database files. And I'm decidedly not the sole 'real user' on 'our development machines' which provide access to resources people don't usually have 'at home', eg, publically reachable IPv4 addresses associated with machines with reasonable network connections. There are also a lot more 'real users' on production installations (sometimes hundreds or thousands) which de facto do 'time-sharing[*]', except that their access to the system is limited to interacting with 'certain applications' instead of being allowed to use them for general-purpose computing. [*] Every 'web server' offering users to option to access some applications via browser is nothing but a time-sharing system, just not a 'general purpose times-sharing system'. > You wish to "educate" me so that "you do understand that programs, > shared libraries and stuff like 'documentation' are ultimatively all > files accessible via the filesystem namespace?" > > The answer is no, what I understand is perhaps different from what you > think I should understand. > > Files are irrelevant, it is the storage and retrieval of named data > that is relevant; "files" are a conceptual artifact held over from > historic storage technology, a special case of the more general case > "named data". 'file' is nothing but a term referring to some collection of 'data' which can be accessed 'by name', ie, 'named data' ... > You seem to like the term "namespace" and yes, it has been all the > rage for the past decade or so to recognize the concept of a > "namespace" and realize that a given name has one meaning in one > namespace, another possibly quite different meaning in another > namespace. However the "namespace" concept is really nothing new, it > was embodied in IBM's OS/360 as introduced in 1964 which used > hierarchical dot-qualification to separate "namespaces", essentially > the same thing as a linux/Unix "directory tree" but using periods > instead of slashes. ... and organizing 'names' referring to 'collections of data' in form of a hierarchical structure where 'generality' decreases with distance from the root of the tree is way older than that: Libraries usually organize books in this way. > The idea of "namespaces" is little more than a technique for > question-begging and unintentional obfuscation, since an individual > element is only uniquely addressed by its fully-qualified name, > regardless of how many "namespaces" are considered to exist. Apart from that, we're using the term 'namespace' in a different way: 'The filesystem namespace' is something which enables resources which are 'somehow accessible' to be named uniquely and independently of the way the resource are actually accessed. Eg, an application can use the name /bin/sh to execute a command interpreter with certain properties (assuming a system conforming to certain 'standards and conventions') without the slightest bit of an idea what kind of real or 'emulated' device will ultimatively supply the data corresponding with the name. [...] > The interfaces provided by Unix, and hence by linux, for working with > "files", no longer match the reality existent when those interfaces > were initially defined. In those days things were simpler, "files" > were stored on devices that either worked or not; there were fewer > problems in that world, if the device didn't respond "immediately" it > was never going to respond because it had malfunctioned. It wasn't a > case of a response being delayed because the file-server was > undergoing maintenance and was temporarily unavailable, or because > there was a DNS update and just where a given domain was happened to > be a bit wiggly, there were fewer cases and the interfaces provided in > those early days only provided what was necessary, they didn't account > for cases that wouldn't come into existence until years later when > communication technology had emerged in a world where "machines" were > cheap and plentiful. I have to admit that I can't really make sense of that (eg, what interfaces are you referring to?) > Linux seems still locked into those "ancient" interfaces, and what I > say is that it's time to set aside "Unix compatability" and build > something more modern so that the increasingly complex world does not > ambush us. Judging from something you wrote earlier, you probably consider 'ACLs' 'more modern' than the simpler UNIX(*) ugo-model. They're, however, actually older as they originated with the Multics 'secondary storage system'. And something similar is true for a great many other things nowadays labelled as modern: The actually are resurrected 'ancient concepts' sold to 'sufficiently young people' who are not familiar with 'ancient concepts' as 'more modern' than something older than them they believe to be familiar with. Eg, UNIX(*) (AFAIK) pioneered the idea of implementing 'system boot' in an interpreted, 'easy to use' (for people willing to learn it or roughly familiar with 'languages used in the 1960s and 70s) programming language supporting 'automatic memory management'. This in the process of being abandoned in favor of the 'modern concept' of using a notoriously difficult to use low-level language instead. For "performance reasons" since - clearly - hardware got so much more feeble since the 1980s that the more user-friendly, _newer_ idea can't be employed anymore. > Some seem to disagree with that view, but then some people blindly > parse command output with sed and use that to drive system-critical > functionality; go figure. I did a lot of shell-programming when I was working on 'embeddded Linux project' some years ago because that was the most convenient programming language available to me in that environment (ARM9 machine language and C being the other options) and 'sed' was the most sophisticated text-processing tool available to me. What's wrong with using it?
[toc] | [prev] | [next] | [standalone]
| From | crankypuss <crankypuss@nomail.invalid> |
|---|---|
| Date | 2014-05-05 16:02 -0600 |
| Subject | Re: UNIX(*)/ Linux history & system design |
| Message-ID | <lk91qa$9l3$1@dont-email.me> |
| In reply to | #674 |
On 05/05/2014 02:31 PM, Rainer Weikusat wrote: > crankypuss <crankypuss@nomail.invalid> writes: <lots snipped> >> Some seem to disagree with that view, but then some people blindly >> parse command output with sed and use that to drive system-critical >> functionality; go figure. > > I did a lot of shell-programming when I was working on 'embeddded Linux > project' some years ago because that was the most convenient programming > language available to me in that environment (ARM9 machine language and C > being the other options) and 'sed' was the most sophisticated > text-processing tool available to me. What's wrong with using it? I know nothing about "ARM9 machine language" but if C was available, yet you used bash/sed, you must have been in one bigassed hurry or had some extremely simple requirements. I'm sure your performed your job as expected and were rewarded accordingly.
[toc] | [prev] | [next] | [standalone]
| From | David Brown <david.brown@hesbynett.no> |
|---|---|
| Date | 2014-05-06 01:17 +0200 |
| Subject | Re: UNIX(*)/ Linux history & system design |
| Message-ID | <lk965t$3o3$1@dont-email.me> |
| In reply to | #675 |
On 06/05/14 00:02, crankypuss wrote: > On 05/05/2014 02:31 PM, Rainer Weikusat wrote: >> crankypuss <crankypuss@nomail.invalid> writes: > > <lots snipped> > >>> Some seem to disagree with that view, but then some people blindly >>> parse command output with sed and use that to drive system-critical >>> functionality; go figure. >> >> I did a lot of shell-programming when I was working on 'embeddded Linux >> project' some years ago because that was the most convenient programming >> language available to me in that environment (ARM9 machine language and C >> being the other options) and 'sed' was the most sophisticated >> text-processing tool available to me. What's wrong with using it? > > I know nothing about "ARM9 machine language" but if C was available, yet > you used bash/sed, you must have been in one bigassed hurry or had some > extremely simple requirements. I'm sure your performed your job as > expected and were rewarded accordingly. Someone who writes a C program when a simple bash/sed script could do the task, would be failing to do their job as expected. One of the reasons Linux (and other *nix flavours) have multiple languages and tools is that different tools are better for different jobs, and that often it is the developer's time that is important rather than the systems run time. A developer who thinks C is the answer to every question is like a carpenter armed only with a hammer. In the embedded Linux systems I have worked on, the final system used C, Python, Perl, C++, and bash. I don't think sed was used, except as part of the makefiles.
[toc] | [prev] | [next] | [standalone]
| From | crankypuss <crankypuss@nomail.invalid> |
|---|---|
| Date | 2014-05-06 01:46 -0600 |
| Subject | Re: UNIX(*)/ Linux history & system design |
| Message-ID | <lka3vv$r0j$1@dont-email.me> |
| In reply to | #677 |
On 05/05/2014 05:17 PM, David Brown wrote: > On 06/05/14 00:02, crankypuss wrote: >> On 05/05/2014 02:31 PM, Rainer Weikusat wrote: >>> crankypuss <crankypuss@nomail.invalid> writes: >> >> <lots snipped> >> >>>> Some seem to disagree with that view, but then some people blindly >>>> parse command output with sed and use that to drive system-critical >>>> functionality; go figure. >>> >>> I did a lot of shell-programming when I was working on 'embeddded Linux >>> project' some years ago because that was the most convenient programming >>> language available to me in that environment (ARM9 machine language >>> and C >>> being the other options) and 'sed' was the most sophisticated >>> text-processing tool available to me. What's wrong with using it? >> >> I know nothing about "ARM9 machine language" but if C was available, yet >> you used bash/sed, you must have been in one bigassed hurry or had some >> extremely simple requirements. I'm sure your performed your job as >> expected and were rewarded accordingly. > > Someone who writes a C program when a simple bash/sed script could do > the task, would be failing to do their job as expected. One of the > reasons Linux (and other *nix flavours) have multiple languages and > tools is that different tools are better for different jobs, and that > often it is the developer's time that is important rather than the > systems run time. A developer who thinks C is the answer to every > question is like a carpenter armed only with a hammer. > > In the embedded Linux systems I have worked on, the final system used C, > Python, Perl, C++, and bash. I don't think sed was used, except as part > of the makefiles. > Indeed, it is the developer's time that is important, because that is what the employer has purchased. Most employees are not very good at building tools, and are as shortsighted as their employers. Tools like sed have been magically provided to them, nobody ever had reason to write those because they were just magically there. Far better to simply use what has been provided, as expected, rather than building a library of C subroutines that puts knocking out a small utility on the same level of effort as debugging a bash/sed script where a single misplaced character can cause unexpected errors that are difficult to find. Not that C syntax is much less finicky, but it at least provides a few diagnostic clues regarding syntax errors. So, you mistake my words, I do not claim that C is the answer to every problem, or even that there is no value in the bash/sed combination (though I personally can't think of a situation where I'd use that particular combination). One uses the best tools at hand, and if one finds that the best tools at hand are inadequate to the task, he obtains a more appropriate tool, or if none are available, builds what is needed. I've used some gawdawful tools over the years, still use some on a daily basis; some of them annoy me and they are on the to-do list. People are different, some are more compliant than others, some simply grind through a task manually without even the use of something like bash/sed. Some use the tools at hand, others realize that their own hands are their best tools. Since all programmers are equivalent economic units, they are easily replaced if they fail to perform as expected: the hive mind has no love for independent workers.
[toc] | [prev] | [next] | [standalone]
| From | Rainer Weikusat <rweikusat@mobileactivedefense.com> |
|---|---|
| Date | 2014-05-06 15:09 +0100 |
| Subject | Re: UNIX(*)/ Linux history & system design |
| Message-ID | <87zjivatio.fsf@sable.mobileactivedefense.com> |
| In reply to | #678 |
crankypuss <crankypuss@nomail.invalid> writes: > On 05/05/2014 05:17 PM, David Brown wrote: >> On 06/05/14 00:02, crankypuss wrote: >>> On 05/05/2014 02:31 PM, Rainer Weikusat wrote: >>>> crankypuss <crankypuss@nomail.invalid> writes: >>> >>> <lots snipped> >>> >>>>> Some seem to disagree with that view, but then some people blindly >>>>> parse command output with sed and use that to drive system-critical >>>>> functionality; go figure. >>>> >>>> I did a lot of shell-programming when I was working on 'embeddded Linux >>>> project' some years ago because that was the most convenient programming >>>> language available to me in that environment (ARM9 machine language >>>> and C >>>> being the other options) and 'sed' was the most sophisticated >>>> text-processing tool available to me. What's wrong with using it? >>> >>> I know nothing about "ARM9 machine language" but if C was available, yet >>> you used bash/sed, you must have been in one bigassed hurry or had some >>> extremely simple requirements. I'm sure your performed your job as >>> expected and were rewarded accordingly. >> >> Someone who writes a C program when a simple bash/sed script could do >> the task, would be failing to do their job as expected. One of the >> reasons Linux (and other *nix flavours) have multiple languages and >> tools is that different tools are better for different jobs, and that >> often it is the developer's time that is important rather than the >> systems run time. A developer who thinks C is the answer to every >> question is like a carpenter armed only with a hammer. >> >> In the embedded Linux systems I have worked on, the final system used C, >> Python, Perl, C++, and bash. I don't think sed was used, except as part >> of the makefiles. >> > > Indeed, it is the developer's time that is important, because that is > what the employer has purchased. Most employees are not very good at > building tools, and are as shortsighted as their employers. Tools > like sed have been magically provided to them, nobody ever had reason > to write those because they were just magically there. Far better to > simply use what has been provided, as expected, rather than building a > library of C subroutines that puts knocking out a small utility on the > same level of effort as debugging a bash/sed script [...] This has indeed been done in the past: The guy who couldn't get his (Perl) scripts to work reasonably well within a reasonable time was called Rasmus Lehrdorf and he named his library of C subroutines the 'Personal Homepage Tools', better known as 'PHP'.
[toc] | [prev] | [next] | [standalone]
Page 2 of 4 — ← Prev page 1 [2] 3 4 Next page →
Back to top | Article view | comp.os.linux.development.system
csiph-web