Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #236574 > unrolled thread

Whole Disk Encryption + SSD

Back to article view | Back to linux.debian.user

Contents

  Whole Disk Encryption + SSD piorunz <piorunz@gmx.com> - 2021-06-28 17:00 +0200
    Re: Whole Disk Encryption + SSD Jochen Spieker <ml@well-adjusted.de> - 2021-06-28 18:00 +0200
    Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-06-28 22:40 +0200
      Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-06-29 00:00 +0200
      Re: Whole Disk Encryption + SSD Stefan Monnier <monnier@iro.umontreal.ca> - 2021-06-29 02:00 +0200
        Re: Whole Disk Encryption + SSD <tomas@tuxteam.de> - 2021-06-29 09:50 +0200
          Re: Whole Disk Encryption + SSD Stefan Monnier <monnier@iro.umontreal.ca> - 2021-06-30 00:50 +0200
          Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-06-30 01:00 +0200
      Re: Whole Disk Encryption + SSD Jonathan Dowland <jon+debian-user@dow.land> - 2021-06-29 11:30 +0200
      Re: Whole Disk Encryption + SSD piorunz <piorunz@gmx.com> - 2021-06-29 14:10 +0200
        Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-06-30 02:20 +0200
      Re: Whole Disk Encryption + SSD David Wright <deblis@lionunicorn.co.uk> - 2021-07-02 05:00 +0200
        Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 05:50 +0200
          Useless use of "dd" Teemu Likonen <tlikonen@iki.fi> - 2021-07-02 07:50 +0200
            Useful use of dd [was: Useless use of "dd"] <tomas@tuxteam.de> - 2021-07-02 09:50 +0200
              Re: Useful use of dd [was: Useless use of "dd"] Teemu Likonen <tlikonen@iki.fi> - 2021-07-02 12:30 +0200
                Re: Useful use of dd "Thomas Schmitt" <scdbackup@gmx.net> - 2021-07-02 13:00 +0200
                  Re: Useful use of dd <tomas@tuxteam.de> - 2021-07-02 13:20 +0200
                  Re: Useful use of dd Teemu Likonen <tlikonen@iki.fi> - 2021-07-02 13:30 +0200
                    Re: Useful use of dd David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 23:50 +0200
                Re: Useful use of dd [was: Useless use of "dd"] <tomas@tuxteam.de> - 2021-07-02 13:20 +0200
                Re: Useful use of dd Michael Stone <mstone@debian.org> - 2021-07-02 14:50 +0200
                Re: Useful use of dd [was: Useless use of "dd"] Kushal Kumaran <kushal@locationd.net> - 2021-07-02 21:10 +0200
                  Creation of empty files [was: Useful use of dd] David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 23:40 +0200
                    Re: Creation of empty files [was: Useful use of dd] David Christensen <dpchrist@holgerdanske.com> - 2021-07-03 00:00 +0200
                    Re: Creation of empty files [was: Useful use of dd] Michael Stone <mstone@debian.org> - 2021-07-03 15:50 +0200
                      Re: Creation of empty files [was: Useful use of dd] David Christensen <dpchrist@holgerdanske.com> - 2021-07-03 23:40 +0200
                        Re: Creation of empty files [was: Useful use of dd] Michael Stone <mstone@debian.org> - 2021-07-04 06:00 +0200
              Re: Useful use of dd [was: Useless use of "dd"] Greg Wooledge <greg@wooledge.org> - 2021-07-02 13:40 +0200
                Re: Useful use of dd [was: Useless use of "dd"] <tomas@tuxteam.de> - 2021-07-02 13:40 +0200
                  Re: Useful use of dd [was: Useless use of "dd"] David Wright <deblis@lionunicorn.co.uk> - 2021-07-02 17:10 +0200
                    Re: Useful use of dd [was: Useless use of "dd"] <tomas@tuxteam.de> - 2021-07-02 17:40 +0200
                    Re: Useful use of dd [was: Useless use of "dd"] Gunnar Gervin <dofeelok@gmail.com> - 2021-07-03 09:20 +0200
                      Re: Useful use of dd [was: Useless use of "dd"] <tomas@tuxteam.de> - 2021-07-03 09:30 +0200
              Useless use of shell pipelines [was: Useful use of dd] David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 23:40 +0200
                Re: Useless use of shell pipelines [was: Useful use of dd] l0f4r0@tuta.io - 2021-07-03 10:30 +0200
                  Re: Useless use of shell pipelines [was: Useful use of dd] Greg Wooledge <greg@wooledge.org> - 2021-07-03 15:40 +0200
            Re: Useless use of "dd" David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 21:40 +0200
          Re: Whole Disk Encryption + SSD David Wright <deblis@lionunicorn.co.uk> - 2021-07-02 17:10 +0200
            Re: Whole Disk Encryption + SSD Michael Stone <mstone@debian.org> - 2021-07-02 17:30 +0200
            Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 20:40 +0200

Page 1 of 3  [1] 2 3  Next page →

#236574 — Whole Disk Encryption + SSD

Hi all,

I've got about 5 years old HP laptop with SSD SATA drive 240 GB. Debian
Bullseye will be installed on it once it's released, as my secondary
computer to use.
I have question regarding whole disk encryption. What technology should
I use, to have encryption of everything, or at least /home, but preserve
free blocks and have TRIM? I don't want encryption to use entire drive
as "full" blob, I want to preserve SSDs life. What solutions should I
use? Thanks!

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀

[toc] | [next] | [standalone]

#236576

[Multipart message — attachments visible in raw view] — view raw

piorunz:
> 
> I have question regarding whole disk encryption. What technology should
> I use, to have encryption of everything, or at least /home, but preserve
> free blocks and have TRIM?

The canonical answer is "LUKS". You can configure it during installation
if you want to. I always use LVM as well. It is up to you whether you
want to use LVM on LUKS or the other way round.

I am not sure how well full-disk encrpytion is supported nowadays. For
common scenarios (like loss or simple theft of the storage medium, no
state-level attackers) you do not need it, in my opinion.

Oh, the buster release notes mention that encrypted /boot is not
supported, everything else may be encrypted, even the root filesystem.

https://www.debian.org/releases/buster/amd64/ch06s03.en.html#partman-crypto

If you think you need protection against somebody tampering with your
boot loader and/or kernel, you need to configure Secure Boot which I
have never really looked at. I guess this is overkill for now.

> I don't want encryption to use entire drive
> as "full" blob, I want to preserve SSDs life.

I am not sure what this means and whether there is any relation between
"full blob" and life-preserving measures. But let me assure you that
your SSD will be fine, not matter how you are setting up encryption. You
can set up both LUKS and LVM to pass through the "discard" command which
you need for TRIM to work and this is more of a performance measure than
a method to reliably lengthen the lifespan of your SSD.

How long do you think you will need your SSD? I recently removed an
Intel X25m from an old system. It was more than ten years old and was
first used heavily in a laptop and later on ran 24/7 for several years
as OS drive in a NAS system. And that SSD didn't even support TRIM!

Do not worry about the lifetime of your SSD.  Worry about backups.

You use LUKS by telling it which disk partition it should encrypt. You
then get a new logical block device which you can treat like any "real"
partition. Most importantly, you can create a regular filesystem on it
(or an LVM physical volume) which is encrypted before anything is
written to the physical disk. The amount of free space in your
filesystem is not dependent on having a LUKS container beneath it.

> What solutions should I
> use? Thanks!

Get familiar with LUKS and possibly LVM. There are options like ecryptfs
which work on regular filesystem and encrypt individual files as well as
their names. But those are 

-- 
When standing at the top of beachy head I find the rocks below very
attractive.
[Agree]   [Disagree]
                 <http://archive.slowlydownward.com/NODATA/data_enter2.html>

[toc] | [prev] | [next] | [standalone]

#236579

On 6/28/21 7:52 AM, piorunz wrote:
> Hi all,
> 
> I've got about 5 years old HP laptop with SSD SATA drive 240 GB. Debian
> Bullseye will be installed on it once it's released, as my secondary
> computer to use.
> I have question regarding whole disk encryption. What technology should
> I use, to have encryption of everything, or at least /home, but preserve
> free blocks and have TRIM? I don't want encryption to use entire drive
> as "full" blob, I want to preserve SSDs life. What solutions should I
> use? Thanks!

Booting a computer involves the motherboard firmware and one or more 
stages of on-disk boot loaders.  Boot loader stages, initial RAM 
filesystem (initramfs), kernel, device drivers, system configuration 
files, etc., must be available as plaintext when needed.


Software encryption (dm-crypt, Linux Unified Key System (LUKS), etc.) 
for a system drive is typically applied to the swap, root, and/or data 
partitions, but the master boot record (partition table and boot 
loader), extensible firmware interface (EFI) system partition contents, 
and boot partition contents are plaintext and easily modified by an 
attacker with physical access.  You will want a CPU with AES-NI or 
equivalent to accelerate encryption/ decryption (should get 90%+ 
performance).  Without AES-NI, performance will be fractional.


The simplest way to protect a drive at rest (e.g. powered off) is to get 
a self-encrypting drive (SED).  The motherboard firmware prompts for the 
passphrase after the power on self test (POST) and before reading the 
drive.  Once the SED passphrase is entered, the entire drive appears as 
plaintext.  Encryption/ decryption is handled by hardware inside the 
drive controller, at full performance.


I do not set the 'discard' (trim) option in fstab(5).  If and when I 
want to erase unused blocks (such as before taking an image), I use 
fstrim(8).


I would not worry about wearing out a good SSD in a daily driver laptop. 
  I have been using Intel SSD 520 Series 2.5" SATA in my SOHO laptops, 
desktops, and servers for many years; they all work and have available 
lifespans in the high 90%'s.


Along with SED, I suggest that you also implement Secure Boot.  This 
provides cryptographic signatures and chain of trust for critical files, 
such as boot loaders and the kernel, whenever those files are to be 
executed (e.g. boot, dynamic loading, etc.).


Reset the CMOS settings to factory defaults, enable/ verify Unified EFI 
(UEFI) mode, enable/ verify Secure Boot, and boot the Debian Installer 
(d-i).  Hopefully, everything should Just Work.  If not, you may have to 
modify the drive settings (Dell factory default for drives is 'RAID'; 
'ACPI' may be required).  Take good notes, STFW, RTFM, and/or post here 
if you have problems.  Post your solution when you figure it out.


David

[toc] | [prev] | [next] | [standalone]

#236580

On 6/28/21 1:36 PM, David Christensen wrote:

> (Dell factory default for drives is 'RAID'; 'ACPI' may be required).

Correction: AHCI.


David

[toc] | [prev] | [next] | [standalone]

#236582

> Along with SED, I suggest that you also implement Secure Boot.

Can someone give me pointers to actually known attacks (not
hypothetical ones, which I can invent myself without much difficulty)
that would have been prevented by Secure Boot?

I can see that subverting the early boot might be a good way for
rootkits to install themselves in a way that's hard to detect and/or
remove, but it's not like there aren't plenty of other ways to get
pretty much the same result.

IOW it sounds to me a bit like putting a reinforced steel frame around
a cardboard door.


        Stefan

[toc] | [prev] | [next] | [standalone]

#236590

[Multipart message — attachments visible in raw view] — view raw

On Mon, Jun 28, 2021 at 07:56:47PM -0400, Stefan Monnier wrote:
> > Along with SED, I suggest that you also implement Secure Boot.
> 
> Can someone give me pointers to actually known attacks (not
> hypothetical ones, which I can invent myself without much difficulty)
> that would have been prevented by Secure Boot?

Basically, subverting the unencrypted loader amounts to what is known
as "evil maid attack" [1]: the most practical variant being that the
subverted loader records your passphrase (or whatever auth thingie you
provide) and either "phones home" or stashes it away in a place your
opponent can retrieve it.

The second time they have control over your device, they can unlock
the disk.

The whole thing is well described in Wikipedia [2], along with some
accounts of actual cases.

So /if/ you leave your laptop unsupervised and have the hunch that
someone might have a chance at it, make sure you reinstall :-)

There is another, low-tech alternative to the monstrous Secure
Boot [3] thingies bandied around here: carry your real boot partition
with you, either in an USB stick or (nicer form factor) an SD card.
Bonus points: you can leave a fake boot partition in your hard disk
which can be checked at each boot; if it changed, you can go "Hmmm...
someone tried to fool me..." and perhaps send them some passphrase.
The wrong one, of course.

Cheers

[1] Yes, a sexist term, but it stuck, unfortunately. OTOH, perhaps
   it's realistic in that it acknowledges that underpaid jobs are
   usually carried out by women. Sigh.

[2] https://en.wikipedia.org/wiki/Evil_maid_attack

[3] I always have the impression that, with Secure Boot, Microsoft
   has more control of the hardware (I paid for dearly, dammit!)
   than myself. Don't ask me why, but I thoroughly dislike that
   impression. So far I try to steer clear of it.

-- tomás

[toc] | [prev] | [next] | [standalone]

#236637

>> > Along with SED, I suggest that you also implement Secure Boot.

>> Can someone give me pointers to actually known attacks (not
>> hypothetical ones, which I can invent myself without much difficulty)
>> that would have been prevented by Secure Boot?

> [2] https://en.wikipedia.org/wiki/Evil_maid_attack

Thanks.


        Stefan

[toc] | [prev] | [next] | [standalone]

#236638

On 6/29/21 12:47 AM, tomas@tuxteam.de wrote:
> On Mon, Jun 28, 2021 at 07:56:47PM -0400, Stefan Monnier wrote:
>>> Along with SED, I suggest that you also implement Secure Boot.
>>
>> Can someone give me pointers to actually known attacks (not
>> hypothetical ones, which I can invent myself without much difficulty)
>> that would have been prevented by Secure Boot?
> 
> Basically, subverting the unencrypted loader amounts to what is known
> as "evil maid attack" [1]: the most practical variant being that the
> subverted loader records your passphrase (or whatever auth thingie you
> provide) and either "phones home" or stashes it away in a place your
> opponent can retrieve it.
> 
> The second time they have control over your device, they can unlock
> the disk.
> 
> The whole thing is well described in Wikipedia [2], along with some
> accounts of actual cases.
> 
> So /if/ you leave your laptop unsupervised and have the hunch that
> someone might have a chance at it, make sure you reinstall :-)
> 
> There is another, low-tech alternative to the monstrous Secure
> Boot [3] thingies bandied around here: carry your real boot partition
> with you, either in an USB stick or (nicer form factor) an SD card.
> Bonus points: you can leave a fake boot partition in your hard disk
> which can be checked at each boot; if it changed, you can go "Hmmm...
> someone tried to fool me..." and perhaps send them some passphrase.
> The wrong one, of course.
> 
> Cheers
> 
> [1] Yes, a sexist term, but it stuck, unfortunately. OTOH, perhaps
>     it's realistic in that it acknowledges that underpaid jobs are
>     usually carried out by women. Sigh.
> 
> [2] https://en.wikipedia.org/wiki/Evil_maid_attack
> 
> [3] I always have the impression that, with Secure Boot, Microsoft
>     has more control of the hardware (I paid for dearly, dammit!)
>     than myself. Don't ask me why, but I thoroughly dislike that
>     impression. So far I try to steer clear of it.


Thank you for that information.  :-)


I am reminded of the strategy of Defense in Depth [4].


SED, dm-crypt, and LUKS are mostly orthogonal to Secure Boot.  The 
former protects all or part of the drive contents, but only when the 
computer is off.  The latter protects critical system files, both when 
the computer is off and when the computer is booting and/or running.


SED provides all of the defenses dm-crypt and LUKS provide when the 
dm-crypt and LUKS partitions are on the SED.  But if the dm-crypt and 
LUKS partitions are moved off the SED in raw encrypted form (e.g. image/ 
clone), dm-crypt and LUKS have the advantage that security is preserved. 
  (If the partitions are decrypted and then moved -- e.g. backup/ 
restore -- security is lost.)


The only way to get all of the defenses is to implement all of the 
technologies -- SED, dm-crypt, LUKS, and Secure Boot.


This should be possible with Debian and a ~5 year old laptop -- SED is a 
purchasing decision, the laptop should have UEFI firmware that supports 
SED and Secure Boot, and Debian 10 supports Secure Boot, dm-crypt and 
LUKS.  But, as always, the only way to find out for a specific 
combination of Debian installer and computer is to try it.


David


[4] https://en.wikipedia.org/wiki/Security-in-depth

[toc] | [prev] | [next] | [standalone]

#236599

On Mon, Jun 28, 2021 at 01:36:35PM -0700, David Christensen wrote:
>I do not set the 'discard' (trim) option in fstab(5).  If and when I 
>want to erase unused blocks (such as before taking an image), I use 
>fstrim(8).

I believe this is installed and enabled by default in Bullseye (at least
new installs from currently released Bullseye install media).



-- 
Please do not CC me for listmail.

👱🏻	Jonathan Dowland
✎	 jmtd@debian.org
🔗	https://jmtd.net

[toc] | [prev] | [next] | [standalone]

#236606

Hi David,

Thanks for your reply.

On 28/06/2021 21:36, David Christensen wrote:

> Software encryption (dm-crypt, Linux Unified Key System (LUKS), etc.)
> for a system drive is typically applied to the swap, root, and/or data
> partitions, but the master boot record (partition table and boot
> loader), extensible firmware interface (EFI) system partition contents,
> and boot partition contents are plaintext and easily modified by an
> attacker with physical access.  You will want a CPU with AES-NI or
> equivalent to accelerate encryption/ decryption (should get 90%+
> performance).  Without AES-NI, performance will be fractional.

Thanks. I will check out AES-NI on this CPU.

> The simplest way to protect a drive at rest (e.g. powered off) is to get
> a self-encrypting drive (SED).  The motherboard firmware prompts for the
> passphrase after the power on self test (POST) and before reading the
> drive.  Once the SED passphrase is entered, the entire drive appears as
> plaintext.  Encryption/ decryption is handled by hardware inside the
> drive controller, at full performance.

I don't trust SED, after listening to Steve Gibson analysis on state of
this feature.

Audio podcast: http://media.GRC.com/sn/SN-689.mp3
Transcript: https://www.grc.com/sn/sn-689.pdf

His findings were sourced, among other things, on work of security
researchers at the Radboud University in the Netherlands, titled:
"Self-Encrypting Deception: Weaknesses in the encryption of solid-
state drives."
https://ieeexplore.ieee.org/abstract/document/8835339

> I do not set the 'discard' (trim) option in fstab(5).  If and when I
> want to erase unused blocks (such as before taking an image), I use
> fstrim(8).

Yes, I use fstrim on weekly basis via crontab on all my SSD based
computers. I don't use discard option in fstab.
Will fstrim work with Debian-encrypted /home partition? fstrim will show
trimmed gigabytes, just like on normal system?
If yes then that's in, my enquiry is solved.

> I would not worry about wearing out a good SSD in a daily driver laptop.
>   I have been using Intel SSD 520 Series 2.5" SATA in my SOHO laptops,
> desktops, and servers for many years; they all work and have available
> lifespans in the high 90%'s.

I prefer to preserve SSD life if I can. And with this problem, it's a
matter of proper configuring it ONCE during install, and then reap the
benefits for years to come. I don't want to throw away free performance
and longevity boost.

> Along with SED, I suggest that you also implement Secure Boot.  This
> provides cryptographic signatures and chain of trust for critical files,
> such as boot loaders and the kernel, whenever those files are to be
> executed (e.g. boot, dynamic loading, etc.).

Thanks, I plan to use that. I already use it on my desktop.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]

#236639

On 6/29/21 5:02 AM, piorunz wrote:

> I don't trust SED, after listening to Steve Gibson analysis on state of
> this feature.
> 
> Audio podcast: http://media.GRC.com/sn/SN-689.mp3
> Transcript: https://www.grc.com/sn/sn-689.pdf
> 
> His findings were sourced, among other things, on work of security
> researchers at the Radboud University in the Netherlands, titled:
> "Self-Encrypting Deception: Weaknesses in the encryption of solid-
> state drives."
> https://ieeexplore.ieee.org/abstract/document/8835339

Interesting.  The IEEE paper seems to cover Crucial, Samsung, and 
Western Digital/Sandisk devices.  I'm using Intel SSD 520 Series 2.5" 
SATA drives.  STFW I don't see any CVE's.


AIUI for my Intel SSD's [1], the encryption key is stored inside the 
drive controller and is inaccessible to the outside world.  Encryption/ 
decryption is always running and the bits in the storage transistors are 
always encrypted.  Performing a Secure Erase or Enhanced Secure Erase 
will clear the storage transistors and generate a new encryption key. 
The passphrase is a feature that is independent of the encryption key 
and stored bits.  The passphrase is accessible via the motherboard CMOS 
Setup program (and specific utilities).  Changing the passphrase does 
not change the encryption key nor the bits in the storage transistors; 
from a human viewpoint, it is instantaneous.  Done correctly, data 
stored on the drive remains accessible and unchanged.


I would suggest shopping for a good SSD.  Per my other post, it will 
likely have SED.  If so, verify that the drive has no known SED 
problems.  You should be able to set or clear the passphrase as desired 
(verify with testing as soon as you get the drive).


> Will fstrim work with Debian-encrypted /home partition? 
> fstrim will show
> trimmed gigabytes, just like on normal system?
> If yes then that's in, my enquiry is solved.


fstrim(8) works on an ext4 filesystem on LUKS:

2021-06-29 16:16:32 root@dipsy ~/dipsy.tracy.holgerdanske.com/etc
# fstrim -v /scratch
/scratch: 27.7 GiB (29705924608 bytes) trimmed

2021-06-29 16:18:42 root@dipsy ~
# mount | grep scratch
/dev/mapper/sda4_crypt on /scratch type ext4 (rw,relatime)

2021-06-29 16:18:49 root@dipsy ~
# grep scratch /etc/fstab
/dev/mapper/sda4_crypt				/scratch			ext4		defaults			0	2


If you use file systems other than ext4, you should test.


>> I would not worry about wearing out a good SSD in a daily driver laptop.
>>   I have been using Intel SSD 520 Series 2.5" SATA in my SOHO laptops,
>> desktops, and servers for many years; they all work and have available
>> lifespans in the high 90%'s.
> 
> I prefer to preserve SSD life if I can. And with this problem, it's a
> matter of proper configuring it ONCE during install, and then reap the
> benefits for years to come. I don't want to throw away free performance
> and longevity boost.


When I first started using SSD's, I read various articles and worried 
about over-provisioning, trim, performance, lifetime, etc..  My 
experience has been that simple OOTB settings are just fine for my SOHO 
workloads.  People running servers with workloads that hammer the SSD's 
24x7 are the ones who need to worry.


David



[1] 
https://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/ssd-520-aes-tech-brief.pdf

[toc] | [prev] | [next] | [standalone]

#236729

On Mon 28 Jun 2021 at 13:36:35 (-0700), David Christensen wrote:

> I do not set the 'discard' (trim) option in fstab(5).  If and when I
> want to erase unused blocks (such as before taking an image), I use
> fstrim(8).

Can you elaborate on a couple of things:

How do you "take an image". Is this equivalent to a conventional
dd if=/dev/sda …, or to some other process?

When I copy an entire conventional drive or partition, all the
free blocks/unused sectors are carefully transferred to the copy.
What improvement does erasing unused blocks achieve?

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#236731

On 7/1/21 7:55 PM, David Wright wrote:
> On Mon 28 Jun 2021 at 13:36:35 (-0700), David Christensen wrote:
> 
>> I do not set the 'discard' (trim) option in fstab(5).  If and when I
>> want to erase unused blocks (such as before taking an image), I use
>> fstrim(8).
> 
> Can you elaborate on a couple of things:
> 
> How do you "take an image". Is this equivalent to a conventional
> dd if=/dev/sda …, or to some other process?

I wrote a script.  To "take an image", the script invokes dd(1) and 
pipes the output to gzip(1), copying raw device octets to a file.  To 
"restore an image", the process is reversed.


> When I copy an entire conventional drive or partition, all the
> free blocks/unused sectors are carefully transferred to the copy.

Same here.


> What improvement does erasing unused blocks achieve?

Zero blocks are readily compressed, reducing the size of the image file.


David

[toc] | [prev] | [next] | [standalone]

#236733 — Useless use of "dd"

[Multipart message — attachments visible in raw view] — view raw

* 2021-07-01 20:43:09-0700, David Christensen wrote:

> To "take an image", the script invokes dd(1) and pipes the output to
> gzip(1), copying raw device octets to a file. To "restore an image",
> the process is reversed.

Sounds like the classic "useless use of dd". For general educational
purposes I remind (not necessarily you) that "dd" is just a file copying
(and conversion) tool, almost like "cat". By default "dd" is a slow
version of "cat" because it uses smaller buffers. "dd" can be made
faster by increasing its buffer size but that will not make it more than
just "cat".

So "dd" is not a special tool for accessing device files. Device files
don't need special tools. They are files. The following command:

    dd if=/dev/sdX | gzip >image.gz

is slower but functionally the same as either of these:

    gzip </dev/sdX >image.gz
    gzip --to-stdout /dev/sdX >image.gz

Usually we don't need "dd" for anything but there are some options which
are useful sometimes. For example, "dd" can report progress to standard
error stream and it can seek and limit read to some blocks or bytes.

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[toc] | [prev] | [next] | [standalone]

#236736 — Useful use of dd [was: Useless use of "dd"]

[Multipart message — attachments visible in raw view] — view raw

On Fri, Jul 02, 2021 at 08:40:39AM +0300, Teemu Likonen wrote:

[...]

>     dd if=/dev/sdX | gzip >image.gz
> 
> is slower but functionally the same as either of these:
> 
>     gzip </dev/sdX >image.gz
>     gzip --to-stdout /dev/sdX >image.gz

Quite right -- akin to "useless use of cat".

FWIW, I've found something which could be deemed to be an "useful use
of dd" which somehow bears a hidden symmetry. As a replacement for
`cat' whenever you need to put a name on the output file.

I'll try to explain: sometimes, you use `cat' to populate a file from
stdin -- say from a script like so:

  #!/bin/sh
  cat <<__EOF >> /etc/hosts
  127.0.0.1 www.google-analytics.com
  __EOF

Now assume you don't want your whole script running under root. A
`sudo cat' (with a suitable setup) or some moral equivalent would
suffice. But, alas! the output redirect `>>' is the one needing the
higher privileges, and that's done by the shell around cat!

The customary pattern here is `tee': it passes stdin through to stdout,
and makes a "side copy" into a named file. So you'll see things like
this

   blah blah | sudo tee myfile > /dev/null

i.e. use `tee's "side copy" and throw away the straight stream.

Now OCD [1] seems to be a professional risk with some computer folks.
This "splitting the channel to throw away one copy" somehow kept tickling
me (although the folks at Combinatory Logic [2] don't seem to have
problems with that kind of wastefulness).

Well, that's where dd rescued me:

  sudo dd of=/etc/hosts oflags=append

(note that if=- is the default). Season to taste :-)

From the "semi-useful ramblings". Perhaps it amuses someone, then it was
downright useful.

FWIW [2], I still use `dd' where I could use `cp' or `cat'. Sometimes it
does align better with my hands. But I agree that it's better left out
when not needed, as is `cat'.

Now to another pet peeve of mine: useless use of grep:

  grep xxxx <myfile> | sed -e 's/bla/foo/' ...

Cheers

[1] Obsessive Compulsive Disorder
[2] https://en.wikipedia.org/wiki/Combinatory_logic#Examples_of_combinators

-- t

[toc] | [prev] | [next] | [standalone]

#236740 — Re: Useful use of dd [was: Useless use of "dd"]

[Multipart message — attachments visible in raw view] — view raw

* 2021-07-02 09:49:23+0200, tomas@tuxteam.de wrote:

> FWIW, I've found something which could be deemed to be an "useful use
> of dd" which somehow bears a hidden symmetry. As a replacement for
> `cat' whenever you need to put a name on the output file.

For this new subject I will add another use: quickly create empty file
of specific size, for example 5 * 1024 bytes:

    $ dd of=empty obs=1024 seek=5 count=0

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[toc] | [prev] | [next] | [standalone]

#236742 — Re: Useful use of dd

Hi,

Teemu Likonen wrote:
> For this new subject I will add another use: quickly create empty file
> of specific size, for example 5 * 1024 bytes:
>    $ dd of=empty obs=1024 seek=5 count=0

Not to forget creation of sparse files with few disk consumption.

For testing zisofs i have this in a script:

  #!/bin/bash
  # 100gb with big holes
  for i in {1..100}
  do
    echo $i | dd of=100gb bs=1 seek="$i"G 2>/dev/null
  done

The resulting file "100gb" occupies 404 KB in an ext4 filesystem.
(It shall not contain entirely zeros. So it is not the smallest possible
100 GiB file.)


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#236745 — Re: Useful use of dd

[Multipart message — attachments visible in raw view] — view raw

On Fri, Jul 02, 2021 at 12:52:53PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> Teemu Likonen wrote:
> > For this new subject I will add another use: quickly create empty file
> > of specific size, for example 5 * 1024 bytes:
> >    $ dd of=empty obs=1024 seek=5 count=0
> 
> Not to forget creation of sparse files with few disk consumption.

Nice.

And then, there's "oflag=sync". I hate it when the system says
"Done!" when writing to an USB stick, then I say "sync", and only
then the writing starts. For an unknown amount of time. Combine
with "status=progress" to taste :)

> For testing zisofs i have this in a script:
> 
>   #!/bin/bash
>   # 100gb with big holes
>   for i in {1..100}
>   do
>     echo $i | dd of=100gb bs=1 seek="$i"G 2>/dev/null
>   done
> 
> The resulting file "100gb" occupies 404 KB in an ext4 filesystem.
> (It shall not contain entirely zeros. So it is not the smallest possible
> 100 GiB file.)

nice, thanks.

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

#236747 — Re: Useful use of dd

[Multipart message — attachments visible in raw view] — view raw

* 2021-07-02 12:52:53+0200, Thomas Schmitt wrote:

> Teemu Likonen wrote:
>> For this new subject I will add another use: quickly create empty file
>> of specific size, for example 5 * 1024 bytes:
>>    $ dd of=empty obs=1024 seek=5 count=0
>
> Not to forget creation of sparse files with few disk consumption.

My example above already does that.

    $ dd of=empty obs=100G seek=1 count=0
    $ ls -gGlhs empty 
    0 -rw-r--r-- 1 100G 2021-07-02 14:21:18 empty

Zero blocks actually allocated, so far.

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[toc] | [prev] | [next] | [standalone]

#236792 — Re: Useful use of dd

On 7/2/21 4:25 AM, Teemu Likonen wrote:
> * 2021-07-02 12:52:53+0200, Thomas Schmitt wrote:
> 
>> Teemu Likonen wrote:
>>> For this new subject I will add another use: quickly create empty file
>>> of specific size, for example 5 * 1024 bytes:
>>>     $ dd of=empty obs=1024 seek=5 count=0
>>
>> Not to forget creation of sparse files with few disk consumption.
> 
> My example above already does that.
> 
>      $ dd of=empty obs=100G seek=1 count=0
>      $ ls -gGlhs empty
>      0 -rw-r--r-- 1 100G 2021-07-02 14:21:18 empty
> 
> Zero blocks actually allocated, so far.


Thanks for the clue regarding viewing sparse files:

2021-07-02 14:39:52 dpchrist@dipsy ~/sandbox/dd
$ dd if=/dev/zero of=zero bs=1M count=5
5+0 records in
5+0 records out
5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0125097 s, 419 MB/s

2021-07-02 14:41:57 dpchrist@dipsy ~/sandbox/dd
$ dd if=/dev/urandom of=urandom bs=1M count=5
5+0 records in
5+0 records out
5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.026771 s, 196 MB/s

2021-07-02 14:44:11 dpchrist@dipsy ~/sandbox/dd
$ ls -gGls --block-size=1 [a-z]*
       0 -rw-r--r-- 1 5242880 Jul  2 14:21 dd-sparse
       0 -rw-r--r-- 1 5242880 Jul  2 14:24 truncate-sparse
5242880 -rw-r--r-- 1 5242880 Jul  2 14:42 urandom
5242880 -rw-r--r-- 1 5242880 Jul  2 14:40 zero


David

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.debian.user

csiph-web