Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #236574 > unrolled thread

Whole Disk Encryption + SSD

Back to article view | Back to linux.debian.user

Contents

  Whole Disk Encryption + SSD piorunz <piorunz@gmx.com> - 2021-06-28 17:00 +0200
    Re: Whole Disk Encryption + SSD Jochen Spieker <ml@well-adjusted.de> - 2021-06-28 18:00 +0200
    Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-06-28 22:40 +0200
      Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-06-29 00:00 +0200
      Re: Whole Disk Encryption + SSD Stefan Monnier <monnier@iro.umontreal.ca> - 2021-06-29 02:00 +0200
        Re: Whole Disk Encryption + SSD <tomas@tuxteam.de> - 2021-06-29 09:50 +0200
          Re: Whole Disk Encryption + SSD Stefan Monnier <monnier@iro.umontreal.ca> - 2021-06-30 00:50 +0200
          Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-06-30 01:00 +0200
      Re: Whole Disk Encryption + SSD Jonathan Dowland <jon+debian-user@dow.land> - 2021-06-29 11:30 +0200
      Re: Whole Disk Encryption + SSD piorunz <piorunz@gmx.com> - 2021-06-29 14:10 +0200
        Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-06-30 02:20 +0200
      Re: Whole Disk Encryption + SSD David Wright <deblis@lionunicorn.co.uk> - 2021-07-02 05:00 +0200
        Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 05:50 +0200
          Useless use of "dd" Teemu Likonen <tlikonen@iki.fi> - 2021-07-02 07:50 +0200
            Useful use of dd [was: Useless use of "dd"] <tomas@tuxteam.de> - 2021-07-02 09:50 +0200
              Re: Useful use of dd [was: Useless use of "dd"] Teemu Likonen <tlikonen@iki.fi> - 2021-07-02 12:30 +0200
                Re: Useful use of dd "Thomas Schmitt" <scdbackup@gmx.net> - 2021-07-02 13:00 +0200
                  Re: Useful use of dd <tomas@tuxteam.de> - 2021-07-02 13:20 +0200
                  Re: Useful use of dd Teemu Likonen <tlikonen@iki.fi> - 2021-07-02 13:30 +0200
                    Re: Useful use of dd David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 23:50 +0200
                Re: Useful use of dd [was: Useless use of "dd"] <tomas@tuxteam.de> - 2021-07-02 13:20 +0200
                Re: Useful use of dd Michael Stone <mstone@debian.org> - 2021-07-02 14:50 +0200
                Re: Useful use of dd [was: Useless use of "dd"] Kushal Kumaran <kushal@locationd.net> - 2021-07-02 21:10 +0200
                  Creation of empty files [was: Useful use of dd] David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 23:40 +0200
                    Re: Creation of empty files [was: Useful use of dd] David Christensen <dpchrist@holgerdanske.com> - 2021-07-03 00:00 +0200
                    Re: Creation of empty files [was: Useful use of dd] Michael Stone <mstone@debian.org> - 2021-07-03 15:50 +0200
                      Re: Creation of empty files [was: Useful use of dd] David Christensen <dpchrist@holgerdanske.com> - 2021-07-03 23:40 +0200
                        Re: Creation of empty files [was: Useful use of dd] Michael Stone <mstone@debian.org> - 2021-07-04 06:00 +0200
              Re: Useful use of dd [was: Useless use of "dd"] Greg Wooledge <greg@wooledge.org> - 2021-07-02 13:40 +0200
                Re: Useful use of dd [was: Useless use of "dd"] <tomas@tuxteam.de> - 2021-07-02 13:40 +0200
                  Re: Useful use of dd [was: Useless use of "dd"] David Wright <deblis@lionunicorn.co.uk> - 2021-07-02 17:10 +0200
                    Re: Useful use of dd [was: Useless use of "dd"] <tomas@tuxteam.de> - 2021-07-02 17:40 +0200
                    Re: Useful use of dd [was: Useless use of "dd"] Gunnar Gervin <dofeelok@gmail.com> - 2021-07-03 09:20 +0200
                      Re: Useful use of dd [was: Useless use of "dd"] <tomas@tuxteam.de> - 2021-07-03 09:30 +0200
              Useless use of shell pipelines [was: Useful use of dd] David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 23:40 +0200
                Re: Useless use of shell pipelines [was: Useful use of dd] l0f4r0@tuta.io - 2021-07-03 10:30 +0200
                  Re: Useless use of shell pipelines [was: Useful use of dd] Greg Wooledge <greg@wooledge.org> - 2021-07-03 15:40 +0200
            Re: Useless use of "dd" David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 21:40 +0200
          Re: Whole Disk Encryption + SSD David Wright <deblis@lionunicorn.co.uk> - 2021-07-02 17:10 +0200
            Re: Whole Disk Encryption + SSD Michael Stone <mstone@debian.org> - 2021-07-02 17:30 +0200
            Re: Whole Disk Encryption + SSD David Christensen <dpchrist@holgerdanske.com> - 2021-07-02 20:40 +0200

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#236744 — Re: Useful use of dd [was: Useless use of "dd"]

[Multipart message — attachments visible in raw view] — view raw

On Fri, Jul 02, 2021 at 01:26:23PM +0300, Teemu Likonen wrote:
> * 2021-07-02 09:49:23+0200, tomas@tuxteam.de wrote:
> 
> > FWIW, I've found something which could be deemed to be an "useful use
> > of dd" which somehow bears a hidden symmetry. As a replacement for
> > `cat' whenever you need to put a name on the output file.
> 
> For this new subject I will add another use: quickly create empty file
> of specific size, for example 5 * 1024 bytes:
> 
>     $ dd of=empty obs=1024 seek=5 count=0

Clever :)

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

#236752 — Re: Useful use of dd

On Fri, Jul 02, 2021 at 01:26:23PM +0300, Teemu Likonen wrote:                                                                                                                
>* 2021-07-02 09:49:23+0200, tomas@tuxteam.de wrote:                                                                                                                          
>                                                                                                                                                                             
>> FWIW, I've found something which could be deemed to be an "useful use                                                                                                      
>> of dd" which somehow bears a hidden symmetry. As a replacement for                                                                                                         
>> `cat' whenever you need to put a name on the output file.                                                                                                                  
>                                                                                                                                                                             
>For this new subject I will add another use: quickly create empty file                                                                                                       
>of specific size, for example 5 * 1024 bytes:                                                                                                                                
>                                                                                                                                                                             
>    $ dd of=empty obs=1024 seek=5 count=0                                                                                                                                    

For this it is IMO more clear to use truncate:
truncate -s 5k empty

On Fri, Jul 02, 2021 at 07:31:24AM -0400, Greg Wooledge wrote:
>On Fri, Jul 02, 2021 at 09:49:23AM +0200, tomas@tuxteam.de wrote:
>>   sudo dd of=/etc/hosts oflags=append
>
>This appears to be a typo for "oflag=append", which is a GNU extension,
>not part of the standard POSIX dd.  No wonder I didn't know about it. ;-)

Because you've been asleep since the 80s when POSIX was relevant? :-D

>The bullseye version of GNU coreutils dd also gives a warning message
>with this particular flag:
>
>unicorn:~$ echo g | dd of=foo oflag=append
>dd: you probably want conv=notrunc with oflag=append
>0+1 records in
>0+1 records out
>2 bytes copied, 9.141e-05 s, 21.9 kB/s
>
>Not sure if that's unique to bullseye.  But anyway, yes, the warning
>is accurate; without conv=notrunc, my test file "foo" got overwritten
>with just the "g".

It's been like that for more than a decade, sarge in the debian 
timeline. The warning was added for obvious reasons. :-)  (There is a 
narrow use case for append without notrunc, which is why it's a warning 
and not a fatal error.)

[toc] | [prev] | [next] | [standalone]

#236779 — Re: Useful use of dd [was: Useless use of "dd"]

On Fri, Jul 02 2021 at 01:26:23 PM, Teemu Likonen <tlikonen@iki.fi> wrote:
> * 2021-07-02 09:49:23+0200, tomas@tuxteam.de wrote:
>
>> FWIW, I've found something which could be deemed to be an "useful use
>> of dd" which somehow bears a hidden symmetry. As a replacement for
>> `cat' whenever you need to put a name on the output file.
>
> For this new subject I will add another use: quickly create empty file
> of specific size, for example 5 * 1024 bytes:
>
>     $ dd of=empty obs=1024 seek=5 count=0

There's

    $ truncate --size 5k testfile

for this purpose.

-- 
regards,
kushal

[toc] | [prev] | [next] | [standalone]

#236789 — Creation of empty files [was: Useful use of dd]

On 7/2/21 11:51 AM, Kushal Kumaran wrote:
> On Fri, Jul 02 2021 at 01:26:23 PM, Teemu Likonen <tlikonen@iki.fi> wrote:

>> For this new subject I will add another use: quickly create empty file
>> of specific size, for example 5 * 1024 bytes:
>>
>>      $ dd of=empty obs=1024 seek=5 count=0


2021-07-02 14:20:47 dpchrist@dipsy ~/sandbox/dd
$ dd of=dd-sparse bs=1M seek=5 count=0
0+0 records in
0+0 records out
0 bytes copied, 6.9482e-05 s, 0.0 kB/s

2021-07-02 14:21:16 dpchrist@dipsy ~/sandbox/dd
$ ll dd-sparse
-rw-r--r-- 1 dpchrist dpchrist 5242880 2021-07-02 14:21:16 dd-sparse

2021-07-02 14:21:40 dpchrist@dipsy ~/sandbox/dd
$ du --bytes dd-sparse
5242880	dd-sparse


> There's
> 
>      $ truncate --size 5k testfile
> 
> for this purpose.


2021-07-02 14:23:34 dpchrist@dipsy ~/sandbox/dd
$ time truncate --size 5M truncate-sparse

real	0m0.002s
user	0m0.002s
sys	0m0.001s

2021-07-02 14:24:25 dpchrist@dipsy ~/sandbox/dd
$ ll truncate-sparse
-rw-r--r-- 1 dpchrist dpchrist 5242880 2021-07-02 14:24:25 truncate-sparse

2021-07-02 14:24:30 dpchrist@dipsy ~/sandbox/dd
$ du --bytes truncate-sparse
5242880	truncate-sparse


I expected sparse files, but du(1) does not indicate such (?).


Comments?


David

[toc] | [prev] | [next] | [standalone]

#236794 — Re: Creation of empty files [was: Useful use of dd]

On 7/2/21 2:30 PM, David Christensen wrote:

> I expected sparse files, but du(1) does not indicate such (?).
> 
> 
> Comments?

RTFM ls(1) and the '-s' and '--block-size' options.


David

[toc] | [prev] | [next] | [standalone]

#236803 — Re: Creation of empty files [was: Useful use of dd]

On Fri, Jul 02, 2021 at 02:30:50PM -0700, David Christensen wrote:
>2021-07-02 14:24:30 dpchrist@dipsy ~/sandbox/dd
>$ du --bytes truncate-sparse
>5242880	truncate-sparse
>
>
>I expected sparse files, but du(1) does not indicate such (?).

You used --bytes, which per the man page implies --apparent-size

[toc] | [prev] | [next] | [standalone]

#236823 — Re: Creation of empty files [was: Useful use of dd]

On 7/3/21 6:44 AM, Michael Stone wrote:
> On Fri, Jul 02, 2021 at 02:30:50PM -0700, David Christensen wrote:
>> 2021-07-02 14:24:30 dpchrist@dipsy ~/sandbox/dd
>> $ du --bytes truncate-sparse
>> 5242880    truncate-sparse
>>
>>
>> I expected sparse files, but du(1) does not indicate such (?).
> 
> You used --bytes, which per the man page implies --apparent-size


RTFM du(1):

       -b, --bytes
               equivalent to '--apparent-size --block-size=1'


2021-07-03 14:12:30 dpchrist@dipsy ~/sandbox/dd
$ du --block-size=1 [a-z]*
0	dd-sparse
0	truncate-sparse
5242880	urandom
5242880	zero


Thank you.  :-)


I suppose '-b' is Huffman Coding [1] for somebody's use case (?).


I prefer the Principle of Least Surprise [2] -- 'du' means "disk usage".


David


[1] https://en.wikipedia.org/wiki/Huffman_encoding

[2] https://en.wikipedia.org/wiki/Principle_of_least_surprise

[toc] | [prev] | [next] | [standalone]

#236831 — Re: Creation of empty files [was: Useful use of dd]

On Sat, Jul 03, 2021 at 02:34:56PM -0700, David Christensen wrote:
>On 7/3/21 6:44 AM, Michael Stone wrote:
>>On Fri, Jul 02, 2021 at 02:30:50PM -0700, David Christensen wrote:
>>>2021-07-02 14:24:30 dpchrist@dipsy ~/sandbox/dd
>>>$ du --bytes truncate-sparse
>>>5242880    truncate-sparse
>>>
>>>
>>>I expected sparse files, but du(1) does not indicate such (?).
>>
>>You used --bytes, which per the man page implies --apparent-size
>
>
>RTFM du(1):
>
>      -b, --bytes
>              equivalent to '--apparent-size --block-size=1'
>
>
>2021-07-03 14:12:30 dpchrist@dipsy ~/sandbox/dd
>$ du --block-size=1 [a-z]*
>0	dd-sparse
>0	truncate-sparse
>5242880	urandom
>5242880	zero
>
>
>Thank you.  :-)
>
>
>I suppose '-b' is Huffman Coding [1] for somebody's use case (?).
>
>
>I prefer the Principle of Least Surprise [2] -- 'du' means "disk usage".

In the case that a file doesn't contain an even multiple of the 
filesystem block size, du will essentially round up the size of every 
file. If for some reason you want to know the cumulative lengths of a 
set of files rather than the number of blocks used, maybe to put the 
files into a serialized form for network copy or somesuch, du -b will do 
that thing. Although there may have been a more obsolete reason it was 
added.[1]

It was like that in the very first version of debian (via fileutils) 
then broken in later versions of fileutils, then fixed in coreutils to 
revert to the original behavior. The breakage came when du (and other 
utilities) gained the ability to use arbitrary block sizes; du -b was 
changed to mean "report blocks used, but display in units of 1 byte 
blocks". The problem is that the original meaning can only be achieved 
by looking at the file size rather than file blocks, while the new 
meaning looked at blocks and then converted into bytes. Different things 
for any file that isn't an even multiple of the block size, and very 
different for sparse files. When this was noticed it was put back the 
way it was, --apparent-size was added, and the man page was clarified.

The distinction was intuitive when du (and df!) only reported in blocks 
and is much more esoteric now that people usually use megabytes or such. 
Anyway, when you're looking at file size (vs blocks) you can't represent 
sparseness at the same time. The man page has gotten better over time to 
try to make it more obvious that a couple of things are going on when 
you use -b, but nobody wants a wall of text like this in a man page. 



[1] I suspect this all has to do with compatability with something 
ancient (posix docs refer to -b being considered then dropped) along the 
lines of "instead of arguing about what size blocks to report, just use 
bytes". (Yes, there was a block size war.) It tried to solve a real 
problem (though it wasn't widely adopted probably because it created a 
new problem) but the thing it was trying to solve became moot after 
nobody at all wanted to talk in terms of blocks. That was about the time 
that people started using megabytes as a common unit for file sizes and 
we could no longer get away with a 32 bit int for off_t on larger 
systems. Prior to that it was fairly common to think in terms of blocks. 
(E.g., it was how file sizes were displayed on vms.)

[toc] | [prev] | [next] | [standalone]

#236749 — Re: Useful use of dd [was: Useless use of "dd"]

On Fri, Jul 02, 2021 at 09:49:23AM +0200, tomas@tuxteam.de wrote:
>   sudo dd of=/etc/hosts oflags=append

This appears to be a typo for "oflag=append", which is a GNU extension,
not part of the standard POSIX dd.  No wonder I didn't know about it. ;-)

The bullseye version of GNU coreutils dd also gives a warning message
with this particular flag:

unicorn:~$ echo g | dd of=foo oflag=append
dd: you probably want conv=notrunc with oflag=append
0+1 records in
0+1 records out
2 bytes copied, 9.141e-05 s, 21.9 kB/s

Not sure if that's unique to bullseye.  But anyway, yes, the warning
is accurate; without conv=notrunc, my test file "foo" got overwritten
with just the "g".

[toc] | [prev] | [next] | [standalone]

#236750 — Re: Useful use of dd [was: Useless use of "dd"]

[Multipart message — attachments visible in raw view] — view raw

On Fri, Jul 02, 2021 at 07:31:24AM -0400, Greg Wooledge wrote:
> On Fri, Jul 02, 2021 at 09:49:23AM +0200, tomas@tuxteam.de wrote:
> >   sudo dd of=/etc/hosts oflags=append
> 
> This appears to be a typo for "oflag=append", which is a GNU extension,
> not part of the standard POSIX dd.  No wonder I didn't know about it. ;-)
> 
> The bullseye version of GNU coreutils dd also gives a warning message
> with this particular flag:
> 
> unicorn:~$ echo g | dd of=foo oflag=append
> dd: you probably want conv=notrunc with oflag=append
> 0+1 records in
> 0+1 records out
> 2 bytes copied, 9.141e-05 s, 21.9 kB/s
> 
> Not sure if that's unique to bullseye.  But anyway, yes, the warning
> is accurate; without conv=notrunc, my test file "foo" got overwritten
> with just the "g".

Eek. Thanks.

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

#236755 — Re: Useful use of dd [was: Useless use of "dd"]

On Fri 02 Jul 2021 at 13:35:17 (+0200), tomas@tuxteam.de wrote:
> On Fri, Jul 02, 2021 at 07:31:24AM -0400, Greg Wooledge wrote:
> > On Fri, Jul 02, 2021 at 09:49:23AM +0200, tomas@tuxteam.de wrote:
> > >   sudo dd of=/etc/hosts oflags=append
> > 
> > This appears to be a typo for "oflag=append", which is a GNU extension,
> > not part of the standard POSIX dd.  No wonder I didn't know about it. ;-)
> > 
> > The bullseye version of GNU coreutils dd also gives a warning message
> > with this particular flag:
> > 
> > unicorn:~$ echo g | dd of=foo oflag=append
> > dd: you probably want conv=notrunc with oflag=append
> > 0+1 records in
> > 0+1 records out
> > 2 bytes copied, 9.141e-05 s, 21.9 kB/s
> > 
> > Not sure if that's unique to bullseye.  But anyway, yes, the warning
> > is accurate; without conv=notrunc, my test file "foo" got overwritten
> > with just the "g".
> 
> Eek. Thanks.

This reminds me of the sentiment expressed in
https://lists.debian.org/debian-user/2019/08/msg01454.html

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#236762 — Re: Useful use of dd [was: Useless use of "dd"]

[Multipart message — attachments visible in raw view] — view raw

On Fri, Jul 02, 2021 at 10:02:49AM -0500, David Wright wrote:
> On Fri 02 Jul 2021 at 13:35:17 (+0200), tomas@tuxteam.de wrote:

[...]

> > Eek. Thanks.
> 
> This reminds me of the sentiment expressed in
> https://lists.debian.org/debian-user/2019/08/msg01454.html

:-)

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

#236798 — Re: Useful use of dd [was: Useless use of "dd"]

[Multipart message — attachments visible in raw view] — view raw

Hi, Debian user 'club':
How can I get off this list in Android mobile? It's filling my mailbox;
5-10 emails a day!!!
Gunnar

On Fri, 2 Jul 2021, 18:03 David Wright, <deblis@lionunicorn.co.uk> wrote:

> On Fri 02 Jul 2021 at 13:35:17 (+0200), tomas@tuxteam.de wrote:
> > On Fri, Jul 02, 2021 at 07:31:24AM -0400, Greg Wooledge wrote:
> > > On Fri, Jul 02, 2021 at 09:49:23AM +0200, tomas@tuxteam.de wrote:
> > > >   sudo dd of=/etc/hosts oflags=append
> > >
> > > This appears to be a typo for "oflag=append", which is a GNU extension,
> > > not part of the standard POSIX dd.  No wonder I didn't know about it.
> ;-)
> > >
> > > The bullseye version of GNU coreutils dd also gives a warning message
> > > with this particular flag:
> > >
> > > unicorn:~$ echo g | dd of=foo oflag=append
> > > dd: you probably want conv=notrunc with oflag=append
> > > 0+1 records in
> > > 0+1 records out
> > > 2 bytes copied, 9.141e-05 s, 21.9 kB/s
> > >
> > > Not sure if that's unique to bullseye.  But anyway, yes, the warning
> > > is accurate; without conv=notrunc, my test file "foo" got overwritten
> > > with just the "g".
> >
> > Eek. Thanks.
>
> This reminds me of the sentiment expressed in
> https://lists.debian.org/debian-user/2019/08/msg01454.html
>
> Cheers,
> David.
>
>

[toc] | [prev] | [next] | [standalone]

#236799 — Re: Useful use of dd [was: Useless use of "dd"]

[Multipart message — attachments visible in raw view] — view raw

On Sat, Jul 03, 2021 at 10:14:46AM +0300, Gunnar Gervin wrote:
> Hi, Debian user 'club':
> How can I get off this list in Android mobile?

No idea about Android mobile (whatever that is ;-) but every mail you
receive via this list carries the instructions. Look at the headers
(however your mail client may show them to you). You'll find a couple
of them starting with "List-", here they are for the mail I'm answering
to:

  List-Id: <debian-user.lists.debian.org>
  List-URL: <https://lists.debian.org/debian-user/>
  List-Post: <mailto:debian-user@lists.debian.org>
  List-Help: <mailto:debian-user-request@lists.debian.org?subject=help>
  List-Subscribe: <mailto:debian-user-request@lists.debian.org?subject=subscribe>
  List-Unsubscribe: <mailto:debian-user-request@lists.debian.org?subject=unsubscribe>

Of special interest is the last one, named "List-Unsubscribe". It says
<mailto:debian-user-request@lists.debian.org?subject=unsubscribe>. So
just do this: send a mail to <debian-user-request@lists.debian.org>
with the subject "unsubscribe".

Depending on your mail program, just clicking on this <mailto:> URL
does this for you.

You'll receive a confirmation mail (just to double-check that it was
you who sent the mail, as opposed to someone playing a prank on you).
Follow the instructions there, done.

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

#236791 — Useless use of shell pipelines [was: Useful use of dd]

On 7/2/21 12:49 AM, tomas@tuxteam.de wrote:

> Now to another pet peeve of mine: useless use of grep:
> 
>    grep xxxx <myfile> | sed -e 's/bla/foo/' ...


Perhaps the underlying issue is useless use of shell pipelines ("The 
Unix Way"):

2021-07-02 12:44:43 dpchrist@dipsy ~/sandbox/perl
$ cat useless-use-of-grep
This is the first line: bla bla bla.
This is line xxxx: bla bla bla.
This is the last line: bla bla bla.

2021-07-02 12:45:38 dpchrist@dipsy ~/sandbox/perl
$ grep xxxx useless-use-of-grep | sed -e 's/bla/foo/'
This is line xxxx: foo bla bla.


Versus "all-in-one":

2021-07-02 12:45:39 dpchrist@dipsy ~/sandbox/perl
$ perl -ne 's/bla/foo/ && print if /xxxx/' useless-use-of-grep
This is line xxxx: foo bla bla.


The former is better from a "golfing" standpoint (e.g. fewer characters 
to type).


But, it is interesting that the benchmark results defy the common 
perceptions of "userland tools are are fast" and "Perl is slow":

2021-07-02 12:59:55 dpchrist@dipsy ~/sandbox/perl
$ time for n in {1..10000}; do grep xxxx useless-use-of-grep | sed -e 
's/bla/foo/' > /dev/null ; done

real	0m42.162s
user	0m32.925s
sys	0m45.787s

2021-07-02 13:00:45 dpchrist@dipsy ~/sandbox/perl
$ time for n in {1..10000}; do perl -ne 's/bla/foo/ && print if /xxxx/' 
useless-use-of-grep > /dev/null; done

real	0m23.624s
user	0m14.308s
sys	0m10.826s


Perhaps this is because processing is trivial and a pipeline of two 
commands has twice the process creation costs of a single command (?).


David

[toc] | [prev] | [next] | [standalone]

#236800 — Re: Useless use of shell pipelines [was: Useful use of dd]

Hi,

2 juil. 2021, 23:30 de dpchrist@holgerdanske.com:

> Perhaps the underlying issue is useless use of shell pipelines ("The Unix Way"):
>
> 2021-07-02 12:44:43 dpchrist@dipsy ~/sandbox/perl
> $ cat useless-use-of-grep
> This is the first line: bla bla bla.
> This is line xxxx: bla bla bla.
> This is the last line: bla bla bla.
>
> 2021-07-02 12:45:38 dpchrist@dipsy ~/sandbox/perl
> $ grep xxxx useless-use-of-grep | sed -e 's/bla/foo/'
> This is line xxxx: foo bla bla.
>
>
> Versus "all-in-one":
>
> 2021-07-02 12:45:39 dpchrist@dipsy ~/sandbox/perl
> $ perl -ne 's/bla/foo/ && print if /xxxx/' useless-use-of-grep
> This is line xxxx: foo bla bla.
>
>
> The former is better from a "golfing" standpoint (e.g. fewer characters to type).
>
Here is a better `sed` if you want a all-in-one: 
sed -n /xxxx/{s/bla/foo/p} useless-use-of-grep

> But, it is interesting that the benchmark results defy the common perceptions of "userland tools are are fast" and "Perl is slow":
>
> 2021-07-02 12:59:55 dpchrist@dipsy ~/sandbox/perl
> $ time for n in {1..10000}; do grep xxxx useless-use-of-grep | sed -e 's/bla/foo/' > /dev/null ; done
>
> real	0m42.162s
> user	0m32.925s
> sys	0m45.787s
>
> 2021-07-02 13:00:45 dpchrist@dipsy ~/sandbox/perl
> $ time for n in {1..10000}; do perl -ne 's/bla/foo/ && print if /xxxx/' useless-use-of-grep > /dev/null; done
>
> real	0m23.624s
> user	0m14.308s
> sys	0m10.826s
>
I'm not in front of my Linux terminal, can someone launch the `time` command for my `sed` above please ? :)

Best regards, 
l0f4r0

[toc] | [prev] | [next] | [standalone]

#236802 — Re: Useless use of shell pipelines [was: Useful use of dd]

On Sat, Jul 03, 2021 at 10:23:39AM +0200, l0f4r0@tuta.io wrote:
> 2 juil. 2021, 23:30 de dpchrist@holgerdanske.com:
> > $ cat useless-use-of-grep
> > This is the first line: bla bla bla.
> > This is line xxxx: bla bla bla.
> > This is the last line: bla bla bla.

> Here is a better `sed` if you want a all-in-one: 
> sed -n /xxxx/{s/bla/foo/p} useless-use-of-grep

Also:
awk '/xxxx/{sub("bla", "foo"); print}' useless-use-of-grep

> > $ time for n in {1..10000}; do perl -ne 's/bla/foo/ && print if /xxxx/' useless-use-of-grep > /dev/null; done
> >
> > real	0m23.624s
> > user	0m14.308s
> > sys	0m10.826s

> I'm not in front of my Linux terminal, can someone launch the `time` command for my `sed` above please ? :)

unicorn:~$ time for n in {1..10000}; do perl -ne 's/bla/foo/ && print if /xxxx/' useless-use-of-grep; done >/dev/null
real 16.034  user 11.555  sys 4.847

unicorn:~$ time for n in {1..10000}; do sed -n '/xxxx/{s/bla/foo/p}' useless-use-of-grep; done >/dev/null
real 11.489  user 8.670  sys 3.251

unicorn:~$ time for n in {1..10000}; do awk '/xxxx/{sub("bla", "foo"); print}' useless-use-of-grep; done >/dev/null
real 15.690  user 11.575  sys 4.532

Sed wins this particular race, not surprisingly.

[toc] | [prev] | [next] | [standalone]

#236782 — Re: Useless use of "dd"

On 7/1/21 10:40 PM, Teemu Likonen wrote:
> * 2021-07-01 20:43:09-0700, David Christensen wrote:
> 
>> To "take an image", the script invokes dd(1) and pipes the output to
>> gzip(1), copying raw device octets to a file. To "restore an image",
>> the process is reversed.
> 
> Sounds like the classic "useless use of dd". For general educational
> purposes I remind (not necessarily you) that "dd" is just a file copying
> (and conversion) tool, almost like "cat". By default "dd" is a slow
> version of "cat" because it uses smaller buffers. "dd" can be made
> faster by increasing its buffer size but that will not make it more than
> just "cat".
> 
> So "dd" is not a special tool for accessing device files. Device files
> don't need special tools. They are files. The following command:
> 
>      dd if=/dev/sdX | gzip >image.gz
> 
> is slower but functionally the same as either of these:
> 
>      gzip </dev/sdX >image.gz
>      gzip --to-stdout /dev/sdX >image.gz
> 
> Usually we don't need "dd" for anything but there are some options which
> are useful sometimes. For example, "dd" can report progress to standard
> error stream and it can seek and limit read to some blocks or bytes.


My description was a high-level overview.  As always, "the devil is in 
the details".  dd(1) has features that would require extra work with 
cat(1) or gzip(1).  The script provides all of the options to dd(1) that 
you mention, and more.  'bs=1048576' gives good performance, makes math 
easy, and dovetails with 'parted ... u MiB'.  I also omitted 
functionality like MD5 and SHA256 checksum files, byte-for-byte 
verification, Bash code generation from Perl, and opportunities for 
concurrency (yet to be implemented).  It's all good stuff that you 
encounter when you've been doing imaging by hand and decide to "up your 
game" with scripting.


David

[toc] | [prev] | [next] | [standalone]

#236754

On Thu 01 Jul 2021 at 20:43:09 (-0700), David Christensen wrote:
> On 7/1/21 7:55 PM, David Wright wrote:
> > On Mon 28 Jun 2021 at 13:36:35 (-0700), David Christensen wrote:
> > 
> > > I do not set the 'discard' (trim) option in fstab(5).  If and when I
> > > want to erase unused blocks (such as before taking an image), I use
> > > fstrim(8).
> > 
> > Can you elaborate on a couple of things:
> > 
> > How do you "take an image". Is this equivalent to a conventional
> > dd if=/dev/sda …, or to some other process?
> 
> I wrote a script.  To "take an image", the script invokes dd(1) and
> pipes the output to gzip(1), copying raw device octets to a file.  To
> "restore an image", the process is reversed.
> 
> 
> > When I copy an entire conventional drive or partition, all the
> > free blocks/unused sectors are carefully transferred to the copy.
> 
> Same here.
> 
> 
> > What improvement does erasing unused blocks achieve?
> 
> Zero blocks are readily compressed, reducing the size of the image file.

Ah, I see. So a spinning rust analogy might be:

. Take an old conventional disk, sdz, where df shows 10% uasge,
. cp /dev/zero a-file-on-sdz
. rm a-file-on-sdz
. dd if=/dev/sdz … and compressing it, saving ~90% space.

But can I tweak my question a little—

Say that /dev/sdz contained one partition, sdz1, which filled the disk
and was a LUKS encrypted /home. In this case, the above would fail to
save space because the raw device sdz would be full of "random" data.

But what happens with an SSD? If, after the rm step above, you
# fstrim /home
the mountpoint, where /etc/fstab has the line
  /dev/mapper/luks-fedcba98-7654-3210-… LABEL1 ext4 /home
then what gets zeroed, the container or the contained?
I can't put my finger on any documentation that spells it out.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#236757

On Fri, Jul 02, 2021 at 10:02:18AM -0500, David Wright wrote:
>But what happens with an SSD? If, after the rm step above, you
># fstrim /home
>the mountpoint, where /etc/fstab has the line
>  /dev/mapper/luks-fedcba98-7654-3210-… LABEL1 ext4 /home
>then what gets zeroed

If everything's appropriately configured the free sectors will be marked 
as unused by the SSD and erased for reuse.

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | linux.debian.user

csiph-web