Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #181537 > unrolled thread

Trying to understand man page for dd

Back to article view | Back to linux.debian.user

Contents

  Trying to understand man page for dd Richard Owlett <rowlett@cloud85.net> - 2017-05-28 14:00 +0200
    Re: Trying to understand man page for dd JPlews <jw@plews.org.uk> - 2017-05-28 14:10 +0200
      Re: Trying to understand man page for dd Richard Owlett <rowlett@cloud85.net> - 2017-05-28 14:30 +0200
      Re: Trying to understand man page for dd Ulf Volmer <u.volmer@u-v.de> - 2017-06-02 00:00 +0200
        Re: Trying to understand man page for dd Curt <curty@free.fr> - 2017-06-03 14:00 +0200
    Re: Trying to understand man page for dd The Wanderer <wanderer@fastmail.fm> - 2017-05-28 14:10 +0200
      Re: Trying to understand man page for dd Richard Owlett <rowlett@cloud85.net> - 2017-05-28 14:30 +0200
    Re: Trying to understand man page for dd "Thomas Schmitt" <scdbackup@gmx.net> - 2017-05-28 14:50 +0200
      Re: Trying to understand man page for dd Richard Owlett <rowlett@cloud85.net> - 2017-05-28 15:00 +0200
    Re: Trying to understand man page for dd David <bouncingcats@gmail.com> - 2017-05-29 02:50 +0200
      Re: Trying to understand man page for dd "Thomas Schmitt" <scdbackup@gmx.net> - 2017-05-29 08:30 +0200
        Re: Trying to understand man page for dd Richard Owlett <rowlett@cloud85.net> - 2017-05-29 14:10 +0200
          Re: Trying to understand man page for dd David Wright <deblis@lionunicorn.co.uk> - 2017-05-29 15:40 +0200
            Re: Trying to understand man page for dd Richard Owlett <rowlett@cloud85.net> - 2017-05-29 16:20 +0200
              Re: Trying to understand man page for dd "Thomas Schmitt" <scdbackup@gmx.net> - 2017-05-29 17:20 +0200
              Re: Trying to understand man page for dd David Wright <deblis@lionunicorn.co.uk> - 2017-05-29 17:20 +0200
          Re: Trying to understand man page for dd David <bouncingcats@gmail.com> - 2017-05-30 04:30 +0200
      Re: Trying to understand man page for dd mbroe@oakland.edu - 2017-05-31 15:20 +0200

#181537 — Trying to understand man page for dd

It says in part:
> Sending a USR1 signal to a running 'dd' process makes it print I/O
> statistics to standard error and then resume copying.
>
>     $ dd if=/dev/zero of=/dev/null& pid=$!
>
>     $ kill -USR1 $pid; sleep 1; kill $pid

I think it is trying to tell me what I need to know.
But there is too much going on there,
Guidance please.
TIA

[toc] | [next] | [standalone]

#181538

On 28/05/17 12:58, Richard Owlett wrote:
> It says in part:
>> Sending a USR1 signal to a running 'dd' process makes it print I/O
>> statistics to standard error and then resume copying.
>>
>>     $ dd if=/dev/zero of=/dev/null& pid=$!
>>
>>     $ kill -USR1 $pid; sleep 1; kill $pid
> 
> I think it is trying to tell me what I need to know.
> But there is too much going on there,
> Guidance please.
> TIA
> 
> 
> 

have a look at status=progress

[toc] | [prev] | [next] | [standalone]

#181541

On 05/28/2017 07:05 AM, JPlews wrote:
> On 28/05/17 12:58, Richard Owlett wrote:
>> It says in part:
>>> Sending a USR1 signal to a running 'dd' process makes it print I/O
>>> statistics to standard error and then resume copying.
>>>
>>>     $ dd if=/dev/zero of=/dev/null& pid=$!
>>>
>>>     $ kill -USR1 $pid; sleep 1; kill $pid
>>
>> I think it is trying to tell me what I need to know.
>> But there is too much going on there,
>> Guidance please.
>> TIA
>>
>>
>>
>
> have a look at status=progress
>
>

Where?

[toc] | [prev] | [next] | [standalone]

#181675

On 05/28/2017 02:05 PM, JPlews wrote:

>>>     $ dd if=/dev/zero of=/dev/null& pid=$!
>>>     $ kill -USR1 $pid; sleep 1; kill $pid

> have a look at status=progress

This option does not exist in debian stable.

best regards
Ulf

[toc] | [prev] | [next] | [standalone]

#181715

On 2017-06-01, Ulf Volmer <u.volmer@u-v.de> wrote:
> On 05/28/2017 02:05 PM, JPlews wrote:
>
>>>>     $ dd if=/dev/zero of=/dev/null& pid=$!
>>>>     $ kill -USR1 $pid; sleep 1; kill $pid
>
>> have a look at status=progress
>
> This option does not exist in debian stable.

Would 'pv' help?

 dd if=/dev/zero | pv | of=/dev/null

Apparently if you know the approximate 'size' it will give you will an
ETA.

 dd if=/dev/sdb | pv -s 2G | dd of=DriveCopy1.dd bs=4096

It can also be used directly as in the man page example

 pv file | nc -w 1 somewhere.com 3000

In this latter case it evaluates the file size directly.


> best regards
> Ulf
>
>


-- 
"It might be a vision--of a shell, of a wheelbarrow, of a fairy kingdom on the
far side of the hedge; or it might be the glory of speed; no one knew." --Mrs.
Ramsay, speculating on why her little daughter might be dashing about, in "To
the Lighthouse," by Virginia Woolf.

[toc] | [prev] | [next] | [standalone]

#181539

[Multipart message — attachments visible in raw view] — view raw

On 2017-05-28 at 07:58, Richard Owlett wrote:

> It says in part:
> 
>> Sending a USR1 signal to a running 'dd' process makes it print I/O
>> statistics to standard error and then resume copying.
>>
>>     $ dd if=/dev/zero of=/dev/null& pid=$!
>>
>>     $ kill -USR1 $pid; sleep 1; kill $pid
> 
> I think it is trying to tell me what I need to know.
> But there is too much going on there,
> Guidance please.
> TIA

Can you clarify what it is that you don't understand, or what it is that
you need to know?

Because the part you quoted (which I do recognize, and have made use of
myself in the past) seems perfectly straightforward and understandable
to me, and you haven't provided any context to indicate what your
problem is or what it is that you don't understand.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#181540

On 05/28/2017 07:04 AM, The Wanderer wrote:
> On 2017-05-28 at 07:58, Richard Owlett wrote:
>
>> It says in part:
>>
>>> Sending a USR1 signal to a running 'dd' process makes it print I/O
>>> statistics to standard error and then resume copying.
>>>
>>>     $ dd if=/dev/zero of=/dev/null& pid=$!
>>>
>>>     $ kill -USR1 $pid; sleep 1; kill $pid
>>
>> I think it is trying to tell me what I need to know.
>> But there is too much going on there,
>> Guidance please.
>> TIA
>
> Can you clarify what it is that you don't understand, or what it is
> that you need to know?
>
> Because the part you quoted (which I do recognize, and have made use
> of myself in the past) seems perfectly straightforward and
> understandable to me, and you haven't provided any context to
> indicate what your problem is or what it is that you don't understand.
>

I suppose what I want is a translation of those two lines of script.
The man page just tosses it at me cold.
I didn't say what I was thinking because it would be pure noise.
Than man page was trying to convey something.
It did not.

[toc] | [prev] | [next] | [standalone]

#181542

Hi,

Richard Owlett wrote:
> It says in part:
> >     $ dd if=/dev/zero of=/dev/null& pid=$!
> >     $ kill -USR1 $pid; sleep 1; kill $pid
> I think it is trying to tell me what I need to know.

dd is started as background process, busily copying bytes from nowhere
to nowhere:

  dd if=/dev/zero of=/dev/null &

Because of the ampersand "&", the shell prompt comes back immediately
after the command line is interpreted and performed. This line contains
the additional assignment of the new process id number to variable "pid":

  pid=$!

The first command in the second line then sends signal USR1 to the
process that is busily copying bytes in background:

  kill -USR1 $pid

This causes the output lines shown in the example

  18335302+0  records  in  18335302+0 records out 9387674624 bytes
  (9.4 GB) copied, 34.6279 seconds, 271 MB/s

The second command simply waits a second before it ends:

  sleep 1

The third command sends signal TERM to the dd process:

  kill $pid

which causes it to end without message.


In the full documentation at
  https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation
a paragraph near the page end gives a more detailed example:
  "Sending an ‘INFO’ signal (or ‘USR1’ signal where that is unavailable)
   to a running dd process makes it print I/O statistics to standard error
   ...
   # Output stats every second.
   ...
   "

Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#181544

On 05/28/2017 07:40 AM, Thomas Schmitt wrote:
> Hi,
>
> Richard Owlett wrote:
>> It says in part:
>>>     $ dd if=/dev/zero of=/dev/null& pid=$!
>>>     $ kill -USR1 $pid; sleep 1; kill $pid
>> I think it is trying to tell me what I need to know.
>
> dd is started as background process, busily copying bytes from nowhere
> to nowhere:
>
>   dd if=/dev/zero of=/dev/null &
>
> Because of the ampersand "&", the shell prompt comes back immediately
> after the command line is interpreted and performed. This line contains
> the additional assignment of the new process id number to variable "pid":
>
>   pid=$!
>
> The first command in the second line then sends signal USR1 to the
> process that is busily copying bytes in background:
>
>   kill -USR1 $pid
>
> This causes the output lines shown in the example
>
>   18335302+0  records  in  18335302+0 records out 9387674624 bytes
>   (9.4 GB) copied, 34.6279 seconds, 271 MB/s
>
> The second command simply waits a second before it ends:
>
>   sleep 1
>
> The third command sends signal TERM to the dd process:
>
>   kill $pid
>
> which causes it to end without message.
>
>
> In the full documentation at
>   https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation
> a paragraph near the page end gives a more detailed example:
>   "Sending an ‘INFO’ signal (or ‘USR1’ signal where that is unavailable)
>    to a running dd process makes it print I/O statistics to standard error
>    ...
>    # Output stats every second.
>    ...
>    "
>
> Have a nice day :)
>
> Thomas
>
>

*THANK YOU* !!!
I was closer to understanding it than I thought.
I've done a first read of the link you gave.
Not so much for this particular question, but in combination with your 
explanation it will lead to "grokking" some other things.
[For non-R.A.Heinlein readers, see 
https://www.merriam-webster.com/dictionary/grok ;]

[toc] | [prev] | [next] | [standalone]

#181554

On 28 May 2017 at 21:58, Richard Owlett <rowlett@cloud85.net> wrote:
> It says in part:
>>
>> Sending a USR1 signal to a running 'dd' process makes it print I/O
>> statistics to standard error and then resume copying.
>>
>>     $ dd if=/dev/zero of=/dev/null& pid=$!
>>
>>     $ kill -USR1 $pid; sleep 1; kill $pid
>
>
> I think it is trying to tell me what I need to know.
> But there is too much going on there,
> Guidance please.

If you browse to explainshell.com and paste shell commands that you
don't understand,
it parses the commands and gives excerpts from relevant docs, which
might help you
in future.

I pasted your two command above, and received the following results.
(If perhaps these links dont work, just paste the commands yourself).

https://explainshell.com/explain?cmd=dd+if%3D%2Fdev%2Fzero+of%3D%2Fdev%2Fnull%26+pid%3D%24!

https://explainshell.com/explain?cmd=kill+-USR1+%24pid%3B+sleep+1%3B+kill+%24pid

[toc] | [prev] | [next] | [standalone]

#181556

Hi,

David wrote:
> https://explainshell.com/explain?cmd=dd+if%3D%2Fdev%2Fzero+of%3D%2Fdev%2Fnul
l%26+pid%3D%24!

For a machine this is not a bad answer.

> https://explainshell.com/explain?cmd=kill+-USR1+%24pid%3B+sleep+1%3B+kill+%2
4pid

But this one missed the point quite clearly.
(It gets no clue that the signals are sent to dd. So the result can only
 be unspecific info.)


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#181559

On 05/29/2017 01:20 AM, Thomas Schmitt wrote:
> Hi,
>
> David wrote:
>> https://explainshell.com/explain?cmd=dd+if%3D%2Fdev%2Fzero+of%3D%2Fdev%2Fnul
> l%26+pid%3D%24!
>
> For a machine this is not a bad answer.
>
>> https://explainshell.com/explain?cmd=kill+-USR1+%24pid%3B+sleep+1%3B+kill+%24pid
>
> But this one missed the point quite clearly.
> (It gets no clue that the signals are sent to dd. So the result can
> only be unspecific info.)
>

His post did have impact.

It reminded me of <https://explainshell.com>. I've used it a few times 
but it hadn't come to mind. If I had remembered, I would have asked a 
different {better????} question.

Your reply then caused me to re-read slowly and _carefully_ your 
previous post.

I think the is a subtle bug in the man page.
I introduces the two lines of code saying:
   > Sending a USR1 signal to a running 'dd' process makes it print
   > I/O statistics to standard error and then resume copying.
I believe that to accurately describe the code at
 
https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation
*NOT* the code on the man page.

I realize now that was the discrepancy that prompted my question.
I had gone to the man page to double check syntax. Spotted the code a 
thought "a way to display progress information when copying".

I can "sort of" understand syntactically correct code. The less said of 
my programming skills the better ;/ How I "read" the code just didn't 
jibe with the description.

Am I correct that there is a problem with the man page?

[toc] | [prev] | [next] | [standalone]

#181562

On Mon 29 May 2017 at 06:59:23 (-0500), Richard Owlett wrote:

> I think the is a subtle bug in the man page.
> I introduces the two lines of code saying:
>   > Sending a USR1 signal to a running 'dd' process makes it print
>   > I/O statistics to standard error and then resume copying.
> I believe that to accurately describe the code at
> 
> https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation
> *NOT* the code on the man page.

The two lines you quote don't "introduce" anything. They describe
the action of dd when you send a USR1 signal to it, a primary
purpose of the man page.

> I realize now that was the discrepancy that prompted my question.
> I had gone to the man page to double check syntax. Spotted the code
> a thought "a way to display progress information when copying".
> 
> I can "sort of" understand syntactically correct code. The less said
> of my programming skills the better ;/ How I "read" the code just
> didn't jibe with the description.
> 
> Am I correct that there is a problem with the man page?

Not that I can see. A terse illustration of program output might
be seen as an infrequent but welcome addition to a man page.
However, others might prefer not to lengthen the man page (which
is a one chunk affair) with tutorials, or even examples,
preferring to leave that for info nodes or other more extensive
documentation.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#181563

On 05/29/2017 08:36 AM, David Wright wrote:
> On Mon 29 May 2017 at 06:59:23 (-0500), Richard Owlett wrote:
>
>> I think the is a subtle bug in the man page.
>> I introduces the two lines of code saying:
>>   > Sending a USR1 signal to a running 'dd' process makes it print
>>   > I/O statistics to standard error and then resume copying.
>> I believe that to accurately describe the code at
>>
>> https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation
>> *NOT* the code on the man page.
>
> The two lines you quote don't "introduce" anything. They describe
> the action of dd when you send a USR1 signal to it, a primary
> purpose of the man page.
>
>> I realize now that was the discrepancy that prompted my question.
>> I had gone to the man page to double check syntax. Spotted the code
>> a thought "a way to display progress information when copying".
>>
>> I can "sort of" understand syntactically correct code. The less said
>> of my programming skills the better ;/ How I "read" the code just
>> didn't jibe with the description.
>>
>> Am I correct that there is a problem with the man page?
>
> Not that I can see. A terse illustration of program output might
> be seen as an infrequent but welcome addition to a man page.
> However, others might prefer not to lengthen the man page (which
> is a one chunk affair) with tutorials, or even examples,
> preferring to leave that for info nodes or other more extensive
> documentation.
>
> Cheers,
> David.
>
>

Not quite. It says "...and then resume copying."
The script copies bytes until the second line is encountered.
It outputs data and *STOPS DEAD*
richard@stretch-2nd:~$ dd if=/dev/zero of=/dev/null& pid=$!
[1] 3600
richard@stretch-2nd:~$ kill -USR1 $pid; sleep 1; kill $pid
53590611+0 records in
53590610+0 records out
27438392320 bytes (27 GB, 26 GiB) copied, 21.6988 s, 1.3 GB/s
richard@stretch-2nd:~$

[toc] | [prev] | [next] | [standalone]

#181566

Hi,

Richard Owlett wrote:
> Am I correct that there is a problem with the man page?

Not with that example. It works as to be expected from the description.
The example in the coreutils manual gives a better motivation why one
would want to send USR1 signals to the running dd process.


> The [example] script copies bytes until the second line is encountered.
> It outputs data and *STOPS DEAD*

That's because the second kill command sent the default signal TERM,
which other than USR1 causes dd to end.
Without the final
  kill $pid
the dd process would run in background until you shut down your computer.

If you send more USR1 by

  kill -USR1 $pid

before the TERM signal then you get more output lines with increasing
byte counts:

  106330856+0 records in
  106330856+0 records out
  54441398272 bytes (54 GB) copied, 21.859 s, 2.5 GB/s
  113139254+0 records in
  113139254+0 records out
  57927298048 bytes (58 GB) copied, 23.2586 s, 2.5 GB/s
  121925708+0 records in
  121925707+0 records out
  62425961984 bytes (62 GB) copied, 25.0622 s, 2.5 GB/s

That you get
  richard@stretch-2nd:~$
from the kill command line does not actually mean that the dd process
has ended (although it has in this special case). This is because of
the ampersand at the end of the dd command.

But the next shell command line after the end of the dd process should
report its demise before the commands in that line get executed:

  [1]+  Terminated              dd if=/dev/zero of=/dev/null


Further you may check by

  ps -ef | fgrep 'dd if=/dev/zero'

whether the process is still running. If so, you will see an output
line like

  thomas   12084  1716 99 17:06 pts/8    00:00:03 dd if=/dev/zero of=/dev/null

where 12084 is the process id number of the dd run.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#181567

On Mon 29 May 2017 at 09:13:21 (-0500), Richard Owlett wrote:
> On 05/29/2017 08:36 AM, David Wright wrote:
> >On Mon 29 May 2017 at 06:59:23 (-0500), Richard Owlett wrote:
> >
> >>I think the is a subtle bug in the man page.
> >>I introduces the two lines of code saying:
> >>  > Sending a USR1 signal to a running 'dd' process makes it print
> >>  > I/O statistics to standard error and then resume copying.
> >>I believe that to accurately describe the code at
> >>
> >>https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation
> >>*NOT* the code on the man page.
> >
> >The two lines you quote don't "introduce" anything. They describe
> >the action of dd when you send a USR1 signal to it, a primary
> >purpose of the man page.
> >
> >>I realize now that was the discrepancy that prompted my question.
> >>I had gone to the man page to double check syntax. Spotted the code
> >>a thought "a way to display progress information when copying".
> >>
> >>I can "sort of" understand syntactically correct code. The less said
> >>of my programming skills the better ;/ How I "read" the code just
> >>didn't jibe with the description.
> >>
> >>Am I correct that there is a problem with the man page?
> >
> >Not that I can see. A terse illustration of program output might
> >be seen as an infrequent but welcome addition to a man page.
> >However, others might prefer not to lengthen the man page (which
> >is a one chunk affair) with tutorials, or even examples,
> >preferring to leave that for info nodes or other more extensive
> >documentation.
> >
> >Cheers,
> >David.
> >
> >
> 
> Not quite. It says "...and then resume copying."
> The script copies bytes until the second line is encountered.
> It outputs data and *STOPS DEAD*
> richard@stretch-2nd:~$ dd if=/dev/zero of=/dev/null& pid=$!
> [1] 3600
> richard@stretch-2nd:~$ kill -USR1 $pid; sleep 1; kill $pid
> 53590611+0 records in
> 53590610+0 records out
> 27438392320 bytes (27 GB, 26 GiB) copied, 21.6988 s, 1.3 GB/s
> richard@stretch-2nd:~$

With respect, dd does not stop after printing out the statistics.
It carries on running exactly as the man page specifies.
You can demonstrate this for yourself by a small modification
of the final command: send the signal INT instead of TERM:

$ dd if=/dev/zero of=/dev/null& pid=$!
[1] 12121
$ kill -USR1 $pid; sleep 1; kill -INT $pid
8113119+0 records in
8113118+0 records out
4153916416 bytes (4.2 GB) copied, 10.6885 s, 389 MB/s
$ 8895227+0 records in
8895226+0 records out
4554355712 bytes (4.6 GB) copied, 11.714 s, 389 MB/s
                                                        ← I pressed <RETURN>
[1]+  Interrupt               dd if=/dev/zero of=/dev/null
$ 

So 0.4GB was copied during the one-second sleep because dd resumed
copying.

Alternatively, because there's no cost associated with typing
extra lines on your terminal, type the following, slowly:

$ dd if=/dev/zero of=/dev/null& pid=$!
$ kill -USR1 $pid
$ kill -USR1 $pid
$ kill -USR1 $pid # until you are convinced
$ kill -USR1 $pid
$ kill $pid

Once again, you've tried to overinterpret this:

  Sending a USR1 signal to a running 'dd' process makes it print
  I/O statistics to standard error and then resume copying.

as a description of this:

              $ dd if=/dev/zero of=/dev/null& pid=$!
              $ kill -USR1 $pid; sleep 1; kill $pid

It isn't. It's a description of the behaviour of dd.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#181581

On 29 May 2017 at 21:59, Richard Owlett <rowlett@cloud85.net> wrote:
>
> His post did have impact.

https://youtu.be/tNapRQILbTA?t=93

[toc] | [prev] | [next] | [standalone]

#181615

David <bouncingcats@gmail.com> writes:

> If you browse to explainshell.com and paste shell commands that you
> don't understand, it parses the commands and gives excerpts from
> relevant docs, which might help you in future.

Just a quick thank you for pointing out this resource! This will
definitely be going into my UNIX bag of tricks.
-- 
Mitchell Roe
mitchell.roe@member.fsf.org

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web