Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #187248 > unrolled thread

dd to clone a drive

Back to article view | Back to linux.debian.user

Contents

  dd to clone a drive Jack Dangler <tdldev@gmail.com> - 2017-09-26 16:00 +0200
    Re: dd to clone a drive Roberto C. Sánchez <roberto@debian.org> - 2017-09-26 16:20 +0200
    Re: dd to clone a drive Michael Stone <mstone@debian.org> - 2017-09-26 16:20 +0200
    Re: dd to clone a drive "Thomas Schmitt" <scdbackup@gmx.net> - 2017-09-26 16:40 +0200
      Re: dd to clone a drive Michael Stone <mstone@debian.org> - 2017-09-26 17:30 +0200
    Re: dd to clone a drive Pascal Hambourg <pascal@plouf.fr.eu.org> - 2017-09-26 16:50 +0200
    Re: dd to clone a drive Gene Heskett <gheskett@shentel.net> - 2017-09-26 17:40 +0200
      Re: dd to clone a drive Jack Dangler <tdldev@gmail.com> - 2017-09-29 13:40 +0200

#187248 — dd to clone a drive

I have an existing drive near EOL (judging from the sounds). I got a 
replacement drive for it (same size).

I plugged the replacement into a USB port and started a byte-for-byte 
copy with

dd if=/dev/sda of=/dev/sdc

The process ran quietly for almost 30 hours with no discernable results 
so i killed it. Apparently, it had been running the whole time and 
resulted in approximately 300 of 500Gb copied. Is it 'usual' to have dd 
take upwards of 2 days to copy a drive ?

The source drive is a 500G 5400rpm WD, and the target is a 500G 7200rpm 
WD black.

Thanks for any input.

Regards

Jack

[toc] | [next] | [standalone]

#187250

On Tue, Sep 26, 2017 at 09:57:57AM -0400, Jack Dangler wrote:
> I have an existing drive near EOL (judging from the sounds). I got a
> replacement drive for it (same size).
> 
> I plugged the replacement into a USB port and started a byte-for-byte copy
> with
> 
> dd if=/dev/sda of=/dev/sdc
> 
> The process ran quietly for almost 30 hours with no discernable results so i
> killed it. Apparently, it had been running the whole time and resulted in
> approximately 300 of 500Gb copied. Is it 'usual' to have dd take upwards of
> 2 days to copy a drive ?
> 
> The source drive is a 500G 5400rpm WD, and the target is a 500G 7200rpm WD
> black.
> 

That all depends on many factors.  If your machine has poor I/O
performance to/from USB (which may be the case depending on whether it
is a USB 1.1, 2, or 3 port you chose and the chipset in the USB
enclosure) you might see transfer as low as 20 MB/sec.  If you do some
math:

(20 MB/sec) x (1 GB/1024 MB) x (3600 sec/hr) = ~70 GB/hr

If your machine is experiencing other loads, that may further reduce
performance and if the chipset in the USB enclosure or the USB
controller on the motherboard happens be buggy then it could be worse.

In your case with a failing drive in the mix, I suspect that the
controller is bogged down at least partially with attempting to do error
correction.  My experience with transferring an image of a failing drive
a few weeks ago was that it took far longer than I expected for a drive
of its size and performance characteristics.

There is an example given in the dd(1) man page for how to monitor the
progress of a dd operation:

       Sending a USR1 signal to a running 'dd' process makes it print
       I/O statistics to  standard  error  and  then resume copying.

              $ dd if=/dev/zero of=/dev/null& pid=$!
              $ kill -USR1 $pid; sleep 1; kill $pid

              18335302+0  records  in 18335302+0 records out 9387674624
              bytes (9.4 GB) copied, 34.6279 seconds, 271 MB/s

I like to use watch to give me periodic updates on the command.  You
could do something similar like this:

dd if=/dev/sda of=/dev/sdc& pid=$!
watch -n 60 kill -USR1 $pid

That will give you a running progress of the command at 60 second
intervals.  That would like you know if the process is uniformly slow or
if it is varying and you could fairly easily estimate the remaining
time.

Regards,

-Roberto
-- 
Roberto C. Sánchez

[toc] | [prev] | [next] | [standalone]

#187251

On Tue, Sep 26, 2017 at 09:57:57AM -0400, Jack Dangler wrote:
>I have an existing drive near EOL (judging from the sounds). I got a 
>replacement drive for it (same size).
>
>I plugged the replacement into a USB port and started a byte-for-byte 
>copy with
>
>dd if=/dev/sda of=/dev/sdc
>
>The process ran quietly for almost 30 hours with no discernable 
>results so i killed it. Apparently, it had been running the whole time 
>and resulted in approximately 300 of 500Gb copied. Is it 'usual' to 
>have dd take upwards of 2 days to copy a drive ?
>
>The source drive is a 500G 5400rpm WD, and the target is a 500G 
>7200rpm WD black.

Try adding bs=64k to the command line. Your current version copies 512 
bytes at a time, which is painfully slow. If both drives are USB and on 
the same bus, that's going to be another source of performance issues.  
With USB2 you'll get about 20MByte/s if both are on the same bus, and 
about 40MByte/s if there's just one. (You can see what bus you're 
attached to with with "lsusb", and you can use "lsusb -t" to see the 
speed the device is currently using in megabits per second. If you have 
other busses available you can try changing ports.)

Also, you can send dd the USR1 signal to get it to show the current 
progress. (Try something like "pkill -x -USR1 dd")

Mike Stone

[toc] | [prev] | [next] | [standalone]

#187253

Hi,

> Is it 'usual' to have dd take upwards of 2 days to copy a drive ?

Not really. An old 500 GiB SATA disk may deliver about 50 MiB/s.
That's 10000 seconds, a bit less than 3 hours.
(If you copy from filesystem to filesystem it may last much longer
 due to the lookup times in the trees.)

The first suspect for slow dd is small block size.
So you should in any case ask dd for larger chunks as already proposed
by Michael Stone. For copying Debian ISOs to USB sticks the FAQ proposes
4 MiB. But i think 1 MiB is surely enough:

  dd if=/dev/sda of=/dev/sdc bs=1M

Next you should get some progress indicating pacifier.
I see that Roberto C. Sánchez already showed the USR1 method to make dd
print progress.
Here is another way:

  dd if=/dev/sda bs=1M | pv | dd of=/dev/sdc bs=1M

(With if=/dev/zero and of=/dev/null i get 5 GiB/s here. So the pipes
 are not a problem in respect to throughput.)


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#187256

On Tue, Sep 26, 2017 at 04:28:10PM +0200, Thomas Schmitt wrote:
>The first suspect for slow dd is small block size.
>So you should in any case ask dd for larger chunks as already proposed
>by Michael Stone. For copying Debian ISOs to USB sticks the FAQ proposes
>4 MiB. But i think 1 MiB is surely enough:

32k is surely enough. :) You're not going to see much performance 
difference on a USB drive after 8k or so. Eventually the performance may 
actually start going down as the block size gets more ridiculous, and 
very large blocks will tend to exhibit inconsistent behavior as you're 
generally observing transient quirks in the interaction with the OS 
cache rather than real performance differences. (In fact, regardless of 
what block size you use in a this particular dd command line, the OS is 
going to repackage it into a different size that's optimal for the 
device before actually writing it out. In my experience that's usually 
something around 1KByte for USB drives. So why use a block larger than 
1k at all?  The point of isn't to send a giant block to the disk, it's 
to reduce the overhead of calling the kernel to write a block. With the 
default 512 byte blocks the overhead is large enough to restrict 
performance. After a few kbytes the overhead is low enough that the USB 
drive is the bottleneck. I have no idea why anyone would have suggested 
4M as a good starting point, but that's well into the range of 
diminishing returns.)

Back to the original question, it's also worth mentioning that if the 
source disk really is failing it might run extremely slowly as it 
retries blocks over and over again. Looking at the progress through one 
of the mechanisms already mentioned will give you an idea of whether the 
process has gotten stuck in that way. If that's the problem, you can 
just leave it running and if the disk isn't too bad it may eventually 
finish. If it errors out, adding "conv=noerror,sync" will make dd write 
zeroes to the new disk when it hits a bad block, but keep going. (Be 
aware that means there will spots with garbage data, maybe in a file, 
maybe making the filesystem unreadable.) Note that the entire input 
block of lost data will be zeroed, so you may want something like 
dd ibs=512 obs=64k conv=sync,noerror ...
to minimize the amount of lost data. There is no point to reading less 
than 512 bytes at a time, as the hardware block will be at least that 
large.

Mike Stone

[toc] | [prev] | [next] | [standalone]

#187254

Le 26/09/2017 à 15:57, Jack Dangler a écrit :
> I have an existing drive near EOL (judging from the sounds). I got a 
> replacement drive for it (same size).
> 
> I plugged the replacement into a USB port and started a byte-for-byte 
> copy with
> 
> dd if=/dev/sda of=/dev/sdc

dd is not the best tool to copy a failing disk. Other tools such as 
ddrescue and dcfldd have useful features to deal with flaky sectors.

Also, you are not doing this while any filesystem or anything on the 
disk is mounted or in use, are you ? Otherwise the copy could be in an 
inconsistent state.

[toc] | [prev] | [next] | [standalone]

#187261

On Tuesday 26 September 2017 09:57:57 Jack Dangler wrote:

> I have an existing drive near EOL (judging from the sounds). I got a
> replacement drive for it (same size).
>
> I plugged the replacement into a USB port and started a byte-for-byte
> copy with
>
> dd if=/dev/sda of=/dev/sdc
>
> The process ran quietly for almost 30 hours with no discernable
> results so i killed it. Apparently, it had been running the whole time
> and resulted in approximately 300 of 500Gb copied. Is it 'usual' to
> have dd take upwards of 2 days to copy a drive ?
>
> The source drive is a 500G 5400rpm WD, and the target is a 500G
> 7200rpm WD black.
>
> Thanks for any input.
>
I think when no bs size is specified, it does a sector copy and likely 
verifies it. In writing sd cards in a usb reader/writer here, dd's 
execution time can be cut to maybe 5 minutes for a 2GB image by the use 
of the "bs=4096" option in the above command line.  That should mean 
your 500GB copy operation would take about 20 hours, probably much less 
since the hd can write faster than my sd cards can on a sustained basis 
like 500GB.

> Regards
>
> Jack


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#187406

On 09/26/2017 11:35 AM, Gene Heskett wrote:
> On Tuesday 26 September 2017 09:57:57 Jack Dangler wrote:
>
>> I have an existing drive near EOL (judging from the sounds). I got a
>> replacement drive for it (same size).
>>
>> I plugged the replacement into a USB port and started a byte-for-byte
>> copy with
>>
>> dd if=/dev/sda of=/dev/sdc
>>
>> The process ran quietly for almost 30 hours with no discernable
>> results so i killed it. Apparently, it had been running the whole time
>> and resulted in approximately 300 of 500Gb copied. Is it 'usual' to
>> have dd take upwards of 2 days to copy a drive ?
>>
>> The source drive is a 500G 5400rpm WD, and the target is a 500G
>> 7200rpm WD black.
>>
>> Thanks for any input.
>>
> I think when no bs size is specified, it does a sector copy and likely
> verifies it. In writing sd cards in a usb reader/writer here, dd's
> execution time can be cut to maybe 5 minutes for a 2GB image by the use
> of the "bs=4096" option in the above command line.  That should mean
> your 500GB copy operation would take about 20 hours, probably much less
> since the hd can write faster than my sd cards can on a sustained basis
> like 500GB.
>
>> Regards
>>
>> Jack
>
> Cheers, Gene Heskett
Thanks to Roberto Sanchez, Michael Stone, Thomas Schmitt, Pascal 
Hambourg, Michael Stone, and Gene Haskett for the great input! A little 
more reading and a little experimentation, and I've got this working 
well. Changed the process so that each partition was separately cloned, 
but also changed block sizes and made sure that both source and target 
were unmounted. The partitions took far less time than the original, and 
the verification was successful on all. I even cloned my entire 
installation and booted from it to make sure that it, too, would 
function. Thanks again for all the invaluable help!

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web