Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #4116 > unrolled thread

fast method of cloning a hard drive?

Back to article view | Back to comp.os.linux.misc

Contents

  fast method of cloning a hard drive? "Orson Cart" <ex-privat@parts.org> - 2012-02-05 13:46 +0100
    Re: fast method of cloning a hard drive? John Reiser <jreiserfl@comcast.net> - 2012-02-05 06:59 -0800
      Re: fast method of cloning a hard drive? unruh <unruh@invalid.ca> - 2012-02-06 01:03 +0000
        Re: fast method of cloning a hard drive? patrick <0001patrick@gmail.com> - 2012-02-06 21:04 +0000
          Re: fast method of cloning a hard drive? The Natural Philosopher <tnp@invalid.invalid> - 2012-02-06 21:50 +0000
          Re: fast method of cloning a hard drive? unruh <unruh@invalid.ca> - 2012-02-07 09:43 +0000
            Re: fast method of cloning a hard drive? patrick <0001patrick@gmail.com> - 2012-02-21 04:20 +0000
              Re: fast method of cloning a hard drive? unruh <unruh@invalid.ca> - 2012-02-21 05:40 +0000
      Re: fast method of cloning a hard drive? "Orson Cart" <ex-privat@parts.org> - 2012-02-06 14:11 +0100
    Re: fast method of cloning a hard drive? ray <ray@zianet.com> - 2012-02-05 15:15 +0000
    Re: fast method of cloning a hard drive? Dan C <youmustbejoking@lan.invalid> - 2012-02-05 16:30 +0000
    Re: fast method of cloning a hard drive? Robert Heller <heller@deepsoft.com> - 2012-02-05 10:55 -0600
      Re: fast method of cloning a hard drive? Johann Klammer <klammerj@NOSPAM.a1.net> - 2012-02-06 03:17 +0100
      Re: fast method of cloning a hard drive? Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> - 2012-02-06 23:38 +0100
        Re: fast method of cloning a hard drive? The Natural Philosopher <tnp@invalid.invalid> - 2012-02-06 23:01 +0000
          Re: fast method of cloning a hard drive? Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> - 2012-02-07 00:17 +0100
            Re: fast method of cloning a hard drive? The Natural Philosopher <tnp@invalid.invalid> - 2012-02-07 01:39 +0000
      Re: fast method of cloning a hard drive? Eli the Bearded <*@eli.users.panix.com> - 2012-02-06 23:50 +0000
        Re: fast method of cloning a hard drive? Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> - 2012-02-21 21:27 +0100

#4116 — fast method of cloning a hard drive?

I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
I use dd if=/dev/sdb of=/dev/sda
This is going for hours.  In system monitor, I see one CPU going
about 30% for 20 seconds, then another CPU goes 30% for 30 seconds.
Just flip-flop forever. The HD LED is flashing. 
System is Core 2 Quad with 6 GiB of RAM. 
I think if I copied it to a USB disk it would have taken less time...

[toc] | [next] | [standalone]

#4119

> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
Are the PATA and SATA interfaces native to the same CPU motherboard?

> I use dd if=/dev/sdb of=/dev/sda
> This is going for hours.  In system monitor, I see one CPU going
> about 30% for 20 seconds, then another CPU goes 30% for 30 seconds.
> Just flip-flop forever. The HD LED is flashing. 
It should be ON nearly all the time.  Any OFF duration should be
almost unnoticeable.

> System is Core 2 Quad with 6 GiB of RAM. 
> I think if I copied it to a USB disk it would have taken less time...

250GB in + 250GB out ==> 500GB transferred;  @50MB/s ==> 10,000 seconds;
about 2.8 hours.  Near-perfect overlap of physical read and write
still requires almost 1.5 hours.  The usual practice is to specify
"bs=32k" or "bs=64k" to reduce syscall overhead when using dd for
such a large copy.  (The default blocksize for dd is bs=512 [bytes].)
Or, use ordinary /bin/cp instead of dd.

-- 

[toc] | [prev] | [next] | [standalone]

#4131

On 2012-02-05, John Reiser <jreiserfl@comcast.net> wrote:
>> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
> Are the PATA and SATA interfaces native to the same CPU motherboard?
>
>> I use dd if=/dev/sdb of=/dev/sda
>> This is going for hours.  In system monitor, I see one CPU going
>> about 30% for 20 seconds, then another CPU goes 30% for 30 seconds.
>> Just flip-flop forever. The HD LED is flashing. 
> It should be ON nearly all the time.  Any OFF duration should be
> almost unnoticeable.
>
>> System is Core 2 Quad with 6 GiB of RAM. 
>> I think if I copied it to a USB disk it would have taken less time...
>
> 250GB in + 250GB out ==> 500GB transferred;  @50MB/s ==> 10,000 seconds;
> about 2.8 hours.  Near-perfect overlap of physical read and write
> still requires almost 1.5 hours.  The usual practice is to specify
> "bs=32k" or "bs=64k" to reduce syscall overhead when using dd for
> such a large copy.  (The default blocksize for dd is bs=512 [bytes].)
> Or, use ordinary /bin/cp instead of dd.

Use rsync not cp. Faster but also you will be sure that what is copied
is the same as what was original.


>

[toc] | [prev] | [next] | [standalone]

#4170

On Mon, 06 Feb 2012 01:03:52 +0000, unruh wrote:

> On 2012-02-05, John Reiser <jreiserfl@comcast.net> wrote:
>>> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
>> Are the PATA and SATA interfaces native to the same CPU motherboard?
>>
>>> I use dd if=/dev/sdb of=/dev/sda This is going for hours.  In system
>>> monitor, I see one CPU going about 30% for 20 seconds, then another
>>> CPU goes 30% for 30 seconds. Just flip-flop forever. The HD LED is
>>> flashing.
>> It should be ON nearly all the time.  Any OFF duration should be almost
>> unnoticeable.
>>
>>> System is Core 2 Quad with 6 GiB of RAM.
>>> I think if I copied it to a USB disk it would have taken less time...
>>
>> 250GB in + 250GB out ==> 500GB transferred;  @50MB/s ==> 10,000
>> seconds;
>> about 2.8 hours.  Near-perfect overlap of physical read and write still
>> requires almost 1.5 hours.  The usual practice is to specify "bs=32k"
>> or "bs=64k" to reduce syscall overhead when using dd for such a large
>> copy.  (The default blocksize for dd is bs=512 [bytes].) Or, use
>> ordinary /bin/cp instead of dd.
> 
> Use rsync not cp. Faster but also you will be sure that what is copied
> is the same as what was original.
> 
> 
>

rsync -av source target     is supposed to be equal to
cp -av source target


Both are quick.

[toc] | [prev] | [next] | [standalone]

#4171

patrick wrote:
> On Mon, 06 Feb 2012 01:03:52 +0000, unruh wrote:
> 
>> On 2012-02-05, John Reiser <jreiserfl@comcast.net> wrote:
>>>> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
>>> Are the PATA and SATA interfaces native to the same CPU motherboard?
>>>
>>>> I use dd if=/dev/sdb of=/dev/sda This is going for hours.  In system
>>>> monitor, I see one CPU going about 30% for 20 seconds, then another
>>>> CPU goes 30% for 30 seconds. Just flip-flop forever. The HD LED is
>>>> flashing.
>>> It should be ON nearly all the time.  Any OFF duration should be almost
>>> unnoticeable.
>>>
>>>> System is Core 2 Quad with 6 GiB of RAM.
>>>> I think if I copied it to a USB disk it would have taken less time...
>>> 250GB in + 250GB out ==> 500GB transferred;  @50MB/s ==> 10,000
>>> seconds;
>>> about 2.8 hours.  Near-perfect overlap of physical read and write still
>>> requires almost 1.5 hours.  The usual practice is to specify "bs=32k"
>>> or "bs=64k" to reduce syscall overhead when using dd for such a large
>>> copy.  (The default blocksize for dd is bs=512 [bytes].) Or, use
>>> ordinary /bin/cp instead of dd.
>> Use rsync not cp. Faster but also you will be sure that what is copied
>> is the same as what was original.
>>
>>
>>
> 
> rsync -av source target     is supposed to be equal to
> cp -av source target
> 
> 
> Both are quick.
> 
as long as the target is a formatted partition..

[toc] | [prev] | [next] | [standalone]

#4182

On 2012-02-06, patrick <0001patrick@gmail.com> wrote:
> On Mon, 06 Feb 2012 01:03:52 +0000, unruh wrote:
>
>> On 2012-02-05, John Reiser <jreiserfl@comcast.net> wrote:
>>>> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
>>> Are the PATA and SATA interfaces native to the same CPU motherboard?
>>>
>>>> I use dd if=/dev/sdb of=/dev/sda This is going for hours.  In system
>>>> monitor, I see one CPU going about 30% for 20 seconds, then another
>>>> CPU goes 30% for 30 seconds. Just flip-flop forever. The HD LED is
>>>> flashing.
>>> It should be ON nearly all the time.  Any OFF duration should be almost
>>> unnoticeable.
>>>
>>>> System is Core 2 Quad with 6 GiB of RAM.
>>>> I think if I copied it to a USB disk it would have taken less time...
>>>
>>> 250GB in + 250GB out ==> 500GB transferred;  @50MB/s ==> 10,000
>>> seconds;
>>> about 2.8 hours.  Near-perfect overlap of physical read and write still
>>> requires almost 1.5 hours.  The usual practice is to specify "bs=32k"
>>> or "bs=64k" to reduce syscall overhead when using dd for such a large
>>> copy.  (The default blocksize for dd is bs=512 [bytes].) Or, use
>>> ordinary /bin/cp instead of dd.
>> 
>> Use rsync not cp. Faster but also you will be sure that what is copied
>> is the same as what was original.
>> 
>> 
>>
>
> rsync -av source target     is supposed to be equal to
> cp -av source target

but it is not. It is a different program from cp.

>
>
> Both are quick.
>

[toc] | [prev] | [next] | [standalone]

#4397

On Tue, 07 Feb 2012 09:43:55 +0000, unruh wrote:

> On 2012-02-06, patrick <0001patrick@gmail.com> wrote:
>> On Mon, 06 Feb 2012 01:03:52 +0000, unruh wrote:
>>
>>> On 2012-02-05, John Reiser <jreiserfl@comcast.net> wrote:
>>>>> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
>>>> Are the PATA and SATA interfaces native to the same CPU motherboard?
>>>>
>>>>> I use dd if=/dev/sdb of=/dev/sda This is going for hours.  In system
>>>>> monitor, I see one CPU going about 30% for 20 seconds, then another
>>>>> CPU goes 30% for 30 seconds. Just flip-flop forever. The HD LED is
>>>>> flashing.
>>>> It should be ON nearly all the time.  Any OFF duration should be
>>>> almost unnoticeable.
>>>>
>>>>> System is Core 2 Quad with 6 GiB of RAM. I think if I copied it to a
>>>>> USB disk it would have taken less time...
>>>>
>>>> 250GB in + 250GB out ==> 500GB transferred;  @50MB/s ==> 10,000
>>>> seconds;
>>>> about 2.8 hours.  Near-perfect overlap of physical read and write
>>>> still requires almost 1.5 hours.  The usual practice is to specify
>>>> "bs=32k" or "bs=64k" to reduce syscall overhead when using dd for
>>>> such a large copy.  (The default blocksize for dd is bs=512 [bytes].)
>>>> Or, use ordinary /bin/cp instead of dd.
>>> 
>>> Use rsync not cp. Faster but also you will be sure that what is copied
>>> is the same as what was original.
>>> 
>>> 
>>> 
>>>
>> rsync -av source target     is supposed to be equal to cp -av source
>> target
> 
> but it is not. It is a different program from cp.

You're right.

cp -av source target
rsync -aHAX -v source target

Nothing wrong with the cp command.

[toc] | [prev] | [next] | [standalone]

#4399

On 2012-02-21, patrick <0001patrick@gmail.com> wrote:
> On Tue, 07 Feb 2012 09:43:55 +0000, unruh wrote:
>
>> On 2012-02-06, patrick <0001patrick@gmail.com> wrote:
>>> On Mon, 06 Feb 2012 01:03:52 +0000, unruh wrote:
>>>
>>>> On 2012-02-05, John Reiser <jreiserfl@comcast.net> wrote:
>>>>>> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
>>>>> Are the PATA and SATA interfaces native to the same CPU motherboard?
>>>>>
>>>>>> I use dd if=/dev/sdb of=/dev/sda This is going for hours.  In system
>>>>>> monitor, I see one CPU going about 30% for 20 seconds, then another
>>>>>> CPU goes 30% for 30 seconds. Just flip-flop forever. The HD LED is
>>>>>> flashing.
>>>>> It should be ON nearly all the time.  Any OFF duration should be
>>>>> almost unnoticeable.
>>>>>
>>>>>> System is Core 2 Quad with 6 GiB of RAM. I think if I copied it to a
>>>>>> USB disk it would have taken less time...
>>>>>
>>>>> 250GB in + 250GB out ==> 500GB transferred;  @50MB/s ==> 10,000
>>>>> seconds;
>>>>> about 2.8 hours.  Near-perfect overlap of physical read and write
>>>>> still requires almost 1.5 hours.  The usual practice is to specify
>>>>> "bs=32k" or "bs=64k" to reduce syscall overhead when using dd for
>>>>> such a large copy.  (The default blocksize for dd is bs=512 [bytes].)
>>>>> Or, use ordinary /bin/cp instead of dd.
>>>> 
>>>> Use rsync not cp. Faster but also you will be sure that what is copied
>>>> is the same as what was original.
>>>> 
>>>> 
>>>> 
>>>>
>>> rsync -av source target     is supposed to be equal to cp -av source
>>> target
>> 
>> but it is not. It is a different program from cp.
>
> You're right.
>
> cp -av source target
> rsync -aHAX -v source target
>
> Nothing wrong with the cp command.

except rsync also hashes the source and target to make sure that what is
transfered is the same as what was supposed to be transfered. 

And if the target is almost the same as source, it transfers only the
parts that are different.

[toc] | [prev] | [next] | [standalone]

#4154

John Reiser <jreiserfl@comcast.net> wrote:
>> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
>Are the PATA and SATA interfaces native to the same CPU motherboard?
>
Um no, it has ICH8 or ICH9 which have no IDE anymore, so there is a Jmicron
chip for the legacy I/O.           

[toc] | [prev] | [next] | [standalone]

#4120

On Sun, 05 Feb 2012 13:46:29 +0100, Orson Cart wrote:

> I try to clone a 250 GB disk PATA to a SATA drive with same CHS. I use
> dd if=/dev/sdb of=/dev/sda
> This is going for hours.  In system monitor, I see one CPU going about
> 30% for 20 seconds, then another CPU goes 30% for 30 seconds. Just
> flip-flop forever. The HD LED is flashing. System is Core 2 Quad with 6
> GiB of RAM. I think if I copied it to a USB disk it would have taken
> less time...

I doubt you have 250gb of data. It would probably have been much faster 
to use partimage to backup the partition(s) to an external usb drive and 
then restore to the new one.

[toc] | [prev] | [next] | [standalone]

#4121

On Sun, 05 Feb 2012 13:46:29 +0100, Orson Cart wrote:

> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
> I use dd if=/dev/sdb of=/dev/sda This is going for hours.  In system
> monitor, I see one CPU going about 30% for 20 seconds, then another CPU
> goes 30% for 30 seconds. Just flip-flop forever. The HD LED is flashing.
> System is Core 2 Quad with 6 GiB of RAM.
> I think if I copied it to a USB disk it would have taken less time...

Clonezilla.


-- 
"Ubuntu" -- an African word, meaning "Slackware is too hard for me".
"Bother!" said Pooh, as Lorena Bobbitt tied him to the bed.
Usenet Improvement Project: http://twovoyagers.com/improve-usenet.org/
Thanks, Obama: http://brandybuck.site40.net/pics/politica/thanks.jpg

[toc] | [prev] | [next] | [standalone]

#4123

At 5 Feb 2012 13:46:29 +0100 "Orson Cart" <ex-privat@parts.org> wrote:

> 
> 
> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
> I use dd if=/dev/sdb of=/dev/sda
> This is going for hours.  In system monitor, I see one CPU going
> about 30% for 20 seconds, then another CPU goes 30% for 30 seconds.
> Just flip-flop forever. The HD LED is flashing. 
> System is Core 2 Quad with 6 GiB of RAM. 
> I think if I copied it to a USB disk it would have taken less time...

Unless the source disk is 100% full, doing a file system copy (eg tar,
dump/restore, cpio, etc.) will almost allways be faster.  Much of your
transfer file will be copying empty (unused) sectors (a complete waste
of time) and stuff like the swap partition(s) (also a complete waste of
time). Yes, going file system by file system is more tedious, but you
will save lots of time...

>                                                                                                        

-- 
Robert Heller             -- 978-544-6933 / heller@deepsoft.com
Deepwoods Software        -- http://www.deepsoft.com/
()  ascii ribbon campaign -- against html e-mail
/\  www.asciiribbon.org   -- against proprietary attachments


                                                                     

[toc] | [prev] | [next] | [standalone]

#4133

Robert Heller wrote:
> At 5 Feb 2012 13:46:29 +0100 "Orson Cart"<ex-privat@parts.org>  wrote:
>
>
> Unless the source disk is 100% full, doing a file system copy (eg tar,
> dump/restore, cpio, etc.) will almost allways be faster.  Much of your
> transfer file will be copying empty (unused) sectors (a complete waste
> of time) and stuff like the swap partition(s) (also a complete waste of
> time). Yes, going file system by file system is more tedious, but you
> will save lots of time...
>

Hi there,

File system copy is definitely the way to go. But...
You have to be careful what and how you copy things. I vaguely remember 
ending up with a unusable system after such a stunt.

The Hard-Disk-Upgrade HOWTO is a decent starting point for those cp... 
incantations.

Regarding speed, the LINUX vm system is IMHO broken. It likes to retain 
rarely used recently read pages in memory, while happily swapping out 
application code. This generates unnecessary pagefaults and slows down 
the system. For sequential reads/writes this is somewhat deadly.

One may do
'watch echo 3 \> /proc/sys/vm/drop_caches'
as root to make the system more responsive in those cases. The command 
drops clean filesystem caches every 2 seconds. It helps for a lot of 
filesystem intensive things, unless you manage to fill up the pagecache 
in less than two seconds.

When using dd, be sure to read its manpage, especially the part about 
direct IO. The differences are significant.

regards,
JK

[toc] | [prev] | [next] | [standalone]

#4172

Hello,

Robert Heller a écrit :
> At 5 Feb 2012 13:46:29 +0100 "Orson Cart" <ex-privat@parts.org> wrote:
>>
>> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
>> I use dd if=/dev/sdb of=/dev/sda
[...]
> Unless the source disk is 100% full, doing a file system copy (eg tar,
> dump/restore, cpio, etc.) will almost allways be faster.

Or unless the data on the filesystem are horribly fragmented, e.g. lots
of small files. Access time kills the throughput on magnetic disks.

[toc] | [prev] | [next] | [standalone]

#4175

Pascal Hambourg wrote:
> Hello,
> 
> Robert Heller a écrit :
>> At 5 Feb 2012 13:46:29 +0100 "Orson Cart" <ex-privat@parts.org> wrote:
>>> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
>>> I use dd if=/dev/sdb of=/dev/sda
> [...]
>> Unless the source disk is 100% full, doing a file system copy (eg tar,
>> dump/restore, cpio, etc.) will almost allways be faster.
> 
> Or unless the data on the filesystem are horribly fragmented, e.g. lots
> of small files. Access time kills the throughput on magnetic disks.

BUT at least you don't end up with a fragged target disk.

And its arguable with decent caching and RAM whether reading a fragged 
disk is all that much slower anyway.

[toc] | [prev] | [next] | [standalone]

#4176

The Natural Philosopher a écrit :
> 
> And its arguable with decent caching and RAM whether reading a fragged 
> disk is all that much slower anyway.

How does decent caching and RAM help in that matter ?

[toc] | [prev] | [next] | [standalone]

#4179

Pascal Hambourg wrote:
> The Natural Philosopher a écrit :
>> And its arguable with decent caching and RAM whether reading a fragged 
>> disk is all that much slower anyway.
> 
> How does decent caching and RAM help in that matter ?
  the subsystem reads whole blocks and holds them in RAM till they are 
needed.

[toc] | [prev] | [next] | [standalone]

#4177

In comp.os.linux.misc, Robert Heller  <heller@deepsoft.com> wrote:
> At 5 Feb 2012 13:46:29 +0100 "Orson Cart" <ex-privat@parts.org> wrote:
> > I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
> > I use dd if=/dev/sdb of=/dev/sda

Set the block size to match the blocks used by the actual devices. For
historical reasons dd defaults to a block size of 1/2 kilobyte. Modern
devices are more likely to use a block size of 8 kilobytes.

	bs=8k

If the two devices have different block sizes, dd can rechunk for you
by setting separate ibs=N (input block size) and obs=M (output) values.

> Unless the source disk is 100% full, doing a file system copy (eg tar,
> dump/restore, cpio, etc.) will almost allways be faster.  Much of your
> transfer file will be copying empty (unused) sectors (a complete waste

Faster != clone. If you are doing forensic analysis, the "empty" sectors
are unlikely a waste.

Faster is not certain. Some filesystems may be faster to copy as raw data
than as individual files. Consider ext2 with tens of thousands of files
in a single directory. I've seen cpio (and rsync) run out of memory when
trying to deal with filesystems that employ millions of hard linked
files: those programs try to cache the hard link information.

> of time) and stuff like the swap partition(s) (also a complete waste of
> time). Yes, going file system by file system is more tedious, but you
> will save lots of time...

dd can copy filesystems your kernel cannot understand. dd can copy
dying disks with the need to fsck on the bad drive. dd can swap byte
endianness for you (conv=swab).

Without more information about the original use case, you can't be
certain what the best approach is.

Elijah
------
has used dd for backing up Mac HFS volumes

[toc] | [prev] | [next] | [standalone]

#4402

Eli the Bearded a écrit :
>>> I try to clone a 250 GB disk PATA to a SATA drive with same CHS.
>>> I use dd if=/dev/sdb of=/dev/sda
> 
> Set the block size to match the blocks used by the actual devices. For
> historical reasons dd defaults to a block size of 1/2 kilobyte. Modern
> devices are more likely to use a block size of 8 kilobytes.

Any source, please ?
AFAIK, the Advanced Format sector size is 4 KiB.
Anyway, even with 512-byte sector size, I observed it is faster to use a
bigger block size such as 4 KiB.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.misc

csiph-web