Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #7875 > unrolled thread

is there a cp command feature showing the percentage progress of copying?

Back to article view | Back to comp.os.linux.misc

Contents

  is there a cp command feature showing the percentage progress of copying? www <xsli2@yahoo.com> - 2013-04-17 11:45 -0700
    Re: is there a cp command feature showing the percentage progress of copying? Richard Kettlewell <rjk@greenend.org.uk> - 2013-04-17 20:31 +0100
    Re: is there a cp command feature showing the percentage progress of copying? unruh <unruh@invalid.ca> - 2013-04-17 20:38 +0000
      Re: is there a cp command feature showing the percentage progress of copying? John Hasler <jhasler@newsguy.com> - 2013-04-17 16:07 -0500
        Re: is there a cp command feature showing the percentage progress of copying? Marc Haber <mh+usenetspam1118@zugschl.us> - 2013-04-18 11:29 +0200
          Re: is there a cp command feature showing the percentage progress of copying? unruh <unruh@invalid.ca> - 2013-04-18 15:44 +0000
            Re: is there a cp command feature showing the percentage progress of copying? Marc Haber <mh+usenetspam1118@zugschl.us> - 2013-04-18 18:38 +0200
              Re: is there a cp command feature showing the percentage progress of copying? unruh <unruh@invalid.ca> - 2013-04-18 16:50 +0000
                Re: is there a cp command feature showing the percentage progress of copying? Marc Haber <mh+usenetspam1118@zugschl.us> - 2013-04-18 23:32 +0200
                  Re: is there a cp command feature showing the percentage progress of copying? unruh <unruh@invalid.ca> - 2013-04-18 23:13 +0000
                  Re: is there a cp command feature showing the percentage progress of copying? Richard Kettlewell <rjk@greenend.org.uk> - 2013-04-19 08:57 +0100
                    Re: is there a cp command feature showing the percentage progress of copying? Marc Haber <mh+usenetspam1118@zugschl.us> - 2013-04-19 13:31 +0200
      Re: is there a cp command feature showing the percentage progress of copying? Bit Twister <BitTwister@mouse-potato.com> - 2013-04-17 22:23 +0000
        Re: is there a cp command feature showing the percentage progress of copying? Robert Riches <spamtrap42@jacob21819.net> - 2013-04-18 03:53 +0000
    Re: is there a cp command feature showing the percentage progress of copying? "Chris F.A. Johnson" <cfajohnson@gmail.com> - 2013-04-17 16:53 -0400
      Re: is there a cp command feature showing the percentage progress of copying? Chick Tower <c.tower@deadspam.com> - 2013-04-19 01:05 +0000
        Re: is there a cp command feature showing the percentage progress of copying? Whiskers <catwheezel@operamail.com> - 2013-04-23 15:23 +0100
    Re: is there a cp command feature showing the percentage progress of copying? Kirk_Von_Rockstein <Kirk_Von_Rockstein@nowhere.invalid> - 2013-04-17 22:44 +0000
      Re: is there a cp command feature showing the percentage progress of copying? Kirk_Von_Rockstein <Kirk_Von_Rockstein@nowhere.invalid> - 2013-04-17 23:04 +0000
    Re: is there a cp command feature showing the percentage progress  of   copying? Fritz Wuehler <fritz@spamexpire-201304.rodent.frell.theremailer.net> - 2013-04-20 00:37 +0200
      Re: is there a cp command feature showing the percentage progress  of copying? Aragorn <stryder@telenet.be.invalid> - 2013-04-20 18:21 +0200
      Re: is there a cp command feature showing the percentage progress  of   copying? Robert Riches <spamtrap42@jacob21819.net> - 2013-04-21 03:41 +0000
    Re: is there a cp command feature showing the percentage progress of copying? www <xsli2@yahoo.com> - 2013-04-22 05:36 -0700
      Re: is there a cp command feature showing the percentage progress of copying? root <NoEMail@home.org> - 2013-04-22 14:55 +0000
      Re: is there a cp command feature showing the percentage progress of copying? Chris Davies <chris-usenet@roaima.co.uk> - 2013-04-22 16:11 +0100
      Re: is there a cp command feature showing the percentage progress of copying? Kirk_Von_Rockstein <Kirk_Von_Rockstein@nowhere.invalid> - 2013-04-22 17:36 +0000

Page 1 of 2  [1] 2  Next page →

#7875 — is there a cp command feature showing the percentage progress of copying?

I am copying a lot of files from one place to another. It would be helpful for me to know it has finished 20%, 30%, 90% etc. Is there a way to know that?

Thank you.

[toc] | [next] | [standalone]

#7879

www <xsli2@yahoo.com> writes:

> I am copying a lot of files from one place to another. It would be
> helpful for me to know it has finished 20%, 30%, 90% etc. Is there a
> way to know that?

https://chris-lamb.co.uk/posts/can-you-get-cp-to-give-a-progress-bar-like-wget

(No, this isn’t a serious suggestion.)

-- 
http://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#7885

On 2013-04-17, www <xsli2@yahoo.com> wrote:
> I am copying a lot of files from one place to another. It would be helpful for me to know it has finished 20%, 30%, 90% etc. Is there a way to know that?
>
> Thank you.

Use scp or rsync, but they will only tell you the percentage of the
current file transfered not the percentage of the total. 

[toc] | [prev] | [next] | [standalone]

#7887

unruh writes:
> Use scp or rsync, but they will only tell you the percentage of the
> current file transfered not the percentage of the total.

Rsynch's "--progress" option will tell you how many files are left to
copy.
-- 
John Hasler 
jhasler@newsguy.com
Dancing Horse Hill
Elmwood, WI USA

[toc] | [prev] | [next] | [standalone]

#7896

John Hasler <jhasler@newsguy.com> wrote:
>unruh writes:
>> Use scp or rsync, but they will only tell you the percentage of the
>> current file transfered not the percentage of the total.
>
>Rsynch's "--progress" option will tell you how many files are left to
>copy.

rsync ist, however, limiting the possible speed if one uses it to copy
local data. On my system (an AMD hexacore with plenty of RAM) a plain
cp is about five times faster than rsync. As to see a percentage of
the copy, I have resorted to using mc, if I am working interactively
anyway.

Greetings
Marc
-- 
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber         |   " Questions are the         | Mailadresse im Header
Mannheim, Germany  |     Beginning of Wisdom "     | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

[toc] | [prev] | [next] | [standalone]

#7900

On 2013-04-18, Marc Haber <mh+usenetspam1118@zugschl.us> wrote:
> John Hasler <jhasler@newsguy.com> wrote:
>>unruh writes:
>>> Use scp or rsync, but they will only tell you the percentage of the
>>> current file transfered not the percentage of the total.
>>
>>Rsynch's "--progress" option will tell you how many files are left to
>>copy.
>
> rsync ist, however, limiting the possible speed if one uses it to copy
> local data. On my system (an AMD hexacore with plenty of RAM) a plain
> cp is about five times faster than rsync. As to see a percentage of
> the copy, I have resorted to using mc, if I am working interactively
> anyway.

Well, it does do an md4 sum to make sure that the copy really really did
copy correctly.

>
> Greetings
> Marc

[toc] | [prev] | [next] | [standalone]

#7901

unruh <unruh@invalid.ca> wrote:
>On 2013-04-18, Marc Haber <mh+usenetspam1118@zugschl.us> wrote:
>> rsync ist, however, limiting the possible speed if one uses it to copy
>> local data. On my system (an AMD hexacore with plenty of RAM) a plain
>> cp is about five times faster than rsync. As to see a percentage of
>> the copy, I have resorted to using mc, if I am working interactively
>> anyway.
>
>Well, it does do an md4 sum to make sure that the copy really really did
>copy correctly.

The man page does not mention that. Where can I read up on that? I
know that it uses checksum algorithms to verify whether the files are
identical _before_ starting the copy, but that only makes sense over a
slow link and not on a local hard disk where write is almost as fast
as read.

Anyway, even if one would do a manual SHA1 check afterwards, it would
still be faster.

Greetings
Marc
-- 
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber         |   " Questions are the         | Mailadresse im Header
Mannheim, Germany  |     Beginning of Wisdom "     | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

[toc] | [prev] | [next] | [standalone]

#7902

On 2013-04-18, Marc Haber <mh+usenetspam1118@zugschl.us> wrote:
> unruh <unruh@invalid.ca> wrote:
>>On 2013-04-18, Marc Haber <mh+usenetspam1118@zugschl.us> wrote:
>>> rsync ist, however, limiting the possible speed if one uses it to copy
>>> local data. On my system (an AMD hexacore with plenty of RAM) a plain
>>> cp is about five times faster than rsync. As to see a percentage of
>>> the copy, I have resorted to using mc, if I am working interactively
>>> anyway.
>>
>>Well, it does do an md4 sum to make sure that the copy really really did
>>copy correctly.
>
> The man page does not mention that. Where can I read up on that? I
> know that it uses checksum algorithms to verify whether the files are
> identical _before_ starting the copy, but that only makes sense over a
> slow link and not on a local hard disk where write is almost as fast
> as read.

No. It only does a checksum before hand if you give it the -c flag.
Otherwise what it does if the file exists on the receiving end is to do
a nested series of checksums on the two files to determine which parts
of the file to send over.

man rsync (under the explanation for --checksum)
   Note that rsync always verifies that each transferred  file  was
    correctly  reconstructed  on  the  receiving  side by checking a
    whole-file checksum that is generated  as  the  file  is trans-
    ferred,  but  that automatic after-the-transfer verification has
    nothing to do with this option's before-the-transfer "Does this
    file need to be updated?" check.

>
> Anyway, even if one would do a manual SHA1 check afterwards, it would
> still be faster.
>
> Greetings
> Marc

[toc] | [prev] | [next] | [standalone]

#7904

unruh <unruh@invalid.ca> wrote:
>man rsync (under the explanation for --checksum)
>   Note that rsync always verifies that each transferred  file  was
>    correctly  reconstructed  on  the  receiving  side by checking a
>    whole-file checksum that is generated  as  the  file  is trans-
>    ferred,  but  that automatic after-the-transfer verification has
>    nothing to do with this option's before-the-transfer "Does this
>    file need to be updated?" check.

On local rsyncs, --checksum is not the default, and the actual copy
still take five times of the stupid copy. I'd guess it would be even
slower with --checksum activated.

Greetings
Marc
-- 
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber         |   " Questions are the         | Mailadresse im Header
Mannheim, Germany  |     Beginning of Wisdom "     | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

[toc] | [prev] | [next] | [standalone]

#7905

On 2013-04-18, Marc Haber <mh+usenetspam1118@zugschl.us> wrote:
> unruh <unruh@invalid.ca> wrote:
>>man rsync (under the explanation for --checksum)
>>   Note that rsync always verifies that each transferred  file  was
>>    correctly  reconstructed  on  the  receiving  side by checking a
>>    whole-file checksum that is generated  as  the  file  is trans-
>>    ferred,  but  that automatic after-the-transfer verification has
>>    nothing to do with this option's before-the-transfer "Does this
>>    file need to be updated?" check.
>
> On local rsyncs, --checksum is not the default, and the actual copy
> still take five times of the stupid copy. I'd guess it would be even
> slower with --checksum activated.

While the above is under the --checksum it does NOT describe the
--checksum. It describes the behaviour of rsync ALWAYS. It always does a
running md4 check while it is transfering to ensure that what is
transfered is the same as the original. If you do not care, then cp
would work.

>
> Greetings
> Marc

[toc] | [prev] | [next] | [standalone]

#7911

Marc Haber <mh+usenetspam1118@zugschl.us> writes:
> unruh <unruh@invalid.ca> wrote:
>>man rsync (under the explanation for --checksum)
>>    Note that rsync always verifies that each transferred  file  was
>>    correctly  reconstructed  on  the  receiving  side by checking a
>>    whole-file checksum that is generated  as  the  file  is trans-
>>    ferred,  but  that automatic after-the-transfer verification has
>>    nothing to do with this option's before-the-transfer "Does this
>>    file need to be updated?" check.
>
> On local rsyncs, --checksum is not the default, and the actual copy
> still take five times of the stupid copy. I'd guess it would be even
> slower with --checksum activated.

He’s not talking about --checksum, which uses a hash to decide what to
transfer, he’s talking about he integrity check applied to all files.
For a large copy from spinning disk that’s not likely to slow things
down appreciably; the IO will be the limiting factor (e.g. my 2007-era
CPU can do MD5 about three times as fast as the attached spinning disk
can keep it fed.)

-- 
http://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#7912

Richard Kettlewell <rjk@greenend.org.uk> wrote:
>For a large copy from spinning disk that’s not likely to slow things
>down appreciably; the IO will be the limiting factor (e.g. my 2007-era
>CPU can do MD5 about three times as fast as the attached spinning disk
>can keep it fed.)

rsync's structure (I suspect it's the client-server model that is
still used even for a local copy) slows things down considerably.

A cp /home/mh/foo /home/mh/bar (with a 2 GB /home/mh/foo) yields a
transfer rate of about 140 Mbyte/s (which is in the range of what the
spinning disk can do) while rsync only transfers about 30 Mbyte/s.

Greetings
Marc
-- 
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber         |   " Questions are the         | Mailadresse im Header
Mannheim, Germany  |     Beginning of Wisdom "     | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

[toc] | [prev] | [next] | [standalone]

#7891

On Wed, 17 Apr 2013 20:38:08 GMT, unruh wrote:
> On 2013-04-17, www <xsli2@yahoo.com> wrote:
>> I am copying a lot of files from one place to another. It would be helpful for me to know it has finished 20%, 30%, 90% etc. Is there a way to know that?

I have no experience with it, you might try pv

$ man pv 
PV(1)                                                User Manuals                                                PV(1)

NAME
       pv - monitor the progress of data through a pipe

SYNOPSIS
       pv [OPTION] [FILE]...
       pv [-h|-V]

DESCRIPTION
       pv allows a user to see the progress of data through a
       pipeline, by giving information such as time elapsed,
       percentage completed (with progress bar), current throughput
       rate, total data transferred, and ETA.

       To use it, insert it in a pipeline between two processes, with
       the appropriate options.  Its standard input will be passed
       through to its standard output and progress will be shown on
       standard error.

       pv will copy each supplied FILE in turn to standard output (-
       means standard input), or if no FILEs are speci‐ fied just
       standard input is copied. This is the same behaviour as cat(1).

       A simple example to watch how quickly a file is transferred using nc(1):

              pv file | nc -w 1 somewhere.com 3000

[toc] | [prev] | [next] | [standalone]

#7894

On 2013-04-17, Bit Twister <BitTwister@mouse-potato.com> wrote:
> On Wed, 17 Apr 2013 20:38:08 GMT, unruh wrote:
>> On 2013-04-17, www <xsli2@yahoo.com> wrote:
>>> I am copying a lot of files from one place to another. It would be helpful for me to know it has finished 20%, 30%, 90% etc. Is there a way to know that?
>
> I have no experience with it, you might try pv
>
> $ man pv 
> PV(1)                                                User Manuals                                                PV(1)
>
> NAME
>        pv - monitor the progress of data through a pipe
>
> SYNOPSIS
>        pv [OPTION] [FILE]...
>        pv [-h|-V]
>
> DESCRIPTION
>        pv allows a user to see the progress of data through a
>        pipeline, by giving information such as time elapsed,
>        percentage completed (with progress bar), current throughput
>        rate, total data transferred, and ETA.
>
>        To use it, insert it in a pipeline between two processes, with
>        the appropriate options.  Its standard input will be passed
>        through to its standard output and progress will be shown on
>        standard error.
>
>        pv will copy each supplied FILE in turn to standard output (-
>        means standard input), or if no FILEs are speci??? fied just
>        standard input is copied. This is the same behaviour as cat(1).
>
>        A simple example to watch how quickly a file is transferred using nc(1):
>
>               pv file | nc -w 1 somewhere.com 3000

I've used it.  It's pretty good for the intended purpose.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

[toc] | [prev] | [next] | [standalone]

#7886

On 2013-04-17, www wrote:
>
> I am copying a lot of files from one place to another. It would be
> helpful for me to know it has finished 20%, 30%, 90% etc. Is there a
> way to know that?

   Use gentoo (the file manager, not the distro):
   <http://www.obsession.se/gentoo/>

-- 
   Chris F.A. Johnson, <http://cfajohnson.com>
   Author:
   Pro Bash Programming: Scripting the GNU/Linux Shell (2009, Apress)
   Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)

[toc] | [prev] | [next] | [standalone]

#7907

On 2013-04-17, Chris F.A. Johnson <cfajohnson@gmail.com> wrote:
> On 2013-04-17, www wrote:
>>
>> I am copying a lot of files from one place to another. It would be
>> helpful for me to know it has finished 20%, 30%, 90% etc. Is there a
>> way to know that?
>
>    Use gentoo (the file manager, not the distro):
>    <http://www.obsession.se/gentoo/>
>

Or Midnight Commander (mc), which is included by default in some
distros, and easily installed in the rest.
-- 
                                 Chick Tower

For e-mail:  colm DOT sent DOT towerboy AT xoxy DOT net

[toc] | [prev] | [next] | [standalone]

#7966

On 2013-04-19, Chick Tower <c.tower@deadspam.com> wrote:
> On 2013-04-17, Chris F.A. Johnson <cfajohnson@gmail.com> wrote:
>> On 2013-04-17, www wrote:
>>>
>>> I am copying a lot of files from one place to another. It would be
>>> helpful for me to know it has finished 20%, 30%, 90% etc. Is there a
>>> way to know that?
>>
>>    Use gentoo (the file manager, not the distro):
>>    <http://www.obsession.se/gentoo/>
>>
>
> Or Midnight Commander (mc), which is included by default in some
> distros, and easily installed in the rest.

It seems the simple approach is too simple for some!

-- 
-- ^^^^^^^^^^
--  Whiskers 
-- ~~~~~~~~~~

[toc] | [prev] | [next] | [standalone]

#7892

On 2013-04-17, www <xsli2@yahoo.com> wrote:
> I am copying a lot of files from one place to another. 
> It would be helpful for me to know it has finished 20%, 
> 30%, 90% etc. Is there a way to know that?
>
> Thank you.

Install gcp and python-progressbar. That will give you
the results you desire.

http://dev.man-online.org/man1/gcp/

You may be able to pipe cp through pv.
I have not done so with cp, but did use
it a while back with dd to get a progress bar
using it, while imaging a large partition.

http://linux.die.net/man/1/pv

[toc] | [prev] | [next] | [standalone]

#7893

On 2013-04-17, Kirk_Von_Rockstein <Kirk_Von_Rockstein@nowhere.invalid> wrote:
> On 2013-04-17, www <xsli2@yahoo.com> wrote:
>> I am copying a lot of files from one place to another. 
>> It would be helpful for me to know it has finished 20%, 
>> 30%, 90% etc. Is there a way to know that?
>>
>> Thank you.
>
> Install gcp and python-progressbar. That will give you
> the results you desire.
>
> http://dev.man-online.org/man1/gcp/
>
> You may be able to pipe cp through pv.
> I have not done so with cp, but did use
> it a while back with dd to get a progress bar
> using it, while imaging a large partition.
>
> http://linux.die.net/man/1/pv

Example gcp:
root@aptosid-msi:/home/kirk# gcp -rf /home/kirk/.claws-mail /mnt/sdb15
Copying 6.61 MiB 100% |########<snip>#######|   3.38 M/s Time: 00:00:02
                  ^                   ^
		  |                   |
             Percent completed     Bar increases
	     shows here

[toc] | [prev] | [next] | [standalone]

#7932 — Re: is there a cp command feature showing the percentage progress of copying?

www <xs...@yahoo.com> [w]:
w> I am copying a lot of files from one place to another. It would
w> be helpful for me to know it has finished 20%, 30%, 90% etc. Is
w> there a way to know that?

If you have a rough idea of the input data size, use a tool like pipemeter

	tar -cf - -C /source/directory . |
        pipemeter -s SIZE_OF_INPUT_DATA | 
	ssh user@some_other_locaion  \  
        	tar -xf - -C /target/directory

It will give you a nice progress bar and a rough ETA.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.os.linux.misc

csiph-web