Groups > linux.debian.user > #208474

Re: Speed Problem Copying Files

From	"Thomas Schmitt" <scdbackup@gmx.net>
Newsgroups	linux.debian.user
Subject	Re: Speed Problem Copying Files
Date	2019-05-09 17:20 +0200
Message-ID	<xVSZX-7DF-1@gated-at.bofh.it> (permalink)
References	<xVQEO-65h-3@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

Hi,

Lothar Schilling wrote:
> Fast enough...
> dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s

So this is sufficiently fast, but

  cp /daten/testfile /daten/testfile2

lasts 2000 seconds ?

> dd if=/daten/testfile of=/dev/null
> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s

It is not about the number of read/write operations.
(How fast is bs=1G on the second try ?)

------------------------------------------------------------------------

It is also not about the filesystem and its health status.
So where could be any difference ?

dd is an i/o dinosaur. It performs read/write without much mind.

cp uses a centralized copy facility by a function named "copy ()".
This facility makes much ado about sparse files:
  https://sources.debian.org/src/coreutils/8.30-3/src/copy.c/#L387
  https://sources.debian.org/src/coreutils/8.30-3/src/copy.c/#L224

man cp says:

  By default, sparse SOURCE files are detected by a crude  heuristic  and
  the corresponding DEST file is made sparse as well.  That is the behav‐
  ior selected by --sparse=auto.  Specify  --sparse=always  to  create  a
  sparse  DEST  file  whenever  the  SOURCE  file  contains a long enough
  sequence of zero bytes.  Use  --sparse=never  to  inhibit  creation  of
  sparse files.

So i'd try cp with --sparse=never in order to see whether it then resembles
the behavior of dd.

Have a nice day :)

Thomas

Thread

Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 11:00 +0200
  Re: Speed Problem Copying Files Jonas Smedegaard <jonas@jones.dk> - 2019-05-09 11:20 +0200
    Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 11:50 +0200
      Re: Speed Problem Copying Files Kevin DAGNEAUX <kevin.dagneaux@fiitelcom.fr> - 2019-05-09 12:00 +0200
        Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 12:30 +0200
          Re: Speed Problem Copying Files Martin <keine-eile@online.de> - 2019-05-09 13:30 +0200
            Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 14:50 +0200
              Re: Speed Problem Copying Files Martin <keine-eile@online.de> - 2019-05-09 16:00 +0200
              Re: Speed Problem Copying Files "Thomas Schmitt" <scdbackup@gmx.net> - 2019-05-09 17:20 +0200
      Re: Speed Problem Copying Files Jonas Smedegaard <jonas@jones.dk> - 2019-05-09 12:30 +0200
        Re: Speed Problem Copying Files Jonas Smedegaard <jonas@jones.dk> - 2019-05-09 13:00 +0200
          Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 13:20 +0200
            Re: Speed Problem Copying Files Keith Christian <keith1christian@gmail.com> - 2019-05-09 13:30 +0200
              Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-09 14:50 +0200
  Re: Speed Problem Copying Files Gene Heskett <gheskett@shentel.net> - 2019-05-09 16:20 +0200
    Re: Speed Problem Copying Files "Jeremy Nicoll" <jn.ml.dbn.25@letterboxes.org> - 2019-05-09 16:30 +0200
      Re: Speed Problem Copying Files Dan Ritter <dsr@randomstring.org> - 2019-05-09 17:00 +0200
      Re: Speed Problem Copying Files Gene Heskett <gheskett@shentel.net> - 2019-05-09 17:00 +0200
  Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-10 07:50 +0200
    Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-10 11:00 +0200
      Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-13 10:40 +0200
        Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-13 11:00 +0200
          Re: Speed Problem Copying Files Tixy <tixy@yxit.co.uk> - 2019-05-13 11:20 +0200
          Re: Speed Problem Copying Files Curt <curty@free.fr> - 2019-05-13 11:40 +0200
          Re: Speed Problem Copying Files Michael Stone <mstone@debian.org> - 2019-05-14 13:40 +0200
        Re: Speed Problem Copying Files Tixy <tixy@yxit.co.uk> - 2019-05-13 11:00 +0200
        Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-14 02:30 +0200
          Re: Speed Problem Copying Files Lothar Schilling <ls@proasyl.de> - 2019-05-14 09:40 +0200
            Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-14 18:30 +0200
    Re: Speed Problem Copying Files Michael Stone <mstone@debian.org> - 2019-05-14 13:30 +0200
      Re: Speed Problem Copying Files David Christensen <dpchrist@holgerdanske.com> - 2019-05-14 21:20 +0200
        Re: Speed Problem Copying Files Michael Stone <mstone@debian.org> - 2019-05-14 22:10 +0200

csiph-web