Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #170643 > unrolled thread

Debian server for backups of Windows clients

Back to article view | Back to linux.debian.user

Contents

  Debian server for backups of Windows clients Daniel Bareiro <daniel-listas@gmx.net> - 2016-08-02 23:20 +0200
    Re: Debian server for backups of Windows clients David Christensen <dpchrist@holgerdanske.com> - 2016-08-03 05:30 +0200
      Re: Debian server for backups of Windows clients Daniel Bareiro <daniel-listas@gmx.net> - 2016-08-08 20:30 +0200
        Re: Debian server for backups of Windows clients David Christensen <dpchrist@holgerdanske.com> - 2016-08-09 05:30 +0200
          Re: Debian server for backups of Windows clients Daniel Bareiro <daniel-listas@gmx.net> - 2016-08-10 01:30 +0200
            Re: Debian server for backups of Windows clients David Christensen <dpchrist@holgerdanske.com> - 2016-08-10 04:00 +0200
              Re: Debian server for backups of Windows clients Celejar <celejar@gmail.com> - 2016-09-09 21:00 +0200
                Re: Debian server for backups of Windows clients Daniel Bareiro <daniel-listas@gmx.net> - 2016-09-09 21:50 +0200
                  Re: Debian server for backups of Windows clients Celejar <celejar@gmail.com> - 2016-09-09 23:20 +0200
                    Re: Debian server for backups of Windows clients Daniel Bareiro <daniel-listas@gmx.net> - 2016-09-11 14:30 +0200
                  Re: Debian server for backups of Windows clients deloptes <deloptes@gmail.com> - 2016-09-10 00:10 +0200
                    Re: Debian server for backups of Windows clients Daniel Bareiro <daniel-listas@gmx.net> - 2016-09-11 14:10 +0200
                Re: Debian server for backups of Windows clients David Christensen <dpchrist@holgerdanske.com> - 2016-09-10 05:40 +0200
                  Re: Debian server for backups of Windows clients David Wright <deblis@lionunicorn.co.uk> - 2016-09-10 06:20 +0200
                    Re: Debian server for backups of Windows clients "Neal P. Murphy" <neal.p.murphy@alum.wpi.edu> - 2016-09-10 07:30 +0200
                      Re: Debian server for backups of Windows clients Dan Ritter <dsr@randomstring.org> - 2016-09-10 14:50 +0200
                        Re: Debian server for backups of Windows clients rhkramer@gmail.com - 2016-09-10 16:30 +0200
                          Re: Debian server for backups of Windows clients Gene Heskett <gheskett@shentel.net> - 2016-09-10 16:50 +0200
                            Re: Debian server for backups of Windows clients rhkramer@gmail.com - 2016-09-10 17:00 +0200
                              Re: Debian server for backups of Windows clients David Christensen <dpchrist@holgerdanske.com> - 2016-09-10 17:40 +0200
                              Re: Debian server for backups of Windows clients Gene Heskett <gheskett@shentel.net> - 2016-09-10 19:40 +0200
                              Re: Debian server for backups of Windows clients "Neal P. Murphy" <neal.p.murphy@alum.wpi.edu> - 2016-09-11 07:50 +0200
                    Re: Debian server for backups of Windows clients David Christensen <dpchrist@holgerdanske.com> - 2016-09-10 08:20 +0200
                  Re: Debian server for backups of Windows clients Celejar <celejar@gmail.com> - 2016-09-11 04:30 +0200
              Re: Debian server for backups of Windows clients Daniel Bareiro <daniel-listas@gmx.net> - 2016-09-09 21:50 +0200
                Re: Debian server for backups of Windows clients David Christensen <dpchrist@holgerdanske.com> - 2016-09-10 05:50 +0200
                  Re: Debian server for backups of Windows clients Celejar <celejar@gmail.com> - 2016-09-11 04:30 +0200
                    Re: Debian server for backups of Windows clients David Christensen <dpchrist@holgerdanske.com> - 2016-09-11 05:10 +0200
    Re: Debian server for backups of Windows clients didier gaumet <didier.gaumet@gmail.com> - 2016-08-03 10:40 +0200
      Re: Debian server for backups of Windows clients Daniel Bareiro <daniel-listas@gmx.net> - 2016-08-10 01:40 +0200
        Re: Debian server for backups of Windows clients didier gaumet <didier.gaumet@gmail.com> - 2016-08-11 18:50 +0200
          Re: Debian server for backups of Windows clients didier gaumet <didier.gaumet@gmail.com> - 2016-08-11 19:00 +0200
    Re: Debian server for backups of Windows clients Darac Marjal <mailinglist@darac.org.uk> - 2016-08-08 11:10 +0200
      Re: Debian server for backups of Windows clients Daniel Bareiro <daniel-listas@gmx.net> - 2016-08-08 22:10 +0200
        Re: Debian server for backups of Windows clients David Christensen <dpchrist@holgerdanske.com> - 2016-08-09 05:30 +0200
          Re: Debian server for backups of Windows clients Mike McGinn <mikemcginn@mcginnweb.net> - 2016-08-09 15:40 +0200
      Re: Debian server for backups of Windows clients Glenn English <ghe@slsware.net> - 2016-08-08 23:40 +0200
        Re: Debian server for backups of Windows clients Daniel Bareiro <daniel-listas@gmx.net> - 2016-08-09 03:20 +0200
    Re: Debian server for backups of Windows clients Joel Wirāmu Pauling <joel@aenertia.net> - 2016-08-10 02:20 +0200

Page 1 of 2  [1] 2  Next page →

#170643 — Debian server for backups of Windows clients

[Multipart message — attachments visible in raw view] — view raw

Hi all!

I'm thinking deploy a Debian backup server using Dirvish (which is based
on rsync --- indeed, we have packaged it in Debian). On previous
occasions I implemented these solutions seamlessly with GNU/Linux
clients, but now I would like add Windows clients.

The idea of using Dirvish is because I had a very good experience.
Besides using rsync with hard links for backups of files that do not
change from backup to the next allows a considerable saving of disk space.

But to use Dirvish with Windows clients I will need to install an SSH
server. I had thought that an alternative would be to use Cygwin, but
was looking for documentation and I have not found any uniform process
to install and configure a Cygwin SSH server on Windows.

I would like to know if anyone has had any experience in this regard
that could share.


Thanks in advance.

Kind regards,
Daniel

[toc] | [next] | [standalone]

#170649

On 08/02/2016 02:12 PM, Daniel Bareiro wrote:
> I'm thinking deploy a Debian backup server using Dirvish (which is based
> on rsync --- indeed, we have packaged it in Debian). On previous
> occasions I implemented these solutions seamlessly with GNU/Linux
> clients, but now I would like add Windows clients.
> 
> The idea of using Dirvish is because I had a very good experience.
> Besides using rsync with hard links for backups of files that do not
> change from backup to the next allows a considerable saving of disk space.
> 
> But to use Dirvish with Windows clients I will need to install an SSH
> server. I had thought that an alternative would be to use Cygwin, but
> was looking for documentation and I have not found any uniform process
> to install and configure a Cygwin SSH server on Windows.
> 
> I would like to know if anyone has had any experience in this regard
> that could share.

Cygwin sshd and rsync are okay for interactive use.  There is a shell
script (ssh-host-config) provided with the Cygwin openssh package for
setting up sshd as a service.


Unfortunately, Cygwin rsync is notorious for working for a while, and
then hanging in the middle of a transfer.  I've seen this for years, and
I saw it on up-to-date installs less than a week ago.  So for automated
backups, you need to detect this failure mode and deal with it.  The
only way I found to get rsync working again was to reboot.


You might find more encouraging answers on the Cygwin mailing list:

    https://cygwin.com/lists.html


Currently, Windows Backup and Restore is the most reliable solution I've
found for automated backups.  Either it works, or it doesn't (the last
Windows Vista box I maintain has broken Volume Shadow Copy, which breaks
Backup and Restore).


David

[toc] | [prev] | [next] | [standalone]

#170852

[Multipart message — attachments visible in raw view] — view raw

Hi, David.

On 03/08/16 00:23, David Christensen wrote:

>> I'm thinking deploy a Debian backup server using Dirvish (which is based
>> on rsync --- indeed, we have packaged it in Debian). On previous
>> occasions I implemented these solutions seamlessly with GNU/Linux
>> clients, but now I would like add Windows clients.
>>
>> The idea of using Dirvish is because I had a very good experience.
>> Besides using rsync with hard links for backups of files that do not
>> change from backup to the next allows a considerable saving of disk space.
>>
>> But to use Dirvish with Windows clients I will need to install an SSH
>> server. I had thought that an alternative would be to use Cygwin, but
>> was looking for documentation and I have not found any uniform process
>> to install and configure a Cygwin SSH server on Windows.
>>
>> I would like to know if anyone has had any experience in this regard
>> that could share.

> Cygwin sshd and rsync are okay for interactive use.  There is a shell
> script (ssh-host-config) provided with the Cygwin openssh package for
> setting up sshd as a service.
> 
> Unfortunately, Cygwin rsync is notorious for working for a while, and
> then hanging in the middle of a transfer.  I've seen this for years, and
> I saw it on up-to-date installs less than a week ago.  So for automated
> backups, you need to detect this failure mode and deal with it.  The
> only way I found to get rsync working again was to reboot.
> 
> You might find more encouraging answers on the Cygwin mailing list:
> 
>     https://cygwin.com/lists.html

It sounds like a blocking issue. Researching about it, I read something
about what you mention here [1]. As it says, it seems that this is
resolved in new versions of Cygwin, although you say that a week ago you
had this problem (with the latest published version?). But this article
is "a bit" old.

He also mentions a problem to doing backup over open files. Have you
experienced that problem?

> Currently, Windows Backup and Restore is the most reliable solution I've
> found for automated backups.  Either it works, or it doesn't (the last
> Windows Vista box I maintain has broken Volume Shadow Copy, which breaks
> Backup and Restore).

It does not sound very encouraging. An alternative that I thought as a
last resort was to mount the remote filesystem on the backup server
using Samba and then to use rsync on the mount point, although I'm not
sure how efficient it can be.


Thanks for your reply.

Kind regards,
Daniel

[1]
http://www.trueblade.com/techblog/backing-up-windows-computers-with-dirvish

[toc] | [prev] | [next] | [standalone]

#170866

On 08/08/2016 11:25 AM, Daniel Bareiro wrote:
> He also mentions a [Cygwin rsync] problem to doing backup over open files. Have you
> experienced that problem?

I'm not sure.  It's been many moons since I tried to do automated
backups of Windows machines using rsync.


> An alternative that I thought as a
> last resort was to mount the remote filesystem on the backup server
> using Samba and then to use rsync on the mount point, although I'm not
> sure how efficient it can be.

Try it and let us know how well it works.


David

[toc] | [prev] | [next] | [standalone]

#170890

[Multipart message — attachments visible in raw view] — view raw

Hi, David.

On 09/08/16 00:21, David Christensen wrote:

>> He also mentions a [Cygwin rsync] problem to doing backup over open files. Have you
>> experienced that problem?

> I'm not sure.  It's been many moons since I tried to do automated
> backups of Windows machines using rsync.

Well, I've been testing with rsync+ssh from Cygwin on a KVM virtual
machine with Windows Server 2012. So far everything has worked smoothly:

-------------------------------------------------------------------
viper@orion:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.5 (jessie)
Release:        8.5
Codename:       jessie
-------------------------------------------------------------------
viper@orion:~$ rsync --stats --progress -vae ssh
Administrator@10.1.0.33:/cygdrive/c/Users/Administrator/Documents
/ImagenCorporativa /tmp/Backup/
Administrator@10.1.0.33's password:
receiving incremental file list
(...)
ImagenCorporativa/disk01.img
  3,221,225,472 100%    1.19MB/s    0:43:02 (xfr#2, to-chk=53/56)
(...)

Number of files: 56 (reg: 48, dir: 8)
Number of created files: 56 (reg: 48, dir: 8)
Number of deleted files: 0
Number of regular files transferred: 48
Total file size: 3,226,528,378 bytes
Total transferred file size: 3,226,528,378 bytes
Literal data: 3,226,528,378 bytes
Matched data: 0 bytes
File list size: 1,360
File list generation time: 0.012 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 996
Total bytes received: 3,227,319,409

sent 996 bytes  received 3,227,319,409 bytes  1,244,868.04 bytes/sec
total size is 3,226,528,378  speedup is 1.00
-------------------------------------------------------------------
viper@orion:~$ rsync --stats --progress -vae ssh
Administrator@10.1.0.33:/cygdrive/c/Users/Administrator/Documents/ImagenCorporativa
/tmp/Backup/
Administrator@10.1.0.33's password:
receiving incremental file list

Number of files: 56 (reg: 48, dir: 8)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 3,226,528,378 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 1,360
File list generation time: 0.015 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 32
Total bytes received: 1,379

sent 32 bytes  received 1,379 bytes  166.00 bytes/sec
total size is 3,226,528,378  speedup is 2,286,696.23
-------------------------------------------------------------------

As you can see, the transfer was over than 3 GB and it were not hung. I
did several tests and all were without problems.

I wonder if in the mentioned episodes of hangs you remember whether the
transferred volume was higher or lower than in this case (or it hung
randomly).

As a side note, the larger file (disk01.img) took more than 40 minutes
to be transferred. So the rsync was running quite some time without
hanging. While it does not have to do with the topic of this thread, in
rsync progress data we can see that the average transfer rate was 10
Mbps. I guess it will have to do with that I'm going through a wireless
network. In this testing the Debian computer is a notebook connected to
the wireless router and the KVM Windows is on the wired network. May it
be so large the decrease in transfer speed? The wireless router is
TPLink WDR3600 with OpenWRT.


Kind regards,
Daniel

[toc] | [prev] | [next] | [standalone]

#170904

On 08/09/2016 04:27 PM, Daniel Bareiro wrote:
> As you can see, the transfer was over than 3 GB and it were not hung. I
> did several tests and all were without problems.
> 
> I wonder if in the mentioned episodes of hangs you remember whether the
> transferred volume was higher or lower than in this case (or it hung
> randomly).

Script it and run it every night for a week.  If it works every time,
try again for 30 days.  Then 90.  Then 365.


> 
> As a side note, the larger file (disk01.img) took more than 40 minutes
> to be transferred. So the rsync was running quite some time without
> hanging. While it does not have to do with the topic of this thread, in
> rsync progress data we can see that the average transfer rate was 10
> Mbps. I guess it will have to do with that I'm going through a wireless
> network. In this testing the Debian computer is a notebook connected to
> the wireless router and the KVM Windows is on the wired network. May it
> be so large the decrease in transfer speed? The wireless router is
> TPLink WDR3600 with OpenWRT.

My laptop has 802.11 a/b/g WiFi and Fast Ethernet.  Wireless data
transfers are slow (~50 Mbps).  Wired is twice as fast (100 Mbps); still
slow.  Newer WiFi (n, ac) should be faster, but only the newest WiFi
hardware can match or beat Gigabit.


For the initial full backup, I have found that scp is faster than rsync.


When I know that I've added a bunch of new and/or large files on the
sender, I sometimes try the rsync 'whole-file' option.  As I haven't
benchmarked it, I don't know if/when it is helping.


My biggest problem with rsync is when I reorganize file/ directory trees
on my file server; especially big stuff -- raw video, movies, disk
images, ISO images, etc..  I have yet to figure out an rsync incantation
that does the corresponding moves on the destination, rather than
mindlessly copying and deleting 100's of GB.  I have often considered
writing an rsync prelude script for just this case.


David

[toc] | [prev] | [next] | [standalone]

#171955

On Tue, 9 Aug 2016 18:57:02 -0700
David Christensen <dpchrist@holgerdanske.com> wrote:

...

> My laptop has 802.11 a/b/g WiFi and Fast Ethernet.  Wireless data
> transfers are slow (~50 Mbps).  Wired is twice as fast (100 Mbps); still
> slow.  Newer WiFi (n, ac) should be faster, but only the newest WiFi
> hardware can match or beat Gigabit.

You get ~50Mbps over a/b/g? 54Mbps is the theoretical maximum, and
everything I've read says that 20-24Mbps is the real-world maximum.

Celejar

[toc] | [prev] | [next] | [standalone]

#171959

[Multipart message — attachments visible in raw view] — view raw

Hi, Celejar.

On 09/09/16 15:51, Celejar wrote:

>> My laptop has 802.11 a/b/g WiFi and Fast Ethernet.  Wireless data
>> transfers are slow (~50 Mbps).  Wired is twice as fast (100 Mbps); still
>> slow.  Newer WiFi (n, ac) should be faster, but only the newest WiFi
>> hardware can match or beat Gigabit.

> You get ~50Mbps over a/b/g? 54Mbps is the theoretical maximum, and
> everything I've read says that 20-24Mbps is the real-world maximum.

Still, 20-24 Mbps is more than 10 Mpbs I was seeing with rsync. There
could be a bottleneck somewhere?


Kind regards,
Daniel

[toc] | [prev] | [next] | [standalone]

#171965

On Fri, 9 Sep 2016 16:46:35 -0300
Daniel Bareiro <daniel-listas@gmx.net> wrote:

> Hi, Celejar.
> 
> On 09/09/16 15:51, Celejar wrote:
> 
> >> My laptop has 802.11 a/b/g WiFi and Fast Ethernet.  Wireless data
> >> transfers are slow (~50 Mbps).  Wired is twice as fast (100 Mbps); still
> >> slow.  Newer WiFi (n, ac) should be faster, but only the newest WiFi
> >> hardware can match or beat Gigabit.
> 
> > You get ~50Mbps over a/b/g? 54Mbps is the theoretical maximum, and
> > everything I've read says that 20-24Mbps is the real-world maximum.
> 
> Still, 20-24 Mbps is more than 10 Mpbs I was seeing with rsync. There
> could be a bottleneck somewhere?

As per your own suggestion in another message, definitely benchmark
with iperf to see if that's better. And as we discussed in another
thread some time ago, (especially) if you're using wireless, benchmark
throughput in *both* directions, since the transmitter (or receiver) may
be better on one machine than on another.

Celejar

[toc] | [prev] | [next] | [standalone]

#172039

[Multipart message — attachments visible in raw view] — view raw

Hi, Celejar

On 09/09/16 18:18, Celejar wrote:

>>>> My laptop has 802.11 a/b/g WiFi and Fast Ethernet.  Wireless data
>>>> transfers are slow (~50 Mbps).  Wired is twice as fast (100 Mbps); still
>>>> slow.  Newer WiFi (n, ac) should be faster, but only the newest WiFi
>>>> hardware can match or beat Gigabit.

>>> You get ~50Mbps over a/b/g? 54Mbps is the theoretical maximum, and
>>> everything I've read says that 20-24Mbps is the real-world maximum.

>> Still, 20-24 Mbps is more than 10 Mpbs I was seeing with rsync. There
>> could be a bottleneck somewhere?

> As per your own suggestion in another message, definitely benchmark
> with iperf to see if that's better.

Yes, it can be. I was thinking about what I said in a previous message
about the control information added by rsync on the packets sent.

I think this would be important only if we focus on the performance
(number of bits of data sent / total number of bits sent). In this case,
the focus is the transfer rate, for which the amount of control bits
used would be irrelevant since I think we need to know how many bits per
second we are getting, regardless of the utility have those bits.

> And as we discussed in another thread some time ago, (especially) if 
> you're using wireless, benchmark throughput in *both* directions,
> since the transmitter (or receiver) may be better on one machine than
> on another.

Interesting sidelight. Thanks for sharing.


Kind regards,
Daniel

[toc] | [prev] | [next] | [standalone]

#171966

Daniel Bareiro wrote:

> Still, 20-24 Mbps is more than 10 Mpbs I was seeing with rsync. There
> could be a bottleneck somewhere?

In my case it was the IO on the disk - I couldn't do more than 12Mbps even
on wired connection, because I have encrypted disk ... it took me a while
to understand why though.

[toc] | [prev] | [next] | [standalone]

#172036

[Multipart message — attachments visible in raw view] — view raw

Hi, deloptes.

On 09/09/16 19:06, deloptes wrote:

>> Still, 20-24 Mbps is more than 10 Mpbs I was seeing with rsync. There
>> could be a bottleneck somewhere?

> In my case it was the IO on the disk - I couldn't do more than 12Mbps even
> on wired connection, because I have encrypted disk ... it took me a while
> to understand why though.

This is an interesting fact. Because 'orion' (the notebook used in the
mentioned test) also has an encrypted disk. In the test, the notebook
was pulling the files on the Windows VM on the wired network.

root@orion:~# dmsetup ls --target crypt
sda5_crypt      (254, 0)

root@orion:~# cryptsetup luksDump /dev/sda5 | grep Version -A3
Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1

viper@orion:~$ lsblk --fs
NAME             FSTYPE LABEL UUID MOUNTPOINT
sda
├─sda1                             /boot
├─sda2
└─sda5
  └─sda5_crypt
    ├─main-swap                    [SWAP]
    ├─main-root                    /
    └─main-datos                   /datos
sr0


I did not think this could affect so strongly in the network transfer.


Kind regards,
Daniel

[toc] | [prev] | [next] | [standalone]

#171968

On 09/09/2016 11:51 AM, Celejar wrote:
> On Tue, 9 Aug 2016 18:57:02 -0700
> David Christensen <dpchrist@holgerdanske.com> wrote:
> 
> ...
> 
>> My laptop has 802.11 a/b/g WiFi and Fast Ethernet.  Wireless data
>> transfers are slow (~50 Mbps).  Wired is twice as fast (100 Mbps); still
>> slow.  Newer WiFi (n, ac) should be faster, but only the newest WiFi
>> hardware can match or beat Gigabit.
> 
> You get ~50Mbps over a/b/g? 54Mbps is the theoretical maximum, and
> everything I've read says that 20-24Mbps is the real-world maximum.
> 
> Celejar
> 

Benchmarking using WiFi (48 Mb/s):

    2016-09-09 20:18:51 dpchrist@t7400 ~
    $ time dd if=/dev/urandom of=urandom.100M bs=1M count=100
    100+0 records in
    100+0 records out
    104857600 bytes (105 MB) copied, 12.6709 s, 8.3 MB/s

    real	0m12.703s
    user	0m0.000s
    sys	0m12.481s

    2016-09-09 20:19:32 dpchrist@t7400 ~
    $ time scp -p urandom.100M samba:.
    urandom.100M


                          100%  100MB   1.5MB/s   01:08

    real	1m16.023s
    user	0m4.548s
    sys	0m0.744s


So, 1048576900 bytes * 8 bits / byte / 76.024 seconds

= 110341671 bits/second


Testing again using Fast Ethernet (100 Mb/s):

    2016-09-09 20:29:54 dpchrist@t7400 ~
    $ time scp -p urandom.100M samba:.
    urandom.100M


                          100%  100MB   2.4MB/s   00:42

    real	0m43.377s
    user	0m4.476s
    sys	0m0.876s


So, 1048576900 bytes * 8 bits / byte / 43.377 seconds

= 193388552. bits/second


Wow.  Even worse than I was expecting...


David

[toc] | [prev] | [next] | [standalone]

#171970

On Fri 09 Sep 2016 at 20:36:39 (-0700), David Christensen wrote:
> On 09/09/2016 11:51 AM, Celejar wrote:
> > On Tue, 9 Aug 2016 18:57:02 -0700
> > David Christensen <dpchrist@holgerdanske.com> wrote:
> > 
> > ...
> > 
> >> My laptop has 802.11 a/b/g WiFi and Fast Ethernet.  Wireless data
> >> transfers are slow (~50 Mbps).  Wired is twice as fast (100 Mbps); still
> >> slow.  Newer WiFi (n, ac) should be faster, but only the newest WiFi
> >> hardware can match or beat Gigabit.
> > 
> > You get ~50Mbps over a/b/g? 54Mbps is the theoretical maximum, and
> > everything I've read says that 20-24Mbps is the real-world maximum.
> > 
> > Celejar
> > 
> 
> Benchmarking using WiFi (48 Mb/s):
> 
>     2016-09-09 20:18:51 dpchrist@t7400 ~
>     $ time dd if=/dev/urandom of=urandom.100M bs=1M count=100
>     100+0 records in
>     100+0 records out
>     104857600 bytes (105 MB) copied, 12.6709 s, 8.3 MB/s
> 
>     real	0m12.703s
>     user	0m0.000s
>     sys	0m12.481s
> 
>     2016-09-09 20:19:32 dpchrist@t7400 ~
>     $ time scp -p urandom.100M samba:.
>     urandom.100M
> 
> 
>                           100%  100MB   1.5MB/s   01:08
> 
>     real	1m16.023s
>     user	0m4.548s
>     sys	0m0.744s
> 
> 
> So, 1048576900 bytes * 8 bits / byte / 76.024 seconds
             ↑

What's this 9?

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#171972

On Fri, 9 Sep 2016 23:14:30 -0500
David Wright <deblis@lionunicorn.co.uk> wrote:

> On Fri 09 Sep 2016 at 20:36:39 (-0700), David Christensen wrote:
> > On 09/09/2016 11:51 AM, Celejar wrote:
> > > On Tue, 9 Aug 2016 18:57:02 -0700
> > > David Christensen <dpchrist@holgerdanske.com> wrote:
> > > 
> > > ...
> > > 
> > >> My laptop has 802.11 a/b/g WiFi and Fast Ethernet.  Wireless data
> > >> transfers are slow (~50 Mbps).  Wired is twice as fast (100 Mbps); still
> > >> slow.  Newer WiFi (n, ac) should be faster, but only the newest WiFi
> > >> hardware can match or beat Gigabit.
> > > 
> > > You get ~50Mbps over a/b/g? 54Mbps is the theoretical maximum, and
> > > everything I've read says that 20-24Mbps is the real-world maximum.
> > > 
> > > Celejar
> > > 
> > 
> > Benchmarking using WiFi (48 Mb/s):
> > 
> >     2016-09-09 20:18:51 dpchrist@t7400 ~
> >     $ time dd if=/dev/urandom of=urandom.100M bs=1M count=100
> >     100+0 records in
> >     100+0 records out
> >     104857600 bytes (105 MB) copied, 12.6709 s, 8.3 MB/s
> > 
> >     real	0m12.703s
> >     user	0m0.000s
> >     sys	0m12.481s
> > 
> >     2016-09-09 20:19:32 dpchrist@t7400 ~
> >     $ time scp -p urandom.100M samba:.
> >     urandom.100M
> > 
> > 
> >                           100%  100MB   1.5MB/s   01:08
> > 
> >     real	1m16.023s
> >     user	0m4.548s
> >     sys	0m0.744s
> > 
> > 
> > So, 1048576900 bytes * 8 bits / byte / 76.024 seconds
>              ↑
> 
> What's this 9?
> 
> Cheers,
> David.
> 

Assuming the talk is about transfer rates over the medium, not something like pre-compression data rates (which might be called 'marketing-speak').

Good eye! I was going to say it's not possible to get 110Mb/s over 802.11g; 40-50 is closer tothe best I get. And 193Mb/s over 100Mb/s ethernet is right out; best I've ever managed is maybe 97Mb/s, and 92-95 is more typical. 11,034,157Mb/s on W/L and 19,338,838Mb/s on wired is *much* more believable.

Unless one has a very fast multicore CPU with hardware crypto assistance, very fast RAM and the data to be transferred cached in RAM, one will probably never saturate a fastE or gigE link where one end must decrypt the data from disk/cache then encrypt the data to scp, and the other end must decrypt the data from scp then encrypt the data to disk. Even simple compression slows transfer down far too much.

Now if one had many CPUs, hacked scp to open as many sockets and thread/child procs as there are CPUs, and had each thread work on a small-ish block of data at a time, one *might* be able to speed up the tranfser.

[toc] | [prev] | [next] | [standalone]

#171980

On Sat, Sep 10, 2016 at 01:22:45AM -0400, Neal P. Murphy wrote:
> On Fri, 9 Sep 2016 23:14:30 -0500
> David Wright <deblis@lionunicorn.co.uk> wrote:
> 
> Good eye! I was going to say it's not possible to get 110Mb/s over 802.11g; 40-50 is closer tothe best I get. And 193Mb/s over 100Mb/s ethernet is right out; best I've ever managed is maybe 97Mb/s, and 92-95 is more typical. 11,034,157Mb/s on W/L and 19,338,838Mb/s on wired is *much* more believable.
> 
> Unless one has a very fast multicore CPU with hardware crypto assistance, very fast RAM and the data to be transferred cached in RAM, one will probably never saturate a fastE or gigE link where one end must decrypt the data from disk/cache then encrypt the data to scp, and the other end must decrypt the data from scp then encrypt the data to disk. Even simple compression slows transfer down far too much.

SSDs can routinely read 400-600 MB/s. No need to have everything
cached in RAM.

In 2010, the first generation of i5 CPUs with hardware support for AES
could encrypt at about 15 MB/s, more than filling a 100 Mb/s pipe.

Here's a table of recent CPUs with AES support, running with
OpenSSL/LibreSSL. https://calomel.org/aesni_ssl_performance.html

It's in megabytes per second, so assume 1000/8 = 250 MB/s is the
bandwidth of a gigabit ethernet NIC. Anything which can do 2x
that can approach encrypting/decrypting from SSD, then
decrypting/encrypting over an SSH connection.

There are a lot of 500s and above on that chart.

And that's per-core, so even the 250+ CPUs can fill a gig-e pipe
while reading from SSD.

Nor are they monstrously expensive: an AMD FX-6300 is $90, a
motherboard for it could be another $90, and you can get a
decent SSD for $100 these days. A $400 desktop can be put
together that can saturate a gig-E link with encrypted traffic
from an encrypted disk.

Truly we live in marvelous times.

-dsr-

[toc] | [prev] | [next] | [standalone]

#171981

On Saturday, September 10, 2016 08:41:53 AM Dan Ritter wrote:
> It's in megabytes per second, so assume 1000/8 = 250 MB/s is the
> bandwidth of a gigabit ethernet NIC.

Sorry, I tend to pick at nits, but, for the record, 1000/8 is 125 Mb/s.  It 
doesn't (really) change your conclusions.

regards,
Randy Kramer

[toc] | [prev] | [next] | [standalone]

#171982

On Saturday 10 September 2016 10:26:15 rhkramer@gmail.com wrote:

> On Saturday, September 10, 2016 08:41:53 AM Dan Ritter wrote:
> > It's in megabytes per second, so assume 1000/8 = 250 MB/s is the
> > bandwidth of a gigabit ethernet NIC.
>
> Sorry, I tend to pick at nits, but, for the record, 1000/8 is 125
> Mb/s.  It doesn't (really) change your conclusions.
>
> regards,
> Randy Kramer

You make an assumption many folks do, but theres a start bit and a stop 
bit so the math is more like 1000/10=100 Mb/s.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#171984

On Saturday, September 10, 2016 10:40:26 AM Gene Heskett wrote:
> On Saturday 10 September 2016 10:26:15 rhkramer@gmail.com wrote:
> > On Saturday, September 10, 2016 08:41:53 AM Dan Ritter wrote:
> > > It's in megabytes per second, so assume 1000/8 = 250 MB/s is the
> > > bandwidth of a gigabit ethernet NIC.
> > 
> > Sorry, I tend to pick at nits, but, for the record, 1000/8 is 125
> > Mb/s.  It doesn't (really) change your conclusions.
> > 
> > regards,
> > Randy Kramer
> 
> You make an assumption many folks do, but theres a start bit and a stop
> bit so the math is more like 1000/10=100 Mb/s.


Well, 1000/8 is still 125 ;-) but I wouldn't have written back just to say 
that.  Isn't it the case that there is something less than 1 start and 1 stop 
for every byte--maybe like 1 stop bit for every several bytes?  (I am just 
(slightly) curious.)

And, iirc, there are variations (which may be obsolete--I seem to remember one 
protocol that had either 2 start or 2 stop bits?

regards,
Randy Kramer

[toc] | [prev] | [next] | [standalone]

#171988

On 09/10/2016 07:53 AM, rhkramer@gmail.com wrote:
> On Saturday, September 10, 2016 10:40:26 AM Gene Heskett wrote:
>> You make an assumption many folks do, but theres a start bit and a stop
>> bit so the math is more like 1000/10=100 Mb/s.
> 
> 
> Well, 1000/8 is still 125 ;-) but I wouldn't have written back just to say 
> that.  Isn't it the case that there is something less than 1 start and 1 stop 
> for every byte--maybe like 1 stop bit for every several bytes?  (I am just 
> (slightly) curious.)
> 
> And, iirc, there are variations (which may be obsolete--I seem to remember one 
> protocol that had either 2 start or 2 stop bits?

I remember start/stop bits from RS-232/485, but Gigabit Ethernet
signaling is more advanced:

    https://en.wikipedia.org/wiki/Gigabit_Ethernet


David

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web