Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #255609 > unrolled thread

what method do you prefer for data transfer between nodes?

Back to article view | Back to linux.debian.user

Contents

  what method do you prefer for data transfer between nodes? Ken Young <ken@highwinds.cloud> - 2023-03-05 09:30 +0100
    Re: what method do you prefer for data transfer between nodes? <tomas@tuxteam.de> - 2023-03-05 10:20 +0100
      Re: what method do you prefer for data transfer between nodes? Michel Verdier <mv524@free.fr> - 2023-03-05 11:40 +0100
        Re: what method do you prefer for data transfer between nodes? <tomas@tuxteam.de> - 2023-03-05 12:00 +0100
          Re: what method do you prefer for data transfer between nodes? Ken Young <ken@highwinds.cloud> - 2023-03-05 12:30 +0100
            Re: what method do you prefer for data transfer between nodes? tomas@tuxteam.de - 2023-03-05 14:10 +0100
              Re: what method do you prefer for data transfer between nodes? Ken Young <ken@highwinds.cloud> - 2023-03-05 14:50 +0100
                Re: what method do you prefer for data transfer between nodes? Rodrigo Cunha <rodrigo.root.rj@gmail.com> - 2023-03-05 20:00 +0100
              Re: what method do you prefer for data transfer between nodes? Celejar <celejar@gmail.com> - 2023-03-07 02:30 +0100
                Re: what method do you prefer for data transfer between nodes? Ken Young <ken@highwinds.cloud> - 2023-03-07 02:30 +0100
    Re: what method do you prefer for data transfer between nodes? Michel Verdier <mv524@free.fr> - 2023-03-05 11:40 +0100
    Re: what method do you prefer for data transfer between          nodes? Linux-Fan <Ma_Sys.ma@web.de> - 2023-03-05 20:50 +0100
      Re: what method do you prefer for data transfer between nodes? Ken Young <ken@highwinds.cloud> - 2023-03-06 00:50 +0100
    Re: what method do you prefer for data transfer between nodes? Charles Curley <charlescurley@charlescurley.com> - 2023-03-06 01:20 +0100
    Re: what method do you prefer for data transfer between nodes? David Christensen <dpchrist@holgerdanske.com> - 2023-03-06 12:10 +0100
    Re: what method do you prefer for data transfer between nodes? Max Nikulin <manikulin@gmail.com> - 2023-03-06 12:50 +0100
      Re: what method do you prefer for data transfer between nodes? <tomas@tuxteam.de> - 2023-03-06 13:20 +0100
    Re: what method do you prefer for data transfer between nodes? Vincent Lefevre <vincent@vinc17.net> - 2023-03-07 03:40 +0100

#255609 — what method do you prefer for data transfer between nodes?

[Multipart message — attachments visible in raw view] — view raw

Hello,

The methods I know,

1. scp
pros: the native tool in the OS
cons: you will either input password or put key pairs into servers for
authentication.

2. rsync
pros: it can transfer data by increasement
cons: you need to setup rsyncd server and make the correct authorization.

3. ftp/ftps
pros: easy to use
cons: need to setup ftpd server, and the way is not that secure?

4. rclone
pros:easy to use
cons: hard to setup (you may need a cloud storage for middleware).

For me I most often use scp + rsync. and what's your choice?

Regards,
Ken

[toc] | [next] | [standalone]

#255613

[Multipart message — attachments visible in raw view] — view raw

On Sun, Mar 05, 2023 at 04:22:25PM +0800, Ken Young wrote:
> Hello,
> 
> The methods I know,
> 
> 1. scp
> pros: the native tool in the OS
> cons: you will either input password or put key pairs into servers for
> authentication.

Sometimes. From the command line, and if I only have to transfer a
hadnful of files, this one is the most convenient.

> 2. rsync
> pros: it can transfer data by increasement
> cons: you need to setup rsyncd server and make the correct authorization.

This. Hands down.

Pros:
Your list of pros is just too short. A corollary
of your first pro is that you can interrupt the proces at any time
and it'll pick up more or less where it left. Plus you can ask it
what it would do (--dry-run). Plus you you can tell it to make
back-up files at the other end, and to delete non-existing files.
Yadda, yadda.

Cons:
 - you need to setup rsyncd server
   Wrong: you just need one "transport", SSH is fine.
   If you can access the other box via SSH, you're set.

 - and make the correct authorization.
   I don't know what you mean with that: I'd hope any
   method to put files on a box would need proper
   authentificationn/authorization.

The others I'd only use when forced to. Well, rclone, I think,
I'd even refuse when forced to. Eek.

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#255614

Le 5 mars 2023 tomas a écrit :

> The others I'd only use when forced to. Well, rclone, I think,
> I'd even refuse when forced to. Eek.

Do you have an alternate solution for cloud storage like google one ?

[toc] | [prev] | [next] | [standalone]

#255616

[Multipart message — attachments visible in raw view] — view raw

On Sun, Mar 05, 2023 at 11:31:42AM +0100, Michel Verdier wrote:
> Le 5 mars 2023 tomas a écrit :
> 
> > The others I'd only use when forced to. Well, rclone, I think,
> > I'd even refuse when forced to. Eek.
> 
> Do you have an alternate solution for cloud storage like google one ?

I don't "do" clouds. But around here, I'd recommend a provider who offers
mail and nextcloud for a small monthly amount (1 EUR, I think).

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#255619

[Multipart message — attachments visible in raw view] — view raw

What provider do you recommend then?

Sincerely,
Ken Young


On Sun, Mar 5, 2023 at 6:54 PM <tomas@tuxteam.de> wrote:

> On Sun, Mar 05, 2023 at 11:31:42AM +0100, Michel Verdier wrote:
> > Le 5 mars 2023 tomas a écrit :
> >
> > > The others I'd only use when forced to. Well, rclone, I think,
> > > I'd even refuse when forced to. Eek.
> >
> > Do you have an alternate solution for cloud storage like google one ?
>
> I don't "do" clouds. But around here, I'd recommend a provider who offers
> mail and nextcloud for a small monthly amount (1 EUR, I think).
>
> Cheers
> --
> t
>

[toc] | [prev] | [next] | [standalone]

#255624

[Multipart message — attachments visible in raw view] — view raw

On Sun, Mar 05, 2023 at 07:23:49PM +0800, Ken Young wrote:
> What provider do you recommend then?

Around here (West Europe), for example, mailbox.org [1]. But they
are far from the only one (I'm not a customer, nor associated with
them, but I do know a few happy customers).

Whenever they don't cost any money you'll have to ask yourself
what their business model is.

Cheers

[1] https://mailbox.org/en/
-- 
t

[toc] | [prev] | [next] | [standalone]

#255626

[Multipart message — attachments visible in raw view] — view raw

Thanks for the suggestion. I will give it a try on mailbox.org.


Sincerely,
Ken Young


On Sun, Mar 5, 2023 at 9:05 PM <tomas@tuxteam.de> wrote:

> On Sun, Mar 05, 2023 at 07:23:49PM +0800, Ken Young wrote:
> > What provider do you recommend then?
>
> Around here (West Europe), for example, mailbox.org [1]. But they
> are far from the only one (I'm not a customer, nor associated with
> them, but I do know a few happy customers).
>
> Whenever they don't cost any money you'll have to ask yourself
> what their business model is.
>
> Cheers
>
> [1] https://mailbox.org/en/
> --
> t
>

[toc] | [prev] | [next] | [standalone]

#255629

[Multipart message — attachments visible in raw view] — view raw

Hi Ken,
For short, I use scp and rsync too. Work fine with my scripts. Ftp works
fine on old servers. I saw old servers work with this, but with monitoring
in the FTP port for recovery. Another is to mount an NFS directory at
/etc/fstab. Work fine, too. If you have a critical script at the
scheduler(crontab) may be monitoring this with logs.

On Sun, Mar 5, 2023 at 10:48 AM Ken Young <ken@highwinds.cloud> wrote:

> Thanks for the suggestion. I will give it a try on mailbox.org.
>
>
> Sincerely,
> Ken Young
>
>
> On Sun, Mar 5, 2023 at 9:05 PM <tomas@tuxteam.de> wrote:
>
>> On Sun, Mar 05, 2023 at 07:23:49PM +0800, Ken Young wrote:
>> > What provider do you recommend then?
>>
>> Around here (West Europe), for example, mailbox.org [1]. But they
>> are far from the only one (I'm not a customer, nor associated with
>> them, but I do know a few happy customers).
>>
>> Whenever they don't cost any money you'll have to ask yourself
>> what their business model is.
>>
>> Cheers
>>
>> [1] https://mailbox.org/en/
>> --
>> t
>>
>

-- 
Atenciosamente,
Rodrigo da Silva Cunha
São Gonçalo, RJ - Brasil

[toc] | [prev] | [next] | [standalone]

#255661

On Sun, 5 Mar 2023 14:05:37 +0100
tomas@tuxteam.de wrote:

> On Sun, Mar 05, 2023 at 07:23:49PM +0800, Ken Young wrote:
> > What provider do you recommend then?
> 
> Around here (West Europe), for example, mailbox.org [1]. But they
> are far from the only one (I'm not a customer, nor associated with
> them, but I do know a few happy customers).

€3 / month for 5 GB, €9 / month for 50 GB - that's rather overpriced
for cloud storage:

https://mailbox.org/en/services#tariffs

> Whenever they don't cost any money you'll have to ask yourself
> what their business model is.

The big cloud providers (e.g. Backblaze) - whose business model does
indeed involve charging money - charge as little as $5 / month per
*terabyte* (plus egress), in addition to giving a few or few dozen GB
free:

https://www.backblaze.com/b2/cloud-storage-pricing.html

I'm personally using Scaleway with rclone, which offers 75 GB free (but
€12 / month per terabyte beyond that).

Other prominent vendors (besides the obvious ones) include Wasabi and
rsync.net. Pricing is tricky, as they break the fees into different
categories (hot and cold storage, egress, etc.)

-- 
Celejar

[toc] | [prev] | [next] | [standalone]

#255662

[Multipart message — attachments visible in raw view] — view raw

For standalone backup I am using Liteserver's VPS, 512G disk for only 2.4
EUR/m.
Then I setup rsyncd on it, and rsync data from clients via crontab for
backup purpose.


Sincerely,
Ken Young


On Tue, Mar 7, 2023 at 9:21 AM Celejar <celejar@gmail.com> wrote:

> On Sun, 5 Mar 2023 14:05:37 +0100
> tomas@tuxteam.de wrote:
>
> > On Sun, Mar 05, 2023 at 07:23:49PM +0800, Ken Young wrote:
> > > What provider do you recommend then?
> >
> > Around here (West Europe), for example, mailbox.org [1]. But they
> > are far from the only one (I'm not a customer, nor associated with
> > them, but I do know a few happy customers).
>
> €3 / month for 5 GB, €9 / month for 50 GB - that's rather overpriced
> for cloud storage:
>
> https://mailbox.org/en/services#tariffs
>
> > Whenever they don't cost any money you'll have to ask yourself
> > what their business model is.
>
> The big cloud providers (e.g. Backblaze) - whose business model does
> indeed involve charging money - charge as little as $5 / month per
> *terabyte* (plus egress), in addition to giving a few or few dozen GB
> free:
>
> https://www.backblaze.com/b2/cloud-storage-pricing.html
>
> I'm personally using Scaleway with rclone, which offers 75 GB free (but
> €12 / month per terabyte beyond that).
>
> Other prominent vendors (besides the obvious ones) include Wasabi and
> rsync.net. Pricing is tricky, as they break the fees into different
> categories (hot and cold storage, egress, etc.)
>
> --
> Celejar
>
>

[toc] | [prev] | [next] | [standalone]

#255615

Le 5 mars 2023 Ken Young a écrit :

> 2. rsync
> pros: it can transfer data by increasement
> cons: you need to setup rsyncd server and make the correct authorization.

Definitely rsync over ssh : easy and secure. You can further filter
commands and automate with proper ssh keys.

> 4. rclone
> pros:easy to use
> cons: hard to setup (you may need a cloud storage for middleware).

Only for sync with cloud storage

[toc] | [prev] | [next] | [standalone]

#255633 — Re: what method do you prefer for data transfer between nodes?

[Multipart message — attachments visible in raw view] — view raw

Ken Young writes:

> Hello,[1;5B
>
>
> The methods I know,
>
> 1. scp
> pros: the native tool in the OS
> cons: you will either input password or put key pairs into servers for  
> authentication.

Works for simple cases.

> 2. rsync
> pros: it can transfer data by increasement 
> cons: you need to setup rsyncd server and make the correct authorization.

Works for simple and complex cases.

> 3. ftp/ftps
> pros: easy to use
> cons: need to setup ftpd server, and the way is not that secure?

Whenever possible, I'd prefer 1 or 2 over this.

> 4. rclone
> pros:easy to use
> cons: hard to setup (you may need a cloud storage for middleware).

I only use rclone when I want to target a cloud storage.
A „cloud storage for middleware” does not seem sensible to me when I can  
copy using methods 1 and 2 without using such a middleware.

> For me I most often use scp + rsync. and what's your choice?

These are my standard choices, too. In automated scenarios I often prefer  
rsync over scp due to more flexibility in configuration.

My additional tools for special purposes:

5. lsyncd
If you need to keep directories in sync continuously, there is a tool called  
`lsyncd` that automates repeated invocation of `rsync` in a smart way.

6. tar + netcat (or tar + ssh in very rare cases)
Using tar sacrifices all the flexibility of rsync but may attain a  
significantly higher performance and does not need a lot of flags to do the  
right thing by default (i.e. preserve everything when acting as root). I  
prefer this variant when migrating to a new disk or PC because it seems to  
be the most efficient variant in a "local trusted network and no speedup  
from incremental copying" scenario.

I documented my approach to this here:
https://masysma.net/37/data_transfer_netcat_tar.xhtml

HTH and YMMV
Linux-Fan

öö

[toc] | [prev] | [next] | [standalone]

#255644

[Multipart message — attachments visible in raw view] — view raw

That's smart. Thanks

Sincerely,
Ken Young


On Mon, Mar 6, 2023 at 3:43 AM Linux-Fan <Ma_Sys.ma@web.de> wrote:

> Ken Young writes:
>
> > Hello,[1;5B
> >
> >
> > The methods I know,
> >
> > 1. scp
> > pros: the native tool in the OS
> > cons: you will either input password or put key pairs into servers for
> > authentication.
>
> Works for simple cases.
>
> > 2. rsync
> > pros: it can transfer data by increasement
> > cons: you need to setup rsyncd server and make the correct authorization.
>
> Works for simple and complex cases.
>
> > 3. ftp/ftps
> > pros: easy to use
> > cons: need to setup ftpd server, and the way is not that secure?
>
> Whenever possible, I'd prefer 1 or 2 over this.
>
> > 4. rclone
> > pros:easy to use
> > cons: hard to setup (you may need a cloud storage for middleware).
>
> I only use rclone when I want to target a cloud storage.
> A „cloud storage for middleware” does not seem sensible to me when I can
> copy using methods 1 and 2 without using such a middleware.
>
> > For me I most often use scp + rsync. and what's your choice?
>
> These are my standard choices, too. In automated scenarios I often prefer
> rsync over scp due to more flexibility in configuration.
>
> My additional tools for special purposes:
>
> 5. lsyncd
> If you need to keep directories in sync continuously, there is a tool
> called
> `lsyncd` that automates repeated invocation of `rsync` in a smart way.
>
> 6. tar + netcat (or tar + ssh in very rare cases)
> Using tar sacrifices all the flexibility of rsync but may attain a
> significantly higher performance and does not need a lot of flags to do
> the
> right thing by default (i.e. preserve everything when acting as root). I
> prefer this variant when migrating to a new disk or PC because it seems
> to
> be the most efficient variant in a "local trusted network and no speedup
> from incremental copying" scenario.
>
> I documented my approach to this here:
> https://masysma.net/37/data_transfer_netcat_tar.xhtml
>
> HTH and YMMV
> Linux-Fan
>
> öö
>

[toc] | [prev] | [next] | [standalone]

#255646

On Sun, 5 Mar 2023 16:22:25 +0800
Ken Young <ken@highwinds.cloud> wrote:

> Hello,
> 
> The methods I know,
> 
> 1. scp
> pros: the native tool in the OS
> cons: you will either input password or put key pairs into servers for
> authentication.

Not so much of a con. I now automatically add my public keys to a
machine as part of the bring-up process after installation.

> 
> 2. rsync
> pros: it can transfer data by increasement
> cons: you need to setup rsyncd server and make the correct
> authorization.

Nope. rsync over ssh handles that.

> 
> 3. ftp/ftps
> pros: easy to use
> cons: need to setup ftpd server, and the way is not that secure?

I use ftp only for one server where I don't have rsync over ssh
service. One of several reasons I want to abandon that server.

> 
> 4. rclone
> pros:easy to use
> cons: hard to setup (you may need a cloud storage for middleware).

I don't use cloud storage other than my own nextcloud server.

> 
> For me I most often use scp + rsync. and what's your choice?

It depends on what I am doing. I use rsnapshot for backups. scp for
one-offs. I use emacs' tramp mode to edit files on other machines. (It
uses scp by default bu you can specify other transport mechanisms.)

I also use unison, nextcloud and syncthing to sync user data between my
desktop and my laptops. The former is manual for occasional use, the
latter two run automatically as files change. The first two I use when I
am away to back up the laptop; the latter I don't use when away. (Hotels
tend to have crappy networks.)



-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[toc] | [prev] | [next] | [standalone]

#255653

On 3/5/23 00:22, Ken Young wrote:
> Hello,
> 
> The methods I know,
> 
> 1. scp
> pros: the native tool in the OS
> cons: you will either input password or put key pairs into servers for
> authentication.
> 
> 2. rsync
> pros: it can transfer data by increasement
> cons: you need to setup rsyncd server and make the correct authorization.
> 
> 3. ftp/ftps
> pros: easy to use
> cons: need to setup ftpd server, and the way is not that secure?
> 
> 4. rclone
> pros:easy to use
> cons: hard to setup (you may need a cloud storage for middleware).
> 
> For me I most often use scp + rsync. and what's your choice?
> 
> Regards,
> Ken


Please tell us about your data, your nodes, and the mechanism(s) 
available for transferring data.


David

[toc] | [prev] | [next] | [standalone]

#255654

On 05/03/2023 15:22, Ken Young wrote:
> 
> 1. scp

Notice that in bookworm (next release) scp uses SFTP protocol. This 
ssh-based protocol can be used in the current stable release as well. Do 
not confuse it with ftps that is traditional FTP over TLS.

[toc] | [prev] | [next] | [standalone]

#255656

[Multipart message — attachments visible in raw view] — view raw

On Mon, Mar 06, 2023 at 06:41:58PM +0700, Max Nikulin wrote:
> On 05/03/2023 15:22, Ken Young wrote:
> > 
> > 1. scp
> 
> Notice that in bookworm (next release) scp uses SFTP protocol. This
> ssh-based protocol can be used in the current stable release as well. Do not
> confuse it with ftps that is traditional FTP over TLS.

"Classical" scp is going away. It allowed one to play nasty tricks no
one really wants played. For background, see [1].

The "easy cases" many people mentioned in this thread probably won't
notice the fake scp coming now :-)

Cheers

[1] https://lwn.net/Articles/835962/
-- 
t

[toc] | [prev] | [next] | [standalone]

#255663

On 2023-03-05 16:22:25 +0800, Ken Young wrote:
> 1. scp
> pros: the native tool in the OS
> cons: you will either input password or put key pairs into servers for
> authentication.

I would not see this as a cons.

> 2. rsync
> pros: it can transfer data by increasement
> cons: you need to setup rsyncd server and make the correct authorization.

When I use rsync, I use it always over SSH (for security), and
no need for rsyncd in this case.

> 3. ftp/ftps
> pros: easy to use
> cons: need to setup ftpd server, and the way is not that secure?

I use sftp to transfer files between my Debian laptop and my
Android phones. This is secure, as this is a protocol over SSH.

I also use Subversion (over SSH) for files I want to keep.

And unison for my mail (stored in Maildir format).

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web