Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #231684 > unrolled thread

rsync to NAS for backup

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  rsync to NAS for backup mick crane <mick.crane@gmail.com> - 2021-02-13 10:30 +0100
    Re: rsync to NAS for backup didier gaumet <didier.gaumet@gmail.com> - 2021-02-13 14:00 +0100
      Re: rsync to NAS for backup Toni Mas Soler <antomassol@protonmail.com> - 2021-02-13 14:00 +0100
    Re: rsync to NAS for backup Charles Curley <charlescurley@charlescurley.com> - 2021-02-13 14:20 +0100
    Re: rsync to NAS for backup David Christensen <dpchrist@holgerdanske.com> - 2021-02-13 20:30 +0100
      Re: rsync to NAS for backup Gary Dale <gary@extremeground.com> - 2021-02-18 17:20 +0100
        Re: rsync to NAS for backup Teemu Likonen <tlikonen@iki.fi> - 2021-02-18 18:10 +0100
          Re: rsync to NAS for backup <tomas@tuxteam.de> - 2021-02-18 18:30 +0100
            Re: rsync to NAS for backup Charles Curley <charlescurley@charlescurley.com> - 2021-02-19 01:10 +0100
              Re: rsync to NAS for backup <tomas@tuxteam.de> - 2021-02-19 10:00 +0100
          Re: rsync to NAS for backup <tomas@tuxteam.de> - 2021-02-18 18:30 +0100
            Re: rsync to NAS for backup Gary Dale <gary@extremeground.com> - 2021-02-18 20:10 +0100
          Re: rsync to NAS for backup Andrei POPESCU <andreimpopescu@gmail.com> - 2021-02-18 18:30 +0100
      Re: rsync to NAS for backup Mirko Parthey <mirko.parthey@web.de> - 2021-02-19 16:30 +0100

#231684 — rsync to NAS for backup

I made a mistake and instead of getting a PC for backup I got a NAS.
I'm struggling to get to grips with it.
If rsync from PC to NAS NAS changes the owner/group of files to me/users 
which is probably no good for backing up.
There's that problem then another that it won't let me login as root.
I asked on Synology forum but not getting a lot of joy.
https://community.synology.com/enu/forum/1/post/141137
Anybody used these things can advise ?

mick

-- 
Key ID    4BFEBB31

[toc] | [next] | [standalone]

#231686

Hello,

Disclaimer: I do not use and am not familiar with Sinology hardware and 
software and generally speaking, I am not knowledgeable in networking

I would say that:

- the owner:group names of a file on the PC you backup and the 
owner:group names of the backup files on the synology files might be 
different, even if you try to maintain ownership and rights. What really 
counts here are owner:group identifiers (UID:GID). Bob_user:Bob_group on 
your PC might equate to Alice_user:John_group on your NAS. Upon 
restoration that would be reversed to Bob_user:Bob_group.
  That would be typical without something like a LDAP server.

- SSH root login seems to be discouraged for security reasons. Sinology 
probably adhere to this principle and the appropriate way to do what you 
want would probably be to access a shell on the Synology software to 
issue a sudo or su -c command.

- editing /etc/sudoers is generally done via the visudo command

- if that is of interest to you, there is a way to install Debian in 
chroot on your NAS

[toc] | [prev] | [next] | [standalone]

#231687

[Multipart message — attachments visible in raw view] — view raw

Is there an alternative if you want an incremental backup?

Obviously you could use tar-ed archives with unprivileged permissions. If you did, you would get a huge network overhead.

thks


Toni Mas
GPG 3F42A21D84D7E950

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
En dissabte 13 de febrer de 2021 a les 13:50, didier gaumet <didier.gaumet@gmail.com> va escriure:

> Hello,
> 

> Disclaimer: I do not use and am not familiar with Sinology hardware and
> software and generally speaking, I am not knowledgeable in networking
> 

> I would say that:
> 

> -   the owner:group names of a file on the PC you backup and the
>     owner:group names of the backup files on the synology files might be
>     different, even if you try to maintain ownership and rights. What really
>     counts here are owner:group identifiers (UID:GID). Bob_user:Bob_group on
>     your PC might equate to Alice_user:John_group on your NAS. Upon
>     restoration that would be reversed to Bob_user:Bob_group.
>     That would be typical without something like a LDAP server.
>     

> -   SSH root login seems to be discouraged for security reasons. Sinology
>     probably adhere to this principle and the appropriate way to do what you
>     want would probably be to access a shell on the Synology software to
>     issue a sudo or su -c command.
>     

> -   editing /etc/sudoers is generally done via the visudo command
> -   if that is of interest to you, there is a way to install Debian in
>     chroot on your NAS
>

[toc] | [prev] | [next] | [standalone]

#231688

On Sat, 13 Feb 2021 09:27:54 +0000
mick crane <mick.crane@gmail.com> wrote:

> I made a mistake and instead of getting a PC for backup I got a NAS.
> I'm struggling to get to grips with it.
> If rsync from PC to NAS NAS changes the owner/group of files to
> me/users which is probably no good for backing up.

Can you set up a network file system such as NFS on the NAS, then tar
your data to the NAS?

Can you install a backup server such as amanda on the NAS?


-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[toc] | [prev] | [next] | [standalone]

#231705

On 2021-02-13 01:27, mick crane wrote:
> I made a mistake and instead of getting a PC for backup I got a NAS.
> I'm struggling to get to grips with it.
> If rsync from PC to NAS NAS changes the owner/group of files to me/users 
> which is probably no good for backing up.
> There's that problem then another that it won't let me login as root.
> I asked on Synology forum but not getting a lot of joy.
> https://community.synology.com/enu/forum/1/post/141137
> Anybody used these things can advise ?

What is the model of the Synology NAS?  What options -- CPU, memory, 
disks, bays, interfaces, PSU, whatever?  Support page URL?


Reading the forum post, it sounds like you damaged the sudoers file. 
The fix would appear to be doing a Mode 2 reset per Synology's instructions:

https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS


Once the NAS has been reset, figure out how to meet your needs within 
the framework provided by Synology.  Follow the User Guide.  Follow the 
Admin Guide.  Do not mess around "under the hood" with a terminal and 
sudo.  Make Synology earn your money.


But if you want complete control, buy or build an x86_64/amd64 server, 
install Debian, and have at it.


David

[toc] | [prev] | [next] | [standalone]

#231827

On 2021-02-18 10:57, mick crane wrote:
> On 2021-02-15 12:39, mick crane wrote:
>> On 2021-02-13 19:20, David Christensen wrote:
>>> On 2021-02-13 01:27, mick crane wrote:
>>>> I made a mistake and instead of getting a PC for backup I got a NAS.
>>>> I'm struggling to get to grips with it.
>>>> If rsync from PC to NAS NAS changes the owner/group of files to 
>>>> me/users which is probably no good for backing up.
>>>> There's that problem then another that it won't let me login as root.
>>>> I asked on Synology forum but not getting a lot of joy.
>>>> https://community.synology.com/enu/forum/1/post/141137
>>>> Anybody used these things can advise ?
>>>
>>> What is the model of the Synology NAS?  What options -- CPU, memory,
>>> disks, bays, interfaces, PSU, whatever?  Support page URL?
>>>
>>>
>>> Reading the forum post, it sounds like you damaged the sudoers file.
>>> The fix would appear to be doing a Mode 2 reset per Synology's
>>> instructions:
>>>
>>> https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS 
>>>
>>>
>>>
>>> Once the NAS has been reset, figure out how to meet your needs within
>>> the framework provided by Synology.  Follow the User Guide. Follow
>>> the Admin Guide.  Do not mess around "under the hood" with a terminal
>>> and sudo.  Make Synology earn your money.
>>>
>>>
>>> But if you want complete control, buy or build an x86_64/amd64 server,
>>> install Debian, and have at it.
>>>
>>>
>>
>> thanks for advices folks.
>> It was indeed user error with being in a rush and blurred eyesight
>> mistook "%" for "#"
>> We are making progress.
>> Appears that to retain permissions need root at both ends of rsync.
>> Have keys working with ssh for users to NAS ( not helped by default
>> permissions for .ssh files being wrong) and can su to root so now need
>> to get ssh working with keys with no passphrase for root and all
>> should be good.
>
> further to this if it helps anybody can start sshd with -d switch and 
> at same time do client with -vv switch then can see where is falling 
> down. Having telnet available helps if break sshd_config can still 
> telnet and mend it.
> mick
>
rsync is a quick & dirty backup tactic but it's got limitations.

1) files may stay around forever in the backup even if you've deleted 
them from your main computer because you don't need them.

2) you only have one copy of a file and that only lasts until the next 
rsync. This limits your ability to restore from a backup before it is 
overwritten.


Using a real backup program, which can run on you main computer to 
backup to the NAS, lets you define a retention policy so files no longer 
needed can be purged while you have multiple backups of files you are 
currently working on.

rsync is not a good substitute for backups.

[toc] | [prev] | [next] | [standalone]

#231828

[Multipart message — attachments visible in raw view] — view raw

* 2021-02-18 11:13:25-0500, Gary Dale wrote:

> rsync is a quick & dirty backup tactic but it's got limitations.
>
> 1) files may stay around forever in the backup even if you've deleted 
> them from your main computer because you don't need them.
>
> 2) you only have one copy of a file and that only lasts until the next 
> rsync. This limits your ability to restore from a backup before it is 
> overwritten.

> rsync is not a good substitute for backups.

Rsync is great backup program with "--link-dest" option. Here is the
idea in simplified code:

    cd "$destination"
    new=$(date +%Y%m%dT%H%M%S%z)
    if mkdir "$new" && \
        rsync -aHAX --link-dest=latest_backup [...] \
        "$source" "$new"
    then
        # Recreate symlink to point to the new backup directory.
        ln -sfn "$new" latest_backup
        # Delete old backups.
        find . -mindepth 1 -maxdepth 1 -type d -mtime +365 \
            -exec /bin/rm -fr -- {} +
    fi

With that sort of code every backup is a new complete directory tree in
a time stamped directory. If files have not changed since the latest
backup "--link-dest" creates hard links. Old backups can be deleted by
removing old (or any) directory tree. The trees don't depend on each
other.

What else do we need? OK, some people may need compression but usually
hard disk space is cheap.

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[toc] | [prev] | [next] | [standalone]

#231830

[Multipart message — attachments visible in raw view] — view raw

On Thu, Feb 18, 2021 at 07:21:01PM +0200, Andrei POPESCU wrote:

[...]

> In my opinion the point still stands, rsync (by itself) has significant 
> limitations as a backup program, which is probably also the reason why 
> several backup programs using rsync exist.

Care to name some of those limitations?

(of course, rsync /is not/ a backup program in itself, but lets you
build one with 10-30 lines around it).

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#231845

[Multipart message — attachments visible in raw view] — view raw

On Thu, 18 Feb 2021 18:24:43 +0100
<tomas@tuxteam.de> wrote:

> Care to name some of those limitations?
> 
> (of course, rsync /is not/ a backup program in itself, but lets you
> build one with 10-30 lines around it).

Why build one (which I have done) when you can get a perfectly good one
from the Debian repos? E.g.: rsnapshot. Others have been mentioned on
this thread.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[toc] | [prev] | [next] | [standalone]

#231854

[Multipart message — attachments visible in raw view] — view raw

On Thu, Feb 18, 2021 at 05:06:33PM -0700, Charles Curley wrote:
> On Thu, 18 Feb 2021 18:24:43 +0100
> <tomas@tuxteam.de> wrote:
> 
> > Care to name some of those limitations?
> > 
> > (of course, rsync /is not/ a backup program in itself, but lets you
> > build one with 10-30 lines around it).
> 
> Why build one (which I have done) when you can get a perfectly good one
> from the Debian repos? E.g.: rsnapshot. Others have been mentioned on
> this thread.

My approach is: if reading the doc takes me significantly longer than
building, I do the latter.

In this case, this is... uh... the case :-)

YMMV, TANSTAAFL and all that.

For me, this is the end of the thread (I feel we've been everywhere,
everyone's happy with her solution and that).

Cheers

> -- 
> Does anybody read signatures any more?

I do, yes.

 - t

[toc] | [prev] | [next] | [standalone]

#231832

[Multipart message — attachments visible in raw view] — view raw

On Thu, Feb 18, 2021 at 06:59:03PM +0200, Teemu Likonen wrote:
> * 2021-02-18 11:13:25-0500, Gary Dale wrote:
> 
> > rsync is a quick & dirty backup tactic but it's got limitations.
> >
> > 1) files may stay around forever in the backup even if you've deleted 
> > them from your main computer because you don't need them.
> >
> > 2) you only have one copy of a file and that only lasts until the next 
> > rsync. This limits your ability to restore from a backup before it is 
> > overwritten.
> 
> > rsync is not a good substitute for backups.

No, it's not. It is a fantastic tool for backups :-)

> Rsync is great backup program with "--link-dest" option. Here is the
> idea in simplified code:

[...]

Absolutely. Time travel!

Actually, I've implemented this at a customer's place. They were
delighted.

Where rsync shows some weaknesses is on big, fat files (think
videos, one or several GB).

Really huge directories (tens to hundreds of TB) were once a
challenge, too, but I hear that they refined the scanning
part in the meantime. No direct experience, though.

And, oh, Gary: if you want to delete files which disappeared
in the source, check out the --delete option.

But this time-staggered backup with --link-dest is really great.

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#231834

On 2021-02-18 12:22, tomas@tuxteam.de wrote:
> On Thu, Feb 18, 2021 at 06:59:03PM +0200, Teemu Likonen wrote:
>> * 2021-02-18 11:13:25-0500, Gary Dale wrote:
>>
>>> rsync is a quick & dirty backup tactic but it's got limitations.
>>>
>>> 1) files may stay around forever in the backup even if you've deleted
>>> them from your main computer because you don't need them.
>>>
>>> 2) you only have one copy of a file and that only lasts until the next
>>> rsync. This limits your ability to restore from a backup before it is
>>> overwritten.
>>> rsync is not a good substitute for backups.
> No, it's not. It is a fantastic tool for backups :-)
>
>> Rsync is great backup program with "--link-dest" option. Here is the
>> idea in simplified code:
> [...]
>
> Absolutely. Time travel!
>
> Actually, I've implemented this at a customer's place. They were
> delighted.
>
> Where rsync shows some weaknesses is on big, fat files (think
> videos, one or several GB).
>
> Really huge directories (tens to hundreds of TB) were once a
> challenge, too, but I hear that they refined the scanning
> part in the meantime. No direct experience, though.
>
> And, oh, Gary: if you want to delete files which disappeared
> in the source, check out the --delete option.
>
> But this time-staggered backup with --link-dest is really great.
>
> Cheers

While you can twist any tool to fit a task, real backup programs don't 
need to be twisted and do a better job. For example backup retention 
policy is intuitive and easy to set. Some backup programs even factor 
out common blocks for de-duplication, which can save a lot of space. 
Hard-links only do that if the file name is the same.

And when you need to restore a file, backup programs usually let you see 
when the files changed then let you choose which version to restore.

As for the delete option, it makes the rsync script even more 
complicated. A backup program will simply expire the file at the end of 
the retention period.

[toc] | [prev] | [next] | [standalone]

#231833

[Multipart message — attachments visible in raw view] — view raw

On Jo, 18 feb 21, 18:59:03, Teemu Likonen wrote:
> * 2021-02-18 11:13:25-0500, Gary Dale wrote:
> 
> > rsync is a quick & dirty backup tactic but it's got limitations.
> >
> > 1) files may stay around forever in the backup even if you've deleted 
> > them from your main computer because you don't need them.
> >
> > 2) you only have one copy of a file and that only lasts until the next 
> > rsync. This limits your ability to restore from a backup before it is 
> > overwritten.
> 
> > rsync is not a good substitute for backups.
> 
> Rsync is great backup program with "--link-dest" option. Here is the
> idea in simplified code:
 
[...]
 
> With that sort of code every backup is a new complete directory tree in
> a time stamped directory. If files have not changed since the latest
> backup "--link-dest" creates hard links. Old backups can be deleted by
> removing old (or any) directory tree. The trees don't depend on each
> other.
> 
> What else do we need? OK, some people may need compression but usually
> hard disk space is cheap.

In my opinion the point still stands, rsync (by itself) has significant 
limitations as a backup program, which is probably also the reason why 
several backup programs using rsync exist.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[toc] | [prev] | [next] | [standalone]

#231863

On Mon, Feb 15, 2021 at 12:39:29PM +0000, mick crane wrote:
> Appears that to retain permissions need root at both ends of rsync.

Not necessarily. If the server filesystem supports xattrs, you can use
the --fake-super option with the rsync server, running as a non-root user
that can write to the storage.

To set the option on the client side for rsync over ssh, use:

--rsync-path "rsync --fake-super"

Regards,
Mirko

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web