Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > alt.os.linux > #54222 > unrolled thread
| Started by | Soviet_Mario <SovietMario@CCCP.MIR> |
|---|---|
| First post | 2019-02-26 19:47 +0100 |
| Last post | 2019-02-26 19:51 +0000 |
| Articles | 20 on this page of 79 — 14 participants |
Back to article view | Back to alt.os.linux
freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-26 19:47 +0100
Re: freshly installed SW not found "J.O. Aho" <user@example.net> - 2019-02-26 20:28 +0100
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-26 21:51 +0100
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-26 21:46 +0000
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 12:52 +0100
Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-02-27 12:44 +0000
Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-02-27 13:27 +0000
Re: freshly installed SW not found Richard Kettlewell <invalid@invalid.invalid> - 2019-02-27 15:16 +0000
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 16:59 +0100
Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-02-27 16:02 +0000
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 20:46 +0100
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-27 20:18 +0000
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-28 12:47 +0100
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-28 18:32 +0000
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-03-01 01:14 +0100
Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-01 01:54 +0100
Re: freshly installed SW not found Jasen Betts <jasen@xnet.co.nz> - 2019-03-01 03:09 +0000
Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-02-27 20:21 +0000
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-27 23:02 +0000
Re: freshly installed SW not found Richard Kettlewell <invalid@invalid.invalid> - 2019-02-27 23:24 +0000
Re: freshly installed SW not found Jasen Betts <jasen@xnet.co.nz> - 2019-02-28 04:46 +0000
Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-02-28 10:59 +0000
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-28 18:51 +0100
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-28 18:24 +0000
Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-03-01 13:49 +0000
Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-01 18:22 +0100
Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-03-01 19:06 +0000
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-03-03 01:08 +0100
Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-03-03 00:14 +0000
Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-03-03 13:00 +0000
Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-03-03 12:46 +0000
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-28 18:15 +0000
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-27 16:38 +0000
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 20:51 +0100
Re: freshly installed SW not found Jasen Betts <jasen@xnet.co.nz> - 2019-02-28 04:39 +0000
Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-02-27 14:30 +0100
Re: freshly installed SW not found "J.O. Aho" <user@example.net> - 2019-02-27 07:05 +0100
Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-02-27 14:29 +0100
Re: freshly installed SW not found Paul <nospam@needed.invalid> - 2019-02-27 08:50 -0500
Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-02-28 00:05 +0100
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 16:56 +0100
Re: freshly installed SW not found Chris <ithinkiam@gmail.com> - 2019-03-01 14:58 +0000
Re: freshly installed SW not found Chris <ithinkiam@gmail.com> - 2019-03-01 15:04 +0000
Re: freshly installed SW not found "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-03-01 10:50 -0500
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-03-01 18:40 +0000
Re: freshly installed SW not found "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-03-01 14:40 -0500
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-03-01 23:38 +0000
Re: freshly installed SW not found "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-03-01 19:09 -0500
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-03-02 03:47 +0000
Re: freshly installed SW not found "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-03-01 22:55 -0500
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-03-02 07:08 +0000
Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-02 11:27 +0100
Re: freshly installed SW not found Paul <nospam@needed.invalid> - 2019-03-02 04:10 -0500
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-03-02 16:29 +0000
Re: freshly installed SW not found Paul <nospam@needed.invalid> - 2019-03-02 11:54 -0500
Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-02 11:24 +0100
Re: freshly installed SW not found Jasen Betts <jasen@xnet.co.nz> - 2019-03-02 00:28 +0000
Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-02 11:31 +0100
Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-03-02 11:31 +0000
Re: freshly installed SW not found Melzzzzz <Melzzzzz@zzzzz.com> - 2019-03-02 11:34 +0000
Re: freshly installed SW not found Aragorn <thorongil@telenet.be> - 2019-03-02 11:48 +0000
Re: freshly installed SW not found Melzzzzz <Melzzzzz@zzzzz.com> - 2019-03-02 12:15 +0000
Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-02 23:15 +0100
Re: freshly installed SW not found Chris Elvidge <chris@mshome.net> - 2019-03-03 00:33 +0000
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-03-03 01:38 +0100
Compressed filesystems [Was: Re: freshly installed SW not found] "Carlos E.R." <robin_listas@es.invalid> - 2019-03-03 13:11 +0100
Re: Compressed filesystems [Was: Re: freshly installed SW not found] Soviet_Mario <SovietMario@CCCP.MIR> - 2019-03-03 19:49 +0100
Re: Compressed filesystems [Was: Re: freshly installed SW not found] "Carlos E.R." <robin_listas@es.invalid> - 2019-03-03 21:17 +0100
Re: freshly installed SW not found Dan Purgert <dan@djph.net> - 2019-03-03 12:49 +0000
Re: freshly installed SW not found Melzzzzz <Melzzzzz@zzzzz.com> - 2019-03-03 12:35 +0000
Re: freshly installed SW not found "Carlos E.R." <robin_listas@es.invalid> - 2019-03-02 11:22 +0100
Re: freshly installed SW not found Wildman <best_lay@yahoo.com> - 2019-02-26 13:31 -0600
Re: freshly installed SW not found Wildman <best_lay@yahoo.com> - 2019-02-26 15:42 -0600
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-26 21:51 +0000
Re: freshly installed SW not found "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2019-02-26 14:36 -0500
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-26 21:58 +0100
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-26 21:49 +0000
Re: freshly installed SW not found Soviet_Mario <SovietMario@CCCP.MIR> - 2019-02-27 12:54 +0100
Re: freshly installed SW not found William Unruh <unruh@invalid.ca> - 2019-02-26 19:51 +0000
Page 3 of 4 — ← Prev page 1 2 [3] 4 Next page →
| From | Soviet_Mario <SovietMario@CCCP.MIR> |
|---|---|
| Date | 2019-02-27 16:56 +0100 |
| Message-ID | <q56bvu$25f$1@dont-email.me> |
| In reply to | #54241 |
On 27/02/19 14:29, Carlos E.R. wrote: > On 26/02/2019 21.51, Soviet_Mario wrote: >> On 26/02/19 20:28, J.O. Aho wrote: > > >> yes, tnx. I located them. >> >> I'm using it but there is a trick to take into account if one (like me) >> wants to make a full backup : the /MEDIA (lowercase) folder must be >> excluded. >> As I was unsure i used both the --exclude-path and --exclude-dir >> >> Otherwise, and i slammed the face on it, an infinite recursion takes >> place. It was very vierd experience to see this sort of self inclusion >> and reinclusion. >> >> Anyway (using gzip) the programs is darn thunderbolt FAST. >> >> By now (one third of progress) It has grinded (or grounded ?) 150 GB of >> compressed data size, since 20:12 to 21:48. >> >> I'd really like some front end ... but well, it works and it does what I >> needed. I notice the "compressed contentS of each folder is named >> dbackup0.tar.gz > > There is a problem with tar.gz archives: a single byte error in the > backup media and the entire archive is lost. :( I'm really unlucky. I was on the brink of choosing bz2 and then gzip :\ > -- 1) Resistere, resistere, resistere. 2) Se tutti pagano le tasse, le tasse le pagano tutti Soviet_Mario - (aka Gatto_Vizzato)
[toc] | [prev] | [next] | [standalone]
| From | Chris <ithinkiam@gmail.com> |
|---|---|
| Date | 2019-03-01 14:58 +0000 |
| Message-ID | <q5bhbd$q5p$1@dont-email.me> |
| In reply to | #54241 |
On 27/02/2019 13:29, Carlos E.R. wrote: > On 26/02/2019 21.51, Soviet_Mario wrote: >> On 26/02/19 20:28, J.O. Aho wrote: > > >> yes, tnx. I located them. >> >> I'm using it but there is a trick to take into account if one (like me) >> wants to make a full backup : the /MEDIA (lowercase) folder must be >> excluded. >> As I was unsure i used both the --exclude-path and --exclude-dir >> >> Otherwise, and i slammed the face on it, an infinite recursion takes >> place. It was very vierd experience to see this sort of self inclusion >> and reinclusion. >> >> Anyway (using gzip) the programs is darn thunderbolt FAST. >> >> By now (one third of progress) It has grinded (or grounded ?) 150 GB of >> compressed data size, since 20:12 to 21:48. >> >> I'd really like some front end ... but well, it works and it does what I >> needed. I notice the "compressed contentS of each folder is named >> dbackup0.tar.gz > > There is a problem with tar.gz archives: a single byte error in the > backup media and the entire archive is lost. Although true, it's a rare risk these days. The simple solution, however, is to gzip first and then tar. That way any errors only affect one file at a time.
[toc] | [prev] | [next] | [standalone]
| From | Chris <ithinkiam@gmail.com> |
|---|---|
| Date | 2019-03-01 15:04 +0000 |
| Message-ID | <q5bhll$vgu$1@dont-email.me> |
| In reply to | #54227 |
On 26/02/2019 20:51, Soviet_Mario wrote: > I'd really like some front end ... but well, it works and it does what I > needed. I notice the "compressed contentS of each folder is named > dbackup0.tar.gz > > I'm hoping the sw would have some hidden feature as to make > incremental/differential backup, generating updates like > dbackup1.tar.gz, dbackup2.tar.gz etc on different runs. > > It'd be too good to be true perhaps :) > Try LuckyBackup. You may be pleasantly surprised :) http://luckybackup.sourceforge.net/manual.html
[toc] | [prev] | [next] | [standalone]
| From | "David W. Hodgins" <dwhodgins@nomail.afraid.org> |
|---|---|
| Date | 2019-03-01 10:50 -0500 |
| Message-ID | <op.zxzcy6dla3w0dxdave@hodgins.homeip.net> |
| In reply to | #54270 |
On Fri, 01 Mar 2019 10:04:07 -0500, Chris <ithinkiam@gmail.com> wrote: > Try LuckyBackup. You may be pleasantly surprised :) > http://luckybackup.sourceforge.net/manual.html Very nice! Pleasantly surprised to find it's available in my distribution (Mageia). Two things it doesn't have builtin - compression - hardlinking files in the snapshots that have not changed Both of these can be handled in post task scripts, which it makes it easy to add. A simple loop over the destination tree could be used to compress the files. The program freedups can be used to hardlink identical files in the destination tree. Really nice to see it doesn't try to dumb down rsync. While it makes it easy to use with sensible defaults, it still allows full control. Just fyi, in my current backup scripts I use rsync commands such as ... time nice -n 19 ionice -n 7 rsync -auvxSHXAP --specials --sparse --delete --exclude="lost+found" /home/dave/ /aback/home/dave/ Determining what options takes research. :-) I'll definitely be recommending this program for people who prefer a gui. I used to recommend various programs that were good for setting up backups on optical discs, but have stopped using them as hard drive capacity and prices have come down. Also found too many times optical disks give i/o errors. Thanks for the suggestion! Regards, Dave Hodgins -- Change dwhodgins@nomail.afraid.org to davidwhodgins@teksavvy.com for email replies.
[toc] | [prev] | [next] | [standalone]
| From | William Unruh <unruh@invalid.ca> |
|---|---|
| Date | 2019-03-01 18:40 +0000 |
| Message-ID | <q5bua9$du9$1@dont-email.me> |
| In reply to | #54271 |
On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: > On Fri, 01 Mar 2019 10:04:07 -0500, Chris <ithinkiam@gmail.com> wrote: > >> Try LuckyBackup. You may be pleasantly surprised :) >> http://luckybackup.sourceforge.net/manual.html > > Very nice! Pleasantly surprised to find it's available in my distribution > (Mageia). > > Two things it doesn't have builtin > - compression > - hardlinking files in the snapshots that have not changed > > Both of these can be handled in post task scripts, which it makes it > easy to add. A simple loop over the destination tree could be used to Unfortunately no. If you compress, then it becomes impossible to test if the next day's file is the same as the previous day's. without uncompressing the previous day's file. They are not the same. And if you uncompress, then you have the problem that rsync's default test for "the same" -- date and size-- no longer works either, making it very slow. > compress the files. The program freedups can be used to hardlink > identical files in the destination tree. > > Really nice to see it doesn't try to dumb down rsync. While it makes it > easy to use with sensible defaults, it still allows full control. > > Just fyi, in my current backup scripts I use rsync commands such as ... > time nice -n 19 ionice -n 7 rsync -auvxSHXAP --specials --sparse --delete --exclude="lost+found" /home/dave/ /aback/home/dave/ > > Determining what options takes research. :-) > > I'll definitely be recommending this program for people who prefer a gui. > > I used to recommend various programs that were good for setting up backups > on optical discs, but have stopped using them as hard drive capacity and > prices have come down. Also found too many times optical disks give i/o > errors. > > Thanks for the suggestion! > > Regards, Dave Hodgins >
[toc] | [prev] | [next] | [standalone]
| From | "David W. Hodgins" <dwhodgins@nomail.afraid.org> |
|---|---|
| Date | 2019-03-01 14:40 -0500 |
| Message-ID | <op.zxznlbyza3w0dxdave@hodgins.homeip.net> |
| In reply to | #54273 |
On Fri, 01 Mar 2019 13:40:10 -0500, William Unruh <unruh@invalid.ca> wrote: > On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >> Two things it doesn't have builtin >> - compression >> - hardlinking files in the snapshots that have not changed >> >> Both of these can be handled in post task scripts, which it makes it >> easy to add. A simple loop over the destination tree could be used to > > Unfortunately no. If you compress, then it becomes impossible to test if > the next day's file is the same as the previous day's. without > uncompressing the previous day's file. They are not the same. > And if you uncompress, then you have the problem that rsync's default > test for "the same" -- date and size-- no longer works either, making it > very slow. What I was thinking about is to run luckybackup creating a new snapshot, which is a full copy of the files/directories selected for backing up. No compression, no hard linking of duplicates. Then compress each file in the newly created snapshot. Then use freedups to remove one copy of the compressed copies of the file, when it's exactly the same as the previously created compressed copy of the file. Time consuming, still requires the full space for the initial creation of each snapshot, but reduces the total space needed when multiple snapshots are wanted. Compressing each individual file allows keeping only one copy of files that do not change between snapshots, at the expense of compression being more efficient when the files are put into a tar archive prior to compression. It really depends on what percentage of the files tend to change between snapshots, and how well they compress, as to whether or not it's worth doing the extra steps. It also depends on whether multiple snapshots will be kept in one file system, or, for example, archived to optical disc after creation. Regards, Dave Hodgins -- Change dwhodgins@nomail.afraid.org to davidwhodgins@teksavvy.com for email replies.
[toc] | [prev] | [next] | [standalone]
| From | William Unruh <unruh@invalid.ca> |
|---|---|
| Date | 2019-03-01 23:38 +0000 |
| Message-ID | <q5cfp2$rfr$1@dont-email.me> |
| In reply to | #54275 |
On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: > On Fri, 01 Mar 2019 13:40:10 -0500, William Unruh <unruh@invalid.ca> wrote: > >> On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >>> Two things it doesn't have builtin >>> - compression >>> - hardlinking files in the snapshots that have not changed >>> >>> Both of these can be handled in post task scripts, which it makes it >>> easy to add. A simple loop over the destination tree could be used to >> >> Unfortunately no. If you compress, then it becomes impossible to test if >> the next day's file is the same as the previous day's. without >> uncompressing the previous day's file. They are not the same. >> And if you uncompress, then you have the problem that rsync's default >> test for "the same" -- date and size-- no longer works either, making it >> very slow. > > What I was thinking about is to run luckybackup creating a new snapshot, > which is a full copy of the files/directories selected for backing up. > No compression, no hard linking of duplicates. > > Then compress each file in the newly created snapshot. > > Then use freedups to remove one copy of the compressed copies of the > file, when it's exactly the same as the previously created compressed > copy of the file. > > Time consuming, still requires the full space for the initial creation > of each snapshot, but reduces the total space needed when multiple > snapshots are wanted. Compressing each individual file allows keeping > only one copy of files that do not change between snapshots, at the > expense of compression being more efficient when the files are put into > a tar archive prior to compression. This is really slow. You would have to use MD5 hash to decide if the two compressed copies are the same, which is slow (rsync by default uses the date and time of the file to decide if they are the same.) If you have thousands of hard links ( which Soviet implied he had) that will take forever. > > It really depends on what percentage of the files tend to change between > snapshots, and how well they compress, as to whether or not it's worth > doing the extra steps. But you have to compress the new file, compare it, using md5 to the old file to decide if they are the same. If you have a large system, you are entering the territory where the saving takes longer than the time between backups. Memory is cheap. Just buy more disks and do not compress them. You are going to regret compressing them. > > It also depends on whether multiple snapshots will be kept in one file > system, or, for example, archived to optical disc after creation. In which case you cannot hard link them. > > Regards, Dave Hodgins >
[toc] | [prev] | [next] | [standalone]
| From | "David W. Hodgins" <dwhodgins@nomail.afraid.org> |
|---|---|
| Date | 2019-03-01 19:09 -0500 |
| Message-ID | <op.zxzz16nca3w0dxdave@hodgins.homeip.net> |
| In reply to | #54276 |
On Fri, 01 Mar 2019 18:38:10 -0500, William Unruh <unruh@invalid.ca> wrote: > On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >> Then use freedups to remove one copy of the compressed copies of the >> file, when it's exactly the same as the previously created compressed >> copy of the file. > This is really slow. You would have to use MD5 hash to decide if the two > compressed copies are the same, which is slow (rsync by default uses the > date and time of the file to decide if they are the same.) > If you have thousands of hard links ( which Soviet implied he had) that > will take forever. The first time it's run, yes. It stores the md5 hash values in a file in the /root directory. After that it only needs to calculate them for new or changed files. >> It really depends on what percentage of the files tend to change between >> snapshots, and how well they compress, as to whether or not it's worth >> doing the extra steps. > But you have to compress the new file, compare it, using md5 to the old > file to decide if they are the same. If you have a large system, you are > entering the territory where the saving takes longer than the time > between backups. For it to take longer than the time between backups, it would have to be a very slow system with a very large disk. I doubt most people would be running a full backup every day. I normally backup my home directory and other data files daily, but the whole installation once a week. I only backup my main, highly customized day to day install. I have a dozen or so other installs that I use for testing. Those I can easily recreate, if needed. > Memory is cheap. Just buy more disks and do not compress them. You are > going to regret compressing them. Agreed, but if you are limited for any reason, and don't mind the time it takes, then some people may prefer the reduced space usage. >> It also depends on whether multiple snapshots will be kept in one file >> system, or, for example, archived to optical disc after creation. > In which case you cannot hard link them. Correct, which is the point of including that statement. The decisions of how to handle backups are very subjective. My point is simply that there are options available, which people are free to choose to use, if they suits their needs. Regards, Dave Hodgins -- Change dwhodgins@nomail.afraid.org to davidwhodgins@teksavvy.com for email replies.
[toc] | [prev] | [next] | [standalone]
| From | William Unruh <unruh@invalid.ca> |
|---|---|
| Date | 2019-03-02 03:47 +0000 |
| Message-ID | <q5cud4$s7b$1@dont-email.me> |
| In reply to | #54277 |
On 2019-03-02, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: > On Fri, 01 Mar 2019 18:38:10 -0500, William Unruh <unruh@invalid.ca> wrote: > >> On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >>> Then use freedups to remove one copy of the compressed copies of the >>> file, when it's exactly the same as the previously created compressed >>> copy of the file. > >> This is really slow. You would have to use MD5 hash to decide if the two >> compressed copies are the same, which is slow (rsync by default uses the >> date and time of the file to decide if they are the same.) >> If you have thousands of hard links ( which Soviet implied he had) that >> will take forever. > > The first time it's run, yes. It stores the md5 hash values in a file in > the /root directory. After that it only needs to calculate them for new > or changed files. But the only way you know if the file is changed is to run the md5hash. That is not fast. Youhave to run the hash on every file every time. > >>> It really depends on what percentage of the files tend to change between >>> snapshots, and how well they compress, as to whether or not it's worth >>> doing the extra steps. > >> But you have to compress the new file, compare it, using md5 to the old >> file to decide if they are the same. If you have a large system, you are >> entering the territory where the saving takes longer than the time >> between backups. > > For it to take longer than the time between backups, it would have to be a > very slow system with a very large disk. I doubt most people would be > running a full backup every day. I normally backup my home directory and > other data files daily, but the whole installation once a week. I only > backup my main, highly customized day to day install. I have a dozen or so > other installs that I use for testing. Those I can easily recreate, if > needed. I "backup" everything using rsnapshot. Must stuff never gets transfered. The files are the same, based on their date and size as the "hash". Certainly not cryptographically secure, but then my worry is not that someone will sneak in and change one of my files. > >> Memory is cheap. Just buy more disks and do not compress them. You are >> going to regret compressing them. > > Agreed, but if you are limited for any reason, and don't mind the time > it takes, then some people may prefer the reduced space usage. > >>> It also depends on whether multiple snapshots will be kept in one file >>> system, or, for example, archived to optical disc after creation. > >> In which case you cannot hard link them. > > Correct, which is the point of including that statement. The decisions > of how to handle backups are very subjective. My point is simply that > there are options available, which people are free to choose to use, if > they suits their needs. Of course, but they should be aware of the costs beforehand. > > Regards, Dave Hodgins >
[toc] | [prev] | [next] | [standalone]
| From | "David W. Hodgins" <dwhodgins@nomail.afraid.org> |
|---|---|
| Date | 2019-03-01 22:55 -0500 |
| Message-ID | <op.zx0ajfcja3w0dxdave@hodgins.homeip.net> |
| In reply to | #54279 |
On Fri, 01 Mar 2019 22:47:49 -0500, William Unruh <unruh@invalid.ca> wrote: > On 2019-03-02, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >> The first time it's run, yes. It stores the md5 hash values in a file in >> the /root directory. After that it only needs to calculate them for new >> or changed files. > But the only way you know if the file is changed is to run the md5hash. > That is not fast. Youhave to run the hash on every file every time. It has options for how to do the comparison. One option is to only check the filename, date, and time haven't changed. Another option is to do as you've said, and recalculate the hash for every file. It's the user's choice. >> Correct, which is the point of including that statement. The decisions >> of how to handle backups are very subjective. My point is simply that >> there are options available, which people are free to choose to use, if >> they suits their needs. > Of course, but they should be aware of the costs beforehand. On that I fully agree. Regards, Dave Hodgins -- Change dwhodgins@nomail.afraid.org to davidwhodgins@teksavvy.com for email replies.
[toc] | [prev] | [next] | [standalone]
| From | William Unruh <unruh@invalid.ca> |
|---|---|
| Date | 2019-03-02 07:08 +0000 |
| Message-ID | <q5da66$8vf$1@dont-email.me> |
| In reply to | #54280 |
On 2019-03-02, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: > On Fri, 01 Mar 2019 22:47:49 -0500, William Unruh <unruh@invalid.ca> wrote: > >> On 2019-03-02, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >>> The first time it's run, yes. It stores the md5 hash values in a file in >>> the /root directory. After that it only needs to calculate them for new >>> or changed files. > >> But the only way you know if the file is changed is to run the md5hash. >> That is not fast. Youhave to run the hash on every file every time. > > It has options for how to do the comparison. One option is to only > check the filename, date, and time haven't changed. Another option > is to do as you've said, and recalculate the hash for every file. > It's the user's choice. Agreed. But if you have compressed the file, then the date/time has changed. Now you could of course set the date/time to the date/time of the previous file, but that leaves just the length of the compressed file as the comparison, and that is really very weak. There is a good chance that the file has changed, but the compressed length has not changed, and you will not save a change that should be saved. Ie, it becomes important that you do the hash as a comparison if you do not want to loose changes, and that takes a long time for a large file. A backup which does not backup some changes is pretty ropy. Or you could do a traversal of the filesystem again, and set the time/date of the compressed file the same as the time/date of the orginal. But you still have to compress the file before you know if it is the same length as the previous one-- again a long process. And this now has to be done for every file on the system. > > >>> Correct, which is the point of including that statement. The decisions >>> of how to handle backups are very subjective. My point is simply that >>> there are options available, which people are free to choose to use, if >>> they suits their needs. > >> Of course, but they should be aware of the costs beforehand. > > On that I fully agree. > > Regards, Dave Hodgins >
[toc] | [prev] | [next] | [standalone]
| From | "Carlos E.R." <robin_listas@es.invalid> |
|---|---|
| Date | 2019-03-02 11:27 +0100 |
| Message-ID | <hd2qkf-fvg.ln1@Telcontar.valinor> |
| In reply to | #54281 |
On 02/03/2019 08.08, William Unruh wrote: > On 2019-03-02, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >> On Fri, 01 Mar 2019 22:47:49 -0500, William Unruh <unruh@invalid.ca> wrote: >> >>> On 2019-03-02, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >>>> The first time it's run, yes. It stores the md5 hash values in a file in >>>> the /root directory. After that it only needs to calculate them for new >>>> or changed files. >> >>> But the only way you know if the file is changed is to run the md5hash. >>> That is not fast. Youhave to run the hash on every file every time. >> >> It has options for how to do the comparison. One option is to only >> check the filename, date, and time haven't changed. Another option >> is to do as you've said, and recalculate the hash for every file. >> It's the user's choice. > > Agreed. But if you have compressed the file, then the date/time has > changed. Now you could of course set the date/time to the date/time of > the previous file, but that leaves just the length of the compressed > file as the comparison, and that is really very weak. There is a good > chance that the file has changed, but the compressed length has not > changed, and you will not save a change that should be saved. Not if the program saves a catalogue. -- Cheers, Carlos.
[toc] | [prev] | [next] | [standalone]
| From | Paul <nospam@needed.invalid> |
|---|---|
| Date | 2019-03-02 04:10 -0500 |
| Message-ID | <q5dh9n$em7$1@dont-email.me> |
| In reply to | #54279 |
William Unruh wrote:
> On 2019-03-02, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote:
>> On Fri, 01 Mar 2019 18:38:10 -0500, William Unruh <unruh@invalid.ca> wrote:
>>
>>> On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote:
>>>> Then use freedups to remove one copy of the compressed copies of the
>>>> file, when it's exactly the same as the previously created compressed
>>>> copy of the file.
>>> This is really slow. You would have to use MD5 hash to decide if the two
>>> compressed copies are the same, which is slow (rsync by default uses the
>>> date and time of the file to decide if they are the same.)
>>> If you have thousands of hard links ( which Soviet implied he had) that
>>> will take forever.
>> The first time it's run, yes. It stores the md5 hash values in a file in
>> the /root directory. After that it only needs to calculate them for new
>> or changed files.
>
> But the only way you know if the file is changed is to run the md5hash.
> That is not fast. Youhave to run the hash on every file every time.
This in fact, loads the system read cache. If the MD5sum is
changed from previously, now when you want to copy the file,
the "source" file is already present in memory and ready to be
copied somewhere. The "I/O effort" isn't wasted.
MD5 is decently fast in my book. For HDD systems, it could well
be the read-rate of the HDD that limits performance. (A quick
bench of md5deep64 here yields about 450MB/sec. Almost a match
for a SATAIII SSD. Someone with a 5GHz turbo processor could
easily beat my bench (there is one processor that will turbo
on two cores at 5GHz, to make this happen).)
Paul
[toc] | [prev] | [next] | [standalone]
| From | William Unruh <unruh@invalid.ca> |
|---|---|
| Date | 2019-03-02 16:29 +0000 |
| Message-ID | <q5eb10$114$1@dont-email.me> |
| In reply to | #54282 |
On 2019-03-02, Paul <nospam@needed.invalid> wrote: > William Unruh wrote: >> On 2019-03-02, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >>> On Fri, 01 Mar 2019 18:38:10 -0500, William Unruh <unruh@invalid.ca> wrote: >>> >>>> On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >>>>> Then use freedups to remove one copy of the compressed copies of the >>>>> file, when it's exactly the same as the previously created compressed >>>>> copy of the file. >>>> This is really slow. You would have to use MD5 hash to decide if the two >>>> compressed copies are the same, which is slow (rsync by default uses the >>>> date and time of the file to decide if they are the same.) >>>> If you have thousands of hard links ( which Soviet implied he had) that >>>> will take forever. >>> The first time it's run, yes. It stores the md5 hash values in a file in >>> the /root directory. After that it only needs to calculate them for new >>> or changed files. >> >> But the only way you know if the file is changed is to run the md5hash. >> That is not fast. Youhave to run the hash on every file every time. > > This in fact, loads the system read cache. If the MD5sum is > changed from previously, now when you want to copy the file, > the "source" file is already present in memory and ready to be > copied somewhere. The "I/O effort" isn't wasted. It it is small it may be in memory. But MD5 is a serial hash. You start at the beginning and go, byte by byte to the end. There is no need to save the file and certainly md5sum does not. > > MD5 is decently fast in my book. For HDD systems, it could well > be the read-rate of the HDD that limits performance. (A quick > bench of md5deep64 here yields about 450MB/sec. Almost a match > for a SATAIII SSD. Someone with a 5GHz turbo processor could > easily beat my bench (there is one processor that will turbo > on two cores at 5GHz, to make this happen).) Parallelism does nothing, AFAIK for MD5. You start at the beginning and go to the end, and you cannot do anything with any piece of data that helps until you get to it, as I understand. > > Paul
[toc] | [prev] | [next] | [standalone]
| From | Paul <nospam@needed.invalid> |
|---|---|
| Date | 2019-03-02 11:54 -0500 |
| Message-ID | <q5ecfl$c20$1@dont-email.me> |
| In reply to | #54292 |
William Unruh wrote:
>
> Parallelism does nothing, AFAIK for MD5. You start at the beginning and
> go to the end, and you cannot do anything with any piece of data that
> helps until you get to it, as I understand.
True.
How "hashdeep" handles this, is to have parallelism
available at the file level. Something like...
hashdeep -C MD5 -j 12 /home/wally
Each thread of execution works on a file. Some threads
finish before others, and such threads then start work
on the next file.
This is only a "productive" option on an SSD, as an
SSD has zero seek time and doesn't mind having the
"head" thrown all over the place by all that random
seeking.
If you use that expression on a regular HDD, the "thrashing"
that results is counter-productive. Switching to -j 1
may help in such cases (serialization).
When operated in that way, the order of output from the
program (with -j 12 threads) is never twice in the same order.
Depending on the intended application, this may or may
not matter (I fix this by "sorting" the output file).
On an SSD, that option really flies... You could have 12
threads checking the hash on files and determining which
file(s) need backup. Twelve threads running at 450MB/sec
each could handle all the read rate available on an
NVMe flash drive.
Paul
[toc] | [prev] | [next] | [standalone]
| From | "Carlos E.R." <robin_listas@es.invalid> |
|---|---|
| Date | 2019-03-02 11:24 +0100 |
| Message-ID | <l82qkf-fvg.ln1@Telcontar.valinor> |
| In reply to | #54279 |
On 02/03/2019 04.47, William Unruh wrote: > On 2019-03-02, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >> On Fri, 01 Mar 2019 18:38:10 -0500, William Unruh <unruh@invalid.ca> wrote: >> >>> On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >>>> Then use freedups to remove one copy of the compressed copies of the >>>> file, when it's exactly the same as the previously created compressed >>>> copy of the file. >> >>> This is really slow. You would have to use MD5 hash to decide if the two >>> compressed copies are the same, which is slow (rsync by default uses the >>> date and time of the file to decide if they are the same.) >>> If you have thousands of hard links ( which Soviet implied he had) that >>> will take forever. >> >> The first time it's run, yes. It stores the md5 hash values in a file in >> the /root directory. After that it only needs to calculate them for new >> or changed files. > > But the only way you know if the file is changed is to run the md5hash. > That is not fast. Youhave to run the hash on every file every time. Only on those with a more recent datestamp. -- Cheers, Carlos.
[toc] | [prev] | [next] | [standalone]
| From | Jasen Betts <jasen@xnet.co.nz> |
|---|---|
| Date | 2019-03-02 00:28 +0000 |
| Message-ID | <q5cioa$b8r$1@gonzo.revmaps.no-ip.org> |
| In reply to | #54276 |
On 2019-03-01, William Unruh <unruh@invalid.ca> wrote: > On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >> On Fri, 01 Mar 2019 13:40:10 -0500, William Unruh <unruh@invalid.ca> wrote: >> >>> On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >> snapshots are wanted. Compressing each individual file allows keeping >> only one copy of files that do not change between snapshots, at the >> expense of compression being more efficient when the files are put into >> a tar archive prior to compression. > > This is really slow. You would have to use MD5 hash to decide if the two > compressed copies are the same, which is slow (rsync by default uses the > date and time of the file to decide if they are the same.) > If you have thousands of hard links ( which Soviet implied he had) that > will take forever. perhaps use filesystem compression? -- When I tried casting out nines I made a hash of it.
[toc] | [prev] | [next] | [standalone]
| From | "Carlos E.R." <robin_listas@es.invalid> |
|---|---|
| Date | 2019-03-02 11:31 +0100 |
| Message-ID | <gk2qkf-fvg.ln1@Telcontar.valinor> |
| In reply to | #54278 |
On 02/03/2019 01.28, Jasen Betts wrote: > On 2019-03-01, William Unruh <unruh@invalid.ca> wrote: >> On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: >>> On Fri, 01 Mar 2019 13:40:10 -0500, William Unruh <unruh@invalid.ca> wrote: >>> >>>> On 2019-03-01, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote: > >>> snapshots are wanted. Compressing each individual file allows keeping >>> only one copy of files that do not change between snapshots, at the >>> expense of compression being more efficient when the files are put into >>> a tar archive prior to compression. >> >> This is really slow. You would have to use MD5 hash to decide if the two >> compressed copies are the same, which is slow (rsync by default uses the >> date and time of the file to decide if they are the same.) >> If you have thousands of hard links ( which Soviet implied he had) that >> will take forever. > > perhaps use filesystem compression? Do you know such a filesystem? ext4 has the field, but it was never developed btrfs I think has it (man 5 btrfs). I don't know if it is reliable. Some other? -- Cheers, Carlos.
[toc] | [prev] | [next] | [standalone]
| From | Aragorn <thorongil@telenet.be> |
|---|---|
| Date | 2019-03-02 11:31 +0000 |
| Message-ID | <q5dpj2$1ro$1@dont-email.me> |
| In reply to | #54286 |
On Sat, 02 Mar 2019 11:31:12 +0100, Carlos E.R. scribbled: > On 02/03/2019 01.28, Jasen Betts wrote: > >> perhaps use filesystem compression? > > Do you know such a filesystem? > > ext4 has the field, but it was never developed > > btrfs I think has it (man 5 btrfs). I don't know if it is reliable. > > Some other? reiser4 also has it, but few (if any) distributions support reiser4. It's also not supported in the vanilla kernel. -- With respect, = Aragorn =
[toc] | [prev] | [next] | [standalone]
| From | Melzzzzz <Melzzzzz@zzzzz.com> |
|---|---|
| Date | 2019-03-02 11:34 +0000 |
| Message-ID | <zTteE.13144$pP5.8636@fx01.am4> |
| In reply to | #54287 |
On 2019-03-02, Aragorn <thorongil@telenet.be> wrote: > On Sat, 02 Mar 2019 11:31:12 +0100, Carlos E.R. scribbled: > >> On 02/03/2019 01.28, Jasen Betts wrote: >> >>> perhaps use filesystem compression? >> >> Do you know such a filesystem? >> >> ext4 has the field, but it was never developed >> >> btrfs I think has it (man 5 btrfs). I don't know if it is reliable. >> >> Some other? > > reiser4 also has it, but few (if any) distributions support reiser4. > It's also not supported in the vanilla kernel. > zfs -- press any key to continue or any other to quit...
[toc] | [prev] | [next] | [standalone]
Page 3 of 4 — ← Prev page 1 2 [3] 4 Next page →
Back to top | Article view | alt.os.linux
csiph-web