Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #221827 > unrolled thread

(none)

Back to article view | Back to linux.debian.user

Contents

  (none) Default User <hunguponcontent@gmail.com> - 2020-04-22 23:00 +0200
    Re: David <bouncingcats@gmail.com> - 2020-04-23 12:00 +0200
    Re: "Alexander V. Makartsev" <avbetev@gmail.com> - 2020-04-23 16:30 +0200
    Re: David Christensen <dpchrist@holgerdanske.com> - 2020-04-24 00:10 +0200
      Re: Default User <hunguponcontent@gmail.com> - 2020-04-24 03:00 +0200
        Re: David Christensen <dpchrist@holgerdanske.com> - 2020-04-24 11:00 +0200
        Backup ideas Andy Smith <andy@strugglers.net> - 2020-04-24 17:40 +0200
          Re: Backup ideas Celejar <celejar@gmail.com> - 2020-04-24 18:30 +0200
            Re: Backup ideas Default User <hunguponcontent@gmail.com> - 2020-04-28 17:50 +0200
              Re: Backup ideas Andrei POPESCU <andreimpopescu@gmail.com> - 2020-04-28 18:20 +0200
                Re: Backup ideas Default User <hunguponcontent@gmail.com> - 2020-04-28 20:30 +0200
                  Re: Backup ideas Default User <hunguponcontent@gmail.com> - 2020-04-28 21:20 +0200
                    Re: Backup ideas David Christensen <dpchrist@holgerdanske.com> - 2020-04-28 22:30 +0200
                    Re: Backup ideas Bob Weber <bobrweber@gmail.com> - 2020-04-28 22:50 +0200
              Re: Backup ideas Bob Weber <bobrweber@gmail.com> - 2020-04-28 20:20 +0200
              Re: Backup ideas Andy Smith <andy@strugglers.net> - 2020-04-28 22:40 +0200
                Re: Backup ideas Celejar <celejar@gmail.com> - 2020-04-29 06:40 +0200
              Re: Backup ideas Celejar <celejar@gmail.com> - 2020-04-29 06:30 +0200
              Re: Backup ideas Alessandro Baggi <alessandro.baggi@gmail.com> - 2020-05-01 12:40 +0200
                Re: Backup ideas Default User <hunguponcontent@gmail.com> - 2020-05-04 03:30 +0200

#221827 — (none)

Hey,

A recent thread got me to thinking. So I checked my primary (only
installed) ssd:

sudo smartctl --test=long /dev/sda

which promised to run the tests, but returned to a command prompt,
with no further messages.
So after the promised test end time I did:

sudo smartctl --all /dev/sda

which gave this:

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.5.0-2-amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     SandForce Driven SSDs
Device Model:     KINGSTON SV300S37A120G
Serial Number:    [redacted]
LU WWN Device Id: 5 0026b7 7490a7ce8
Firmware Version: 541ABBF0
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS, ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Apr 21 23:45:15 2020 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (    0) seconds.
Offline data collection
capabilities: (0x79) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: (   1) minutes.
Extended self-test routine
recommended polling time: (  48) minutes.
Conveyance self-test routine
recommended polling time: (   2) minutes.
SCT capabilities:       (0x0025) SCT Status supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   095   095   050    Old_age
Always       -       0/5466601
  5 Retired_Block_Count     0x0033   100   100   003    Pre-fail
Always       -       0
  9 Power_On_Hours_and_Msec 0x0032   091   091   000    Old_age
Always       -       8197h+53m+29.020s
 12 Power_Cycle_Count       0x0032   097   097   000    Old_age
Always       -       3549
171 Program_Fail_Count      0x000a   100   100   000    Old_age
Always       -       0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age
Always       -       0
174 Unexpect_Power_Loss_Ct  0x0030   000   000   000    Old_age
Offline      -       175
177 Wear_Range_Delta        0x0000   000   000   000    Old_age
Offline      -       1
181 Program_Fail_Count      0x000a   100   100   000    Old_age
Always       -       0
182 Erase_Fail_Count        0x0032   100   100   000    Old_age
Always       -       0
187 Reported_Uncorrect      0x0012   100   100   000    Old_age
Always       -       0
189 Airflow_Temperature_Cel 0x0000   035   070   000    Old_age
Offline      -       35 (Min/Max 21/70)
194 Temperature_Celsius     0x0022   035   070   000    Old_age
Always       -       35 (Min/Max 21/70)
195 ECC_Uncorr_Error_Count  0x001c   120   120   000    Old_age
Offline      -       0/5466601
196 Reallocated_Event_Count 0x0033   100   100   003    Pre-fail
Always       -       0
201 Unc_Soft_Read_Err_Rate  0x001c   120   120   000    Old_age
Offline      -       0/5466601
204 Soft_ECC_Correct_Rate   0x001c   120   120   000    Old_age
Offline      -       0/5466601
230 Life_Curve_Status       0x0013   100   100   000    Pre-fail
Always       -       100
231 SSD_Life_Left           0x0013   100   100   010    Pre-fail
Always       -       0
233 SandForce_Internal      0x0032   000   000   000    Old_age
Always       -       2748
234 SandForce_Internal      0x0032   000   000   000    Old_age
Always       -       3157
241 Lifetime_Writes_GiB     0x0032   000   000   000    Old_age
Always       -       3157
242 Lifetime_Reads_GiB      0x0032   000   000   000    Old_age
Always       -       2340

SMART Error Log not supported

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      8197         -
# 2  Short offline       Completed without error       00%      8196         -
# 3  Short offline       Completed without error       00%      7883         -
# 4  Short offline       Completed without error       00%      7883         -
# 5  Short offline       Completed without error       00%      7883         -
# 6  Extended offline    Completed without error       00%      7003         -
# 7  Extended offline    Completed without error       00%      7002         -
# 8  Short offline       Completed without error       00%      6999         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

--------------------------------------------------------------------------------------------------

I don't really know how to interpret this, but it looks . . .  bad,
with all those "old_age" and "Pre-fail" notations.

Well, I have an old, blank ssd  as a spare, so I checked it too:

--------------------------------------------------------------------------------------------------

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.5.0-2-amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     SandForce Driven SSDs
Device Model:     SanDisk SDSSDA240G
Serial Number:    [redacted]
LU WWN Device Id: 5 001b44 4a4c2fb82
Firmware Version: Z32070RL
User Capacity:    240,057,409,536 bytes [240 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 22 12:10:40 2020 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test
routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  120) seconds.
Offline data collection
capabilities:              (0x11) SMART execute Offline immediate.
                    No Auto Offline data collection support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    No Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (  10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  5 Retired_Block_Count     0x0032   100   100   000    Old_age
Always       -       0
  9 Power_On_Hours_and_Msec 0x0032   000   100   000    Old_age
Always       -       2498h+00m+00.000s
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age
Always       -       1079
165 Unknown_Attribute       0x0032   100   100   000    Old_age
Always       -       476755132619
166 Unknown_Attribute       0x0032   100   100   000    Old_age
Always       -       4
167 Unknown_Attribute       0x0032   100   100   000    Old_age
Always       -       0
168 Unknown_Attribute       0x0032   100   100   000    Old_age
Always       -       5
169 Unknown_Attribute       0x0032   100   100   000    Old_age
Always       -       0
170 Reserve_Block_Count     0x0032   100   100   000    Old_age
Always       -       0
171 Program_Fail_Count      0x0032   100   100   000    Old_age
Always       -       0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age
Always       -       0
173 Unknown_SandForce_Attr  0x0032   100   100   000    Old_age
Always       -       4
174 Unexpect_Power_Loss_Ct  0x0032   100   100   000    Old_age
Always       -       36
187 Reported_Uncorrect      0x0032   100   100   000    Old_age
Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age
Always       -       0
194 Temperature_Celsius     0x0022   050   018   000    Old_age
Always       -       50 (Min/Max 0/82)
199 SATA_CRC_Error_Count    0x0032   100   100   000    Old_age
Always       -       0
230 Life_Curve_Status       0x0032   100   100   000    Old_age
Always       -       287767068739
232 Available_Reservd_Space 0x0033   100   100   004    Pre-fail
Always       -       100
233 SandForce_Internal      0x0032   100   100   000    Old_age
Always       -       865
234 SandForce_Internal      0x0032   100   100   000    Old_age
Always       -       1887
241 Lifetime_Writes_GiB     0x0030   253   253   000    Old_age
Offline      -       1306
242 Lifetime_Reads_GiB      0x0030   253   253   000    Old_age
Offline      -       1543
244 Unknown_Attribute       0x0032   000   100   000    Old_age
Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      2498         -
# 2  Short offline       Completed without error       00%      2498         -
# 3  Extended offline    Completed without error       00%      2225         -
# 4  Extended offline    Self-test routine in progress 90%      2225         -
# 5  Short offline       Completed without error       00%      2225         -
# 6  Short offline       Self-test routine in progress 90%      2225         -

Selective Self-tests/Logging not supported

---------------------------------------------------------------------------------------

Doesn't seem good, either.

Since, I recall, that ssd drives do tend to fail quickly and without
notice, is it time to get a new drive and reinstall from scratch?
If so, recommendations?

I think I want to continue to use an ssd, because even though it can
fail without warning, this would be in a laptop, and I want to avoid
shock damage to a mechanical drive at all costs.

I an currently running Debian Unstable, on a 5-year-old laptop (Dell
Inspiron 3000 series, 8Gb ram, Intel i3 processor, traditional BIOS
booting). It has space for only one, 2.5-inch drive.

lsblk:
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 111.8G  0 disk
├─sda1   8:1    0    28G  0 part /
├─sda2   8:2    0     1K  0 part
├─sda5   8:5    0   7.9G  0 part [SWAP]
└─sda6   8:6    0    76G  0 part /home
sr0     11:0    1  1024M  0 rom

The 120Gb ssd is "adequate"; root partition is 38% full, and home
partition is 45% full.
Swap partition is rarely if ever used; not sure i it is really necessary.

User case is conventional, nothing taxing. No video editing, gaming, etc.

And yes, I do back up, home partition only, using rsync to an external
usb drive:

sudo rsync -avvzHAXPish --delete /home/default
/media/default/USBHD005/Backup_of_home_directory_of_Dell_Debian_dimwit

I suppose I could [learn to] do a full system backup and restore to a
new drive, if I had to.

And if I do need to try to copy my current install to a new drive
(instead of a fresh install), what is the EASIEST way to do that?

[toc] | [next] | [standalone]

#221852 — Re:

On Thu, 23 Apr 2020 at 06:53, Default User <hunguponcontent@gmail.com> wrote:

> A recent thread got me to thinking. So I checked my primary (only
> installed) ssd:

> sudo smartctl --test=long /dev/sda

> I don't really know how to interpret this,

Ok, if you dont know how to interpret this, then I'd suggest that
the first step might be to use your favourite internet
search engine to learn about the meaning of your test results.

That's what I did to find this:
https://media.kingston.com/support/downloads/MKP_306_SMART_attribute.pdf

The meaning of the SMART attribute values is defined by the manufacturer,
so it's best to check with the manufacturer of your drive.

> but it looks . . .  bad,

I spent a few seconds glancing at the results you posted of your first drive,
I didn't notice anything concerning.

> with all those "old_age" and "Pre-fail" notations.

Those are in the column headed "TYPE", they are informative about the
nature of the failure criterion, they are not test results.

[toc] | [prev] | [next] | [standalone]

#221862 — Re:

[Multipart message — attachments visible in raw view] — view raw

On 23.04.2020 01:52, Default User wrote:
> Hey,
>
> A recent thread got me to thinking. So I checked my primary (only
> installed) ssd:
>
> === START OF INFORMATION SECTION ===
> Model Family:     SandForce Driven SSDs
> Device Model:     KINGSTON SV300S37A120G
This drive is based on MLC (2-bit) NAND type chips. According to
datasheet [1] your drive is rated for 64TB TBW.

> 241 Lifetime_Writes_GiB     0x0032   000   000   000    Old_age
> Always       -       3157
> 242 Lifetime_Reads_GiB      0x0032   000   000   000    Old_age
> Always       -       2340
And you has written to it only 3.1TB. This means your drive should
continue to serve you well for a long time, since your workload is low.
The rest of SMART attributes only support my opinion.

> === START OF INFORMATION SECTION ===
> Model Family:     SandForce Driven SSDs
> Device Model:     SanDisk SDSSDA240G
This drive doesn't have any public information about type of NAND memory
it based on, and manufacturer doesn't even rated it for TBW.
If manufacturer this stingy to disclose technical specifications about
his products, usually means that drive is based on cheap TLC (3-bit) or
even QLC (4-bit) NAND type chips.
These kind of chips do not have as much lifespan as MLC, so first drive
is still better, except for its volume 128GB vs 240GB.

> User case is conventional, nothing taxing. No video editing, gaming, etc.
>
> And yes, I do back up, home partition only, using rsync to an external
> usb drive:
>
> sudo rsync -avvzHAXPish --delete /home/default
> /media/default/USBHD005/Backup_of_home_directory_of_Dell_Debian_dimwit
>
> I suppose I could [learn to] do a full system backup and restore to a
> new drive, if I had to.
>
> And if I do need to try to copy my current install to a new drive
> (instead of a fresh install), what is the EASIEST way to do that?
>
If you already doing regular backups you'll be fine. There are no signs
of first drive to be failing anytime soon.
As for full system backup, I usually use CloneZilla project [2]. They
have LiveCD which could be run from USB flash drive to perform efficient
offline copy of entire disk image to another disk or over the network.
If online full system backup is required then I hope somebody else will
chime in with recommendation.


[1] https://www.kingston.com/datasheets/sv300s3_us.pdf
[2] https://clonezilla.org/

-- 
With kindest regards, Alexander.

⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀ 

[toc] | [prev] | [next] | [standalone]

#221883 — Re:

On 2020-04-22 13:52, Default User wrote:
> Hey,
> 
> A recent thread got me to thinking. So I checked my primary (only
> installed) ssd:
> 
> sudo smartctl --test=long /dev/sda
> 
> which promised to run the tests, but returned to a command prompt,
> with no further messages.
> So after the promised test end time I did:
> 
> sudo smartctl --all /dev/sda
> 
> which gave this:

> SMART overall-health self-assessment test result: PASSED

I pay attention to this item.


> General SMART Values:
> Offline data collection status:  (0x02) Offline data collection activity
> was completed without error.

> Self-test execution status:      (   0) The previous self-test routine completed
> without error or no self-test has ever
> been run.

I pay attention to these items.


> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
> UPDATED  WHEN_FAILED RAW_VALUE
>    1 Raw_Read_Error_Rate     0x0032   095   095   050    Old_age
> Always       -       0/5466601
>    5 Retired_Block_Count     0x0033   100   100   003    Pre-fail
> Always       -       0
>    9 Power_On_Hours_and_Msec 0x0032   091   091   000    Old_age
> Always       -       8197h+53m+29.020s

I ignore the "old age" and "pre-fail" notations.


> SMART Self-test log structure revision number 1
> Num  Test_Description    Status                  Remaining
> LifeTime(hours)  LBA_of_first_error
> # 1  Extended offline    Completed without error       00%      8197         -
> # 2  Short offline       Completed without error       00%      8196         -
> # 3  Short offline       Completed without error       00%      7883         -
> # 4  Short offline       Completed without error       00%      7883         -
> # 5  Short offline       Completed without error       00%      7883         -
> # 6  Extended offline    Completed without error       00%      7003         -
> # 7  Extended offline    Completed without error       00%      7002         -
> # 8  Short offline       Completed without error       00%      6999         -

I pay attention to these items.


> Since, I recall, that ssd drives do tend to fail quickly and without
> notice, is it time to get a new drive and reinstall from scratch?

I have seen lots of HDD failures and a few USB flash drive failures, but 
no SSD failures (yet).  The USB flash drive failures do tend towards 
all-or-nothing or all-or-very-slow, sometimes accompanied with the smell 
of roasting electronics.


> If so, recommendations?

I prefer Intel and Samsung, but any major brand should do.


> I think I want to continue to use an ssd, because even though it can
> fail without warning, this would be in a laptop, and I want to avoid
> shock damage to a mechanical drive at all costs.

The only HDD's I buy are large capacity 3.5" SATA enterprise HDD's for 
storage and backups.


> I an currently running Debian Unstable, on a 5-year-old laptop (Dell
> Inspiron 3000 series, 8Gb ram, Intel i3 processor, traditional BIOS
> booting). It has space for only one, 2.5-inch drive.
> 
> lsblk:
> NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
> sda      8:0    0 111.8G  0 disk
> ├─sda1   8:1    0    28G  0 part /
> ├─sda2   8:2    0     1K  0 part
> ├─sda5   8:5    0   7.9G  0 part [SWAP]
> └─sda6   8:6    0    76G  0 part /home
> sr0     11:0    1  1024M  0 rom
> 
> The 120Gb ssd is "adequate"; root partition is 38% full, and home
> partition is 45% full.
> Swap partition is rarely if ever used; not sure i it is really necessary.

A ~10.6 GB root is in the ballpark for a Debian graphical workstation.


I put my bulk data on a Samba server.


I tried running without swap -- those systems crashed.  Now I do 1-2 GB.


> User case is conventional, nothing taxing. No video editing, gaming, etc.
> 
> And yes, I do back up, home partition only, using rsync to an external
> usb drive:
> 
> sudo rsync -avvzHAXPish --delete /home/default
> /media/default/USBHD005/Backup_of_home_directory_of_Dell_Debian_dimwit

You should get two more -- keep one on-site, keep one near-site, keep 
one off-site, and rotate them periodically (weekly, bi-monthly, etc.).


> I suppose I could [learn to] do a full system backup and restore to a
> new drive, if I had to.

I have never done an old-school dump(8)/ restore(8).  I keep my system 
images small enough to fit on "16 GB" devices.  I take and restore 
images with dd(1), which is available in the Debian installer rescue 
shell (but preferably with a USB live drive and a Perl script I wrote). 
As other readers have mentioned, Clonezilla is an option.


> And if I do need to try to copy my current install to a new drive
> (instead of a fresh install), what is the EASIEST way to do that?

Building up a blank disk into a working system drive requires many 
tedious steps -- you have to move the partition table(s), bootloader, 
all partitions, and the contents of those partitions (e.g. swap, 
filesystems, whatever).  Things get harder if you add encryption, LVM, 
or non-standard filesystems like ZFS.  The installer takes care of many 
of those details for you.  I believe Clonezilla also does so.


A raw binary copy with dd(1) bypasses all of those messy details except 
for relocating the backup GPT partition table if the device size has 
changed.  That is one reason why I still use MBR.


David

[toc] | [prev] | [next] | [standalone]

#221889 — Re:

[Multipart message — attachments visible in raw view] — view raw

On Thu, Apr 23, 2020, 18:09 David Christensen <dpchrist@holgerdanske.com>
wrote:

> On 2020-04-22 13:52, Default User wrote:
> > Hey,
> >
> > A recent thread got me to thinking. So I checked my primary (only
> > installed) ssd:
> >
> > sudo smartctl --test=long /dev/sda
> >
> > which promised to run the tests, but returned to a command prompt,
> > with no further messages.
> > So after the promised test end time I did:
> >
> > sudo smartctl --all /dev/sda
> >
> > which gave this:
>
> > SMART overall-health self-assessment test result: PASSED
>
> I pay attention to this item.
>
>
> > General SMART Values:
> > Offline data collection status:  (0x02) Offline data collection activity
> > was completed without error.
>
> > Self-test execution status:      (   0) The previous self-test routine
> completed
> > without error or no self-test has ever
> > been run.
>
> I pay attention to these items.
>
>
> > SMART Attributes Data Structure revision number: 10
> > Vendor Specific SMART Attributes with Thresholds:
> > ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
> > UPDATED  WHEN_FAILED RAW_VALUE
> >    1 Raw_Read_Error_Rate     0x0032   095   095   050    Old_age
> > Always       -       0/5466601
> >    5 Retired_Block_Count     0x0033   100   100   003    Pre-fail
> > Always       -       0
> >    9 Power_On_Hours_and_Msec 0x0032   091   091   000    Old_age
> > Always       -       8197h+53m+29.020s
>
> I ignore the "old age" and "pre-fail" notations.
>
>
> > SMART Self-test log structure revision number 1
> > Num  Test_Description    Status                  Remaining
> > LifeTime(hours)  LBA_of_first_error
> > # 1  Extended offline    Completed without error       00%      8197
>      -
> > # 2  Short offline       Completed without error       00%      8196
>      -
> > # 3  Short offline       Completed without error       00%      7883
>      -
> > # 4  Short offline       Completed without error       00%      7883
>      -
> > # 5  Short offline       Completed without error       00%      7883
>      -
> > # 6  Extended offline    Completed without error       00%      7003
>      -
> > # 7  Extended offline    Completed without error       00%      7002
>      -
> > # 8  Short offline       Completed without error       00%      6999
>      -
>
> I pay attention to these items.
>
>
> > Since, I recall, that ssd drives do tend to fail quickly and without
> > notice, is it time to get a new drive and reinstall from scratch?
>
> I have seen lots of HDD failures and a few USB flash drive failures, but
> no SSD failures (yet).  The USB flash drive failures do tend towards
> all-or-nothing or all-or-very-slow, sometimes accompanied with the smell
> of roasting electronics.
>
>
> > If so, recommendations?
>
> I prefer Intel and Samsung, but any major brand should do.
>
>
> > I think I want to continue to use an ssd, because even though it can
> > fail without warning, this would be in a laptop, and I want to avoid
> > shock damage to a mechanical drive at all costs.
>
> The only HDD's I buy are large capacity 3.5" SATA enterprise HDD's for
> storage and backups.
>
>
> > I an currently running Debian Unstable, on a 5-year-old laptop (Dell
> > Inspiron 3000 series, 8Gb ram, Intel i3 processor, traditional BIOS
> > booting). It has space for only one, 2.5-inch drive.
> >
> > lsblk:
> > NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
> > sda      8:0    0 111.8G  0 disk
> > ├─sda1   8:1    0    28G  0 part /
> > ├─sda2   8:2    0     1K  0 part
> > ├─sda5   8:5    0   7.9G  0 part [SWAP]
> > └─sda6   8:6    0    76G  0 part /home
> > sr0     11:0    1  1024M  0 rom
> >
> > The 120Gb ssd is "adequate"; root partition is 38% full, and home
> > partition is 45% full.
> > Swap partition is rarely if ever used; not sure i it is really necessary.
>
> A ~10.6 GB root is in the ballpark for a Debian graphical workstation.
>
>
> I put my bulk data on a Samba server.
>
>
> I tried running without swap -- those systems crashed.  Now I do 1-2 GB.
>
>
> > User case is conventional, nothing taxing. No video editing, gaming, etc.
> >
> > And yes, I do back up, home partition only, using rsync to an external
> > usb drive:
> >
> > sudo rsync -avvzHAXPish --delete /home/default
> > /media/default/USBHD005/Backup_of_home_directory_of_Dell_Debian_dimwit
>
> You should get two more -- keep one on-site, keep one near-site, keep
> one off-site, and rotate them periodically (weekly, bi-monthly, etc.).
>
>
> > I suppose I could [learn to] do a full system backup and restore to a
> > new drive, if I had to.
>
> I have never done an old-school dump(8)/ restore(8).  I keep my system
> images small enough to fit on "16 GB" devices.  I take and restore
> images with dd(1), which is available in the Debian installer rescue
> shell (but preferably with a USB live drive and a Perl script I wrote).
> As other readers have mentioned, Clonezilla is an option.
>
>
> > And if I do need to try to copy my current install to a new drive
> > (instead of a fresh install), what is the EASIEST way to do that?
>
> Building up a blank disk into a working system drive requires many
> tedious steps -- you have to move the partition table(s), bootloader,
> all partitions, and the contents of those partitions (e.g. swap,
> filesystems, whatever).  Things get harder if you add encryption, LVM,
> or non-standard filesystems like ZFS.  The installer takes care of many
> of those details for you.  I believe Clonezilla also does so.
>
>
> A raw binary copy with dd(1) bypasses all of those messy details except
> for relocating the backup GPT partition table if the device size has
> changed.  That is one reason why I still use MBR.
>
>
> David
>



Hi.

Thanks to all for the information.

I may just stay with what I have, for now. If it ain't broke, don't fix it?

BTW, aside from neglecting to put a subject line in the original post, l
should have mentioned that both the root and home partitions are regular
ext4 (no lvm or encryption).

And by BIOS booting, I meant using a traditional mbr, not gpt. And I use
grub2, not lilo or anything else.

May I ask one additional question?
I back up my home partition only, because:

1)  I figure that I always reinstall the system from scratch, and either
re-attach the home partition, or add back the data from it bit by bit if
re-attaching the home partition doesn't work (well).

2)  Years ago, I tried to back up my whole system using rsync.  That
created a bizarre, seemingly recursive monstrosity which ended only when
the entire drive was filled.  I never forgot that.

Note: years later, I did read something on the Arch wiki about excluding
something when backing up a system using rsync, to avoid that recursive
loop problem. But I have no desire to stick my hand back into the hornet's
nest again.

So, in backing up my home partition, is use:

sudo rsync -avvzHAXPish --delete /home/default
/media/default/USBHD005/Backup_of_home_directory_of_Dell_Debian_dimwit

to backup my home directory to an external (local) usb hdd.

I just sort of came up with that process (and the command syntax) on my
own.

Good? Bad? Indifferent?
Comments welcome. Especially on the rsync command syntax.

[toc] | [prev] | [next] | [standalone]

#221899 — Re:

On 2020-04-23 17:52, Default User wrote:

> Thanks to all for the information.
> 
> I may just stay with what I have, for now. If it ain't broke, don't fix it?

I would not make any changes because of the smartclt reports.


> BTW, aside from neglecting to put a subject line in the original post, l
> should have mentioned that both the root and home partitions are regular
> ext4 (no lvm or encryption).
> 
> And by BIOS booting, I meant using a traditional mbr, not gpt. And I use
> grub2, not lilo or anything else.
> 
> May I ask one additional question?
> I back up my home partition only, because:
> 
> 1)  I figure that I always reinstall the system from scratch, and either
> re-attach the home partition, or add back the data from it bit by bit if
> re-attaching the home partition doesn't work (well).
> 
> 2)  Years ago, I tried to back up my whole system using rsync.  That
> created a bizarre, seemingly recursive monstrosity which ended only when
> the entire drive was filled.  I never forgot that.

Without seeing the exact command issued, it would be hard to figure out. 
  But, I do use the -x (--one-file-system) option when invoking rsync(1) 
to prevent trying to backup network mounts (see below).


> Note: years later, I did read something on the Arch wiki about excluding
> something when backing up a system using rsync, to avoid that recursive
> loop problem. But I have no desire to stick my hand back into the hornet's
> nest again.
> 
> So, in backing up my home partition, is use:
> 
> sudo rsync -avvzHAXPish --delete /home/default
> /media/default/USBHD005/Backup_of_home_directory_of_Dell_Debian_dimwit
> 
> to backup my home directory to an external (local) usb hdd.
> 
> I just sort of came up with that process (and the command syntax) on my
> own.
> 
> Good? Bad? Indifferent?
> Comments welcome. Especially on the rsync command syntax.

I keep my system images small enough to fit on 16 GB devices -- USB 
flash drives, SSD's, HDD's, whatever.  My backup strategy is many-fold:

1.  I wrote a script that calls dd(1) to copy the raw contents to a 
tar(1) gzip(1) file.  I run it from a USB flash drive with Debian 
installed on it.  The script also generates MD5 and SHA256 checksum 
files, and can reverse the process.  I call this "imaging".

2.  Whenever I create or modify a system configuration file (e.g. 
/etc/*), I check-in the file to CVS.  I call this "configuration 
management".

3.  I wrote a script that invokes rsync(1) to copy my system drives to 
backup directories on my SOHO server.  I call this "backup".  Here is 
the rsync(1) command issued by the script to backup the system drive in 
my Debian daily driver.  I have replaced the spaces with newlines to 
avoid e-mail wrapping:

+ /usr/local/bin/rsync
  -a
  --delete
  -e /usr/bin/ssh
  -x
  root@tinkywinky.tracy.holgerdanske.com:/
  /var/local/backup/tinkywinky.tracy.holgerdanske.com//

Your rsync(1) command has a lot more switches.  I am not going to try to 
figure them out.  I prefer to KISS and use the bare minimum.

4.  The server data is on ZFS.  I wrote another script to snapshot and 
replicate the ZFS filesystems to other pools -- one on-site, one 
near-site, and one off-site.  The on-site pool is a mirror is inside 
another server, but the near-site and off-site drives are in mobile dock 
drawers; I rotate them bi-monthly.  I call this "redundant backups".


David

[toc] | [prev] | [next] | [standalone]

#221902 — Backup ideas

Hello,

On Thu, Apr 23, 2020 at 08:52:05PM -0400, Default User wrote:

[vast mounts of quoted text snipped - please don't quote too much!]
> So, in backing up my home partition, is use:
> 
> sudo rsync -avvzHAXPish --delete /home/default
> /media/default/USBHD005/Backup_of_home_directory_of_Dell_Debian_dimwit
> 
> to backup my home directory to an external (local) usb hdd.

Note that unless you regularly store this HDD somewhere else, it
isn't a great backup.

Your home could, for example, burn down, or if you are a victim of
theft they will probably take any attached drives also.

A remote or off-site backup would be better.

> Good? Bad? Indifferent?
> Comments welcome. Especially on the rsync command syntax.

An issue with your backup methodology is that you only get one copy
of your data: whatever was most recent at the time that the command
ran.

The most common form of data loss is human error: you delete
something or alter something, and need to get back what you had
before.

With your current method you are relying on spotting that issue
before your next backup runs, because once your next backup runs it
will overwrite the existing backup.

Your scheme could be improved by keeping more history.

It doesn't look like you are encrypting your backups. Maybe you are
and this just wasn't evident in the information that you supplied.
If you aren't doing so, it would be a good idea to encrypt your
source files and your backups so that if someone steals the external
HDD then the data will be useless to them.

There are many different backup solutions, with different strengths
and weaknesses. I would suggest first looking into a way of keeping
some amount of history, secondly to work out how to get an off-site
or remote backup, and thirdly to look at how you can keep encrypted
backups.

Something like restic backup is a very good all around solution.

You could consider backing up to a cloud like Amazon S3. Using the
Infrequent Access or Glacier storage classes you can store 100s of
gigabytes of data for just a few US cents per day. There are many
different backup solutions that support clouds like these.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#221905 — Re: Backup ideas

On Fri, 24 Apr 2020 15:34:24 +0000
Andy Smith <andy@strugglers.net> wrote:

...

> Something like restic backup is a very good all around solution.

OP: also look at borg.

> You could consider backing up to a cloud like Amazon S3. Using the
> Infrequent Access or Glacier storage classes you can store 100s of
> gigabytes of data for just a few US cents per day. There are many

It should just be noted that anyone considering Glacier should also
price in its considerable retrieval costs.

Celejar

[toc] | [prev] | [next] | [standalone]

#222038 — Re: Backup ideas

On Fri, Apr 24, 2020 at 12:28 PM Celejar <celejar@gmail.com> wrote:
>
> On Fri, 24 Apr 2020 15:34:24 +0000
> Andy Smith <andy@strugglers.net> wrote:
>
> ...
>
> > Something like restic backup is a very good all around solution.
>
> OP: also look at borg.
>
> > You could consider backing up to a cloud like Amazon S3. Using the
> > Infrequent Access or Glacier storage classes you can store 100s of
> > gigabytes of data for just a few US cents per day. There are many
>
> It should just be noted that anyone considering Glacier should also
> price in its considerable retrieval costs.
>
> Celejar


Hi Andy and Celejar!

Sorry it took several days to get back to this.  I have had some
personal things going on, and I have also been spending some quality
time with Clonezilla and rsync.

1 - First, regarding:
"[vast mounts of quoted text snipped - please don't quote too much!]"

I didn't realize some trimming might be needed.  I guess I just
figured you can't scroll through an email for information that has
been trimmed.  And on some lists, like OpenBSD and Arch, people will
actually yell at you for trimming or not providing enough information.

2 - I had originally posted in another thread about checking an ssd
and replacing it if needed, imaging the old system and applying the
image(s) to the new drive.  I did test Clonezilla, and it did image
and restore all partitions to the same ssd.  I have not tested
restoring to a different drive, though.

Clonezilla (using partclone by default) does work.  But the user
interface is really ugly, and how to use it is somewhat non-intuitive
(not good if you are using it for the first  time in an emergency).

3 - Now on to the subject of backing up a system. I am using rsync as
my primary backup tool.  Celejar, thank you for suggesting borg.  I
did install it and look at it briefly.  But for several reasons, I'm
not sure I want to use that right now.

It is of course one more thing to learn, use, and maintain.

I get the sense that it stores data as a mass of chunks, rather than
as individual files, so if something goes wrong it could really go
wrong?  It is newer, and perhaps less proven than some older
alternatives.

Also, the borg website seems to suggest that they will be developing
aggressively, and breaking compatibility over time.  Where else have I
heard that lately?  (*cough* SystemD *cough* . . .)

And, I always worry that today's free/libre can be tomorrow's
proprietary/unfree.

But I still might try it out later.

Andy, you mentioned restic, which I am not familiar with.  Similar
considerations would seem to apply to that also.  But I might also try
that out later.

4 - So I am still using rsync.  I back up a single system to a local
external usb drive.  That drive is then backed up to another local
external usb drive. So I do have some redundancy.  I could go even
further, backing up to a third drive, and using the --checksum option.
But that might be excessive, and I do have a life to live.

Andy, you suggested having off-site backups.  You are correct that
local-only backups do not protect against things like fire or theft.

But off-site backups require a place for them to be, the ongoing
effort of getting them there (and back, in a timely manner, if
needed), and some cost involved.  And the storage place is also
subject to "disasters" , or just going out of business overnight.

On-line backups have the same problems, in addition to the fact that
once your data is "out there", even if it is encrypted, it can be
"cracked" by governments, corporations, or individuals, at their
leisure.

I'm not sure my threat model really justifies off-site backups at this point.

5 - You also mentioned encrypted backups.  Of course, you have to to
trust the encryption methodology.  And it should not be such a hassle
that you don't actually use it.

And . . .   don't lose your key!  That could really ruin your day.  Or life.

I have actually considered (and still am) encrypting the backups,
perhaps using full disk encryption.  Maybe LUKS or something similar.
But . . .  I do remember using disk encryption on the home partition
of my system, years ago.  I worked fine for a short time, and then it
just didn't. Not a lost password, it just stopped working.  Lost data
forever.  Not fun.  Never forgot that.

6 - Finally, using rsync I actually am doing two separate backups:

date; sudo rsync -avvzHAXPSish --delete --stats
--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/home/default/*"}
/ /media/default/USBHD005/Backup_of_Dell_Debian_dimwit/root_partition

[This backs up the filesystem EXCEPT for the home directory.]

And:

date; sudo rsync -avvzHAXPSish --delete --stats
--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"}
/home/default /media/default/USBHD005/Backup_of_Dell_Debian_dimwit/home_partition

[This backs up the home directory.]

Note: my home directory is on a separate (extended) partition from my
root directory.

Another note: rsync did NOT like the -x option.  I wanted to use that
to prevent getting into a recursive loop while backing up. Rsync just
refused, complaining with a sparse non-informative error code message.
But without it, it seems to work okay.  Go figure.

I have been using rsync to backup live, from within Debian.  Maybe not
a good idea.  I could instead try using rsync from a live usb, such as
SystemRescueCD, etc.  I'll try that later.  After all, it does seem to
make more sense to back up a dead filesystem from outside it than a
live filesystem from inside it.

Finally, after well over seven hours into rsyncing (with no end in
sight) from the first external usb drive to the second one (both are
HDD), I am beginning to wonder if that is a good idea.  Those first
full backups always take forever . . .

[toc] | [prev] | [next] | [standalone]

#222042 — Re: Backup ideas

[Multipart message — attachments visible in raw view] — view raw

On Ma, 28 apr 20, 11:43:22, Default User wrote:
> > Andy Smith <andy@strugglers.net> wrote:
> 
> 1 - First, regarding:
> > [vast mounts of quoted text snipped - please don't quote too much!]
> 
> I didn't realize some trimming might be needed.  I guess I just
> figured you can't scroll through an email for information that has
> been trimmed.  And on some lists, like OpenBSD and Arch, people will
> actually yell at you for trimming or not providing enough information.

Ideally every message should be as short as possible while still being 
as much as possible self-contained.

Proper attribution of quotes is also much appreciated.
 
> And, I always worry that today's free/libre can be tomorrow's
> proprietary/unfree.

See "Tentacles of Evil" test. 

https://en.wikipedia.org/wiki/Debian_Free_Software_Guidelines#debian-legal_tests_for_DFSG_compliance

Short version: once a software has been released under a free software 
license not even the author can change it. They can only change the 
license for future versions. Anybody else is free to fork from the last 
free version.

This is not just theoretical, it has already happened in several cases 
and will likely happen again if the software in question has any 
significant user base.
 
> But off-site backups require a place for them to be, the ongoing
> effort of getting them there (and back, in a timely manner, if
> needed), and some cost involved.  And the storage place is also
> subject to "disasters" , or just going out of business overnight.

At the same time with a "local" disaster? Highly unlikely.

> On-line backups have the same problems, in addition to the fact that
> once your data is "out there", even if it is encrypted, it can be
> "cracked" by governments, corporations, or individuals, at their
> leisure.

[citation needed]

With current encryption methods it's probably easier to compromise the 
source computer instead. Or https://xkcd.com/538/.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[toc] | [prev] | [next] | [standalone]

#222052 — Re: Backup ideas

On Tue, Apr 28, 2020 at 12:16 PM Andrei POPESCU
<andreimpopescu@gmail.com> wrote:
>
> On Ma, 28 apr 20, 11:43:22, Default User wrote:
> > > Andy Smith <andy@strugglers.net> wrote:
> >
> > 1 - First, regarding:
> > > [vast mounts of quoted text snipped - please don't quote too much!]
> >
> > I didn't realize some trimming might be needed.  I guess I just
> > figured you can't scroll through an email for information that has
> > been trimmed.  And on some lists, like OpenBSD and Arch, people will
> > actually yell at you for trimming or not providing enough information.
>
> Ideally every message should be as short as possible while still being
> as much as possible self-contained.
>
> Proper attribution of quotes is also much appreciated.
>
> > And, I always worry that today's free/libre can be tomorrow's
> > proprietary/unfree.
>
> See "Tentacles of Evil" test.
>
> https://en.wikipedia.org/wiki/Debian_Free_Software_Guidelines#debian-legal_tests_for_DFSG_compliance
>
> Short version: once a software has been released under a free software
> license not even the author can change it. They can only change the
> license for future versions. Anybody else is free to fork from the last
> free version.
>
> This is not just theoretical, it has already happened in several cases
> and will likely happen again if the software in question has any
> significant user base.
>
> > But off-site backups require a place for them to be, the ongoing
> > effort of getting them there (and back, in a timely manner, if
> > needed), and some cost involved.  And the storage place is also
> > subject to "disasters" , or just going out of business overnight.
>
> At the same time with a "local" disaster? Highly unlikely.
>
> > On-line backups have the same problems, in addition to the fact that
> > once your data is "out there", even if it is encrypted, it can be
> > "cracked" by governments, corporations, or individuals, at their
> > leisure.
>
> [citation needed]
>
> With current encryption methods it's probably easier to compromise the
> source computer instead. Or https://xkcd.com/538/.
>
> Kind regards,
> Andrei
> --
> http://wiki.debian.org/FAQsFromDebianUser



Hi Andrei.

On 2020-04-28 [TU] at 12:16 EDT, Andrei POPESCU
<andreimpopescu@gmail.com> wrote:
>
> Ideally every message should be as short as possible while still being
> as much as possible self-contained.
>
> Proper attribution of quotes is also much appreciated.
>

Okay.  Do you mean like this?

>
> See "Tentacles of Evil" test.
>
> https://en.wikipedia.org/wiki/Debian_Free_Software_Guidelines#debian-legal_tests_for_DFSG_compliance
>

Noted. And read.

>
> > But off-site backups require a place for them to be, the ongoing
> > effort of getting them there (and back, in a timely manner, if
> > needed), and some cost involved.  And the storage place is also
> > subject to "disasters" , or just going out of business overnight.
>
> At the same time with a "local" disaster? Highly unlikely.
>

Well, things can (and do) happen, anywhere at any time.  And things
that are away from me are not under my control.

("Just because I'm paranoid doesn't mean they're not out to get me!")

>
> > On-line backups have the same problems, in addition to the fact that
> > once your data is "out there", even if it is encrypted, it can be
> > "cracked" by governments, corporations, or individuals, at their
> > leisure.
>
> [citation needed]
>

I would have thought that by now, this would be almost universally
accepted and considered virtually self-evident.

("Just because I'm paranoid doesn't mean they're not out to get me!")

>
> With current encryption methods it's probably easier to compromise the
> source computer instead. Or https://xkcd.com/538/.
>

There is truth in that.  Yes, I am familiar with "lead-pipe"
cryptanalysis.  But:
("Just because I'm paranoid doesn't mean they're not out to get me!")
- and these days, they don't even have to get out of their chair to
get information . . .

[toc] | [prev] | [next] | [standalone]

#222055 — Re: Backup ideas

On 2020-04-28 [TU] at 14:18 EDT, Bob Weber <bobrweber@gmail.com> said:

> According to the manual the -x option is:
>
> -x, --one-file-system       don't cross filesystem boundaries
>
> I use that option all the time to keep from backing up my large home > directory when I only
> want the system directories under root.  It even > keeps rsync from copying system directories
> like /proc /dev and
> /sys.

> Before I do a system update/upgrade I run rsnapshot (debian
> package) on the root system directories so I can get the system
> back in case of major failure in some of the updates (I run testing so > I have to be careful).  I do > run this on a live system and on 3 or 4
> occasions I have had to restore from the snapshot successfully
> getting my system back alive.

> Do you have another system you could backup to?  I can get around > 50 mega bytes per
> second transfer over 1Gb Ethernet so you might > try that.

> My main backup is done by backuppc on a dedicated server .  I
> have 4 or 5 systems that get unattended daily backups with versions > going back about a year. > All my systems use a 2 drive raid1  array >
> so I can survive a single disk crash without having to resort to restoring a backup.  Every few
> months I install an extra drive in the backuppc server and have raid sync it to the 2 drives in the > server.  After syncing I pull the drive and put it in my barn for offsite storage.  Since it is a raid1
> full copy you can take that drive and mount it on another system and get the files back if you
> need to (running the raid array in a degraded mode).


Hi, Bob!

I don't recall the specific error code I got; just that it refused to
do a sync using the -x option.  And I can't try it now, as I am now
into the 11th or 12th hour of rsyncing one external usb drive to
another.  At this rate it could take days to complete!  I had no idea.

Unfortunately, I only have one computer.  I am not on a LAN and have
no NAS, etc.  And raid setups are "above my pay grade" anyway.

Question:

When you use rsync, do you ever do it on a live, mounted filesystem
from within said machine/filesystem (that is, using the same machine)?

Or do you do it on a "dormant" unmounted filesystem, either from
another machine or from a "live [usb or .iso] utility distribution or
boot disk from which you have booted the same machine?

Most references to rsync I have seen just seem to accept as a given,
that you are doing it remotely, from across a LAN (or across the
world).

And don't seem to address whether the machine/filesystem they are
rsyncing to/from is "live" (mounted), or can/should be unmounted (like
it would be when imaging a disk with dd or Clonezilla, for example.

[toc] | [prev] | [next] | [standalone]

#222057 — Re: Backup ideas

On 2020-04-28 12:15, Default User wrote:

> I don't recall the specific error code I got; just that it refused to
> do a sync using the -x option.  

I keep a plain text system administration log file for each computer in 
CVS.  I cut and paste console sessions into it.  If a given command 
produces a lot of output, I paste it into a separate file or create the 
file with tee(1).


> And I can't try it now, as I am now
> into the 11th or 12th hour of rsyncing one external usb drive to
> another.  At this rate it could take days to complete!  I had no idea.

Use df(1) to find out how much data is on the source disk.


Install nmon(1) and use it to find out the transfer rate.  If you use 
the Xfce desktop, install the Disk Performance Monitor panel applet.


Divide the data size by the transfer rate to get an estimate of how long 
the transfer will take.  Understand that the transfer rate of HDD's 
slows down as the block number increases (transfer rate of the last 
block is about half that of the first block).


> Unfortunately, I only have one computer. 

Ouch.  You need another computer.  I would suggest a used server with 
ECC memory and a decent number of drive bays.  But, a $20 garage sale 
used tower is better than nothing.


> I am not on a LAN 

Assuming you have a residential broadband Internet connection, many 
gateways include a 4 port Gigabit switch and Wi-Fi.  If you have more 
than one device connected to the gateway/ Internet, you have a LAN.


> and have no NAS, etc. 

NAS devices give you functionality without having to understand the 
details.  If you want to understand the details, building your own file 
server with Debian and Samba is an intermediate-level project.


> And raid setups are "above my pay grade" anyway.

Setting up a mirror with md(4) and mdadm(8) is a beginner/ 
intermediate-level project.  If your one computer is a desktop and it 
has two spare bays, you should consider it.


> Question:
> 
> When you use rsync, do you ever do it on a live, mounted filesystem
> from within said machine/filesystem (that is, using the same machine)?
> 
> Or do you do it on a "dormant" unmounted filesystem, either from
> another machine or from a "live [usb or .iso] utility distribution or
> boot disk from which you have booted the same machine?
> 
> Most references to rsync I have seen just seem to accept as a given,
> that you are doing it remotely, from across a LAN (or across the
> world).
> 
> And don't seem to address whether the machine/filesystem they are
> rsyncing to/from is "live" (mounted), or can/should be unmounted (like
> it would be when imaging a disk with dd or Clonezilla, for example.

rsync(1) is for mounted filesystems.  If you need consistency of files 
and/or directories, the source filesystem should be mounted read-only. 
I use rsync(1) for data and for system configuration files.  I do not 
use rsync(1) to backup, restore, or migrate operating system images.


I use dd(1) to backup, restore, and migrate operating system raw drive 
images.  To make this practical, I keep my system images small enough to 
fit on a "16 GB" device (USB flash drive, SSD, HDD), and I do not use RAID.


It is useful to have a live Debian system on a USB flash drive for doing 
these kinds of operations.


David

[toc] | [prev] | [next] | [standalone]

#222060 — Re: Backup ideas

[Multipart message — attachments visible in raw view] — view raw

On 4/28/20 3:15 PM, Default User wrote:
> On 2020-04-28 [TU] at 14:18 EDT, Bob Weber <bobrweber@gmail.com> said:
>
>> According to the manual the -x option is:
>>
>> -x, --one-file-system       don't cross filesystem boundaries
>>
>> Question:
>>
>> When you use rsync, do you ever do it on a live, mounted filesystem
>> from within said machine/filesystem (that is, using the same machine)?
>>
>> Or do you do it on a "dormant" unmounted filesystem, either from
>> another machine or from a "live [usb or .iso] utility distribution or
>> boot disk from which you have booted the same machine?
>>
>> Most references to rsync I have seen just seem to accept as a given,
>> that you are doing it remotely, from across a LAN (or across the
>> world).
>>
>> And don't seem to address whether the machine/filesystem they are
>> rsyncing to/from is "live" (mounted), or can/should be unmounted (like
>> it would be when imaging a disk with dd or Clonezilla, for example.
>>
Yes that is the way I use rsnapshot (which uses rsync) ... on a live system.  I 
do this before an upgrade.  rsnapshot copies the files to a directory under 
/home.  I have a very big /home filesystem.  The files I am interested in are 
under /usr /etc and the various bin opt and lib directories.  These files will 
just be open for reading but not for writing.  There are log files and mail 
files under /var that may be copied in an open for write state but I can loose 
those files if necessary.  Also, files under /tmp and /run may be lost but those 
directories are usually cleared on reboot.  I do run PostgreSQL but the data 
files are under /home (mounted from another partition)  ... these are the kind 
of files that should be backed up on a "dormant" filesystem or after PostgreSQL 
is shut down.  I also run several virtual machines but their files are also 
under /home.

To restore the rsnapshot backup if I need to I run sysrescuecd. I mount the 
filesystems on /root/src and /root/dst and use rsync with the correct options 
including  --delete to get rid of any extra files that were just upgraded.  I 
usually use  -aHXA --one-file-system --progress for rsync backup options.  That 
way I can be sure the file attributes will be preserved.

A note about borgbackup.  I used it a few years ago but I found that ALL 
attributes were not backed up.  Maybe I didn't use the correct options.   The 
many options can be confusing.  The backup medium should be a journaling 
filesystem according to the docs. rsnapshot just duplicates the file structure 
of the source so you can just get one file back if you want rather than having 
to use borg to get the backed up file.


-- 


*...Bob*

[toc] | [prev] | [next] | [standalone]

#222050 — Re: Backup ideas

[Multipart message — attachments visible in raw view] — view raw

> 6 - Finally, using rsync I actually am doing two separate backups:
>
> date; sudo rsync -avvzHAXPSish --delete --stats
> --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/home/default/*"}
> / /media/default/USBHD005/Backup_of_Dell_Debian_dimwit/root_partition
>
> [This backs up the filesystem EXCEPT for the home directory.]
>
> And:
>
> date; sudo rsync -avvzHAXPSish --delete --stats
> --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"}
> /home/default /media/default/USBHD005/Backup_of_Dell_Debian_dimwit/home_partition
>
> [This backs up the home directory.]
>
> Note: my home directory is on a separate (extended) partition from my
> root directory.
>
> Another note: rsync did NOT like the -x option.  I wanted to use that
> to prevent getting into a recursive loop while backing up. Rsync just
> refused, complaining with a sparse non-informative error code message.
> But without it, it seems to work okay.  Go figure.

According to the manual the -x option is:

-x, --one-file-system       don't cross filesystem boundaries

I use that option all the time to keep from backing up my large home directory 
when I only want the system directories under root.  It even keeps rsync from 
copying system directories like /proc /dev and /sys.

Before I do a system update/upgrade I run rsnapshot (debian package) on the root 
system directories so I can get the system back in case of major failure in some 
of the updates (I run testing so I have to be careful).  I do run this on a live 
system and on 3 or 4 occasions I have had to restore from the snapshot 
successfully getting my system back alive.

Do you have another system you could backup to?  I can get around 50 mega bytes 
per second transfer over 1Gb Ethernet so you might try that.

My main backup is done by backuppc on a dedicated server .  I have 4 or 5 
systems that get unattended daily backups with versions going back about a 
year.  All my systems use a 2 drive raid1  array so I can survive a single disk 
crash without having to resort to restoring a backup.  Every few months I 
install an extra drive in the backuppc server and have raid sync it to the 2 
drives in the server.  After syncing I pull the drive and put it in my barn for 
offsite storage.  Since it is a raid1 full copy you can take that drive and 
mount it on another system and get the files back if you need to (running the 
raid array in a degraded mode).

> I have been using rsync to backup live, from within Debian.  Maybe not
> a good idea.  I could instead try using rsync from a live usb, such as
> SystemRescueCD, etc.  I'll try that later.  After all, it does seem to
> make more sense to back up a dead filesystem from outside it than a
> live filesystem from inside it.
>
> Finally, after well over seven hours into rsyncing (with no end in
> sight) from the first external usb drive to the second one (both are
> HDD), I am beginning to wonder if that is a good idea.  Those first
> full backups always take forever . . .
>

-- 


*...Bob*

[toc] | [prev] | [next] | [standalone]

#222058 — Re: Backup ideas

Hello,

On Tue, Apr 28, 2020 at 11:43:22AM -0400, Default User wrote:
> Andy, you mentioned restic, which I am not familiar with.  Similar
> considerations would seem to apply to that also.  But I might also try
> that out later.

Note that you did not state any of these requirements for only using
tools you are familiar with, so we cannot really be blamed for
suggesting things just because we think they are good. You didn't
really state many requirements at all, only asked for pointers as to
what makes a good backup system, so only generalities were possible.
I did say there are many many different solutions.

> 4 - So I am still using rsync.  I back up a single system to a local
> external usb drive.  That drive is then backed up to another local
> external usb drive. So I do have some redundancy.  I could go even
> further, backing up to a third drive, and using the --checksum option.
> But that might be excessive, and I do have a life to live.

If you wish to continue using rsync then I would suggest looking at
rsnapshot. It uses rsync. It provides for incremental backups. It
stores things as the actual files which makes them easy to
investigate and restore.

When you come to features like deduplication and encryption either
the remote filesystem has to support it, or else the backup tool has
to support it. That is why so many of these solutions store data in
big blocks in their own format, so they can provide those features.

rsnapshot gives you some limited form of deduplication by hard
linking identical versions of each file together. It is far from
perfect but is better than no deduplication.

> Andy, you suggested having off-site backups.  You are correct that
> local-only backups do not protect against things like fire or theft.
> 
> But off-site backups require a place for them to be, the ongoing
> effort of getting them there (and back, in a timely manner, if
> needed), and some cost involved.  And the storage place is also
> subject to "disasters" , or just going out of business overnight.

The thing about cloud storage is that it is generic and it is easy.
Once you come up with a means to backup to an S3-compatible storage,
should your data disappear you can just point it at another
S3-compatible storage. Amazon's S3 was of course the original but
there are many competitors to Amazon now that use the same protocol.

Anyone using cloud ("someone else's computer") services needs to
think about what they will do if/when the cloud ("someone else's
computer") goes away. The usual answer is "switch everything over to
another cloud".

So the concern when using such a service usually isn't whether they
are going to go out of business, but more often about keeping
control of costs. At some point there may be a time where it's
cheaper to do yourself, or scale up the amount of responsibility you
take. For example, if you require many terabytes of storage then it
will be cheaper for you to rent a whole server from someone instead.
That server will almost certainly have a lower reliability than the
cloud storage vendor, but if it's only for backups then maybe that
is not as important to you as the cost saving.

I would not suggest having "the cloud" as one's ONLY backup of
important data. Multiple different clouds may be safe enough, though
the advantage of having a copy of the data local is large.

> On-line backups have the same problems, in addition to the fact that
> once your data is "out there", even if it is encrypted, it can be
> "cracked" by governments, corporations, or individuals, at their
> leisure.

Any organisation that wants to spend that amount of resources
accessing your data has already been in your home and cloned your
storage while you were out/asleep.

> I'm not sure my threat model really justifies off-site backups at this point.

Fair enough. Fire and theft seem pretty common to me, but it's your
call.

> 5 - You also mentioned encrypted backups.  Of course, you have to to
> trust the encryption methodology.  And it should not be such a hassle
> that you don't actually use it.

It is perfectly feasible to have it be entirely automated. I would
not recommend any kind of backup that's not automated!

> Finally, after well over seven hours into rsyncing (with no end in
> sight) from the first external usb drive to the second one (both are
> HDD), I am beginning to wonder if that is a good idea.  Those first
> full backups always take forever . . .

…and USB is a fairly awful way to use storage beyond personal
gadgets (camera, phone, etc)…

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#222069 — Re: Backup ideas

On Tue, 28 Apr 2020 20:37:27 +0000
Andy Smith <andy@strugglers.net> wrote:

> Hello,
> 
> On Tue, Apr 28, 2020 at 11:43:22AM -0400, Default User wrote:

...

> The thing about cloud storage is that it is generic and it is easy.
> Once you come up with a means to backup to an S3-compatible storage,
> should your data disappear you can just point it at another
> S3-compatible storage. Amazon's S3 was of course the original but
> there are many competitors to Amazon now that use the same protocol.

And furthermore, there's the wonderful rclone, which supports literally
dozens of cloud types in a single utility:

https://rclone.org/

Celejar

[toc] | [prev] | [next] | [standalone]

#222068 — Re: Backup ideas

On Tue, 28 Apr 2020 11:43:22 -0400
Default User <hunguponcontent@gmail.com> wrote:

> On Fri, Apr 24, 2020 at 12:28 PM Celejar <celejar@gmail.com> wrote:

...

> 3 - Now on to the subject of backing up a system. I am using rsync as
> my primary backup tool.  Celejar, thank you for suggesting borg.  I
> did install it and look at it briefly.  But for several reasons, I'm
> not sure I want to use that right now.
> 
> It is of course one more thing to learn, use, and maintain.
> 
> I get the sense that it stores data as a mass of chunks, rather than
> as individual files, so if something goes wrong it could really go
> wrong?  It is newer, and perhaps less proven than some older
> alternatives.

All true enough, but note that my impression at least is that it's not
some kind of small, hobby project, but a very sophisticated, highly
polished, widely used piece of software.

Celejar

[toc] | [prev] | [next] | [standalone]

#222166 — Re: Backup ideas

> 3 - Now on to the subject of backing up a system. I am using rsync as
> my primary backup tool.  Celejar, thank you for suggesting borg.  I
> did install it and look at it briefly.  But for several reasons, I'm
> not sure I want to use that right now.
>
> It is of course one more thing to learn, use, and maintain.
>
> I get the sense that it stores data as a mass of chunks, rather than
> as individual files, so if something goes wrong it could really go
> wrong?  It is newer, and perhaps less proven than some older
> alternatives.
Hi, I would spend some words for borgbackup. borgbackup is an amazing 
tool (all of us know that it does).

Don't worry about deduplication. Deduplication is a feature. 
Deduplication currently is taking great place in many envs because it is 
a secure practices if well done and the amount of data is increased in 
last years.
The problem you stated about "if something goes wrong" is a real problem 
today and loosing data is bad. I would say that things could go wrong 
using zfs, vdo, rsync + hardlink, borgbackup, restic and any other 
tool/fs that provides deduplication, all in the same way. Suppose that 
you have a broken deduped chuck, all files pointing to this chunk are 
faulty. This type of problem could happen also with any devices that 
encounter bad blocks, bit rotting and so on. Deduplication is a features 
that permit you to save space. "If something should go bad, It will go 
bad with/without deduplication" (the biggest errors are human errors) 
and you will know that your backup is really working well when you 
try/need restore (this is why a periodical restore test is needed). You 
could not use deduplication but you could lost all data because your 
drive dies. So nothing changed, it is a features. About how borgbackup 
saves data inside the repository I would say that you can encounter this 
problem also using tool like bacula, bareos or with rsync + hardlink. 
Imagine that using bacula you saved 3 backup cycles and having 3 full 
backups of 1TB plus 30 daily incremental for each full backup (monthly 
cycle). If the volume that contains the full backup get faulty, all 
backups based on this full backup are not usefull, so the first backup 
cycle is incosistent and red flagged. To avoid problem with data 
corruption, you should use raid, better hardware, ECC ram to avoid data 
corruption.

Compressed archives have the same problem so you should not compress 
your data.

In the case of borgbackup, you have the ability to perform checks on 
repository and archives inside repository and check if something is broken.
rsync + hardlink is an inefficient way to do deduplication..it works, 
save some space on devices but nothing more.(I'm not saying that rsync 
is not a very good tool).

> Also, the borg website seems to suggest that they will be developing
> aggressively, and breaking compatibility over time.  Where else have I
> heard that lately?  (*cough* SystemD *cough* . . .)
About this. Backup is a practice that permits you to protect your data. 
If you really care your data you should use a stable system and you 
should not use the latest version when released. If your data are 
important and you update too much frequently the software, you will 
encounter at 100% some problems. This is why stable distribution like 
debian, centos, ubuntu and other maintain the same software for many 
years with changing the version and push security or critical fixes. 
Always about borg when a new release is released the maintainer will 
report any incompability with previous version and always if you care 
your data you should read notes before any update. This is a good 
practice and not only for backup.
>
> And, I always worry that today's free/libre can be tomorrow's
> proprietary/unfree.

About this it is not really a problem. If borgbackup will not be open 
source/free, someone probably will fork the latest open source version 
and that's all. [I know that this is off-topic for debian but this is 
only an example, sorry] For example when big blue bought redhat, many 
centos users were worried about the future of centos. If big blue (is 
some case) will drop centos, the community will fork it without any 
problem and again if your preferred software, in this case borgbackup, 
will be non-free you could use it until you deploy another solution with 
another free/open source software and replace it (or you could pay it 
considering as a support donation). I don't think that actually, when 
open-source is a business model, that a software, released as 
open-source, will change its license. Generally they release open source 
software and services like support or other. borgbackup is around since 
2010 and many companies are selling storage space service for borgbackup 
like rsync.net and other (please don't consider this as spam is only an 
example).

> Andy, you suggested having off-site backups.  You are correct that
> local-only backups do not protect against things like fire or theft.
>
> But off-site backups require a place for them to be, the ongoing
> effort of getting them there (and back, in a timely manner, if
> needed), and some cost involved.  And the storage place is also
> subject to "disasters" , or just going out of business overnight.
>
> On-line backups have the same problems, in addition to the fact that
> once your data is "out there", even if it is encrypted, it can be
> "cracked" by governments, corporations, or individuals, at their
> leisure.
>
> I'm not sure my threat model really justifies off-site backups at this point.
About remote off-site backup, again if your data are important and 
critical for you, you should consider this for 3-2-1 method. If you are 
paranoid about your privacy and data, you can avoid public cloud and get 
a dedicated server/vps with the necessary storage and provide by 
yourself remote storage, it has a cost but if you need it you can do it. 
Again if you data are critical off-site backup could save you and 
off-site backup is only another good practices.
> 5 - You also mentioned encrypted backups.  Of course, you have to to
> trust the encryption methodology.  And it should not be such a hassle
> that you don't actually use it.
>
> And . . .   don't lose your key!  That could really ruin your day.  Or life.
>
> I have actually considered (and still am) encrypting the backups,
> perhaps using full disk encryption.  Maybe LUKS or something similar.
> But . . .  I do remember using disk encryption on the home partition
> of my system, years ago.  I worked fine for a short time, and then it
> just didn't. Not a lost password, it just stopped working.  Lost data
> forever.  Not fun.  Never forgot that.

About encryption, this is a feature. With borgbackup you can disable it 
(like compression). I always avoided encrypting my files due to loss 
risk, but if you need an off-site backup this is a good feature. Do you 
trust all encryption methodology of every protocol/software that you 
use? Crypto can be strong today and weak tomorrow..simply adopt new 
methods when something becomes "obsolete" or deprecated.

Another word about borgbackup. Borgbackup works in a similar way to 
rsync but rsync was born as sync tool (and after used as transfer method 
in backup script based on rsync) and borgbackup was born as a backup 
tool. When borg run through ssh to a remote repository, borg is needed 
on the destination host (like rsync) to perform operations. If you don't 
trust push method (where borgbackup and ssh permit some option to save 
your security) you could use sshfs (or similar) and run in pull mode.

My short story: I'm migrating away from bacula to borgbackup using it as 
a central server that runs command on target host using ssh keys. To do 
this you can use ansible or a simple ssh script. Why? bacula is great 
software but too much complicated over time and borgbackup solve me some 
problems (like encryption, deduplication,offsite backup and backup speed 
== less time).

Ah, I'm not an expert about borgbackup, but it seems very useful when 
working with system like ansible to run backup on a great number of 
hosts because it has not the barriers that tools like bacula have due to 
the their client/server model (director/storage).

Suggestion: if you want use it, try it well and see if suits your needs 
and (sorry for this) before replace your current solution makes 
appropriate test. It is not the best backup tool on the world and it is 
not enteprise but hey, it is amazing and do the job very well. Give only 
a try.

[toc] | [prev] | [next] | [standalone]

#222215 — Re: Backup ideas

[Multipart message — attachments visible in raw view] — view raw

Hi to all.

I want to thank you all very much  for taking the time to reply to this
thread.

Your replies have indeed been most thoughtful and informative, and you have
given me much to consider.

The subject of data protection and preservation seems to be not only quite
extensive, but also seems to be something of an art, as well as a science.
It's hard to get right, but easy to get wrong.

So, not wanting to get it wrong, I have rolled up my sleeves, am studying,
and practicing, practicing, practicing.

Please let me apologize for not replying more promptly to your posts. It
isn't out of disinterest, it's that I have been busy doing rather than
talking.

Again, thank you for your interest and assistance.  I really do appreciate
it!

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web