Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #238432 > unrolled thread

Disk for a small server

Back to article view | Back to linux.debian.user

Contents

  Disk for a small server Leandro Noferini <lnoferin@cybervalley.org> - 2021-08-10 17:20 +0200
    Re: Disk for a small server ghe2001 <ghe2001@protonmail.com> - 2021-08-10 19:10 +0200
      Re: Disk for a small server Leandro Noferini <leandro@cybervalley.org> - 2021-08-10 20:30 +0200
        Re: Disk for a small server ghe2001 <ghe2001@protonmail.com> - 2021-08-10 20:50 +0200
        Re: Disk for a small server rhkramer@gmail.com - 2021-08-10 21:10 +0200
          Re: Disk for a small server Leandro Noferini <lnoferin@cybervalley.org> - 2021-08-10 21:10 +0200
            Re: Disk for a small server Thomas Amm <debili@open-email.net> - 2021-08-10 22:40 +0200
        Re: Disk for a small server Polyna-Maude Racicot-Summerside <debian@polynamaude.com> - 2021-08-11 01:00 +0200
          Re: Disk for a small server Leandro Noferini <leandro@cybervalley.org> - 2021-08-11 11:20 +0200
            Re: Disk for a small server Polyna-Maude Racicot-Summerside <debian@polynamaude.com> - 2021-08-11 12:30 +0200
    Re: Disk for a small server David Christensen <dpchrist@holgerdanske.com> - 2021-08-10 21:50 +0200
      Re: Disk for a small server Dan Ritter <dsr@randomstring.org> - 2021-08-10 22:00 +0200
        Re: Disk for a small server David Christensen <dpchrist@holgerdanske.com> - 2021-08-11 02:40 +0200
          Re: Disk for a small server Celejar <celejar@gmail.com> - 2021-08-11 05:00 +0200
            Re: Disk for a small server David Christensen <dpchrist@holgerdanske.com> - 2021-08-11 12:00 +0200
              Re: Memory and other hardware safety issue regarding ZFS [was Disk  for a small server] Polyna-Maude Racicot-Summerside <debian@polynamaude.com> - 2021-08-11 12:30 +0200
              Re: Disk for a small server Celejar <celejar@gmail.com> - 2021-08-11 16:10 +0200
                Re: Disk for a small server David Christensen <dpchrist@holgerdanske.com> - 2021-08-11 23:50 +0200
                  Re: Disk for a small server Stefan Monnier <monnier@iro.umontreal.ca> - 2021-08-12 00:00 +0200
                    Re: Disk for a small server <tomas@tuxteam.de> - 2021-08-12 08:50 +0200
      Re: Disk for a small server <tomas@tuxteam.de> - 2021-08-10 22:00 +0200
        Re: Disk for a small server David Christensen <dpchrist@holgerdanske.com> - 2021-08-11 03:00 +0200

Page 1 of 2  [1] 2  Next page →

#238432 — Disk for a small server

Ciao a tutti,

I have a little server (debian stable on raspberry) used by my family and a
little set of people (~10) with services like nextcloud (~100GB growing) and
some more.

This server has an external disk for the data, disk that is becoming too little
so I would like to change it with a bigger one (1TB) and I would like to arrange
this new disk in a confgurable fashion, so I would change the partitions on the
way.

I remained to lvm2: there is something newer/better?

-- 
Ciao
leandro

[toc] | [next] | [standalone]

#238438

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, August 10, 2021 9:04 AM, Leandro Noferini <lnoferin@cybervalley.org> wrote:

> I have a little server (debian stable on raspberry) used by my family and a
> little set of people (~10) with services like nextcloud (~100GB growing) and
> some more.
>
> This server has an external disk for the data, disk that is becoming too little
> so I would like to change it with a bigger one (1TB) and I would like to arrange
> this new disk in a confgurable fashion, so I would change the partitions on the
> way.

I have a 1T 2.5", USB disk on on my 'Pi server.  It works and fits nicely under the RPi.  The 4s have USB3 ports.

It's all backed up to tape regularly...

--
Glenn English
>
-----BEGIN PGP SIGNATURE-----
Version: ProtonMail

wsBzBAEBCAAGBQJhErHwACEJEJ/XhjGCrIwyFiEELKJzD0JScCVjQA2Xn9eG
MYKsjDJ2IwgAu22b7LFJXFgjvjcD7CI6X5ZIikQGvvbJ4/HbSDL47KeymVwU
2YCg9K0i6SMTU8NsZQ/9qO1LvL1wbDqCVgDOSlRymzS3uJONF/zbjpPjvpBc
cuE2bP6eANONXVXV7124aKLSAhXyfP0l3BXaOKMspSCVDcW5i/6gJpIRB4sR
I+Sd8tICnn0NbOVZju29Rn63VOLiKlWMpg0/OktxGVaKcjO5cZET92nN2Tbl
qSWF7GnjT2kvvGhM+Hcv5T/Vk4ub3n0fx7Xdx8GvvSVijBUkoctjnEISUKlh
ChmImtjRR+rSQ7nED9qH4jmZxhMSmptWBCaMzp8b7D+0LvRfe26paw==
=/v/K
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#238451

On mar, ago 10, 2021 at 05:06:00 +0000, ghe2001 wrote:

> On Tuesday, August 10, 2021 9:04 AM, Leandro Noferini <lnoferin@cybervalley.org> wrote:
> 
> > I have a little server (debian stable on raspberry) used by my family and a
> > little set of people (~10) with services like nextcloud (~100GB growing) and
> > some more.
> >
> > This server has an external disk for the data, disk that is becoming too little
> > so I would like to change it with a bigger one (1TB) and I would like to arrange
> > this new disk in a confgurable fashion, so I would change the partitions on the
> > way.
> 
> I have a 1T 2.5", USB disk on on my 'Pi server.  It works and fits nicely
> under the RPi.  The 4s have USB3 ports.
> 
> It's all backed up to tape regularly...

Ok, but I need to divide some directories to avoid the fullfilling of the disk.

Do you have only one filesystem in your disk?

-- 
Ciao
leandro

[toc] | [prev] | [next] | [standalone]

#238452

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256




‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, August 10, 2021 12:06 PM, Leandro Noferini <leandro@cybervalley.org> wrote:

> Ok, but I need to divide some directories to avoid the fullfilling of the disk.
>
> Do you have only one filesystem in your disk?

3 partitions, with the same filesystem (ext4) on all of them.  /var/log in there, as is an empty partition, and one where I keep backups of a shopping center in Texas (I'm in Colorado).

--
Glenn English
>
-----BEGIN PGP SIGNATURE-----
Version: ProtonMail

wsBzBAEBCAAGBQJhEshDACEJEJ/XhjGCrIwyFiEELKJzD0JScCVjQA2Xn9eG
MYKsjDK8bgf9H9NRbfDZTESFPEvWlqjtFA/tfMM90xhSigwX6BUHZ8hSEZ7H
AHbJ732tPsXd7g2Zh2hZ4jYCkxALloqRM/3VzoxRvzkb81xl3ROKg8+vx8te
5NllapMHHvMukEucmuppB0NIErWStLWs4Jd8af5H51e605Cgh7RJsWgbcK3A
Tz1EkPaMW9klOR1Fa4lNwNu3gBPnsOHsRTnfvi2HcHAuzubycNJWofrPPQfW
fSlcy1NF3fO7ckAlpE5oS7LQuaRzJcMPoRkxMG/M1ktT2i5LxkDuubnlYZP4
JoxAbxgoeYn4fZs6VRlZZTxu33DdELbuzAacgHxQoQ+ltul2rfePqg==
=d0sL
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#238457

On Tuesday, August 10, 2021 02:06:46 PM Leandro Noferini wrote:
> Ok, but I need to divide some directories to avoid the fullfilling of the
> disk.
> 
> Do you have only one filesystem in your disk?

Is your issue / question how to put multiple partitions on the disk?  Or how 
to allow future rearrangement of partitions as user's needs change.

In general, the answer might be LVM (which I don't use so won't attempt to 
explain how) or maybe simply how to put multiple partitions on a disk?

If you answer those questions, I'm sure somebody will be able to give you good 
advice.

(Aside: You can put multiple partitions on a disk, and they don't all have to 
be part of the standard Linux filesystem hierarchy (e.g., part of /home) -- for 
what I consider good reasons, I have multiple top level directories / 
partitions / mount points (can all be explained) with names like /<user1>, 
/<user2>, ... (vs. /home/<user1>, /home/<user2> ...)

[toc] | [prev] | [next] | [standalone]

#238458

On mar, ago 10, 2021 at 02:59:37 -0400, rhkramer@gmail.com wrote:
> On Tuesday, August 10, 2021 02:06:46 PM Leandro Noferini wrote:
> > Ok, but I need to divide some directories to avoid the fullfilling of the
> > disk.
> > 
> > Do you have only one filesystem in your disk?
> 
> Is your issue / question how to put multiple partitions on the disk?  Or how
> to allow future rearrangement of partitions as user's needs change.

Quite the second: I would like to know if there is a good solution to use a
complete disk with different partitions to use with different file systems for
different uses, knowing that I could have to change the dimensions of these
partitions in the time.

> In general, the answer might be LVM (which I don't use so won't attempt to 
> explain how) or maybe simply how to put multiple partitions on a disk?

Yes I know (a little) lvm but I would like to know if there are different/better
solutions.

> If you answer those questions, I'm sure somebody will be able to give you good 
> advice.
> 
> (Aside: You can put multiple partitions on a disk, and they don't all have to 
> be part of the standard Linux filesystem hierarchy (e.g., part of /home) -- for 
> what I consider good reasons, I have multiple top level directories / 
> partitions / mount points (can all be explained) with names like /<user1>, 
> /<user2>, ... (vs. /home/<user1>, /home/<user2> ...)

I need the standard file system hierarchy.

-- 
Ciao
leandro

[toc] | [prev] | [next] | [standalone]

#238463

On Tue, 2021-08-10 at 21:08 +0200, Leandro Noferini wrote:
> On mar, ago 10, 2021 at 02:59:37 -0400, rhkramer@gmail.com wrote:
> > On Tuesday, August 10, 2021 02:06:46 PM Leandro Noferini wrote:
> > > Ok, but I need to divide some directories to avoid the
> > > fullfilling of the
> > > disk.
> > > 
> > > Do you have only one filesystem in your disk?
> > 
> > Is your issue / question how to put multiple partitions on the
> > disk?  Or how
> > to allow future rearrangement of partitions as user's needs change.
> 
> Quite the second: I would like to know if there is a good solution to
> use a
> complete disk with different partitions to use with different file
> systems for
> different uses, knowing that I could have to change the dimensions of
> these
> partitions in the time.
> 
> > In general, the answer might be LVM (which I don't use so won't
> > attempt to 
> > explain how) or maybe simply how to put multiple partitions on a
> > disk?
> 
> Yes I know (a little) lvm but I would like to know if there are
> different/better
> solutions.
> 
> > If you answer those questions, I'm sure somebody will be able to
> > give you good 
> > advice.
> > 
> > (Aside: You can put multiple partitions on a disk, and they don't
> > all have to 
> > be part of the standard Linux filesystem hierarchy (e.g., part of
> > /home) -- for 
> > what I consider good reasons, I have multiple top level directories
> > / 
> > partitions / mount points (can all be explained) with names like
> > /<user1>, 
> > /<user2>, ... (vs. /home/<user1>, /home/<user2> ...)
> 
> I need the standard file system hierarchy.
> 

Heavens, why ask complexity to a simple question? 'cause we can - why
else?

 Nevertheless, you've given the answer yourself in your first mail.
LVM2 is the current solution for logical volume management and it does
what it is expected to do. Whatever file systems, partitions, logical
volumes or arbitrary stuff like raw partitions, empty spaces or spares
you want, you can put them on LVM logical volumes and wipe them off
there if you want so.

Cheers,

Tom

[toc] | [prev] | [next] | [standalone]

#238469

[Multipart message — attachments visible in raw view] — view raw

Hi,

On 2021-08-10 2:06 p.m., Leandro Noferini wrote:
> On mar, ago 10, 2021 at 05:06:00 +0000, ghe2001 wrote:
> 
>> On Tuesday, August 10, 2021 9:04 AM, Leandro Noferini <lnoferin@cybervalley.org> wrote:
>>
>>> I have a little server (debian stable on raspberry) used by my family and a
>>> little set of people (~10) with services like nextcloud (~100GB growing) and
>>> some more.
>>>
>>> This server has an external disk for the data, disk that is becoming too little
>>> so I would like to change it with a bigger one (1TB) and I would like to arrange
>>> this new disk in a confgurable fashion, so I would change the partitions on the
>>> way.
>>
>> I have a 1T 2.5", USB disk on on my 'Pi server.  It works and fits nicely
>> under the RPi.  The 4s have USB3 ports.
>>
>> It's all backed up to tape regularly...
> 
> Ok, but I need to divide some directories to avoid the fullfilling of the disk.
> 
Ever had the idea of using quota system ?
> Do you have only one filesystem in your disk?
> 

-- 
Polyna-Maude R.-Summerside
-Be smart, Be wise, Support opensource development

[toc] | [prev] | [next] | [standalone]

#238491

Polyna-Maude Racicot-Summerside <debian@polynamaude.com> writes:


[...]

> Ever had the idea of using quota system ?

Is quota to complicated for my needs?

--
Ciao
leandro

[toc] | [prev] | [next] | [standalone]

#238494

[Multipart message — attachments visible in raw view] — view raw

Hi,

On 2021-08-11 4:55 a.m., Leandro Noferini wrote:
> Polyna-Maude Racicot-Summerside <debian@polynamaude.com> writes:
> 
> 
> [...]
> 
>> Ever had the idea of using quota system ?
> 
> Is quota to complicated for my needs?
> 
We all have different view on what's complicated and what effort we are
ready to put into a certain solution.

For example, there's a easy way to write a scientific paper and that's
using LibreOffice Writer (WYSIWYG) but this may not be optimal for
productivity and there's also using LaTeX to compose the same scientific
paper. The last one will require some time to get acquainted but will be
much more effective, you can edit your document from a terminal screen,
you can process the document thru command line, use *fgrep* for
searching thru the source of all your papers, etc.

On my opinion, I'd say this is not something complex, would be easy to
implement and would be easy on maintenance. It will require planning
regarding the group / user quota allowance and that's probably all to do
more than what you actually have.

There's some visual tool for quota management and also it is included as
part of some configuration management tool too, example *webmin*

Quota is the historic tool used on Linux system to prevent one user from
using all the space on a hard drive.

Here's a description of Quota from Digital Ocean (Server Hosting) :

Quotas are used to limit the amount of disk space a user or group can
use on a filesystem. Without such limits, a user could fill up the
machine’s disk and cause problems for other users and services.


Strange enough, there's no Wiki page for Debian regarding Quota in
English, only a French page. But I've found some nice links for information.

https://debian-handbook.info/browse/stable/sect.quotas.html
The Debian Administrator's Handbook - 9.9 Quota

https://www.howtoforge.com/tutorial/linux-quota-ubuntu-debian/
Linux Quota - installation and configuration on Ubuntu and Debian

> --
> Ciao
> leandro
> 

-- 
Polyna-Maude R.-Summerside
-Be smart, Be wise, Support opensource development

[toc] | [prev] | [next] | [standalone]

#238459

On 8/10/21 8:04 AM, Leandro Noferini wrote:
> Ciao a tutti,
> 
> I have a little server (debian stable on raspberry) used by my family and a
> little set of people (~10) with services like nextcloud (~100GB growing) and
> some more.
> 
> This server has an external disk for the data, disk that is becoming too little
> so I would like to change it with a bigger one (1TB) and I would like to arrange
> this new disk in a confgurable fashion, so I would change the partitions on the
> way.
> 
> I remained to lvm2: there is something newer/better?


https://wiki.debian.org/ZFS


But:

- ZFS wants lots of memory.  The rule of thumb is 5 GB of memory for 
every 1 TB of storage.

- ECC memory is safer than non-ECC memory.

- Consider getting two HDD's and creating a mirror.

- "compression=on" is lightweight and generally useful.

- "dedup" is heavyweight, slow on HDD's (due to seek latency), and not 
recommended for general workloads.

- Consider getting a NAS or an entry-level server.


David

[toc] | [prev] | [next] | [standalone]

#238461

David Christensen wrote: 
> On 8/10/21 8:04 AM, Leandro Noferini wrote:
> 
> https://wiki.debian.org/ZFS
> 
> 
> But:
> 
> - ZFS wants lots of memory.  The rule of thumb is 5 GB of memory for every 1
> TB of storage.

This is a myth.

> - ECC memory is safer than non-ECC memory.

This is true, but there is nothing that makes ZFS more dangerous
than another filesystem using non-ECC memory.

> - Consider getting two HDD's and creating a mirror.

Generally a good idea.

> - "compression=on" is lightweight and generally useful.

Also a good idea.

> - "dedup" is heavyweight, slow on HDD's (due to seek latency), and not
> recommended for general workloads.

Not stated strongly enough. Nobody should turn on dedup; people
who think they are experimenting should definitely not turn on
dedup; only people who have a completely sacrificial system and
a good knowledge of the data to be stored should consider dedup.


> - Consider getting a NAS or an entry-level server.

A Raspberry Pi with two disks *is* a NAS or an entry-level
server.

It's not an awesome one, but it is certainly a cheap one and
useful for many purposes.

-dsr-

[toc] | [prev] | [next] | [standalone]

#238471

On 8/10/21 12:56 PM, Dan Ritter wrote:
> David Christensen wrote:
>> On 8/10/21 8:04 AM, Leandro Noferini wrote:
>>
>> https://wiki.debian.org/ZFS
>>
>>
>> But:
>>
>> - ZFS wants lots of memory.  The rule of thumb is 5 GB of memory for every 1
>> TB of storage.
> 
> This is a myth.


Oracle says [1]:

     "... for good ZFS performance, use at least one GB or more of memory."


The FreeBSD "ZFS Tuning Guide" says [2]:

     "To use ZFS, at least 1 GB of memory is recommended (for all
architectures) but more is helpful as ZFS needs *lots* of memory."

     "There are some resources that suggest that one needs 2GB per TB of
storage with deduplication ... . In practice with FreeBSD, based on
empirical testing and additional reading, it's closer to 5GB per TB."


(I use deduplication, so I recalled the "5 GB per TB".)


STFW there are other recommendations.


Looking ahead, I agree that deduplication does not make sense for the 
OP's use-case.  So, 1 GB of memory is supposed to be enough.  Benchmarks 
would be informative, but STFW I am unable to find such.


>> - ECC memory is safer than non-ECC memory.
> 
> This is true, but there is nothing that makes ZFS more dangerous
> than another filesystem using non-ECC memory.


I think the amount of danger depends upon how you do your risk 
assessment math.  I find used entry-level server hardware with ECC 
memory to be desirable for additional reasons.


>> - "dedup" is heavyweight, slow on HDD's (due to seek latency), and not
>> recommended for general workloads.
> 
> Not stated strongly enough. Nobody should turn on dedup; people
> who think they are experimenting should definitely not turn on
> dedup; only people who have a completely sacrificial system and
> a good knowledge of the data to be stored should consider dedup.


This article has some good information:

https://www.truenas.com/docs/references/zfsdeduplication/


I have been running deduplication for several years on my SOHO servers, 
which have HDD's and SATA3 SSD caches.  Normal read/ write operations 
can fill the Gigabit network, but replication is very slow (5.8 MB/s for 
a recent replication job).


David


[1] https://docs.oracle.com/cd/E18752_01/html/819-5461/gbgxg.html

[2] https://wiki.freebsd.org/ZFSTuningGuide

[toc] | [prev] | [next] | [standalone]

#238474

On Tue, 10 Aug 2021 17:35:32 -0700
David Christensen <dpchrist@holgerdanske.com> wrote:

> On 8/10/21 12:56 PM, Dan Ritter wrote:
> > David Christensen wrote:
> >> On 8/10/21 8:04 AM, Leandro Noferini wrote:
> >>
> >> https://wiki.debian.org/ZFS

...

> >> - ECC memory is safer than non-ECC memory.
> > 
> > This is true, but there is nothing that makes ZFS more dangerous
> > than another filesystem using non-ECC memory.
> 
> 
> I think the amount of danger depends upon how you do your risk 
> assessment math.  I find used entry-level server hardware with ECC 
> memory to be desirable for additional reasons.

Dan's point is that while ECC memory is indeed safer than non-ECC
memory, this is true whether one is using ZFS or some other filesystem;
furthermore, with or without ECC memory, there's no reason to believe
that ZFS is less safe than the alternative.

See:

https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/?comments=1&post=38877683
https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

So while ECC memory is always good, it's not a consideration when
trying to choose between ZFS and other filesystems.

Celejar

[toc] | [prev] | [next] | [standalone]

#238492

On 8/10/21 7:51 PM, Celejar wrote:
> On Tue, 10 Aug 2021 17:35:32 -0700
> David Christensen <dpchrist@holgerdanske.com> wrote:
> 
>> On 8/10/21 12:56 PM, Dan Ritter wrote:
>>> David Christensen wrote:
>>>> On 8/10/21 8:04 AM, Leandro Noferini wrote:
>>>>
>>>> https://wiki.debian.org/ZFS
> 
> ...
> 
>>>> - ECC memory is safer than non-ECC memory.
>>>
>>> This is true, but there is nothing that makes ZFS more dangerous
>>> than another filesystem using non-ECC memory.
>>
>>
>> I think the amount of danger depends upon how you do your risk
>> assessment math.  I find used entry-level server hardware with ECC
>> memory to be desirable for additional reasons.
> 
> Dan's point is that while ECC memory is indeed safer than non-ECC
> memory, this is true whether one is using ZFS or some other filesystem;
> furthermore, with or without ECC memory, there's no reason to believe
> that ZFS is less safe than the alternative.
> 
> See:
> 
> https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/?comments=1&post=38877683
> https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/
> 
> So while ECC memory is always good, it's not a consideration when
> trying to choose between ZFS and other filesystems.


I see two sets of choices:

1.  Memory integrity:

     a.  No error checking or correcting -- non-ECC.

     b.  Error checking and correcting -- ECC.

2.  Operating system storage stack data integrity:

     a.  No data integrity -- md, LVM, ext*, FAT, NTFS.

     b.  Data integrity -- dm-integrity, btrfs, ZFS.


There are four combinations of the above.  I order them from highest 
risk (A) to lowest risk (D) as follows:

A.  Non-ECC memory (1a) and data integrity (2b)

B.  Non-ECC memory (1a) and no data integrity (2a)

C.  ECC memory (1b) and no data integrity (2a)

D.  ECC memory (1b) and data integrity (2b)


I have seen a few computers with failing non-ECC memory and no OS 
storage stack data integrity (case B).  It might take weeks or months to 
identify the problem.  If those computers had had OS storage stack data 
integrity with automatic correction (case A), the "scrub of death" is 
the logical outcome (failure modes and effects analysis); it's just a 
question of time.  Given the eventual catastrophic outcome (fault hazard 
analysis), I see a significant difference in risk between A and B.


I started buying ECC machines specifically for ZFS a few years ago (case 
D), and suffered through a rash of drive, rack, cable, and/or HBA 
failures.  Given RAID, ZFS snapshots, backups, etc.,, I replaced bad 
drives, fixed connections, resilvered, restored, verified, etc., with 
minimal loss.  If I had chosen md, LVM, and ext4 instead (case C), there 
would still be hardware checksums inside the drives, hardware checksums 
on the connections,  and memory checksums.  So, the risk difference C-D 
is less pronounced than A-B.


Holding the data integrity choice constant and comparing memory choices 
(cases A-D and cases B-C), I see more risk with non-ECC memory and less 
risk with ECC memory for both data integrity choices.


So, I do consider memory when choosing the storage stack.  Furthermore, 
my OS storage stack data integrity choice with non-ECC memory is the 
opposite of my choice with ECC memory.  My desktops and laptops have 
non-ECC and ext4 (case B).  My servers have ECC and ZFS (case D).


Therefore, my suggestion of ZFS on RPi contradicts my own practice.  :-/


David

[toc] | [prev] | [next] | [standalone]

#238496 — Re: Memory and other hardware safety issue regarding ZFS [was Disk for a small server]

[Multipart message — attachments visible in raw view] — view raw

Hi,

On 2021-08-11 5:53 a.m., David Christensen wrote:
> On 8/10/21 7:51 PM, Celejar wrote:
>> On Tue, 10 Aug 2021 17:35:32 -0700
>> David Christensen <dpchrist@holgerdanske.com> wrote:
>>
>>> On 8/10/21 12:56 PM, Dan Ritter wrote:
>>>> David Christensen wrote:
>>>>> On 8/10/21 8:04 AM, Leandro Noferini wrote:
>>>>>
>>>>> https://wiki.debian.org/ZFS
>>
>> ...
>>
>>>>> - ECC memory is safer than non-ECC memory.
>>>>
>>>> This is true, but there is nothing that makes ZFS more dangerous
>>>> than another filesystem using non-ECC memory.
>>>
>>>
>>> I think the amount of danger depends upon how you do your risk
>>> assessment math.  I find used entry-level server hardware with ECC
>>> memory to be desirable for additional reasons.
>>
>> Dan's point is that while ECC memory is indeed safer than non-ECC
>> memory, this is true whether one is using ZFS or some other filesystem;
>> furthermore, with or without ECC memory, there's no reason to believe
>> that ZFS is less safe than the alternative.
>>
>> See:
>>
>> https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/?comments=1&post=38877683
>>
>> https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/
>>
>> So while ECC memory is always good, it's not a consideration when
>> trying to choose between ZFS and other filesystems.
> 
> 
> I see two sets of choices:
> 
> 1.  Memory integrity:
> 
>     a.  No error checking or correcting -- non-ECC.
> 
>     b.  Error checking and correcting -- ECC.
> 
> 2.  Operating system storage stack data integrity:
> 
>     a.  No data integrity -- md, LVM, ext*, FAT, NTFS.
> 
>     b.  Data integrity -- dm-integrity, btrfs, ZFS.
> 
> 
> There are four combinations of the above.  I order them from highest
> risk (A) to lowest risk (D) as follows:
> 
> A.  Non-ECC memory (1a) and data integrity (2b)
> 
> B.  Non-ECC memory (1a) and no data integrity (2a)
> 
> C.  ECC memory (1b) and no data integrity (2a)
> 
> D.  ECC memory (1b) and data integrity (2b)
> 
> 
> I have seen a few computers with failing non-ECC memory and no OS
> storage stack data integrity (case B).  It might take weeks or months to
> identify the problem.  If those computers had had OS storage stack data
> integrity with automatic correction (case A), the "scrub of death" is
> the logical outcome (failure modes and effects analysis); it's just a
> question of time.  Given the eventual catastrophic outcome (fault hazard
> analysis), I see a significant difference in risk between A and B.
> 
> 
> I started buying ECC machines specifically for ZFS a few years ago (case
> D), and suffered through a rash of drive, rack, cable, and/or HBA
> failures.  Given RAID, ZFS snapshots, backups, etc.,, I replaced bad
> drives, fixed connections, resilvered, restored, verified, etc., with
> minimal loss.  If I had chosen md, LVM, and ext4 instead (case C), there
> would still be hardware checksums inside the drives, hardware checksums
> on the connections,  and memory checksums.  So, the risk difference C-D
> is less pronounced than A-B.
> 
> 
> Holding the data integrity choice constant and comparing memory choices
> (cases A-D and cases B-C), I see more risk with non-ECC memory and less
> risk with ECC memory for both data integrity choices.
> 
> 
> So, I do consider memory when choosing the storage stack.  Furthermore,
> my OS storage stack data integrity choice with non-ECC memory is the
> opposite of my choice with ECC memory.  My desktops and laptops have
> non-ECC and ext4 (case B).  My servers have ECC and ZFS (case D).
> 
> 
> Therefore, my suggestion of ZFS on RPi contradicts my own practice.  :-/
> 
> 
> David
> 

-- 
Polyna-Maude R.-Summerside
-Be smart, Be wise, Support opensource development

[toc] | [prev] | [next] | [standalone]

#238522

On Wed, 11 Aug 2021 02:53:13 -0700
David Christensen <dpchrist@holgerdanske.com> wrote:

> On 8/10/21 7:51 PM, Celejar wrote:
> > On Tue, 10 Aug 2021 17:35:32 -0700
> > David Christensen <dpchrist@holgerdanske.com> wrote:
> > 
> >> On 8/10/21 12:56 PM, Dan Ritter wrote:
> >>> David Christensen wrote:
> >>>> On 8/10/21 8:04 AM, Leandro Noferini wrote:
> >>>>
> >>>> https://wiki.debian.org/ZFS
> > 
> > ...
> > 
> >>>> - ECC memory is safer than non-ECC memory.
> >>>
> >>> This is true, but there is nothing that makes ZFS more dangerous
> >>> than another filesystem using non-ECC memory.
> >>
> >>
> >> I think the amount of danger depends upon how you do your risk
> >> assessment math.  I find used entry-level server hardware with ECC
> >> memory to be desirable for additional reasons.
> > 
> > Dan's point is that while ECC memory is indeed safer than non-ECC
> > memory, this is true whether one is using ZFS or some other filesystem;
> > furthermore, with or without ECC memory, there's no reason to believe
> > that ZFS is less safe than the alternative.
> > 
> > See:
> > 
> > https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/?comments=1&post=38877683
> > https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/
> > 
> > So while ECC memory is always good, it's not a consideration when
> > trying to choose between ZFS and other filesystems.
> 
> 
> I see two sets of choices:
> 
> 1.  Memory integrity:
> 
>      a.  No error checking or correcting -- non-ECC.
> 
>      b.  Error checking and correcting -- ECC.
> 
> 2.  Operating system storage stack data integrity:
> 
>      a.  No data integrity -- md, LVM, ext*, FAT, NTFS.
> 
>      b.  Data integrity -- dm-integrity, btrfs, ZFS.
> 
> 
> There are four combinations of the above.  I order them from highest 
> risk (A) to lowest risk (D) as follows:
> 
> A.  Non-ECC memory (1a) and data integrity (2b)
> 
> B.  Non-ECC memory (1a) and no data integrity (2a)
> 
> C.  ECC memory (1b) and no data integrity (2a)
> 
> D.  ECC memory (1b) and data integrity (2b)
> 
> 
> I have seen a few computers with failing non-ECC memory and no OS 
> storage stack data integrity (case B).  It might take weeks or months to 
> identify the problem.  If those computers had had OS storage stack data 
> integrity with automatic correction (case A), the "scrub of death" is 
> the logical outcome (failure modes and effects analysis); it's just a 
> question of time.  Given the eventual catastrophic outcome (fault hazard 
> analysis), I see a significant difference in risk between A and B.

I myself have no personal experience or deep understanding of the
issues, but the experts do not accept your position that A is higher
risk than B due to the possibility of the "scrub of death." Here's Jim
Salter (from the second link I gave above):

> Is ZFS and non-ECC worse than not-ZFS and non-ECC? What about the Scrub
> of Death?
> 
> OK, it’s pretty easy to demonstrate that a flipped bit in RAM means
> data corruption: if you write that flipped bit back out to disk,
> congrats, you just wrote bad data. There’s no arguing that. The real
> issue here isn’t whether ECC is good to have, it’s whether non-ECC is
> particularly problematic with ZFS. The scenario usually thrown out is
> the the much-dreaded Scrub Of Death.
> 
> TL;DR version of the scenario: ZFS is on a system with non-ECC RAM that
> has a stuck bit, its user initiates a scrub, and as a result of
> in-memory corruption good blocks fail checksum tests and are
> overwritten with corrupt data, thus instantly murdering an entire pool.
> As far as I can tell, this idea originates with a very prolific user on
> the FreeNAS forums named Cyberjock, and he lays it out in this thread
> here. It’s a scary idea – what if the very thing that’s supposed to
> keep your system safe kills it? A scrub gone mad! Nooooooo!
> 
> The problem is, the scenario as written doesn’t actually make sense.
> For one thing, even if you have a particular address in RAM with a
> stuck bit, you aren’t going to have your entire filesystem run through
> that address. That’s not how memory management works, and if it were
> how memory management works, you wouldn’t even have managed to boot the
> system: it would have crashed and burned horribly when it failed to
> load the operating system in the first place. So no, you might corrupt
> a block here and there, but you’re not going to wring the entire
> filesystem through a shredder block by precious block.
> 
> But we’re being cheap here. Say you only corrupt one block in 5,000
> this way. That would still be hellacious. So let’s examine the more
> reasonable idea of corrupting some data due to bad RAM during a scrub.
> And let’s assume that we have RAM that not only isn’t working 100%
> properly, but is actively goddamn evil and trying its naive but
> enthusiastic best to specifically kill your data during a scrub:
> 
> First, you read a block. This block is good. It is perfectly good data
> written to a perfectly good disk with a perfectly matching checksum.
> But that block is read into evil RAM, and the evil RAM flips some bits.
> Perhaps those bits are in the data itself, or perhaps those bits are in
> the checksum. Either way, your perfectly good block now does not appear
> to match its checksum, and since we’re scrubbing, ZFS will attempt to
> actually repair the “bad” block on disk. Uh-oh! What now?
> 
> Next, you read a copy of the same block – this copy might be a
> redundant copy, or it might be reconstructed from parity, depending on
> your topology. The redundant copy is easy to visualize – you literally
> stored another copy of the block on another disk. Now, if your evil RAM
> leaves this block alone, ZFS will see that the second copy matches its
> checksum, and so it will overwrite the first block with the same data
> it had originally – no data was lost here, just a few wasted disk
> cycles. OK. But what if your evil RAM flips a bit in the second copy?
> Since it doesn’t match the checksum either, ZFS doesn’t overwrite
> anything. It logs an unrecoverable data error for that block, and
> leaves both copies untouched on disk. No data has been corrupted. A
> later scrub will attempt to read all copies of that block and validate
> them just as though the error had never happened, and if this time
> either copy passes, the error will be cleared and the block will be
> marked valid again (with any copies that don’t pass validation being
> overwritten from the one that did).
> 
> Well, huh. That doesn’t sound so bad. So what does your evil RAM need
> to do in order to actually overwrite your good data with corrupt data
> during a scrub? Well, first it needs to flip some bits during the
> initial read of every block that it wants to corrupt. Then, on the
> second read of a copy of the block from parity or redundancy, it needs
> to not only flip bits, it needs to flip them in such a way that you get
> a hash collision. In other words, random bit-flipping won’t do – you
> need some bit flipping in the data (with or without some more
> bit-flipping in the checksum) that adds up to the corrupt data
> correctly hashing to the value in the checksum. By default, ZFS uses
> 256-bit SHA validation hashes, which means that a single bit-flip has a
> 1 in 2^256 chance of giving you a corrupt block which now matches its
> checksum. To be fair, we’re using evil RAM here, so it’s probably going
> to do lots of experimenting, and it will try flipping bits in both the
> data and the checksum itself, and it will do so multiple times for any
> single block. However, that’s multiple 1 in 2^256 (aka roughly 1 in
> 10^77) chances, which still makes it vanishingly unlikely to actually
> happen… and if your RAM is that damn evil, it’s going to kill your data
> whether you’re using ZFS or not. ...

[snipped the rest of Jim's analysis]

> I don’t care about your logic! I wish to appeal to authority!
> 
> OK. “Authority” in this case doesn’t get much better than Matthew
> Ahrens, one of the cofounders of ZFS at Sun Microsystems and current
> ZFS developer at Delphix. In the comments to one of my filesystem
> articles on Ars Technica, Matthew said “There’s nothing special about
> ZFS that requires/encourages the use of ECC RAM more so than any other
> filesystem.”
> 
> Hope that helps. =)

Celejar

[toc] | [prev] | [next] | [standalone]

#238541

On 8/11/21 7:00 AM, Celejar wrote:

> I myself have no personal experience or deep understanding of the 
> issues, but the experts do not accept your position that [non-ECC
> memory combined with operating system storage stack integrity
> checking] is higher risk than [ECC memory combined with operating
> system storage stack integrity checking] due to the possibility of
> the "scrub of death." Here's Jim Salter (from the second link I gave
> above):


Without a thorough review of all the engineering for all of the various 
computer hardware, software, and systems under discussion, definitive 
answers cannot be found through analysis alone.  That leaves opinion. 
"Academics", "experts", "creators", etc., are typically favored, but 
sometimes the nobodies with real-world experience are "right" (to a 
greater or less degree).


That is why there is the scientific method.  Please cite relevant 
article(s) with reproducible laboratory results and/or analysis of 
long-term real-world data regarding failure modes, effects, and hazards 
of non-ECC memory vs. ECC memory when paired with operating systems with 
vs. without storage stack integrity checking and correction.


David

[toc] | [prev] | [next] | [standalone]

#238542

David Christensen [2021-08-11 14:48:05] wrote:
> That is why there is the scientific method.  Please cite relevant article(s)
> with reproducible laboratory results and/or analysis of long-term real-world
> data regarding failure modes, effects, and hazards of non-ECC memory vs. ECC
> memory when paired with operating systems with vs. without storage stack
> integrity checking and correction.

And until such empirical data shows up, I'll give the benefit of the
doubt to the scientists/academics.


        Stefan

[toc] | [prev] | [next] | [standalone]

#238548

[Multipart message — attachments visible in raw view] — view raw

On Wed, Aug 11, 2021 at 05:54:24PM -0400, Stefan Monnier wrote:
> David Christensen [2021-08-11 14:48:05] wrote:
> > That is why there is the scientific method.  Please cite relevant article(s)

[...]

> And until such empirical data shows up, I'll give the benefit of the
> doubt to the scientists/academics.

That's what we pay them for, after all ;-D

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web