Groups | Search | Server Info | Login | Register

Groups > alt.os.linux > #81680

Re: GlusterFS with replica 3

Show all headers | View raw

On 07/07/2025 22.52, ^Bart wrote:
>> Have you seen anything in the logs? Maybe check /var/log/glusterfs/ 
>> glusterd.log
> 
> There's nothing wrong in the normal mode but when the system starts the 
> backup of some db, more or less six, the node changes from Y to N but 
> just with the most important db 1,9GB and it works well with other dbs 
> untill 1,4GB and on the logs of gluster I can read something like "the 
> node is disconnect" because can't read other peers.

Could this be that you are reaching max transfer on your network, which 
could lead to that there ain't enough bandwidth for both file transfer 
and checking of nodes are up?

Another alternative is that the nodes can't handle the speed the 
incoming traffic has for a longer time, as it's on 100% disk 
utilization, network traffic will also suffer (I have seen this, when 
disk is slow, everything else seems to go into a crawl and network 
connections fail as everything is queued up and while queued packages 
times out).


>> It can be lock that hasn't been released, sadly only fix is to restart 
>> glusterd.
> 
> It's very sadly I can just restart the daemon to fix the "N" :\ but I 
> could try to add more ram and get other 2GB so change from 8GB to 10GB.

If the RAM is used a cache before things are written down to disk, this 
could help for a while, until the extra 2 GB is used up. If lucky that 
is more than needed and then nothing ill will happen until you have more 
data to backup.


>> It's quite many years since I used GlusterFS, but back then at work we 
>> had quite large DELL servers (64GB RAM, 2 CPU with 8 cores each) with 
>> SAN based storage as nodes, the system got degradation with having high 
> 
> I think GlusterFS needs more than 8GB of ram to prevent spikes when the 
> system does backup, ok also without backup job there isn't a lot of free 
> memory (300-400MB) but there aren't down nodes!
> 
>> read/write. In the end those was replaced by standard NFS servers 
>> which gave more stability and then have replication from one SAN to 
>> another, sure not a fully HA solution.
> 
> I'm watching cephfs but on internet I read it needs more ram than what I 
> use now so... like what I wrote above I think now I could try to upgrade 
> ram and run tests on GlusterFS because to change a "production cluster" 
> is not easy like a charm but I know also there aren't future plans about 
> gluster and I heard it will be closed so... cephfs will be the only 
> alternative.

Yeah, it's a load of work, think we had like 48h downtime when we 
switched from gluster to nfs, customers wasn't that happy.

>> gluster.org do write, for basic nodes: 2 CPU’s, 4GB of RAM each, 1 
>> Gigabit network.
> 
> I read it but in a real production environment I think the cpu and ram 
> quantities are little bit different...

All depends on what you are doing, small amount cpu/ram works fine in 
lab environments as you usually don't have 300+ clients trying to write.

-- 
  //Aho

Back to alt.os.linux | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

GlusterFS with replica 3 ^Bart <none@none.it> - 2025-07-05 08:41 +0200
  Re: GlusterFS with replica 3 Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-07-05 07:18 +0000
  Re: GlusterFS with replica 3 Daniel70 <daniel47@eternal-september.org> - 2025-07-05 20:08 +1000
    Re: GlusterFS with replica 3 vallor <vallor@cultnix.org> - 2025-07-06 02:09 +0000
      Re: GlusterFS with replica 3 Paul <nospam@needed.invalid> - 2025-07-06 04:12 -0400
  Re: GlusterFS with replica 3 "J.O. Aho" <user@example.net> - 2025-07-05 17:25 +0200
    Re: GlusterFS with replica 3 ^Bart <none@none.it> - 2025-07-07 22:52 +0200
      Re: GlusterFS with replica 3 "J.O. Aho" <user@example.net> - 2025-07-08 09:10 +0200
        Re: GlusterFS with replica 3 ^Bart <none@none.it> - 2025-10-26 21:25 +0100
          Re: GlusterFS with replica 3 Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-10-26 20:56 +0000
            Re: GlusterFS with replica 3 Joerg Walther <joerg.walther@magenta.de> - 2025-10-27 17:40 +0100
              Re: GlusterFS with replica 3 Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-10-27 21:15 +0000
                Re: GlusterFS with replica 3 "J.O. Aho" <user@example.net> - 2025-10-28 09:10 +0100
                Re: GlusterFS with replica 3 Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-10-28 23:09 +0000

csiph-web