Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #171925 > unrolled thread

Re: Problems communicating with and between servers after upgrade - correction

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Problems communicating with and between servers after upgrade -  correction Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-09 01:00 +0200
    Re: Problems communicating with and between servers after upgrade -  correction Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-09 10:20 +0200
      Re: Problems communicating with and between servers after upgrade -  correction Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-09 10:30 +0200
    Systemd and Init was Problems communicating with and between servers  after upgrade - correction Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-12 20:50 +0200
      Re: Systemd and Init was Problems communicating with and between servers after upgrade - correction Greg Wooledge <wooledg@eeg.ccf.org> - 2016-09-12 21:20 +0200
        Re: Systemd and Init was Problems communicating with and between  servers after upgrade - correction Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-12 22:30 +0200

#171925 — Re: Problems communicating with and between servers after upgrade - correction

On 08/09/16 23:07, Clive Menzies wrote:
> Hi
>
> We've suffered a series of seemingly disconnect problems on 4 machines 
> since upgrading jessie on Monday:
>
> apt/log:
>
> Start-Date: 2016-09-05  12:17:10
> Commandline: apt-get upgrade
> Upgrade: libgcrypt20:i386 (1.6.3-2+deb8u1, 1.6.3-2+deb8u2), gnupg:i386 
> (1.4.18-7+deb8u1, 1.4.18-7+deb8u2), linux-libc-dev:i386 
> (3.16.7-ckt25-2+deb8u3, 3.16.36-1+deb8u1), 
> linux-image-3.16.0-4-686-pae:i386 (3.16.7-ckt25-2+deb8u3, 
> 3.16.36-1+deb8u1), gpgv:i386 (1.4.18-7+deb8u1, 1.4.18-7+deb8u2), 
> libidn11:i386 (1.29-1+deb8u1, 1.29-1+deb8u2)
> End-Date: 2016-09-05  12:19:22
>
> First there was a DMA error on bootup on file and mail server_U, we 
> were alerted to by no email being delivered from dovecot on the 
> server. We fsck'd the disk offline and no errors were reported. 
> Although the system would boot, dovecot wouldn't work. We've been 
> through so many permutations and combinations since that I can't 
> remember each step we took after that, some of which we repeated. 
> Eventually, in spite of the fsck result, we replaced the disk and 
> reinstalled. The samba installation worked out of the box but dovecot 
> and rsync (for automated remote backups) didn't. It turned out to be a 
> certification problem which required creating the certs while making 
> sure dovecot knows where to look (not straightforward). I can't 
> remember the specifics of the rsync issue; it may have been 
> self-inflicted. Eventually, all was well and everything was working, 
> including the remote backup.
>
> Tuesday: we'd lost ssh connection to the two remote backup servers via 
> the VPN; I've no idea of their state other than getting someone onsite 
> to reboot them using the power button - they appear to be working.
>
> We then found that laptop_T can access smb shares (using both windows 
> and debian systems) on file and mail server_M but two other linux 
> machines couldn't, nor could a remote windows laptop via VPN, (but 
> that may be because the user's machine is "broken" but the timing is 
> suspicious). Nor can he get to dovecot with his Thunderbird email 
> client which may be related to the same upgrade.
>
> We've compared /etc/fstab on laptop_T which mounts the shares with no 
> problem to that on laptop_D which doesn't. Same user, same share and 
> they are identical in respect of mounting the shares but one works, 
> one doesn't.
>
> As server_U is working after reinstallation, following much 
> exploration, we reinstalled jessie 8.3 on server_M and committed to 
> systemd to avoid potential progressive sysv-init problems we'd learned 
> of during our investigation. After a reinstall of the system and 
> subsequently samba (twice), we resorted to the maintainer's version of 
> /etc/smb.conf and customised it for our setup. We tried to keep the 
> configuration as vanilla as possible but there was no improvement in 
> terms of access from the two debian machines.
>
> In comparing the /etc/smb.conf with that on server_U, we noticed that 
> the winserver IP address on U was uncommented and gave its LAN IP (it 
> is acting as the winserver for the workgroup). We edited server_M 
> /etc/fstab to include the winserver IP and debian workstation_E saw 
> the shares in file manager but the share didn't show up in df -h. The 
> shares appeared to be unmounted but were accessible through 
> Thunar(FM). We commented it out and access broke, uncommented it and 
> it worked again. On laptop_D the IP "fix" didn't have any effect - 
> won't mount and can't be seen.
Sorry, brain disengaged. We edited server_M /etc/smb.conf to include the 
winserver IP NOT /etc/fstab
>
> At this point we're stuck which gives pause for reflection. These 4 
> servers have been running stable debian for over 10 years and apart 
> from the odd hardware issue have been rock solid. Two of the machines 
> have been replaced more than once over the years but the other two are 
> the original boxes. Most upgrades were pretty seamless and if there 
> were problems, a short burst of intensive exploration, trial and 
> error, quickly resolved them.
>
> This nightmare of expanding problems has been going on for three days, 
> since Monday afternoon. Never before have I questioned the decision to 
> base our business (and our lives) on Debian and I remain a firm 
> advocate. I also recognise that over successive releases, 
> accommodating a plethora of configurations becomes harder and that at 
> some point a step changes in the foundations of the system are 
> required. I'm presuming that the transition to systemd from sysv-init 
> was an essential step and understand that backwards compatibility 
> becomes more challenging as time goes on.
>
> Whether this systemd transition is related to the remote connectivity 
> with the servers and the samba issue, I don't know but this number of 
> seemingly random but mission critical series of problems has shaken 
> our confidence.
>
> Apologies if this sounds like a complaint, it's not. It is a concern, 
> which someone may be able to allay, that Debian is not as rock solid 
> as it was.
>
> You guys have done brilliant work and I'm aware that my contribution 
> to the project has been very small and pretty non-existent for the 
> last few years - other priorities. So thank you.
>
> Regards
>
> Clive
>
> PS If anyone has any ideas to help, they'd be much appreciated.
>

-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [next] | [standalone]

#171933

On 08/09/16 23:50, Clive Menzies wrote:
> On 08/09/16 23:07, Clive Menzies wrote:
>> This nightmare of expanding problems has been going on for three 
>> days, since Monday afternoon. Never before have I questioned the 
>> decision to base our business (and our lives) on Debian and I remain 
>> a firm advocate. I also recognise that over successive releases, 
>> accommodating a plethora of configurations becomes harder and that at 
>> some point a step changes in the foundations of the system are 
>> required. I'm presuming that the transition to systemd from sysv-init 
>> was an essential step and understand that backwards compatibility 
>> becomes more challenging as time goes on.
>>
Just when I thought life couldn't get anymore bizarre, dovecot-imapd and 
core have disappeared off server_M this morning along with the config 
files which had the cert info in. The mail on server_U is working fine. 
I've looked in the apt-log and there's no indication it was uninstalled 
which is not surprising because it was working fine in the early hours 
of this morning.

-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [prev] | [next] | [standalone]

#171935

Please ignore this last message about dovecot - my brain was scrambled. 
I hadn't installed it yet.


On 09/09/16 09:16, Clive Menzies wrote:
> On 08/09/16 23:50, Clive Menzies wrote:
>> On 08/09/16 23:07, Clive Menzies wrote:
>>> This nightmare of expanding problems has been going on for three 
>>> days, since Monday afternoon. Never before have I questioned the 
>>> decision to base our business (and our lives) on Debian and I remain 
>>> a firm advocate. I also recognise that over successive releases, 
>>> accommodating a plethora of configurations becomes harder and that 
>>> at some point a step changes in the foundations of the system are 
>>> required. I'm presuming that the transition to systemd from 
>>> sysv-init was an essential step and understand that backwards 
>>> compatibility becomes more challenging as time goes on.
>>>
> Just when I thought life couldn't get anymore bizarre, dovecot-imapd 
> and core have disappeared off server_M this morning along with the 
> config files which had the cert info in. The mail on server_U is 
> working fine. I've looked in the apt-log and there's no indication it 
> was uninstalled which is not surprising because it was working fine in 
> the early hours of this morning.
>

-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [prev] | [next] | [standalone]

#172091 — Systemd and Init was Problems communicating with and between servers after upgrade - correction

On 08/09/16 23:50, Clive Menzies wrote:
> On 08/09/16 23:07, Clive Menzies wrote:
>> Hi
>>
>> We've suffered a series of seemingly disconnect problems on 4 
>> machines since upgrading jessie on Monday:
>>
>> apt/log:
>>
>> Start-Date: 2016-09-05  12:17:10
>> Commandline: apt-get upgrade
>> Upgrade: libgcrypt20:i386 (1.6.3-2+deb8u1, 1.6.3-2+deb8u2), 
>> gnupg:i386 (1.4.18-7+deb8u1, 1.4.18-7+deb8u2), linux-libc-dev:i386 
>> (3.16.7-ckt25-2+deb8u3, 3.16.36-1+deb8u1), 
>> linux-image-3.16.0-4-686-pae:i386 (3.16.7-ckt25-2+deb8u3, 
>> 3.16.36-1+deb8u1), gpgv:i386 (1.4.18-7+deb8u1, 1.4.18-7+deb8u2), 
>> libidn11:i386 (1.29-1+deb8u1, 1.29-1+deb8u2)
>> End-Date: 2016-09-05  12:19:22
>>
>> First there was a DMA error on bootup on file and mail server_U, we 
>> were alerted to by no email being delivered from dovecot on the 
>> server. We fsck'd the disk offline and no errors were reported. 
>> Although the system would boot, dovecot wouldn't work. We've been 
>> through so many permutations and combinations since that I can't 
>> remember each step we took after that, some of which we repeated. 
>> Eventually, in spite of the fsck result, we replaced the disk and 
>> reinstalled. The samba installation worked out of the box but dovecot 
>> and rsync (for automated remote backups) didn't. It turned out to be 
>> a certification problem which required creating the certs while 
>> making sure dovecot knows where to look (not straightforward). I 
>> can't remember the specifics of the rsync issue; it may have been 
>> self-inflicted. Eventually, all was well and everything was working, 
>> including the remote backup.
>>
>> Tuesday: we'd lost ssh connection to the two remote backup servers 
>> via the VPN; I've no idea of their state other than getting someone 
>> onsite to reboot them using the power button - they appear to be 
>> working.
>>
>> We then found that laptop_T can access smb shares (using both windows 
>> and debian systems) on file and mail server_M but two other linux 
>> machines couldn't, nor could a remote windows laptop via VPN, (but 
>> that may be because the user's machine is "broken" but the timing is 
>> suspicious). Nor can he get to dovecot with his Thunderbird email 
>> client which may be related to the same upgrade.
>>
>> We've compared /etc/fstab on laptop_T which mounts the shares with no 
>> problem to that on laptop_D which doesn't. Same user, same share and 
>> they are identical in respect of mounting the shares but one works, 
>> one doesn't.
>>
>> As server_U is working after reinstallation, following much 
>> exploration, we reinstalled jessie 8.3 on server_M and committed to 
>> systemd to avoid potential progressive sysv-init problems we'd 
>> learned of during our investigation. After a reinstall of the system 
>> and subsequently samba (twice), we resorted to the maintainer's 
>> version of /etc/smb.conf and customised it for our setup. We tried to 
>> keep the configuration as vanilla as possible but there was no 
>> improvement in terms of access from the two debian machines.
>>
>> In comparing the /etc/smb.conf with that on server_U, we noticed that 
>> the winserver IP address on U was uncommented and gave its LAN IP (it 
>> is acting as the winserver for the workgroup). We edited server_M 
>> /etc/fstab to include the winserver IP and debian workstation_E saw 
>> the shares in file manager but the share didn't show up in df -h. The 
>> shares appeared to be unmounted but were accessible through 
>> Thunar(FM). We commented it out and access broke, uncommented it and 
>> it worked again. On laptop_D the IP "fix" didn't have any effect - 
>> won't mount and can't be seen.
> Sorry, brain disengaged. We edited server_M /etc/smb.conf to include 
> the winserver IP NOT /etc/fstab
>>
>> At this point we're stuck which gives pause for reflection. These 4 
>> servers have been running stable debian for over 10 years and apart 
>> from the odd hardware issue have been rock solid. Two of the machines 
>> have been replaced more than once over the years but the other two 
>> are the original boxes. Most upgrades were pretty seamless and if 
>> there were problems, a short burst of intensive exploration, trial 
>> and error, quickly resolved them.
>>
>> This nightmare of expanding problems has been going on for three 
>> days, since Monday afternoon. Never before have I questioned the 
>> decision to base our business (and our lives) on Debian and I remain 
>> a firm advocate. I also recognise that over successive releases, 
>> accommodating a plethora of configurations becomes harder and that at 
>> some point a step changes in the foundations of the system are 
>> required. I'm presuming that the transition to systemd from sysv-init 
>> was an essential step and understand that backwards compatibility 
>> becomes more challenging as time goes on.
>>
>> Whether this systemd transition is related to the remote connectivity 
>> with the servers and the samba issue, I don't know but this number of 
>> seemingly random but mission critical series of problems has shaken 
>> our confidence.
>>
>> Apologies if this sounds like a complaint, it's not. It is a concern, 
>> which someone may be able to allay, that Debian is not as rock solid 
>> as it was.
>>
>> You guys have done brilliant work and I'm aware that my contribution 
>> to the project has been very small and pretty non-existent for the 
>> last few years - other priorities. So thank you.
>>
>>
We've managed to sort out the urgent mission critical problems, only 
some of which seem related to the upgrade. Coincidental problems arose 
which really obscured what was going on. So far we've our two main 
servers up and one of the backup servers at the remote location. The 
other one we've recovered because we'll reinstall it - it is the same 
hardware configuration as server_M.

For reasons I can't specify, IDE seems to be an issue and so we put a 
SATA disk in server_M, reinstalled and it seems to be working OK.

However, studying documentation on systemd v init, I'm a bit confused. I 
assumed the reinstall would implement systemd for all services and init 
wouldn't be visible although symlinks will use init where necessary. 
systemd-sysv is installed as per the Debian wiki.

Looking at other setups on the net, a ps aux should assign the first 
process (1) to systemd but on server_M it is /sbin/init

Is this right?

Thanks

Clive

-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [prev] | [next] | [standalone]

#172093 — Re: Systemd and Init was Problems communicating with and between servers after upgrade - correction

On Mon, Sep 12, 2016 at 07:32:52PM +0100, Clive Menzies wrote:
> However, studying documentation on systemd v init, I'm a bit confused. I 
> assumed the reinstall would implement systemd for all services and init 
> wouldn't be visible although symlinks will use init where necessary. 
> systemd-sysv is installed as per the Debian wiki.

Most of the common server-oriented packages in jessie have systemd
units written for them.  E.g. openssh-server:

~$ dpkg -L openssh-server | grep systemd
/lib/systemd
/lib/systemd/system
/lib/systemd/system/ssh.service
/lib/systemd/system/ssh.socket
/lib/systemd/system/ssh@.service

However, there are still several packages that were not updated to use
systemd facilities in time for jessie.  These packages still ship with
sysvinit scripts.

When a given service has sysvinit script and does *not* have systemd
unit files, it falls back to the sysvinit scripts.

> Looking at other setups on the net, a ps aux should assign the first 
> process (1) to systemd but on server_M it is /sbin/init
> 
> Is this right?

The kernel still runs /sbin/init because the systemd changes are not
intrusive all the way up inside the kernel.  Thus, the kernel does not
know or care which init system you're using.

On a jessie-systemd (default) installation, /sbin/init is a symlink:

~$ ls -ld /sbin/init
lrwxrwxrwx 1 root root 20 Mar 12  2016 /sbin/init -> /lib/systemd/systemd

[toc] | [prev] | [next] | [standalone]

#172096 — Re: Systemd and Init was Problems communicating with and between servers after upgrade - correction

On 12/09/16 19:57, Greg Wooledge wrote:
> On Mon, Sep 12, 2016 at 07:32:52PM +0100, Clive Menzies wrote:
> Most of the common server-oriented packages in jessie have systemd
> units written for them.  E.g. openssh-server:
>
> ~$ dpkg -L openssh-server | grep systemd
> /lib/systemd
> /lib/systemd/system
> /lib/systemd/system/ssh.service
> /lib/systemd/system/ssh.socket
> /lib/systemd/system/ssh@.service
>
> However, there are still several packages that were not updated to use
> systemd facilities in time for jessie.  These packages still ship with
> sysvinit scripts.
>
> When a given service has sysvinit script and does *not* have systemd
> unit files, it falls back to the sysvinit scripts.
>
>
Thanks Greg

That makes sense. Does this mean init will disappear from Debian in a 
future release?

It is interesting that you gave ssh as an example because we seem to 
have encountered strange problems with mixed IDE/SATA systems: 
specifically ssh, samba and dovecot.

The first server(U) we reinstalled (twice) seemed to exhibit samba and 
dovecot communications problems with some clients (but it was 
complicated by problems we later discovered on those) after the first 
reinstall. The second time, we changed the system disk from IDE to SATA 
(making it exclusively SATA) and it cleared most of the client 
communication issues.

We reinstalled serverM on SATA, similarly the communications problems 
disappeared.

The working back up server is all SATA but the "broken" one isn't; it's 
IDE  for the system, and software RAIDed SATA for the data. We've 
ordered another PCI SATA card to change it.

Has anyone else seen this?

Regards

Clive



-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web