Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #247901 > unrolled thread

wtf just happened to my local staging web server

Back to article view | Back to linux.debian.user

Contents

  wtf just happened to my local staging web server Gary Dale <gary@extremeground.com> - 2022-05-04 19:20 +0200
    Re: wtf just happened to my local staging web server Greg Wooledge <greg@wooledge.org> - 2022-05-04 19:30 +0200
      Re: wtf just happened to my local staging web server Gary Dale <gary@extremeground.com> - 2022-05-04 20:00 +0200
    Re: wtf just happened to my local staging web server Brian <ad44@cityscape.co.uk> - 2022-05-04 20:40 +0200
      Re: wtf just happened to my local staging web server Greg Wooledge <greg@wooledge.org> - 2022-05-04 20:50 +0200
        Re: wtf just happened to my local staging web server Brian <ad44@cityscape.co.uk> - 2022-05-04 21:00 +0200
        Re: wtf just happened to my local staging web server Klaus Singvogel <deb-user-ml@singvogel.net> - 2022-05-05 09:50 +0200
          Re: wtf just happened to my local staging web server Stephan Seitz <stse+debian@rootsland.net> - 2022-05-05 10:00 +0200
            Re: wtf just happened to my local staging web server Gary Dale <gary@extremeground.com> - 2022-05-07 06:20 +0200
            Re: wtf just happened to my local staging web server Richard Hector <richard@walnut.gen.nz> - 2022-05-12 04:30 +0200
    Re: wtf just happened to my local staging web server Claudio Kuenzler <ck@claudiokuenzler.com> - 2022-05-05 08:20 +0200
    Re: wtf just happened to my local staging web server Erwan David <erwan@rail.eu.org> - 2022-05-05 08:40 +0200
      Re: wtf just happened to my local staging web server Greg Wooledge <greg@wooledge.org> - 2022-05-05 13:10 +0200
      Re: wtf just happened to my local staging web server Gary Dale <gary@extremeground.com> - 2022-05-07 18:50 +0200

#247901 — wtf just happened to my local staging web server

[Multipart message — attachments visible in raw view] — view raw

My Apache2 file/print/web server is running Bullseye. I had to restart 
it yesterday evening to replace a disk drive. Otherwise the last reboot 
was a couple of weeks ago - I recall some updates to Jitsi - but I don't 
think there were any updates since then.

Today I find that I can't get through to any of the sites on the server. 
Instead I get the Apache2 default web page. This happens with both 
Firefox and Chromium. This happens for all the staging sites (that I 
access as ".loc" through entries in my hosts file). My jitsi and 
nextcloud servers simply report failure to get to the server.

I verified that the site files (-available and -enabled) haven't changed 
in months.

I tried restarting the apache2 service and got an error so I tried 
stopping it then starting it again - same error:

root@TheLibrarian:~# service apache2 start
Job for apache2.service failed because the control process exited with 
error code.
See "systemctl status apache2.service" and "journalctl -xe" for details.
root@TheLibrarian:~# systemctl status apache2.service
●apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; 
vendor preset: enabled)
     Active: failed(Result: exit-code) since Wed 2022-05-04 12:16:55 
EDT; 5s ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 7932 ExecStart=/usr/sbin/apachectl start (code=exited, 
status=1/FAILURE)
        CPU: 29ms

May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP Server...
May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in 
use: AH00072: make_sock: could not bind to addre>
May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in 
use: AH00072: make_sock: could not bind to addre>
May 04 12:16:55 TheLibrarian apachectl[7935]: no listening sockets 
available, shutting down
May 04 12:16:55 TheLibrarian apachectl[7935]: AH00015: Unable to open logs
May 04 12:16:55 TheLibrarian apachectl[7932]: Action 'start' failed.
May 04 12:16:55 TheLibrarian apachectl[7932]: The Apache error log may 
have more information.
May 04 12:16:55 TheLibrarian systemd[1]: apache2.service: Control 
process exited, code=exited, status=1/FAILURE
May 04 12:16:55 TheLibrarian systemd[1]: apache2.service: Failed with 
result 'exit-code'.
May 04 12:16:55 TheLibrarian systemd[1]: Failed to start The Apache HTTP 
Server.

also

root@TheLibrarian:/var/log# journalctl -xe
░░The job identifier is 4527.
May 04 12:50:49 TheLibrarian apachectl[8232]: (98)Address already in 
use: AH00072: make_sock: could not bind to addre>
May 04 12:50:49 TheLibrarian apachectl[8232]: (98)Address already in 
use: AH00072: make_sock: could not bind to addre>
May 04 12:50:49 TheLibrarian apachectl[8232]: no listening sockets 
available, shutting down
May 04 12:50:49 TheLibrarian apachectl[8232]: AH00015: Unable to open logs
May 04 12:50:49 TheLibrarian apachectl[8229]: Action 'start' failed.
May 04 12:50:49 TheLibrarian apachectl[8229]: The Apache error log may 
have more information.
May 04 12:50:49 TheLibrarian systemd[1]: apache2.service: Control 
process exited, code=exited, status=1/FAILURE
░░Subject: Unit process exited
░░Defined-By: systemd
░░Support: https://www.debian.org/support
░░
░░An ExecStart= process belonging to unit apache2.service has exited.
░░
░░The process' exit code is 'exited' and its exit status is 1.
May 04 12:50:49 TheLibrarian systemd[1]: apache2.service: Failed with 
result 'exit-code'.
░░Subject: Unit failed
░░Defined-By: systemd
░░Support: https://www.debian.org/support
░░
░░The unit apache2.service has entered the 'failed' state with result 
'exit-code'.
May 04 12:50:49 TheLibrarian systemd[1]: Failed to start The Apache HTTP 
Server.
░░Subject: A start job for unit apache2.service has failed
░░Defined-By: systemd
░░Support: https://www.debian.org/support
░░
░░A start job for unit apache2.service has finished with a failure.
░░
░░The job identifier is 4527 and the job result is failed.


As I said, I do get the default Apache2 page saying "It works" but that 
appears to be optimistic. ps aux | grep apache2 fails to show the 
service, which confirms the systemctl message that it isn't running.

There is nothing in /var/log/apache2/error.log. The .1 log ends 
yesterday but only contains complaints about php7. Systemctl does report 
(above) "unable to open logs" so that would explain the lack of 
additional messages.  The apache2 directory and its files are root:adm 
with only root having write privileges.

I tried giving the adm group write privileges but that didn't work. 
Turns out the group is empty. Adding www-data to it didn't work either.

Any ideas on how to track down the cause of the failure(s)?

Thanks.

[toc] | [next] | [standalone]

#247902

On Wed, May 04, 2022 at 01:01:58PM -0400, Gary Dale wrote:
> May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP Server...
> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in use:
> AH00072: make_sock: could not bind to addre>
> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in use:
> AH00072: make_sock: could not bind to addre>

Something else is using the ports that Apache wants to use.

Assuming those ports are 80 and 443, you could use commands like this
to see what's using them:

lsof -i :80
lsof -i :443

If your configuration is telling Apache to use some other ports, then
substitute your port numbers.

[toc] | [prev] | [next] | [standalone]

#247903

On 2022-05-04 13:21, Greg Wooledge wrote:
> On Wed, May 04, 2022 at 01:01:58PM -0400, Gary Dale wrote:
>> May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP Server...
>> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in use:
>> AH00072: make_sock: could not bind to addre>
>> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in use:
>> AH00072: make_sock: could not bind to addre>
> Something else is using the ports that Apache wants to use.
>
> Assuming those ports are 80 and 443, you could use commands like this
> to see what's using them:
>
> lsof -i :80
> lsof -i :443
>
> If your configuration is telling Apache to use some other ports, then
> substitute your port numbers.
>
Thanks. Somehow nginx got installed. Wondering if jitsi or nextcloud did 
that because I certainly didn't (doesn't seem likely though because they 
both failed).

I guess I should pay more attention to the packages that get installed 
when I do apt full-upgrade... Usually I just scan to see if there is 
anything that I should reboot over.

[toc] | [prev] | [next] | [standalone]

#247904

On Wed 04 May 2022 at 13:01:58 -0400, Gary Dale wrote:

[...]

My young childre read -user. They asked me what "wtf" means. Please,
explain, for the benefit of us civilised and acronymn-challenged
users, what it stands for.

For extra points, knowing whether it was an essential part of your
query would be interesting.

-- 
Brian.

[toc] | [prev] | [next] | [standalone]

#247905

On Wed, May 04, 2022 at 07:38:35PM +0100, Brian wrote:
> My young childre read -user. They asked me what "wtf" means. Please,
> explain, for the benefit of us civilised and acronymn-challenged
> users, what it stands for.

Clearly just a really bad typo for "what".

[toc] | [prev] | [next] | [standalone]

#247906

On Wed 04 May 2022 at 14:42:15 -0400, Greg Wooledge wrote:

> On Wed, May 04, 2022 at 07:38:35PM +0100, Brian wrote:
> > My young childre read -user. They asked me what "wtf" means. Please,
> > explain, for the benefit of us civilised and acronymn-challenged
> > users, what it stands for.
> 
> Clearly just a really bad typo for "what".

Clearly. Training the brain and fingers to avoid such typos is
indictated.

-- 
Brian.

[toc] | [prev] | [next] | [standalone]

#247915

Greg Wooledge wrote:
> On Wed, May 04, 2022 at 07:38:35PM +0100, Brian wrote:
> > My young childre read -user. They asked me what "wtf" means. Please,
> > explain, for the benefit of us civilised and acronymn-challenged
> > users, what it stands for.
> 
> Clearly just a really bad typo for "what".

I learned, it is usually an acronym for "well that's funny,"

Sometimes its also "what the freak", "what the failure", "world trade
federation", or "world taekwondo federation".

I think there are more.

Best regards,
	Klaus.
-- 
Klaus Singvogel
GnuPG-Key-ID: 1024R/5068792D  1994-06-27

[toc] | [prev] | [next] | [standalone]

#247916

Am Do, Mai 05, 2022 at 09:30:42 +0200 schrieb Klaus Singvogel:
>I think there are more.

Yes, I only know wtf as „what the fuck”.

	Stephan

-- 
|    If your life was a horse, you'd have to shoot it.    |

[toc] | [prev] | [next] | [standalone]

#247966

On 2022-05-05 03:57, Stephan Seitz wrote:
> Am Do, Mai 05, 2022 at 09:30:42 +0200 schrieb Klaus Singvogel:
>> I think there are more.
>
> Yes, I only know wtf as „what the fuck”.
>
>     Stephan
>
Actually, it's "what the frack" - a nod to the Battlestar Galactica 
TV/movie franchise, which uses frack as the expletive of choice.

These days "frack" also refers to a gas extraction process with terrible 
environmental consequences, thereby justifying its use as an expletive 
in the broader world. Fracking is derived from fracturing, the breaking 
of something, which is appropriate in the case of my staging server 
suddenly being broken.

[toc] | [prev] | [next] | [standalone]

#248115

On 5/05/22 19:57, Stephan Seitz wrote:
> Am Do, Mai 05, 2022 at 09:30:42 +0200 schrieb Klaus Singvogel:
>> I think there are more.
> 
> Yes, I only know wtf as ...

Yes, but such language is not permitted on this list.

Richard

[toc] | [prev] | [next] | [standalone]

#247913

[Multipart message — attachments visible in raw view] — view raw

On Wed, May 4, 2022 at 7:18 PM Gary Dale <gary@extremeground.com> wrote:

> May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP
> Server...
> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in use:
> AH00072: make_sock: could not bind to addre>
> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in use:
> AH00072: make_sock: could not bind to addre>
> May 04 12:16:55 TheLibrarian apachectl[7935]: no listening sockets
> available, shutting down
> May 04 12:16:55 TheLibrarian apachectl[7935]: AH00015: Unable to open logs
> May 04 12:16:55 TheLibrarian apachectl[7932]: Action 'start' failed.
> May 04 12:16:55 TheLibrarian apachectl[7932]: The Apache error log may
> have more information.
> May 04 12:16:55 TheLibrarian systemd[1]: apache2.service: Control process
> exited, code=exited, status=1/FAILURE
> May 04 12:16:55 TheLibrarian systemd[1]: apache2.service: Failed with
> result 'exit-code'.
> May 04 12:16:55 TheLibrarian systemd[1]: Failed to start The Apache HTTP
> Server.
>
The errors show that Apache was unable to bind to the listener port
(Address already in use).

Check for other services (maybe Nginx?) which are listening on the same
port as Apache tries to bind to.
Run: netstat -lntup

Also check /etc/apache2/ports.conf for possible misconfigurations.

Are you using HTTP (Port 80) only or also HTTPS (Port 443)?

Just to rule a config error out, run "apache2ctl configtest".

As I said, I do get the default Apache2 page saying "It works" but that
> appears to be optimistic. ps aux | grep apache2 fails to show the service,
> which confirms the systemctl message that it isn't running.
>
That could be your browser cache tricking you. You can verify with "curl
localhost".

[toc] | [prev] | [next] | [standalone]

#247914

Le 04/05/2022 à 19:01, Gary Dale a écrit :
> My Apache2 file/print/web server is running Bullseye. I had to restart 
> it yesterday evening to replace a disk drive. Otherwise the last reboot 
> was a couple of weeks ago - I recall some updates to Jitsi - but I don't 
> think there were any updates since then.
> 
> Today I find that I can't get through to any of the sites on the server. 
> Instead I get the Apache2 default web page. This happens with both 
> Firefox and Chromium. This happens for all the staging sites (that I 
> access as ".loc" through entries in my hosts file). My jitsi and 
> nextcloud servers simply report failure to get to the server.
> 
> I verified that the site files (-available and -enabled) haven't changed 
> in months.
> 
> I tried restarting the apache2 service and got an error so I tried 
> stopping it then starting it again - same error:
> 
> root@TheLibrarian:~# service apache2 start

It looks like you started it, not restart, thus the running apache is 
not killed

[...]

> 
> May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP Server...
> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in 
> use: AH00072: make_sock: could not bind to addre>
> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in 
> use: AH00072: make_sock: could not bind to addre>

This is consistent with former apache still running at that time, and 
using the wanted ports.

[toc] | [prev] | [next] | [standalone]

#247917

On Thu, May 05, 2022 at 08:37:04AM +0200, Erwan David wrote:
> > root@TheLibrarian:~# service apache2 start
> 
> It looks like you started it, not restart, thus the running apache is not
> killed
> 
> [...]
> 
> > 
> > May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP Server...
> > May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in
> > use: AH00072: make_sock: could not bind to addre>
> > May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in
> > use: AH00072: make_sock: could not bind to addre>
> 
> This is consistent with former apache still running at that time, and using
> the wanted ports.

Except that systemd is more clever than that.  If a service is already
running, and you issue a "start" for it, it'll do nothing.

unicorn:~$ systemctl status mariadb
● mariadb.service - MariaDB 10.5.15 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor prese>
     Active: active (running) since Tue 2022-05-03 07:16:46 EDT; 1 day 23h ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 683 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/>
    Process: 691 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_STA>
    Process: 694 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && V>
    Process: 769 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_ST>
    Process: 771 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/S>
   Main PID: 749 (mariadbd)
     Status: "Taking your SQL requests now..."
      Tasks: 8 (limit: 14199)
     Memory: 118.7M
        CPU: 26.273s
     CGroup: /system.slice/mariadb.service
             └─749 /usr/sbin/mariadbd

Warning: some journal files were not opened due to insufficient permissions.
unicorn:~$ sudo systemctl start mariadb
[sudo] password for greg: 
unicorn:~$ systemctl status mariadb
● mariadb.service - MariaDB 10.5.15 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor prese>
     Active: active (running) since Tue 2022-05-03 07:16:46 EDT; 1 day 23h ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 683 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/>
    Process: 691 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_STA>
    Process: 694 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && V>
    Process: 769 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_ST>
    Process: 771 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/S>
   Main PID: 749 (mariadbd)
     Status: "Taking your SQL requests now..."
      Tasks: 8 (limit: 14199)
     Memory: 118.7M
        CPU: 26.283s
     CGroup: /system.slice/mariadb.service
             └─749 /usr/sbin/mariadbd

Warning: some journal files were not opened due to insufficient permissions.
unicorn:~$ sudo journalctl -u mariadb | tail -n5
May 03 07:16:46 unicorn mariadbd[749]: 2022-05-03  7:16:46 0 [Note] Added new Master_info '' to hash table
May 03 07:16:46 unicorn mariadbd[749]: 2022-05-03  7:16:46 0 [Note] /usr/sbin/mariadbd: ready for connections.
May 03 07:16:46 unicorn mariadbd[749]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
May 03 07:16:46 unicorn systemd[1]: Started MariaDB 10.5.15 database server.
May 03 07:16:47 unicorn mariadbd[749]: 2022-05-03  7:16:47 0 [Note] InnoDB: Buffer pool(s) load completed at 220503  7:16:47


Note that this "Started MariaDB ..." happened 2 days ago, consistent with
the "1 day 23h ago" from systemctl status.  The start command that I issued
had no effect at all.  It didn't even add a line to the journal.

[toc] | [prev] | [next] | [standalone]

#247971

On 2022-05-05 02:37, Erwan David wrote:
> Le 04/05/2022 à 19:01, Gary Dale a écrit :
>> My Apache2 file/print/web server is running Bullseye. I had to 
>> restart it yesterday evening to replace a disk drive. Otherwise the 
>> last reboot was a couple of weeks ago - I recall some updates to 
>> Jitsi - but I don't think there were any updates since then.
>>
>> Today I find that I can't get through to any of the sites on the 
>> server. Instead I get the Apache2 default web page. This happens with 
>> both Firefox and Chromium. This happens for all the staging sites 
>> (that I access as ".loc" through entries in my hosts file). My jitsi 
>> and nextcloud servers simply report failure to get to the server.
>>
>> I verified that the site files (-available and -enabled) haven't 
>> changed in months.
>>
>> I tried restarting the apache2 service and got an error so I tried 
>> stopping it then starting it again - same error:
>>
>> root@TheLibrarian:~# service apache2 start
>
> It looks like you started it, not restart, thus the running apache is 
> not killed
>
> [...]
>
>>
>> May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP 
>> Server...
>> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in 
>> use: AH00072: make_sock: could not bind to addre>
>> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in 
>> use: AH00072: make_sock: could not bind to addre>
>
> This is consistent with former apache still running at that time, and 
> using the wanted ports.

If you read my original e-mail, I tried both restarting it and starting 
it. Also, in my original e-mail, I identified that Apache2 wasn't 
running by running ps aux output through grep. Again, this confirms the 
systemctl message - as Greg Wooledge mentions in his reply to you.

Greg Wooledge showed me how to diagnose the problem by identifying the 
process (nginx in this case) that was grabbing the ports Apache2 needed. 
Claudio Kuenzler also provided an alternative method of diagnosing the 
problem.

My problem is I'm not all that conversant in tracking down network 
issues, such as ports. I didn't know that lsof even had a port option. 
And I'm still getting used to systemctl / journalctl.

Anyway, thanks for your attempt to help.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web