Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #200997 > unrolled thread

systemd can't start a dæmon and doesn't give any error either

Back to article view | Back to linux.debian.user

Contents

  systemd can't start a dæmon and doesn't give any error either Weiwu Zhang <a@colourful.land> - 2018-10-08 12:40 +0200
    Re: systemd can't start a dæmon  and doesn't give any error either Dan Ritter <dsr@randomstring.org> - 2018-10-08 15:10 +0200
      Re: systemd can't start a dæmon and doesn't give any error either Weiwu Zhang <a@colourful.land> - 2018-10-08 15:20 +0200
    Re: systemd can't start a dæmon  and doesn't give any error either Greg Wooledge <wooledg@eeg.ccf.org> - 2018-10-08 15:20 +0200
      Re: systemd can't start a  dæmon and doesn't give any error either "AvY" <avy@avyenterprises.com> - 2018-10-08 16:40 +0200
        Re: systemd can't start a dæmon and doesn't give any error either Weiwu Zhang <a@colourful.land> - 2018-10-08 23:10 +0200
      [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either) Didar Hossain <didar.hossain@gmail.com> - 2018-10-09 09:10 +0200
        Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and  doesn't give any error either) <tomas@tuxteam.de> - 2018-10-09 11:30 +0200
          Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either) Jonathan Dowland <jmtd@debian.org> - 2018-10-09 12:00 +0200
            Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and  doesn't give any error either) <tomas@tuxteam.de> - 2018-10-09 12:30 +0200
          Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either) Greg Wooledge <wooledg@eeg.ccf.org> - 2018-10-09 15:10 +0200
            Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and  doesn't give any error either) <tomas@tuxteam.de> - 2018-10-09 18:20 +0200
              Re: Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either) Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM> - 2018-11-17 22:00 +0100
                Re: Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either) Roberto C. Sánchez <roberto@debian.org> - 2018-11-17 22:30 +0100
        Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either) Jonathan Dowland <jmtd@debian.org> - 2018-10-09 11:50 +0200
          Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either) Greg Wooledge <wooledg@eeg.ccf.org> - 2018-10-09 15:10 +0200

#200997 — systemd can't start a dæmon and doesn't give any error either

To show this dæmon itself is properly configured and runs fine on Debian 9.5:

# su bitcoin
$ cd
$ /usr/local/bin/bitcoind -daemon
Bitcoin server starting
$ pgrep bitcoind
3946

However, it won't start if converted to a systemd service:

# cat /etc/systemd/system/bitcoin.service
[Unit]
Description=bitcoind
After=syslog.target

[Service]
User=bitcoin
ExecStart=/usr/local/bin/bitcoind -daemon

[Install]
WantedBy=multi-user.target

# systemctl enable bitcoin.service
# systemctl start bitcoin.service
# pgrep bitcoind || echo "not working"
not working

There were no error messages:

# journalctl -u bitcoin.service
-- Logs begin at Mon 2018-10-08 03:53:55 BST, end at Mon 2018-10-08 11:14:47 BST
Oct 08 11:08:54 lightning systemd[1]: Started bitcoind.

There weren't error messages even if I add these in
/etc/systemd/system/bitcoin.service

StandardOutput=syslog+console
StandardError=syslog+console

I assume this question is best addressed in Debian list because
normally bitcoind would through some error message. No error message
could mean the whole command isn't executed.

Thanks!

[toc] | [next] | [standalone]

#201013 — Re: systemd can't start a dæmon and doesn't give any error either

On Mon, Oct 08, 2018 at 06:17:52PM +0800, Weiwu Zhang wrote:
> To show this dæmon itself is properly configured and runs fine on Debian 9.5:
> 
> # su bitcoin
> $ cd
> $ /usr/local/bin/bitcoind -daemon
> Bitcoin server starting
> $ pgrep bitcoind
> 3946
> 
> However, it won't start if converted to a systemd service:
> 
> # cat /etc/systemd/system/bitcoin.service
> [Unit]
> Description=bitcoind
> After=syslog.target
> 
> [Service]
> User=bitcoin
> ExecStart=/usr/local/bin/bitcoind -daemon

Remove the -daemon flag here and try again.

Supervision services like systemd usually prefer daemons to be running
in the foreground.


-dsr-

[toc] | [prev] | [next] | [standalone]

#201015

El lun., 8 oct. 2018 a las 20:44, Dan Ritter (<dsr@randomstring.org>) escribió:
> Remove the -daemon flag here and try again.
>
> Supervision services like systemd usually prefer daemons to be running
> in the foreground.

Thanks for the advice!

The problem seems to be elsewhere:

# grep bitcoind /etc/systemd/system/bitcoin.service
Description=bitcoind
ExecStart=/usr/local/bin/bitcoind -printtoconsole
# systemctl start bitcoin.service
# pgrep bitcoin || echo failed
failed

# journalctl -u bitcoin.service
-- Logs begin at Mon 2018-10-08 03:53:55 BST, end at Mon 2018-10-08 13:53:12 BST
Oct 08 13:51:57 lightning systemd[1]: Started bitcoind.
Oct 08 13:52:03 lightning systemd[1]: Started bitcoind.

Now, since normally bitcoind print a line like this:

$ /usr/local/bin/bitcoind
Bitcoin server starting

If systemd is honestly recording the output some line like this should
be found in the syslog since I configured

StandardOutput=syslog+console
StandardError=syslog+console

But syslog simply mentioned this and nothing more:

Oct  8 13:58:52 lightning systemd[1]: Started bitcoind.

[toc] | [prev] | [next] | [standalone]

#201014 — Re: systemd can't start a dæmon and doesn't give any error either

On Mon, Oct 08, 2018 at 06:17:52PM +0800, Weiwu Zhang wrote:
> To show this dæmon itself is properly configured and runs fine on Debian 9.5:
> 
> # su bitcoin
> $ cd
> $ /usr/local/bin/bitcoind -daemon
> Bitcoin server starting
> $ pgrep bitcoind
> 3946

You've used an option named "-daemon".  I am guessing that this causes
the process to "daemonize" itself (auto-background, self-background).

This is what you want if you're on a BSD system in 1989.

This is NOT what you want if you're on a systemd system in 2018.

If you omit the -daemon option, does the process run in the foreground?
If so, THAT is what you want.

> However, it won't start if converted to a systemd service:
> 
> # cat /etc/systemd/system/bitcoin.service
> [Unit]
> Description=bitcoind
> After=syslog.target
> 
> [Service]
> User=bitcoin
> ExecStart=/usr/local/bin/bitcoind -daemon
> 
> [Install]
> WantedBy=multi-user.target

You did not specify what TYPE of service this is.  Since you didn't specify
a Type= option, systemd uses the default, which is "simple", which means
that it expects a single process to run in the foreground.

So: either omit that "-daemon" option, or whatever it takes to make the
process run in the foreground like a good process should.

-OR-

Tell systemd that your process uses ancient, broken behavior
(self-backgrounding) by specifying "Type=forking".

I know which one I would pick.  Assuming that dropping -daemon does what
I am guessing it does.  (If my guess is wrong, then you may be forced
to use a non-simple Type= option.)

[toc] | [prev] | [next] | [standalone]

#201017 — Re: systemd can't start a dæmon and doesn't give any error either

> So: either omit that "-daemon" option, or whatever it takes to make the
> process run in the foreground like a good process should.

bitcoind defaults to running as a daemon so '--daemon=0' would be needed
here to put it in the foreground

[toc] | [prev] | [next] | [standalone]

#201055

El lun., 8 oct. 2018 a las 22:30, AvY (<avy@avyenterprises.com>) escribió:
>
> > So: either omit that "-daemon" option, or whatever it takes to make the
> > process run in the foreground like a good process should.
>
> bitcoind defaults to running as a daemon so '--daemon=0' would be needed
> here to put it in the foreground

Thank you. I was under the wrong assumption that systemd tracks the
pid so whether or not it forks does not matter. Also thank Greg
Wooledge and Dan Ritter for the correct deductions!

Regards

[toc] | [prev] | [next] | [standalone]

#201065 — [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

** Apologies for hijacking the thread, I really wanted to start a new one
   but I need the perspective of Greg on this **

On Mon, Oct 08, 2018 at 09:02:18AM -0400, Greg Wooledge wrote:
> On Mon, Oct 08, 2018 at 06:17:52PM +0800, Weiwu Zhang wrote:
> 
> You've used an option named "-daemon".  I am guessing that this causes
> the process to "daemonize" itself (auto-background, self-background).
> 
> This is what you want if you're on a BSD system in 1989.
> 
> This is NOT what you want if you're on a systemd system in 2018.

I am interested in understanding the above points. I am not a hater and do not
intend to start a flame war here about systemd. I am an old dog who is trying to
understand the new way of doing things.


> You did not specify what TYPE of service this is.  Since you didn't specify
> a Type= option, systemd uses the default, which is "simple", which means
> that it expects a single process to run in the foreground.
> 
> So: either omit that "-daemon" option, or whatever it takes to make the
> process run in the foreground like a good process should.
> 
> -OR-
> 
> Tell systemd that your process uses ancient, broken behavior
> (self-backgrounding) by specifying "Type=forking".

> I know which one I would pick.  Assuming that dropping -daemon does what

Please explain as to why "forking" is "broken" behaviour so that I can make
informed decisions whether I should use forking or not in my own scripts.

Again this is not a flame bait, I just want to understand Pros and Cons.

Regards,
Didar

-- 
<Sammy> that's *IT*.  I'm never fucking attempting to install redhat
        again.
<Sammy> this is like the 10th fucking machine on which the installer has
        imploded immediately after I went through the hell of their
	package selection process.
<timball> Sammy: just use debian and never look back
<Sammy> timball: debian iso's are being written at this very moment.

[toc] | [prev] | [next] | [standalone]

#201070 — Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[Multipart message — attachments visible in raw view] — view raw

On Tue, Oct 09, 2018 at 12:13:31PM +0530, Didar Hossain wrote:

[...]

> Please explain as to why "forking" is "broken" behaviour so that I can make
> informed decisions whether I should use forking or not in my own scripts.

Systemd's stance is that it wants to supervise the processes it creates. And
the most effective way to do that is to keep them as children; that's what
old BSD init did, and what other process supervision programs (e.g. runit)
do. They can watch closely when one of their children die and take action.

The whole PID thing came up with SysV init and is a crutch: many things
can happen: process dies, leaving behind the PID file, so you have to
check explicitly whether process is still running (in itself a race
condition), PIDs "wrap around", so if process died a while ago, the
PID may be active again for a totally unrelated process, etc. etc.

Not pretty.

Now systemd is very opinionated [1] software, and "there is just one
way to do it", so the daemons serviced by it have to behave in this
one way (stay in the foreground).

> Again this is not a flame bait, I just want to understand Pros and Cons.

No worries: flame baits tend to have a strong smell :-)

Cheers
[1] That's why some of us don't like systemd: personally, I think it's
   OK for people to have (sometimes strong) opinions, but I don't
   really like software to have opinions, especially if they try to
   override mine.

-- tomás

[toc] | [prev] | [next] | [standalone]

#201072 — Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

On Tue, Oct 09, 2018 at 11:04:21AM +0200, tomas@tuxteam.de wrote:
>Now systemd is very opinionated [1] software, and "there is just one
>way to do it", so the daemons serviced by it have to behave in this
>one way (stay in the foreground).

It certainly is opinionated, but daemons serviced by it do not *have* to
behave in this one way: the version on my system (232-25+deb9u4)
supports "simple" (this no-fork mode that we are discussing), "forking",
"oneshot", "dbus", "notify" and "idle" (via man systemd.service(5)).


-- 

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄⠀⠀⠀⠀ Please do not CC me, I am subscribed to the list.

[toc] | [prev] | [next] | [standalone]

#201074 — Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[Multipart message — attachments visible in raw view] — view raw

On Tue, Oct 09, 2018 at 10:34:23AM +0100, Jonathan Dowland wrote:
> On Tue, Oct 09, 2018 at 11:04:21AM +0200, tomas@tuxteam.de wrote:
> >Now systemd is very opinionated [1] software, and "there is just one
> >way to do it", so the daemons serviced by it have to behave in this
> >one way (stay in the foreground).
> 
> It certainly is opinionated, but daemons serviced by it do not *have* to
> behave in this one way: the version on my system (232-25+deb9u4)
> supports "simple" (this no-fork mode that we are discussing), "forking",
> "oneshot", "dbus", "notify" and "idle" (via man systemd.service(5)).

Thanks for the clarification

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#201080 — Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

On Tue, Oct 09, 2018 at 11:04:21AM +0200, tomas@tuxteam.de wrote:
> Systemd's stance is that it wants to supervise the processes it creates. And
> the most effective way to do that is to keep them as children; that's what
> old BSD init did, and what other process supervision programs (e.g. runit)
> do. They can watch closely when one of their children die and take action.

You're actually thinking of inittab here.  This is what System V's inittab
was intended to do.  It's a monolithic file that controls the starting
and restarting of all services on the system.  You (root) edit the file
using your favorite text editor to add or remove a service, and then
run "init q" (or "telinit q" in later Linux implementations) to have
the running init process re-read the file.

For some reason, this model never caught on.  People didn't like having
a monolithic file that had to be human-edited and which could break the
entire boot process if munged.

So, they built another layer on top of inittab.

> The whole PID thing came up with SysV init and is a crutch: many things
> can happen: process dies, leaving behind the PID file, so you have to
> check explicitly whether process is still running (in itself a race
> condition), PIDs "wrap around", so if process died a while ago, the
> PID may be active again for a totally unrelated process, etc. etc.

This is the additional layer that was added later.  It's sometimes called
"sysv-rc".

In this model, the inittab file is told to run a single script which then
runs a whole bunch of other scripts.  Each of these other scripts is
of the form /etc/init.d/foobar and takes a single argument, like "start"
or "stop".  The objective of the /etc/init.d/foobar script is to start
its designated service as some kind of background program, and capture
its PID and store it in a file, so that later, it can terminate this
process.

The storing of PIDs in files is a catastrophic hack, for the reasons
you've already explained.  The only sane way to make sure you're
controlling the actual service that you started is to stay running,
as the parent of the service.  As the parent process, you have unique
authority over the child-service.  You can be informed immediately
when it terminates, you can receive its exit status, and perhaps most
importantly of all, the child is left in the process table as a zombie
until you, the parent, have asked for its exit status.

Under the sysv-rc model, the child-service processes are orphaned.
There's no responsible parent hanging around waiting for that phone
call from the kernel telling it that Little Johnny has died.  The sysv-rc
services are wards of the state.  They become children of init (PID 1),
and init does not care about their exit status, or about notifying the
next of kin, or any of that.

When an orphaned process dies, the state (init) reaps and discards its
exit status, and the zombie is gone.  The PID is free to be reused by
some other process.  The sysv-rc shell scripts have no way of knowing
whether this has occurred.  So, people tried to layer more and more and
more hacks (Debian's start-stop-daemon is one of these) on top of the
problem.  But going from 99% working to 99.9% working is still not a
good solution.

The only 100% working solution is to keep the parent alive.  MANY people
wrote service management systems to do this.  I won't attempt to list
them all here.  Systemd is just one of them.

For a more detailed analysis of the history of sysv-rc and its failings,
see <http://jdebp.eu./FGA/system-5-rc-problems.html> among many other
pages.

[toc] | [prev] | [next] | [standalone]

#201092 — Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[Multipart message — attachments visible in raw view] — view raw

On Tue, Oct 09, 2018 at 08:47:50AM -0400, Greg Wooledge wrote:

[...]

> You're actually thinking of inittab here [...]

I know, I know. I've been around this for longer than I care.
I saw HP-UX's transition from one to another (9 -> 10). I've
had GNU/Linux distros spawning PostgreSQL off inittab (bad
idea to put respawn there without a sleep...)

Inittab was the original way of doing things for AT&T, BSD
copied that, and SysV grafted /etc/init.d onto it. Slowly
other unices followed (alas!).

I think inittab's ailments could have been cured differently,
but history is like that.

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#202573 — Re: Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

tomas@tuxteam.de:

> Inittab was the original way of doing things for AT&T, BSD copied 
> that, and SysV grafted /etc/init.d onto it. Slowly other unices 
> followed (alas!).
>

Some history seems in order.

There wasn't really an original way of doing things, as in First Edition 
Unix this stuff, including starting up mel's da program, was simply 
hard-coded into the init program itself.  One can enjoy seeing 
/usr/mel/da and the rest in the resurrected 1st Edition source.

By the time of Seventh Edition and 4BSD, init was running /etc/rc and 
terminals were listed in /etc/ttys.  The BSDs stuck with this for 2 to 3 
decades more (depending from the BSD).  AT&T on the other hand changed 
to an inittab system with run-levels in System 3 at the start of the 1980s.

In the AT&T world, inittab was soon augmented and superseded, within a 
decade.  By the time of System 5 Release 4 in 1988 handling of terminal 
login had been moved out of inittab to the Service Access Facility, 
specifically to ttymon.  IBM AIX likewise gained the System Resource 
Controller in version 3.1 two years later, and reduced the number of 
run-levels actually used to just 1 (everything in inittab being marked 
as run level "2").  This left comparatively very little in inittab on 
those systems and in practice did away with the run-level system on Unix 
in the AT&T side of the universe.

By 1990.

Yes, run-levels and a lot of /etc/inittab were obsolete before Linux was 
even invented.

* http://jdebp.eu./FGA/run-levels-are-history.html

Two years after that Miquel van Smoorenburg cloned the old System 5 
init+rc system for Minix, but did not clone the more recent Service 
Access Facility or anything like the SRC; meaning that terminal login 
was back to the old inittab way of operating, inittab was back to being 
full of stuff, and run-levels were back.  Most Linux operating systems 
adopted it.  van Smoorenburg init had several different rcs to accompany 
it.  By the late 1990s there were so-called sysv-rc, the van Smoorenburg 
one, and file-rc developed from Winfried Truemper's r2d2.  van 
Smoorenburg init itself did not stay static, splitting the old single 
user mode into emergency and rescue modes in 1995, although it took 
about 5 years for the names to fully catch on.

* http://jdebp.eu./FGA/emergency-and-rescue-mode-bootstrap.html

In the BSD world, things were quite turbulent as well, as the 
edit-this-single-monolithic-shell-script system was proving troublesome 
in real world use.  One paper from the 1991 LISA conference published by 
USENIX talked about how /etc/rc.local had accrued all of the very same 
problems of /etc/rc that had caused /etc/rc.local to be split out in the 
first place.  FreeBSD re-engineered /etc/rc.local into an 
/etc/rc.local.d/ system with drop-in files in 1995,  and outright got 
rid of /etc/rc.local in 1998.  (The commit that deleted it was made by 
one Matthew Dillon, after encouragement from Jordan Hubbard.)  There was 
a big discussion at the end of the decade, which culminated in Mewburn 
rc replacing the old /etc/rc in NetBSD 1.5 in 2000, and in FreeBSD in 
2002.  Mewburn rc did not have run levels, and learned from experience 
of van Smoorenburg rc, organizing things so that rc scripts were (as 
intended and often, but alas far from always) a few lines of variable 
assignments and just two commands.  Gentoo's OpenRC went down the same 
route years later in 2007.  Debian's variant of van Smoorenburg rc 
started going down the same route almost as long again after that, in 2014.

OpenBSD was the lone holdout, hanging on to the old 4BSD way until 2011 
when it switched to a system similar to, but not the same as, Mewburn rc.

* http://jdebp.eu./FGA/rc.local-is-history.html

In the Linux world, there were then simpleinit, jinit, minit (once 
packaged and part of Debian), runit-init (still packaged and part of 
Debian) depinit, GNU dmd, pinit, initng, eINIT, upstart, finit, systemd, 
nosh system-manager, s6-rc, and Epoch.  Over the same period, Solaris 
switched to the SMF and MacOS switched to launchd.

* http://jdebp.eu./FGA/inittab-is-history.html

* https://blog.darknedgy.net/technology/2015/09/05/0/

[toc] | [prev] | [next] | [standalone]

#202574 — Re: Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

On Sat, Nov 17, 2018 at 08:27:45PM +0000, Jonathan de Boyne Pollard wrote:
> tomas@tuxteam.de:
> 
> > Inittab was the original way of doing things for AT&T, BSD copied that,
> > and SysV grafted /etc/init.d onto it. Slowly other unices followed
> > (alas!).
> > 
> 
> Some history seems in order.
> 
I love reading this stuff :-)

-- 
Roberto C. Sánchez

[toc] | [prev] | [next] | [standalone]

#201071 — Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

On Tue, Oct 09, 2018 at 12:13:31PM +0530, Didar Hossain wrote:
>Please explain as to why "forking" is "broken" behaviour so that I can make
>informed decisions whether I should use forking or not in my own scripts.
>
>Again this is not a flame bait, I just want to understand Pros and Cons.

If you a program that doesn't fork, but you want to run it in the
background, you can: either by invoking it with '&' in an interactive
shell, or suspending it in an interactive shell and issuing the "bg"
command. There are similar ways in non-interactive environments, and
a myriad other ways (execute via cron, or at, etc.)

If you have a program that does fork, but you don't want it to, there's
nothing you can do to stop it.

Modern init systems (like systemd) can manage programs that do not fork,
so there's no need to do when being used in conjunction with such
systems.

So not forking gives the user more control. It's also the simpler
implementation, and not necessary with modern init systems.



-- 

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄⠀⠀⠀⠀ Please do not CC me, I am subscribed to the list.

[toc] | [prev] | [next] | [standalone]

#201081 — Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

On Tue, Oct 09, 2018 at 10:31:41AM +0100, Jonathan Dowland wrote:
> If you have a program that does fork, but you don't want it to, there's
> nothing you can do to stop it.

There's DJB's fghack program, but as the name implies, it's a hack, and
not guaranteed to work.

https://cr.yp.to/daemontools/fghack.html

Fortunately, upstream maintainers of daemons and other services have
come to understand that self-backgrounding is no longer considered
appropriate mandatory behavior for a daemon.  All the recent releases
of major daemons have options to STOP self-backgrounding, or they no
longer do it by default, and offer an option to do it if desired.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web