Respawning a dead daemon

To stream files to my XBox (yes, I admit I have an XBox 360 console) I am using uShare, an outstanding piece of software that is actually compatible with Microsoft’s horrible DLNA specification. (That’s also the reason why I am so passionate about 973295 and 880076)

The default startup script simply launches the daemon as root and does nothing to keep it alive. As I found out one can make uShare exit by sending too much requests to reload the contents via its web server (this resulted from a bug in my inotify monitoring script, though). How do we ask upstart to restart the daemon?

# uShare UPnP A/V & DLNA Media Server

description     "uShare UPnP A/V & DLNA Media Server"
author          "Roman Yepishev <rtg@rtg.in.ua>"

start on filesystem
stop on runlevel [016]

respawn

setuid nobody
setgid nogroup

exec /usr/bin/ushare

Original script reads $USHARE_OPTIONS from /etc/default/ushare which does not actually exist so I left this out.

What we have above is a simple script that:

1. Starts uShare when filesystem is available.

2. Stops uShare when system is rebooted or shut down.

3. Respawns the process if it exits.

4. Makes the process run as nobody:nogroup because I don’t want it to run as root.

$ sudo start ushare
ushare start/running, process 5892
$ sudo kill 5892
$ sudo status ushare
ushare start/running, process 7800

Yep, it works as expected. respawn stanza (that’s how these directives are called in Upstart) by default have a respawn limit 10 5 – 10 respawns per 5 seconds. If the application respawns faster than that (basically meaning that it cannot start successfully), the init will say

init: ushare respawning too fast, stopped

While respawn limit can be made unlimited such rate of failure signals about a real problem.

Please note that in order to control the respawn limit, you need to provide both respawn and respawn limit stanzas. respawn limit does not enable respawn:

respawn
respawn limit 10 5

Monitoring the respawn limit because we can

uShare crashing is not such a big deal but it was still upsetting my wife since it meant I had to log into the server and start the process again (before I fixed a bug in my script). But then I thought that notifying an administrator about such kind of an issue is a good thing.

# upstart-respawn-monitor.conf

description "Notify root about ushare respawn limit reached"
author "Roman Yepishev <rtg@rtg.in.ua>"

# This is a short-lived job
task

start on stopped ushare PROCESS=respawn

script
    echo "uShare reached respawn limit" | mail -s "uShared crashed" root
end script

This is another type of Upstream job, a task. It does not keep running forever and once it has finished it will not be respawned. Same thing as setting up the hostname. But what we just did is we created a simple monitoring task that will be executed once ushare reaches stopped state and a special PROCESS variable will have respawn as the reason why the service stopped. And this is all was done using Upstart only.

Before you start implementing monitoring this way, check whether there are existing solutions already.

Disable Upstart task

By default when you install a service in Ubuntu it will be immediately started. Sometimes, however, you might want to start service manually. Upstart has a concept of overrides that basically let you override any stanza in the original job file by creating a job.override file in /etc/init and placing the stanzas here.

For example, I am using LXC for some machines now and these lack tty[2..6]. Actually I don’t need these but getty keeps trying to start on these ttys. It starts and stops pretty slowly so it is not caught by the respawn limit. Disabling a task in Upstart is trivial, you just need to add manual to the stanzas:

# /etc/init/tty2.override
manual

Yes, it is that easy. Another example when you’d want to use an override file is when you want to use different options on the exec line (you will copy the original exec line and change the options then) or change the start on/stop on conditions.

Starting, stopping and restarting

If the job was converted to upstart, then interacting with it is easy:

# Getting the info about particular job:
$ sudo status cups
cups start/running, process 1503

# Getting the info about all the jobs
$ sudo initctl list
avahi-daemon start/running, process 1526
cgroup-lite start/running
mountall-net stop/waiting
nmbd start/running, process 1649
qemu-kvm start/running
...

# Stopping a job
$ sudo stop cups
cups stop/waiting

# Starting a job
$ sudo start cups
cups start/running, process 7870

# Ask the job to reload the configuration (SIGHUP)
$ sudo reload ushare

For SysV compatibility, the packages that were converted to upstart ship the symlinks pointing from /etc/init.d/something to /lib/init/upstart-job.

When you run e.g. /etc/init.d/cups restart you will receive an explanation how can you run the upstart job without the compatibility layer:

Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service cups restart

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the stop(8) and then start(8) utilities,
e.g. stop cups ; start cups. The restart(8) utility is also available.
cups stop/waiting
cups start/running, process 4028

Cookbook

The Upstart cookbook may look like it is an unnecessary long document but once you start reading it you realise the potential of upstart. I did not want to learn about upstart until I found myself in a need of a startup script that could launch and control my daemon, the whole script ended up having 7 stanzas and I did not even need to handle forking, respawning and privilege dropping in the daemon itself.

Seriously, if you are in doubt, read the Upstart Cookbook. Have fun!