Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.
—Upstart Cookbook upstart.ubuntu.com/cookbook
Upstart scripts have always seemed to be weird to me, somebody who learned about SysV init subsystem in Mandrake and Slackware long time ago. It seemed like Upstart was lacking lot of features but it turns out I just needed to understand what I was looking for.
So, when I found a need of a startup script, I decided to stop treating upstart as magic and here are my Upstart examples.
- Respawning a dead daemon.
- Monitoring the respawn limit.
- Disable an Upstart task.
- Listing, starting, stopping and restarting Upstart jobs.
Respawning a dead daemon
To stream files to my XBox (yes, I admit I have an XBox 360 console) I am using uShare, an outstanding piece of software that is actually compatible with Microsoft’s horrible DLNA specification. (That’s also the reason why I am so passionate about 973295 and 880076)
The default startup script simply launches the daemon as root and does nothing to keep it alive. As I found out one can make uShare exit by sending too much requests to reload the contents via its web server (this resulted from a bug in my inotify monitoring script, though). How do we ask upstart to restart the daemon?
# uShare UPnP A/V & DLNA Media Server description "uShare UPnP A/V & DLNA Media Server" author "Roman Yepishev <email@example.com>" start on filesystem stop on runlevel  respawn setuid nobody setgid nogroup exec /usr/bin/ushare
Original script reads $USHARE_OPTIONS from /etc/default/ushare which does not actually exist so I left this out.
What we have above is a simple script that:
- Starts uShare when filesystem is available.
- Stops uShare when system is rebooted or shut down.
- Respawns the process if it exits.
- Makes the process run as nobody:nogroup because I don’t want it to run as root.
$ sudo start ushare ushare start/running, process 5892 $ sudo kill 5892 $ sudo status ushare ushare start/running, process 7800
Yep, it works as expected. respawn stanza (that’s how these directives are called in Upstart) by default have a respawn limit 10 5 – 10 respawns per 5 seconds. If the application respawns faster than that (basically meaning that it cannot start successfully), the init will say
While respawn limit can be made unlimited such rate of failure signals about a real problem.
Please note that in order to control the respawn limit, you need to provide both respawn and respawn limit stanzas. respawn limit does not enable respawn:
Monitoring the respawn limit because we can
uShare crashing is not such a big deal but it was still upsetting my wife since it meant I had to log into the server and start the process again (before I fixed a bug in my script). But then I thought that notifying an administrator about such kind of an issue is a good thing.
# upstart-respawn-monitor.conf description "Notify root about ushare respawn limit reached" author "Roman Yepishev <firstname.lastname@example.org>" # This is a short-lived job task start on stopped ushare PROCESS=respawn script echo "uShare reached respawn limit" | mail -s "uShared crashed" root end script
This is another type of Upstream job, a task. It does not keep running forever and once it has finished it will not be respawned. Same thing as setting up the hostname. But what we just did is we created a simple monitoring task that will be executed once ushare reaches stopped state and a special PROCESS variable will have respawn as the reason why the service stopped. And this is all was done using Upstart only.
Before you start implementing monitoring this way, check whether there are existing solutions already.
Disable Upstart task
By default when you install a service in Ubuntu it will be immediately started. Sometimes, however, you might want to start service manually. Upstart has a concept of overrides that basically let you override any stanza in the original job file by creating a job.override file in /etc/init and placing the stanzas here.
For example, I am using LXC for some machines now and these lack tty[2..6]. Actually I don’t need these but getty keeps trying to start on these ttys. It starts and stops pretty slowly so it is not caught by the respawn limit. Disabling a task in Upstart is trivial, you just need to add manual to the stanzas:
Yes, it is that easy. Another example when you’d want to use an override file is when you want to use different options on the exec line (you will copy the original exec line and change the options then) or change the start on/stop on conditions.
Starting, stopping and restarting
If the job was converted to upstart, then interacting with it is easy:
# Getting the info about particular job: $ sudo status cups cups start/running, process 1503 # Getting the info about all the jobs $ sudo initctl list avahi-daemon start/running, process 1526 cgroup-lite start/running mountall-net stop/waiting nmbd start/running, process 1649 qemu-kvm start/running ... # Stopping a job $ sudo stop cups cups stop/waiting # Starting a job $ sudo start cups cups start/running, process 7870 # Ask the job to reload the configuration (SIGHUP) $ sudo reload ushare
For SysV compatibility, the packages that were converted to upstart ship the symlinks pointing from /etc/init.d/something to /lib/init/upstart-job.
When you run e.g. /etc/init.d/cups restart you will receive an explanation how can you run the upstart job without the compatibility layer:
Rather than invoking init scripts through /etc/init.d, use the service(8) utility, e.g. service cups restart Since the script you are attempting to invoke has been converted to an Upstart job, you may also use the stop(8) and then start(8) utilities, e.g. stop cups ; start cups. The restart(8) utility is also available. cups stop/waiting cups start/running, process 4028
The Upstart cookbook may look like it is an unnecessary long document but once you start reading it you realise the potential of upstart. I did not want to learn about upstart until I found myself in a need of a startup script that could launch and control my daemon, the whole script ended up having 7 stanzas and I did not even need to handle forking, respawning and privilege dropping in the daemon itself.
Seriously, if you are in doubt, read the Upstart Cookbook. Have fun!