1.1   Document Version

This is document edit 129.

See footer for further details.

1.2   Authors

1.3   Acknowledgements

The Authors are grateful to the following individuals who have provided valuable input to this document:

• Colin Watson (Canonical)
• Scott James Remnant (Canonical, Google), author of Upstart.
• James Page (Canonical)
• Joel Ebel (Google)
• Mark Russell (Canonical)
• Bradley Ayers
• Kenneth Porter

1.4   Purpose

The purpose of this document is multi-faceted. It is intended as:

• A gentle introduction to Upstart.
• A Cookbook of recipes and best-practises for solving common and not so common problems.
• An extended guide to the configuration syntax of Upstart.

It attempts to explain the intricacies of Upstart with worked examples and lots of details.

Note that the reference documentation for Upstart will always be the manual pages: this is merely a supplement to them.

1.5   Suggestions and Errata

Bad documentation is often worse than no documentation. If you find a problem with this document, however small...

• spelling error
• grammatical error
• factual error
• inconsistency
• lack of clarity
• ambiguous or misleading content
• missing information
• et cetera

... or if you'd like to see some particular feature covered please raise a bug report on the Upstart Cookbook project website so that we can improve this work:

• https://bugs.launchpad.net/upstart-cookbook/+filebug

As an incentive you will be credited in the Acknowledgements section.

1.6   Coverage

There are essentially two major versions of Upstart covered by this document:

1.7   Audience

This document is targeted at:

• Users interested in learning about Upstart.
• System Administrators looking to make the most of the capabilities of Upstart.
• Developers and Packagers who wish to package their application to work with Upstart.

1.8   Document Preparation

This document is written in reStructuredText, a textual markup language. The document was prepared using the following tools:

• Vim editor.
• Emacs editor with Org-Mode for tables.
• Jave for ASCII graphics.

1.9   Document Availability

The source for this document is available here:

• https://code.launchpad.net/~upstart-documenters/upstart-cookbook/trunk

The latest version of this document should always be available from:

• http://upstart.ubuntu.com/cookbook/
• http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf

1.10   Warning

This document aims to aid understanding of Upstart and identify some hopefully useful "canned" solutions and advice to common problems and questions.

The authors have taken as much care as possible in the preparation of this document. However, you are advised strongly to exercise extreme caution when changing critical system facilities such as the init daemon. Most situations are recoverable and advice is provided in this document, but if your system explodes in a ball of fire or becomes unusable as a result of a suggestion from this document, you alone have the intellectual pleasure of fixing your systems.

2.1   Commands and configuration stanzas

Throughout this document a fixed-width font such as this will be used to denote commands, brief command output and configuration stanzas.

2.2   User Input and Command Output

An indented block will be used to denote user input and command output.

$ echo hello
hello

# whoami
root

2.3   Configuration Examples

An indented block is also used to show examples of job configuration:

script
  # a config file
end script

3.1   What is Upstart?

Quoting from http://upstart.ubuntu.com/,

Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.

The "init" or "system initialisation" process on Unix and Linux systems has process ID (PID) "1". That is to say, it is the first process to start when the system boots (ignoring the initrd/initramfs). As the quote shows, Upstart is an "init" replacement for the traditional Unix "System V" "init" system. Upstart provides the same facilities as the traditional "init" system, but surpasses it in many ways.

4.1   Job

A "unit of work" - generally either a "Task" or a "Service". Each Job is defined in a Job configuration file.

4.2   Job Configuration File

A Job is defined in a Job Configuration File (or mor esimply a conf file) which is a plain text file containing one or more stanzas. Job configuration files are named:

<name>.conf

Where "<name>" should reflect the application being run or the service being provided.

Job configuration files can exist in two types of location, depending on whether they are a System Job or a User Job.

Note that it is common to refer to a Job configuration file as a "job", although technically a job is a running instance of a Job configuration file.

/etc/init/

$HOME/.init/

/etc/dbus-1/system.d/Upstart.conf

4.3   Event

A notification sent by Upstart to all interested parties (either jobs or other events). They can generally be thought of as "signals", "methods", or "hooks" [20], depending on how they are emitted and/or consumed.

Events are emitted (created and then broadcast) to the entire Upstart system. Note that it is not possible to stop any other job or event from seeing an event when it is emitted.

If there are no jobs which have registered an interest in an event in either their start on or stop on conditions, the event has no effect on the system.

Events can be created by an administrator at any time using:

# initctl emit <event>

Note that some events are "special". See the upstart-events(7) manual page for a list.

Note also that an event name with the same name as a job is allowed.

Jobs are often started or stopped as a result of other jobs starting or stopping. Upstart has a special set of events that it emits to announce these job state transitions. You'll probably notice that these events have the same names as some of the job states described in Job States, however it's important to appreciate that these are not describing the same thing. Task states are not events, and events are not task states. See Events, not States for details.

These events are as follows:

starting

This event is emitted by Upstart when a job has been scheduled to run and is about to start executing.

started

This event is emitted by Upstart when a job is now running. Note that a job does not have to have an associated program or script so "running" does not necessarily imply that any additional process is executing.

stopping

This event is emitted by Upstart when a job is about to be stopped.

stopped

This event is emitted by Upstart when a job has completed (successfully or otherwise).

See Job Lifecycle for further details.

To help reinforce the difference, consider how Upstart itself starts: See the Startup Process.

1. It performs its internal initialization.
2. Upstart itself emits a single event called startup(7). This event triggers the rest of the system to initialize. Note that there is no "startup" job (and hence no /etc/init/startup.conf file).
3. init(8) runs the mountall job (as defined in /etc/init/mountall.conf) since the startup(7) event satisfies mountall(8)'s requirement: "start on startup".
4. The mountall(8) job in turn emits a number of events (including local-filesystems(7) and all-swaps(7)). See upstart-events(7) for further details.

Upstart provides three different types of Events.

# initctl emit --no-wait mysignal

# initctl emit mymethod

start on mymethod
task
exec /usr/bin/myapp $ACTION

# initctl emit mymethod ACTION=do_something
[ $? -ne 0 ] && { echo "ERROR: myapp failed"; exit 1; }

4.4   Job Lifecycle

4.5   Ordering

start on event-A
stop  on event-A

script
  sleep 999
end script

# initctl log-priority debug

# status odd
odd stop/waiting
# initctl emit event-A
# status odd
odd start/running, process 9474

event_new: Pending event-A event               # Upstart emitted the event.
Handling event-A event
event_pending_handle_jobs: New instance odd    # Job instance created.
odd goal changed from stop to start            # Since job not running,
odd state changed from waiting to starting     # change goal to "start".
event_new: Pending starting event
Handling starting event
event_finished: Finished starting event
odd state changed from starting to pre-start
odd state changed from pre-start to spawned
odd main process (9474)                        # Start script section.
odd state changed from spawned to post-start
odd state changed from post-start to running   # Job now fully started.
event_new: Pending started event
Handling started event
event_finished: Finished started event
event_finished: Finished event-A event

# status odd
odd stop/waiting
# start odd
odd start/running, process 11416    # Note this PID!
# status odd
odd start/running, process 11416
# initctl emit event-A
# status odd
odd start/running, process 11428    # Look! It changed!

odd goal changed from stop to start
odd state changed from waiting to starting
event_new: Pending starting event
Handling starting event
event_finished: Finished starting event
odd state changed from starting to pre-start
odd state changed from pre-start to spawned
odd main process (11416)
odd state changed from spawned to post-start
odd state changed from post-start to running
event_new: Pending started event
Handling started event
event_finished: Finished started event

event_new: Pending event-A event
Handling event-A event
odd goal changed from start to stop             # Job already running, so stop it.
odd state changed from running to pre-stop
odd state changed from pre-stop to stopping
event_new: Pending stopping event
event_pending_handle_jobs: New instance odd
odd goal changed from stop to start
Handling stopping event
event_finished: Finished stopping event
odd state changed from stopping to killed
Sending TERM signal to odd main process (11416) # Forcibly stop existing job process.
odd main process (11416) killed by TERM signal  # Successfully stopped it.
odd state changed from killed to post-stop
odd state changed from post-stop to starting
event_new: Pending starting event
Handling starting event
event_finished: Finished starting event
odd state changed from starting to pre-start
odd state changed from pre-start to spawned
odd main process (11428)                        # New instance of job started with new PID.
odd state changed from spawned to post-start
odd state changed from post-start to running
event_new: Pending started event
Handling started event
event_finished: Finished started event
event_finished: Finished event-A event

4.6   Runlevels

A runlevel is a single-byte name for a particular system configuration. Runlevels for Debian and Ubuntu systems are generally as follows [29]:

• 0 : System halt.
• 1 : Single-User mode.
• 2 : Graphical multi-user plus networking (DEFAULT)
• 3 : Same as "2", but not used.
• 4 : Same as "2", but not used.
• 5 : Same as "2", but not used.
• 6 : System reboot.

There are also a few pseudo-runlevels:

• N : The previous runlevel cannot be determined.
• S : Alias for Single-User mode.

$ runlevel
N 2

env DEFAULT_RUNLEVEL=1

DEFAULT_RUNLEVEL=1

5.1   Startup

At boot, after the initramfs system has been run (for setting up RAID, unlocking encrypted file system volumes, et cetera), Upstart will be given control. The initramfs environment will exec(3) /sbin/init (this is the main Upstart binary) and cause it to run as PID 1.

5.2   Shutdown

5.3   Reboot

To initiate a reboot, perform one of the following actions:

• Click "Restart..." (or equivalent) in your graphical environment (for example Gnome)

• Run the shutdown(8) command specifying the "-r" option, for example:

  # shutdown -r now
  

• Run the reboot(8) command:

  # reboot
  

The following will steps will now be taken:

1. Assuming the current runlevel is "2", whichever command is run above will cause Upstart to emit the runlevel(7) event like this:

   runlevel RUNLEVEL=6 PREVLEVEL=2
   

2. The job /etc/init/rc.conf will be run.

   This job calls /etc/init.d/rc passing it the new runlevel ("6").

3. The SystemV system will then invoke the necessary scripts in /etc/rc6.d/ to stop SystemV services.

4. One of the scripts run is /etc/init.d/sendsigs.

   This script will kill any remaining processes not already stopped (including Upstart processes).

5.4   Single-User Mode

When booting direct into single-user mode, the runlevel command will show:

# runlevel
N S

See Runlevels.

5.5   Recovery Mode ()

Ubuntu provides a recovery mode in case your system experiences problems. This is handled by the friendly-recovery package. If you select a "recovery mode" option on the Grub menu. This makes the initramfs pass a flag to Upstart which ensures that the /etc/init/friendly-recovery.conf Upstart job is the first job run after Upstart starts. As a result, this job has full control over the system and provides a friendly menu that allows users to check disks with fsck(8), repair your package database and so on.

5.6   Failsafe Mode ()

This is a new phase introduced in Ubuntu 11.10 that borrows an idea from Google's Chrome OS. A new job called failsafe has been introduced that checks to ensure the system has reached a particular state. If the expected state is not attained, the job reboots the system automatically.

6.1   Stanzas by Category

6.2   author

Syntax:

author <string>

Quoted name (and maybe contact details) of author of this Job Configuration File.

Example:

author "Scott James Remnant <scott@netsplit.com>"

6.3   console

For all versions of Upstart prior to v1.4, the default value for console was console none. As of Upstart 1.4, the default value is console log. If you are using Upstart 1.4 or later and wish to retain the old default, boot specifying the --no-log command-line option. An alternative is to boot using the --default-console <value> option which allows the default console value for jobs to be specified. Using this option it is possible to set the default to none but still honour jobs that specify explicitly console log.

6.4   chdir

Syntax:

chdir <directory>

Runs the job's processes with a working directory in the specified directory instead of the root of the filesystem.

Example:

chdir /var/mydaemon

6.5   chroot

Syntax:

chroot <directory>

Runs the job's processes in a chroot(8) environment underneath the specified directory.

Note that the specified directory must have all the necessary system libraries for the process to be run, often including /bin/sh.

Example:

chroot /srv/chroots/oneiric

6.6   description

Syntax:

description <string>

One line quoted description of Job Configuration File. For example:

description "OpenSSH server"

6.7   emits

Syntax:

emits <values>

Specifies the events the job configuration file generates (directly or indirectly via a child process). This stanza can be specified multiple times for each event emitted. This stanza can also use the following shell wildcard meta-characters to simplify the specification:

• asterisk ("*")
• question mark ("?")
• square brackets ("[" and "]")

For example, upstart-udev-bridge can emit a large number of events. Rather than having to specify every possible event, since the form of the event names is consistent, a single emits stanza can be specified to cover all possible events:

emits *-device-*

Further Examples:

emits foo-event bar-event wibble-event
emits hello

6.8   end script

This psuedo-stanza acts as a terminator for script sections:

• script.
• pre-start script.
• post-start script.
• pre-stop script.
• post-start script`.

6.9   env

Syntax:

env KEY[=VALUE]

Allows an environment variable to be set which is accessible in all script sections.

Example:

env myvar="hello world"

script
  echo "myvar='$myvar'" > /run/script.log
end script

See Environment Variables.

6.10   exec

Syntax:

exec COMMAND [ ARG ]...

Stanza that allows the specification of a single-line command to run. Note that if this command-line contains any shell meta-characters, it will be passed through a shell prior to being executed. This ensures that shell redirection and variable expansion occur as expected.

Example:

exec /usr/bin/my-daemon --option foo -v

6.11   expect

Upstart will keep track of the process ID that it thinks belongs to a job (using ptrace(2)). If a job has specified the instance stanza, Upstart will track the PIDS for each unique instance of that job.

If you do not specify the expect stanza, Upstart will track the life cycle of the first PID that it executes in the exec or script stanzas. However, most Unix services will "daemonize", meaning that they will create a new process (using fork(2)) which is a child of the initial process. Often services will "double fork" to ensure they have no association whatsoever with the initial process. (Note that no services will fork more than twice initially since there is no additional benefit in doing so).

In this case, Upstart must have a way to track it, so you can use expect fork, or expect daemon.

To allow Upstart to determine the final process ID for a job, it needs to know how many times that process will call fork(2). Upstart itself cannot know the answer to this question since once a daemon is running, it could then fork a number of "worker" processes which could themselves fork any number of times. Upstart cannot be expected to know which PID is the "master" in this case, considering it does not know if worker processes will be created at all, let alone how many times, or how many times the process will fork initially. As such, it is necessary to tell Upstart which PID is the "master" or parent PID. This is achieved using the expect stanza.

The syntax is simple, but you do need to know how many times your service forks.

Note that most daemons fork twice.

If your daemon has a "don't daemonize" or "run in the foreground" mode, then it's much simpler to use that and not run with fork following. One issue with that though, is that Upstart will emit the started JOB=yourjob event as soon as it has executed your daemon, which may be before it has had time to listen for incoming connections or fully initialize.

A final point: the expect stanza only applies to exec and script stanzas: it has no effect on pre-start and post-start.

# Trace all children of /usr/bin/myapp
$ sudo strace -o /tmp/strace.log -fFv /usr/bin/myapp --arg foo --hello wibble &

# After allowing some "reasonable" time for the app to start, kill it and strace
$ sudo killall -9 strace

# Display the number of forks
#
#   1 => specify "expect fork"
#   2 => specify "expect daemon"
#
$ sudo egrep "\<(fork|clone)\>\(" /tmp/strace.log | wc | awk '{print $1}'

6.12   export

Export variables previously set with env to all events that result from this job. See for example Job Lifecycle.

Note that no leading dollar sign ($) is specified.

Example:

env myvar="hello world"
export myvar

6.13   instance

Sometimes you want to run the same job, but with different arguments. The variable that defines the unique instance of this job is defined with instance.

Lets say that once memcached is up and running, we want to start a queue worker for each directory in /var/lib/queues:

# queue-workers

start on started memcached

task

script
  for dir in `ls /var/lib/queues` ; do
    start queue-worker QUEUE=$dir
  done
end script

And now:

# queue-worker

stop on stopping memcached

respawn

instance $QUEUE

exec /usr/local/bin/queue-worker $QUEUE

In this way, Upstart will keep them all running with the specified arguments, and stop them if memcached is ever stopped.

The instance stanza is designed to make a running job unique.

Notes:

• the stanza isn't restricted to a single value. You can do silly things like the following if you wish:

  instance ${myvar1}hello${myvar2}-foo/\wibble${var3}{$JOB}
  

  See Multiple Running Job Instances Without PID for another crazy real-life example.

• You must include at least one variable and it must have a leading dollar sign ($):

  # GOOD (value can be changed by specifying different values
  # for the variable called 'foo')
  instance $foo
  
  # BAD (value will always be the string literal "foo")
  instance foo
  

• If you attempt to start a job with the instance stanza, but forget to provide the required variables, you will get an error since Upstart cannot then guarantee uniqueness. For example, if you have a job configuration file foo.conf such as this:

  instance $bar
  
  script
    sleep 999
  end script
  

  Attempting to start it without specifying a value for foo will fail:

  # start foo
  start: Unknown parameter: bar
  

  Let's try again:

  # start foo bar=1
  foo (1) start/running, process 30003
  

  And now let's start another instance:

  # start foo bar="hello 1,2,3"
  foo (hello 1,2,3) start/running, process 30008
  

  Finally, let's see the current state of our two job instances:

  $ initctl list|grep ^foo
  foo (1) start/running, process 30003
  foo (hello 1,2,3) start/running, process 30008
  

# /etc/init/trickery.conf
start on foo

instance $UPSTART_EVENTS
env UPSTART_EVENTS=

# start trickery

6.14   kill signal

Specifies the stopping signal, SIGTERM by default, a job's main process will receive when stopping the running job.

Example:

kill signal INT

Note that if you are running an older version of Upstart without this feature, and you have an application which breaks with the normal conventions for shutdown signal, you can simulate it to some degree by using start-stop-daemon(8) with the --signal option:

start on some-event

exec start-stop-daemon --signal SIGQUIT --start --exec /usr/bin/foo

6.15   kill timeout

The number of seconds Upstart will wait before killing a process. The default is 5 seconds.

Example:

kill timeout 20

6.16   limit

Provides the ability to specify resource limits for a job.

For example, to allow a job to open any number of files, specify:

limit nofile unlimited unlimited

For further details on the available limits see init(5) and getrlimit(2).

6.17   manual

Added in Upstart v0.6.7

This stanza will tell Upstart to ignore the start on / stop on stanzas. It is useful for keeping the logic and capability of a job on the system while not having it automatically start at boot-up.

Example:

manual

6.18   nice

Change the jobs scheduling priority from the default. See nice(1).

Example:

# run with lowest priority
nice 19

6.19   normal exit

Used to change Upstart's idea of what a "normal" exit status is. Conventionally, processes exit with status "0" (zero) to denote success and non-zero to denote failure. If your application can exit with exit status "13" and you want Upstart to consider this as an normal (successful) exit, then you can specify:

normal exit 0 13

You can even specify signals. For example, to consider exit codes "0", "13" as success and also to consider the program to have completed successfully if it exits on signal "SIGUSR1" and "SIGWINCH", specify:

normal exit 0 13 SIGUSR1 SIGWINCH

6.20   oom score

Linux has an "Out of Memory" killer facility. This is a feature of the kernel that will detect if a process is consuming increasingly more memory. Once "triggered", the kernel automatically takes action by killing the rogue process to avoid it impacting the system adversely.

Normally the OOM killer regards all processes equally, this stanza advises the kernel to treat this job differently.

The "adjustment" value provided to this stanza may be an integer value from -999 (very unlikely to be killed by the OOM killer) up to 1000 (very likely to be killed by the OOM killer). It may also be the special value never to have the job ignored by the OOM killer entirely (potentially dangerous unless you really trust the application in all possible system scenarios).

Example:

# this application is a "resource hog"
oom score 1000

expect daemon
respawn
exec /usr/bin/leaky-app

6.21   post-start

Syntax:

post-start exec|script

Script or process to run after the main process has been spawned, but before the started(7) event has been emitted.

Use this stanza when a delay (or some arbitrary condition) must be satisfied before an executed job is considered "started". An example is MySQL. After executing it, it may need to perform recovery operations before accepting network traffic. Rather than start dependent services, you can have a post-start like this:

post-start script
  while ! mysqladmin ping localhost ; do sleep 1 ; done
end script

6.22   post-stop

Syntax:

post-stop exec|script

There are times where the cleanup done in pre-start is not enough. Ultimately, the cleanup should be done both pre-start and post-stop, to ensure the service starts with a consistent environment, and does not leave behind anything that it shouldn't.

exec /some/directory/script

If it is possible, you'll want to run your daemon with a simple exec line. Something like this:

exec /usr/bin/mysqld

If you need to do some scripting before starting the daemon, script works fine here. Here is one example of using a script stanza that may be non-obvious:

# statd - NSM status monitor

description   "NSM status monitor"
author                "Steve Langasek <steve.langasek@canonical.com>"

start on (started portmap or mounting TYPE=nfs)
stop on stopping portmap

expect fork
respawn

env DEFAULTFILE=/etc/default/nfs-common

pre-start script
    if [ -f "$DEFAULTFILE" ]; then
        . "$DEFAULTFILE"
    fi

    [ "x$NEED_STATD" != xno ] || { stop; exit 0; }

    start portmap || true
    status portmap | grep -q start/running
    exec sm-notify
end script

script
    if [ -f "$DEFAULTFILE" ]; then
        . "$DEFAULTFILE"
    fi

    if [ "x$NEED_STATD" != xno ]; then
        exec rpc.statd -L $STATDOPTS
    fi
end script

Because this job is marked respawn, an exit of 0 is "ok" and will not force a respawn (only exiting with a non-0 exit or being killed by an unexpected signal causes a respawn), this script stanza is used to start the optional daemon rpc.statd based on the defaults file. If NEED_STATD=no is in /etc/default/nfs-common, this job will run this snippet of script, and then the script will exit with 0 as its return code. Upstart will not respawn it, but just gracefully see that it has stopped on its own, and return to stopped status. If, however, rpc.statd had been run, it would stay in the start/running state and be tracked normally.

6.23   pre-start

Syntax:

pre-start exec|script

Use this stanza to prepare the environment for the job. Clearing out cache/tmp dirs is a good idea, but any heavy logic is discouraged, as Upstart job files should read like configuration files, not so much like complicated software.

pre-start script
  [ -d "/var/cache/squid" ] || squid -k
end script

Another possibility is to cancel the start of the job for some reason. One good reason is that it's clear from the system configuration that a service is not needed:

pre-start script
  if ! grep -q 'parent=foo' /etc/bar.conf ; then
    stop ; exit 0
  fi
end script

Note that the "stop" command did not receive any arguments. This is a shortcut available to jobs where the "stop" command will look at the current environment and determine that you mean to stop the current job.

6.24   pre-stop

Syntax:

pre-stop exec|script

The pre-stop stanza will be executed before the job's stopping(7) event is emitted and before the main process is killed.

Stopping a job involves sending SIGTERM to it. If there is anything that needs to be done before SIGTERM, do it here. Arguably, services should handle SIGTERM very gracefully, so this shouldn't be necessary. However, if the service takes more than kill timeout seconds (default, 5 seconds) then it will be sent SIGKILL, so if there is anything critical, like a flush to disk, and raising kill timeout is not an option, pre-stop is not a bad place to do it. [15]

You can also use this stanza to cancel the stop, in a similar fashion to the way one can cancel the start in the pre-start.

6.25   respawn

Without this stanza, a job that exits quietly transitions into the stop/waiting state, no matter how it exited.

With this stanza, whenever the main script/exec exits, without the goal of the job having been changed to stop, the job will be started again. This includes running pre-start, post-start and post-stop. Note that pre-stop will not be run.

There are a number of reasons why you may or may not want to use this. For most traditional network services this makes good sense. If the tracked process exits for some reason that wasn't the administrator's intent, you probably want to start it back up again.

Likewise, for tasks, (see below), respawning means that you want that task to be retried until it exits with zero (0) as its exit code.

One situation where it may seem like respawn should be avoided, is when a daemon does not respond well to SIGTERM for stopping it. You may believe that you need to send the service its shutdown command without Upstart being involved, and therefore, you don't want to use respawn because Upstart will keep trying to start your service back up when you told it to shutdown.

However, the appropriate way to handle that situation is a pre-stop which runs this shutdown command. Since the job's goal will already be 'stop' when a pre-stop is run, you can shutdown the process through any means, and the process won't be re-spawned (even with the respawn stanza).

6.26   respawn limit

Yes, this is different to a plain respawn: specifying respawn limit does not imply respawn.

Syntax:

respawn limit COUNT INTERVAL

Example:

# respawn the job up to 10 times within a 5 second period.
# If the job exceeds these values, it will be stopped and
# marked as failed.
respawn
respawn limit 10 5

Respawning is subject to a limit. If the job is respawned more than COUNT times in INTERVAL seconds, it will be considered to be having deeper problems and will be stopped. Default COUNT is 10. Default INTERVAL is 5 seconds.

Note that this only applies to automatic respawns and not the restart(8) command.

6.27   script

Allows the specification of a multi-line block of shell code to be executed. Block is termined by end script.

6.28   setgid

Added in Upstart v1.4

Syntax:

setgid <groupname>

Changes to the group <groupname> before running the job's process.

If this stanza is unspecified, the primary group of the user specified in the setuid block is used. If both stanzas are unspecified, the job will run with its group ID set to 0 in the case of system jobs, and as the primary group of the user in the case of User Jobs.

Example:

setgid apache

6.29   setuid

Added in Upstart v1.4

Syntax:

setuid <username>

Changes to the user <username> before running the job's process.

If this stanza is unspecified, the job will run as root in the case of system jobs, and as the user in the case of User Jobs.

Note that System jobs using the setuid stanza are still system jobs, and can not be controlled by an unprivileged user, even if the setuid stanza specifies that user.

6.30   start on

Note that the start on stanza expects a token to follow on the same line. Thus:

# ERROR: invalid
start on
  foo or bar

# OK
start on foo or bar

start on (local-filesystems and net-device-up IFACE!=lo)

start on runlevel [2345]

start on started network-services

start on started other-service

start on starting other-service

start on starting apache2
stop on stopped apache2
respawn

exec /usr/sbin/memcached

6.31   stop on

stop on runlevel [016]

stop on stopping network-services

stop on stopping other-service

start on starting other-service

stop on stopped other-service

6.32   task

In concept, a task is just a short lived job. In practice, this is accomplished by changing how the transition from a goal of "stop" to "start" is handled.

Without the 'task' keyword, the events that cause the job to start will be unblocked as soon as the job is started. This means the job has emitted a starting(7) event, run its pre-start, begun its script/exec, and post-start, and emitted its started(7) event.

With task, the events that lead to this job starting will be blocked until the job has completely transitioned back to stopped. This means that the job has run up to the previously mentioned started(7) event, and has also completed its post-stop, and emitted its stopped(7) event.

Typically, task is for something that you just want to run and finish completely when a certain event happens.

# pre-warm-memcache

start on started memcached

task

exec /path/to/pre-warm-memcached

So you can have another job that starts your background queue worker once the local memcached is pre-warmed:

# queue-worker

start on stopped pre-warm-memcache
stop on stopping memcached

respawn

exec /usr/local/bin/queue-worker

The key concept demonstrated above is that we "start on stopped pre-warm-memcache". This means that we don't start until the task has completed. If we were to use started instead of stopped, we would start our queue worker as soon as /path/to/pre-warm-memcached had been started running.

We could also accomplish this without mentioning the pre-warm in the queue-worker job by doing this:

# queue-worker

start on started memcached
stop on stopping memcached

respawn

exec /usr/local/bin/queue-worker

# pre-warm-memcache

start on starting queue-worker
task
exec /path/to/pre-warm-memcache

If we did not use "task" in the above example, queue-worker would be allowed to start as soon as we executed /path/to/pre-warm-memcache, which means it might potentially start before the cache was warmed.

6.33   umask

Syntax:

umask <value>

Set the file mode creation mask for the process. <value>" should be an octal value for the mask. See umask(2) for more details.

Example:

umask 0002

6.34   version

Syntax:

version <string>

This stanza may contain version information about the job, such as revision control or package version number. It is not used or interpreted by init(8) in any way.

Example:

version "1.0.2a-beta4"

7.1   Really understanding start on and stop on

(Note: This section focuses on start on, but the information also applies to stop on unless explicitly specified).

The start on stanza needs careful contemplation. Consider this example:

start on started mysql

The syntax above is actually a short-hand way of writing:

start on started JOB=mysql

Remember that started(7) is an event which Upstart emits automatically when the mysql job has started to run. The whole start on stanza can be summarized as:

start on <event> [<vars_to_match_event_on>]

Where <vars_to_match_event_on> is optional, but if specified comprises one or more variables.

A slight variation of the above:

start on started JOB=mydb DBNAME=foobar

This example shows that the fictitious job above would only be started when the mydb database server brings the foobar database on-line. Correspondingly, file /etc/init/mydb.conf would need to specify "export DBNAME" and be started like this:

start mydb DBNAME=foobar

Looking at a slightly more complex real-life example:

# /etc/init/alsa-mixer-save.conf
start on starting rc RUNLEVEL=[06]

This job says,

"Run when the rc job emits the starting(7) event, but only if the

environment variable RUNLEVEL equals either 0 (halt) or 6 (reboot)".

If we again add in the implicit variable it becomes clearer:

# /etc/init/alsa-mixer-save.conf
start on starting JOB=rc RUNLEVEL=[06]

But where does the RUNLEVEL environment variable come from? Well, variables are exported in a job configuration file to related jobs. Thus, the answer is The rc Job.

If you look at this job configuration file, you will see, as deduced:

export RUNLEVEL

# /etc/init/rc.conf
start on runlevel [0123456]
stop on runlevel [!$RUNLEVEL]

export RUNLEVEL
export PREVLEVEL

console output
env INIT_VERBOSE

task

exec /etc/init.d/rc $RUNLEVEL

stop on runlevel [!$RUNLEVEL]

stop on runlevel [!2]

7.2   Environment Variables

Upstart allows you to set environment variables which will be accessible to the jobs whose job configuration files they are defined in. Environment variables are set using the env keyword.

For example:

# /etc/init/env.conf
env TESTING=123

script
  # prints "TESTING='123'" to system log
  logger -t $0 "TESTING='$TESTING'"
end script

Further, we can pass environment variables defined in events to jobs using the env stanza and the export stanza. Assume we have two job configuration files, A.conf and B.conf:

# /etc/init/A.conf
start on wibble
export foo

# /etc/init/B.conf
start on A
script
  logger "value of foo is '$foo'"
end script

If we now run the following command, both jobs A and B will run, causing B to write "value of foo is 'bar'" to the system log:

# initctl emit wibble foo=bar

Note that a variables value can always be overridden by specifying a new value on the command-line. For example:

start on wibble
env var=hello

script
  logger "value of var is '$var'"
end script

When we emit the required event...:

# initctl emit wibble var=world

... the system log will have recorded:

value of var is 'world'

Note that a Job Configuration File does not have access to a user's environment variables, not even the superuser. This is not possible since all job processes created are children of init which does not have a user's environment.

However, using the technique above, it is possible to inject a variable from a user's environment into a job indirectly:

# initctl emit wibble foo=bar USER=$USER

As another example of environment variables, consider this job configuration file [15]:

env var=bar
export var

pre-start script
  logger "pre-start: before: var=$var"

  var=pre-start
  export var

  logger "pre-start: after: var=$var"
end script

post-start script
  logger "post-start: before: var=$var"

  var=post-start
  export var

  logger "post-start: after: var=$var"
end script

script
  logger "script: before: var=$var"

  var=main
  export var

  logger "script: after: var=$var"
end script

post-stop script
  logger "post-stop: before: var=$var"

  var=post-stop
  export var

  logger "post-stop: after: var=$var"
end script

This will generate output in your system log as follows (the timestamp and hostname have been removed, and the output formatted to make it clearer):

logger: pre-start:  before: var=bar
logger: pre-start:   after: var=pre-start

logger: post-start: before: var=bar
logger: post-start:  after: var=post-start

logger: script:     before: var=bar
logger: script:      after: var=main

logger: post-stop:  before: var=bar
logger: post-stop:   after: var=post-stop

As shown, every script section receives the value of $var as bar, but if any script section changes the value, it only affects that particular script sections copy of the variable. To summarize:

A script section cannot modify the value of a variable defined in a job configuration file for other script sections.

env FOO=bar
start on $FOO

# job above *NOT* started
initctl emit bar

# job above started!
initctl emit '$FOO'

start on starting $FOO
start on starting JOB=$FOO

start on starting foo

JOB=foo
UPSTART_JOB=bar
UPSTART_EVENTS=starting
INSTANCE=

7.3   Job with Multiple Duplicate Stanzas

The way in which Upstart parses the job configuration files means that "the last entry wins". That is to say, every job configuration file must be syntactically correct, but if you had a file such as:

start on event-A
start on starting job-B
start on event-C or starting job-D

This job will have a start on condition of:

start on event-C or starting job-D

...since that is the last start on condition specified.

For start on, stop on and emits stanzas, you can confirm Upstart's decision, you can use the initctl show-config command like this:

initctl show-config myjob

For the example above, the output would be:

start on event-C or starting job-D

7.4   Job Specifying Same Condition in start on on stop on

See Ordering of Stop/Start Operations.

8.1   D-Bus Service Activation

As of D-Bus version 1.4.1-0ubuntu2 (in Ubuntu), you can have Upstart start a D-Bus service rather than D-Bus. This is useful because it is then possible to create Upstart jobs that start or stop when D-Bus services start.

See Run a Job When a User Logs in for an example.

9.1   Utilities

initctl reload <job>

initctl restart <job>

initctl start <job>

# start myjob
start: Job is already running: myjob

# start myjob
start: Unknown parameter: foo

# start myjob foo="hello, world"

initctl stop <job>

# stop myjob
stop: unknown instance

# stop myjob
stop: Unknown parameter: foo

# stop myjob foo=...

<job> [ (<instance>)]<goal>/<status>[, process <PID>]
        [<section> process <PID>]

<job> <goal>/<status>

ufw start/running

description     "Uncomplicated firewall"

# Make sure we start before an interface receives traffic
start on (starting network-interface
          or starting network-manager
          or starting networking)

stop on runlevel [!023456]

console output

pre-start exec /lib/ufw/ufw-init start quiet
post-stop exec /lib/ufw/ufw-init stop

<job> <goal>/<status>, process <PID>

cups start/running, process 1733

<job> <goal>/<status>, process <PID>
  <section> process <PID>

ureadahead stop/pre-stop, process 227
      pre-stop process 5579

<job> (<instance>) <goal>/<status> (<instance>)
<job> (<instance>) <goal>/<status> (<instance>)

network-interface (lo) start/running
network-interface (eth0) start/running

network-interface-security (network-manager) start/running
network-interface-security (network-interface/eth0) start/running
network-interface-security (network-interface/lo) start/running
network-interface-security (networking) start/running

start on (starting network-interface
          or starting network-manager
          or starting networking)

instance $JOB${INTERFACE:+/}${INTERFACE:-}

pre-start script
  # ...
end script

<job> (<instance>) <goal>/<status> (<instance>), process <PID>

foo (1) start/running, process 30003
foo (hello 1,2,3) start/running, process 30008

<job> (<instance>) <goal>/<status> (<instance>), process <PID>
        <section> process <PID>

myjob (foo) stop/pre-stop, process 31677
        pre-stop process 31684
myjob (bar) stop/pre-stop, process 31679
        pre-stop process 31687
myjob (bzr) stop/pre-stop, process 31681
        pre-stop process 31690

instance $foo

exec sleep 999

pre-stop script
  sleep 999
end script

# for i in foo bar baz; do start -n myjob foo=$i; done

# initctl list|grep -A 1 ^inst
myjob start/running (foo), process 31677
myjob start/running (bar), process 31679
myjob start/running (baz), process 31681

# for i in foo bar baz; do stop -n myjob foo=$i; done
myjob (foo) stop/pre-stop, process 31677
      pre-stop process 31684
myjob (bar) stop/pre-stop, process 31679
      pre-stop process 31687
myjob (baz) stop/pre-stop, process 31681
      pre-stop process 31690

<job> <goal>/<status>

rc stop/waiting

initctl log-priority <priority>

# : same as "--verbose"
# initctl log-priority info

# : same as "--debug"
# initctl log-priority debug

$ # search all job configuration files for "unreachable" conditions
$ initctl check-config

$ # search specified job configuration file for unreachable conditions
$ initctl check-config <job>

$ initctl check-config plymouth
$ initctl check-config --warn plymouth
plymouth
  start on: unknown job uxlaunch
  start on: unknown job lightdm
  start on: unknown job lxdm
  start on: unknown job xdm
  start on: unknown job kdm
$

$ init-checkconf myjob.conf

+------------------------------------------------------------+---------------------+
|mounting MOUNTPOINT=/virtual-1                              |  mounting TYPE=swap |
|mounted  MOUNTPOINT=/virtual-1                              |  mounted  TYPE=swap |
|     :                                                      |  all-swaps          |
|mounting MOUNTPOINT=/virtual-n                              |                     |
|mounted  MOUNTPOINT=/virtual-n                              |                     |
|virtual-filesystems                                         |                     |
+-----------------------------+------------------------------+                     |
|mounting MOUNTPOINT=/local-1 |mounting MOUNTPOINT=/remote-1 |                     |
|mounted  MOUNTPOINT=/local-1 |mounted  MOUNTPOINT=/remote-1 |                     |
|     :                       |     :                        |                     |
|mounting MOUNTPOINT=/local-n |mounting MOUNTPOINT=/remote-n |                     |
|mounted  MOUNTPOINT=/local-n |mounted  MOUNTPOINT=/remote-n |                     |
|local-filesystems            |remote-filesystems            |                     |
+-----------------------------+------------------------------+---------------------+
|filesystem                                                                        |
+----------------------------------------------------------------------------------+

start on (local-filesystems
      or (mounting
      or (mounted
      or (virtual-filesystems
      or (remote-filesystems
      or (all-swaps or filesystem))))))

script
  echo "\n`env`" >> /dev/.initramfs/mountall.log
end script

MOUNTPOINT=/proc
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=nodev,noexec,nosuid
TYPE=proc
UPSTART_EVENTS=mounted
PWD=/
DEVICE=proc

MOUNTPOINT=/sys/fs/fuse/connections
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=optional
TYPE=fusectl
UPSTART_EVENTS=mounted
PWD=/
DEVICE=fusectl

MOUNTPOINT=/dev/pts
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=noexec,nosuid,gid=tty,mode=0620
TYPE=devpts
UPSTART_EVENTS=mounted
PWD=/
DEVICE=none

MOUNTPOINT=/sys/kernel/debug
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=optional
TYPE=debugfs
UPSTART_EVENTS=mounted
PWD=/
DEVICE=none

MOUNTPOINT=/sys/kernel/security
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=optional
TYPE=securityfs
UPSTART_EVENTS=mounting
PWD=/
DEVICE=none

MOUNTPOINT=/sys/kernel/security
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=optional
TYPE=securityfs
UPSTART_EVENTS=mounted
PWD=/
DEVICE=none

MOUNTPOINT=/dev/shm
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=nosuid,nodev
TYPE=tmpfs
UPSTART_EVENTS=mounting
PWD=/
DEVICE=none

MOUNTPOINT=/dev/shm
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=nosuid,nodev
TYPE=tmpfs
UPSTART_EVENTS=mounted
PWD=/
DEVICE=none

MOUNTPOINT=/var/run
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=mode=0755,nosuid,showthrough
TYPE=tmpfs
UPSTART_EVENTS=mounting
PWD=/
DEVICE=none

MOUNTPOINT=/var/run
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=mode=0755,nosuid,showthrough
TYPE=tmpfs
UPSTART_EVENTS=mounted
PWD=/
DEVICE=none

MOUNTPOINT=/var/lock
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=nodev,noexec,nosuid,showthrough
TYPE=tmpfs
UPSTART_EVENTS=mounting
PWD=/
DEVICE=none

MOUNTPOINT=/var/lock
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=nodev,noexec,nosuid,showthrough
TYPE=tmpfs
UPSTART_EVENTS=mounted
PWD=/
DEVICE=none

MOUNTPOINT=/lib/init/rw
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=mode=0755,nosuid,optional
TYPE=tmpfs
UPSTART_EVENTS=mounted
PWD=/
DEVICE=none

UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
UPSTART_EVENTS=virtual-filesystems
PWD=/

UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
UPSTART_EVENTS=remote-filesystems
PWD=/

MOUNTPOINT=none
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=sw
TYPE=swap
UPSTART_EVENTS=mounting
PWD=/
DEVICE=/dev/disk/by-uuid/b67802dc-35f9-4153-9957-ef04c7af6a1f

MOUNTPOINT=none
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=sw
TYPE=swap
UPSTART_EVENTS=mounted
PWD=/
DEVICE=/dev/disk/by-uuid/b67802dc-35f9-4153-9957-ef04c7af6a1f

UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
UPSTART_EVENTS=all-swaps
PWD=/

MOUNTPOINT=/
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=errors=remount-ro
TYPE=ext4
UPSTART_EVENTS=mounting
PWD=/
DEVICE=/dev/disk/by-uuid/b68c4bc0-6342-411c-878a-a576b3a255b3

MOUNTPOINT=/
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=errors=remount-ro
TYPE=ext4
UPSTART_EVENTS=mounted
PWD=/
DEVICE=/dev/disk/by-uuid/b68c4bc0-6342-411c-878a-a576b3a255b3

MOUNTPOINT=/tmp
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=defaults
TYPE=none
UPSTART_EVENTS=mounting
PWD=/
DEVICE=none

MOUNTPOINT=/tmp
UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
OPTIONS=defaults
TYPE=none
UPSTART_EVENTS=mounted
PWD=/
DEVICE=none

UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
UPSTART_EVENTS=local-filesystems
PWD=/

UPSTART_INSTANCE=
UPSTART_JOB=get_mountall
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
UPSTART_EVENTS=filesystem
PWD=/

9.2   Bridges

Bridges react to events from some other (non-Upstart) source and create corresponding Upstart events.

<subsystem>-device-<action>

for subsystem in /sys/class/*
do
  for action in added changed removed
  do
    echo "${subsystem}-device-${action}"
  done
done

# udevadm info --export-db

$ udevadm monitor --environment

start on (graphics-device-added or drm-device-added)

start on (graphics-device-added or drm-device-added)
script
  echo "`env`" > /dev/.initramfs/job-A.log
end script

DEV_LOG=3
DEVNAME=/dev/fb0
UPSTART_INSTANCE=
ACTION=add
SEQNUM=1176
MAJOR=29
KERNEL=fb0
DEVPATH=/devices/platform/efifb.0/graphics/fb0
UPSTART_JOB=job-A
TERM=linux
SUBSYSTEM=graphics
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
MINOR=0
UPSTART_EVENTS=graphics-device-added
PWD=/
PRIMARY_DEVICE_FOR_DISPLAY=1

ID_BUS=pci
UDEV_LOG=3
UPSTART_INSTANCE=
ID_VENDOR_FROM_DATABASE=Realtek Semiconductor Co., Ltd.
ACTION=add
SEQNUM=1171
MATCHADDR=52:54:00:12:34:56
IFINDEX=2
KERNEL=eth0
DEVPATH=/devices/pci0000:00/0000:00:03.0/net/eth0
UPSTART_JOB=job-A
TERM=linux
SUBSYSTEM=net
ID_MODEL_ID=0x8139
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
ID_MM_CANDIDATE=1
ID_MODEL_FROM_DATABASE=RTL-8139/8139C/8139C+
UPSTART_EVENTS=net-device-added
INTERFACE=eth0
PWD=/
MATCHIFTYPE=1
ID_VENDOR_ID=0x10ec

DEV_LOG=3
DEVNAME=/dev/input/event12
UPSTART_INSTANCE=
ACTION=add
SEQNUM=2689
XKBLAYOUT=gb
MAJOR=13
ID_INPUT=1
KERNEL=event12
DEVPATH=/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/input/input33/event12
UPSTART_JOB=test_camera
TERM=linux
DEVLINKS=/dev/char/13:76 /dev/input/by-path/pci-0000:00:1d.0-event
SUBSYSTEM=input
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
MINOR=76
DISPLAY=:0.0
ID_INPUT_KEY=1
ID_PATH=pci-0000:00:1d.0
UPSTART_EVENTS=input-device-added
PWD=/

10.1   List All Jobs

To list all jobs on the system along with their states, run:

$ initctl list

See initctl.

10.2   List All Jobs With No stop on Condition

# list all jobs (stopped and running instances), and compact down
# to actual job names.
initctl list | awk '{print $1}' | sort -u | while read job
do
  # identify jobs with no "stop on"
  initctl show-config -e $job | grep -q "^  stop on" || echo "$job"
done

10.3   List All Events That Jobs Are Interested In On Your System

Here is another example of how initctl show-config can be useful:

initctl show-config -e | egrep -i "(start|stop) on" | awk '{print $3}' | sort -u

10.4   Create an Event

To create, or "emit" an event, use initctl(8) specifying the emit command.

For example, to emit the hello event, you would run:

# initctl emit hello

This event will be "broadcast" to all Upstart jobs.

If you are creating a job configuration file for a new application, you probably do not need to do this though, since Upstart emits events on behalf of a job whenever the job changes state.

A simple configuration file like that shown below may suffice for your application:

# /etc/init/myapp.conf
description "run my app under Upstart"
task
exec /path/to/myapp

10.5   Create an Event Alias

Say you have an event, but want to create a different name for it, you can simulate a new name by creating a new job which:

• has a start on that matches the event you want to "rename"
• is a task
• emits the new name for the event

For example, if you wanted to create an alias for a particular flavour of the runlevel event called "shutdown" which would be emitted when the system was shutdown, you could create a job configuration file called /etc/init/shutdown.conf containing:

start on runlevel RUNLEVEL=0
task
exec initctl emit shutdown

Note that this isn't a true alias since:

• there are now two events which will be generated when the system is shutting down:
  • runlevel RUNLEVEL=0
  • shutdown
• the two events will be delivered by Upstart at slightly different times (shutdown will be emitted just fractionally before runlevel RUNLEVEL=0).

However, the overall result might suffice for your purposes such that you could create a job configuration file like the following which will run (and complete) just before your system changes to runlevel 0 (in other words halts):

start on shutdown
task
exec backup_my_machine.sh

10.6   Synchronisation

Upstart is very careful to ensure when a condition becomes true that it starts all relevant jobs in sequence (see Order in Which Jobs Which start on the Same Event are Run). However, although Upstart has started them one after another they might still be running at the same time. For example, assume the following:

• /etc/init/X.conf

  start on event-A
  script
    echo "`date`: $UPSTART_JOB started" >> /tmp/test.log
    sleep 2
    echo "`date`: $UPSTART_JOB stopped" >> /tmp/test.log
  end script
  

• /etc/init/Y.conf

  start on event-A
  
  script
    echo "`date`: $UPSTART_JOB started" >> /tmp/test.log
    sleep 2
    echo "`date`: $UPSTART_JOB stopped" >> /tmp/test.log
  end script
  

• /etc/init/Z.conf

  start on event-A
  
  script
    echo "`date`: $UPSTART_JOB started" >> /tmp/test.log
    sleep 2
    echo "`date`: $UPSTART_JOB stopped" >> /tmp/test.log
  end script
  

Running the following will cause all the jobs above to run in some order:

# initctl emit event-A

Here is sample output of /tmp/test.log:

Thu Mar 31 10:20:44 BST 2011: Y started
Thu Mar 31 10:20:44 BST 2011: X started
Thu Mar 31 10:20:44 BST 2011: Z started
Thu Mar 31 10:20:46 BST 2011: Y stopped
Thu Mar 31 10:20:46 BST 2011: Z stopped
Thu Mar 31 10:20:46 BST 2011: X stopped

There are a few points to note about this output:

• All jobs start "around the same time" but are started sequentially.
• The order the jobs are initiated by Upstart cannot be predicted.
• All three jobs are running concurrently.

It is possible with a bit of thought to create a simple framework for synchronisation. Take the following job configuration file /etc/init/synchronise.conf:

manual

This one-line Abstract Job configuration file is extremely interesting in that:

• Since it includes the manual keyword, a job created from it can only be started manually.
• Only a single instance of a job created from this configuration can exist (since no instance stanza has been specified).

What this means is that we can use a job based on this configuration as a simple synchronisation device.

The astute reader may observe that synchronise has similar semantics to a POSIX pthread condition variable.

Now we have our synchronisation primitive, how do we use it? Here is an example which we'll call /etc/init/test_synchronise.conf:

start on stopped synchronise

# allow multiple instances
instance $N

# this is not a service
task

pre-start script
  # "lock"
  start synchronise || true
end script

script
  # do something here, knowing that you have exclusive access
  # to some resource that you are using the "synchronise"
  # job to protect.
  echo "`date`: $UPSTART_JOB ($N) started" >> /tmp/test.log
  sleep 2
  echo "`date`: $UPSTART_JOB ($N) stopped" >> /tmp/test.log
end script

post-stop script
  # "unlock"
  stop synchronise || true
end script

For example, to run 3 instances of this job, run:

for n in $(seq 3)
do
  start test_synchronise N=$n
done

Here is sample output of /tmp/test.log:

Thu Mar 31 10:32:20 BST 2011: test_synchronise (1) started
Thu Mar 31 10:32:22 BST 2011: test_synchronise (1) stopped
Thu Mar 31 10:32:22 BST 2011: test_synchronise (2) started
Thu Mar 31 10:32:24 BST 2011: test_synchronise (2) stopped
Thu Mar 31 10:32:25 BST 2011: test_synchronise (3) started
Thu Mar 31 10:32:27 BST 2011: test_synchronise (3) stopped

The main observation here:

• Each instance of the job started and stopped before any other instance ran.

Like condition variables, this technique require collaboration from all parties. Note that you cannot know the order in which each instance of the test_synchronise job will run.

Note too that it is not necessary to use instances here. All that is required is that your chosen set of jobs all collaborate in their handling of the "lock". Instances make this simple since you can spawn any number of jobs from a single "template" job configuration file.

10.7   Determine if Job was Started by an Event or by "start"

A job that specifies a start on condition can be started in two ways:

• by Upstart itself when the start on condition becomes true.
• by running, "start <job>".

Interestingly, it is possible for a job to establish how it was started by considering the UPSTART_EVENTS variable:

• If the UPSTART_EVENTS variable is set in the job environment, the job was started by an event.
• If the UPSTART_EVENTS variable is not set in the job environment, the job was started by the start command.

Note that this technique does not allow you to determine definitively if the job was started manually by an Administrator since it is possible that if the UPSTART_EVENTS variable is not set that the job was started by another job calling start inside a script section.

10.8   Stop a Job from Running if A pre-start Condition Fails

If you wish a job to not be run if a pre-start condition fails:

pre-start script
  # main process will not be run if /some/file does not exist
  test -f /some/file || { stop ; exit 0; }
end script

script
  # main process is run here
end script

10.9   Run a Job Only When an Event Variable Matches Some Value

By default, Upstart will run your job if the start on condition matches the events listed:

start on event-A

But if event-A provides a number of environment variables, you can restrict your job to starting only when one or more of these variables matches some value. For example:

start on event-A FOO=hello BAR=wibble

Now, Upstart will only run your job if all of the following are true:

• the event-A is emitted
• the value of the $FOO variable in event-A's environment is "hello".
• the value of the $BAR variable in event-A's environment is "wibble".

10.10   Run a Job when an Event Variable Does Not Match Some Value

Upstart supports negation of environment variable values such that you can say:

start on event-A FOO=hello BAR!=wibble

Now, Upstart will only run your job if all of the following are true:

• the event-A is emitted
• the value of the $FOO variable in event-A's environment is "hello".
• the value of the $BAR variable in event-A's environment is not "wibble".

10.11   Run a Job as Soon as Possible After Boot

(Note: we ignore the initramfs in this section).

To start a job as early as possible, simply "start on" the startup event. This is the first event Upstart emits and all other events and jobs follow from this:

start on startup

10.12   Run a Job When a User Logs in Graphically ()

Assuming a graphical login, this can be achieved using a start on condition of:

start on desktop-session-start

This requires the display manager emit the event in question. See the upstart-events(7) man page on an Ubuntu system for the 2 events a Display Manager is expected to emit. If your Display Manager does not emit these event, check its documentation to see if it allows scripts to be called at appropriate points and then you can easily conform to the reference implementations behaviour:

# A user has logged in
/sbin/initctl -q emit desktop-session-start \
  DISPLAY_MANAGER=some_name USER=$USER

# Display Manager has initialized and displayed a login screen
# (if appropriate)
/sbin/initctl -q emit login-session-start \
  DISPLAY_MANAGER=some_name

10.13   Run a Job When a User Logs in

This makes use of D-Bus Service Activation.

1. Add "UpstartJob=true" to file "/usr/share/dbus-1/system-services/org.freedesktop.ConsoleKit.service".

2. Create a job configuration file corresponding to the D-Bus service, say /etc/init/user-login.conf [11]:

   start on dbus-activation org.freedesktop.ConsoleKit
   exec /usr/sbin/console-kit-daemon --no-daemon
   

3. Ensure that the D-Bus daemon ("dbus-daemon") is started with the --activation=upstart option (see /etc/init/dbus.conf).

Now, when a user logs in, D-Bus will emit the dbus-activation event, specifying the D-Bus service started. You can now create other jobs that start on user-login.

UPSTART_INSTANCE=
DBUS_STARTER_BUS_TYPE=system
UPSTART_JOB=user-login
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
SERVICE=org.freedesktop.ConsoleKit
DBUS_SYSTEM_BUS_ADDRESS=unix:path=/var/run/dbus/system_bus_socket,guid=e86f5a01fbb7f5f1c22131090000000a
UPSTART_EVENTS=dbus-activation
PWD=/
DBUS_STARTER_ADDRESS=unix:path=/var/run/dbus/system_bus_socket,guid=e86f5a01fbb7f5f1c22131090000000a

10.14   Run a Job For All of a Number of Conditions

If you have a job configuration file like this:

start on (event-A or (event-B or event-C))

script
  echo "`date`: ran in environment: `env`" >> /tmp/myjob.log
end script

Upstart will run this job when any of the following events is emitted:

• event-A
• event-B
• event-C

You cannot know the order in which the events will arrive in, but the specified start on condition has told Upstart that any of them will suffice for your purposes. So, if event-B is emitted first, Upstart will run the job and only consider re-running the job if and when the job has finished running. If event-B is emitted and the job is running and then (before the job finishes running) event-A is emitted, the job will not be re-run.

However, what if you wanted to run the script for all the events? If you know that all of these events will be emitted at some point, you could change the start on to be:

start on (event-A and (event-B and event-C))

Here, the job will only run at the time when the last of the three events is received.

Is it possible to run this job for each event as soon as each event arrives? Yes it is:

start on (event-A or (event-B or event-C))

instance $UPSTART_EVENTS

script
  echo "`date`: ran in environment: `env`" >> /tmp/myjob.log
end script

By adding the instance keyword, you ensure that whenever any of the events listed in your start on condition is emitted, an instance of the job will be run. Therefore, if all three events are emitted very close together in time, three jobs instances will now be run.

See the Instance section for further details.

10.15   Run a Job Before Another Job

If you wish to run a particular job before some other job, simply make your jobs start on condition specify the starting(7) event. Since the starting(7) event is emitted just before the job in question starts, this provides the behaviour you want since your job will be run first.

For example, assuming your job is called job-B and you want it to start before job-A, in /etc/init/job-B.conf you would specify:

start on starting job-A

10.16   Run a Job After Another Job

If you have a job you wish to run after job "job-A", your start on condition would need to make use of the stopped(7) event like this:

start on stopped job-A

10.17   Run a Job Once After Some Other Job Ends

Imagine a job configuration file myjob.conf such as the following which might result in a job which is restarted a number of times:

start on event-A

script
  # do something
end script

Is it possible to run a job only once after job myjob ends? Yes if you create a job configuration file myjob-sync.conf such as:

start on stopped myjob and event-B

script
  # do something
end script

Now, when event-A is emitted, job myjob will start and if and when job myjob finishes and event event-B is emitted, job myjob-sync will be run.

However, crucially, even if job myjob is restarted, the myjob-sync job will not be restarted.

10.18   Run a Job Before Another Job and Stop it After that Job Stops

If you have a job you wish to be running before job "job-A" starts, but which you want to stop as soon as job-A stops:

start on starting job-A
stop on stopped job-A

10.19   Run a Job Only If Another Job Succeeds

To have a job start only when job-A succeeds, use the $RESULT variable from the stopped(7) event like this:

start on stopped job-A RESULT=ok

10.20   Run a Job Only If Another Job Fails

To have a job start only when job-A fails, use the $RESULT variable from the stopped(7) event like this:

start on stopped job-A RESULT=failed

Note that you could also specify this condition as:

start on stopped job-A RESULT!=ok

10.21   Run a Job Only If One Job Succeeds and Another Fails

This would be a strange scenario to want, but it is quite easy to specify. Assuming we want a job to start only if job-A succeeds and if job-B fails:

start on stopped job-A RESULT=ok and stopped job-B RESULT=failed

10.22   Run a Job If Another Job Exits with a particular Exit Code

Imagine you have a database server process that exits with a particular exit code (say 7) to denote that it needs some sort of cleanup process to be run before it can be re-started. To handle this you could create /etc/init/mydb-cleanup.conf with a start on condition like this:

start on stopped mydb EXIT_STATUS=7

script
  # handle cleanup...

  # assuming the cleanup was successful, restart the server
  start mydb
end script

10.23   Use Details of a Failed Job from Another Job

Although you cannot see the exact environment another job ran in, you can access some details. For example, if your job specified /etc/init/job-B.conf as:

start on stopped job-A RESULT=fail

script
  exec 1>>/tmp/log.file
  echo "Environment of job $JOB was:"
  env
  echo
end script

The file /tmp/log.file might contain something like this:

UPSTART_INSTANCE=
EXIT_STATUS=7
INSTANCE=
UPSTART_JOB=B
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
PROCESS=main
UPSTART_EVENTS=stopped
PWD=/
RESULT=failed
JOB=A

Here, job-B can see that:

• job-A exited in its "main" process. This is a special name for the script section. All other script sections are named as expected. For example, if the pre-start section had failed, the PROCESS variable would be set to pre-start, and if in post-stop, the variable would have been set to post-stop.
• job-A exited with exit code 7.
• job-A only had 1 instance (since the INSTANCE variable is set to the null value.
• job-A ran in the root ("/") directory.
• UPSTART_JOB is the name of the job running the script (ie job-B).
• JOB is the name of the job that we are starting on (here job-A).
• UPSTART_EVENTS is a list of the events that caused UPSTART_JOB (ie job-B) to start. Here, the event is starting(7) showing that job-B started as a result of job-A being sent the stopped(7) event.

10.24   Stop a Job when Another Job Starts

If we wish job-A to stop when job-B starts, specify the following in /etc/init/job-A.conf:

stop on starting job-B

# /etc/init/job-A.conf
start on stopped job-B

script
  # do something when job-B is stopped
end script

# /etc/init/job-B.conf
start on stopped job-A

script
  # do something when job-A is stopped
end script

# start job-A

10.25   Run a Job Periodically

This cannot currently be handled by Upstart directly. However, the "Temporal Events" feature is being worked on now will address this.

Until Temporal Events are available you should either use cron(8), or something like:

# /etc/init/timer.conf

instance $JOB_TO_RUN

script
  for var in SLEEP JOB_TO_RUN
  do
    eval val=\${$var}
    if [ -z "$val" ]
    then
      logger -t $0 "ERROR: variable $var not specified"
      exit 1
    fi
  done

  eval _sleep=\${SLEEP}
  eval _job=\${JOB_TO_RUN}

  while [ 1 ]
  do
    stop  $_job || true
    sleep $_sleep
    start $_job || true
  done
end script

Note well the contents of the while loop. We ensure that the commands that might fail are converted into expressions guaranteed to pass. If we did not do this, timer.conf would fail, which would be undesirable. Note too the use of instance to allow more than one instance of the timer job to be running at any one time.

10.26   Migration from System V initialization scripts

With SysV init scripts, the Administrator decides the order that jobs are started in by assigning numeric values to each service. Such a system is simple, but non-optimal since:

• The SysV init system runs each job sequentially.

  This disallows running jobs in parallel, to make full use of system resources. Due to the limited nature of the SysV system, many SysV services put services that take a long time to start into the background to give the illusion that the boot is progressing quickly. However, this makes it difficult for Administrators to know if a required service is running by the time their later service starts.

• The Administrator cannot know the best order to run jobs in.

  Since the only meta information encoded for services is a numeric value used purely for ordering jobs, the system cannot optimize the services since it knows nothing about the requirements for each job.

In summary, the SysV init system is designed to be easy for the Administrator to use, not easy for the system to optimize.

In order to migrate a service from SysV to Upstart, it is necessary to change your mindset somewhat. Rather than trying to decide which two services to "slot" your service between, you need to consider the conditions that your service needs before it can legitimately be started.

So, if you wished to add a new service that traditionally started before cron(8) or atd(8) you do not need to change the configuration files cron.conf or atd.conf. You can "insert" your new service by specifying a simple:

# /etc/init/my-service.conf
start on (starting cron or starting atd)

In English, this says,

"start the "my-service" service just before either the cron or the atd services start".

Whether crond or atd actually start first is not a concern for my-service: Upstart ensures that the my-service service will be started before either of them. Even if cron normally starts before atd but for some reason one day atd starts first, Upstart will ensure that my-service will be started before atd.

Note therefore that introducing a new service should not generally require existing job configuration files to be updated.

10.27   How to Establish a Jobs start on and stop on Conditions

How do you establish what values you should specify for a jobs start on and stop on conditions?

stop on runlevel [016]

10.28   Guarantee that a job will only run once

If you have a job which must only be run once, but which depends on multiple conditions, the naive approach won't necessarily work:

task
start on (A or B)

If event 'A' is emitted, the task will run. But assuming the task has completed and event 'B' is then emitted, the task will run again.

task
start on startup and (A or B)

10.29   Stop a Job That is About to Start

Upstart will start a job when its "start on" condition becomes true.

Although somewhat unusual, it is quite possible to stop a job from starting when Upstart tries to start it:

start on starting job-A

script
  stop $JOB
end script

10.30   Stop a Job That is About to Start From Within That Job

You can in fact stop a job that Upstart has decided it needs to start from within that job:

pre-start script
  stop
end script

This is actually just an alias for:

pre-start script
  stop $UPSTART_JOB
end script

Of course, you could set the pre-start using the Override Files facility.

10.31   Stop a Job When Some Other Job is about to Start

Here, we create /etc/init/job-C.conf which will stop job-B when job-A is about to start:

start on starting job-A

script
  stop job-B
end script

10.32   Start a Job when a Particular Filesystem is About to be Mounted

Here, we start a job when the /apps mountpoint is mounted read-only as an NFS-v4 filesystem:

start on mounting TYPE=nfs4 MOUNTPOINT=/apps OPTION=ro

Here's another example:

start on mounted MOUNTPOINT=/var/run TYPE=tmpfs

Another example where a job would be started when any non-virtual filesystem is mounted:

start on mounted DEVICE=[/UL]*

The use of the $DEVICE variable is interesting. It is used here to specify succinctly any device that:

• is a real device (starts with "/" (to denote a normal "/dev/..." mount)).
• is a device specified by its filesystem:
  • label (starts with "L" (to denote a "LABEL=" mount)).
  • UUID (starts with "U" (to denote a "UUID=" mount)).

Another example where a job is started when a non-root filesystem is mounted:

start on mounting MOUNTPOINT!=/ TYPE!=swap

10.33   Start a Job when a Device is Hot-Plugged

Hot-plug kernel events create udev(7) events under Linux and Upstart events are created from udev events by the upstart-udev-bridge(8).

Added to this the ifup and ifdown commands are run at boot when network devices are available for use.

start on net-device-added INTERFACE=eth0

start on net-device-up IFACE=eth0

10.34   Stopping a Job if it Runs for Too Long

To stop a running job after a certain period of time, create a generic job configuration file like this:

# /etc/init/timeout.conf
stop on stopping JOB=$JOB_TO_WAIT_FOR
kill timeout 1
manual

export JOB_TO_WAIT_FOR
export TIMEOUT

script
  sleep $TIMEOUT
  initctl stop $JOB_TO_WAIT_FOR
end script

Now, you can control a job using a timeout:

start myjob
start timeout JOB_TO_WAIT_FOR=myjob TIMEOUT=5

This will start job myjob running and then wait for 5 seconds. If job "myjob" is still running after this period of time, the job will be stopped using the initctl(8) command. Note the stop on stanza which will cause the timeout job not to run if the job being waited for has already started to stop.

10.35   Run a Job When a File or Directory is Created/Deleted

If you need to start a Job only when a certain file is created, you could create a generic job configuration file such as the following:

# /etc/init/wait_for_file.conf
instance FILE_PATH
export   FILE_PATH

script
  while [ ! -e "$FILE_PATH" ]
  do
    sleep 1
  done

  initctl emit file FILE_PATH="$FILE_PATH"
end script

Having done this, you can now make use of it. To have another job start if say file /var/run/foo.dat gets created, you first need to create a job configuration file stating this:

# /etc/init/myapp.conf
start on file FILE_PATH=/var/run/foo.dat

script
  # ...
end script

Lastly, kick of the process by starting an instance of wait_for_file:

start wait_for_file FILE_PATH=/var/run/foo.dat

Now, when file /var/run/foo.dat is created, the following will happen:

1. The myapp job will emit the file event, passing the path of the file which you just specified in that events environment.
2. Upstart will see that the start on condition for the myapp job configuration file is satisfied.
3. Upstart will create a myapp job, and start it.

You can modify this strategy slightly to run a job when a file is:

• modified
• deleted
• contains certain content
• et cetera

See test(1), or your shells documentation for available file tests.

Note that this is very simplistic. A better approach would be to use inotify(7).

10.36   Run a Job Each Time a Condition is True

This is the default way Upstart works when you have defined a task:

# /etc/init/myjob.conf
task
exec /some/program
start on (A or B)

Job "myjob" will run every time either event 'A' or event 'B' are emitted. However, there is a corner condition: if event 'A' has been emitted and the task is currently running when event 'B' is emitted, job "myjob" will not be run. To avoid this situation, use instances:

# /etc/init/myjob2.conf
task
instance $SOME_VARIABLE
exec /some/program
start on (A or B)

Now, as long variable $SOME_VARIABLE is defined with a unique value each time either event 'A' or 'B' is emitted, Upstart will run job "myjob2" multiple times.

10.37   Run a Job When a Particular Runlevel is Entered and Left

To run a job when a particular runlevel is entered and also run it when that same runlevel is left, you could specify:

start on runlevel RUNLEVEL=5 or runlevel PREVLEVEL=5

See runlevel(7) and the Runlevels section for more details.

10.38   Pass State From a Script Section to its Job Configuration File

Assume you have a job configuration file like this:

script
   # ...
end script

exec /bin/some-program $ARG

How can you get the script section to set $ARG and have the job configuration file use that value in the "exec" stanza? This isn't as easy as you might imagine for the simple reason that Upstart runs the script section in a new process. As such, by the time Upstart gets to the exec stanza the process spawned to handle the script section has now ended. This implies they cannot communicate directly.

A way to achieve the required goal is as follows:

# set a variable which is the name of a file this job will use
# to pass information between script sections.
env ARG_FILE="/var/myapp/myapp.dat"

# make the variable accessible to all script sections (ie sub-shells)
export ARG

pre-start script
   # decide upon arguments and write them to
   # $ARG_FILE, which is available in this sub-shell.
end script

script
  # read back the contents of the arguments file
  # and pass the values to the program to run.
  ARGS="$(cat $ARG_FILE)"
  exec /bin/some-program $ARGS
end script

10.39   Pass State From Job Configuration File to a Script Section

To pass a value from a job configuration file to one of its script sections, simply use the env stanza:

env CONF_FILE=/etc/myapp/myapp.cfg

script
  exec /bin/myapp -c $CONF_FILE
end script

This example is a little pointless, but the following slightly modified example is much more useful:

start on an-event
export CONF_FILE

script
  exec /bin/myapp -c $CONF_FILE
end script

By dropping the use of the env stanza we can now pass the value in via an event:

# initctl emit an-event CONF_FILE=/etc/myapp/myapp.cfg

This is potentially much more useful since the value passed into myapp.conf can be varied without having to modify the job configuration file.

10.40   Run a Job as a Different User

exec start-stop-daemon --start -c myuser --exec command

script
  exec su -c command myuser
end script

script
  exec sudo -u myuser command
end script

10.41   Disabling a Job from Automatically Starting

With Upstart 0.6.7, to stop Upstart automatically starting a job, you can either:

• Rename the job configuration file such that it does not end with ".conf".
• Edit the job configuration file and comment out the "start on" stanza using a leading '#'.

To re-enable the job, just undo the change.

# echo "manual" >> /etc/init/myjob.override

# echo "manual" >> /etc/init/myjob.conf

10.42   Jobs that "Run Forever"

To create a job that runs continuously from the time it is manually started(7) until the time it is manually stopped(7), create a job configuration file without any process definition (exec and script) or event definition (start on for example) stanzas:

# /etc/init/runforever.conf
description "job that runs until stopped manually"

This job can only be started by the administrator running:

# start runforever

The status of this job will now be "start/running" until the administrator subsequently runs:

# stop runforever

These "Abstract Job" types have other uses as covered in other parts of this document. See for example Synchronisation.

10.43   Run a Java Application

Running a Java application is no different to any other, but Java suffers from the inability to switch users without extra helper classes.

If your Java daemon needs to run as a different user and you are running Upstart 1.4, you can use the setuid and setgid stanzas.

However, if you are using an older version, you will have to use a facility such as su(1). Also, you may wish to define some variables to simplify the invocation:

env ROOT_DIR=/apps/myapp
env HTTP_PORT=8080
env USER=java_user
env JAVA_HOME=/usr/lib/jvm/java-6-openjdk
env JVM_OPTIONS="-Xms64m -Xmx256m"
env APP_OPTIONS="--httpPort=$HTTP_PORT"
env LOGFILE=/var/log/myapp.log

script
  exec su -c "$JAVA_HOME/bin/java $JVM_OPTIONS \
    -jar $ROOT_DIR/myjar.jar $APP_OPTIONS > $LOGFILE 2>&1" $USER
end script

You should read the Changing User section section before using this technique though.

exec start-stop-daemon --start --exec $JAVA_HOME/bin/java \
  -- $JAVA_OPTS -jar $SOMEWHERE/file.war

10.44   Ensure a Directory Exists Before Starting a Job

This is a good use of the pre-start stanza:

env DIR=/var/run/myapp
env USER=myuser
env GROUP=mygroup
env PERMS=0755

pre-start script
  mkdir $DIR             || true
  chmod $PERMS $DIR      || true
  chown $USER:$GROUP DIR || true
end script

10.45   Run a GUI Application

To have Upstart start a GUI application, you first need to ensure that the user who will be running it has access to the X display. This is achieved using the xhost command.

Once the user has access, the method is the same as usual:

env DISPLAY=:0.0
exec xclock -update 1

10.46   Run an Application through GNU Screen

If you want Upstart to create a GNU Screen (or Byobu) session to run your application in, this is equally simple:

exec su myuser -c "screen -D -m -S MYAPP java -jar MyApp.jar"

10.47   Run Upstart in a chroot Environment

dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

10.48   Record all Jobs and Events which Emit an Event

For example, if you want to record all jobs which emit a started event:

# /etc/init/debug.conf
start on started
script
  exec 1>>/tmp/log.file
  echo "$0:$$:`date`:got called. Environment of job $JOB was:"
  env
  echo
end script

You could also log details of all jobs (except the debug job itself) which are affected by the main events:

# /etc/init/debug.conf
start on ( starting JOB!=debug \
  or started JOB!=debug \
  or stopping JOB!=debug \
  or stopped JOB!=debug )
script
  exec 1>>/tmp/log.file
  echo -n "$UPSTART_JOB/$UPSTART_INSTANCE ($0):$$:`date`:"
  echo    "Job $JOB/$INSTANCE $UPSTART_EVENTS. Environment was:"
  env
  echo
end script

Note that the $UPSTART_JOB and $UPSTART_INSTANCE environment variables refer to the debug job itself, whereas $JOB and $INSTANCE refer to the job which the debug job is triggered by.

10.49   Integrating your New Application with Upstart

Integrating your application into Upstart is actually very simple. However, you need to remember that Upstart is NOT "System V" (aka "SysV"), so you need to think in a different way.

With SysV you slot your service script between other service scripts by specifying a startup number. The SysV init system then runs each script in numerical order. This is very simple to understand and use, but highly inefficient in practical terms since it means the boot cannot be parallelised and thus cannot be optimized.

10.50   Block Another Job Until Yours has Started

It is common that a particular piece of software, when installed, will need to be started before another. The logical conclusion is to use the 'starting' event of the other job:

start on starting foo

This will indeed, block foo from starting until our job has started.

But what if we have multiple events that we need to delay:

start on starting foo or starting network-services

This would seem to make sense. However, if we have a time-line like this:

starting foo
starting our job
starting network-services
started network-services

Network-services will actually NOT be blocked. This is because upstart only blocks an event if that event causes change in the goal of the service. So, we need to make sure upstart waits every time. This can be done by using a "wait job":

# myjob-wait
start on starting foo or starting network-services
stop on started myjob or stopped myjob
instance $JOB
normal exit 2
task
script
  status myjob | grep -q 'start/running' && exit 0
  start myjob || :
  sleep 3600
end script

This is a bit of a hack to get around the lack of state awareness in Upstart. Eventually this should be built in to upstart. The job above will create an instance for each JOB that causes it to start. It will try and check to see if it's already running, and if so, let the blocked job go with exit 0. If it's not running, it will set the ball in motion for it to start. By doing this, we make it very likely that the stopped or started event for myjob will be emitted (the only thing that will prevent this, is a script line in 'myjob' that runs 'stop'). Because we know we will get one of those start or stopped events, we can just sleep for an hour waiting for upstart to kill us when the event happens.

10.51   Controlling Upstart using D-Bus

Upstart contains its own D-Bus server which means that initctl and any other D-Bus application can control Upstart. The examples below use dbus-send, but any of the D-Bus bindings could be used.

$ dbus-send --system --print-reply --dest=com.ubuntu.Upstart /com/ubuntu/Upstart com.ubuntu.Upstart0_6.GetAllJobs

$ job=myjob
$ dbus-send --system --print-reply --dest=com.ubuntu.Upstart /com/ubuntu/Upstart/jobs/${job}/_ org.freedesktop.DBus.Properties.GetAll string:''

# job=myjob
# dbus-send --system --print-reply --dest=com.ubuntu.Upstart /com/ubuntu/Upstart/jobs/${job} com.ubuntu.Upstart0_6.Job.Start array:string: boolean:true

# job=myjob
# dbus-send --system --print-reply --dest=com.ubuntu.Upstart /com/ubuntu/Upstart/jobs/${job} com.ubuntu.Upstart0_6.Job.Stop array:string: boolean:true

# job=myjob
# dbus-send --system --print-reply --dest=com.ubuntu.Upstart /com/ubuntu/Upstart/jobs/${job} com.ubuntu.Upstart0_6.Job.Restart array:string: boolean:true

10.52   Establish Blocking Job

Image you have just run the following command and it has "blocked" (appeared to hang):

# initctl emit event-A

The reason for the block is that the event-A event changes the goal of "some job", and until the goal has changed, the initctl command will block.

But which job is being slow to change goal? It is now possible to hone in on the problem using initctl show-config in a script such as this:

#!/bin/sh
# find_blocked_job.sh

[ $# -ne 1 ] && { echo "ERROR: usage: $0 <event>"; exit 1; }
event="$1"

# obtain a list of jobs (removing instances)
initctl list | awk '{print $1}' | sort -u | while read job
do
  initctl show-config -e "$job" |\
    egrep "(start|stop) on \<event\>" >/dev/null 2>&1
  [ $? -eq 0 ] && echo $job
done

This will return a list of jobs, one per line. One of these will be the culprit. Having identified the problematic job, you can debug using techniques from the Debugging section.

10.53   Determine if a Job is Disabled

To determine if a job has been disabled from starting automatically:

$ job=foo
$ initctl show-config $job | grep -q "^  start on" && echo enabled || echo disabled

10.54   Visualising Jobs and Events

Use the initctl2dot(8) facility. See [24] for further details and examples.

10.55   Sourcing Files

You need to take care when "sourcing" a script or configuration file into a script section for a number of reasons. Suppose we have the following:

script
. /etc/default/myapp.cfg
. /etc/myapp/myapp.cfg
echo hello > /tmp/myapp.log
end script

Assume that file /etc/myapp/myapp.cfg does NOT exist.

script
  files="\
  /etc/default/myapp.cfg
  /etc/myapp/myapp.cfg
  "

  for file in $files
  do
    [ -f "$file" ] && . "$file"
  done
  echo hello > /tmp/myapp.log
end script

10.56   Determining How to Stop a Job with Multiple Running Instances

As explained in the initctl status section, a job that has multiple running instances will show the specific (unique) instance value within brackets:

$ initctl list | grep ^network-interface-security
network-interface-security (network-manager) start/running
network-interface-security (network-interface/eth0) start/running
network-interface-security (network-interface/lo) start/running
network-interface-security (networking) start/running

In the example output above there are four instances of the network-interface-security job running with the unique instances values of:

• "network-manager"
• "network-interface/eth0"
• "network-interface/lo"
• "networking"

So how do we stop one of these jobs? Lets try to work this out without looking at initctl(8) manual page:

# stop network-interface-security network-interface/eth0
stop: Env must be KEY=VALUE pairs

That clearly doesn't work. The problem is that we have provided the value to the instance variable, but we haven't named the instance variable that the given value corresponds to. But how do we establish the instance variable name?

There are 2 options:

• look at the corresponding Job Configuration File.

  /etc/init/network-interface-security.conf in this example.

• Use a trick to get Upstart to tell you the name:

  $ status network-interface-security
  status: Unknown parameter: JOB
  

  This shows us the name of the instance variable is "JOB".

We are now in a position to stop a particular instance of this job:

# stop network-interface-security JOB=network-interface/eth0
network-interface-security stop/waiting

The job instance has now been stopped. To prove it:

# status network-interface-security JOB=network-interface/eth0
status: Unknown instance: network-interface/eth0
# initctl list | grep ^network-interface-security | grep network-interface/eth0
#

10.57   Logging Boot and Shutdown Times

If you want to create a log of when your system starts and stops, you could do something like this:

start on filesystem or runlevel [06]

env log=/var/log/boot-times.log

script
  action=$(echo "$UPSTART_EVENTS" | grep -q filesystem && echo boot || echo shutdown)
  echo "`date`: $action" >> $log
end script

Note that you do not need to specify a stop on condition: you want this job to start both "around" the time of system startup (when the disks are writeable, hence the use of the fileysystem event) and shutdown.

If you want a more accurate method, you would need to have a job start on startup. The slight issue here is that when Upstart emits that first event, there is no guarantee of writeable disks. However, this can be overcome using a bit of thought...

First, create a "record-boot-time.conf" job configuration file to record the time of the "boot" (initial Upstart event):

start on startup

exec initctl emit boot-time TIME=$(date '+%s')

This job emits an event containing a variable specifying the time in seconds since the Epoch.

Now, create a second "log-boot-time.conf" job configuration file to actually log the boot time:

start on boot-time and filesystem

log=/var/log/boot-times.log

script
  echo "system booted at $TIME" >>$log
end script

Since the "log-boot-time" job specifies the "booted" event emitted by the "record-boot-time" job, Upstart will retain knowledge of this event until it is able to run the second job. The "record-boot-time" job can then simply make use of the "TIME" variable set by the first job.

10.58   Running an Alternative Job on a tty

Here's a silly example of how to run a custom job on a particular tty. It asks the user to guess a random number. If after 3 attempts they fail to guess the correct number, the job ends. However, if they guess successfully, the are allowed to login. This won't win any scripting competitions, but you get the idea.

WARNING - DO NOT USE THIS ON A REAL SYSTEM unless you want to get hacked, or fired or both!:

# Get the user to guess the number. If they get it right, let them
# login.

start on runlevel [23]
stop on runlevel [!23]

env tty=tty9

# XXX: Ensure job is connected to the terminal device
console output

script
  # XXX: Ensure all standard streams are connected to the console
  exec 0</dev/$tty >/dev/$tty 2>&1
  clear
  trap '' INT TERM HUP
  RANDOM=$(dd if=/dev/urandom count=1 2>/dev/null|cksum|cut -f1 -d' ')
  answer=$(((RANDOM % 100) + 1))
  attempt=0
  max=3
  got=0

  while [ $attempt -lt $max ]
  do
    attempt=$((attempt+1))
    echo -n "Guess the number (1-100, attempt $attempt of $max): "
    read guess
    if [ "$guess" -eq "$answer" ]
    then
      got=1
      break
    else
      echo "Wrong"
    fi
  done

  [ "$got" = 0 ] && stop

  exec /sbin/getty -8 38400 $tty
end script

The important lines are:

console output

... and:

exec 0</dev/$tty >/dev/$tty 2>&1

11.1   Questions about start on

Consider the following start on condition:

start on startup or starting stopped or stopping started

Questions (answers provided in footnote links):

11.2   General Questions

What is wrong with the following job configuration file?:

start on startup

script
  echo hello > /tmp/foo.log
end script

Answer: [5]

Whas is wrong with the following job configuration file?:

start on runlevel [2345]

env CONFIG=/etc/default/myapp

expect fork
respawn

script
  enabled=$(grep ENABLED=1 $CONFIG)
  [ -z "$enabled" ] && exit 0
  /usr/bin/myapp
end script

Answer: [6]

12.1   Cannot Start a Job

If you have just created or modified a job configuration file such as /etc/init/myjob.conf, but start gives the following error when you attempt to start it:

start: Unknown job: myjob

The likelihood is that the file contains a syntax error. The easiest way to establish if this is true is by running the init-checkconf command.

If you are wondering why the original error couldn't be more helpful, it is important to remember that the job control commands (start, stop and restart) and initctl communicate with Upstart over D-Bus. The problem here is that Upstart rejected the invalid myjob.conf, so attempting to control that job over D-Bus is nonsensical - the job does not exist.

12.2   Cannot stop a job

If start is hanging or seems to be behaving oddly, the chances are you have misspecified the expect stanza. See expect and How to Establish Fork Count.

12.3   Strange Error When Running start/stop/restart or initctl emit

If you attempt to run a job command, or emit an event and you get a D-Bus error like this:

$ start myjob
start: Rejected send message, 1 matched rules; type="method_call", sender=":1.58" (uid=1000 pid=5696 comm="start) interface="com.ubuntu.Upstart0_6.Job" member="Start" error name="(unset)" requested_reply=0 destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init"))

The problem is caused by not running the command as root. To resolve it, either "su -" to root or use a facility such as sudo(8):

# start myjob
myjob start/running, process 1234

The reason for the very cryptic error is that the job control commands (start, stop and restart) and initctl communicate with Upstart over D-Bus.

12.4   The initctl command shows "the wrong PID"

The likelihood is that you have mis-specified the type of application you are running in the job configuration file. Since Upstart traces or follows fork(2) calls, it needs to know how many forks to expect. If your application forks once, specify the following in the job configuration file:

expect fork

However, if your application forks twice (which all daemon processes should do), specify:

expect daemon

See also Alternative Method.

12.5   Symbolic Links don't work in /etc/init

Upstart does not monitor files which are symbolic links since it needs to be able to guarantee behaviour and if a link is broken or cannot be followed (it might refer to a filesystem that hasn't yet been mounted for example), behaviour would be unexpected, and thus undesirable. As such, all system job configuration files must live in or below /etc/init (although user jobs can live in other locations).

12.6   Sometimes status shows PID, but other times does not

You may have noticed that when you start certain jobs manually using start, sometimes the output will show the PID of the process associated with that job. However, other times, no PID is shown. Why?

This behaviour is observed when the job runs to completion very quickly. If your system has minimal load the job will start and finish before the initctl status command has a chance to query its PID from Upstart. Whereas if your system is busy you may well see a PID displayed since Upstart was able to return the PID details to status before the job finished.

The behaviour is similar to the following shell code:

(sleep 0.01 &) ; ps -fU $USER | grep sleep | grep -v grep

It is unlikely that you will get any output from this command (since the sleep 0.01 command will run to completion before the grep(1) calls get a chance filter the ps(1) output. However, change the time for that subshell to run, and you will see the PID:

(sleep 5 &) ; ps -fU $USER | grep sleep | grep -v grep

See initctl status.

15.1   Determining the value of expect

The Expect section explains how to determine the value of the expect stanza. Note that you should not introduce the respawn stanza until you are fully satisfied you have specifed the expect stanza correctly.

15.2   start on and stop on condition

See How to Establish a Jobs start on and stop on Conditions.

15.3   Services

• If your job is a service, identify the correct value for the expect stanza.

  Once you have decided on the correct value:

  1. start the job:

     $ sudo start myjob
     

  2. Check the PID of the job matches the expected PID:

     $ actual_pid=$(pidof myapp)
     $ upstart_pid=$(status myjob | awk '{print $NF}')
     $ [ "$actual_pid" = "$upstart_pid" ] || echo "ERROR: pid "
     

  3. Stop the job:

     $ sudo stop myjob
     

  4. Ensure the PID no longer exists:

     $ [ -z "$(pidof myapp)" ] || echo "ERROR: myapp still running"
     

• Only once you have specified the correct expect stanza should you introduce the respawn stanza since if you introduce it at the outset, this will just confuse your understanding, particulary if the expect stanza has been misspecified.

16.1   Obtaining a List of Events

To obtain a list of events that have been generated by your system, do one of the following:

grep init: /var/log/syslog

grep init: /var/log/kern.log /var/log/syslog

16.2   See the Environment a Job Runs In

To get a log of the environment variables set when Upstart ran a job you can add simple debug to the appropriate script section. For example:

script
  echo "DEBUG: `set`" >> /tmp/myjob.log

  # rest of script follows...
end script

Alternatively you could always have the script log to the system log:

script
  logger -t "$0" "DEBUG: `set`"

  # rest of script follows...
end script

Or, have it pop up a GUI window for you:

env DISPLAY=:0.0

script
  env | zenity --title="got event $UPSTART_EVENTS" --text-info &
end script

16.3   Obtaining a log of a Script Section

This relies on a trick relating to the early boot process on an Ubuntu system. On the first line below script stanza, add:

exec >>/dev/.initramfs/myjob.log 2>&1
set -x

This will ensure that /bin/sh will log its progress to the file named. The location of this file is special in that /dev/.initramfs/ will be available early on in the boot sequence (before the root filesystem has been mounted read-write).

16.4   Checking a Job Configuration File for Syntax Errors

See init-checkconf.

16.5   Check a Script Section for Errors

Upstart runs your job using /bin/sh -e for safety reasons: scripts running as the root user need to be well-written! But how can you check to ensure that your script sections contain valid (syntactically correct at least) shell fragments? Simply run the init-checkconf script, which performs these checks automatically.

$ /bin/sh -n <(sed -n '/^script/,/^end script/p' myjob.conf)

$ /bin/sh -n <(sed -n '/^pre-start script/,/^end script/p' myjob.conf)

#!/bin/sh
# check-upstart-script-sections.sh

[ $# -ne 1 ] && { echo "ERROR: usage: $0 <conf_file>"; exit 1; }
file="$1"

[ ! -f "$file" ] && { echo "ERROR: file $file does not exist" >&2; exit 1; }

for v in pre-start post-start script pre-stop post-stop
do
  if egrep -q "\<${v}\>" $file
  then
    sed -n "/^ *${v}/,/^ *end script/p" $file | \
      sh -n || echo "ERROR in $v section"
  fi
done

$ check-upstart-script-sections.sh myjob.conf

16.6   Debugging a Script Which Appears to be Behaving Oddly

If a script section appears to be behaving in an odd fashion, the chances are that one of the commands is failing. Remember that Upstart runs every script section using /bin/sh -e. This means that if any simple command fails, the shell will exit. For example, if file /etc/does-not-exist.cfg does not exist in the example below the script will exit before the shell runs the if test:

script
  grep foo /etc/does-not-exist.cfg >/dev/null 2>&1
  if [ $? -eq 0 ]
  then
    echo ok
  else
    echo bad
  fi
end script

In other words, you will get no output from this script if the file grep is attempting to operate on does not exist.

The common idiom to handle possible errors of this type is to convert the simple expression into an expression guaranteed to return true:

script
  # ensure this statement always evaluates to true
  command-that-might-fail || true

  # ditto
  another-command || :
end script

See man sh for further details.

17.1   Boot into Recovery Mode

Select the "recovery" option in the Grub boot menu

This assumes that Upstart (init(8) itself) is usable.

Note that you need to hold down the SHIFT key to see the Grub boot menu.

17.2   Boot to a shell directly

If Upstart (init(8)) itself has broken, you'll need to follow the steps below. By specifying an alternate "initial process" (here a shell) it is possible to repair the system.

1. Hold down SHIFT key before the splash screen appears (this will then display the grub menu).

2. Type, "e" to edit the default kernel command-line.

3. Use the arrow keys to go to the end of the line which starts "linux /boot/vmlinuz ...".

4. Press the END key (or use arrows) to go to end of the line.

5. Add a space followed by "init=/bin/sh".

6. If the line you are editing contains "quiet" and/or "splash", remove them.

7. Press CONTROL+x to boot with this modified kernel command line.

8. When the shell appears you will need to remount the root filesystem read-write like this:

   # mount -oremount,rw /
   

You can now make changes to your system as necessary.

18.1   Changing the Default Shell

By default, Upstart uses "/bin/sh" to execute script sections. If you wish to change this behaviour, you have the following options:

• Link /bin/sh to your chosen shell [8].

• Copy your chosen shell to /bin/sh.

• Recompile Upstart specifying an alternative shell as follows:

  # XXX: Note the careful quoting to retain double-quotes around the shell!
  export CFLAGS=-DSHELL='\"/bin/bash\"'
  ./configure && make && sudo make install
  

  Note that you should consider such a change carefully since Upstart has to rely upon the shell. Remember too that Upstart runs all script sections as the root user.

• Use a "here document" (assuming your chosen shell supports them) within the Job Configuration Files you wish to run with a different shell:

  script
  /bin/bash <<EOT
  
  echo "Hi - I am running under the bash shell"
  
  date
  
  echo "and so am I :)"
  
  EOT
  end script
  

  Note that currently, this technique is the only way (without modifying the Upstart source code) to run a shell without specifying the "-e" option (see dash(1) or bash(1) for details).

18.2   Running a script Section with Python

To run a script section with Python:

script

python - <<END

from datetime import datetime

today = datetime.now().strftime("%A")

fh = open("/tmp/file.txt", "w")
print >>fh, "Today is %s" % today
fh.close()

END

end script

18.3   Running a script Section with Perl

To run a script section with Perl:

script

perl - <<END

use strict;
use warnings;
use POSIX;

my $fh;
my $today = POSIX::strftime("%A", localtime);

open($fh, ">/tmp/file.txt");
printf $fh "Today is %s\n", $today;
close($fh);

END

end script

19.1   Unit Tests

Every major feature in Upstart needs to be accompanied with comprehensive unit tests. To run the tests:

$ autoreconf -fi
$ ./configure --enable-compiler-coverage ...
$ make check 2>&1|tee make-check.log

Note that as of Upstart 1.3, some of these tests cannot be run from within a chroot(2) environment unless D-Bus is installed and configured within the chroot. This scenario is detected, a warning about bug 728988 is logged and those tests are automatically skipped. Hence, to run all the tests, please ensure you run "make check" outside of a chroot(2) environment.

$ cd init
$ gcov -bf event.c

19.2   Enable Full Compiler Warnings

If you want to start submitting changes to Upstart, you need to ensure you build it as follows to catch any warnings and errors the compiler can flag:

./configure --disable-silent-rules --enable-compiler-warnings --disable-compiler-optimisations --disable-linker-optimisations --enable-compiler-coverage

19.3   Running Upstart as a Non-Privileged User

Upstart 1.3 introduced a number of options to help with testing. The "--session" command-line option allows you to run Upstart as a non-privileged user since it makes Upstart connect to the D-Bus session bus for which each user has their own:

$ /sbin/init --session --debug --confdir $HOME/conf/ --no-sessions

This is useful since you can now try out new features, debug with GDB, et cetera without having to install Upstart and run it as root. Once you've got your second instance of Upstart running, you can then use the same option on initctl to manipulate jobs:

$ initctl --session emit foo

The caveat here is that running Upstart as a non-privileged user with a PID other than 1 changes its behaviour slightly. So, only use this technique for unit/functional testing and remember that any changes you post for inclusion should have been tested in a real scenario where Upstart is run as root and used to boot a system.

19.4   Useful tools for Debugging with D-Bus

If you are debugging initctl(8), you'll need to understand D-Bus. These tools are invaluable:

• dbus-send(1)
• D-Feet

19.5   Debugging a Job

There is a magic stanza called debug which will start the job via fork(2) and then pause it. This can be useful. Assuming you have a job "debug.conf" such as:

# XXX: magic stanza!
debug

script
  /bin/true
end script

You could now trace the job process like this:

# start debug
debug start/running, process 12345
# strace -p 12345 -o /tmp/debug.log -Ff -s 1024 -v
status debug debug stop/waiting

After the call to start, the job process will be "running", but paused. The strace(1) will resume the job and you will then have a log of what happened in file "/tmp/debug.log".

19.6   Debugging Another Instance of Upstart Running as root with PID 1

$ sudo \
  gdb --args \
  clone -e DBUS_SYSTEM_BUS_ADDRESS=$DBUS_SESSION_BUS_ADDRESS \
  -f CLONE_NEWPID,SIGCHLD,CLONE_PTRACE -- \
  init/init --debug --confdir /my/conf/dir --no-startup-event
    --no-sessions

$ sudo lxc-start -n natty
$ upstart_pid=$(pgrep -f /sbin/init|grep -v '^1$')
$ sudo gdb /sbin/init $upstart_pid

19.7   Debugger Magic

Debugging in gdb initially seems rather difficult, but you just need to know the right tricks. The complication comes from the fact that Upstart uses the NIH Utility Library, which uses macros (such as NIH_LIST_FOREACH and NIH_HASH_FOREACH) for performance.

However, how do you access a data structure such as an NihList whose only method of iteration is a macro? Like this:

# first entry
(gdb) print *(JobClass *)job_classes->next

# 2nd entry
(gdb) print *(JobClass *)job_classes->next->next

# 3rd entry
(gdb) print *(JobClass *)job_classes->next->next->next

# ConfSource NihWatch for 1st entry in conf_sources list
(gdb) print *((ConfSource *)conf_sources->next)->watch

# size of JobClass->instances hash list
# XXX: this is the capacity, *NOT* the number of entries!
print class->instances->size

# first entry in job_classes global hash
print *(JobClass *)job_classes->bins->next

20.1   Restarting Jobs with Complex Conditions

The and and or operators allowed with start on and stop on do not work intuitively: operands to the right of either operator are only evaluated when the specified event is emitted. This can lead to jobs with complex start on or stop on conditions not behaving as expected when restarted. For example, if a job specifies the following condition:

start on A and (B or C)

When the events "A" and "B" are emitted, the condition is satisfied so the job will be run. If the job fails to start, or is stopped later, there is no guarantee that "A" will be emitted again, and the fact that it happened before is no longer known to Upstart. Meanwhile, events "C" or "B" may occur, but the job will not be transitioned back to a start goal, until event "A" is emitted again.

20.2   Using expect with script sections

Using the expect stanza with a job that uses a script section will lead to trouble if your script spawns any processes (likely!). Consider:

expect fork
respawn
script
  ARGS=$(cat /etc/default/grub)
  exec echo "ARGS=$ARGS" > /tmp/myjob.log
end script

This job configuration file is somewhat nonsensical, but it does demonstrate the problem. The main issue here is that by specifying expect fork, Upstart will attempt to follow only the first fork(2) call. The first process that this job will spawn is... cat(1), NOT echo. As such, starting the job will show something like this:

# start myjob
myjob start/running, process 12345
# status myjob
myjob start/running, process 12345
# ps --no-headers -p 12345
# kill 12345
-su: kill: (12345) - No such process

As the ps(1) call shows, the (cat) process is no longer running, but Upstart thinks it is.

Unfortunately, since Upstart will wait forever until it is able to stop the pid (which no longer exits). A manual attempt to either "stop myjob" or "start myjob" will also hang.

The only solution to clear this "stuck job" is to reboot. See [17] and Recovery on Misspecification of expect. Note that this "zombie job" isn't actually causing any problems for Upstart, but it is annoying and potentially confusing seeing it listed in initctl output. It will of course also be consuming a very small amount of memory.

Note however, that if you are working on a development system (hopefully you are whilst developing your job configuration file!), what you can do to keep working is to copy the problematic job configuration file to a new name, ignore the old job entirely and keep working using the new job!

20.3   Bugs

Upstart bugs

https://bugs.launchpad.net/upstart

Ubuntu-specific Upstart bugs

https://launchpad.net/ubuntu/+source/upstart/+bugs

22.1   Manual Pages

man 5 init

Configuration syntax reference.

man 8 init

Options for running the Upstart init daemon.

man 8 initctl

Explanation of the Upstart control command.

man 7 upstart-events

Comprehensive summary of all "well-known" Upstart system events on Ubuntu.

22.2   Web Sites

http://upstart.ubuntu.com/

Main Ubuntu page for Upstart.

http://launchpad.net/upstart

The main Upstart Bazaar project page.

http://upstart.at

The New Upstart Blog site.

http://netsplit.com/category/tech/upstart/

Scotts Original Upstart blog with useful overviews of features and Concepts.

https://wiki.ubuntu.com/ReplacementInit

Original Specification.

22.3   Mailing List

• https://lists.ubuntu.com/mailman/listinfo/upstart-devel

26.1   Ubuntu Well-Known Events ()

The information in this section is taken from the upstart-events(7) manual page.

Name

   upstart-events — Well-known Upstart events summary

Event Summary

   This manual page summarizes well-known events generated by the Upstart init(8) daemon. It is
   not an exhaustive list of all possible events, but rather details a standard set of events
   expected to be generated on any Ubuntu system running Upstart.

   The primary table, Table 1, encodes the well-known events, along with the type of each event
   (listed in Table 2), the emitter of the event (see Table 3) and the approximate time at which
   the event could be generated. Additionally, the Note column indexes into Table 4 for further
   details on a particular event.

   The Ref (Reference) column is used to refer to individual events succinctly in the Time
   column.

   Note that the '<' and '>' characters in the Time column denote that the event in the Event
   column occurs respectively before or after the event specified in the Time column (for
   example, the mounting(7) event occurs "at some time" after the startup(7) event, and the
   virtual-filesystems(7) event occurs after the last mounted(7) event relating to a virtual
   filesystem has been emitted).

   For further details on events, consult the manual pages and the job configuration files,
   usually located in /etc/init.

   Table 1. Table 1: Well-Known Event Summary.

   +--------------------------------------------------------------------------------------+
   | Ref |            Event             | Type | Emit |            Time            | Note |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | all-swaps                    |  S   |  M   | > (5)                      |      |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | control-alt-delete(7)        |  S   |  A   | > (5)                      |  A   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | dbus-activation              |  S   |  B   | > D-Bus client request     |      |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | deconfiguring-networking     |  H   |  V   | < non-local IFs down       |  P   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | desktop-session-start        |  H   |  D   | > X(7) session created     |  B   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | desktop-shutdown             |  H   |  D   | > X(7) session ended       |  O   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | device-not-ready             |  H   |  M   | > (2)                      |  N   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | drm-device-added             |  S   |  U   | > (5)                      |  C   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | filesystem                   |  S   |  M   | After last (1)             |  D   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | graphics-device-added        |  S   |  U   | > (5)                      |  C   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | keyboard-request(7)          |  S   |  A   | > (5)                      |  E   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | local-filesystems(7)         |  S   |  M   | > (6)                      |      |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | login-session-start          |  H   |  D   | < DM running               |  F   |
   |-----+------------------------------+------+------+----------------------------+------|
   |  1  | mounted(7)                   |  H   |  M   | > associated (2)           |  G   |
   |-----+------------------------------+------+------+----------------------------+------|
   |  2  | mounting(7)                  |  H   |  M   | > (5)                      |  H   |
   |-----+------------------------------+------+------+----------------------------+------|
   |  3  | net-device-added             |  S   |  U   | > (5)                      |  C   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | net-device-changed           |  S   |  U   | > (5)                      |  C   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | net-device-down              |  S   |  F   | < (4)                      |  C   |
   |-----+------------------------------+------+------+----------------------------+------|
   |  4  | net-device-removed           |  S   |  U   | > (5)                      |  C   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | net-device-up                |  S   | F,N  | > (3)                      |  C   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | power-status-changed(7)      |  S   |  I   | > (5)                      |  I   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | remote-filesystems(7)        |  S   |  M   | > (6)                      |      |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | runlevel(7)                  |  M   |  T   | > (5)                      |      |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | socket(7)                    |  S   |  S   | > socket connection        |      |
   |-----+------------------------------+------+------+----------------------------+------|
   |  5  | startup(7)                   |  S   |  I   | Boot                       |  J   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | started(7)                   |  S   |  I   | > job started              |  K   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | starting(7)                  |  H   |  I   | < job starts               |  K   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | static-network-up            |  S   |  I   | > last static IF up        |      |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | stopped(7)                   |  S   |  I   | > job stopped              |  K   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | stopping(7)                  |  H   |  I   | < job stops                |  K   |
   |-----+------------------------------+------+------+----------------------------+------|
   |     | unmounted-remote-filesystems |  H   |  V   | > last remote FS unmounted |  L   |
   |-----+------------------------------+------+------+----------------------------+------|
   |  6  | virtual-filesystems(7)       |  S   |  M   | > last virtual FS (1)      |  M   |
   +--------------------------------------------------------------------------------------+

   Key: 'DM' is an abbreviation for Display Manager. 'FS' is an abbreviation for filesystem. 'IF'
   is an abbreviation for Network Interface.

   Table 2. Table 2: Event Types.

   +------------------------------------------------------------------------------------+
   | Ref | Event Type | Notes                                                           |
   |-----+------------+-----------------------------------------------------------------|
   |  H  | Hook       | Blocking. Waits for events that start on or stop on this event. |
   |-----+------------+-----------------------------------------------------------------|
   |  M  | Method     | Blocking task.                                                  |
   |-----+------------+-----------------------------------------------------------------|
   |  S  | Signal     | Non-blocking.                                                   |
   +------------------------------------------------------------------------------------+

   Table 3. Table 3: Event Emitters.

   +--------------------------------------------------------------------------+
   | Ref | Emitter                          | Notes                           |
   |-----+----------------------------------+---------------------------------|
   |  A  | System Administrator (initiator) | Technically emitted by init(8). |
   |-----+----------------------------------+---------------------------------|
   |  B  | dbus-daemon(1)                   | Run with "--activation=upstart" |
   |-----+----------------------------------+---------------------------------|
   |  D  | Display Manager                  | e.g. lightdm/gdm/kdm/xdm.       |
   |-----+----------------------------------+---------------------------------|
   |  F  | ifup(8) or ifdown(8)             | See /etc/network/.              |
   |-----+----------------------------------+---------------------------------|
   |  I  | init(8)                          |                                 |
   |-----+----------------------------------+---------------------------------|
   |  M  | mountall(8)                      |                                 |
   |-----+----------------------------------+---------------------------------|
   |  N  | network-interface job            |                                 |
   |-----+----------------------------------+---------------------------------|
   |  S  | upstart-socket-bridge(8)         |                                 |
   |-----+----------------------------------+---------------------------------|
   |  T  | telinit(8), shutdown(8)          |                                 |
   |-----+----------------------------------+---------------------------------|
   |  U  | upstart-udev-bridge(8)           |                                 |
   |-----+----------------------------------+---------------------------------|
   |  V  | System V init system             |                                 |
   +--------------------------------------------------------------------------+

   Table 4. Table 4: Event Summary Notes.

   +--------------------------------------------------------------------------------------------+
   | Note | Detail                                                                              |
   |------+-------------------------------------------------------------------------------------|
   |  A   | Requires administrator to press Control-Alt-Delete key combination on the console.  |
   |------+-------------------------------------------------------------------------------------|
   |  B   | Event generated when user performs graphical login.                                 |
   |------+-------------------------------------------------------------------------------------|
   |      | These are specific examples. upstart-udev-bridge(8) will emit events which match    |
   |  C   | the pattern, "S-device-A" where 'S' is the udev subsystem and 'A' is the udev       |
   |      | action. See udev(7) and for further details. If you have sysfs mounted, you can     |
   |      | look in /sys/class/ for possible values for subsystem.                              |
   |------+-------------------------------------------------------------------------------------|
   |  D   | Note this is in the singular - there is no 'filesystems' event.                     |
   |------+-------------------------------------------------------------------------------------|
   |  E   | Emitted when administrator presses Alt-UpArrow key combination on the console.      |
   |------+-------------------------------------------------------------------------------------|
   |  F   | Denotes Display Manager running (about to be displayed), but no users logged in     |
   |      | yet.                                                                                |
   |------+-------------------------------------------------------------------------------------|
   |  G   | Generated for each mount that completes successfully.                               |
   |------+-------------------------------------------------------------------------------------|
   |  H   | Emitted when mount attempt for single entry from fstab(5) for any filesystem type   |
   |      | is about to begin.                                                                  |
   |------+-------------------------------------------------------------------------------------|
   |  I   | Emitted when Upstart receives the SIGPWR signal.                                    |
   |------+-------------------------------------------------------------------------------------|
   |  J   | Initial event.                                                                      |
   |------+-------------------------------------------------------------------------------------|
   |  K   | Although the events are emmitted by init(8), the instigator may be initctl(8) if a  |
   |      | System Administrator has manually started or stopped a job.                         |
   |------+-------------------------------------------------------------------------------------|
   |  L   | /etc/init/umountnfs.sh.                                                             |
   |------+-------------------------------------------------------------------------------------|
   |  M   | Emitted when all virtual filesystems (such as /proc) mounted.                       |
   |------+-------------------------------------------------------------------------------------|
   |  N   | Emitted when the --dev-wait-time timeout is exceeded for mountall(8). This defaults |
   |      | to 30 seconds.                                                                      |
   |------+-------------------------------------------------------------------------------------|
   |  O   | Emitted when the X(7) display manager exits at shutdown or reboot, to hand off to   |
   |      | the shutdown splash manager.                                                        |
   |------+-------------------------------------------------------------------------------------|
   |  P   | Emitted by /etc/init.d/networking just prior to stopping all non-local network      |
   |      | interfaces.                                                                         |
   +--------------------------------------------------------------------------------------------+