Debugging Upstart

The document describes some helpful techniques for debugging. These methods can be used for new Upstart development, porting Upstart to a new Linux distribution, or authoring job definition files.

Increase Upstart Verbosity

Add a "--debug" command line to the kernel options after the init=/opt/upstart/sbin/init command. The "--debug" option causes Upstart to display additional messages to the console.

Add Persistent Log Entries to Job Definition Files

It may be helpful to keep a persistent record of what happens during the boot process. By writing to a file in /tmp we can watch boot messages after a successful boot. If the boot isn't successful, we can review the /tmp file after rebooting into a known good boot.

    # this will work only if you've mounted the fs read write
    echo "Debug Message" >> /tmp/upstart

Make sure to delete the /tmp files after each debug boot to not confuse two boots.

You can also use

    exec &> /tmp/my-log
    set -x

this works very well to debug {pre,post}-{start,stop,} job scripts

     exec upstart -v

will make upstart log all events

If your jobs hang at

    Mar 10 13:39:29 nixos init: sshd pre-start process (4928)

and you can't any longer find the process using

    ps -p 4928

the kernel probably doesn't send SIGCHD signals. This happens on some vservers. You can further debug this by

    strace -fF -p 1 # attach to upstart and trace all signals etc..

Initctl Commands

Once your system is up, Upstart provides useful commands for debugging jobs. "Initctl events" displays events as they are emitted in real time. Events can be generated from services or user generated from the "initctl emit <event>" command.

"Initctl jobs" displays verbose information in real time from Upstart about the actions taken on jobs. upstart-0.5.1 even supports "Initctl log-priority" to change verbosity of logging

Both commands do not display history, they only display activities as they occur. So when you first start the command you will not see anything happening. Both commands are exited with a CTRL+C.

"Initctl list" shows a complete listing of jobs and their current status. "Initctl status <job>" shows the same status as "initctl list", but for only a single job.

Boot Problems

If there is ever a problem booting, you can always change your init on the kernel command line. Depending on your distribution, access your kernel command line and add:


Even if you have replaced init with Upstart's init, you can boot with a minimal setup. From the shell prompt you can modify configuration files, or replace binaries as needed.

-- CategoryDoc

Debugging (last edited 2009-03-10 14:53:20 by Marc Weber)