Differences between revisions 14 and 15
Revision 14 as of 2011-06-25 11:45:20
Size: 4544
Comment: nice to see these docs are being updated
Revision 15 as of 2011-08-26 04:10:18
Size: 4544
Editor: localhost
Comment: converted to 1.6 markup
No differences found!

Debugging Upstart

The document describes some helpful techniques for debugging. These methods can be used for new Upstart development, porting Upstart to a new Linux distribution, or authoring job definition files.

Increase Upstart Verbosity

Add a "--verbose" command line to the kernel options. You can do this easily if you are using grub. The "--verbose" option causes Upstart to display additional messages to the console.

If you have upstart installed in a non-standard location, you probably already have an init= line in your kernel command line such as "init=/opt/upstart/sbin/init". In this case, add the "--verbose" switch after the "init=" string.

Add Persistent Log Entries to Job Definition Files

It may be helpful to keep a persistent record of what happens during the boot process. By writing to a file in /tmp we can watch boot messages after a successful boot. If the boot isn't successful, we can review the /tmp file after rebooting into a known good boot.

    # this will work only if you've mounted the fs read write
    echo "Debug Message" >> /tmp/upstart

Make sure to delete the /tmp files after each debug boot to not confuse two boots.

  • This is not clear enough: how do I activate the persistant record? What do I have to do? Does upstart check at the startup if there is a file /tmp/upstart with the content "Debug Message"?

You can also use

    exec > /tmp/my-log 2>&1
    set -x

this works very well to debug {pre,post}-{start,stop,} job scripts

If you need to log the stdout and stderr of an exec process within a job file, you can do:

exec foo >> /tmp/upstart.log 2>&1

It will append the stdout and stderr of the process that is exec'd to /tmp/upstart.log

     exec upstart -v

will make upstart log all events

If your jobs hang at

    Mar 10 13:39:29 nixos init: sshd pre-start process (4928)

and you can't any longer find the process using

    ps -p 4928

the kernel probably doesn't send SIGCHD signals. This happens on some vservers. You can further debug this by

    strace -fF -p 1 # attach to upstart and trace all signals etc..

Initctl Commands

Once your system is up, Upstart provides useful commands for debugging jobs. "Initctl events" displays events as they are emitted in real time. Events can be generated from services or user generated from the "initctl emit <event>" command.

"Initctl jobs" displays verbose information in real time from Upstart about the actions taken on jobs. (perhaps on old versions, certainly not on 0.9.) upstart-0.5.1 even supports "Initctl log-priority" to change verbosity of logging

Both commands do not display history, they only display activities as they occur. So when you first start the command you will not see anything happening. Both commands are exited with a CTRL+C.

"Initctl list" shows a complete listing of jobs and their current status. "Initctl status <job>" shows the same status as "initctl list", but for only a single job.

In newer versions of Upstart, "initctl log-priority <priority>" will set the level of logging. See the initctl manpage for valid priority levels. The debug and message priority levels can be useful for debugging job definitions.

Boot Problems

If there is ever a problem booting, you can always change your init on the kernel command line. Depending on your distribution, access your kernel command line and add:

init=/bin/sh

Even if you have replaced init with Upstart's init, you can boot with a minimal setup. From the shell prompt you can modify configuration files, or replace binaries as needed.

You can try forking a shell to aid in debugging by starting:

getty -8 -n -l /bin/sh 38400 tty2 &

Then starting Upstart from the original shell with:

exec init

NFS issues

Upstart uses inotify to detect changes to files. Inotify doesn't work over NFS. Thus, if you edit a job file that resides on an NFS volume, Upstart won't pick up the changes. This is an issue with network booted machines, or with symbolically linking a job file on an NFS mount to the jobs directory. The solution in the latter case is to copy the job.

-- CategoryDoc

Debugging (last edited 2011-08-26 04:10:18 by localhost)