1.\" $NetBSD: crash.8,v 1.8 2002/07/04 17:35:44 wiz Exp $ 2.\" 3.\" Copyright (c) 1980, 1991, 1993 4.\" The Regents of the University of California. All rights reserved. 5.\" 6.\" Redistribution and use in source and binary forms, with or without 7.\" modification, are permitted provided that the following conditions 8.\" are met: 9.\" 1. Redistributions of source code must retain the above copyright 10.\" notice, this list of conditions and the following disclaimer. 11.\" 2. Redistributions in binary form must reproduce the above copyright 12.\" notice, this list of conditions and the following disclaimer in the 13.\" documentation and/or other materials provided with the distribution. 14.\" 3. All advertising materials mentioning features or use of this software 15.\" must display the following acknowledgement: 16.\" This product includes software developed by the University of 17.\" California, Berkeley and its contributors. 18.\" 4. Neither the name of the University nor the names of its contributors 19.\" may be used to endorse or promote products derived from this software 20.\" without specific prior written permission. 21.\" 22.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND 23.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 24.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 25.\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE 26.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 27.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 28.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 29.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 30.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 31.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 32.\" SUCH DAMAGE. 33.\" 34.\" from: @(#)crash.8 8.1 (Berkeley) 6/5/93 35.\" 36.Dd June 5, 1993 37.Os 38.Dt CRASH 8 39.Sh NAME 40.Nm crash 41.Nd UNIX system failures 42.Sh DESCRIPTION 43This section explains what happens when the system crashes 44and (very briefly) how to analyze crash dumps. 45.Pp 46When the system crashes voluntarily it prints a message of the form 47.Pp 48.Dl panic: why i gave up the ghost 49.Pp 50on the console, takes a dump on a mass storage peripheral, 51and then invokes an automatic reboot procedure as 52described in 53.Xr reboot 8 . 54(If auto-reboot is disabled on the front panel of the machine the system 55will simply halt at this point.) 56Unless some unexpected inconsistency is encountered in the state 57of the file systems due to hardware or software failure, the system 58will then resume multi-user operations. 59.Pp 60The system has a large number of internal consistency checks; if one 61of these fails, then it will panic with a very short message indicating 62which one failed. 63In many instances, this will be the name of the routine which detected 64the error, or a two-word description of the inconsistency. 65A full understanding of most panic messages requires perusal of the 66source code for the system. 67.Pp 68The most common cause of system failures is hardware failure, which 69can reflect itself in different ways. Here are the messages which 70are most likely, with some hints as to causes. 71Left unstated in all cases is the possibility that hardware or software 72error produced the message in some unexpected way. 73.Bl -tag -width 8n -compact 74.It iinit 75This cryptic panic message results from a failure to mount the root filesystem 76during the bootstrap process. 77Either the root filesystem has been corrupted, 78or the system is attempting to use the wrong device as root filesystem. 79Usually, an alternative copy of the system binary or an alternative root 80filesystem can be used to bring up the system to investigate. 81.It Can't exec /sbin/init 82This is not a panic message, as reboots are likely to be futile. 83Late in the bootstrap procedure, the system was unable to locate 84and execute the initialization process, 85.Xr init 8 . 86The root filesystem is incorrect or has been corrupted, or the mode 87or type of /sbin/init forbids execution. 88.It IO err in push 89.It hard IO err in swap 90The system encountered an error trying to write to the paging device 91or an error in reading critical information from a disk drive. 92The offending disk should be fixed if it is broken or unreliable. 93.It realloccg: bad optim 94.It ialloc: dup alloc 95.It alloccgblk: cyl groups corrupted 96.It ialloccg: map corrupted 97.It free: freeing free block 98.It free: freeing free frag 99.It ifree: freeing free inode 100.It alloccg: map corrupted 101These panic messages are among those that may be produced 102when filesystem inconsistencies are detected. 103The problem generally results from a failure to repair damaged filesystems 104after a crash, hardware failures, or other condition that should not 105normally occur. 106A filesystem check will normally correct the problem. 107.It timeout table overflow 108This really shouldn't be a panic, but until the data structure 109involved is made to be extensible, running out of entries causes a crash. 110If this happens, make the timeout table bigger. 111.It KSP not valid 112.It SBI fault 113.It CHM? in kernel 114These indicate either a serious bug in the system or, more often, 115a glitch or failing hardware. 116If SBI faults recur, check out the hardware or call 117field service. If the other faults recur, there is likely a bug somewhere 118in the system, although these can be caused by a flakey processor. 119Run processor microdiagnostics. 120.It machine check %x: Em description 121.It \0\0\0machine dependent machine-check information 122Machine checks are different on each type of CPU. 123Most of the internal processor registers are saved at the time of the fault 124and are printed on the console. 125For most processors, there is one line that summarizes the type of machine 126check. 127Often, the nature of the problem is apparent from this messaage 128and/or the contents of key registers. 129The VAX Hardware Handbook should be consulted, 130and, if necessary, your friendly field service people should be informed 131of the problem. 132.It trap type %d, code=%x, pc=%x 133A unexpected trap has occurred within the system; the trap types are: 134.Bd -literal -offset indent 1350 reserved addressing fault 1361 privileged instruction fault 1372 reserved operand fault 1383 bpt instruction fault 1394 xfc instruction fault 1405 system call trap 1416 arithmetic trap 1427 ast delivery trap 1438 segmentation fault 1449 protection fault 14510 trace trap 14611 compatibility mode fault 14712 page fault 14813 page table fault 149.Ed 150.Pp 151The favorite trap types in system crashes are trap types 8 and 9, 152indicating 153a wild reference. The code is the referenced address, and the pc at the 154time of the fault is printed. These problems tend to be easy to track 155down if they are kernel bugs since the processor stops cold, but random 156flakiness seems to cause this sometimes. 157The debugger can be used to locate the instruction and subroutine 158corresponding to the PC value. 159If that is insufficient to suggest the nature of the problem, 160more detailed examination of the system status at the time of the trap 161usually can produce an explanation. 162.It init died 163The system initialization process has exited. This is bad news, as no new 164users will then be able to log in. Rebooting is the only fix, so the 165system just does it right away. 166.It out of mbufs: map full 167The network has exhausted its private page map for network buffers. 168This usually indicates that buffers are being lost, and rather than 169allow the system to slowly degrade, it reboots immediately. 170The map may be made larger if necessary. 171.El 172.Pp 173That completes the list of panic types you are likely to see. 174.Pp 175When the system crashes it writes (or at least attempts to write) 176an image of memory into the back end of the dump device, 177usually the same as the primary swap 178area. After the system is rebooted, the program 179.Xr savecore 8 180runs and preserves a copy of this core image and the current 181system in a specified directory for later perusal. See 182.Xr savecore 8 183for details. 184.Pp 185To analyze a dump you should begin by running 186.Ic adb 187with the 188.Fl k 189flag on the system load image and core dump. 190If the core image is the result of a panic, 191the panic message is printed. 192Normally the command 193.Dq $c 194will provide a stack trace from the point of 195the crash and this will provide a clue as to 196what went wrong. 197For more detail 198see 199.Dq Using ADB to Debug the UNIX Kernel . 200.Sh SEE ALSO 201.Xr gdb 1 , 202.Xr reboot 8 203.br 204.Dq VAX 11/780 System Maintenance Guide 205and 206.Dq VAX Hardware Handbook 207for more information about machine checks. 208.br 209.Dq Using ADB to Debug the UNIX Kernel 210