1.\" $NetBSD: crash.8,v 1.12 2009/03/09 19:24:32 joerg Exp $ 2.\" 3.\" Copyright (c) 1980, 1991, 1993 4.\" The Regents of the University of California. All rights reserved. 5.\" 6.\" Redistribution and use in source and binary forms, with or without 7.\" modification, are permitted provided that the following conditions 8.\" are met: 9.\" 1. Redistributions of source code must retain the above copyright 10.\" notice, this list of conditions and the following disclaimer. 11.\" 2. Redistributions in binary form must reproduce the above copyright 12.\" notice, this list of conditions and the following disclaimer in the 13.\" documentation and/or other materials provided with the distribution. 14.\" 3. Neither the name of the University nor the names of its contributors 15.\" may be used to endorse or promote products derived from this software 16.\" without specific prior written permission. 17.\" 18.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND 19.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 21.\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE 22.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 24.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 25.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 26.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 27.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 28.\" SUCH DAMAGE. 29.\" 30.\" from: @(#)crash.8 8.1 (Berkeley) 6/5/93 31.\" 32.Dd June 5, 1993 33.Dt CRASH 8 vax 34.Os 35.Sh NAME 36.Nm crash 37.Nd UNIX system failures 38.Sh DESCRIPTION 39This section explains what happens when the system crashes 40and (very briefly) how to analyze crash dumps. 41.Pp 42When the system crashes voluntarily it prints a message of the form 43.Pp 44.Dl panic: why i gave up the ghost 45.Pp 46on the console, takes a dump on a mass storage peripheral, 47and then invokes an automatic reboot procedure as 48described in 49.Xr reboot 8 . 50(If auto-reboot is disabled on the front panel of the machine the system 51will simply halt at this point.) 52Unless some unexpected inconsistency is encountered in the state 53of the file systems due to hardware or software failure, the system 54will then resume multi-user operations. 55.Pp 56The system has a large number of internal consistency checks; if one 57of these fails, then it will panic with a very short message indicating 58which one failed. 59In many instances, this will be the name of the routine which detected 60the error, or a two-word description of the inconsistency. 61A full understanding of most panic messages requires perusal of the 62source code for the system. 63.Pp 64The most common cause of system failures is hardware failure, which 65can reflect itself in different ways. Here are the messages which 66are most likely, with some hints as to causes. 67Left unstated in all cases is the possibility that hardware or software 68error produced the message in some unexpected way. 69.Bl -tag -width 8n -compact 70.It iinit 71This cryptic panic message results from a failure to mount the root filesystem 72during the bootstrap process. 73Either the root filesystem has been corrupted, 74or the system is attempting to use the wrong device as root filesystem. 75Usually, an alternative copy of the system binary or an alternative root 76filesystem can be used to bring up the system to investigate. 77.It Can't exec /sbin/init 78This is not a panic message, as reboots are likely to be futile. 79Late in the bootstrap procedure, the system was unable to locate 80and execute the initialization process, 81.Xr init 8 . 82The root filesystem is incorrect or has been corrupted, or the mode 83or type of /sbin/init forbids execution. 84.It IO err in push 85.It hard IO err in swap 86The system encountered an error trying to write to the paging device 87or an error in reading critical information from a disk drive. 88The offending disk should be fixed if it is broken or unreliable. 89.It realloccg: bad optim 90.It ialloc: dup alloc 91.It alloccgblk: cyl groups corrupted 92.It ialloccg: map corrupted 93.It free: freeing free block 94.It free: freeing free frag 95.It ifree: freeing free inode 96.It alloccg: map corrupted 97These panic messages are among those that may be produced 98when filesystem inconsistencies are detected. 99The problem generally results from a failure to repair damaged filesystems 100after a crash, hardware failures, or other condition that should not 101normally occur. 102A filesystem check will normally correct the problem. 103.It timeout table overflow 104This really shouldn't be a panic, but until the data structure 105involved is made to be extensible, running out of entries causes a crash. 106If this happens, make the timeout table bigger. 107.It KSP not valid 108.It SBI fault 109.It CHM? in kernel 110These indicate either a serious bug in the system or, more often, 111a glitch or failing hardware. 112If SBI faults recur, check out the hardware or call 113field service. If the other faults recur, there is likely a bug somewhere 114in the system, although these can be caused by a flakey processor. 115Run processor microdiagnostics. 116.It machine check %x: Em description 117.It \0\0\0machine dependent machine-check information 118Machine checks are different on each type of CPU. 119Most of the internal processor registers are saved at the time of the fault 120and are printed on the console. 121For most processors, there is one line that summarizes the type of machine 122check. 123Often, the nature of the problem is apparent from this message 124and/or the contents of key registers. 125The VAX Hardware Handbook should be consulted, 126and, if necessary, your friendly field service people should be informed 127of the problem. 128.It trap type %d, code=%x, pc=%x 129A unexpected trap has occurred within the system; the trap types are: 130.Bd -literal -offset indent 1310 reserved addressing fault 1321 privileged instruction fault 1332 reserved operand fault 1343 bpt instruction fault 1354 xfc instruction fault 1365 system call trap 1376 arithmetic trap 1387 ast delivery trap 1398 segmentation fault 1409 protection fault 14110 trace trap 14211 compatibility mode fault 14312 page fault 14413 page table fault 145.Ed 146.Pp 147The favorite trap types in system crashes are trap types 8 and 9, 148indicating 149a wild reference. The code is the referenced address, and the pc at the 150time of the fault is printed. These problems tend to be easy to track 151down if they are kernel bugs since the processor stops cold, but random 152flakiness seems to cause this sometimes. 153The debugger can be used to locate the instruction and subroutine 154corresponding to the PC value. 155If that is insufficient to suggest the nature of the problem, 156more detailed examination of the system status at the time of the trap 157usually can produce an explanation. 158.It init died 159The system initialization process has exited. This is bad news, as no new 160users will then be able to log in. Rebooting is the only fix, so the 161system just does it right away. 162.It out of mbufs: map full 163The network has exhausted its private page map for network buffers. 164This usually indicates that buffers are being lost, and rather than 165allow the system to slowly degrade, it reboots immediately. 166The map may be made larger if necessary. 167.El 168.Pp 169That completes the list of panic types you are likely to see. 170.Pp 171When the system crashes it writes (or at least attempts to write) 172an image of memory into the back end of the dump device, 173usually the same as the primary swap 174area. After the system is rebooted, the program 175.Xr savecore 8 176runs and preserves a copy of this core image and the current 177system in a specified directory for later perusal. See 178.Xr savecore 8 179for details. 180.Pp 181To analyze a dump you should begin by running 182.Ic adb 183with the 184.Fl k 185flag on the system load image and core dump. 186If the core image is the result of a panic, 187the panic message is printed. 188Normally the command 189.Dq $c 190will provide a stack trace from the point of 191the crash and this will provide a clue as to 192what went wrong. 193For more detail 194see 195.Dq Using ADB to Debug the UNIX Kernel . 196.Sh SEE ALSO 197.Xr gdb 1 , 198.Xr reboot 8 199.br 200.Dq VAX 11/780 System Maintenance Guide 201and 202.Dq VAX Hardware Handbook 203for more information about machine checks. 204.br 205.Dq Using ADB to Debug the UNIX Kernel 206