All rights reserved.
%sccs.include.redist.roff%
@(#)6.t 6.5 (Berkeley) 04/17/91
\\$1\|\\$2 .. .nr H1 6 .nr H2 0 .bp
6. SYSTEM OPERATION .R
This section describes procedures used to operate a \*(4B UNIX system. Procedures described here are used periodically, to reboot the system, analyze error messages from devices, do disk backups, monitor system performance, recompile system software and control local changes. Bootstrap and shutdown procedures
In a normal reboot, the system checks the disks and comes up multi-user without intervention at the console. Such a reboot can be stopped (after it prints the date) with a ^C (interrupt). This will leave the system in single-user mode, with only the console terminal active. It is also possible to allow the filesystem checks to complete and then to return to single-user mode by signaling fsck(8) with a QUIT signal (^\\).
If booting from the console command level is needed, then the command #>\|fb will boot from the default device.
You can boot a system up single user by doing #>\|p23 2.#>y.#>\|fb
Other possibilities are: #>\|p23 3.#>y.#>\|fb to do a full bootstrap, or #>\|p23 3.#>y.#>\|fr /boot to run the bootstrap without performing self-tests and reloading microcode; it can be used after a full bootstrap has been done once. .\}
If booting from the console command level is needed, then the command >>>B will boot from the default device. On an 8600, 8200, 11/780, or 11/730 the default device is determined by a ``DEPOSIT'' command stored on the console boot device in the file ``DEFBOO.CMD'' (``DEFBOO.COM'' on an 8600); on an 11/750 the default device is determined by the setting of a switch on the front panel.
You can boot a system up single user on an 8600, 780, or 730 by doing >>>B xxS where xx is one of HP, HK, UP, RA, or RB. The corresponding command on an 11/750 is >>>B/2 On an 8200, use >>>B/R5:800 (node and memory test values) BOOT58> @XXSBOO.CMD
For second vendor storage modules on the UNIBUS or MASSBUS of an 11/750 you will need to have a boot prom. Most vendors will sell you such proms for their controllers; contact your vendor if you don't have one.
Other possibilities are: >>>B ANY or, on an 8200, >>>B/R5:800 BOOT58>@ANYBOO.CMD or, on an 11/750 >>>B/3 .\} These commands boot and ask for the name of the system to be booted. They can be used after building a new test system to give the boot program the name of the test version of the system.* .FS * Additional bootflags are used when a system is configured with the kernel debugger; consult kdb(4) for details. .FE
To bring the system up to a multi-user configuration from the single-user status, all you have to do is hit ^D on the console. The system will then execute /etc/rc, a multi-user restart script (and /etc/rc.local), and come up on the terminals listed as active in the file /etc/ttys. See init\|(8) and ttys\|(5) for more details. Note, however, that this does not cause a file system check to be performed. Unless the system was taken down cleanly, you should run ``fsck -p'' or force a reboot with reboot\|(8) to have the disks checked.
To take the system down to a single user state you can use # kill 1 or use the shutdown\|(8) command (which is much more polite, if there are other users logged in) when you are running multi-user. Either command will kill all processes and give you a shell on the console, as if you had just booted. File systems remain mounted after the system is taken single-user. If you wish to come up multi-user again, you should do this by: # cd / # /etc/umount -a # ^D
Each system shutdown, crash, processor halt and reboot is recorded in the system log with its cause. Device errors and diagnostics
When serious errors occur on peripherals or in the system, the system prints a warning diagnostic on the console. These messages are collected by the system error logging process syslogd (8) and written into a system error log file /usr/adm/messages. Less serious errors are sent directly to syslogd, which may log them on the console. The error priorities that are logged and the locations to which they are logged are controlled by /etc/syslog.conf. See syslogd (8) for further details.
Error messages printed by the devices in the system are described with the drivers for the devices in section 4 of the programmer's manual. If errors occur suggesting hardware problems, you should contact your hardware support group or field service. It is a good idea to examine the error log file regularly (e.g. with the command tail -r /usr/adm/messages). File system checks, backups and disaster recovery
Periodically (say every week or so in the absence of any problems) and always (usually automatically) after a crash, all the file systems should be checked for consistency by fsck\|(1). The procedures of reboot\|(8) should be used to get the system to a state where a file system check can be performed manually or automatically.
Dumping of the file systems should be done regularly, since once the system is going it is easy to become complacent. Complete and incremental dumps are easily done with dump\|(8). You should arrange to do a towers-of-hanoi dump sequence; we tune ours so that almost all files are dumped on two tapes and kept for at least a week in most every case. We take full dumps every month (and keep these indefinitely). Operators can execute ``dump w'' at login that will tell them what needs to be dumped (based on the /etc/fstab information). Be sure to create a group operator in the file /etc/group so that dump can notify logged-in operators when it needs help.
More precisely, we have three sets of dump tapes: 10 daily tapes, 5 weekly sets of 2 tapes, and fresh sets of three tapes monthly. We do daily dumps circularly on the daily tapes with sequence `3 2 5 4 7 6 9 8 9 9 9 ...'. Each weekly is a level 1 and the daily dump sequence level restarts after each weekly dump. Full dumps are level 0 and the daily sequence restarts after each full dump also.
Thus a typical dump sequence would be:
.KS
| tape name level number date opr size |
| FULL 0 Nov 24, 1979 jkf 137K |
| D1 3 Nov 28, 1979 jkf 29K |
| D2 2 Nov 29, 1979 rrh 34K |
| D3 5 Nov 30, 1979 rrh 19K |
| D4 4 Dec 1, 1979 rrh 22K |
| W1 1 Dec 2, 1979 etc 40K |
| D5 3 Dec 4, 1979 rrh 15K |
| D6 2 Dec 5, 1979 jkf 25K |
| D7 5 Dec 6, 1979 jkf 15K |
| D8 4 Dec 7, 1979 rrh 19K |
| W2 1 Dec 9, 1979 etc 118K |
| D9 3 Dec 11, 1979 rrh 15K |
| D10 2 Dec 12, 1979 rrh 26K |
| D1 5 Dec 15, 1979 rrh 14K |
| W3 1 Dec 17, 1979 etc 71K |
| D2 3 Dec 18, 1979 etc 13K |
| FULL 0 Dec 22, 1979 etc 135K |
Dumping of files by name is best done by tar\|(1) but the amount of data that can be moved in this way is limited to a single tape. Finally if there are enough drives entire disks can be copied with dd\|(1) using the raw special files and an appropriate blocking factor; the number of sectors per track is usually a good value to use, consult /etc/disktab.
It is desirable that full dumps of the root file system be made regularly. This is especially true when only one disk is available. Then, if the root file system is damaged by a hardware or software failure, you can rebuild a workable disk doing a restore in the same way that the initial root file system was created.
Exhaustion of user-file space is certain to occur now and then; disk quotas may be imposed, or if you prefer a less fascist approach, try using the programs du\|(1), df\|(1), and quot\|(8), combined with threatening messages of the day, and personal letters. Moving file system data
If you have the resources, the best way to move a file system is to dump it to a spare disk partition, or magtape, using dump\|(8), use newfs\|(8) to create the new file system, and restore the file system using restore\|(8). Filesystems may also be moved by piping the output of dump to restore. The restore program uses an ``in-place'' algorithm that allows file system dumps to be restored without concern for the original size of the file system. Further, portions of a file system may be selectively restored using a method similar to the tape archive program.
If you have to merge a file system into another, existing one, the best bet is to use tar\|(1). If you must shrink a file system, the best bet is to dump the original and restore it onto the new file system. If you are playing with the root file system and only have one drive, the procedure is more complicated. If the only drive is a Winchester disk, this procedure may not be used without overwriting the existing root or another partition. What you do is the following:
Note that if you change the disk partition tables or add new disk drivers they should also be added to the standalone system in /sys/\*(mCstand, and the default disk partition tables in /etc/disktab should be modified. Monitoring System Performance
The systat program provided with the system is designed to be an aid to monitoring systemwide activity. The default ``pigs'' mode shows a dynamic ``ps''. By running in the ``vmstat'' mode when the system is active you can judge the system activity in several dimensions: job distribution, virtual memory load, paging and swapping activity, device interrupts, and disk and cpu utilization. Ideally, there should be few blocked (b) jobs, there should be little paging or swapping activity, there should be available bandwidth on the disk devices (most single arms peak out at 20-30 tps in practice), and the user cpu utilization (us) should be high (above 50%).
If the system is busy, then the count of active jobs may be large, and several of these jobs may often be blocked (b). If the virtual memory is active, then the paging demon will be running (sr will be non-zero). It is healthy for the paging demon to free pages when the virtual memory gets active; it is triggered by the amount of free memory dropping below a threshold and increases its pace as free memory goes to zero.
If you run in the ``vmstat'' mode when the system is busy, you can find imbalances by noting abnormal job distributions. If many processes are blocked (b), then the disk subsystem is overloaded or imbalanced. If you have several non-dma devices or open teletype lines that are ``ringing'', or user programs that are doing high-speed non-buffered input/output, then the system time may go high (60-70% or higher). It is often possible to pin down the cause of high system time by looking to see if there is excessive context switching (cs), interrupt activity (in) and per-device interrupt counts, or system call activity (sy). Cumulatively on one of our large machines we average about 60-100 context switches and interrupts per second and about 70-120 system calls per second.
If the system is heavily loaded, or if you have little memory for your load (2M is little in most any case), then the system may be forced to swap. This is likely to be accompanied by a noticeable reduction in system performance and pregnant pauses when interactive jobs such as editors swap out. If you expect to be in a memory-poor environment for an extended period you might consider administratively limiting system load. Recompiling and reinstalling system software
It is easy to regenerate the system, and it is a good idea to try rebuilding pieces of the system to build confidence in the procedures. The system consists of two major parts: the kernel itself (/sys) and the user programs (/usr/src and subdirectories). The major part of this is /usr/src.
The three major libraries are the C library in /usr/src/lib/libc and the \s-2FORTRAN\s0 libraries /usr/src/usr.lib/libI77 and /usr/src/usr.lib/libF77. In each case the library is remade by changing into the corresponding directory and doing # make and then installed by # make install Similar to the system, # make clean cleans up.
The source for all other libraries is kept in subdirectories of /usr/src/usr.lib; each has a makefile and can be recompiled by the above recipe.
If you look at /usr/src/Makefile, you will see that you can recompile the entire system source with one command. To recompile a specific program, find out where the source resides with the whereis\|(1) command, then change to that directory and remake it with the Makefile present in the directory. For instance, to recompile ``date'', all one has to do is # whereis date date: /usr/src/bin/date.c /bin/date # cd /usr/src/bin # make date this will create an unstripped version of the binary of ``date'' in the current directory. To install the binary image, use the install command as in # install -s date -o bin -g bin -m 755 /bin/date The -s option will insure the installed version of date has its symbol table stripped. The install command should be used instead of mv or cp as it understands how to install programs even when the program is currently in use.
If you wish to recompile and install all programs in a particular target area you can override the default target by doing: # make # make DESTDIR=pathname install
To regenerate all the system source you can do # cd /usr/src # make clean; make depend; make
If you modify the C library, say to change a system call, and want to rebuild and install everything from scratch you have to be a little careful. You must insure that the libraries are installed before the remainder of the source, otherwise the loaded images will not contain the new routine from the library. The following sequence will accomplish this. # cd /usr/src # make clean # make depend # make build # make installsrc The make clean removes any existing binary or object files in the source trees to insure that everything will be recompiled and reloaded. The make depend recreates all of the dependencies. See mkdep(1) for further details. The make build compiles and installs the libraries and compilers, then recompiles the libraries and compilers and the remainder of the sources. The make installsrc installs all of the commands not installed as part of the make build. This will take approximately 10 hours on a reasonably configured Tahoe. .\} Making local modifications
Locally written commands that aren't distributed are kept in /usr/src/local and their binaries are kept in /usr/local. This allows /usr/bin, /usr/ucb, and /bin to correspond to the distribution tape (and to the manuals that people can buy). People using local commands should be made aware that they aren't in the base manual. Manual pages for local commands should be installed in /usr/src/local/man and installed in /usr/local/man/cat[1-8]. The man(1) command automatically finds manual pages placed in /usr/local/man/cat[1-8] to facilitate this practice. Accounting
UNIX optionally records two kinds of accounting information: connect time accounting and process resource accounting. The connect time accounting information is stored in the file /usr/adm/wtmp, which is summarized by the program ac (8). The process time accounting information is stored in the file /usr/adm/acct after it is enabled by accton (8), and is analyzed and summarized by the program sa (8).
If you need to recharge for computing time, you can develop procedures based on the information provided by these commands. A convenient way to do this is to give commands to the clock daemon /etc/cron to be executed every day at a specified time. This is done by adding lines to /usr/adm/crontab; see cron (8) for details. Resource control
Resource control in the current version of UNIX is more elaborate than in most UNIX systems. The disk quota facilities developed at the University of Melbourne have been incorporated in the system and allow control over the number of files and amount of disk space each user may use on each file system. In addition, the resources consumed by any single process can be limited by the mechanisms of setrlimit\|(2). As distributed, the latter mechanism is voluntary, though sites may choose to modify the login mechanism to impose limits not covered with disk quotas.
To use the disk quota facilities, the system must be configured with ``options QUOTA''. File systems may then be placed under the quota mechanism by creating a null file quotas at the root of the file system, running quotacheck (8), and modifying /etc/fstab to show that the file system is read-write with disk quotas (an ``rq'' type field). The quotaon (8) program may then be run to enable quotas.
Individual quotas are applied by using the quota editor edquota (8). Users may view their quotas (but not those of other users) with the quota (1) program. The repquota (8) program may be used to summarize the quotas and current space usage on a particular file system or file systems.
Quotas are enforced with soft and hard limits. When a user first reaches a soft limit on a resource, a message is generated on his/her terminal. If the user fails to lower the resource usage below the soft limit the next time they log in to the system the login program will generate a warning about excessive usage. Should three login sessions go by with the soft limit breached the system then treats the soft limit as a hard limit and disallows any allocations until enough space is reclaimed to bring the user back below the soft limit. Hard limits are enforced strictly resulting in errors when a user tries to create or write a file. Each time a hard limit is exceeded the system will generate a message on the user's terminal.
Consult the auxiliary document, ``Disc Quotas in a UNIX Environment'' and the appropriate manual entries for more information. Network troubleshooting
If you have anything more than a trivial network configuration, from time to time you are bound to run into problems. Before blaming the software, first check your network connections. On networks such as the Ethernet a loose cable tap or misplaced power cable can result in severely deteriorated service. The netstat\|(1) program may be of aid in tracking down hardware malfunctions. In particular, look at the -i and -s options in the manual page.
Should you believe a communication protocol problem exists, consult the protocol specifications and attempt to isolate the problem in a packet trace. The SO_DEBUG option may be supplied before establishing a connection on a socket, in which case the system will trace all traffic and internal actions (such as timers expiring) in a circular trace buffer. This buffer may then be printed out with the trpt\|(8C) program. Most of the servers distributed with the system accept a -d option forcing all sockets to be created with debugging turned on. Consult the appropriate manual pages for more information. Files that need periodic attention
We conclude the discussion of system operations by listing the files that require periodic attention or are system specific:
..
| /etc/fstab how disk partitions are used |
| /etc/disktab default disk partition sizes/labels |
| /etc/printcap printer data base |
| /etc/gettytab terminal type definitions |
| /etc/remote names and phone numbers of remote machines for tip(1) |
| /etc/group group memberships |
| /etc/motd message of the day |
| /etc/passwd password file; each account has a line |
| /etc/rc.local local system restart script; runs reboot; starts daemons |
| /etc/inetd.conf local internet servers |
| /etc/hosts host name data base |
| /etc/networks network name data base |
| /etc/services network services data base |
| /etc/hosts.equiv hosts under same administrative control |
| /etc/syslog.conf error log configuration for syslogd\|(8) |
| /etc/ttys enables/disables ports |
| /usr/lib/crontab commands that are run periodically |
| /usr/lib/aliases mail forwarding and distribution groups |
| /usr/adm/acct raw process account data |
| /usr/adm/messages system error log |
| /usr/adm/wtmp login session accounting |