1# @(#)TOUR 5.2 (Berkeley) 07/15/92 2 3NOTE -- This is the original TOUR paper distributed with ash and 4does not represent the current state of the shell. It is provided anyway 5since it provides helpful information for how the shell is structured, 6but be warned that things have changed -- the current shell is 7still under development. 8 9================================================================ 10 11 A Tour through Ash 12 13 Copyright 1989 by Kenneth Almquist. 14 15 16DIRECTORIES: The subdirectory bltin contains commands which can 17be compiled stand-alone. The rest of the source is in the main 18ash directory. 19 20SOURCE CODE GENERATORS: Files whose names begin with "mk" are 21programs that generate source code. A complete list of these 22programs is: 23 24 program intput files generates 25 ------- ------------ --------- 26 mkbuiltins builtins builtins.h builtins.c 27 mkinit *.c init.c 28 mknodes nodetypes nodes.h nodes.c 29 mksignames - signames.h signames.c 30 mksyntax - syntax.h syntax.c 31 mktokens - token.def 32 bltin/mkexpr unary_op binary_op operators.h operators.c 33 34There are undoubtedly too many of these. Mkinit searches all the 35C source files for entries looking like: 36 37 INIT { 38 x = 1; /* executed during initialization */ 39 } 40 41 RESET { 42 x = 2; /* executed when the shell does a longjmp 43 back to the main command loop */ 44 } 45 46 SHELLPROC { 47 x = 3; /* executed when the shell runs a shell procedure */ 48 } 49 50It pulls this code out into routines which are when particular 51events occur. The intent is to improve modularity by isolating 52the information about which modules need to be explicitly 53initialized/reset within the modules themselves. 54 55Mkinit recognizes several constructs for placing declarations in 56the init.c file. 57 INCLUDE "file.h" 58includes a file. The storage class MKINIT makes a declaration 59available in the init.c file, for example: 60 MKINIT int funcnest; /* depth of function calls */ 61MKINIT alone on a line introduces a structure or union declara- 62tion: 63 MKINIT 64 struct redirtab { 65 short renamed[10]; 66 }; 67Preprocessor #define statements are copied to init.c without any 68special action to request this. 69 70INDENTATION: The ash source is indented in multiples of six 71spaces. The only study that I have heard of on the subject con- 72cluded that the optimal amount to indent is in the range of four 73to six spaces. I use six spaces since it is not too big a jump 74from the widely used eight spaces. If you really hate six space 75indentation, use the adjind (source included) program to change 76it to something else. 77 78EXCEPTIONS: Code for dealing with exceptions appears in 79exceptions.c. The C language doesn't include exception handling, 80so I implement it using setjmp and longjmp. The global variable 81exception contains the type of exception. EXERROR is raised by 82calling error. EXINT is an interrupt. EXSHELLPROC is an excep- 83tion which is raised when a shell procedure is invoked. The pur- 84pose of EXSHELLPROC is to perform the cleanup actions associated 85with other exceptions. After these cleanup actions, the shell 86can interpret a shell procedure itself without exec'ing a new 87copy of the shell. 88 89INTERRUPTS: In an interactive shell, an interrupt will cause an 90EXINT exception to return to the main command loop. (Exception: 91EXINT is not raised if the user traps interrupts using the trap 92command.) The INTOFF and INTON macros (defined in exception.h) 93provide uninterruptable critical sections. Between the execution 94of INTOFF and the execution of INTON, interrupt signals will be 95held for later delivery. INTOFF and INTON can be nested. 96 97MEMALLOC.C: Memalloc.c defines versions of malloc and realloc 98which call error when there is no memory left. It also defines a 99stack oriented memory allocation scheme. Allocating off a stack 100is probably more efficient than allocation using malloc, but the 101big advantage is that when an exception occurs all we have to do 102to free up the memory in use at the time of the exception is to 103restore the stack pointer. The stack is implemented using a 104linked list of blocks. 105 106STPUTC: If the stack were contiguous, it would be easy to store 107strings on the stack without knowing in advance how long the 108string was going to be: 109 p = stackptr; 110 *p++ = c; /* repeated as many times as needed */ 111 stackptr = p; 112The folloing three macros (defined in memalloc.h) perform these 113operations, but grow the stack if you run off the end: 114 STARTSTACKSTR(p); 115 STPUTC(c, p); /* repeated as many times as needed */ 116 grabstackstr(p); 117 118We now start a top-down look at the code: 119 120MAIN.C: The main routine performs some initialization, executes 121the user's profile if necessary, and calls cmdloop. Cmdloop is 122repeatedly parses and executes commands. 123 124OPTIONS.C: This file contains the option processing code. It is 125called from main to parse the shell arguments when the shell is 126invoked, and it also contains the set builtin. The -i and -j op- 127tions (the latter turns on job control) require changes in signal 128handling. The routines setjobctl (in jobs.c) and setinteractive 129(in trap.c) are called to handle changes to these options. 130 131PARSING: The parser code is all in parser.c. A recursive des- 132cent parser is used. Syntax tables (generated by mksyntax) are 133used to classify characters during lexical analysis. There are 134three tables: one for normal use, one for use when inside single 135quotes, and one for use when inside double quotes. The tables 136are machine dependent because they are indexed by character vari- 137ables and the range of a char varies from machine to machine. 138 139PARSE OUTPUT: The output of the parser consists of a tree of 140nodes. The various types of nodes are defined in the file node- 141types. 142 143Nodes of type NARG are used to represent both words and the con- 144tents of here documents. An early version of ash kept the con- 145tents of here documents in temporary files, but keeping here do- 146cuments in memory typically results in significantly better per- 147formance. It would have been nice to make it an option to use 148temporary files for here documents, for the benefit of small 149machines, but the code to keep track of when to delete the tem- 150porary files was complex and I never fixed all the bugs in it. 151(AT&T has been maintaining the Bourne shell for more than ten 152years, and to the best of my knowledge they still haven't gotten 153it to handle temporary files correctly in obscure cases.) 154 155The text field of a NARG structure points to the text of the 156word. The text consists of ordinary characters and a number of 157special codes defined in parser.h. The special codes are: 158 159 CTLVAR Variable substitution 160 CTLENDVAR End of variable substitution 161 CTLBACKQ Command substitution 162 CTLBACKQ|CTLQUOTE Command substitution inside double quotes 163 CTLESC Escape next character 164 165A variable substitution contains the following elements: 166 167 CTLVAR type name '=' [ alternative-text CTLENDVAR ] 168 169The type field is a single character specifying the type of sub- 170stitution. The possible types are: 171 172 VSNORMAL $var 173 VSMINUS ${var-text} 174 VSMINUS|VSNUL ${var:-text} 175 VSPLUS ${var+text} 176 VSPLUS|VSNUL ${var:+text} 177 VSQUESTION ${var?text} 178 VSQUESTION|VSNUL ${var:?text} 179 VSASSIGN ${var=text} 180 VSASSIGN|VSNUL ${var=text} 181 182In addition, the type field will have the VSQUOTE flag set if the 183variable is enclosed in double quotes. The name of the variable 184comes next, terminated by an equals sign. If the type is not 185VSNORMAL, then the text field in the substitution follows, ter- 186minated by a CTLENDVAR byte. 187 188Commands in back quotes are parsed and stored in a linked list. 189The locations of these commands in the string are indicated by 190CTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether 191the back quotes were enclosed in double quotes. 192 193The character CTLESC escapes the next character, so that in case 194any of the CTL characters mentioned above appear in the input, 195they can be passed through transparently. CTLESC is also used to 196escape '*', '?', '[', and '!' characters which were quoted by the 197user and thus should not be used for file name generation. 198 199CTLESC characters have proved to be particularly tricky to get 200right. In the case of here documents which are not subject to 201variable and command substitution, the parser doesn't insert any 202CTLESC characters to begin with (so the contents of the text 203field can be written without any processing). Other here docu- 204ments, and words which are not subject to splitting and file name 205generation, have the CTLESC characters removed during the vari- 206able and command substitution phase. Words which are subject 207splitting and file name generation have the CTLESC characters re- 208moved as part of the file name phase. 209 210EXECUTION: Command execution is handled by the following files: 211 eval.c The top level routines. 212 redir.c Code to handle redirection of input and output. 213 jobs.c Code to handle forking, waiting, and job control. 214 exec.c Code to to path searches and the actual exec sys call. 215 expand.c Code to evaluate arguments. 216 var.c Maintains the variable symbol table. Called from expand.c. 217 218EVAL.C: Evaltree recursively executes a parse tree. The exit 219status is returned in the global variable exitstatus. The alter- 220native entry evalbackcmd is called to evaluate commands in back 221quotes. It saves the result in memory if the command is a buil- 222tin; otherwise it forks off a child to execute the command and 223connects the standard output of the child to a pipe. 224 225JOBS.C: To create a process, you call makejob to return a job 226structure, and then call forkshell (passing the job structure as 227an argument) to create the process. Waitforjob waits for a job 228to complete. These routines take care of process groups if job 229control is defined. 230 231REDIR.C: Ash allows file descriptors to be redirected and then 232restored without forking off a child process. This is accom- 233plished by duplicating the original file descriptors. The redir- 234tab structure records where the file descriptors have be dupli- 235cated to. 236 237EXEC.C: The routine find_command locates a command, and enters 238the command in the hash table if it is not already there. The 239third argument specifies whether it is to print an error message 240if the command is not found. (When a pipeline is set up, 241find_command is called for all the commands in the pipeline be- 242fore any forking is done, so to get the commands into the hash 243table of the parent process. But to make command hashing as 244transparent as possible, we silently ignore errors at that point 245and only print error messages if the command cannot be found 246later.) 247 248The routine shellexec is the interface to the exec system call. 249 250EXPAND.C: Arguments are processed in three passes. The first 251(performed by the routine argstr) performs variable and command 252substitution. The second (ifsbreakup) performs word splitting 253and the third (expandmeta) performs file name generation. If the 254"/u" directory is simulated, then when "/u/username" is replaced 255by the user's home directory, the flag "didudir" is set. This 256tells the cd command that it should print out the directory name, 257just as it would if the "/u" directory were implemented using 258symbolic links. 259 260VAR.C: Variables are stored in a hash table. Probably we should 261switch to extensible hashing. The variable name is stored in the 262same string as the value (using the format "name=value") so that 263no string copying is needed to create the environment of a com- 264mand. Variables which the shell references internally are preal- 265located so that the shell can reference the values of these vari- 266ables without doing a lookup. 267 268When a program is run, the code in eval.c sticks any environment 269variables which precede the command (as in "PATH=xxx command") in 270the variable table as the simplest way to strip duplicates, and 271then calls "environment" to get the value of the environment. 272There are two consequences of this. First, if an assignment to 273PATH precedes the command, the value of PATH before the assign- 274ment must be remembered and passed to shellexec. Second, if the 275program turns out to be a shell procedure, the strings from the 276environment variables which preceded the command must be pulled 277out of the table and replaced with strings obtained from malloc, 278since the former will automatically be freed when the stack (see 279the entry on memalloc.c) is emptied. 280 281BUILTIN COMMANDS: The procedures for handling these are scat- 282tered throughout the code, depending on which location appears 283most appropriate. They can be recognized because their names al- 284ways end in "cmd". The mapping from names to procedures is 285specified in the file builtins, which is processed by the mkbuil- 286tins command. 287 288A builtin command is invoked with argc and argv set up like a 289normal program. A builtin command is allowed to overwrite its 290arguments. Builtin routines can call nextopt to do option pars- 291ing. This is kind of like getopt, but you don't pass argc and 292argv to it. Builtin routines can also call error. This routine 293normally terminates the shell (or returns to the main command 294loop if the shell is interactive), but when called from a builtin 295command it causes the builtin command to terminate with an exit 296status of 2. 297 298The directory bltins contains commands which can be compiled in- 299dependently but can also be built into the shell for efficiency 300reasons. The makefile in this directory compiles these programs 301in the normal fashion (so that they can be run regardless of 302whether the invoker is ash), but also creates a library named 303bltinlib.a which can be linked with ash. The header file bltin.h 304takes care of most of the differences between the ash and the 305stand-alone environment. The user should call the main routine 306"main", and #define main to be the name of the routine to use 307when the program is linked into ash. This #define should appear 308before bltin.h is included; bltin.h will #undef main if the pro- 309gram is to be compiled stand-alone. 310 311CD.C: This file defines the cd and pwd builtins. The pwd com- 312mand runs /bin/pwd the first time it is invoked (unless the user 313has already done a cd to an absolute pathname), but then 314remembers the current directory and updates it when the cd com- 315mand is run, so subsequent pwd commands run very fast. The main 316complication in the cd command is in the docd command, which 317resolves symbolic links into actual names and informs the user 318where the user ended up if he crossed a symbolic link. 319 320SIGNALS: Trap.c implements the trap command. The routine set- 321signal figures out what action should be taken when a signal is 322received and invokes the signal system call to set the signal ac- 323tion appropriately. When a signal that a user has set a trap for 324is caught, the routine "onsig" sets a flag. The routine dotrap 325is called at appropriate points to actually handle the signal. 326When an interrupt is caught and no trap has been set for that 327signal, the routine "onint" in error.c is called. 328 329OUTPUT: Ash uses it's own output routines. There are three out- 330put structures allocated. "Output" represents the standard out- 331put, "errout" the standard error, and "memout" contains output 332which is to be stored in memory. This last is used when a buil- 333tin command appears in backquotes, to allow its output to be col- 334lected without doing any I/O through the UNIX operating system. 335The variables out1 and out2 normally point to output and errout, 336respectively, but they are set to point to memout when appropri- 337ate inside backquotes. 338 339INPUT: The basic input routine is pgetc, which reads from the 340current input file. There is a stack of input files; the current 341input file is the top file on this stack. The code allows the 342input to come from a string rather than a file. (This is for the 343-c option and the "." and eval builtin commands.) The global 344variable plinno is saved and restored when files are pushed and 345popped from the stack. The parser routines store the number of 346the current line in this variable. 347 348DEBUGGING: If DEBUG is defined in shell.h, then the shell will 349write debugging information to the file $HOME/trace. Most of 350this is done using the TRACE macro, which takes a set of printf 351arguments inside two sets of parenthesis. Example: 352"TRACE(("n=%d0, n))". The double parenthesis are necessary be- 353cause the preprocessor can't handle functions with a variable 354number of arguments. Defining DEBUG also causes the shell to 355generate a core dump if it is sent a quit signal. The tracing 356code is in show.c. 357