1# @(#)TOUR 8.1 (Berkeley) 5/31/93 2# $FreeBSD: src/bin/sh/TOUR,v 1.7 2006/04/16 11:54:01 schweikh Exp $ 3# $DragonFly: src/bin/sh/TOUR,v 1.3 2007/01/04 06:24:11 pavalos Exp $ 4 5NOTE -- This is the original TOUR paper distributed with ash and 6does not represent the current state of the shell. It is provided anyway 7since it provides helpful information for how the shell is structured, 8but be warned that things have changed -- the current shell is 9still under development. 10 11================================================================ 12 13 A Tour through Ash 14 15 Copyright 1989 by Kenneth Almquist. 16 17 18DIRECTORIES: The subdirectory bltin contains commands which can 19be compiled stand-alone. The rest of the source is in the main 20ash directory. 21 22SOURCE CODE GENERATORS: Files whose names begin with "mk" are 23programs that generate source code. A complete list of these 24programs is: 25 26 program input files generates 27 ------- ----------- --------- 28 mkbuiltins builtins builtins.h builtins.c 29 mkinit *.c init.c 30 mknodes nodetypes nodes.h nodes.c 31 mksignames - signames.h signames.c 32 mksyntax - syntax.h syntax.c 33 mktokens - token.h 34 bltin/mkexpr unary_op binary_op operators.h operators.c 35 36There are undoubtedly too many of these. Mkinit searches all the 37C source files for entries looking like: 38 39 INIT { 40 x = 1; /* executed during initialization */ 41 } 42 43 RESET { 44 x = 2; /* executed when the shell does a longjmp 45 back to the main command loop */ 46 } 47 48 SHELLPROC { 49 x = 3; /* executed when the shell runs a shell procedure */ 50 } 51 52It pulls this code out into routines which are when particular 53events occur. The intent is to improve modularity by isolating 54the information about which modules need to be explicitly 55initialized/reset within the modules themselves. 56 57Mkinit recognizes several constructs for placing declarations in 58the init.c file. 59 INCLUDE "file.h" 60includes a file. The storage class MKINIT makes a declaration 61available in the init.c file, for example: 62 MKINIT int funcnest; /* depth of function calls */ 63MKINIT alone on a line introduces a structure or union declara- 64tion: 65 MKINIT 66 struct redirtab { 67 short renamed[10]; 68 }; 69Preprocessor #define statements are copied to init.c without any 70special action to request this. 71 72INDENTATION: The ash source is indented in multiples of six 73spaces. The only study that I have heard of on the subject con- 74cluded that the optimal amount to indent is in the range of four 75to six spaces. I use six spaces since it is not too big a jump 76from the widely used eight spaces. If you really hate six space 77indentation, use the adjind (source included) program to change 78it to something else. 79 80EXCEPTIONS: Code for dealing with exceptions appears in 81exceptions.c. The C language doesn't include exception handling, 82so I implement it using setjmp and longjmp. The global variable 83exception contains the type of exception. EXERROR is raised by 84calling error. EXINT is an interrupt. EXSHELLPROC is an excep- 85tion which is raised when a shell procedure is invoked. The pur- 86pose of EXSHELLPROC is to perform the cleanup actions associated 87with other exceptions. After these cleanup actions, the shell 88can interpret a shell procedure itself without exec'ing a new 89copy of the shell. 90 91INTERRUPTS: In an interactive shell, an interrupt will cause an 92EXINT exception to return to the main command loop. (Exception: 93EXINT is not raised if the user traps interrupts using the trap 94command.) The INTOFF and INTON macros (defined in exception.h) 95provide uninterruptible critical sections. Between the execution 96of INTOFF and the execution of INTON, interrupt signals will be 97held for later delivery. INTOFF and INTON can be nested. 98 99MEMALLOC.C: Memalloc.c defines versions of malloc and realloc 100which call error when there is no memory left. It also defines a 101stack oriented memory allocation scheme. Allocating off a stack 102is probably more efficient than allocation using malloc, but the 103big advantage is that when an exception occurs all we have to do 104to free up the memory in use at the time of the exception is to 105restore the stack pointer. The stack is implemented using a 106linked list of blocks. 107 108STPUTC: If the stack were contiguous, it would be easy to store 109strings on the stack without knowing in advance how long the 110string was going to be: 111 p = stackptr; 112 *p++ = c; /* repeated as many times as needed */ 113 stackptr = p; 114The following three macros (defined in memalloc.h) perform these 115operations, but grow the stack if you run off the end: 116 STARTSTACKSTR(p); 117 STPUTC(c, p); /* repeated as many times as needed */ 118 grabstackstr(p); 119 120We now start a top-down look at the code: 121 122MAIN.C: The main routine performs some initialization, executes 123the user's profile if necessary, and calls cmdloop. Cmdloop 124repeatedly parses and executes commands. 125 126OPTIONS.C: This file contains the option processing code. It is 127called from main to parse the shell arguments when the shell is 128invoked, and it also contains the set builtin. The -i and -j op- 129tions (the latter turns on job control) require changes in signal 130handling. The routines setjobctl (in jobs.c) and setinteractive 131(in trap.c) are called to handle changes to these options. 132 133PARSING: The parser code is all in parser.c. A recursive des- 134cent parser is used. Syntax tables (generated by mksyntax) are 135used to classify characters during lexical analysis. There are 136three tables: one for normal use, one for use when inside single 137quotes, and one for use when inside double quotes. The tables 138are machine dependent because they are indexed by character vari- 139ables and the range of a char varies from machine to machine. 140 141PARSE OUTPUT: The output of the parser consists of a tree of 142nodes. The various types of nodes are defined in the file node- 143types. 144 145Nodes of type NARG are used to represent both words and the con- 146tents of here documents. An early version of ash kept the con- 147tents of here documents in temporary files, but keeping here do- 148cuments in memory typically results in significantly better per- 149formance. It would have been nice to make it an option to use 150temporary files for here documents, for the benefit of small 151machines, but the code to keep track of when to delete the tem- 152porary files was complex and I never fixed all the bugs in it. 153(AT&T has been maintaining the Bourne shell for more than ten 154years, and to the best of my knowledge they still haven't gotten 155it to handle temporary files correctly in obscure cases.) 156 157The text field of a NARG structure points to the text of the 158word. The text consists of ordinary characters and a number of 159special codes defined in parser.h. The special codes are: 160 161 CTLVAR Variable substitution 162 CTLENDVAR End of variable substitution 163 CTLBACKQ Command substitution 164 CTLBACKQ|CTLQUOTE Command substitution inside double quotes 165 CTLESC Escape next character 166 167A variable substitution contains the following elements: 168 169 CTLVAR type name '=' [ alternative-text CTLENDVAR ] 170 171The type field is a single character specifying the type of sub- 172stitution. The possible types are: 173 174 VSNORMAL $var 175 VSMINUS ${var-text} 176 VSMINUS|VSNUL ${var:-text} 177 VSPLUS ${var+text} 178 VSPLUS|VSNUL ${var:+text} 179 VSQUESTION ${var?text} 180 VSQUESTION|VSNUL ${var:?text} 181 VSASSIGN ${var=text} 182 VSASSIGN|VSNUL ${var:=text} 183 184In addition, the type field will have the VSQUOTE flag set if the 185variable is enclosed in double quotes. The name of the variable 186comes next, terminated by an equals sign. If the type is not 187VSNORMAL, then the text field in the substitution follows, ter- 188minated by a CTLENDVAR byte. 189 190Commands in back quotes are parsed and stored in a linked list. 191The locations of these commands in the string are indicated by 192CTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether 193the back quotes were enclosed in double quotes. 194 195The character CTLESC escapes the next character, so that in case 196any of the CTL characters mentioned above appear in the input, 197they can be passed through transparently. CTLESC is also used to 198escape '*', '?', '[', and '!' characters which were quoted by the 199user and thus should not be used for file name generation. 200 201CTLESC characters have proved to be particularly tricky to get 202right. In the case of here documents which are not subject to 203variable and command substitution, the parser doesn't insert any 204CTLESC characters to begin with (so the contents of the text 205field can be written without any processing). Other here docu- 206ments, and words which are not subject to splitting and file name 207generation, have the CTLESC characters removed during the vari- 208able and command substitution phase. Words which are subject to 209splitting and file name generation have the CTLESC characters re- 210moved as part of the file name phase. 211 212EXECUTION: Command execution is handled by the following files: 213 eval.c The top level routines. 214 redir.c Code to handle redirection of input and output. 215 jobs.c Code to handle forking, waiting, and job control. 216 exec.c Code to do path searches and the actual exec sys call. 217 expand.c Code to evaluate arguments. 218 var.c Maintains the variable symbol table. Called from expand.c. 219 220EVAL.C: Evaltree recursively executes a parse tree. The exit 221status is returned in the global variable exitstatus. The alter- 222native entry evalbackcmd is called to evaluate commands in back 223quotes. It saves the result in memory if the command is a buil- 224tin; otherwise it forks off a child to execute the command and 225connects the standard output of the child to a pipe. 226 227JOBS.C: To create a process, you call makejob to return a job 228structure, and then call forkshell (passing the job structure as 229an argument) to create the process. Waitforjob waits for a job 230to complete. These routines take care of process groups if job 231control is defined. 232 233REDIR.C: Ash allows file descriptors to be redirected and then 234restored without forking off a child process. This is accom- 235plished by duplicating the original file descriptors. The redir- 236tab structure records where the file descriptors have been dupli- 237cated to. 238 239EXEC.C: The routine find_command locates a command, and enters 240the command in the hash table if it is not already there. The 241third argument specifies whether it is to print an error message 242if the command is not found. (When a pipeline is set up, 243find_command is called for all the commands in the pipeline be- 244fore any forking is done, so to get the commands into the hash 245table of the parent process. But to make command hashing as 246transparent as possible, we silently ignore errors at that point 247and only print error messages if the command cannot be found 248later.) 249 250The routine shellexec is the interface to the exec system call. 251 252EXPAND.C: Arguments are processed in three passes. The first 253(performed by the routine argstr) performs variable and command 254substitution. The second (ifsbreakup) performs word splitting 255and the third (expandmeta) performs file name generation. If the 256"/u" directory is simulated, then when "/u/username" is replaced 257by the user's home directory, the flag "didudir" is set. This 258tells the cd command that it should print out the directory name, 259just as it would if the "/u" directory were implemented using 260symbolic links. 261 262VAR.C: Variables are stored in a hash table. Probably we should 263switch to extensible hashing. The variable name is stored in the 264same string as the value (using the format "name=value") so that 265no string copying is needed to create the environment of a com- 266mand. Variables which the shell references internally are preal- 267located so that the shell can reference the values of these vari- 268ables without doing a lookup. 269 270When a program is run, the code in eval.c sticks any environment 271variables which precede the command (as in "PATH=xxx command") in 272the variable table as the simplest way to strip duplicates, and 273then calls "environment" to get the value of the environment. 274There are two consequences of this. First, if an assignment to 275PATH precedes the command, the value of PATH before the assign- 276ment must be remembered and passed to shellexec. Second, if the 277program turns out to be a shell procedure, the strings from the 278environment variables which preceded the command must be pulled 279out of the table and replaced with strings obtained from malloc, 280since the former will automatically be freed when the stack (see 281the entry on memalloc.c) is emptied. 282 283BUILTIN COMMANDS: The procedures for handling these are scat- 284tered throughout the code, depending on which location appears 285most appropriate. They can be recognized because their names al- 286ways end in "cmd". The mapping from names to procedures is 287specified in the file builtins, which is processed by the mkbuilt- 288ins command. 289 290A builtin command is invoked with argc and argv set up like a 291normal program. A builtin command is allowed to overwrite its 292arguments. Builtin routines can call nextopt to do option pars- 293ing. This is kind of like getopt, but you don't pass argc and 294argv to it. Builtin routines can also call error. This routine 295normally terminates the shell (or returns to the main command 296loop if the shell is interactive), but when called from a builtin 297command it causes the builtin command to terminate with an exit 298status of 2. 299 300The directory bltins contains commands which can be compiled in- 301dependently but can also be built into the shell for efficiency 302reasons. The makefile in this directory compiles these programs 303in the normal fashion (so that they can be run regardless of 304whether the invoker is ash), but also creates a library named 305bltinlib.a which can be linked with ash. The header file bltin.h 306takes care of most of the differences between the ash and the 307stand-alone environment. The user should call the main routine 308"main", and #define main to be the name of the routine to use 309when the program is linked into ash. This #define should appear 310before bltin.h is included; bltin.h will #undef main if the pro- 311gram is to be compiled stand-alone. 312 313CD.C: This file defines the cd and pwd builtins. The pwd com- 314mand runs /bin/pwd the first time it is invoked (unless the user 315has already done a cd to an absolute pathname), but then 316remembers the current directory and updates it when the cd com- 317mand is run, so subsequent pwd commands run very fast. The main 318complication in the cd command is in the docd command, which 319resolves symbolic links into actual names and informs the user 320where the user ended up if he crossed a symbolic link. 321 322SIGNALS: Trap.c implements the trap command. The routine set- 323signal figures out what action should be taken when a signal is 324received and invokes the signal system call to set the signal ac- 325tion appropriately. When a signal that a user has set a trap for 326is caught, the routine "onsig" sets a flag. The routine dotrap 327is called at appropriate points to actually handle the signal. 328When an interrupt is caught and no trap has been set for that 329signal, the routine "onint" in error.c is called. 330 331OUTPUT: Ash uses it's own output routines. There are three out- 332put structures allocated. "Output" represents the standard out- 333put, "errout" the standard error, and "memout" contains output 334which is to be stored in memory. This last is used when a buil- 335tin command appears in backquotes, to allow its output to be col- 336lected without doing any I/O through the UNIX operating system. 337The variables out1 and out2 normally point to output and errout, 338respectively, but they are set to point to memout when appropri- 339ate inside backquotes. 340 341INPUT: The basic input routine is pgetc, which reads from the 342current input file. There is a stack of input files; the current 343input file is the top file on this stack. The code allows the 344input to come from a string rather than a file. (This is for the 345-c option and the "." and eval builtin commands.) The global 346variable plinno is saved and restored when files are pushed and 347popped from the stack. The parser routines store the number of 348the current line in this variable. 349 350DEBUGGING: If DEBUG is defined in shell.h, then the shell will 351write debugging information to the file $HOME/trace. Most of 352this is done using the TRACE macro, which takes a set of printf 353arguments inside two sets of parenthesis. Example: 354"TRACE(("n=%d0, n))". The double parenthesis are necessary be- 355cause the preprocessor can't handle functions with a variable 356number of arguments. Defining DEBUG also causes the shell to 357generate a core dump if it is sent a quit signal. The tracing 358code is in show.c. 359