1<title>CGI Server Extensions</title> 2 3<h2>1.0 Introduction</h2> 4 5If you have a [./server/|Fossil server] for your project, 6you can add [./aboutcgi.wiki|CGI] 7extensions to that server. These extensions work like 8any other CGI program, except that they also have access to the Fossil 9login information and can (optionally) leverage the "[./customskin.md|skins]" 10of Fossil so that they appear to be more tightly integrated into the project. 11 12An example of where this is useful is the 13[https://sqlite.org/src/ext/checklist|checklist application] on 14the [https://sqlite.org/|SQLite] project. The checklist 15helps the SQLite developers track which release tests have passed, 16or failed, or are still to be done. The checklist program began as a 17stand-alone CGI which kept its own private user database and implemented 18its own permissions and login system and provided its own CSS. By 19converting checklist into a Fossil extension, the same login that works 20for the [https://sqlite.org/src|main SQLite source repository] also works 21for the checklist. Permission to change elements of the checklist 22is tied on permission to check-in to the main source repository. And 23the standard Fossil header menu and footer appear on each page of 24the checklist. 25 26<h2>2.0 How It Works</h2> 27 28CGI Extensions are disabled by default. 29An administrator activates the CGI extension mechanism by specifying 30an "Extension Root Directory" or "extroot" as part of the 31[./server/index.html|server setup]. 32If the Fossil server is itself run as 33[./server/any/cgi.md|CGI], then add a line to the 34[./cgi.wiki#extroot|CGI script file] that says: 35 36<blockquote><pre> 37extroot: <i>DIRECTORY</i> 38</pre></blockquote> 39 40Or, if the Fossil server is being run using the 41"[./server/any/none.md|fossil server]" or 42"[./server/any/none.md|fossil ui]" or 43"[./server/any/inetd.md|fossil http]" commands, then add an extra 44"--extroot <i>DIRECTORY</i>" option to that command. 45 46The <i>DIRECTORY</i> is the DOCUMENT_ROOT for the CGI. 47Files in the DOCUMENT_ROOT are accessed via URLs like this: 48 49<blockquote> 50https://example-project.org/ext/<i>FILENAME</i> 51</blockquote> 52 53In other words, access files in DOCUMENT_ROOT by appending the filename 54relative to DOCUMENT_ROOT to the [/help?cmd=/ext|/ext] 55page of the Fossil server. 56Files that are readable but not executable are returned as static 57content. Files that are executable are run as CGI. 58 59<h3>2.1 Example #1</h3> 60 61The source code repository for SQLite is a Fossil server that is run 62as CGI. The URL for the source code repository is [https://sqlite.org/src]. 63The CGI script looks like this: 64 65<blockquote><verbatim> 66#!/usr/bin/fossil 67repository: /fossil/sqlite.fossil 68errorlog: /logs/errors.txt 69extroot: /sqlite-src-ext 70</verbatim></blockquote> 71 72The "extroot: /sqlite-src-ext" line tells Fossil that it should look for 73extension CGIs in the /sqlite-src-ext directory. (All of this is happening 74inside of a chroot jail, so putting the document root in a top-level 75directory is a reasonable thing to do.) 76 77When a URL like "https://sqlite.org/src/ext/checklist" is received by the 78main webserver, it figures out that the /src part refers to the main 79Fossil CGI script and so it runs that script. Fossil gets the remainder 80of the URL to work with: "/ext/checklist". Fossil extracts the "/ext" 81prefix and uses that to determine that this a CGI extension request. 82Then it takes the leftover "/checklist" part and appends it to the 83"extroot" to get the filename "/sqlite-src-ext/checklist". Fossil finds 84that file to be executable, so it runs it as CGI and returns the result. 85 86The /sqlite-src-ext/checklist file is a 87[https://wapp.tcl.tk|Wapp program]. The current source code to the 88this program can be seen at 89[https://www.sqlite.org/src/ext/checklist/3070700/self] and 90recent historical versions are available at 91[https://sqlite.org/docsrc/finfo/misc/checklist.tcl] with 92older legacy at [https://sqlite.org/checklistapp/timeline?n1=all] 93 94There is a cascade of CGIs happening here. The web server that receives 95the initial HTTP request runs Fossil as a CGI based on the 96"https://sqlite.org/src" portion of the URL. The Fossil instance then 97runs the checklist sub-CGI based on the "/ext/checklists" suffix. The 98output of the sub-CGI is read by Fossil and then relayed on to the 99main web server which in turn relays the result back to the original client. 100 101<h3>2.2 Example #2</h3> 102 103The [https://fossil-scm.org/home|Fossil self-hosting repository] is also 104a CGI that looks like this: 105 106<blockquote><verbatim> 107#!/usr/bin/fossil 108repository: /fossil/fossil.fossil 109errorlog: /logs/errors.txt 110extroot: /fossil-extroot 111</verbatim></blockquote> 112 113The extroot for this Fossil server is /fossil-extroot and in that directory 114is an executable file named "fileup1" - another [https://wapp.tcl.tk|Wapp] 115script. (The extension mechanism is not required to use Wapp. You can use 116any kind of program you like. But the creator of SQLite and Fossil is fond 117of [https://www.tcl.tk|Tcl/Tk] and so he tends to gravitate toward Tcl-based 118technologies like Wapp.) The fileup1 script is a demo program that lets 119the user upload a file using a form, and then displays that file in the reply. 120There is a link on the page that causes the fileup1 script to return a copy 121of its own source-code, so you can see how it works. 122 123<h2>3.0 CGI Inputs</h2> 124 125The /ext extension mechanism is an ordinary CGI interface. Parameters 126are passed to the CGI program using environment variables. The following 127standard CGI environment variables are supported: 128 129 * AUTH_TYPE 130 * AUTH_CONTENT 131 * CONTENT_LENGTH 132 * CONTENT_TYPE 133 * DOCUMENT_ROOT 134 * GATEWAY_INTERFACE 135 * HTTPS 136 * HTTP_ACCEPT 137 * HTTP_ACCEPT_ENCODING 138 * HTTP_COOKIE 139 * HTTP_HOST 140 * HTTP_IF_MODIFIED_SINCE 141 * HTTP_IF_NONE_MATCH 142 * HTTP_REFERER 143 * HTTP_USER_AGENT 144 * PATH_INFO 145 * QUERY_STRING 146 * REMOTE_ADDR 147 * REMOTE_USER 148 * REQUEST_METHOD 149 * REQUEST_SCHEME 150 * REQUEST_URI 151 * SCRIPT_DIRECTORY 152 * SCRIPT_FILENAME 153 * SCRIPT_NAME 154 * SERVER_NAME 155 * SERVER_PORT 156 * SERVER_PROTOCOL 157 158Do a web search for 159"[https://duckduckgo.com/?q=cgi+environment_variables|cgi environment variables]" 160to find more detail about what each of the above variables mean and how 161they are used. 162Live listings of the values of some or all of these environment variables 163can be found at links like these: 164 165 * [https://fossil-scm.org/home/test_env] 166 * [https://sqlite.org/src/ext/checklist/top/env] 167 168In addition to the standard CGI environment variables listed above, 169Fossil adds the following: 170 171 * FOSSIL_CAPABILITIES 172 * FOSSIL_NONCE 173 * FOSSIL_REPOSITORY 174 * FOSSIL_URI 175 * FOSSIL_USER 176 177The FOSSIL_USER string is the name of the logged-in user. This variable 178is missing or is an empty string if the user is not logged in. The 179FOSSIL_CAPABILITIES string is a list of 180[./caps/ref.html|Fossil capabilities] that 181indicate what permissions the user has on the Fossil repository. 182The FOSSIL_REPOSITORY environment variable gives the filename of the 183Fossil repository that is running. The FOSSIL_URI variable shows the 184prefix of the REQUEST_URI that is the Fossil CGI script, or is an 185empty string if Fossil is being run by some method other than CGI. 186 187The [https://sqlite.org/src/ext/checklist|checklist application] uses the 188FOSSIL_USER environment variable to determine the name of the user and 189the FOSSIL_CAPABILITIES variable to determine if the user is allowed to 190mark off changes to the checklist. Only users with check-in permission 191to the Fossil repository are allowed to mark off checklist items. That 192means that the FOSSIL_CAPABILITIES string must contain the letter "i". 193Search for "FOSSIL_CAPABILITIES" in the 194[https://sqlite.org/src/ext/checklist/top/self|source listing] to see how 195this happens. 196 197If the CGI output is one of the forms for which Fossil inserts its own 198header and footer, then the inserted header will include a 199Content Security Policy (CSP) restriction on the use of javascript within 200the webpage. Any <script>...</script> elements within the 201CGI output must include a nonce or else they will be suppressed by the 202web browser. The FOSSIL_NONCE variable contains the value of that nonce. 203So, in other words, to get javascript to work, it must be enclosed in: 204 205<blockquote><verbatim> 206<script nonce='$FOSSIL_NONCE'>...</script> 207</verbatim></blockquote> 208 209Except, of course, the $FOSSIL_NONCE is replaced by the value of the 210FOSSIL_NONCE environment variable. 211 212<h3>3.1 Input Content</h3> 213 214If the HTTP request includes content (for example if this is a POST request) 215then the CONTENT_LENGTH value will be positive and the data for the content 216will be readable on standard input. 217 218 219<h2>4.0 CGI Outputs</h2> 220 221CGI programs construct a reply by writing to standard output. The first 222few lines of output are parameters intended for the web server that invoked 223the CGI. These are followed by a blank line and then the content. 224 225Typical parameter output looks like this: 226 227<blockquote><verbatim> 228Status: 200 OK 229Content-Type: text/html 230</verbatim></blockquote> 231 232CGI programs can return any content type they want - they are not restricted 233to text replies. It is OK for a CGI program to return (for example) 234image/png. 235 236The fields of the CGI response header can be any valid HTTP header fields. 237Those that Fossil does not understand are simply relayed back to up the 238line to the requester. 239 240Fossil takes special action with some content types. If the Content-Type 241is "text/x-fossil-wiki" or "text/x-markdown" then Fossil 242converts the content from [/wiki_rules|Fossil-Wiki] or 243[/md_rules|Markdown] into HTML, adding its 244own header and footer text according to the repository skin. Content 245of type "text/html" is normally passed straight through 246unchanged. However, if the text/html content is of the form: 247 248<blockquote><verbatim> 249<div class='fossil-doc' data-title='DOCUMENT TITLE'> 250... HTML content there ... 251</div> 252</verbatim></blockquote> 253 254In other words, if the outer-most markup of the HTML is a <div> 255element with a single class of "fossil-doc", 256then Fossil will adds its own header and footer to the HTML. The 257page title contained in the added header will be extracted from the 258"data-title" attribute. 259 260Except for the three cases noted above, Fossil makes no changes or 261additions to the CGI-generated content. Fossil just passes the verbatim 262content back up the stack towards the requester. 263 264<h3>4.1 <tt>GATEWAY_INTERFACE</tt> and Recursive Calls to fossil</h3> 265 266Like many CGI-aware applications, if fossil sees the environment 267variable <tt>GATEWAY_INTERFACE</tt> when it starts up, it assumes it 268is running in a CGI environment and behaves differently than when it 269is run in a non-CGI interactive session. If you intend to run fossil 270itself from within an extension CGI script, e.g. to run a query 271against the repository or simply fetch the fossil binary version, make 272sure to <em>unset</em> the <tt>GATEWAY_INTERFACE</tt> environment 273variable before doing so, otherwise the invocation will behave as if 274it's being run in CGI mode. 275 276<h2>5.0 Filename Restrictions</h2> 277 278For security reasons, Fossil places restrictions on the names of files 279in the extroot directory that can participate in the extension CGI 280mechanism: 281 282 1. Filenames must consist of only ASCII alphanumeric characters, 283 ".", "_", and "-", and of course "/" as the file separator. 284 Files with names that includes spaces or 285 other punctuation or special characters are ignored. 286 287 2. No element of the pathname can begin with "." or "-". Files or 288 directories whose names begin with "." or "-" are ignored. 289 290If a CGI program requires separate data files, it is safe to put those 291files in the same directory as the CGI program itself as long as the names 292of the data files contain special characters that cause them to be ignored 293by Fossil. 294 295<h2>6.0 Access Permissions</h2> 296 297CGI extension files and programs are accessible to everyone. 298 299When CGI extensions have been enabled (using either "extroot:" in the 300CGI file or the --extroot option for other server methods) all files 301in the extension root directory hierarchy, except special filenames 302identified previously, are accessible to all users. Users do not 303have to have "Read" privilege, or any other privilege, in order to 304access the extensions. 305 306This is by design. The CGI extension mechanism is intended to operate 307in the same way as a traditional web-server. 308 309CGI programs that want to restrict access 310can examine the FOSSIL_CAPABILITIES and/or FOSSIL_USER environment variables. 311In other words, access control is the responsibility of the individual 312extension programs. 313 314 315<h2>7.0 Trouble-Shooting Hints</h2> 316 317Remember that the /ext will return any file in the extroot directory 318hierarchy as static content if the file is readable but not executable. 319When initially setting up the /ext mechanism, it is sometimes helpful 320to verify that you are able to receive static content prior to starting 321work on your CGIs. Also remember that CGIs must be 322executable files. 323 324Fossil likes to run inside a chroot jail, and will automatically put 325itself inside a chroot jail if it can. The sub-CGI program will also 326run inside this same chroot jail. Make sure all embedded pathnames 327have been adjusted accordingly and that all resources needed by the 328CGI program are available within the chroot jail. 329 330If anything goes wrong while trying to process an /ext page, Fossil 331returns a 404 Not Found error with no details. However, if the requester 332is logged in as a user that has <b>[./caps/ref.html#D | Debug]</b> capability 333then additional diagnostic information may be included in the output. 334 335If the /ext page has a "fossil-ext-debug=1" query parameter and if 336the requester is logged in as a user with Debug privilege, then the 337CGI output is returned verbatim, as text/plain and with the original 338header intact. This is useful for diagnosing problems with the 339CGI script. 340