1# Terminal support 2 3In some cases, GitLab can provide in-browser terminal access to an 4environment (which is a running server or container, onto which a 5project has been deployed) through a WebSocket. Workhorse manages 6the WebSocket upgrade and long-lived connection to the terminal for 7the environment, which frees up GitLab to process other requests. 8 9This document outlines the architecture of these connections. 10 11## Introduction to WebSockets 12 13A websocket is an "upgraded" HTTP/1.1 request. Their purpose is to 14permit bidirectional communication between a client and a server. 15**Websockets are not HTTP**. Clients can send messages (known as 16frames) to the server at any time, and vice-versa. Client messages 17are not necessarily requests, and server messages are not necessarily 18responses. WebSocket URLs have schemes like `ws://` (unencrypted) or 19`wss://` (TLS-secured). 20 21When requesting an upgrade to WebSocket, the browser sends a HTTP/1.1 22request that looks like this: 23 24``` 25GET /path.ws HTTP/1.1 26Connection: upgrade 27Upgrade: websocket 28Sec-WebSocket-Protocol: terminal.gitlab.com 29# More headers, including security measures 30``` 31 32At this point, the connection is still HTTP, so this is a request and 33the server can send a normal HTTP response, including `404 Not Found`, 34`500 Internal Server Error`, etc. 35 36If the server decides to permit the upgrade, it will send a HTTP 37`101 Switching Protocols` response. From this point, the connection 38is no longer HTTP. It is a WebSocket and frames, not HTTP requests, 39will flow over it. The connection will persist until the client or 40server closes the connection. 41 42In addition to the subprotocol, individual websocket frames may 43also specify a message type - examples include `BinaryMessage`, 44`TextMessage`, `Ping`, `Pong` or `Close`. Only binary frames can 45contain arbitrary data - other frames are expected to be valid 46UTF-8 strings, in addition to any subprotocol expectations. 47 48## Browser to Workhorse 49 50GitLab serves a JavaScript terminal emulator to the browser on 51a URL like `https://gitlab.com/group/project/environments/1/terminal`. 52This opens a websocket connection to, e.g., 53`wss://gitlab.com/group/project/environments/1/terminal.ws`, 54This endpoint doesn't exist in GitLab - only in Workhorse. 55 56When receiving the connection, Workhorse first checks that the 57client is authorized to access the requested terminal. It does 58this by performing a "preauthentication" request to GitLab. 59 60If the client has the appropriate permissions and the terminal 61exists, GitLab responds with a successful response that includes 62details of the terminal that the client should be connected to. 63Otherwise, it returns an appropriate HTTP error response. 64 65Errors are passed back to the client as HTTP responses, but if 66GitLab returns valid terminal details to Workhorse, it will 67connect to the specified terminal, upgrade the browser to a 68WebSocket, and proxy between the two connections for as long 69as the browser's credentials are valid. Workhorse will also 70send regular `PingMessage` control frames to the browser, to 71keep intervening proxies from terminating the connection 72while the browser is present. 73 74The browser must request an upgrade with a specific subprotocol: 75 76### `terminal.gitlab.com` 77 78This subprotocol considers `TextMessage` frames to be invalid. 79Control frames, such as `PingMessage` or `CloseMessage`, have 80their usual meanings. 81 82`BinaryMessage` frames sent from the browser to the server are 83arbitrary terminal input. 84 85`BinaryMessage` frames sent from the server to the browser are 86arbitrary terminal output. 87 88These frames are expected to contain ANSI terminal control codes 89and may be in any encoding. 90 91### `base64.terminal.gitlab.com` 92 93This subprotocol considers `BinaryMessage` frames to be invalid. 94Control frames, such as `PingMessage` or `CloseMessage`, have 95their usual meanings. 96 97`TextMessage` frames sent from the browser to the server are 98base64-encoded arbitrary terminal input (so the server must 99base64-decode them before inputting them). 100 101`TextMessage` frames sent from the server to the browser are 102base64-encoded arbitrary terminal output (so the browser must 103base64-decode them before outputting them). 104 105In their base64-encoded form, these frames are expected to 106contain ANSI terminal control codes, and may be in any encoding. 107 108## Workhorse to GitLab 109 110Before upgrading the browser, Workhorse sends a normal HTTP 111request to GitLab on a URL like 112`https://gitlab.com/group/project/environments/1/terminal.ws/authorize`. 113This returns a JSON response containing details of where the 114terminal can be found, and how to connect it. In particular, 115the following details are returned in case of success: 116 117* WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1` 118* WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]` 119* Headers to send, e.g.: `Authorization: Token xxyyz..` 120* Certificate authority to verify `wss` connections with (optional) 121 122Workhorse periodically re-checks this endpoint, and if it gets an 123error response, or the details of the terminal change, it will 124terminate the websocket session. 125 126## Workhorse to Terminal 127 128In GitLab, environments may have a deployment service (e.g., 129`KubernetesService`) associated with them. This service knows 130where the terminals for an environment may be found, and these 131details are returned to Workhorse by GitLab. 132 133These URLs are *also* WebSocket URLs, and GitLab tells Workhorse 134which subprotocols to speak over the connection, along with any 135authentication details required by the remote end. 136 137Before upgrading the browser's connection to a websocket, 138Workhorse opens a HTTP client connection, according to the 139details given to it by Workhorse, and attempts to upgrade 140that connection to a websocket. If it fails, an error 141response is sent to the browser; otherwise, the browser is 142also upgraded. 143 144Workhorse now has two websocket connections, albeit with 145differing subprotocols. It decodes incoming frames from the 146browser, re-encodes them to the terminal's subprotocol, and 147sends them to the terminal. Similarly, it decodes incoming 148frames from the terminal, re-encodes them to the browser's 149subprotocol, and sends them to the browser. 150 151When either connection closes or enters an error state, 152Workhorse detects the error and closes the other connection, 153terminating the terminal session. If the browser is the 154connection that has disconnected, Workhorse will send an ANSI 155`End of Transmission` control code (the `0x04` byte) to the 156terminal, encoded according to the appropriate subprotocol. 157Workhorse will automatically reply to any websocket ping frame 158sent by the terminal, to avoid being disconnected. 159 160Currently, Workhorse only supports the following subprotocols. 161Supporting new deployment services will require new subprotocols 162to be supported: 163 164### `channel.k8s.io` 165 166Used by Kubernetes, this subprotocol defines a simple multiplexed 167channel. 168 169Control frames have their usual meanings. `TextMessage` frames are 170invalid. `BinaryMessage` frames represent I/O to a specific file 171descriptor. 172 173The first byte of each `BinaryMessage` frame represents the file 174descriptor (fd) number, as a `uint8` (so the value `0x00` corresponds 175to fd 0, `STDIN`, while `0x01` corresponds to fd 1, `STDOUT`). 176 177The remaining bytes represent arbitrary data. For frames received 178from the server, they are bytes that have been received from that 179fd. For frames sent to the server, they are bytes that should be 180written to that fd. 181 182### `base64.channel.k8s.io` 183 184Also used by Kubernetes, this subprotocol defines a similar multiplexed 185channel to `channel.k8s.io`. The main differences are: 186 187* `TextMessage` frames are valid, rather than `BinaryMessage` frames. 188* The first byte of each `TextMessage` frame represents the file 189 descriptor as a numeric UTF-8 character, so the character `U+0030`, 190 or "0", is fd 0, STDIN). 191* The remaining bytes represent base64-encoded arbitrary data. 192