1--- 2stage: Enablement 3group: Distribution 4info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments 5--- 6 7# Finding relevant log entries with a correlation ID **(FREE SELF)** 8 9GitLab instances log a unique request tracking ID (known as the 10"correlation ID") for most requests. Each individual request to GitLab gets 11its own correlation ID, which then gets logged in each GitLab component's logs for that 12request. This makes it easier to trace behavior in a 13distributed system. Without this ID it can be difficult or 14impossible to match correlating log entries. 15 16## Identify the correlation ID for a request 17 18The correlation ID is logged in structured logs under the key `correlation_id` 19and in all response headers GitLab sends under the header `x-request-id`. 20You can find your correlation ID by searching in either place. 21 22### Getting the correlation ID in your browser 23 24You can use your browser's developer tools to monitor and inspect network 25activity with the site that you're visiting. See the links below for network monitoring 26documentation for some popular browsers. 27 28- [Network Monitor - Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor) 29- [Inspect Network Activity In Chrome DevTools](https://developer.chrome.com/docs/devtools/network/) 30- [Safari Web Development Tools](https://developer.apple.com/safari/tools/) 31- [Microsoft Edge Network panel](https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/network/) 32 33To locate a relevant request and view its correlation ID: 34 351. Enable persistent logging in your network monitor. Some actions in GitLab will redirect you quickly after you submit a form, so this will help capture all relevant activity. 361. To help isolate the requests you are looking for, you can filter for `document` requests. 371. Click the request of interest to view further detail. 381. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a 39value that was randomly generated by GitLab for the request. 40 41See the following example: 42 43![Firefox's network monitor showing an request ID header](img/network_monitor_xid.png) 44 45### Getting the correlation ID from your logs 46 47Another approach to finding the correct correlation ID is to search or watch 48your logs and find the `correlation_id` value for the log entry that you're 49watching for. 50 51For example, let's say that you want learn what's happening or breaking when 52you reproduce an action in GitLab. You could tail the GitLab logs, filtering 53to requests by your user, and then watch the requests until you see what you're 54interested in. 55 56### Getting the correlation ID from curl 57 58If you're using `curl` then you can use the verbose option to show request and response headers, as well as other debug information. 59 60```shell 61➜ ~ curl --verbose "https://gitlab.example.com/api/v4/projects" 62# look for a line that looks like this 63< x-request-id: 4rAMkV3gof4 64``` 65 66#### Using jq 67 68This example uses [jq](https://stedolan.github.io/jq/) to filter results and 69display values we most likely care about. 70 71```shell 72sudo gitlab-ctl tail gitlab-rails/production_json.log | jq 'select(.username == "bob") | "User: \(.username), \(.method) \(.path), \(.controller)#\(.action), ID: \(.correlation_id)"' 73``` 74 75```plaintext 76"User: bob, GET /root/linux, ProjectsController#show, ID: U7k7fh6NpW3" 77"User: bob, GET /root/linux/commits/master/signatures, Projects::CommitsController#signatures, ID: XPIHpctzEg1" 78"User: bob, GET /root/linux/blob/master/README, Projects::BlobController#show, ID: LOt9hgi1TV4" 79``` 80 81#### Using grep 82 83This example uses only `grep` and `tr`, which are more likely to be installed than `jq`. 84 85```shell 86sudo gitlab-ctl tail gitlab-rails/production_json.log | grep '"username":"bob"' | tr ',' '\n' | egrep 'method|path|correlation_id' 87``` 88 89```plaintext 90{"method":"GET" 91"path":"/root/linux" 92"username":"bob" 93"correlation_id":"U7k7fh6NpW3"} 94{"method":"GET" 95"path":"/root/linux/commits/master/signatures" 96"username":"bob" 97"correlation_id":"XPIHpctzEg1"} 98{"method":"GET" 99"path":"/root/linux/blob/master/README" 100"username":"bob" 101"correlation_id":"LOt9hgi1TV4"} 102``` 103 104## Searching your logs for the correlation ID 105 106Once you have the correlation ID you can start searching for relevant log 107entries. You can filter the lines by the correlation ID itself. 108Combining a `find` and `grep` should be sufficient to find the entries you are looking for. 109 110```shell 111# find <gitlab log directory> -type f -mtime -0 exec grep '<correlation ID>' '{}' '+' 112find /var/log/gitlab -type f -mtime 0 -exec grep 'LOt9hgi1TV4' '{}' '+' 113``` 114 115```plaintext 116/var/log/gitlab/gitlab-workhorse/current:{"correlation_id":"LOt9hgi1TV4","duration_ms":2478,"host":"gitlab.domain.tld","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.domain.tld/root/linux","remote_addr":"68.0.116.160:0","remote_ip":"[filtered]","status":200,"system":"http","time":"2019-09-17T22:17:19Z","uri":"/root/linux/blob/master/README?format=json\u0026viewer=rich","user_agent":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","written_bytes":1743} 117/var/log/gitlab/gitaly/current:{"correlation_id":"LOt9hgi1TV4","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-09-17T22:17:47Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"root/linux","grpc.request.glRepository":"project-1","grpc.request.repoPath":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-09-17T22:17:17Z","grpc.time_ms":2319.161,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","span.kind":"server","system":"grpc","time":"2019-09-17T22:17:19Z"} 118/var/log/gitlab/gitlab-rails/production_json.log:{"method":"GET","path":"/root/linux/blob/master/README","format":"json","controller":"Projects::BlobController","action":"show","status":200,"duration":2448.77,"view":0.49,"db":21.63,"time":"2019-09-17T22:17:19.800Z","params":[{"key":"viewer","value":"rich"},{"key":"namespace_id","value":"root"},{"key":"project_id","value":"linux"},{"key":"id","value":"master/README"}],"remote_ip":"[filtered]","user_id":2,"username":"bob","ua":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","queue_duration":3.38,"gitaly_calls":1,"gitaly_duration":0.77,"rugged_calls":4,"rugged_duration_ms":28.74,"correlation_id":"LOt9hgi1TV4"} 119``` 120 121### Searching in distributed architectures 122 123If you have done some horizontal scaling in your GitLab infrastructure, then 124you will need to search across _all_ of your GitLab nodes. You can do this with 125some sort of log aggregation software like Loki, ELK, Splunk, or others. 126 127You can use a tool like Ansible or PSSH (parallel SSH) that can execute identical commands across your servers in 128parallel, or craft your own solution. 129 130### Viewing the request in the Performance Bar 131 132You can use the [performance bar](../monitoring/performance/performance_bar.md) to view interesting data including calls made to SQL and Gitaly. 133 134To view the data, the correlation ID of the request must match the same session as the user 135viewing the performance bar. For API requests, this means that you must perform the request 136using the session cookie of the signed-in user. 137 138For example, if you want to view the database queries executed for the following API endpoint: 139 140```shell 141https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1 142``` 143 144First, enable the **Developer Tools** panel. See [Getting the correlation ID in your browser](#getting-the-correlation-id-in-your-browser) for details on how to do this. 145 146After developer tools have been enabled, obtain a session cookie as follows: 147 1481. Visit <https://gitlab.com> while logged in. 1491. Optional. Select **Fetch/XHR** request filter in the **Developer Tools** panel. This step is described for Google Chrome developer tools and is not strictly necessary, it just makes it easier to find the correct request. 1501. Select the `results?request_id=<some-request-id>` request on the left hand side. 1511. The session cookie is displayed under the `Request Headers` section of the `Headers` panel. Right-click on the cookie value and select `Copy value`. 152 153![Obtaining a session cookie for request](img/obtaining-a-session-cookie-for-request_v14_3.png) 154 155You have the value of the session cookie copied to your clipboard, for example: 156 157```shell 158experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow 159``` 160 161Use the value of the session cookie to craft an API request by pasting it into a custom header of a `curl` request: 162 163```shell 164$ curl --include "https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1" \ 165--header 'cookie: experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow' 166 167 date: Tue, 28 Sep 2021 03:55:33 GMT 168 content-type: application/json 169 ... 170 x-request-id: 01FGN8P881GF2E5J91JYA338Y3 171 ... 172 [ 173 { 174 "id":27497069, 175 "description":"Analyzer for images used on live K8S containers based on Starboard" 176 }, 177 "container_registry_image_prefix":"registry.gitlab.com/gitlab-org/security-products/analyzers/cluster-image-scanning", 178 "..." 179 ] 180``` 181 182The response contains the data from the API endpoint, and a `correlation_id` value, returned in the `x-request-id` header, as described in the [Identify the correlation ID for a request](#identify-the-correlation-id-for-a-request) section. 183 184You can then view the database details for this request: 185 1861. Paste the `x-request-id` value into the `request details` field of the [performance bar](../monitoring/performance/performance_bar.md) and press <kbd>Enter/Return</kbd>. This example uses the `x-request-id` value `01FGN8P881GF2E5J91JYA338Y3`, returned by the above response: 187 188 ![Paste request ID into progress bar](img/paste-request-id-into-progress-bar_v14_3.png) 189 1901. A new request is inserted into the `Request Selector` dropdown on the right-hand side of the Performance Bar. Select the new request to view the metrics of the API request: 191 192 ![Select request ID from request selector drop down menu](img/select-request-id-from-request-selector-drop-down-menu_v14_3.png) 193 194 <!-- vale gitlab.Substitutions = NO --> 1951. Select the `pg` link in the Progress Bar to view the database queries executed by the API request: 196 197 ![View pg database details](img/view-pg-details_v14_3.png) 198 <!-- vale gitlab.Substitutions = YES --> 199 200 The database query dialog is displayed: 201 202 ![Database query dialog](img/database-query-dialog_v14_3.png) 203