1--- 2stage: Package 3group: Package 4info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments 5--- 6 7# GitLab Container Registry administration **(FREE SELF)** 8 9With the GitLab Container Registry, every project can have its 10own space to store Docker images. 11 12Read more about the Docker Registry in [the Docker documentation](https://docs.docker.com/registry/introduction/). 13 14This document is the administrator's guide. To learn how to use the GitLab Container 15Registry, see the [user documentation](../../user/packages/container_registry/index.md). 16 17## Enable the Container Registry 18 19**Omnibus GitLab installations** 20 21If you installed GitLab by using the Omnibus installation package, the Container Registry 22may or may not be available by default. 23 24The Container Registry is automatically enabled and available on your GitLab domain, port 5050 if: 25 26- You're using the built-in [Let's Encrypt integration](https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration), and 27- You're using GitLab 12.5 or later. 28 29Otherwise, the Container Registry is not enabled. To enable it: 30 31- You can configure it for your [GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain), or 32- You can configure it for [a different domain](#configure-container-registry-under-its-own-domain). 33 34The Container Registry works under HTTPS by default. You can use HTTP 35but it's not recommended and is beyond the scope of this document. 36Read the [insecure Registry documentation](https://docs.docker.com/registry/insecure/) 37if you want to implement this. 38 39**Installations from source** 40 41If you have installed GitLab from source: 42 431. You must [install Registry](https://docs.docker.com/registry/deploying/) by yourself. 441. After the installation is complete, to enable it, you must configure the Registry's 45 settings in `gitlab.yml`. 461. Use the sample NGINX configuration file from under 47 [`lib/support/nginx/registry-ssl`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/nginx/registry-ssl) and edit it to match the 48 `host`, `port`, and TLS certificate paths. 49 50The contents of `gitlab.yml` are: 51 52```yaml 53registry: 54 enabled: true 55 host: registry.gitlab.example.com 56 port: 5005 57 api_url: http://localhost:5000/ 58 key: config/registry.key 59 path: shared/registry 60 issuer: gitlab-issuer 61``` 62 63Where: 64 65| Parameter | Description | 66| --------- | ----------- | 67| `enabled` | `true` or `false`. Enables the Registry in GitLab. By default this is `false`. | 68| `host` | The host URL under which the Registry runs and users can use. | 69| `port` | The port the external Registry domain listens on. | 70| `api_url` | The internal API URL under which the Registry is exposed. It defaults to `http://localhost:5000`. Do not change this unless you are setting up an [external Docker registry](#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). | 71| `key` | The private key location that is a pair of Registry's `rootcertbundle`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). | 72| `path` | This should be the same directory like specified in Registry's `rootdirectory`. Read the [storage configuration documentation](https://docs.docker.com/registry/configuration/#storage). This path needs to be readable by the GitLab user, the web-server user and the Registry user. Read more in [#configure-storage-for-the-container-registry](#configure-storage-for-the-container-registry). | 73| `issuer` | This should be the same value as configured in Registry's `issuer`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). | 74 75A Registry init file is not shipped with GitLab if you install it from source. 76Hence, [restarting GitLab](../restart_gitlab.md#installations-from-source) does not restart the Registry should 77you modify its settings. Read the upstream documentation on how to achieve that. 78 79At the **absolute** minimum, make sure your [Registry configuration](https://docs.docker.com/registry/configuration/#auth) 80has `container_registry` as the service and `https://gitlab.example.com/jwt/auth` 81as the realm: 82 83```yaml 84auth: 85 token: 86 realm: https://gitlab.example.com/jwt/auth 87 service: container_registry 88 issuer: gitlab-issuer 89 rootcertbundle: /root/certs/certbundle 90``` 91 92WARNING: 93If `auth` is not set up, users can pull Docker images without authentication. 94 95## Container Registry domain configuration 96 97There are two ways you can configure the Registry's external domain. Either: 98 99- [Use the existing GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain). 100 The Registry listens on a port and reuses the TLS certificate from GitLab. 101- [Use a completely separate domain](#configure-container-registry-under-its-own-domain) with a new TLS certificate 102 for that domain. 103 104Because the Container Registry requires a TLS certificate, cost may be a factor. 105 106Take this into consideration before configuring the Container Registry 107for the first time. 108 109### Configure Container Registry under an existing GitLab domain 110 111If the Registry is configured to use the existing GitLab domain, you can 112expose the Registry on a port. This way you can reuse the existing GitLab TLS 113certificate. 114 115If the GitLab domain is `https://gitlab.example.com` and the port to the outside world is `5050`, here is what you need to set 116in `gitlab.rb` or `gitlab.yml` if you are using Omnibus GitLab or installed 117GitLab from source respectively. 118 119Ensure you choose a port different than the one that Registry listens to (`5000` by default), 120otherwise conflicts occur. 121 122NOTE: 123Host and container firewall rules must be configured to allow traffic in through the port listed 124under the `registry_external_url` line, rather than the port listed under 125`gitlab_rails['registry_port']` (default `5000`). 126 127**Omnibus GitLab installations** 128 1291. Your `/etc/gitlab/gitlab.rb` should contain the Registry URL as well as the 130 path to the existing TLS certificate and key used by GitLab: 131 132 ```ruby 133 registry_external_url 'https://gitlab.example.com:5050' 134 ``` 135 136 The `registry_external_url` is listening on HTTPS under the 137 existing GitLab URL, but on a different port. 138 139 If your TLS certificate is not in `/etc/gitlab/ssl/gitlab.example.com.crt` 140 and key not in `/etc/gitlab/ssl/gitlab.example.com.key` uncomment the lines 141 below: 142 143 ```ruby 144 registry_nginx['ssl_certificate'] = "/path/to/certificate.pem" 145 registry_nginx['ssl_certificate_key'] = "/path/to/certificate.key" 146 ``` 147 1481. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) 149 for the changes to take effect. 150 1511. Validate using: 152 153 ```shell 154 openssl s_client -showcerts -servername gitlab.example.com -connect gitlab.example.com:5050 > cacert.pem 155 ``` 156 157If your certificate provider provides the CA Bundle certificates, append them to the TLS certificate file. 158 159An administrator may want the container registry listening on an arbitrary port such as `5678`. 160However, the registry and application server are behind an AWS application load balancer that only 161listens on ports `80` and `443`. The admin may simply remove the port number for 162`registry_external_url`, so HTTP or HTTPS is assumed. Then, the rules apply that map the load 163balancer to the registry from ports `80` or `443` to the arbitrary port. This is important if users 164rely on the `docker login` example in the container registry. Here's an example: 165 166```ruby 167registry_external_url 'https://registry-gitlab.example.com' 168registry_nginx['redirect_http_to_https'] = true 169registry_nginx['listen_port'] = 5678 170``` 171 172**Installations from source** 173 1741. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and 175 configure it with the following settings: 176 177 ```yaml 178 registry: 179 enabled: true 180 host: gitlab.example.com 181 port: 5050 182 ``` 183 1841. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 1851. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). 186 187Users should now be able to sign in to the Container Registry with their GitLab 188credentials using: 189 190```shell 191docker login gitlab.example.com:5050 192``` 193 194### Configure Container Registry under its own domain 195 196When the Registry is configured to use its own domain, you need a TLS 197certificate for that specific domain (for example, `registry.example.com`). You might need 198a wildcard certificate if hosted under a subdomain of your existing GitLab 199domain, for example, `registry.gitlab.example.com`. 200 201As well as manually generated SSL certificates (explained here), certificates automatically 202generated by Let's Encrypt are also [supported in Omnibus installs](https://docs.gitlab.com/omnibus/settings/ssl.html#host-services). 203 204Let's assume that you want the container Registry to be accessible at 205`https://registry.gitlab.example.com`. 206 207**Omnibus GitLab installations** 208 2091. Place your TLS certificate and key in 210 `/etc/gitlab/ssl/registry.gitlab.example.com.crt` and 211 `/etc/gitlab/ssl/registry.gitlab.example.com.key` and make sure they have 212 correct permissions: 213 214 ```shell 215 chmod 600 /etc/gitlab/ssl/registry.gitlab.example.com.* 216 ``` 217 2181. After the TLS certificate is in place, edit `/etc/gitlab/gitlab.rb` with: 219 220 ```ruby 221 registry_external_url 'https://registry.gitlab.example.com' 222 ``` 223 224 The `registry_external_url` is listening on HTTPS. 225 2261. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 227 228If you have a [wildcard certificate](https://en.wikipedia.org/wiki/Wildcard_certificate), you must specify the path to the 229certificate in addition to the URL, in this case `/etc/gitlab/gitlab.rb` 230looks like: 231 232```ruby 233registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/certificate.pem" 234registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/certificate.key" 235``` 236 237**Installations from source** 238 2391. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and 240 configure it with the following settings: 241 242 ```yaml 243 registry: 244 enabled: true 245 host: registry.gitlab.example.com 246 ``` 247 2481. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 2491. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). 250 251Users should now be able to sign in to the Container Registry using their GitLab 252credentials: 253 254```shell 255docker login registry.gitlab.example.com 256``` 257 258## Disable Container Registry site-wide 259 260When you disable the Registry by following these steps, you do not 261remove any existing Docker images. This is handled by the 262Registry application itself. 263 264**Omnibus GitLab** 265 2661. Open `/etc/gitlab/gitlab.rb` and set `registry['enable']` to `false`: 267 268 ```ruby 269 registry['enable'] = false 270 ``` 271 2721. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 273 274**Installations from source** 275 2761. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and 277 set `enabled` to `false`: 278 279 ```yaml 280 registry: 281 enabled: false 282 ``` 283 2841. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 285 286## Disable Container Registry for new projects site-wide 287 288If the Container Registry is enabled, then it should be available on all new 289projects. To disable this function and let the owners of a project to enable 290the Container Registry by themselves, follow the steps below. 291 292**Omnibus GitLab installations** 293 2941. Edit `/etc/gitlab/gitlab.rb` and add the following line: 295 296 ```ruby 297 gitlab_rails['gitlab_default_projects_features_container_registry'] = false 298 ``` 299 3001. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 301 302**Installations from source** 303 3041. Open `/home/git/gitlab/config/gitlab.yml`, find the `default_projects_features` 305 entry and configure it so that `container_registry` is set to `false`: 306 307 ```yaml 308 ## Default project features settings 309 default_projects_features: 310 issues: true 311 merge_requests: true 312 wiki: true 313 snippets: false 314 builds: true 315 container_registry: false 316 ``` 317 3181. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 319 320## Configure storage for the Container Registry 321 322NOTE: 323For storage backends that support it, you can use object versioning to preserve, retrieve, and 324restore the non-current versions of every object stored in your buckets. However, this may result in 325higher storage usage and costs. Due to how the registry operates, image uploads are first stored in 326a temporary path and then transferred to a final location. For object storage backends, including S3 327and GCS, this transfer is achieved with a copy followed by a delete. With object versioning enabled, 328these deleted temporary upload artifacts are kept as non-current versions, therefore increasing the 329storage bucket size. To ensure that non-current versions are deleted after a given amount of time, 330you should configure an object lifecycle policy with your storage provider. 331 332You can configure the Container Registry to use various storage backends by 333configuring a storage driver. By default the GitLab Container Registry 334is configured to use the [file system driver](#use-file-system) 335configuration. 336 337The different supported drivers are: 338 339| Driver | Description | 340|--------------|--------------------------------------| 341| `filesystem` | Uses a path on the local file system | 342| `azure` | Microsoft Azure Blob Storage | 343| `gcs` | Google Cloud Storage | 344| `s3` | Amazon Simple Storage Service. Be sure to configure your storage bucket with the correct [S3 Permission Scopes](https://docs.docker.com/registry/storage-drivers/s3/#s3-permission-scopes). | 345| `swift` | OpenStack Swift Object Storage | 346| `oss` | Aliyun OSS | 347 348Although most S3 compatible services (like [MinIO](https://min.io/)) should work with the Container Registry, we only guarantee support for AWS S3. Because we cannot assert the correctness of third-party S3 implementations, we can debug issues, but we cannot patch the registry unless an issue is reproducible against an AWS S3 bucket. 349 350Read more about the individual driver's configuration options in the 351[Docker Registry docs](https://docs.docker.com/registry/configuration/#storage). 352 353### Use file system 354 355If you want to store your images on the file system, you can change the storage 356path for the Container Registry, follow the steps below. 357 358This path is accessible to: 359 360- The user running the Container Registry daemon. 361- The user running GitLab. 362 363All GitLab, Registry, and web server users must 364have access to this directory. 365 366**Omnibus GitLab installations** 367 368The default location where images are stored in Omnibus, is 369`/var/opt/gitlab/gitlab-rails/shared/registry`. To change it: 370 3711. Edit `/etc/gitlab/gitlab.rb`: 372 373 ```ruby 374 gitlab_rails['registry_path'] = "/path/to/registry/storage" 375 ``` 376 3771. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 378 379**Installations from source** 380 381The default location where images are stored in source installations, is 382`/home/git/gitlab/shared/registry`. To change it: 383 3841. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and 385 change the `path` setting: 386 387 ```yaml 388 registry: 389 path: shared/registry 390 ``` 391 3921. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 393 394### Use object storage 395 396If you want to store your images on object storage, you can change the storage 397driver for the Container Registry. 398 399[Read more about using object storage with GitLab](../object_storage.md). 400 401WARNING: 402GitLab does not back up Docker images that are not stored on the 403file system. Enable backups with your object storage provider if 404desired. 405 406**Omnibus GitLab installations** 407 408To configure the `s3` storage driver in Omnibus: 409 4101. Edit `/etc/gitlab/gitlab.rb`: 411 412 ```ruby 413 registry['storage'] = { 414 's3' => { 415 'accesskey' => 's3-access-key', 416 'secretkey' => 's3-secret-key-for-access-key', 417 'bucket' => 'your-s3-bucket', 418 'region' => 'your-s3-region', 419 'regionendpoint' => 'your-s3-regionendpoint' 420 } 421 } 422 ``` 423 424 To avoid using static credentials, use an 425 [IAM role](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) 426 and omit `accesskey` and `secretkey`. Make sure that your IAM profile follows 427 [the permissions documented by Docker](https://docs.docker.com/registry/storage-drivers/s3/#s3-permission-scopes). 428 429 ```ruby 430 registry['storage'] = { 431 's3' => { 432 'bucket' => 'your-s3-bucket', 433 'region' => 'your-s3-region' 434 } 435 } 436 ``` 437 438 If using with an [AWS S3 VPC endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html), 439 then set `regionendpoint` to your VPC endpoint address and set `pathstyle` to false: 440 441 ```ruby 442 registry['storage'] = { 443 's3' => { 444 'accesskey' => 's3-access-key', 445 'secretkey' => 's3-secret-key-for-access-key', 446 'bucket' => 'your-s3-bucket', 447 'region' => 'your-s3-region', 448 'regionendpoint' => 'your-s3-vpc-endpoint', 449 'pathstyle' => false 450 } 451 } 452 ``` 453 454 - `regionendpoint` is only required when configuring an S3 compatible service such as MinIO, or 455 when using an AWS S3 VPC Endpoint. 456 - `your-s3-bucket` should be the name of a bucket that exists, and can't include subdirectories. 457 - `pathstyle` should be set to true to use `host/bucket_name/object` style paths instead of 458 `bucket_name.host/object`. [Set to false for AWS S3](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/). 459 4601. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 461 462**Installations from source** 463 464Configuring the storage driver is done in the registry configuration YML file created 465when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). 466 467`s3` storage driver example: 468 469```yaml 470storage: 471 s3: 472 accesskey: 's3-access-key' # Not needed if IAM role used 473 secretkey: 's3-secret-key-for-access-key' # Not needed if IAM role used 474 bucket: 'your-s3-bucket' 475 region: 'your-s3-region' 476 regionendpoint: 'your-s3-regionendpoint' 477 cache: 478 blobdescriptor: inmemory 479 delete: 480 enabled: true 481``` 482 483`your-s3-bucket` should be the name of a bucket that exists, and can't include subdirectories. 484 485#### Migrate to object storage without downtime 486 487WARNING: 488Using [AWS DataSync](https://aws.amazon.com/datasync/) 489to copy the registry data to or between S3 buckets creates invalid metadata objects in the bucket. 490For additional details, see [Tags with an empty name](#tags-with-an-empty-name). 491To move data to and between S3 buckets, the AWS CLI `sync` operation is recommended. 492 493To migrate storage without stopping the Container Registry, set the Container Registry 494to read-only mode. On large instances, this may require the Container Registry 495to be in read-only mode for a while. During this time, 496you can pull from the Container Registry, but you cannot push. 497 4981. Optional: To reduce the amount of data to be migrated, run the [garbage collection tool without downtime](#performing-garbage-collection-without-downtime). 4991. This example uses the `aws` CLI. If you haven't configured the 500 CLI before, you have to configure your credentials by running `sudo aws configure`. 501 Because a non-administrator user likely can't access the Container Registry folder, 502 ensure you use `sudo`. To check your credential configuration, run 503 [`ls`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/ls.html) to list 504 all buckets. 505 506 ```shell 507 sudo aws --endpoint-url https://your-object-storage-backend.com s3 ls 508 ``` 509 510 If you are using AWS as your back end, you do not need the [`--endpoint-url`](https://docs.aws.amazon.com/cli/latest/reference/#options). 5111. Copy initial data to your S3 bucket, for example with the `aws` CLI 512 [`cp`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html) 513 or [`sync`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html) 514 command. Make sure to keep the `docker` folder as the top-level folder inside the bucket. 515 516 ```shell 517 sudo aws --endpoint-url https://your-object-storage-backend.com s3 sync registry s3://mybucket 518 ``` 519 520 NOTE: 521 If you have a lot of data, you may be able to improve performance by 522 [running parallel sync operations](https://aws.amazon.com/premiumsupport/knowledge-center/s3-improve-transfer-sync-command/). 523 5241. To perform the final data sync, 525 [put the Container Registry in `read-only` mode](#performing-garbage-collection-without-downtime) and 526 [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 5271. Sync any changes since the initial data load to your S3 bucket and delete files that exist in the destination bucket but not in the source: 528 529 ```shell 530 sudo aws --endpoint-url https://your-object-storage-backend.com s3 sync registry s3://mybucket --delete --dryrun 531 ``` 532 533 After verifying the command performs as expected, remove the 534 [`--dryrun`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) 535 flag and run the command. 536 537 WARNING: 538 The [`--delete`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) 539 flag deletes files that exist in the destination but not in the source. 540 If you swap the source and destination, all data in the Registry is deleted. 541 5421. Verify all Container Registry files have been uploaded to object storage 543 by looking at the file count returned by these two commands: 544 545 ```shell 546 sudo find registry -type f | wc -l 547 ``` 548 549 ```shell 550 sudo aws --endpoint-url https://your-object-storage-backend.com s3 ls s3://mybucket --recursive | wc -l 551 ``` 552 553 The output of these commands should match, except for the content in the 554 `_uploads` directories and sub-directories. 5551. Configure your registry to [use the S3 bucket for storage](#use-object-storage). 5561. For the changes to take effect, set the Registry back to `read-write` mode and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 557 558### Disable redirect for storage driver 559 560By default, users accessing a registry configured with a remote backend are redirected to the default backend for the storage driver. For example, registries can be configured using the `s3` storage driver, which redirects requests to a remote S3 bucket to alleviate load on the GitLab server. 561 562However, this behavior is undesirable for registries used by internal hosts that usually can't access public servers. To disable redirects and [proxy download](../object_storage.md#proxy-download), set the `disable` flag to true as follows. This makes all traffic always go through the Registry service. This results in improved security (less surface attack as the storage backend is not publicly accessible), but worse performance (all traffic is redirected via the service). 563 564**Omnibus GitLab installations** 565 5661. Edit `/etc/gitlab/gitlab.rb`: 567 568 ```ruby 569 registry['storage'] = { 570 's3' => { 571 'accesskey' => 's3-access-key', 572 'secretkey' => 's3-secret-key-for-access-key', 573 'bucket' => 'your-s3-bucket', 574 'region' => 'your-s3-region', 575 'regionendpoint' => 'your-s3-regionendpoint' 576 }, 577 'redirect' => { 578 'disable' => true 579 } 580 } 581 ``` 582 5831. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 584 585**Installations from source** 586 5871. Add the `redirect` flag to your registry configuration YML file: 588 589 ```yaml 590 storage: 591 s3: 592 accesskey: 'AKIAKIAKI' 593 secretkey: 'secret123' 594 bucket: 'gitlab-registry-bucket-AKIAKIAKI' 595 region: 'your-s3-region' 596 regionendpoint: 'your-s3-regionendpoint' 597 redirect: 598 disable: true 599 cache: 600 blobdescriptor: inmemory 601 delete: 602 enabled: true 603 ``` 604 6051. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 606 607#### Encrypted S3 buckets 608 609You can use server-side encryption with AWS KMS for S3 buckets that have 610[SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). 611Customer master keys (CMKs) and SSE-C encryption aren't supported since this requires sending the 612encryption keys in every request. 613 614For SSE-S3, you must enable the `encrypt` option in the registry settings. How you do this depends 615on how you installed GitLab. Follow the instructions here that match your installation method. 616 617For Omnibus GitLab installations: 618 6191. Edit `/etc/gitlab/gitlab.rb`: 620 621 ```ruby 622 registry['storage'] = { 623 's3' => { 624 'accesskey' => 's3-access-key', 625 'secretkey' => 's3-secret-key-for-access-key', 626 'bucket' => 'your-s3-bucket', 627 'region' => 'your-s3-region', 628 'regionendpoint' => 'your-s3-regionendpoint', 629 'encrypt' => true 630 } 631 } 632 ``` 633 6341. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) 635 for the changes to take effect. 636 637For installations from source: 638 6391. Edit your registry configuration YML file: 640 641 ```yaml 642 storage: 643 s3: 644 accesskey: 'AKIAKIAKI' 645 secretkey: 'secret123' 646 bucket: 'gitlab-registry-bucket-AKIAKIAKI' 647 region: 'your-s3-region' 648 regionendpoint: 'your-s3-regionendpoint' 649 encrypt: true 650 ``` 651 6521. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) 653 for the changes to take effect. 654 655### Storage limitations 656 657Currently, there is no storage limitation, which means a user can upload an 658infinite amount of Docker images with arbitrary sizes. This setting should be 659configurable in future releases. 660 661## Change the registry's internal port 662 663The Registry server listens on localhost at port `5000` by default, 664which is the address for which the Registry server should accept connections. 665In the examples below we set the Registry's port to `5001`. 666 667**Omnibus GitLab** 668 6691. Open `/etc/gitlab/gitlab.rb` and set `registry['registry_http_addr']`: 670 671 ```ruby 672 registry['registry_http_addr'] = "localhost:5001" 673 ``` 674 6751. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 676 677**Installations from source** 678 6791. Open the configuration file of your Registry server and edit the 680 [`http:addr`](https://docs.docker.com/registry/configuration/#http) value: 681 682 ```yaml 683 http: 684 addr: localhost:5001 685 ``` 686 6871. Save the file and restart the Registry server. 688 689## Disable Container Registry per project 690 691If Registry is enabled in your GitLab instance, but you don't need it for your 692project, you can [disable it from your project's settings](../../user/project/settings/index.md#sharing-and-permissions). 693 694## Use an external container registry with GitLab as an auth endpoint 695 696If you use an external container registry, some features associated with the 697container registry may be unavailable or have [inherent risks](../../user/packages/container_registry/index.md#use-with-external-container-registries). 698 699For the integration to work, the external registry must be configured to 700use a JSON Web Token to authenticate with GitLab. The 701[external registry's runtime configuration](https://docs.docker.com/registry/configuration/#token) 702**must** have the following entries: 703 704```yaml 705auth: 706 token: 707 realm: https://gitlab.example.com/jwt/auth 708 service: container_registry 709 issuer: gitlab-issuer 710 rootcertbundle: /root/certs/certbundle 711``` 712 713Without these entries, the registry logins cannot authenticate with GitLab. 714GitLab also remains unaware of 715[nested image names](../../user/packages/container_registry/#image-naming-convention) 716under the project hierarchy, like 717`registry.example.com/group/project/image-name:tag` or 718`registry.example.com/group/project/my/image-name:tag`, and only recognizes 719`registry.example.com/group/project:tag`. 720 721**Omnibus GitLab** 722 723You can use GitLab as an auth endpoint with an external container registry. 724 7251. Open `/etc/gitlab/gitlab.rb` and set necessary configurations: 726 727 ```ruby 728 gitlab_rails['registry_enabled'] = true 729 gitlab_rails['registry_api_url'] = "https://<external_registry_host>:5000" 730 gitlab_rails['registry_issuer'] = "gitlab-issuer" 731 ``` 732 733 - `gitlab_rails['registry_enabled'] = true` is needed to enable GitLab 734 Container Registry features and authentication endpoint. The GitLab bundled 735 Container Registry service does not start, even with this enabled. 736 - `gitlab_rails['registry_api_url'] = "http://<external_registry_host>:5000"` 737 must be changed to match the host where Registry is installed. 738 It must also specify `https` if the external registry is 739 configured to use TLS. Read more on the 740 [Docker registry documentation](https://docs.docker.com/registry/deploying/). 741 7421. A certificate-key pair is required for GitLab and the external container 743 registry to communicate securely. You need to create a certificate-key 744 pair, configuring the external container registry with the public 745 certificate (`rootcertbundle`) and configuring GitLab with the private key. 746 To do that, add the following to `/etc/gitlab/gitlab.rb`: 747 748 ```ruby 749 # registry['internal_key'] should contain the contents of the custom key 750 # file. Line breaks in the key file should be marked using `\n` character 751 # Example: 752 registry['internal_key'] = "---BEGIN RSA PRIVATE KEY---\nMIIEpQIBAA\n" 753 754 # Optionally define a custom file for Omnibus GitLab to write the contents 755 # of registry['internal_key'] to. 756 gitlab_rails['registry_key_path'] = "/custom/path/to/registry-key.key" 757 ``` 758 759 Each time reconfigure is executed, the file specified at `registry_key_path` 760 gets populated with the content specified by `internal_key`. If 761 no file is specified, Omnibus GitLab defaults it to 762 `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and populates 763 it. 764 7651. To change the container registry URL displayed in the GitLab Container 766 Registry pages, set the following configurations: 767 768 ```ruby 769 gitlab_rails['registry_host'] = "registry.gitlab.example.com" 770 gitlab_rails['registry_port'] = "5005" 771 ``` 772 7731. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) 774 for the changes to take effect. 775 776**Installations from source** 777 7781. Open `/home/git/gitlab/config/gitlab.yml`, and edit the configuration settings under `registry`: 779 780 ```yaml 781 ## Container Registry 782 783 registry: 784 enabled: true 785 host: "registry.gitlab.example.com" 786 port: "5005" 787 api_url: "https://<external_registry_host>:5000" 788 path: /var/lib/registry 789 key: /path/to/keyfile 790 issuer: gitlab-issuer 791 ``` 792 793 [Read more](#enable-the-container-registry) about what these parameters mean. 794 7951. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 796 797## Configure Container Registry notifications 798 799You can configure the Container Registry to send webhook notifications in 800response to events happening within the registry. 801 802Read more about the Container Registry notifications configuration options in the 803[Docker Registry notifications documentation](https://docs.docker.com/registry/notifications/). 804 805You can configure multiple endpoints for the Container Registry. 806 807**Omnibus GitLab installations** 808 809To configure a notification endpoint in Omnibus: 810 8111. Edit `/etc/gitlab/gitlab.rb`: 812 813 ```ruby 814 registry['notifications'] = [ 815 { 816 'name' => 'test_endpoint', 817 'url' => 'https://gitlab.example.com/notify', 818 'timeout' => '500ms', 819 'threshold' => 5, 820 'backoff' => '1s', 821 'headers' => { 822 "Authorization" => ["AUTHORIZATION_EXAMPLE_TOKEN"] 823 } 824 } 825 ] 826 ``` 827 8281. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 829 830**Installations from source** 831 832Configuring the notification endpoint is done in your registry configuration YML file created 833when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). 834 835Example: 836 837```yaml 838notifications: 839 endpoints: 840 - name: alistener 841 disabled: false 842 url: https://my.listener.com/event 843 headers: <http.Header> 844 timeout: 500 845 threshold: 5 846 backoff: 1000 847``` 848 849## Run the Cleanup policy now 850 851WARNING: 852If you're using a distributed architecture and Sidekiq is running on a different node, the cleanup 853policies don't work. To fix this, you must configure the `gitlab.rb` file on the Sidekiq nodes to 854point to the correct registry URL and copy the `registry.key` file to each Sidekiq node. For more 855information, see the [Sidekiq configuration](../sidekiq.md) 856page. 857 858To reduce the amount of [Container Registry disk space used by a given project](../troubleshooting/gitlab_rails_cheat_sheet.md#registry-disk-space-usage-by-project), 859administrators can clean up image tags 860and [run garbage collection](#container-registry-garbage-collection). 861 862To remove image tags by running the cleanup policy, run the following commands in the 863[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md): 864 865```ruby 866# Numeric ID of the project whose container registry should be cleaned up 867P = <project_id> 868 869# Numeric ID of a user with Developer, Maintainer, or Owner role for the project 870U = <user_id> 871 872# Get required details / objects 873user = User.find_by_id(U) 874project = Project.find_by_id(P) 875policy = ContainerExpirationPolicy.find_by(project_id: P) 876 877# Loop through each container repository 878project.container_repositories.find_each do |repo| 879 puts repo.attributes 880 881 # Start the tag cleanup 882 puts Projects::ContainerRepository::CleanupTagsService.new(repo, user, policy.attributes.except("created_at", "updated_at")).execute() 883end 884``` 885 886You can also [run cleanup on a schedule](../../user/packages/container_registry/index.md#cleanup-policy). 887 888## Container Registry garbage collection 889 890Container Registry can use considerable amounts of disk space. To clear up 891some unused layers, the registry includes a garbage collect command. 892 893GitLab offers a set of APIs to manipulate the Container Registry and aid the process 894of removing unused tags. Currently, this is exposed using the API, but in the future, 895these controls should migrate to the GitLab interface. 896 897Users who have the [Maintainer role](../../user/permissions.md) for the project can 898[delete Container Registry tags in bulk](../../api/container_registry.md#delete-registry-repository-tags-in-bulk) 899periodically based on their own criteria, however, this alone does not recycle data, 900it only unlinks tags from manifests and image blobs. To recycle the Container 901Registry data in the whole GitLab instance, you can use the built-in command 902provided by `gitlab-ctl`. 903 904Prerequisites: 905 906- You must have installed GitLab by using an Omnibus package or the 907 [cloud native chart](https://docs.gitlab.com/charts/charts/registry/#garbage-collection). 908- You must set the Registry to [read-only mode](#performing-garbage-collection-without-downtime). 909 Running garbage collection causes downtime for the Container Registry. When you run this command 910 on an instance in an environment where another instances is still writing to the Registry storage, 911 referenced manifests are removed. 912 913### Understanding the content-addressable layers 914 915Consider the following example, where you first build the image: 916 917```shell 918# This builds a image with content of sha256:111111 919docker build -t my.registry.com/my.group/my.project:latest . 920docker push my.registry.com/my.group/my.project:latest 921``` 922 923Now, you do overwrite `:latest` with a new version: 924 925```shell 926# This builds a image with content of sha256:222222 927docker build -t my.registry.com/my.group/my.project:latest . 928docker push my.registry.com/my.group/my.project:latest 929``` 930 931Now, the `:latest` tag points to manifest of `sha256:222222`. However, due to 932the architecture of registry, this data is still accessible when pulling the 933image `my.registry.com/my.group/my.project@sha256:111111`, even though it is 934no longer directly accessible via the `:latest` tag. 935 936### Recycling unused tags 937 938Before you run the built-in command, note the following: 939 940- The built-in command stops the registry before it starts the garbage collection. 941- The garbage collect command takes some time to complete, depending on the 942 amount of data that exists. 943- If you changed the location of registry configuration file, you must 944 specify its path. 945- After the garbage collection is done, the registry should start automatically. 946 947If you did not change the default location of the configuration file, run: 948 949```shell 950sudo gitlab-ctl registry-garbage-collect 951``` 952 953This command takes some time to complete, depending on the amount of 954layers you have stored. 955 956If you changed the location of the Container Registry `config.yml`: 957 958```shell 959sudo gitlab-ctl registry-garbage-collect /path/to/config.yml 960``` 961 962You may also [remove all untagged manifests and unreferenced layers](#removing-untagged-manifests-and-unreferenced-layers), 963although this is a way more destructive operation, and you should first 964understand the implications. 965 966### Removing untagged manifests and unreferenced layers 967 968> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3097) in Omnibus GitLab 11.10. 969 970WARNING: 971This is a destructive operation. 972 973The GitLab Container Registry follows the same default workflow as Docker Distribution: 974retain untagged manifests and all layers, even ones that are not referenced directly. All content 975can be accessed by using context addressable identifiers. 976 977However, in most workflows, you don't care about untagged manifests and old layers if they are not directly 978referenced by a tagged manifest. The `registry-garbage-collect` command supports the 979`-m` switch to allow you to remove all unreferenced manifests and layers that are 980not directly accessible via `tag`: 981 982```shell 983sudo gitlab-ctl registry-garbage-collect -m 984``` 985 986Since this is a way more destructive operation, this behavior is disabled by default. 987You are likely expecting this way of operation, but before doing that, ensure 988that you have backed up all registry data. 989 990When the command is used without the `-m` flag, the Container Registry only removes layers that are not referenced by any manifest, tagged or not. 991 992### Performing garbage collection without downtime 993 994You can perform garbage collection without stopping the Container Registry by putting 995it in read-only mode and by not using the built-in command. On large instances 996this could require Container Registry to be in read-only mode for a while. 997During this time, 998you are able to pull from the Container Registry, but you are not able to 999push. 1000 1001By default, the [registry storage path](#configure-storage-for-the-container-registry) 1002is `/var/opt/gitlab/gitlab-rails/shared/registry`. 1003 1004To enable the read-only mode: 1005 10061. In `/etc/gitlab/gitlab.rb`, specify the read-only mode: 1007 1008 ```ruby 1009 registry['storage'] = { 1010 'filesystem' => { 1011 'rootdirectory' => "<your_registry_storage_path>" 1012 }, 1013 'maintenance' => { 1014 'readonly' => { 1015 'enabled' => true 1016 } 1017 } 1018 } 1019 ``` 1020 10211. Save and reconfigure GitLab: 1022 1023 ```shell 1024 sudo gitlab-ctl reconfigure 1025 ``` 1026 1027 This command sets the Container Registry into the read-only mode. 1028 10291. Next, trigger one of the garbage collect commands: 1030 1031 ```shell 1032 # Recycling unused tags 1033 sudo /opt/gitlab/embedded/bin/registry garbage-collect /var/opt/gitlab/registry/config.yml 1034 1035 # Removing unused layers not referenced by manifests 1036 sudo /opt/gitlab/embedded/bin/registry garbage-collect -m /var/opt/gitlab/registry/config.yml 1037 ``` 1038 1039 This command starts the garbage collection, which might take some time to complete. 1040 10411. Once done, in `/etc/gitlab/gitlab.rb` change it back to read-write mode: 1042 1043 ```ruby 1044 registry['storage'] = { 1045 'filesystem' => { 1046 'rootdirectory' => "<your_registry_storage_path>" 1047 }, 1048 'maintenance' => { 1049 'readonly' => { 1050 'enabled' => false 1051 } 1052 } 1053 } 1054 ``` 1055 10561. Save and reconfigure GitLab: 1057 1058 ```shell 1059 sudo gitlab-ctl reconfigure 1060 ``` 1061 1062### Running the garbage collection on schedule 1063 1064Ideally, you want to run the garbage collection of the registry regularly on a 1065weekly basis at a time when the registry is not being in-use. 1066The simplest way is to add a new crontab job that it runs periodically 1067once a week. 1068 1069Create a file under `/etc/cron.d/registry-garbage-collect`: 1070 1071```shell 1072SHELL=/bin/sh 1073PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin 1074 1075# Run every Sunday at 04:05am 10765 4 * * 0 root gitlab-ctl registry-garbage-collect 1077``` 1078 1079You may want to add the `-m` flag to [remove untagged manifests and unreferenced layers](#removing-untagged-manifests-and-unreferenced-layers). 1080 1081### Stop garbage collection 1082 1083If you anticipate stopping garbage collection, you should manually run garbage collection as 1084described in [Performing garbage collection without downtime](#performing-garbage-collection-without-downtime). 1085You can then stop garbage collection by pressing <kbd>Control</kbd>+<kbd>C</kbd>. 1086 1087Otherwise, interrupting `gitlab-ctl` could leave your registry service in a down state. In this 1088case, you must find the [garbage collection process](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-ctl-commands/registry_garbage_collect.rb#L26-35) 1089itself on the system so that the `gitlab-ctl` command can bring the registry service back up again. 1090 1091Also, there's no way to save progress or results during the mark phase of the process. Only once 1092blobs start being deleted is anything permanent done. 1093 1094## Configuring GitLab and Registry to run on separate nodes (Omnibus GitLab) 1095 1096By default, package assumes that both services are running on the same node. 1097In order to get GitLab and Registry to run on a separate nodes, separate configuration 1098is necessary for Registry and GitLab. 1099 1100### Configuring Registry 1101 1102Below you can find configuration options you should set in `/etc/gitlab/gitlab.rb`, 1103for Registry to run separately from GitLab: 1104 1105- `registry['registry_http_addr']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L50). Needs to be reachable by web server (or LB). 1106- `registry['token_realm']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L53). Specifies the endpoint to use to perform authentication, usually the GitLab URL. 1107 This endpoint needs to be reachable by user. 1108- `registry['http_secret']`, [random string](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L32). A random piece of data used to sign state that may be stored with the client to protect against tampering. 1109- `registry['internal_key']`, default [automatically generated](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb#L113-119). Contents of the key that GitLab uses to sign the tokens. They key gets created on the Registry server, but it is not used there. 1110- `gitlab_rails['registry_key_path']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb#L35). This is the path where `internal_key` contents are written to disk. 1111- `registry['internal_certificate']`, default [automatically generated](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/registry/recipes/enable.rb#L60-66). Contents of the certificate that GitLab uses to sign the tokens. 1112- `registry['rootcertbundle']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/registry/recipes/enable.rb#L60). Path to certificate. This is the path where `internal_certificate` 1113 contents are written to disk. 1114- `registry['health_storagedriver_enabled']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-7-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L88). Configure whether health checks on the configured storage driver are enabled. 1115- `gitlab_rails['registry_issuer']`, [default value](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/attributes/default.rb#L153). This setting needs to be set the same between Registry and GitLab. 1116 1117### Configuring GitLab 1118 1119Below you can find configuration options you should set in `/etc/gitlab/gitlab.rb`, 1120for GitLab to run separately from Registry: 1121 1122- `gitlab_rails['registry_enabled']`, must be set to `true`. This setting 1123 signals to GitLab that it should allow Registry API requests. 1124- `gitlab_rails['registry_api_url']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L52). This is the Registry URL used internally that users do not need to interact with, `registry['registry_http_addr']` with scheme. 1125- `gitlab_rails['registry_host']`, eg. `registry.gitlab.example`. Registry endpoint without the scheme, the address that gets shown to the end user. 1126- `gitlab_rails['registry_port']`. Registry endpoint port, visible to the end user. 1127- `gitlab_rails['registry_issuer']` must match the issuer in the Registry configuration. 1128- `gitlab_rails['registry_key_path']`, path to the key that matches the certificate on the 1129 Registry side. 1130- `gitlab_rails['internal_key']`, contents of the key that GitLab uses to sign the tokens. 1131 1132## Architecture of GitLab Container Registry 1133 1134The GitLab registry is what users use to store their own Docker images. 1135Because of that the Registry is client facing, meaning that we expose it directly 1136on the web server (or load balancers, LB for short). 1137 1138![GitLab Registry diagram](img/gitlab-registry-architecture.png) 1139 1140The flow described by the diagram above: 1141 11421. A user runs `docker login registry.gitlab.example` on their client. This reaches the web server (or LB) on port 443. 11431. Web server connects to the Registry backend pool (by default, using port 5000). Since the user 1144 didn’t provide a valid token, the Registry returns a 401 HTTP code and the URL (`token_realm` from 1145 Registry configuration) where to get one. This points to the GitLab API. 11461. The Docker client then connects to the GitLab API and obtains a token. 11471. The API signs the token with the registry key and hands it to the Docker client 11481. The Docker client now logs in again with the token received from the API. It can now push and pull Docker images. 1149 1150Reference: <https://docs.docker.com/registry/spec/auth/token/> 1151 1152### Communication between GitLab and Registry 1153 1154Registry doesn’t have a way to authenticate users internally so it relies on 1155GitLab to validate credentials. The connection between Registry and GitLab is 1156TLS encrypted. The key is used by GitLab to sign the tokens while the certificate 1157is used by Registry to validate the signature. By default, a self-signed certificate key pair is generated 1158for all installations. This can be overridden as needed. 1159 1160GitLab interacts with the Registry using the Registry private key. When a Registry 1161request goes out, a new short-living (10 minutes) namespace limited token is generated 1162and signed with the private key. 1163The Registry then verifies that the signature matches the registry certificate 1164specified in its configuration and allows the operation. 1165GitLab background jobs processing (through Sidekiq) also interacts with Registry. 1166These jobs talk directly to Registry in order to handle image deletion. 1167 1168## Troubleshooting 1169 1170Before diving in to the following sections, here's some basic troubleshooting: 1171 11721. Check to make sure that the system clock on your Docker client and GitLab server have 1173 been synchronized (for example, via NTP). 1174 11751. If you are using an S3-backed Registry, double check that the IAM 1176 permissions and the S3 credentials (including region) are correct. See [the 1177 sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/) 1178 for more details. 1179 11801. Check the Registry logs (for example `/var/log/gitlab/registry/current`) and the GitLab production logs 1181 for errors (for example `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues 1182 there. 1183 1184### Using self-signed certificates with Container Registry 1185 1186If you're using a self-signed certificate with your Container Registry, you 1187might encounter issues during the CI jobs like the following: 1188 1189```plaintext 1190Error response from daemon: Get registry.example.com/v1/users/: x509: certificate signed by unknown authority 1191``` 1192 1193The Docker daemon running the command expects a cert signed by a recognized CA, 1194thus the error above. 1195 1196While GitLab doesn't support using self-signed certificates with Container 1197Registry out of the box, it is possible to make it work by 1198[instructing the Docker daemon to trust the self-signed certificates](https://docs.docker.com/registry/insecure/#use-self-signed-certificates), 1199mounting the Docker daemon and setting `privileged = false` in the GitLab Runner 1200`config.toml` file. Setting `privileged = true` takes precedence over the Docker daemon: 1201 1202```toml 1203 [runners.docker] 1204 image = "ruby:2.6" 1205 privileged = false 1206 volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] 1207``` 1208 1209Additional information about this: [issue 18239](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/18239). 1210 1211### `unauthorized: authentication required` when pushing large images 1212 1213Example error: 1214 1215```shell 1216docker push gitlab.example.com/myproject/docs:latest 1217The push refers to a repository [gitlab.example.com/myproject/docs] 1218630816f32edb: Preparing 1219530d5553aec8: Preparing 1220... 12214b0bab9ff599: Waiting 1222d1c800db26c7: Waiting 122342755cf4ee95: Waiting 1224unauthorized: authentication required 1225``` 1226 1227GitLab has a default token expiration of 5 minutes for the registry. When pushing 1228larger images, or images that take longer than 5 minutes to push, users may 1229encounter this error. 1230 1231Administrators can increase the token duration in **Admin area > Settings > 1232CI/CD > Container Registry > Authorization token duration (minutes)**. 1233 1234### Docker login attempt fails with: 'token signed by untrusted key' 1235 1236[Registry relies on GitLab to validate credentials](#architecture-of-gitlab-container-registry) 1237If the registry fails to authenticate valid login attempts, you get the following error message: 1238 1239```shell 1240# docker login gitlab.company.com:4567 1241Username: user 1242Password: 1243Error response from daemon: login attempt to https://gitlab.company.com:4567/v2/ failed with status: 401 Unauthorized 1244``` 1245 1246And more specifically, this appears in the `/var/log/gitlab/registry/current` log file: 1247 1248```plaintext 1249level=info msg="token signed by untrusted key with ID: "TOKE:NL6Q:7PW6:EXAM:PLET:OKEN:BG27:RCIB:D2S3:EXAM:PLET:OKEN"" 1250level=warning msg="error authorizing context: invalid token" go.version=go1.12.7 http.request.host="gitlab.company.com:4567" http.request.id=74613829-2655-4f96-8991-1c9fe33869b8 http.request.method=GET http.request.remoteaddr=10.72.11.20 http.request.uri="/v2/" http.request.useragent="docker/19.03.2 go/go1.12.8 git-commit/6a30dfc kernel/3.10.0-693.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.2 \(linux\))" 1251``` 1252 1253GitLab uses the contents of the certificate key pair's two sides to encrypt the authentication token 1254for the Registry. This message means that those contents do not align. 1255 1256Check which files are in use: 1257 1258- `grep -A6 'auth:' /var/opt/gitlab/registry/config.yml` 1259 1260 ```yaml 1261 ## Container Registry Certificate 1262 auth: 1263 token: 1264 realm: https://gitlab.my.net/jwt/auth 1265 service: container_registry 1266 issuer: omnibus-gitlab-issuer 1267 --> rootcertbundle: /var/opt/gitlab/registry/gitlab-registry.crt 1268 autoredirect: false 1269 ``` 1270 1271- `grep -A9 'Container Registry' /var/opt/gitlab/gitlab-rails/etc/gitlab.yml` 1272 1273 ```yaml 1274 ## Container Registry Key 1275 registry: 1276 enabled: true 1277 host: gitlab.company.com 1278 port: 4567 1279 api_url: http://127.0.0.1:5000 # internal address to the registry, is used by GitLab to directly communicate with API 1280 path: /var/opt/gitlab/gitlab-rails/shared/registry 1281 --> key: /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key 1282 issuer: omnibus-gitlab-issuer 1283 notification_secret: 1284 ``` 1285 1286The output of these `openssl` commands should match, proving that the cert-key pair is a match: 1287 1288```shell 1289openssl x509 -noout -modulus -in /var/opt/gitlab/registry/gitlab-registry.crt | openssl sha256 1290openssl rsa -noout -modulus -in /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key | openssl sha256 1291``` 1292 1293If the two pieces of the certificate do not align, remove the files and run `gitlab-ctl reconfigure` 1294to regenerate the pair. If you have overridden the automatically generated self-signed pair with 1295your own certificates and have made sure that their contents align, you can delete the 'registry' 1296section in your `/etc/gitlab/gitlab-secrets.json` and run `gitlab-ctl reconfigure`. 1297 1298### AWS S3 with the GitLab registry error when pushing large images 1299 1300When using AWS S3 with the GitLab registry, an error may occur when pushing 1301large images. Look in the Registry log for the following error: 1302 1303```plaintext 1304level=error msg="response completed with error" err.code=unknown err.detail="unexpected EOF" err.message="unknown error" 1305``` 1306 1307To resolve the error specify a `chunksize` value in the Registry configuration. 1308Start with a value between `25000000` (25MB) and `50000000` (50MB). 1309 1310**For Omnibus installations** 1311 13121. Edit `/etc/gitlab/gitlab.rb`: 1313 1314 ```ruby 1315 registry['storage'] = { 1316 's3' => { 1317 'accesskey' => 'AKIAKIAKI', 1318 'secretkey' => 'secret123', 1319 'bucket' => 'gitlab-registry-bucket-AKIAKIAKI', 1320 'chunksize' => 25000000 1321 } 1322 } 1323 ``` 1324 13251. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1326 1327**For installations from source** 1328 13291. Edit `config/gitlab.yml`: 1330 1331 ```yaml 1332 storage: 1333 s3: 1334 accesskey: 'AKIAKIAKI' 1335 secretkey: 'secret123' 1336 bucket: 'gitlab-registry-bucket-AKIAKIAKI' 1337 chunksize: 25000000 1338 ``` 1339 13401. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 1341 1342### Supporting older Docker clients 1343 1344As of GitLab 11.9, we began shipping version 2.7.1 of the Docker container registry, which disables the schema1 manifest by default. If you are still using older Docker clients (1.9 or older), you may experience an error pushing images. See [omnibus-4145](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4145) for more details. 1345 1346You can add a configuration option for backwards compatibility. 1347 1348**For Omnibus installations** 1349 13501. Edit `/etc/gitlab/gitlab.rb`: 1351 1352 ```ruby 1353 registry['compatibility_schema1_enabled'] = true 1354 ``` 1355 13561. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1357 1358**For installations from source** 1359 13601. Edit the YML configuration file you created when you [deployed the registry](https://docs.docker.com/registry/deploying/). Add the following snippet: 1361 1362 ```yaml 1363 compatibility: 1364 schema1: 1365 enabled: true 1366 ``` 1367 13681. Restart the registry for the changes to take affect. 1369 1370### Docker connection error 1371 1372A Docker connection error can occur when there are special characters in either the group, 1373project or branch name. Special characters can include: 1374 1375- Leading underscore 1376- Trailing hyphen/dash 1377- Double hyphen/dash 1378 1379To get around this, you can [change the group path](../../user/group/index.md#change-a-groups-path), 1380[change the project path](../../user/project/settings/index.md#renaming-a-repository) or change the 1381branch name. Another option is to create a [push rule](../../push_rules/push_rules.md) to prevent 1382this at the instance level. 1383 1384### Image push errors 1385 1386When getting errors or "retrying" loops in an attempt to push an image but `docker login` works fine, 1387there is likely an issue with the headers forwarded to the registry by NGINX. The default recommended 1388NGINX configurations should handle this, but it might occur in custom setups where the SSL is 1389offloaded to a third party reverse proxy. 1390 1391This problem was discussed in a [Docker project issue](https://github.com/docker/distribution/issues/970) 1392and a simple solution would be to enable relative URLs in the Registry. 1393 1394**For Omnibus installations** 1395 13961. Edit `/etc/gitlab/gitlab.rb`: 1397 1398 ```ruby 1399 registry['env'] = { 1400 "REGISTRY_HTTP_RELATIVEURLS" => true 1401 } 1402 ``` 1403 14041. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1405 1406**For installations from source** 1407 14081. Edit the YML configuration file you created when you [deployed the registry](https://docs.docker.com/registry/deploying/). Add the following snippet: 1409 1410 ```yaml 1411 http: 1412 relativeurls: true 1413 ``` 1414 14151. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 1416 1417### Enable the Registry debug server 1418 1419You can use the Container Registry debug server to diagnose problems. The debug endpoint can monitor metrics and health, as well as do profiling. 1420 1421WARNING: 1422Sensitive information may be available from the debug endpoint. 1423Access to the debug endpoint must be locked down in a production environment. 1424 1425The optional debug server can be enabled by setting the registry debug address 1426in your `gitlab.rb` configuration. 1427 1428```ruby 1429registry['debug_addr'] = "localhost:5001" 1430``` 1431 1432After adding the setting, [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) to apply the change. 1433 1434Use curl to request debug output from the debug server: 1435 1436```shell 1437curl "localhost:5001/debug/health" 1438curl "localhost:5001/debug/vars" 1439``` 1440 1441### Access old schema v1 Docker images 1442 1443Support for the [Docker registry v1 API](https://www.docker.com/blog/registry-v1-api-deprecation/), 1444including [schema V1 image manifests](https://docs.docker.com/registry/spec/manifest-v2-1/), 1445was: 1446 1447- [Deprecated in GitLab 13.7](https://about.gitlab.com/releases/2020/12/22/gitlab-13-7-released/#deprecate-pulls-that-use-v1-of-the-docker-registry-api) 1448- [Removed in GitLab 13.9](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#deprecate-pulls-that-use-v1-of-the-docker-registry-api) 1449 1450It's no longer possible to push or pull v1 images from the GitLab Container Registry. 1451 1452If you had v1 images in the GitLab Container Registry, but you did not upgrade them (following the 1453[steps Docker recommends](https://docs.docker.com/registry/spec/deprecated-schema-v1/)) 1454ahead of the GitLab 13.9 upgrade, these images are no longer accessible. If you try to pull them, 1455this error appears: 1456 1457- `Error response from daemon: manifest invalid: Schema 1 manifest not supported` 1458 1459For Self-Managed GitLab instances, you can regain access to these images by temporarily downgrading 1460the GitLab Container Registry to a version lower than `v3.0.0-gitlab`. Follow these steps to regain 1461access to these images: 1462 14631. Downgrade the Container Registry to [`v2.13.1-gitlab`](https://gitlab.com/gitlab-org/container-registry/-/releases/v2.13.1-gitlab). 14641. Upgrade any v1 images. 14651. Revert the Container Registry downgrade. 1466 1467There's no need to put the registry in read-only mode during the image upgrade process. Ensure that 1468you are not relying on any new feature introduced since `v3.0.0-gitlab`. Such features are 1469unavailable during the upgrade process. See the [complete registry changelog](https://gitlab.com/gitlab-org/container-registry/-/blob/master/CHANGELOG.md) 1470for more information. 1471 1472The following sections provide additional details about each installation method. 1473 1474#### Helm chart installations 1475 1476For Helm chart installations: 1477 14781. Override the [`image.tag`](https://docs.gitlab.com/charts/charts/registry/#configuration) 1479 configuration parameter with `v2.13.1-gitlab`. 14801. Restart. 14811. Performing the [images upgrade](#images-upgrade)) steps. 14821. Revert the `image.tag` parameter to the previous value. 1483 1484No other registry configuration changes are required. 1485 1486#### Omnibus installations 1487 1488For Omnibus installations: 1489 14901. Temporarily replace the registry binary that ships with GitLab 13.9+ for one prior to 1491 `v3.0.0-gitlab`. To do so, pull a previous version of the Docker image for the GitLab Container 1492 Registry, such as `v2.13.1-gitlab`. You can then grab the `registry` binary from within this 1493 image, located at `/bin/registry`: 1494 1495 ```shell 1496 id=$(docker create registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v2.13.1-gitlab) 1497 docker cp $id:/bin/registry registry-2.13.1-gitlab 1498 docker rm $id 1499 ``` 1500 15011. Replace the binary embedded in the Omnibus install, located at 1502 `/opt/gitlab/embedded/bin/registry`, with `registry-2.13.1-gitlab`. Make sure to start by backing 1503 up the original binary embedded in Omnibus, and restore it after performing the 1504 [image upgrade](#images-upgrade)) steps. You should [stop](https://docs.gitlab.com/omnibus/maintenance/#starting-and-stopping) 1505 the registry service before replacing its binary and start it right after. No registry 1506 configuration changes are required. 1507 1508#### Source installations 1509 1510For source installations, locate your `registry` binary and temporarily replace it with the one 1511obtained from `v3.0.0-gitlab`, as explained for [Omnibus installations](#omnibus-installations). 1512Make sure to start by backing up the original registry binary, and restore it after performing the 1513[images upgrade](#images-upgrade)) 1514steps. 1515 1516#### Images upgrade 1517 1518Follow the [steps that Docker recommends to upgrade v1 images](https://docs.docker.com/registry/spec/deprecated-schema-v1/). 1519The most straightforward option is to pull those images and push them once again to the registry, 1520using a Docker client version above v1.12. Docker converts images automatically before pushing them 1521to the registry. Once done, all your v1 images should now be available as v2 images. 1522 1523### Tags with an empty name 1524 1525If using [AWS DataSync](https://aws.amazon.com/datasync/) 1526to copy the registry data to or between S3 buckets, an empty metadata object is created in the root 1527path of each container repository in the destination bucket. This causes the registry to interpret 1528such files as a tag that appears with no name in the GitLab UI and API. For more information, see 1529[this issue](https://gitlab.com/gitlab-org/container-registry/-/issues/341). 1530 1531To fix this you can do one of two things: 1532 1533- Use the AWS CLI [`rm`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/rm.html) 1534 command to remove the empty objects from the root of **each** affected repository. Pay special 1535 attention to the trailing `/` and make sure **not** to use the `--recursive` option: 1536 1537 ```shell 1538 aws s3 rm s3://<bucket>/docker/registry/v2/repositories/<path to repository>/ 1539 ``` 1540 1541- Use the AWS CLI [`sync`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html) 1542 command to copy the registry data to a new bucket and configure the registry to use it. This 1543 leaves the empty objects behind. 1544 1545### Advanced Troubleshooting 1546 1547We use a concrete example to illustrate how to 1548diagnose a problem with the S3 setup. 1549 1550#### Unexpected 403 error during push 1551 1552A user attempted to enable an S3-backed Registry. The `docker login` step went 1553fine. However, when pushing an image, the output showed: 1554 1555```plaintext 1556The push refers to a repository [s3-testing.myregistry.com:5050/root/docker-test/docker-image] 1557dc5e59c14160: Pushing [==================================================>] 14.85 kB 155803c20c1a019a: Pushing [==================================================>] 2.048 kB 1559a08f14ef632e: Pushing [==================================================>] 2.048 kB 1560228950524c88: Pushing 2.048 kB 15616a8ecde4cc03: Pushing [==> ] 9.901 MB/205.7 MB 15625f70bf18a086: Pushing 1.024 kB 1563737f40e80b7f: Waiting 156482b57dbc5385: Waiting 156519429b698a22: Waiting 15669436069b92a3: Waiting 1567error parsing HTTP 403 response body: unexpected end of JSON input: "" 1568``` 1569 1570This error is ambiguous, as it's not clear whether the 403 is coming from the 1571GitLab Rails application, the Docker Registry, or something else. In this 1572case, since we know that since the login succeeded, we probably need to look 1573at the communication between the client and the Registry. 1574 1575The REST API between the Docker client and Registry is described 1576[in the Docker documentation](https://docs.docker.com/registry/spec/api/). Normally, one would just 1577use Wireshark or tcpdump to capture the traffic and see where things went 1578wrong. However, since all communications between Docker clients and servers 1579are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even 1580if you know the private key. What can we do instead? 1581 1582One way would be to disable HTTPS by setting up an [insecure 1583Registry](https://docs.docker.com/registry/insecure/). This could introduce a 1584security hole and is only recommended for local testing. If you have a 1585production system and can't or don't want to do this, there is another way: 1586use mitmproxy, which stands for Man-in-the-Middle Proxy. 1587 1588#### mitmproxy 1589 1590[mitmproxy](https://mitmproxy.org/) allows you to place a proxy between your 1591client and server to inspect all traffic. One wrinkle is that your system 1592needs to trust the mitmproxy SSL certificates for this to work. 1593 1594The following installation instructions assume you are running Ubuntu: 1595 15961. [Install mitmproxy](https://docs.mitmproxy.org/stable/overview-installation/). 15971. Run `mitmproxy --port 9000` to generate its certificates. 1598 Enter <kbd>CTRL</kbd>-<kbd>C</kbd> to quit. 15991. Install the certificate from `~/.mitmproxy` to your system: 1600 1601 ```shell 1602 sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt 1603 sudo update-ca-certificates 1604 ``` 1605 1606If successful, the output should indicate that a certificate was added: 1607 1608```shell 1609Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. 1610Running hooks in /etc/ca-certificates/update.d....done. 1611``` 1612 1613To verify that the certificates are properly installed, run: 1614 1615```shell 1616mitmproxy --port 9000 1617``` 1618 1619This command runs mitmproxy on port `9000`. In another window, run: 1620 1621```shell 1622curl --proxy "http://localhost:9000" "https://httpbin.org/status/200" 1623``` 1624 1625If everything is set up correctly, information is displayed on the mitmproxy window and 1626no errors are generated by the curl commands. 1627 1628#### Running the Docker daemon with a proxy 1629 1630For Docker to connect through a proxy, you must start the Docker daemon with the 1631proper environment variables. The easiest way is to shutdown Docker (for example `sudo initctl stop docker`) 1632and then run Docker by hand. As root, run: 1633 1634```shell 1635export HTTP_PROXY="http://localhost:9000" 1636export HTTPS_PROXY="https://localhost:9000" 1637docker daemon --debug 1638``` 1639 1640This command launches the Docker daemon and proxies all connections through mitmproxy. 1641 1642#### Running the Docker client 1643 1644Now that we have mitmproxy and Docker running, we can attempt to sign in and 1645push a container image. You may need to run as root to do this. For example: 1646 1647```shell 1648docker login s3-testing.myregistry.com:5050 1649docker push s3-testing.myregistry.com:5050/root/docker-test/docker-image 1650``` 1651 1652In the example above, we see the following trace on the mitmproxy window: 1653 1654![mitmproxy output from Docker](img/mitmproxy-docker.png) 1655 1656The above image shows: 1657 1658- The initial PUT requests went through fine with a 201 status code. 1659- The 201 redirected the client to the S3 bucket. 1660- The HEAD request to the AWS bucket reported a 403 Unauthorized. 1661 1662What does this mean? This strongly suggests that the S3 user does not have the right 1663[permissions to perform a HEAD request](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html). 1664The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/). 1665Once the right permissions were set, the error goes away. 1666