1--- 2stage: none 3group: unassigned 4info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments 5--- 6 7# Ruby upgrade guidelines 8 9We strive to run GitLab using the latest Ruby MRI releases to benefit from performance and 10security updates and new Ruby APIs. When upgrading Ruby across GitLab, we should do 11so in a way that: 12 13- Is least disruptive to contributors. 14- Optimizes for GitLab SaaS availability. 15- Maintains Ruby version parity across all parts of GitLab. 16 17Before making changes to Ruby versions, read through this document carefully and entirely to get a high-level 18understanding of what changes may be necessary. It is likely that every Ruby upgrade is a little 19different than the one before it, so assess the order and necessity of the documented 20steps. 21 22## Scope of a Ruby upgrade 23 24The first thing to consider when upgrading Ruby is scope. In general, we consider 25the following areas in which Ruby updates may have to occur: 26 27- The main GitLab Rails repository. 28- Any ancillary Ruby system repositories. 29- Any third-party libraries used by systems in these repositories. 30- Any GitLab libraries used by systems in these repositories. 31 32We may not always have to touch all of these. For instance, a patch-level Ruby update is 33unlikely to require updates in third-party gems. 34 35### Patch, minor, and major upgrades 36 37When assessing scope, the Ruby version level matters. For instance, it is harder and riskier 38to upgrade GitLab from Ruby 2.x to 3.x than it is to upgrade from Ruby 2.7.2 to 2.7.4, as 39patch releases are typically restricted to security or bug fixes. 40Be aware of this when preparing an upgrade and plan accordingly. 41 42To help you estimate the scope of future upgrades, see the efforts required for the following upgrades: 43 44- [Patch upgrade 2.7.2 -> 2.7.4](https://gitlab.com/gitlab-org/gitlab/-/issues/335890) 45- [Minor upgrade 2.6.x -> 2.7.x](https://gitlab.com/groups/gitlab-org/-/epics/2380) 46- [Major upgrade 2.x.x -> 3.x.x](https://gitlab.com/groups/gitlab-org/-/epics/5149) 47 48## Affected audiences and targets 49 50Before any upgrade, consider all audiences and targets, ordered by how immediately they are affected by Ruby upgrades: 51 521. **Developers.** We have many contributors to GitLab and related projects both inside and outside the company. Changing files such as `.ruby-version` affects everyone using tooling that interprets these files. 53The developers are affected as soon as they pull from the repository containing the merged changes. 541. **GitLab CI/CD.** We heavily lean on CI/CD for code integration and testing. CI/CD jobs do not interpret files such as `.ruby-version`. 55Instead, they use the Ruby installed in the Docker container they execute in, which is defined in `.gitlab-ci.yml`. 56The container images used in these jobs are maintained in the [`gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) repository. 57When we merge an update to an image, CI/CD jobs are affected as soon as the [image is built](https://gitlab.com/gitlab-org/gitlab-build-images/#pushing-a-rebuild-image). 581. **GitLab SaaS**. GitLab.com is deployed from customized Helm charts that use Docker images from [Cloud Native GitLab (CNG)](https://gitlab.com/gitlab-org/build/CNG). 59Just like CI/CD, `.ruby-version` is meaningless in this environment. Instead, those Docker images must be patched to upgrade Ruby. 60GitLab SaaS is affected with the next deployment. 611. **Self-managed GitLab.** Customers installing GitLab via [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab) use none of the above. 62Instead, their Ruby version is defined by the [Ruby software bundle](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/config/software/ruby.rb) in Omnibus. 63Self-managed customers are affected as soon as they upgrade to the release containing this change. 64 65## Ruby upgrade approach 66 67Timing all steps in a Ruby upgrade correctly is critical. As a general guideline, consider the following: 68 69- For smaller upgrades where production behavior is unlikely to change, aim to keep the version gap between 70repositories and production minimal. Coordinate with stakeholders to merge all changes closely together 71(within a day or two) to avoid drift. In this scenario the likely order is to upgrade developer tooling and 72environments first, production second. 73- For larger changes, the risk of going to production with a new Ruby is significant. In this case, try to get into a 74position where all known incompatibilities with the new Ruby version are already fixed, then work 75with production engineers to deploy the new Ruby to a subset of the GitLab production fleet. In this scenario 76the likely order is to update production first, developer tooling and environments second. This makes rollbacks 77easier in case of critical regressions in production. 78 79Either way, we found that from past experience the following approach works well, with some steps likely only 80necessary for minor and major upgrades. Note that some of these steps can happen in parallel or may have their 81order reversed as described above. 82 83### Create an epic 84 85Tracking this work in an epic is useful to get a sense of progress. For larger upgrades, include a 86timeline in the epic description so stakeholders know when the final switch is expected to go live. 87 88Break changes to individual repositories into separate issues under this epic. 89 90### Communicate the intent to upgrade 91 92Especially for upgrades that introduce or deprecate features, 93communicate early that an upgrade is due, ideally with an associated timeline. Provide links to important or 94noteworthy changes, so developers can start to familiarize themselves with 95changes ahead of time. 96 97GitLab team members should announce the intent in relevant Slack channels (`#backend` and `#development` at minimum) 98and Engineering Week In Review (EWIR). Include a link to the upgrade epic in your 99[communication](https://about.gitlab.com/handbook/engineering/#communication). 100 101### Add new Ruby to CI/CD and development environments 102 103To build and run Ruby gems and the GitLab Rails application with a new Ruby, you must first prepare CI/CD 104and developer environments to include the new Ruby version. 105At this stage, you *must not make it the default Ruby yet*, but make it optional instead. This allows 106for a smoother transition by supporting both old and new Ruby versions for a period of time. 107 108There are two places that require changes: 109 1101. **[GitLab Build Images](https://gitlab.com/gitlab-org/gitlab-build-images).** These are Docker images 111we use for runners and other Docker-based pre-production environments. The kind of change necessary 112depends on the scope. 113 - For [patch level updates](https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/418), it should suffice to increment the patch level of `RUBY_VERSION`. 114All projects building against the same minor release automatically download the new patch release. 115 - For [major and minor updates](https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/320), create a new set of Docker images that can be used side-by-side with existing images during the upgrade process. **Important:** Make sure to copy over all Ruby patch files 116in the `/patches` directory to a new folder matching the Ruby version you upgrade to, or they aren't applied. 1171. **[GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit).** 118Update GDK to add the new Ruby as an additional option for 119developers to choose from. This typically only requires it to be appended to `.tool-versions` so `asdf` 120users will benefit from this. Other users will have to install it manually 121([example](https://gitlab.com/gitlab-org/gitlab-development-kit/-/merge_requests/2136).) 122 123For larger version upgrades, consider working with [Quality Engineering](https://about.gitlab.com/handbook/engineering/quality/) 124to identify and set up a test plan. 125 126### Update third-party gems 127 128For patch releases this is unlikely to be necessary, but 129for minor and major releases, there could be breaking changes or Bundler dependency issues when gems 130pin Ruby to a particular version. A good way to find out is to create a merge request in `gitlab-org/gitlab` 131and see what breaks. 132 133### Update GitLab gems and related systems 134 135This is typically necessary, since gems or Ruby applications that we maintain ourselves contain the build setup such as 136`.ruby-version`, `.tool-versions`, or `.gitlab-ci.yml` files. While there isn't always a technical necessity to 137update these repositories for the GitLab Rails application to work with a new Ruby, 138it is good practice to keep Ruby versions in lock-step across all our repositories. For minor and major 139upgrades, add new CI/CD jobs to these repositories using the new Ruby. 140A [build matrix definition](../ci/yaml/index.md#parallelmatrix) can do this efficiently. 141 142#### Decide which repositories to update 143 144When upgrading Ruby, consider updating the following repositories: 145 146- [Gitaly](https://gitlab.com/gitlab-org/gitaly) ([example](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3771)) 147- [GitLab Labkit](https://gitlab.com/gitlab-org/labkit-ruby) ([example](https://gitlab.com/gitlab-org/labkit-ruby/-/merge_requests/79)) 148- [GitLab Exporter](https://gitlab.com/gitlab-org/gitlab-exporter) ([example](https://gitlab.com/gitlab-org/gitlab-exporter/-/merge_requests/150)) 149- [GitLab Experiment](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) ([example](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment/-/merge_requests/128)) 150- [Gollum Lib](https://gitlab.com/gitlab-org/gollum-lib) ([example](https://gitlab.com/gitlab-org/gollum-lib/-/merge_requests/21)) 151- [GitLab Helm Chart](https://gitlab.com/gitlab-org/charts/gitlab) ([example](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2162)) 152- [GitLab Sidekiq fetcher](https://gitlab.com/gitlab-org/sidekiq-reliable-fetch) ([example](https://gitlab.com/gitlab-org/sidekiq-reliable-fetch/-/merge_requests/33)) 153- [Prometheus Ruby Mmap Client](https://gitlab.com/gitlab-org/prometheus-client-mmap) ([example](https://gitlab.com/gitlab-org/prometheus-client-mmap/-/merge_requests/59)) 154- [GitLab-mail_room](https://gitlab.com/gitlab-org/gitlab-mail_room) ([example](https://gitlab.com/gitlab-org/gitlab-mail_room/-/merge_requests/16)) 155 156To assess which of these repositories are critical to be updated alongside the main GitLab application consider: 157 158- The Ruby version scope. 159- The role that the service or library plays in the overall functioning of GitLab. 160 161Refer to the [list of GitLab projects](https://about.gitlab.com/handbook/engineering/projects/) for a complete 162account of which repositories could be affected. 163For smaller version upgrades, it can be acceptable to delay updating libraries that are non-essential or where 164we are certain that the main application test suite would catch regressions under a new Ruby version. 165 166NOTE: 167Consult with the respective code owners whether it is acceptable to merge these changes ahead 168of updating the GitLab application. It might be best to get the necessary approvals 169but wait to merge the change until everything is ready. 170 171### Prepare the GitLab application MR 172 173With the dependencies updated and the new gem versions released, you can update the main Rails 174application with any necessary changes, similar to the gems and related systems. 175On top of that, update the documentation to reflect the version change in the installation 176and update instructions ([example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68363)). 177 178NOTE: 179Be especially careful with timing this merge request, since as soon as it is merged, all GitLab contributors 180will be affected by it and the changes will be deployed. You must ensure that this MR remains 181open until everything else is ready, but it can be useful to get approval early to reduce lead time. 182 183### Give developers time to upgrade (grace period) 184 185With the new Ruby made available as an option, and all merge requests either ready or merged, 186there should be a grace period (1 week at minimum) during which developers can 187install the new Ruby on their machines. For GDK and `asdf` users this should happen automatically 188via `gdk update`. 189 190This pause is a good time to assess the risk of this upgrade for GitLab SaaS. 191For Ruby upgrades that are high risk, such as major version upgrades, it is recommended to 192coordinate the changes with the infrastructure team through a [change management request](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/). 193Create this issue early to give everyone enough time to schedule and prepare changes. 194 195### Make it the default Ruby 196 197If there are no known version compatibility issues left, and the grace 198period has passed, all affected repositories and developer tools should be updated to make the new Ruby 199default. 200 201At this point, update the [GitLab Compose Kit (GCK)](https://gitlab.com/gitlab-org/gitlab-compose-kit). 202This is an alternative development environment for users that prefer to run GitLab in `docker-compose`. 203This project relies on the same Docker images as our runners, so it should maintain parity with changes 204in that repository. This change is only necessary when the minor or major version changes 205([example](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/merge_requests/176).) 206 207As mentioned above, if the impact of the Ruby upgrade on SaaS availability is uncertain, it is 208prudent to skip this step until you have verified that it runs smoothly in production via a staged 209rollout. In this case, go to the next step first, and then, after the verification period has passed, promote 210the new Ruby to be the new default. 211 212### Update CNG and Omnibus, merge the GitLab MR 213 214The last step is to use the new Ruby in production. This 215requires updating Omnibus and production Docker images to use the new version. 216Helm charts may also have to be updated if there were changes to related systems that maintain 217their own charts (such as `gitlab-exporter`.) 218 219To use the new Ruby in production, update the following projects: 220 221- [Cloud-native GitLab Docker Images (CNG)](https://gitlab.com/gitlab-org/build/CNG) ([example](https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/739)) 222- [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) ([example](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5545)) 223 224If you submit a change management request, coordinate the rollout with infrastructure 225engineers. When dealing with larger upgrades, involve [Release Managers](https://about.gitlab.com/community/release-managers/) 226in the rollout plan. 227 228### Create patch releases and backports for security patches 229 230If the upgrade was a patch release and contains important security fixes, it should be released as a 231GitLab patch release to self-managed customers. Consult our [release managers](https://about.gitlab.com/community/release-managers/) 232for how to proceed. 233 234## Ruby upgrade tooling 235 236There are several tools that ease the upgrade process. 237 238### Deprecation Toolkit 239 240A common problem with Ruby upgrades is that deprecation warnings turn into errors. This means that every single 241deprecation warning must be resolved before making the switch. To avoid new warnings from making it into the 242main application branch, we use [`DeprecationToolkitEnv`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/deprecation_toolkit_env.rb). 243This module observes deprecation warnings emitted from spec runs and turns them into test failures. This prevents 244developers from checking in new code that would fail under a new Ruby. 245 246Sometimes it cannot be avoided to introduce new warnings, for example when a Ruby gem we use emits these warnings 247and we have no control over it. In these cases, add silences, like [this merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68865) did. 248 249### Deprecation Logger 250 251We also log Ruby and Rails deprecation warnings to a dedicated log file, `log/deprecation_json.log` 252(see [GitLab Developers Guide to Logging](logging.md) for where to find GitLab log files), 253which can provide clues when there is code that is not adequately covered by tests and hence would slip past `DeprecationToolkitEnv`. 254 255For GitLab SaaS, GitLab team members can inspect these log events in Kibana 256(`https://log.gprd.gitlab.net/goto/f7cebf1ff05038d901ba2c45925c7e01`). 257 258## Recommendations 259 260During the upgrade process, consider the following recommendations: 261 262- **Front-load as many changes as possible.** Especially for minor and major releases, it is likely that application 263code will break or change. Any changes that are backward compatible should be merged into the main branch and 264released independently ahead of the Ruby version upgrade. This ensures that we move in small increments and 265get feedback from production environments early. 266- **Create an experimental branch for larger updates.** We generally try to avoid long-running topic branches, 267but for purposes of feedback and experimentation, it can be useful to have such a branch to get regular 268feedback from CI/CD when running a newer Ruby. This can be helpful when first assessing what problems 269we might run into, as [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50640) demonstrates. 270These experimental branches are not intended to be merged; they can be closed once all required changes have been broken out 271and merged back independently. 272- **Give yourself enough time to fix problems ahead of a milestone release.** GitLab moves fast. 273As a Ruby upgrade requires many MRs to be sent and reviewed, make sure all changes are merged at least a week 274before the 22nd. This gives us extra time to act if something breaks. If in doubt, it is better to 275postpone the upgrade to the following month, as we [prioritize availability over velocity](https://about.gitlab.com/handbook/engineering/#prioritizing-technical-decisions). 276