1--- 2stage: Create 3group: Gitaly 4info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments 5type: reference 6--- 7 8# Gitaly developers guide 9 10[Gitaly](https://gitlab.com/gitlab-org/gitaly) is a high-level Git RPC service used by GitLab Rails, 11Workhorse and GitLab Shell. 12 13## Deep Dive 14 15<!-- vale gitlab.Spelling = NO --> 16 17In May 2019, Bob Van Landuyt 18hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) 19on the [Gitaly project](https://gitlab.com/gitlab-org/gitaly). It included how to contribute to it as a 20Ruby developer, and shared domain-specific knowledge with anyone who may work in this part of the 21codebase in the future. 22 23<!-- vale gitlab.Spelling = YES --> 24 25You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=BmlEWFS8ORo), and the slides 26on [Google Slides](https://docs.google.com/presentation/d/1VgRbiYih9ODhcPnL8dS0W98EwFYpJ7GXMPpX-1TM6YE/edit) 27and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/a4fdb1026278bda5c1c5bb574379cf80/Create_Deep_Dive__Gitaly_for_Create_Ruby_Devs.pdf). 28 29Everything covered in this deep dive was accurate as of GitLab 11.11, and while specific details may 30have changed, it should still serve as a good introduction. 31 32## Beginner's guide 33 34Start by reading the Gitaly repository's 35[Beginner's guide to Gitaly contributions](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/beginners_guide.md). 36It describes how to set up Gitaly, the various components of Gitaly and what 37they do, and how to run its test suites. 38 39## Developing new Git features 40 41To read or write Git data, a request has to be made to Gitaly. This means that 42if you're developing a new feature where you need data that's not yet available 43in `lib/gitlab/git` changes have to be made to Gitaly. 44 45There should be no new code that touches Git repositories via disk access (for example, 46Rugged, `git`, `rm -rf`) anywhere in the `gitlab` repository. Anything that 47needs direct access to the Git repository *must* be implemented in Gitaly, and 48exposed via an RPC. 49 50It's often easier to develop a new feature in Gitaly if you make the changes to 51GitLab that intends to use the new feature in a separate merge request, to be merged 52immediately after the Gitaly one. This allows you to test your changes before 53they are merged. 54 55- See [below](#running-tests-with-a-locally-modified-version-of-gitaly) for instructions on running GitLab tests with a modified version of Gitaly. 56- In GDK run `gdk install` and restart GDK using `gdk restart` to use a locally modified Gitaly version for development 57 58### `gitaly-ruby` 59 60It is possible to implement and test RPC's in Gitaly using Ruby code, 61in 62[`gitaly-ruby`](https://gitlab.com/gitlab-org/gitaly/tree/master/ruby). 63This should make it easier to contribute for developers who are less 64comfortable writing Go code. 65 66There is documentation for this approach in [the Gitaly 67repository](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/ruby_endpoint.md). 68 69## Gitaly-Related Test Failures 70 71If your test-suite is failing with Gitaly issues, as a first step, try running: 72 73```shell 74rm -rf tmp/tests/gitaly 75``` 76 77During RSpec tests, the Gitaly instance writes logs to `gitlab/log/gitaly-test.log`. 78 79## Legacy Rugged code 80 81While Gitaly can handle all Git access, many of GitLab customers still 82run Gitaly atop NFS. The legacy Rugged implementation for Git calls may 83be faster than the Gitaly RPC due to N+1 Gitaly calls and other 84reasons. See [the 85issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/57317) for more 86details. 87 88Until GitLab has eliminated most of these inefficiencies or the use of 89NFS is discontinued for Git data, Rugged implementations of some of the 90most commonly-used RPCs can be enabled via feature flags: 91 92- `rugged_find_commit` 93- `rugged_get_tree_entries` 94- `rugged_tree_entry` 95- `rugged_commit_is_ancestor` 96- `rugged_commit_tree_entry` 97- `rugged_list_commits_by_oid` 98 99A convenience Rake task can be used to enable or disable these flags 100all together. To enable: 101 102```shell 103bundle exec rake gitlab:features:enable_rugged 104``` 105 106To disable: 107 108```shell 109bundle exec rake gitlab:features:disable_rugged 110``` 111 112Most of this code exists in the `lib/gitlab/git/rugged_impl` directory. 113 114NOTE: 115You should *not* need to add or modify code related to 116Rugged unless explicitly discussed with the 117[Gitaly Team](https://gitlab.com/groups/gl-gitaly/group_members). This code does 118NOT work on GitLab.com or other GitLab instances that do not use NFS. 119 120## `TooManyInvocationsError` errors 121 122During development and testing, you may experience `Gitlab::GitalyClient::TooManyInvocationsError` failures. 123The `GitalyClient` attempts to block against potential n+1 issues by raising this error 124when Gitaly is called more than 30 times in a single Rails request or Sidekiq execution. 125 126As a temporary measure, export `GITALY_DISABLE_REQUEST_LIMITS=1` to suppress the error. This disables the n+1 detection 127in your development environment. 128 129Please raise an issue in the GitLab CE or EE repositories to report the issue. Include the labels ~Gitaly 130~performance ~"technical debt". Please ensure that the issue contains the full stack trace and error message of the 131`TooManyInvocationsError`. Also include any known failing tests if possible. 132 133Isolate the source of the n+1 problem. This is normally a loop that results in Gitaly being called for each 134element in an array. If you are unable to isolate the problem, please contact a member 135of the [Gitaly Team](https://gitlab.com/groups/gl-gitaly/group_members) for assistance. 136 137After the source has been found, wrap it in an `allow_n_plus_1_calls` block, as follows: 138 139```ruby 140# n+1: link to n+1 issue 141Gitlab::GitalyClient.allow_n_plus_1_calls do 142 # original code 143 commits.each { |commit| ... } 144end 145``` 146 147After the code is wrapped in this block, this code path is excluded from n+1 detection. 148 149## Request counts 150 151Commits and other Git data, is now fetched through Gitaly. These fetches can, 152much like with a database, be batched. This improves performance for the client 153and for Gitaly itself and therefore for the users too. To keep performance stable 154and guard performance regressions, Gitaly calls can be counted and the call count 155can be tested against. This requires the `:request_store` flag to be set. 156 157```ruby 158describe 'Gitaly Request count tests' do 159 context 'when the request store is activated', :request_store do 160 it 'correctly counts the gitaly requests made' do 161 expect { subject }.to change { Gitlab::GitalyClient.get_request_count }.by(10) 162 end 163 end 164end 165``` 166 167## Running tests with a locally modified version of Gitaly 168 169Normally, GitLab CE/EE tests use a local clone of Gitaly in 170`tmp/tests/gitaly` pinned at the version specified in 171`GITALY_SERVER_VERSION`. The `GITALY_SERVER_VERSION` file supports also 172branches and SHA to use a custom commit in [the repository](https://gitlab.com/gitlab-org/gitaly). 173 174NOTE: 175With the introduction of auto-deploy for Gitaly, the format of 176`GITALY_SERVER_VERSION` was aligned with Omnibus syntax. 177It no longer supports `=revision`, it evaluates the file content as a Git 178reference (branch or SHA). Only if it matches a semantic version does it prepend a `v`. 179 180If you want to run tests locally against a modified version of Gitaly you 181can replace `tmp/tests/gitaly` with a symlink. This is much faster 182because it avoids a Gitaly re-install each time you run `rspec`. 183 184Make sure this directory contains the files `config.toml` and `praefect.config.toml`. 185You can copy them from `config.toml.example` and `config.praefect.toml.example` respectively. 186After copying, make sure to edit them so everything points to the correct paths. 187 188```shell 189rm -rf tmp/tests/gitaly 190ln -s /path/to/gitaly tmp/tests/gitaly 191``` 192 193Make sure you run `make` in your local Gitaly directory before running 194tests. Otherwise, Gitaly fails to boot. 195 196If you make changes to your local Gitaly in between test runs you need 197to manually run `make` again. 198 199Note that CI tests do not use your locally modified version of 200Gitaly. To use a custom Gitaly version in CI you need to update 201GITALY_SERVER_VERSION as described at the beginning of this section. 202 203To use a different Gitaly repository, such as if your changes are present 204on a fork, you can specify a `GITALY_REPO_URL` environment variable when 205running tests: 206 207```shell 208GITALY_REPO_URL=https://gitlab.com/nick.thomas/gitaly bundle exec rspec spec/lib/gitlab/git/repository_spec.rb 209``` 210 211If your fork of Gitaly is private, you can generate a [Deploy Token](../user/project/deploy_tokens/index.md) 212and specify it in the URL: 213 214```shell 215GITALY_REPO_URL=https://gitlab+deploy-token-1000:token-here@gitlab.com/nick.thomas/gitaly bundle exec rspec spec/lib/gitlab/git/repository_spec.rb 216``` 217 218To use a custom Gitaly repository in CI/CD, for instance if you want your 219GitLab fork to always use your own Gitaly fork, set `GITALY_REPO_URL` 220as a [CI/CD variable](../ci/variables/index.md). 221 222### Use a locally modified version of Gitaly RPC client 223 224If you are making changes to the RPC client, such as adding a new endpoint or adding a new 225parameter to an existing endpoint, follow the guide for 226[Gitaly protobuf specifications](https://gitlab.com/gitlab-org/gitaly/blob/master/proto/README.md). After pushing 227the branch with the changes (`new-feature-branch`, for example): 228 2291. Change the `gitaly` line in the Rails' `Gemfile` to: 230 231 ```ruby 232 gem 'gitaly', git: 'https://gitlab.com/gitlab-org/gitaly.git', branch: 'new-feature-branch' 233 ``` 234 2351. Run `bundle install` to use the modified RPC client. 236 237Re-run `bundle install` in the `gitlab` project each time the Gitaly branch 238changes to embed a new SHA in the `Gemfile.lock` file. 239 240--- 241 242[Return to Development documentation](index.md) 243 244## Wrapping RPCs in Feature Flags 245 246Here are the steps to gate a new feature in Gitaly behind a feature flag. 247 248### Gitaly 249 2501. Create a package scoped flag name: 251 252 ```golang 253 var findAllTagsFeatureFlag = "go-find-all-tags" 254 ``` 255 2561. Create a switch in the code using the `featureflag` package: 257 258 ```golang 259 if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) { 260 // go implementation 261 } else { 262 // ruby implementation 263 } 264 ``` 265 2661. Create Prometheus metrics: 267 268 ```golang 269 var findAllTagsRequests = prometheus.NewCounterVec( 270 prometheus.CounterOpts{ 271 Name: "gitaly_find_all_tags_requests_total", 272 Help: "Counter of go vs ruby implementation of FindAllTags", 273 }, 274 []string{"implementation"}, 275 ) 276 277 func init() { 278 prometheus.Register(findAllTagsRequests) 279 } 280 281 if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) { 282 findAllTagsRequests.WithLabelValues("go").Inc() 283 // go implementation 284 } else { 285 findAllTagsRequests.WithLabelValues("ruby").Inc() 286 // ruby implementation 287 } 288 ``` 289 2901. Set headers in tests: 291 292 ```golang 293 import ( 294 "google.golang.org/grpc/metadata" 295 296 "gitlab.com/gitlab-org/gitaly/internal/featureflag" 297 ) 298 299 //... 300 301 md := metadata.New(map[string]string{featureflag.HeaderKey(findAllTagsFeatureFlag): "true"}) 302 ctx = metadata.NewOutgoingContext(context.Background(), md) 303 304 c, err = client.FindAllTags(ctx, rpcRequest) 305 require.NoError(t, err) 306 ``` 307 308### GitLab Rails 309 310Test in a Rails console by setting the feature flag: 311 312```ruby 313Feature.enable('gitaly_go_find_all_tags') 314``` 315 316Pay attention to the name of the flag and the one used in the Rails console. There is a difference 317between them (dashes replaced by underscores and name prefix is changed). Make sure to prefix all 318flags with `gitaly_`. 319 320NOTE: 321If not set in GitLab, feature flags are read as false from the console and Gitaly uses their 322default value. The default value depends on the GitLab version. 323 324### Testing with GDK 325 326To be sure that the flag is set correctly and it goes into Gitaly, you can check 327the integration by using GDK: 328 3291. The state of the flag must be observable. To check it, you need to enable it 330 by fetching the Prometheus metrics: 331 1. Navigate to GDK's root directory. 332 1. Make sure you have the proper branch checked out for Gitaly. 333 1. Recompile it with `make gitaly-setup` and restart the service with `gdk restart gitaly`. 334 1. Make sure your setup is running: `gdk status | grep praefect`. 335 1. Check what configuration file is used: `cat ./services/praefect/run | grep praefect` value of the `-config` flag 336 1. Uncomment `prometheus_listen_addr` in the configuration file and run `gdk restart gitaly`. 337 3381. Make sure that the flag is not enabled yet: 339 1. Perform whatever action is required to trigger your changes, such as project creation, 340 submitting commit, or observing history. 341 1. Check that the list of current metrics has the new counter for the feature flag: 342 343 ```shell 344 curl --silent "http://localhost:9236/metrics" | grep go_find_all_tags 345 ``` 346 3471. After you observe the metrics for the new feature flag and it increments, you 348 can enable the new feature: 349 1. Navigate to GDK's root directory. 350 1. Start a Rails console: 351 352 ```shell 353 bundle install && bundle exec rails console 354 ``` 355 356 1. Check the list of feature flags: 357 358 ```ruby 359 Feature::Gitaly.server_feature_flags 360 ``` 361 362 It should be disabled `"gitaly-feature-go-find-all-tags"=>"false"`. 363 1. Enable it: 364 365 ```ruby 366 Feature.enable('gitaly_go_find_all_tags') 367 ``` 368 369 1. Exit the Rails console and perform whatever action is required to trigger 370 your changes, such as project creation, submitting commit, or observing history. 371 1. Verify the feature is on by observing the metrics for it: 372 373 ```shell 374 curl --silent "http://localhost:9236/metrics" | grep go_find_all_tags 375 ``` 376