1--- 2stage: Enablement 3group: Geo 4info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments 5type: howto 6--- 7 8# Geo configuration **(PREMIUM SELF)** 9 10## Configuring a new **secondary** site 11 12NOTE: 13This is the final step in setting up a **secondary** Geo site. Stages of the 14setup process must be completed in the documented order. 15Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). 16 17The basic steps of configuring a **secondary** site are to: 18 19- Replicate required configurations between the **primary** site and the **secondary** sites. 20- Configure a tracking database on each **secondary** site. 21- Start GitLab on each **secondary** site. 22 23You are encouraged to first read through all the steps before executing them 24in your testing/production environment. 25 26NOTE: 27**Do not** set up any custom authentication for the **secondary** sites. This is handled by the **primary** site. 28Any change that requires access to the **Admin Area** needs to be done in the 29**primary** site because the **secondary** site is a read-only replica. 30 31### Step 1. Manually replicate secret GitLab values 32 33GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json` 34file which *must* be the same on all of a site's nodes. Until there is 35a means of automatically replicating these between sites (see [issue #3789](https://gitlab.com/gitlab-org/gitlab/-/issues/3789)), 36they must be manually replicated to **all nodes of the secondary site**. 37 381. SSH into a **Rails node on your primary** site, and execute the command below: 39 40 ```shell 41 sudo cat /etc/gitlab/gitlab-secrets.json 42 ``` 43 44 This displays the secrets that need to be replicated, in JSON format. 45 461. SSH **into each node on your secondary Geo site** and login as the `root` user: 47 48 ```shell 49 sudo -i 50 ``` 51 521. Make a backup of any existing secrets: 53 54 ```shell 55 mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` 56 ``` 57 581. Copy `/etc/gitlab/gitlab-secrets.json` from the **Rails node on your primary** site to **each node on your secondary** site, or 59 copy-and-paste the file contents between nodes: 60 61 ```shell 62 sudo editor /etc/gitlab/gitlab-secrets.json 63 64 # paste the output of the `cat` command you ran on the primary 65 # save and exit 66 ``` 67 681. Ensure the file permissions are correct: 69 70 ```shell 71 chown root:root /etc/gitlab/gitlab-secrets.json 72 chmod 0600 /etc/gitlab/gitlab-secrets.json 73 ``` 74 751. Reconfigure **each Rails, Sidekiq and Gitaly nodes on your secondary** site for the change to take effect: 76 77 ```shell 78 gitlab-ctl reconfigure 79 gitlab-ctl restart 80 ``` 81 82### Step 2. Manually replicate the **primary** site's SSH host keys 83 84GitLab integrates with the system-installed SSH daemon, designating a user 85(typically named `git`) through which all access requests are handled. 86 87In a [Disaster Recovery](../disaster_recovery/index.md) situation, GitLab system 88administrators promote a **secondary** site to the **primary** site. DNS records for the 89**primary** domain should also be updated to point to the new **primary** site 90(previously a **secondary** site). Doing so avoids the need to update Git remotes and API URLs. 91 92This causes all SSH requests to the newly promoted **primary** site to 93fail due to SSH host key mismatch. To prevent this, the primary SSH host 94keys must be manually replicated to the **secondary** site. 95 961. SSH into **each node on your secondary** site and login as the `root` user: 97 98 ```shell 99 sudo -i 100 ``` 101 1021. Make a backup of any existing SSH host keys: 103 104 ```shell 105 find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; 106 ``` 107 1081. Copy OpenSSH host keys from the **primary** site: 109 110 If you can access one of the **nodes on your primary** site serving SSH traffic (usually, the main GitLab Rails application nodes) using the **root** user: 111 112 ```shell 113 # Run this from the secondary site, change `<primary_site_fqdn>` for the IP or FQDN of the server 114 scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh 115 ``` 116 117 If you only have access through a user with `sudo` privileges: 118 119 ```shell 120 # Run this from the node on your primary site: 121 sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* 122 123 # Run this on each node on your secondary site: 124 scp <user_with_sudo>@<primary_site_fqdn>:geo-host-key.tar.gz . 125 tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh 126 ``` 127 1281. On **each node on your secondary** site, ensure the file permissions are correct: 129 130 ```shell 131 chown root:root /etc/ssh/ssh_host_*_key* 132 chmod 0600 /etc/ssh/ssh_host_*_key* 133 ``` 134 1351. To verify key fingerprint matches, execute the following command on both primary and secondary nodes on each site: 136 137 ```shell 138 for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done 139 ``` 140 141 You should get an output similar to this one and they should be identical on both nodes: 142 143 ```shell 144 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) 145 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) 146 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) 147 2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA) 148 ``` 149 1501. Verify that you have the correct public keys for the existing private keys: 151 152 ```shell 153 # This will print the fingerprint for private keys: 154 for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done 155 156 # This will print the fingerprint for public keys: 157 for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done 158 ``` 159 160 NOTE: 161 The output for private keys and public keys command should generate the same fingerprint. 162 1631. Restart `sshd` on **each node on your secondary** site: 164 165 ```shell 166 # Debian or Ubuntu installations 167 sudo service ssh reload 168 169 # CentOS installations 170 sudo service sshd reload 171 ``` 172 1731. Verify SSH is still functional. 174 175 SSH into your GitLab **secondary** server in a new terminal. If you are unable to connect, 176 verify the permissions are correct according to the previous steps. 177 178### Step 3. Add the **secondary** site 179 1801. SSH into **each Rails and Sidekiq node on your secondary** site and login as root: 181 182 ```shell 183 sudo -i 184 ``` 185 1861. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your site. You need this in the next steps: 187 188 ```ruby 189 ## 190 ## The unique identifier for the Geo site. See 191 ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings 192 ## 193 gitlab_rails['geo_node_name'] = '<site_name_here>' 194 ``` 195 1961. Reconfigure **each Rails and Sidekiq node on your secondary** site for the change to take effect: 197 198 ```shell 199 gitlab-ctl reconfigure 200 ``` 201 2021. On the top bar, select **Menu > Admin**. 2031. On the left sidebar, select **Geo > Sites**. 2041. Select **New site**. 205 ![Add secondary site](img/adding_a_secondary_v13_3.png) 2061. Fill in **Name** with the `gitlab_rails['geo_node_name']` in 207 `/etc/gitlab/gitlab.rb`. These values must always match *exactly*, character 208 for character. 2091. Fill in **URL** with the `external_url` in `/etc/gitlab/gitlab.rb`. These 210 values must always match, but it doesn't matter if one ends with a `/` and 211 the other doesn't. 2121. Optionally, choose which groups or storage shards should be replicated by the 213 **secondary** site. Leave blank to replicate all. Read more in 214 [selective synchronization](#selective-synchronization). 2151. Select **Add site** to add the **secondary** site. 2161. SSH into **each Rails, and Sidekiq node on your secondary** site and restart the services: 217 218 ```shell 219 gitlab-ctl restart 220 ``` 221 222 Check if there are any common issue with your Geo setup by running: 223 224 ```shell 225 gitlab-rake gitlab:geo:check 226 ``` 227 2281. SSH into a **Rails or Sidekiq server on your primary** site and login as root to verify the 229 **secondary** site is reachable or there are any common issue with your Geo setup: 230 231 ```shell 232 gitlab-rake gitlab:geo:check 233 ``` 234 235Once added to the Geo administration page and restarted, the **secondary** site automatically starts 236replicating missing data from the **primary** site in a process known as **backfill**. 237Meanwhile, the **primary** site starts to notify each **secondary** site of any changes, so 238that the **secondary** site can act on those notifications immediately. 239 240Be sure the _secondary_ site is running and accessible. You can sign in to the 241_secondary_ site with the same credentials as were used with the _primary_ site. 242 243### Step 4. (Optional) Configuring the **secondary** site to trust the **primary** site 244 245You can safely skip this step if your **primary** site uses a CA-issued HTTPS certificate. 246 247If your **primary** site is using a self-signed certificate for *HTTPS* support, you 248need to add that certificate to the **secondary** site's trust store. Retrieve the 249certificate from the **primary** site and follow 250[these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) 251on the **secondary** site. 252 253### Step 5. Enable Git access over HTTP/HTTPS 254 255Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone 256method to be enabled. This is enabled by default, but if converting an existing site to Geo it should be checked: 257 258On the **primary** site: 259 2601. On the top bar, select **Menu > Admin**. 2611. On the left sidebar, select **Settings > General**. 2621. Expand **Visibility and access controls**. 2631. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)". 264 265### Step 6. Verify proper functioning of the **secondary** site 266 267You can sign in to the **secondary** site with the same credentials you used with 268the **primary** site. After you sign in: 269 2701. On the top bar, select **Menu > Admin**. 2711. On the left sidebar, select **Geo > Sites**. 2721. Verify that it's correctly identified as a **secondary** Geo site, and that 273 Geo is enabled. 274 275The initial replication, or 'backfill', is probably still in progress. You 276can monitor the synchronization process on each Geo site from the **primary** 277site's **Geo Sites** dashboard in your browser. 278 279![Geo dashboard](img/geo_dashboard_v14_0.png) 280 281If your installation isn't working properly, check the 282[troubleshooting document](troubleshooting.md). 283 284The two most obvious issues that can become apparent in the dashboard are: 285 2861. Database replication not working well. 2871. Instance to instance notification not working. In that case, it can be 288 something of the following: 289 - You are using a custom certificate or custom CA (see the [troubleshooting document](troubleshooting.md)). 290 - The instance is firewalled (check your firewall rules). 291 292Disabling a **secondary** site stops the synchronization process. 293 294If `git_data_dirs` is customized on the **primary** site for multiple 295repository shards you must duplicate the same configuration on each **secondary** site. 296 297Point your users to the [Using a Geo Site guide](usage.md). 298 299Currently, this is what is synced: 300 301- Git repositories. 302- Wikis. 303- LFS objects. 304- Issues, merge requests, snippets, and comment attachments. 305- Users, groups, and project avatars. 306 307## Selective synchronization 308 309Geo supports selective synchronization, which allows administrators to choose 310which projects should be synchronized by **secondary** sites. 311A subset of projects can be chosen, either by group or by storage shard. The 312former is ideal for replicating data belonging to a subset of users, while the 313latter is more suited to progressively rolling out Geo to a large GitLab 314instance. 315 316It is important to note that selective synchronization: 317 3181. Does not restrict permissions from **secondary** sites. 3191. Does not hide project metadata from **secondary** sites. 320 - Since Geo currently relies on PostgreSQL replication, all project metadata 321 gets replicated to **secondary** sites, but repositories that have not been 322 selected are empty. 3231. Does not reduce the number of events generated for the Geo event log. 324 - The **primary** site generates events as long as any **secondary** sites are present. 325 Selective synchronization restrictions are implemented on the **secondary** sites, 326 not the **primary** site. 327 328### Git operations on unreplicated repositories 329 330> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2562) in GitLab 12.10 for HTTP(S) and in GitLab 13.0 for SSH. 331 332Git clone, pull, and push operations over HTTP(S) and SSH are supported for repositories that 333exist on the **primary** site but not on **secondary** sites. This situation can occur 334when: 335 336- Selective synchronization does not include the project attached to the repository. 337- The repository is actively being replicated but has not completed yet. 338 339## Upgrading Geo 340 341See the [updating the Geo sites document](updating_the_geo_sites.md). 342 343## Troubleshooting 344 345See the [troubleshooting document](troubleshooting.md). 346