1--- 2stage: Configure 3group: Configure 4info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments 5--- 6 7# GitLab Agent for Kubernetes **(FREE)** 8 9> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223061) in GitLab 13.4. 10> - Support for `grpcs` [introduced](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/7) in GitLab 13.6. 11> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300960) in GitLab 13.10, KAS became available on GitLab.com under `wss://kas.gitlab.com` through an Early Adopter Program. 12> - Introduced in GitLab 13.11, the GitLab Agent became available to every project on GitLab.com. 13> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. 14> - [Renamed](https://gitlab.com/groups/gitlab-org/-/epics/7167) from "GitLab Kubernetes Agent" to "GitLab Agent for Kubernetes" in GitLab 14.6. 15 16The [GitLab Agent for Kubernetes](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) ("Agent", for short) 17is an active in-cluster component for connecting Kubernetes clusters to GitLab safely to support cloud-native deployment, management, and monitoring. 18 19The Agent is installed into the cluster through code, providing you with a fast, safe, stable, and scalable solution. 20 21INFO: 22Get Network Security Alerts in GitLab by upgrading to Ultimate. 23[Try a free 30-day trial now](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-cluster-agent-docs). 24 25With GitOps, you can manage containerized clusters and applications from a Git repository that: 26 27- Is the single source of truth of your system. 28- Is the single place where you operate your system. 29- Is a single resource to monitor your system. 30 31By combining GitLab, Kubernetes, and GitOps, it results in a robust infrastructure: 32 33- GitLab as the GitOps operator. 34- Kubernetes as the automation and convergence system. 35- GitLab CI/CD as the Continuous Integration and Continuous Deployment engine. 36 37Beyond that, you can use all the features offered by GitLab as 38the all-in-one DevOps platform for your product and your team. 39 40## Agent's features 41 42By using the Agent, you can: 43 44- Connect GitLab with a Kubernetes cluster behind a firewall or a 45Network Address Translation (NAT). 46- Have real-time access to API endpoints in your cluster from GitLab CI/CD. 47- Use GitOps to configure your cluster through the [Agent's repository](repository.md). 48- Perform pull-based or push-based GitOps deployments. 49- Configure [Network Security Alerts](#kubernetes-network-security-alerts) 50based on [Container Network Policies](../../application_security/policies/index.md#container-network-policy). 51- Track objects applied to your cluster through [inventory objects](../../infrastructure/clusters/deploy/inventory_object.md). 52- Use the [CI/CD Tunnel](ci_cd_tunnel.md) to access Kubernetes clusters 53from GitLab CI/CD jobs while keeping the cluster's APIs safe and unexposed 54to the internet. 55- [Deploy the GitLab Runner in a Kubernetes cluster](https://docs.gitlab.com/runner/install/kubernetes-agent.html). 56 57See the [Agent roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329) to track its development. 58 59To contribute to the Agent, see the [Agent's development documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/doc). 60 61## Agent's GitOps workflow **(PREMIUM)** 62 63The Agent uses multiple GitLab projects to provide a flexible workflow 64that can suit various needs. This diagram shows these repositories and the main 65actors involved in a deployment: 66 67```mermaid 68sequenceDiagram 69 participant D as Developer 70 participant A as Application code repository 71 participant M as Manifest repository 72 participant K as GitLab Agent 73 participant C as Agent configuration repository 74 loop Regularly 75 K-->>C: Grab the configuration 76 end 77 D->>+A: Pushing code changes 78 A->>M: Updating manifest 79 loop Regularly 80 K-->>M: Watching changes 81 M-->>K: Pulling and applying changes 82 end 83``` 84 85For more details, refer to our [architecture documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md#high-level-architecture) in the Agent project. 86 87## Install the Agent in your cluster 88 89See how to [install the Agent in your cluster](install/index.md). 90 91## GitOps deployments **(PREMIUM)** 92 93To perform GitOps deployments with the Agent, you need: 94 95- A properly-configured Kubernetes cluster where the Agent is running. 96- A [configuration repository](repository.md) that contains a 97`config.yaml` file, which tells the Agent the repositories to synchronize 98with the cluster. 99- A manifest repository that contains manifest files. Any changes to manifest files are applied to the cluster. 100 101You can use a single GitLab project or different projects for the Agent 102configuration and manifest files, as follows: 103 104- Single GitLab project (recommended): When you use a single repository to hold 105 both the manifest and the configuration files, these projects can be either 106 private or public. 107- Two GitLab projects: When you use two different GitLab projects (one for 108 manifest files and another for configuration files), the manifests project must 109 be public, while the configuration project can be either private or public. 110 111Support for separated private manifest and configuration repositories is tracked in this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). 112 113## Kubernetes Network Security Alerts **(ULTIMATE)** 114 115The GitLab Agent also provides an integration with Cilium. This integration provides a simple way to 116generate network policy-related alerts and to surface those alerts in GitLab. 117 118There are several components that work in concert for the Agent to generate the alerts: 119 120- A working Kubernetes cluster. 121- Cilium integration through either of these options: 122 - Installation through [cluster management template](../../project/clusters/protect/container_network_security/quick_start_guide.md#use-the-cluster-management-template-to-install-cilium). 123 - Enablement of [hubble-relay](https://docs.cilium.io/en/v1.8/concepts/overview/#hubble) on an 124 existing installation. 125- One or more network policies through any of these options: 126 - Use the [Container Network Policy editor](../../application_security/policies/index.md#container-network-policy-editor) to create and manage policies. 127 - Use an [AutoDevOps](../../application_security/policies/index.md#container-network-policy) configuration. 128 - Add the required labels and annotations to existing network policies. 129- A configuration repository with [Cilium configured in `config.yaml`](repository.md#surface-network-security-alerts-from-cluster-to-gitlab) 130 131The setup process follows the same [Agent's installation steps](install/index.md), 132with the following differences: 133 134- When you define a configuration repository, you must do so with [Cilium settings](repository.md#surface-network-security-alerts-from-cluster-to-gitlab). 135- You do not need to specify the `gitops` configuration section. 136 137## Remove an agent 138 1391. Get the `<cluster-agent-id>` and the `<cluster-agent-token-id>` from a query in the interactive GraphQL explorer. 140For GitLab.com, go to <https://gitlab.com/-/graphql-explorer> to open GraphQL Explorer. 141For self-managed GitLab instances, go to `https://gitlab.example.com/-/graphql-explorer`, replacing `gitlab.example.com` with your own instance's URL. 142 143 ```graphql 144 query{ 145 project(fullPath: "<full-path-to-agent-configuration-project>") { 146 clusterAgent(name: "<agent-name>") { 147 id 148 tokens { 149 edges { 150 node { 151 id 152 } 153 } 154 } 155 } 156 } 157 } 158 ``` 159 1601. Remove an Agent record with GraphQL by deleting the `clusterAgent` and the `clusterAgentToken`. 161 162 ```graphql 163 mutation deleteAgent { 164 clusterAgentDelete(input: { id: "<cluster-agent-id>" } ) { 165 errors 166 } 167 } 168 169 mutation deleteToken { 170 clusterAgentTokenDelete(input: { id: "<cluster-agent-token-id>" }) { 171 errors 172 } 173 } 174 ``` 175 1761. Verify whether the removal occurred successfully. If the output in the Pod logs includes `unauthenticated`, it means that the agent was successfully removed: 177 178 ```json 179 { 180 "level": "warn", 181 "time": "2021-04-29T23:44:07.598Z", 182 "msg": "GetConfiguration.Recv failed", 183 "error": "rpc error: code = Unauthenticated desc = unauthenticated" 184 } 185 ``` 186 1871. Delete the Agent in your cluster: 188 189 ```shell 190 kubectl delete -n gitlab-kubernetes-agent -f ./resources.yml 191 ``` 192 193## Troubleshooting 194 195If you face any issues while using the Agent, read the 196service logs with the following command: 197 198```shell 199kubectl logs -f -l=app=gitlab-kubernetes-agent -n gitlab-kubernetes-agent 200``` 201 202GitLab administrators can additionally view the [GitLab Agent Server logs](../../../administration/clusters/kas.md#troubleshooting). 203 204### Agent logs 205 206#### Transport: Error while dialing failed to WebSocket dial 207 208```json 209{ 210 "level": "warn", 211 "time": "2020-11-04T10:14:39.368Z", 212 "msg": "GetConfiguration failed", 213 "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://gitlab-kas:443/-/kubernetes-agent\\\": dial tcp: lookup gitlab-kas on 10.60.0.10:53: no such host\"" 214} 215``` 216 217This error is shown if there are some connectivity issues between the address 218specified as `kas-address`, and your Agent pod. To fix it, make sure that you 219specified the `kas-address` correctly. 220 221```json 222{ 223 "level": "error", 224 "time": "2021-06-25T21:15:45.335Z", 225 "msg": "Reverse tunnel", 226 "mod_name": "reverse_tunnel", 227 "error": "Connect(): rpc error: code = Unavailable desc = connection error: desc= \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\"" 228} 229``` 230 231This error occurs if the `kas-address` doesn't include a trailing slash. To fix it, make sure that the 232`wss` or `ws` URL ends with a trailing slash, such as `wss://GitLab.host.tld:443/-/kubernetes-agent/` 233or `ws://GitLab.host.tld:80/-/kubernetes-agent/`. 234 235#### ValidationError(Deployment.metadata) 236 237```json 238{ 239 "level": "info", 240 "time": "2020-10-30T08:56:54.329Z", 241 "msg": "Synced", 242 "project_id": "root/kas-manifest001", 243 "resource_key": "apps/Deployment/kas-test001/nginx-deployment", 244 "sync_result": "error validating data: [ValidationError(Deployment.metadata): unknown field \"replicas\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"selector\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"template\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta]" 245} 246``` 247 248This error is shown if a manifest file is malformed, and Kubernetes can't 249create specified objects. Make sure that your manifest files are valid. You 250may try using them to create objects in Kubernetes directly for more troubleshooting. 251 252#### Error while dialing failed to WebSocket dial: failed to send handshake request 253 254```json 255{ 256 "level": "warn", 257 "time": "2020-10-30T09:50:51.173Z", 258 "msg": "GetConfiguration failed", 259 "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent\\\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \\\"\\\\x00\\\\x00\\\\x06\\\\x04\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x05\\\\x00\\\\x00@\\\\x00\\\"\"" 260} 261``` 262 263This error is shown if you configured `wss` as `kas-address` on the agent side, 264but KAS on the server side is not available via `wss`. To fix it, make sure the 265same schemes are configured on both sides. 266 267It's not possible to set the `grpc` scheme due to the issue 268[It is not possible to configure KAS to work with `grpc` without directly editing GitLab KAS deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/276888). To use `grpc` while the 269issue is in progress, directly edit the deployment with the 270`kubectl edit deployment gitlab-kas` command, and change `--listen-websocket=true` to `--listen-websocket=false`. After running that command, you should be able to use 271`grpc://gitlab-kas.<YOUR-NAMESPACE>:8150`. 272 273#### Decompressor is not installed for grpc-encoding 274 275```json 276{ 277 "level": "warn", 278 "time": "2020-11-05T05:25:46.916Z", 279 "msg": "GetConfiguration.Recv failed", 280 "error": "rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"gzip\"" 281} 282``` 283 284This error is shown if the version of the agent is newer that the version of KAS. 285To fix it, make sure that both `agentk` and KAS use the same versions. 286 287#### Certificate signed by unknown authority 288 289```json 290{ 291 "level": "error", 292 "time": "2021-02-25T07:22:37.158Z", 293 "msg": "Reverse tunnel", 294 "mod_name": "reverse_tunnel", 295 "error": "Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent/\\\": x509: certificate signed by unknown authority\"" 296} 297``` 298 299This error is shown if your GitLab instance is using a certificate signed by an internal CA that 300is unknown to the agent. One approach to fixing it is to present the CA certificate file to the agent 301via a Kubernetes `configmap` and mount the file in the agent `/etc/ssl/certs` directory from where it 302will be picked up automatically. 303 304For example, if your internal CA certificate is `myCA.pem`: 305 306```plaintext 307kubectl -n gitlab-kubernetes-agent create configmap ca-pemstore --from-file=myCA.pem 308``` 309 310Then in `resources.yml`: 311 312```yaml 313 spec: 314 serviceAccountName: gitlab-kubernetes-agent 315 containers: 316 - name: agent 317 image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>" 318 args: 319 - --token-file=/config/token 320 - --kas-address 321 - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com. 322 # - wss://gitlab.host.tld:443/-/kubernetes-agent/ 323 # - wss://kas.gitlab.com # for GitLab.com users, use this KAS. 324 # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker. 325 volumeMounts: 326 - name: token-volume 327 mountPath: /config 328 - name: ca-pemstore-volume 329 mountPath: /etc/ssl/certs/myCA.pem 330 subPath: myCA.pem 331 volumes: 332 - name: token-volume 333 secret: 334 secretName: gitlab-kubernetes-agent-token 335 - name: ca-pemstore-volume 336 configMap: 337 name: ca-pemstore 338 items: 339 - key: myCA.pem 340 path: myCA.pem 341``` 342 343Alternatively, you can mount the certificate file at a different location and include it using the 344`--ca-cert-file` agent parameter: 345 346```yaml 347 containers: 348 - name: agent 349 image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>" 350 args: 351 - --ca-cert-file=/tmp/myCA.pem 352 - --token-file=/config/token 353 - --kas-address 354 - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com. 355 # - wss://gitlab.host.tld:443/-/kubernetes-agent/ 356 # - wss://kas.gitlab.com # for GitLab.com users, use this KAS. 357 # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker. 358 volumeMounts: 359 - name: token-volume 360 mountPath: /config 361 - name: ca-pemstore-volume 362 mountPath: /tmp/myCA.pem 363 subPath: myCA.pem 364``` 365