1--- 2stage: Monitor 3group: Monitor 4info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments 5--- 6 7# Working with Prometheus Metrics **(FREE)** 8 9## Adding to the library 10 11We strive to support the 2-4 most important metrics for each common system service that supports Prometheus. If you are looking for support for a particular exporter which has not yet been added to the library, additions can be made [to the `common_metrics.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/prometheus/common_metrics.yml) file. 12 13### Query identifier 14 15The requirement for adding a new metric is to make each query to have an unique identifier which is used to update the metric later when changed: 16 17```yaml 18- group: Response metrics (NGINX Ingress) 19 metrics: 20 - title: "Throughput" 21 y_axis: 22 name: "Requests / Sec" 23 format: "number" 24 precision: 2 25 queries: 26 - id: response_metrics_nginx_ingress_throughput_status_code 27 query_range: 'sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code)' 28 unit: req / sec 29 label: Status Code 30``` 31 32### Update existing metrics 33 34After you add or change an existing common metric, you must [re-run the import script](../administration/raketasks/maintenance.md#import-common-metrics) that queries and updates all existing metrics. 35 36Or, you can create a database migration: 37 38```ruby 39class ImportCommonMetrics < Gitlab::Database::Migration[1.0] 40 def up 41 ::Gitlab::DatabaseImporters::CommonMetrics::Importer.new.execute 42 end 43 44 def down 45 # no-op 46 end 47end 48``` 49 50If a query metric (which is identified by `id:`) is removed, it isn't removed from database by default. 51You might want to add additional database migration that makes a decision what to do with removed one. 52For example: you might be interested in migrating all dependent data to a different metric. 53 54## GitLab Prometheus metrics 55 56GitLab provides [Prometheus metrics](../administration/monitoring/prometheus/gitlab_metrics.md) 57to monitor itself. 58 59### Adding a new metric 60 61This section describes how to add new metrics for self-monitoring 62([example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15440)). 63 641. Select the [type of metric](https://gitlab.com/gitlab-org/prometheus-client-mmap#metrics): 65 66 - `Gitlab::Metrics.counter` 67 - `Gitlab::Metrics.gauge` 68 - `Gitlab::Metrics.histogram` 69 - `Gitlab::Metrics.summary` 70 711. Select the appropriate name for your metric. Refer to the guidelines 72 for [Prometheus metric names](https://prometheus.io/docs/practices/naming/#metric-names). 731. Update the list of [GitLab Prometheus metrics](../administration/monitoring/prometheus/gitlab_metrics.md). 741. Carefully choose what labels you want to add to your metric. Values with high cardinality, 75like `project_path`, or `project_id` are strongly discouraged because they can affect our services 76availability due to the fact that each set of labels is exposed as a new entry in the `/metrics` endpoint. 77For example, a histogram with 10 buckets and a label with 100 values would generate 1000 78entries in the export endpoint. 791. Trigger the relevant page or code that records the new metric. 801. Check that the new metric appears at `/-/metrics`. 81