1--- 2stage: Growth 3group: Product Intelligence 4info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments 5--- 6 7# Service Ping Guide **(FREE SELF)** 8 9> Introduced in GitLab Ultimate 11.2, more statistics. 10 11Service Ping is a GitLab process that collects and sends a weekly payload to GitLab. 12The payload provides important high-level data that helps our product, support, 13and sales teams understand how GitLab is used. The data helps to: 14 15- Compare counts month over month (or week over week) to get a rough sense for how an instance uses 16 different product features. 17- Collect other facts that help us classify and understand GitLab installations. 18- Calculate our stage monthly active users (SMAU), which helps to measure the success of our stages 19 and features. 20 21Service Ping information is not anonymous. It's linked to the instance's hostname, but does 22not contain project names, usernames, or any other specific data. 23 24Sending a Service Ping payload is optional and you can [disable](#disable-service-ping) it on any 25self-managed instance. When Service Ping is enabled, GitLab gathers data from the other instances 26and can show your instance's usage statistics to your users. 27 28## Service Ping terminology 29 30We use the following terminology to describe the Service Ping components: 31 32- **Service Ping**: the process that collects and generates a JSON payload. 33- **Service Data**: the contents of the Service Ping JSON payload. This includes metrics. 34- **Metrics**: primarily made up of row counts for different tables in an instance's database. Each 35 metric has a corresponding [metric definition](metrics_dictionary.md#metrics-definition-and-validation) 36 in a YAML file. 37- **MAU**: monthly active users. 38- **WAU**: weekly active users. 39 40### Why enable Service Ping? 41 42The main purpose of Service Ping is to build a better GitLab. We collect data about how GitLab is used 43to understand feature or stage adoption and usage. This data gives an insight into how GitLab adds 44value and helps our team understand the reasons why people use GitLab, and with this knowledge we're able to 45make better product decisions. 46 47There are several other benefits to enabling Service Ping: 48 49- As a benefit of having Service Ping active, GitLab lets you analyze the users' activities over time of your GitLab installation. 50- As a benefit of having Service Ping active, GitLab provides you with [DevOps Score](../../user/admin_area/analytics/dev_ops_report.md#devops-score), which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. 51- You get better, more proactive support (assuming that our TAMs and support organization used the data to deliver more value). 52- You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? 53- You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. 54- Service Ping is enabled by default. To disable it, see [Disable Service Ping](#disable-service-ping). 55- When Service Ping is enabled, you have the option to participate in our [Registration Features Program](#registration-features-program) and receive free paid features. 56 57### Limitations 58 59- Service Ping does not track frontend events things like page views, link clicks, or user sessions. 60- Service Ping focuses only on aggregated backend events. 61 62Because of these limitations we recommend you: 63 64- Instrument your products with Snowplow for more detailed analytics on GitLab.com. 65- Use Service Ping to track aggregated backend events on self-managed instances. 66 67### Registration Features Program 68 69> Introduced in GitLab 14.1. 70 71In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through Service Ping. Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. 72 73#### Features available in 14.1 and later 74 751. [Email from GitLab](../../tools/email.md). 76 77#### Features available in 14.4 and later 78 791. [Repository size limit](../../user/admin_area/settings/account_and_limit_settings.md#repository-size-limit). 801. [Restrict group access by IP address](../../user/group/index.md#restrict-group-access-by-ip-address). 81 82NOTE: 83Registration is not yet required for participation, but will be added in a future milestone. 84 85#### Enable Registration Features 86 871. Sign in as a user with the [Administrator](../../user/permissions.md) role. 881. On the top bar, select **Menu > Admin**. 891. On the left sidebar, select **Settings > Metrics and profiling**. 901. Expand the **Usage statistics** section. 911. If not enabled, select the **Enable Service Ping** checkbox. 921. Select the **Enable Registration Features** checkbox. 931. Select **Save changes**. 94 95## View the Service Ping payload **(FREE SELF)** 96 97You can view the exact JSON payload sent to GitLab Inc. in the Admin Area. To view the payload: 98 991. Sign in as a user with the [Administrator](../../user/permissions.md) role. 1001. On the top bar, select **Menu > Admin**. 1011. On the left sidebar, select **Settings > Metrics and profiling**. 1021. Expand the **Usage statistics** section. 1031. Select **Preview payload**. 104 105For an example payload, see [Example Service Ping payload](#example-service-ping-payload). 106 107## Disable Service Ping **(FREE SELF)** 108 109NOTE: 110The method to disable Service Ping in the GitLab configuration file does not work in 111GitLab versions 9.3 to 13.12.3. See the [troubleshooting section](#cannot-disable-service-ping-using-the-configuration-file) 112on how to disable it. 113 114You can disable Service Ping either using the GitLab UI, or editing the GitLab 115configuration file. 116 117### Disable Service Ping using the UI 118 119To disable Service Ping in the GitLab UI: 120 1211. Sign in as a user with the [Administrator](../../user/permissions.md) role. 1221. On the top bar, select **Menu > Admin**. 1231. On the left sidebar, select **Settings > Metrics and profiling**. 1241. Expand the **Usage statistics** section. 1251. Clear the **Enable Service Ping** checkbox. 1261. Select **Save changes**. 127 128### Disable Service Ping using the configuration file 129 130To disable Service Ping and prevent it from being configured in the future through 131the Admin Area: 132 133**For installations using the Linux package:** 134 1351. Edit `/etc/gitlab/gitlab.rb`: 136 137 ```ruby 138 gitlab_rails['usage_ping_enabled'] = false 139 ``` 140 1411. Reconfigure GitLab: 142 143 ```shell 144 sudo gitlab-ctl reconfigure 145 ``` 146 147**For installations from source:** 148 1491. Edit `/home/git/gitlab/config/gitlab.yml`: 150 151 ```yaml 152 production: &base 153 # ... 154 gitlab: 155 # ... 156 usage_ping_enabled: false 157 ``` 158 1591. Restart GitLab: 160 161 ```shell 162 sudo service gitlab restart 163 ``` 164 165## Service Ping request flow 166 167The following example shows a basic request/response flow between a GitLab instance, the Versions Application, the License Application, Salesforce, the GitLab S3 Bucket, the GitLab Snowflake Data Warehouse, and Sisense: 168 169```mermaid 170sequenceDiagram 171 participant GitLab Instance 172 participant Versions Application 173 participant Licenses Application 174 participant Salesforce 175 participant S3 Bucket 176 participant Snowflake DW 177 participant Sisense Dashboards 178 GitLab Instance->>Versions Application: Send Service Ping 179 loop Process usage data 180 Versions Application->>Versions Application: Parse usage data 181 Versions Application->>Versions Application: Write to database 182 Versions Application->>Versions Application: Update license ping time 183 end 184 loop Process data for Salesforce 185 Versions Application-xLicenses Application: Request Zuora subscription id 186 Licenses Application-xVersions Application: Zuora subscription id 187 Versions Application-xSalesforce: Request Zuora account id by Zuora subscription id 188 Salesforce-xVersions Application: Zuora account id 189 Versions Application-xSalesforce: Usage data for the Zuora account 190 end 191 Versions Application->>S3 Bucket: Export Versions database 192 S3 Bucket->>Snowflake DW: Import data 193 Snowflake DW->>Snowflake DW: Transform data using dbt 194 Snowflake DW->>Sisense Dashboards: Data available for querying 195 Versions Application->>GitLab Instance: DevOps Score (Conversational Development Index) 196``` 197 198## How Service Ping works 199 2001. The Service Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_service_ping_worker.rb#L24) is set in Sidekiq to run weekly. 2011. When the cron job runs, it calls [`Gitlab::UsageData.to_json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L49). 2021. `Gitlab::UsageData.to_json` [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) to ~400+ other counter method calls. 2031. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L68) into a single JSON payload in `Gitlab::UsageData.to_json`. 2041. The JSON payload is then [posted to the Versions application](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L20) 205 If a firewall exception is needed, the required URL depends on several things. If 206 the hostname is `version.gitlab.com`, the protocol is `TCP`, and the port number is `443`, 207 the required URL is <https://version.gitlab.com/>. 208 209### On a Geo secondary site 210 211We also collect metrics specific to [Geo](../../administration/geo/index.md) secondary sites to send with Service Ping. 212 2131. The [Geo secondary service ping cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/geo/secondary_usage_data_cron_worker.rb) is set in Sidekiq to run weekly. 2141. When the cron job runs, it calls [`SecondaryUsageData.update_metrics!`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo/secondary_usage_data.rb#L33). This collects the relevant metrics from Prometheus and stores the data in the Geo secondary tracking database for transmission to the primary site during a [Geo node status update](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo_node_status.rb#L105). 2151. Geo node status data is sent with the JSON payload in the process described above. The following is an example of the payload where each object in the array represents a Geo node: 216 217 ```json 218 [ 219 { 220 "repository_verification_enabled"=>true, 221 "repositories_replication_enabled"=>true, 222 "repositories_synced_count"=>24, 223 "repositories_failed_count"=>0, 224 "git_fetch_event_count_weekly"=>nil, 225 "git_push_event_count_weekly"=>nil, 226 ... other geo node status fields 227 } 228 ] 229 ``` 230 231## Implementing Service Ping 232 233See the [implement Service Ping](implement.md) guide. 234 235## Example Service Ping payload 236 237The following is example content of the Service Ping payload. 238 239```json 240{ 241 "uuid": "0000000-0000-0000-0000-000000000000", 242 "hostname": "example.com", 243 "version": "12.10.0-pre", 244 "installation_type": "omnibus-gitlab", 245 "active_user_count": 999, 246 "recorded_at": "2020-04-17T07:43:54.162+00:00", 247 "edition": "EEU", 248 "license_md5": "00000000000000000000000000000000", 249 "license_id": null, 250 "historical_max_users": 999, 251 "licensee": { 252 "Name": "ABC, Inc.", 253 "Email": "email@example.com", 254 "Company": "ABC, Inc." 255 }, 256 "license_user_count": 999, 257 "license_starts_at": "2020-01-01", 258 "license_expires_at": "2021-01-01", 259 "license_plan": "ultimate", 260 "license_add_ons": { 261 }, 262 "license_trial": false, 263 "counts": { 264 "assignee_lists": 999, 265 "boards": 999, 266 "ci_builds": 999, 267 ... 268 }, 269 "container_registry_enabled": true, 270 "dependency_proxy_enabled": false, 271 "gitlab_shared_runners_enabled": true, 272 "gravatar_enabled": true, 273 "influxdb_metrics_enabled": true, 274 "ldap_enabled": false, 275 "mattermost_enabled": false, 276 "omniauth_enabled": true, 277 "prometheus_enabled": false, 278 "prometheus_metrics_enabled": false, 279 "reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com", 280 "signup_enabled": true, 281 "web_ide_clientside_preview_enabled": true, 282 "projects_with_expiration_policy_disabled": 999, 283 "projects_with_expiration_policy_enabled": 999, 284 ... 285 "elasticsearch_enabled": true, 286 "license_trial_ends_on": null, 287 "geo_enabled": false, 288 "git": { 289 "version": { 290 "major": 2, 291 "minor": 26, 292 "patch": 1 293 } 294 }, 295 "gitaly": { 296 "version": "12.10.0-rc1-93-g40980d40", 297 "servers": 56, 298 "clusters": 14, 299 "filesystems": [ 300 "EXT_2_3_4" 301 ] 302 }, 303 "gitlab_pages": { 304 "enabled": true, 305 "version": "1.17.0" 306 }, 307 "container_registry_server": { 308 "vendor": "gitlab", 309 "version": "2.9.1-gitlab" 310 }, 311 "database": { 312 "adapter": "postgresql", 313 "version": "9.6.15", 314 "pg_system_id": 6842684531675334351 315 }, 316 "analytics_unique_visits": { 317 "g_analytics_contribution": 999, 318 ... 319 }, 320 "usage_activity_by_stage": { 321 "configure": { 322 "project_clusters_enabled": 999, 323 ... 324 }, 325 "create": { 326 "merge_requests": 999, 327 ... 328 }, 329 "manage": { 330 "events": 999, 331 ... 332 }, 333 "monitor": { 334 "clusters": 999, 335 ... 336 }, 337 "package": { 338 "projects_with_packages": 999 339 }, 340 "plan": { 341 "issues": 999, 342 ... 343 }, 344 "release": { 345 "deployments": 999, 346 ... 347 }, 348 "secure": { 349 "user_container_scanning_jobs": 999, 350 ... 351 }, 352 "verify": { 353 "ci_builds": 999, 354 ... 355 } 356 }, 357 "usage_activity_by_stage_monthly": { 358 "configure": { 359 "project_clusters_enabled": 999, 360 ... 361 }, 362 "create": { 363 "merge_requests": 999, 364 ... 365 }, 366 "manage": { 367 "events": 999, 368 ... 369 }, 370 "monitor": { 371 "clusters": 999, 372 ... 373 }, 374 "package": { 375 "projects_with_packages": 999 376 }, 377 "plan": { 378 "issues": 999, 379 ... 380 }, 381 "release": { 382 "deployments": 999, 383 ... 384 }, 385 "secure": { 386 "user_container_scanning_jobs": 999, 387 ... 388 }, 389 "verify": { 390 "ci_builds": 999, 391 ... 392 } 393 }, 394 "topology": { 395 "duration_s": 0.013836685999194742, 396 "application_requests_per_hour": 4224, 397 "query_apdex_weekly_average": 0.996, 398 "failures": [], 399 "nodes": [ 400 { 401 "node_memory_total_bytes": 33269903360, 402 "node_memory_utilization": 0.35, 403 "node_cpus": 16, 404 "node_cpu_utilization": 0.2, 405 "node_uname_info": { 406 "machine": "x86_64", 407 "sysname": "Linux", 408 "release": "4.19.76-linuxkit" 409 }, 410 "node_services": [ 411 { 412 "name": "web", 413 "process_count": 16, 414 "process_memory_pss": 233349888, 415 "process_memory_rss": 788220927, 416 "process_memory_uss": 195295487, 417 "server": "puma" 418 }, 419 { 420 "name": "sidekiq", 421 "process_count": 1, 422 "process_memory_pss": 734080000, 423 "process_memory_rss": 750051328, 424 "process_memory_uss": 731533312 425 }, 426 ... 427 ], 428 ... 429 }, 430 ... 431 ] 432 } 433} 434``` 435 436## Notable changes 437 438In GitLab 13.5, `pg_system_id` was added to send the [PostgreSQL system identifier](https://www.2ndquadrant.com/en/blog/support-for-postgresqls-system-identifier-in-barman/). 439 440## Export Service Ping SQL queries and definitions 441 442Two Rake tasks exist to export Service Ping definitions. 443 444- The Rake tasks export the raw SQL queries for `count`, `distinct_count`, `sum`. 445- The Rake tasks export the Redis counter class or the line of the Redis block for `redis_usage_data`. 446- The Rake tasks calculate the `alt_usage_data` metrics. 447 448In the home directory of your local GitLab installation run the following Rake tasks for the YAML and JSON versions respectively: 449 450```shell 451# for YAML export 452bin/rake gitlab:usage_data:dump_sql_in_yaml 453 454# for JSON export 455bin/rake gitlab:usage_data:dump_sql_in_json 456 457# You may pipe the output into a file 458bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml 459``` 460 461## Generate Service Ping 462 463To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a detached screen session on a remote server. 464 465### Triggering 466 467#### Trigger Service Ping with Teleport 468 4691. Request temporary [access](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to the required environment. 4701. After your approval is issued, [access the Rails console](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#access-approval). 4711. Run `ServicePing::SubmitService.new.execute`. 472 473#### Trigger Service Ping with a detached screen session 474 4751. Connect to bastion with agent forwarding: 476 477 ```shell 478 ssh -A lb-bastion.gprd.gitlab.com 479 ``` 480 4811. Create named screen: 482 483 ```shell 484 screen -S <username>_usage_ping_<date> 485 ``` 486 4871. Connect to console host: 488 489 ```shell 490 ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal 491 ``` 492 4931. Run: 494 495 ```shell 496 ServicePing::SubmitService.new.execute 497 ``` 498 4991. To detach from screen, press `ctrl + A`, `ctrl + D`. 5001. Exit from bastion: 501 502 ```shell 503 exit 504 ``` 505 506### Verification (After approx 30 hours) 507 508#### Verify with a detached screen session 509 5101. Follow [the steps](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to request a new access to the required environment and connect to the Rails console 5111. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` 5121. Check the when the payload was sent: `RawUsageData.last.sent_at` 513 514#### Verify using detached screen session 515 5161. Reconnect to bastion: 517 518 ```shell 519 ssh -A lb-bastion.gprd.gitlab.com 520 ``` 521 5221. Find your screen session: 523 524 ```shell 525 screen -ls 526 ``` 527 5281. Attach to your screen session: 529 530 ```shell 531 screen -x 14226.mwawrzyniak_usage_ping_2021_01_22 532 ``` 533 5341. Check the last payload in `raw_usage_data` table: 535 536 ```shell 537 RawUsageData.last.payload 538 ``` 539 5401. Check the when the payload was sent: 541 542 ```shell 543 RawUsageData.last.sent_at 544 ``` 545 546### Skip database write operations 547 548To skip database write operations, DevOps report creation, and storage of usage data payload, pass an optional argument: 549 550```shell 551skip_db_write: 552ServicePing::SubmitService.new(skip_db_write: true).execute 553``` 554 555## Troubleshooting 556 557### Cannot disable Service Ping using the configuration file 558 559The method to disable Service Ping using the GitLab configuration file does not work in 560GitLab versions 9.3.0 to 13.12.3. To disable it, you must use the Admin Area in 561the GitLab UI instead. For more information, see 562[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/333269). 563 564GitLab functionality and application settings cannot override or circumvent 565restrictions at the network layer. If Service Ping is blocked by your firewall, 566you are not impacted by this bug. 567 568#### Check if you are affected 569 570You can check if you were affected by this bug by using the Admin Area or by 571checking the configuration file of your GitLab instance: 572 573- Using the Admin Area: 574 575 1. On the top bar, select **Menu > Admin**. 576 1. On the left sidebar, select **Settings > Metrics and profiling**. 577 1. Expand **Usage Statistics**. 578 1. Are you able to check or uncheck the checkbox to disable Service Ping? 579 580 - If _yes_, your GitLab instance is not affected by this bug. 581 - If you can't check or uncheck the checkbox, you are affected by this bug. 582 See the steps on [how to fix this](#how-to-fix-the-cannot-disable-service-ping-bug). 583 584- Checking your GitLab instance configuration file: 585 586 To check whether you're impacted by this bug, check your instance configuration 587 settings. The configuration file in which Service Ping can be disabled depends 588 on your installation and deployment method, but is typically one of the following: 589 590 - `/etc/gitlab/gitlab.rb` for Omnibus GitLab Linux Package and Docker. 591 - `charts.yaml` for GitLab Helm and cloud-native Kubernetes deployments. 592 - `gitlab.yml` for GitLab installations from source. 593 594 To check the relevant configuration file for strings that indicate whether 595 Service Ping is disabled, you can use `grep`: 596 597 ```shell 598 # Linux package 599 grep "usage_ping_enabled'\] = false" /etc/gitlab/gitlab.rb 600 601 # Kubernetes charts 602 grep "enableUsagePing: false" values.yaml 603 604 # From source 605 grep "usage_ping_enabled'\] = false" gitlab/config.yml 606 ``` 607 608 If you see any output after running the relevant command, your GitLab instance 609 may be affected by the bug. Otherwise, your instance is not affected. 610 611#### How to fix the "Cannot disable Service Ping" bug 612 613To work around this bug, you have two options: 614 615- [Update](../../update/index.md) to GitLab 13.12.4 or newer to fix this bug. 616- If you can't update to GitLab 13.12.4 or newer, enable Service Ping in the 617 configuration file, then disable Service Ping in the UI. For example, if you're 618 using the Linux package: 619 620 1. Edit `/etc/gitlab/gitlab.rb`: 621 622 ```ruby 623 gitlab_rails['usage_ping_enabled'] = true 624 ``` 625 626 1. Reconfigure GitLab: 627 628 ```shell 629 sudo gitlab-ctl reconfigure 630 ``` 631 632 1. In GitLab, on the top bar, select **Menu > Admin**. 633 1. On the left sidebar, select **Settings > Metrics and profiling**. 634 1. Expand **Usage Statistics**. 635 1. Clear the **Enable Service Ping** checkbox. 636 1. Select **Save Changes**. 637 638## Related topics 639 640- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) 641- [Snowplow Guide](../snowplow/index.md) 642- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) 643- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) 644- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) 645- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) 646