1# Loading benchmarks 2 3[TOC] 4 5## Overview 6 7The Telemetry loading benchmarks measure Chrome's loading performance under 8different network and caching conditions. 9 10There are currently three loading benchmarks: 11 12- **`loading.desktop`**: A desktop-only benchmark in which each test case 13 measures performance of loading a real world website (e.g: facebook, cnn, 14 alibaba..). 15- **`loading.mobile`**: A mobile-only benchmark that parallels `loading.desktop` 16- **`loading.cluster_telemetry`**: A cluster Telemetry benchmark that uses the 17corpus of top 10 thousands URLs from Alexa. Unlike the other two loading 18benchmarks which are run continuously on the perf waterfall, this benchmark is 19triggered on-demand only. 20 21## Running the tests remotely 22 23If you're just trying to gauge whether your change has caused a loading 24regression, you can run the loading benchmarks through perf try job or through 25Cluster Telemetry service. You can specify a Gerrit patch for them and compare 26the results with and without the patch applied. 27 28### Using Perf Try Bots 29[Perf Try Bots](https://chromium.googlesource.com/chromium/src/+/master/docs/speed/perf_trybots.md) 30lets you run only single story of `loading.desktop` or `loading.mobile`, because 31an entire loading benchmark is too large to run as a perf try job. You should 32only set the `Story` field, and leave the `Story Tags` field blank. Invalid 33combinations of `Story` and `Story Tags` will result in "zero Histograms" result 34page. 35 36See 37[Understanding the loading test cases](#understanding-the-loading-test-cases) 38for story name conventions. 39 40### Using Cluster Telemetry Service 41You can run `loading.cluster_telemetry` (top 10k pages) through 42[Cluster Telemetry service](https://ct.skia.org/) (Cluster Telemetry is for 43Googler only). 44 45## Running the tests locally 46 47For more in-depth analysis and shorter cycle times, it can be helpful to run the tests locally. 48 49First, [prepare your test device for 50Telemetry](https://chromium.googlesource.com/chromium/src/+/master/docs/speed/benchmark/telemetry_device_setup.md). 51 52Once you've done this, you can start the Telemetry benchmark with: 53 54``` 55./tools/perf/run_benchmark <benchmark_name> --browser=<browser> 56``` 57 58where `benchmark_name` can be `loading.desktop` or `loading.mobile`. 59 60## Understanding the loading test cases 61 62The loading test cases are divided into groups based on their network traffic 63settings and cache conditions. 64 65All available traffic settings can be found in [traffic_setting.py](https://chromium.googlesource.com/catapult/+/master/telemetry/telemetry/page/traffic_setting.py) 66 67All available caching conditions can be found in [cache_temperature.py](https://chromium.googlesource.com/catapult/+/master/telemetry/telemetry/page/cache_temperature.py) 68 69Test cases of `loading.desktop` and `loading.mobile` are named with their 70corresponding settings. For example, `DevOpera_cold_3g` test case loads 71`https://dev.opera.com/` with cold cache and 3G network setting. 72 73In additions, the pages are also tagged with labels describing their content. 74e.g: 'global', 'pwa',... 75 76To run only pages of one tags, add `--story-tag-filter=<tag name>` flag to the 77run benchmark command. 78 79## Understanding the loading metrics 80The benchmark output several different loading metrics. The keys one are: 81 * [Time To First Contentful Paint](https://docs.google.com/document/d/1kKGZO3qlBBVOSZTf-T8BOMETzk3bY15SC-jsMJWv4IE/edit#heading=h.27igk2kctj7o) 82 * [Time To First Meaningful Paint](https://docs.google.com/document/d/1BR94tJdZLsin5poeet0XoTW60M0SjvOJQttKT-JK8HI/edit) 83 * [Time to First CPU 84 Idle](https://docs.google.com/document/d/12UHgAW2r7nWo3R6FBerpYuz9EVOdG1OpPm8YmY4yD0c/edit#) 85 86Besides those key metrics, there are also breakdown metrics that are meant to 87to make debugging regressions simpler. These metrics are updated often, for most 88up to date information, you can email speed-metrics-dev@chromium.org 89or chrome-speed-metrics@google.com (Googlers only). 90 91## Adding new loading test cases 92New test cases can be added by modifying 93[loading_desktop.py](https://chromium.googlesource.com/chromium/src/+/master/tools/perf/page_sets/loading_desktop.py) 94or [loading_mobile.py](https://chromium.googlesource.com/chromium/src/+/master/tools/perf/page_sets/loading_mobile.py) page sets. 95 96For example, to add a new case of loading 97`https://en.wikipedia.org/wiki/Cats_and_the_Internet` on 2G and 3G networks with 98warm cache to `news` group to `loading.desktop` benchmark, you would write: 99 100``` 101self.AddStories( 102 tags=['news'], 103 urls=[('https://en.wikipedia.org/wiki/Cats_and_the_Internet', 'wiki_cats')], 104 cache_temperatures=[cache_temperature_module.WARM], 105 traffic_settings=[traffic_setting_module.2G, traffic_setting_module.3G]) 106``` 107 108After adding the new page, record it and upload the page archive to cloud 109storage with: 110 111``` 112$ ./tools/perf/record_wpr loading_desktop --browser=system \ 113 --story-filter=wiki_cats --upload 114``` 115 116If the extra story was added to `loading.mobile`, replace `loading_desktop` in 117the command above with `loading_mobile`. 118