1[![Rust][Rust Badge]][Rust CI Link]
2[![NotImplemented Counter][NotImplemented Badge]][NotImplemented Search]
3[![Fuzzbug days since][Fuzzbug Days Badge]][Fuzzbugs]
4[![Fuzzbug open][Fuzzbug Open Badge]][Open Fuzzbugs]
5
6# Metrics
7
8This is the metrics directory. It follows the evolution of the repository separately from the
9repostory. You can find the actual metrics in the
10[`ci-results`](https://github.com/mozilla-spidermonkey/jsparagus/tree/ci-results) branch of the jsparagus project. This branch is automatically generated using the `create-ci-branch.sh` script found in this directory. If there are issues with your fork, you can remove the `ci-results` branch, and the ci will automatically rerun the `create-ci-branch` script to reset it. Do not push manula data to this repository, it will be lost.
11
12If you find that the `ci-results` branch has disappeared or been corrupted somehow, you can reset it by deleting it and recreating it.
13
14```
15git branch -D ci-results
16cd .metrics
17./create-ci-branch.sh
18```
19
20The `create-ci-branch.sh` file creates the branch, prepares it, and populates it with data from the past.
21
22## Making your own metrics
23Make sure you do not use data that can not be automatically recovered. We cannot rely on the `ci-results` branch always being present, therefore anything that you write must be recoverable on its own, either by relying on external APIs or through some other mechanism.
24
25Please update this README if you make any changes.
26
27## Types of CI Actions
28These actions are all found in the `.github/workflows` directory
29
301) `Rust.yml` - Run on Pull Request
31* runs every time there is a push to master, use for any metrics that are development related. Examples include linting, testing, etc.
322) `ci-push.yml` - Run on Push to `master`
33* runs on self contained metrics. An example is the number of `NotImplemented` errors in the codebase. This does not depend on anything external
343) `ci-daily.yml` - Run Daily
35* a cron task that runs daily. Useful for metrics that need daily updates
364) `ci-issue.yml` - Run on issue open
37* runs each time an issue is opened. Good for tracking types of issues.
38
39
40## Types of data
41
42These are the types of data that this metrics folder tracks.
43
441) Rust Passing
45    * Ensures our internal tests are passing
46    * Updates on every pull request to master. See [this
47        action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/rust.yml)
48
492) NotImplemented Count
50    * counts number of NotImplemented errors in the codebase. This should slowly rundown to zero
51    * Updates on every push to master. See [this
52        action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-push.yml)
53
543) Days Since last Fuzzbug
55    * tracks the last fuzzbug we saw, if it does not exist, return ∞, otherwise return the last date regardless of state.
56    * Updates daily, regardless of push. See [this
57        action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-daily.yml)
58
594) Fuzzbug open count
60    * tracks the number of open fuzzbugs
61    * Updates on issue open. See [this action](https://github.com/mozilla-spidermonkey/jsparagus/.github/workflows/ci-issues.yml)
62
635) Percentage of tests passing with SmooshMonkey
64    * TODO: tracks the number of tests passing without fallback. We should use the try api for this.
65    * Updates daily, regardless of push. See [this
66        action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-daily.yml)
67
686) Percentage of JS compilable with SmooshMonkey
69    * TODO: see comment about writing bytes to a file in [this repo](https://github.com/nbp/seqrec)
70    * implementation is dependant on how we get the data. We need a robust solution for importing this data.
71
72[Rust Badge]: https://github.com/mozilla-spidermonkey/jsparagus/workflows/Rust/badge.svg
73[Rust CI Link]: https://github.com/mozilla-spidermonkey/jsparagus/actions?query=branch%3Amaster
74[NotImplemented Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fnot-implemented.json
75[NotImplemented Search]: https://github.com/mozilla-spidermonkey/jsparagus/search?q=notimplemented&unscoped_q=notimplemented
76[Fuzzbug days Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fsince-last-fuzzbug.json
77[Fuzzbug Open Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fopen-fuzzbug.json
78[Fuzzbugs]: https://github.com/mozilla-spidermonkey/jsparagus/issues?utf8=%E2%9C%93&q=label%3AlibFuzzer+
79[Open Fuzzbugs]: https://github.com/mozilla-spidermonkey/jsparagus/labels/libFuzzer
80