README.md
1# Libc mem* benchmarks
2
3This framework has been designed to evaluate and compare relative performance of memory function implementations on a particular machine.
4
5It relies on two tools:
6 - `libc-benchmark-main` a C++ benchmarking utility producing raw measurements,
7 - `libc-benchmark-analysis.py3` a tool to process the measurements into reports.
8
9## Benchmarking tool
10
11### Setup
12
13```shell
14cd llvm-project
15cmake -B/tmp/build -Sllvm -DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;libc' -DCMAKE_BUILD_TYPE=Release -G Ninja
16ninja -C /tmp/build libc-benchmark-main
17```
18
19> Note: The machine should run in `performance` mode. This is achieved by running:
20```shell
21cpupower frequency-set --governor performance
22```
23
24### Usage
25
26`libc-benchmark-main` can run in two modes:
27 - **stochastic mode** returns the average time per call for a particular size distribution,
28 - **sweep mode** returns the average time per size over a range of sizes.
29
30The tool requires the following flags to be set:
31 - `--study-name`: a name to identify a run and provide label during analysis,
32 - `--function`: the name of the function under test.
33
34It also provides optional flags:
35 - `--num-trials`: repeats the benchmark more times, the analysis tool can take this into account and give confidence intervals.
36 - `--output`: specifies a file to write the report - or standard output if not set.
37 - `--aligned-access`: The alignment to use when accessing the buffers, default is unaligned, 0 disables address randomization.
38
39> Note: `--function` takes a generic function name like `memcpy` or `memset` but the actual function being tested is the llvm-libc implementation (e.g. `__llvm_libc::memcpy`).
40
41### Stochastic mode
42
43This is the preferred mode to use. The function parameters are randomized and the branch predictor is less likely to kick in.
44
45```shell
46/tmp/build/bin/libc-benchmark-main \
47 --study-name="new memcpy" \
48 --function=memcpy \
49 --size-distribution-name="memcpy Google A" \
50 --num-trials=30 \
51 --output=/tmp/benchmark_result.json
52```
53
54The `--size-distribution-name` flag is mandatory and points to one of the [predefined distribution](MemorySizeDistributions.h).
55
56> Note: These distributions are gathered from several important binaries at Google (servers, databases, realtime and batch jobs) and reflect the importance of focusing on small sizes.
57
58Using a profiler to observe size distributions for calls into libc functions, it
59was found most operations act on a small number of bytes.
60
61Function | % of calls with size ≤ 128 | % of calls with size ≤ 1024
62------------------ | --------------------------: | ---------------------------:
63memcpy | 96% | 99%
64memset | 91% | 99.9%
65memcmp<sup>1</sup> | 99.5% | ~100%
66
67_<sup>1</sup> - The size refers to the size of the buffers to compare and not
68the number of bytes until the first difference._
69
70### Sweep mode
71
72This mode is used to measure call latency per size for a certain range of sizes. Because it exercises the same size over and over again the branch predictor can kick in. It can still be useful to compare strength and weaknesses of particular implementations.
73
74```shell
75/tmp/build/bin/libc-benchmark-main \
76 --study-name="new memcpy" \
77 --function=memcpy \
78 --sweep-mode \
79 --sweep-max-size=128 \
80 --output=/tmp/benchmark_result.json
81```
82
83## Analysis tool
84
85### Setup
86
87Make sure to have `matplotlib`, `pandas` and `seaborn` setup correctly:
88
89```shell
90apt-get install python3-pip
91pip3 install matplotlib pandas seaborn
92```
93You may need `python3-gtk` or similar package to display the graphs.
94
95### Usage
96
97```shell
98python3 libc/benchmarks/libc-benchmark-analysis.py3 /tmp/benchmark_result.json ...
99```
100
101When used with __multiple trials Sweep Mode data__ the tool displays the 95% confidence interval.
102
103When providing with multiple reports at the same time, all the graphs from the same machine are displayed side by side to allow for comparison.
104
105The Y-axis unit can be changed via the `--mode` flag:
106 - `time` displays the measured time (this is the default),
107 - `cycles` displays the number of cycles computed from the cpu frequency,
108 - `bytespercycle` displays the number of bytes per cycle (for `Sweep Mode` reports only).
109
110## Under the hood
111
112 To learn more about the design decisions behind the benchmarking framework,
113 have a look at the [RATIONALE.md](RATIONALE.md) file.
114