1=====================
2Test-Suite Extensions
3=====================
4
5.. contents::
6   :depth: 1
7   :local:
8
9Abstract
10========
11
12These are ideas for additional programs, benchmarks, applications and
13algorithms that could be added to the LLVM Test-Suite.
14The test-suite could be much larger than it is now, which would help us
15detecting compiler errors (crashes, miscompiles) during development.
16
17Most probably, the reason why the programs below have not been added to
18the test-suite yet is that nobody has found time to do it. But there
19might be other issues as well, such as
20
21 * Licensing (Support can still be added as external module,
22              like for the SPEC benchmarks)
23
24 * Language (in particular, there is no official LLVM frontend
25             for FORTRAN yet)
26
27 * Parallelism (currently, all programs in test-suite use
28                one thread only)
29
30Benchmarks
31==========
32
33SPEC CPU 2017
34-------------
35https://www.spec.org/cpu2017/
36
37The following have not been included yet because they contain Fortran
38code.
39
40In case of cactuBSSN only a small portion is Fortran. The hosts's
41Fortran compiler could be used for these parts.
42
43Note that CMake's Ninja generator has difficulties with Fortran. See the
44`CMake documentation <https://cmake.org/cmake/help/v3.13/generator/Ninja.html#fortran-support>`_
45for details.
46
47 * 503.bwaves_r/603.bwaves_s
48 * 507.cactuBSSN_r
49 * 521.wrf_r/621.wrf_s
50 * 527.cam4_r/627.cam4_s
51 * 628.pop2_s
52 * 548.exchange2_r/648.exchange2_s
53 * 549.fotonik3d_r/649.fotonik3d_s
54 * 554.roms_r/654.roms_s
55
56SPEC OMP2012
57------------
58https://www.spec.org/omp2012/
59
60 * 350.md
61 * 351.bwaves
62 * 352.nab
63 * 357.bt331
64 * 358.botsalgn
65 * 359.botsspar
66 * 360.ilbdc
67 * 362.fma3d
68 * 363.swim
69 * 367.imagick
70 * 370.mgrid331
71 * 371.applu331
72 * 372.smithwa
73 * 376.kdtree
74
75OpenCV
76------
77https://opencv.org/
78
79OpenMP 4.x SIMD Benchmarks
80--------------------------
81https://github.com/flwende/simd_benchmarks
82
83PWM-benchmarking
84----------------
85https://github.com/tbepler/PWM-benchmarking
86
87SLAMBench
88---------
89https://github.com/pamela-project/slambench
90
91FireHose
92--------
93http://firehose.sandia.gov/
94
95A Benchmark for the C/C++ Standard Library
96------------------------------------------
97https://github.com/hiraditya/std-benchmark
98
99OpenBenchmarking.org CPU / Processor Suite
100------------------------------------------
101https://openbenchmarking.org/suite/pts/cpu
102
103This is a subset of the
104`Phoronix Test Suite <https://github.com/phoronix-test-suite/phoronix-test-suite/>`_
105and is itself a collection of benchmark suites
106
107Parboil Benchmarks
108------------------
109http://impact.crhc.illinois.edu/parboil/parboil.aspx
110
111MachSuite
112---------
113https://breagen.github.io/MachSuite/
114
115Rodinia
116-------
117http://lava.cs.virginia.edu/Rodinia/download_links.htm
118
119Rodinia has already been partially included in
120MultiSource/Benchmarks/Rodinia. Benchmarks still missing are:
121
122 * streamcluster
123 * particlefilter
124 * nw
125 * nn
126 * myocyte
127 * mummergpu
128 * lud
129 * leukocyte
130 * lavaMD
131 * kmeans
132 * hotspot3D
133 * heartwall
134 * cfd
135 * bfs
136 * b+tree
137
138vecmathlib tests harness
139------------------------
140https://bitbucket.org/eschnett/vecmathlib/wiki/Home
141
142PARSEC
143------
144http://parsec.cs.princeton.edu/
145
146Graph500 reference implementations
147----------------------------------
148https://github.com/graph500/graph500/tree/v2-spec
149
150NAS Parallel Benchmarks
151-----------------------
152https://www.nas.nasa.gov/publications/npb.html
153
154The official benchmark is written in Fortran, but an unofficial
155C-translation is available as well:
156https://github.com/benchmark-subsetting/NPB3.0-omp-C
157
158DARPA HPCS SSCA#2 C/OpenMP reference implementation
159---------------------------------------------------
160http://www.highproductivity.org/SSCABmks.htm
161
162This web site does not exist any more, but there seems to be a copy of
163some of the benchmarks
164https://github.com/gtcasl/hpc-benchmarks/tree/master/SSCA2v2.2
165
166Kokkos
167------
168https://github.com/kokkos/kokkos-kernels/tree/master/perf_test
169https://github.com/kokkos/kokkos/tree/master/benchmarks
170
171PolyMage
172--------
173https://github.com/bondhugula/polymage-benchmarks
174
175PolyBench
176---------
177https://sourceforge.net/projects/polybench/
178
179A modified version of Polybench 3.2 is already presented in
180SingleSource/Benchmarks/Polybench. A newer version 4.2.1 is available.
181
182High Performance Geometric Multigrid
183------------------------------------
184https://crd.lbl.gov/departments/computer-science/PAR/research/hpgmg/
185
186RAJA Performance Suite
187----------------------
188https://github.com/LLNL/RAJAPerf
189
190CORAL-2 Benchmarks
191------------------
192https://asc.llnl.gov/coral-2-benchmarks/
193
194Many of its programs have already been integrated in
195MultiSource/Benchmarks/DOE-ProxyApps-C and
196MultiSource/Benchmarks/DOE-ProxyApps-C++.
197
198 * Nekbone
199 * QMCPack
200 * LAMMPS
201 * Kripke
202 * Quicksilver
203 * PENNANT
204 * Big Data Analytic Suite
205 * Deep Learning Suite
206 * Stream
207 * Stride
208 * ML/DL micro-benchmark
209 * Pynamic
210 * ACME
211 * VPIC
212 * Laghos
213 * Parallel Integer Sort
214 * Havoq
215
216NWChem
217------
218http://www.nwchem-sw.org/index.php/Benchmarks
219
220TVM
221----
222https://github.com/dmlc/tvm/tree/main/apps/benchmark
223
224HydroBench
225----------
226https://github.com/HydroBench/Hydro
227
228ParRes
229------
230https://github.com/ParRes/Kernels/tree/default/Cxx11
231
232Applications/Libraries
233======================
234
235GnuPG
236-----
237https://gnupg.org/
238
239Blitz++
240-------
241https://sourceforge.net/projects/blitz/
242
243FFmpeg
244------
245https://ffmpeg.org/
246
247FreePOOMA
248---------
249http://www.nongnu.org/freepooma/
250
251FTensors
252--------
253http://www.wlandry.net/Projects/FTensor
254
255rawspeed
256--------
257https://github.com/darktable-org/rawspeed
258
259Its test dataset is 756 MB in size, which is too large to be included
260into the test-suite repository.
261
262C++ Performance Benchmarks
263--------------------------
264https://gitlab.com/chriscox/CppPerformanceBenchmarks
265
266Generic Algorithms
267==================
268
269Image processing
270----------------
271
272Resampling
273``````````
274
275 * Bilinear
276 * Bicubic
277 * Lanczos
278
279Dither
280``````
281
282 * Threshold
283 * Random
284 * Halftone
285 * Bayer
286 * Floyd-Steinberg
287 * Jarvis
288 * Stucki
289 * Burkes
290 * Sierra
291 * Atkinson
292 * Gradient-based
293
294Feature detection
295`````````````````
296
297 * Harris
298 * Histogram of Oriented Gradients
299
300Color conversion
301````````````````
302
303 * RGB to grayscale
304 * HSL to RGB
305
306Graph
307-----
308
309Search Algorithms
310`````````````````
311
312 * Breadth-First-Search
313 * Depth-First-Search
314 * Dijkstra's algorithm
315 * A-Star
316
317Spanning Tree
318`````````````
319
320 * Kruskal's algorithm
321 * Prim's algorithm
322