docs/Proposals/TestSuite.rst

   1 =====================
   2 Test-Suite Extentions
   3 =====================
   4
   5 .. contents::
   6    :depth: 1
   7    :local:
   8
   9 Abstract
  10 ========
  11
  12 These are ideas for additional programs, benchmarks, applications and
  13 algorithms that could be added to the LLVM Test-Suite.
  14 The test-suite could be much larger than it is now, which would help us
  15 detecting compiler errors (crashes, miscompiles) during development.
  16
  17 Most probably, the reason why the programs below have not been added to
  18 the test-suite yet is that nobody has found time to do it. But there
  19 might be other issues as well, such as
  20
  21  * Licensing (Support can still be added as external module,
  22               like for the SPEC benchmarks)
  23
  24  * Language (in particular, there is no official LLVM frontend
  25              for FORTRAN yet)
  26
  27  * Parallelism (currently, all programs in test-suite use
  28                 one thread only)
  29
  30 Benchmarks
  31 ==========
  32
  33 SPEC CPU 2017
  34 -------------
  35 https://www.spec.org/cpu2017/
  36
  37 The following have not been included yet because they contain Fortran
  38 code.
  39
  40 In case of cactuBSSN only a small portion is Fortran. The hosts's
  41 Fortran compiler could be used for these parts.
  42
  43 Note that CMake's Ninja generator has difficulties with Fortran. See the
  44 `CMake documentation <https://cmake.org/cmake/help/v3.13/generator/Ninja.html#fortran-support>`_
  45 for details.
  46
  47  * 503.bwaves_r/603.bwaves_s
  48  * 507.cactuBSSN_r
  49  * 521.wrf_r/621.wrf_s
  50  * 527.cam4_r/627.cam4_s
  51  * 628.pop2_s
  52  * 548.exchange2_r/648.exchange2_s
  53  * 549.fotonik3d_r/649.fotonik3d_s
  54  * 554.roms_r/654.roms_s
  55
  56 SPEC OMP2012
  57 ------------
  58 https://www.spec.org/omp2012/
  59
  60  * 350.md
  61  * 351.bwaves
  62  * 352.nab
  63  * 357.bt331
  64  * 358.botsalgn
  65  * 359.botsspar
  66  * 360.ilbdc
  67  * 362.fma3d
  68  * 363.swim
  69  * 367.imagick
  70  * 370.mgrid331
  71  * 371.applu331
  72  * 372.smithwa
  73  * 376.kdtree
  74
  75 OpenCV
  76 ------
  77 https://opencv.org/
  78
  79 OpenMP 4.x SIMD Benchmarks
  80 --------------------------
  81 https://github.com/flwende/simd_benchmarks
  82
  83 PWM-benchmarking
  84 ----------------
  85 https://github.com/tbepler/PWM-benchmarking
  86
  87 SLAMBench
  88 ---------
  89 https://github.com/pamela-project/slambench
  90
  91 FireHose
  92 --------
  93 http://firehose.sandia.gov/
  94
  95 A Benchmark for the C/C++ Standard Library
  96 ------------------------------------------
  97 https://github.com/hiraditya/std-benchmark
  98
  99 OpenBenchmarking.org CPU / Processor Suite
 100 ------------------------------------------
 101 https://openbenchmarking.org/suite/pts/cpu
 102
 103 This is a subset of the
 104 `Phoronix Test Suite <https://github.com/phoronix-test-suite/phoronix-test-suite/>`_
 105 and is itself a collection of benchmark suites
 106
 107 Parboil Benchmarks
 108 ------------------
 109 http://impact.crhc.illinois.edu/parboil/parboil.aspx
 110
 111 MachSuite
 112 ---------
 113 https://breagen.github.io/MachSuite/
 114
 115 Rodinia
 116 -------
 117 http://lava.cs.virginia.edu/Rodinia/download_links.htm
 118
 119 Rodinia has already been partially included in
 120 MultiSource/Benchmarks/Rodinia. Benchmarks still missing are:
 121
 122  * streamcluster
 123  * particlefilter
 124  * nw
 125  * nn
 126  * myocyte
 127  * mummergpu
 128  * lud
 129  * leukocyte
 130  * lavaMD
 131  * kmeans
 132  * hotspot3D
 133  * heartwall
 134  * cfd
 135  * bfs
 136  * b+tree
 137
 138 vecmathlib tests harness
 139 ------------------------
 140 https://bitbucket.org/eschnett/vecmathlib/wiki/Home
 141
 142 PARSEC
 143 ------
 144 http://parsec.cs.princeton.edu/
 145
 146 Graph500 reference implementations
 147 ----------------------------------
 148 https://github.com/graph500/graph500/tree/v2-spec
 149
 150 NAS Parallel Benchmarks
 151 -----------------------
 152 https://www.nas.nasa.gov/publications/npb.html
 153
 154 The official benchmark is written in Fortran, but an unofficial
 155 C-translation is available as well:
 156 https://github.com/benchmark-subsetting/NPB3.0-omp-C
 157
 158 DARPA HPCS SSCA#2 C/OpenMP reference implementation
 159 ---------------------------------------------------
 160 http://www.highproductivity.org/SSCABmks.htm
 161
 162 This web site does not exist any more, but there seems to be a copy of
 163 some of the benchmarks
 164 https://github.com/gtcasl/hpc-benchmarks/tree/master/SSCA2v2.2
 165
 166 Kokkos
 167 ------
 168 https://github.com/kokkos/kokkos-kernels/tree/master/perf_test
 169 https://github.com/kokkos/kokkos/tree/master/benchmarks
 170
 171 PolyMage
 172 --------
 173 https://github.com/bondhugula/polymage-benchmarks
 174
 175 PolyBench
 176 ---------
 177 https://sourceforge.net/projects/polybench/
 178
 179 A modified version of Polybench 3.2 is already presented in
 180 SingleSource/Benchmarks/Polybench. A newer version 4.2.1 is available.
 181
 182 High Performance Geometric Multigrid
 183 ------------------------------------
 184 https://crd.lbl.gov/departments/computer-science/PAR/research/hpgmg/
 185
 186 RAJA Performance Suite
 187 ----------------------
 188 https://github.com/LLNL/RAJAPerf
 189
 190 CORAL-2 Benchmarks
 191 ------------------
 192 https://asc.llnl.gov/coral-2-benchmarks/
 193
 194 Many of its programs have already been integreated in
 195 MultiSource/Benchmarks/DOE-ProxyApps-C and
 196 MultiSource/Benchmarks/DOE-ProxyApps-C++.
 197
 198  * Nekbone
 199  * QMCPack
 200  * LAMMPS
 201  * Kripke
 202  * Quicksilver
 203  * PENNANT
 204  * Big Data Analytic Suite
 205  * Deep Learning Suite
 206  * Stream
 207  * Stride
 208  * ML/DL micro-benchmark
 209  * Pynamic
 210  * ACME
 211  * VPIC
 212  * Laghos
 213  * Parallel Integer Sort
 214  * Havoq
 215
 216 NWChem
 217 ------
 218 http://www.nwchem-sw.org/index.php/Benchmarks
 219
 220 TVM
 221 ----
 222 https://github.com/dmlc/tvm/tree/master/apps/benchmark
 223
 224 HydroBench
 225 ----------
 226 https://github.com/HydroBench/Hydro
 227
 228 ParRes
 229 ------
 230 https://github.com/ParRes/Kernels/tree/master/Cxx11
 231
 232 Applications/Libraries
 233 ======================
 234
 235 GnuPG
 236 -----
 237 https://gnupg.org/
 238
 239 Blitz++
 240 -------
 241 https://sourceforge.net/projects/blitz/
 242
 243 FFmpeg
 244 ------
 245 https://ffmpeg.org/
 246
 247 FreePOOMA
 248 ---------
 249 http://www.nongnu.org/freepooma/
 250
 251 FTensors
 252 --------
 253 http://www.wlandry.net/Projects/FTensor
 254
 255 rawspeed
 256 --------
 257 https://github.com/darktable-org/rawspeed
 258
 259 Its test dataset is 756 MB in size, which is too large to be included
 260 into the test-suite repository.
 261
 262 C++ Performance Benchmarks
 263 --------------------------
 264 https://gitlab.com/chriscox/CppPerformanceBenchmarks
 265
 266 Generic Algorithms
 267 ==================
 268
 269 Image processing
 270 ----------------
 271
 272 Resampling
 273 ``````````
 274
 275  * Bilinear
 276  * Bicubic
 277  * Lanczos
 278
 279 Dither
 280 ``````
 281
 282  * Threshold
 283  * Random
 284  * Halftone
 285  * Bayer
 286  * Floyd-Steinberg
 287  * Jarvis
 288  * Stucki
 289  * Burkes
 290  * Sierra
 291  * Atkinson
 292  * Gradient-based
 293
 294 Feature detection
 295 `````````````````
 296
 297  * Harris
 298  * Histogram of Oriented Gradients
 299
 300 Color conversion
 301 ````````````````
 302
 303  * RGB to grayscale
 304  * HSL to RGB
 305
 306 Graph
 307 -----
 308
 309 Search Algorithms
 310 `````````````````
 311
 312  * Breadth-First-Search
 313  * Depth-First-Search
 314  * Dijkstra's algorithm
 315  * A-Star
 316
 317 Spanning Tree
 318 `````````````
 319
 320  * Kruskal's algorithm
 321  * Prim's algorithm