usr.bin/ministat/README

   1 $FreeBSD$
   2
   3 A small tool to do the statistics legwork on benchmarks etc.
   4
   5 Prepare your data into two files, one number per line
   6 run
   7         ./ministat data_before data_after
   8
   9 and see what it says.
  10
  11 You need at least three data points in each data set, but the more
  12 you have the better your result generally gets.
  13
  14 Here are two typical outputs:
  15
  16 x _1
  17 + _2
  18 +--------------------------------------------------------------------------+
  19 |x            +    x+      x            x   x             +           ++   |
  20 |        |_________|______AM_______________|__A___________M_______________||
  21 +--------------------------------------------------------------------------+
  22     N           Min           Max        Median           Avg        Stddev
  23 x   5         36060         36138         36107       36105.6     31.165686
  24 +   5         36084         36187         36163       36142.6     49.952978
  25 No difference proven at 95.0% confidence
  26
  27 Here nothing can be concluded from the numbers.  It _may_ be possible to
  28 prove something if many more measurements are made, but with only five
  29 measurements, nothing is proven.
  30
  31
  32 x _1
  33 + _2
  34 +--------------------------------------------------------------------------+
  35 |                                                               +          |
  36 |                               x                               +         +|
  37 |x                    x         x          x                    +         +|
  38 |         |_______________A_____M_________|                   |_M___A____| |
  39 +--------------------------------------------------------------------------+
  40     N           Min           Max        Median           Avg        Stddev
  41 x   5         0.133         0.137         0.136        0.1354  0.0015165751
  42 +   5         0.139          0.14         0.139        0.1394 0.00054772256
  43 Difference at 95.0% confidence
  44         0.004 +/- 0.00166288
  45         2.95421% +/- 1.22812%
  46         (Student's t, pooled s = 0.00114018)
  47
  48 Here we have a clearcut difference, not very big, but clear and unambiguous.
  49
  50