DieHarder? And some results.

[jrp2014]

I wasn't able to figure out how to run die harder on this library (cabal-plan seems to be some javascript-related doobry) but here, for reference, are some results after adding -O2 -fllvm to the ghc-options in the cabal file. cabal test, which I was expecting to do the diehard validation seems to complete v quickly (and successfully).

⇒  cabal bench                                                         master ✱
Build profile: -w ghc-8.8.3 -O1
In order, the following will be built (use -v for more details):
 - splitmix-0.0.4 (bench:comparison) (first run)
 - splitmix-0.0.4 (bench:range) (first run)
 - splitmix-0.0.4 (bench:simple-sum) (first run)
Preprocessing benchmark 'range' for splitmix-0.0.4..
Preprocessing benchmark 'simple-sum' for splitmix-0.0.4..
Preprocessing benchmark 'comparison' for splitmix-0.0.4..
Building benchmark 'simple-sum' for splitmix-0.0.4..
Building benchmark 'comparison' for splitmix-0.0.4..
Building benchmark 'range' for splitmix-0.0.4..
Running 1 benchmarks...
Benchmark simple-sum: RUNNING...
Summing randoms...
Running 1 benchmarks...
Benchmark comparison: RUNNING...
benchmarking list/random
Running 1 benchmarks...
Benchmark range: RUNNING...
[8,3,6,4,4,5,9,3,8,3,9,2,0,6,8,3,6,1,6,9,9,3,7,9,7,7,6,3,6,3,4,9,2,6,7,0,1,7,4,0,7,4,6,6,5,1,7,8,6,1,0,0,0,3,9,8,2,6,6,0,0,3,8,1,8,4,7,2,5,9]
773728671
Benchmark simple-sum: FINISH
4193221127
loop: 214.406ms
[8,2,3,7,5,4,6,6,3,8,8,9,5,1,3,3,3,2,6,9,5,1,0,0,9,4,8,0,8,8,9,1,5,8,6,3,8,6,9,3,2,9,5,1,4,5,3,3,3,7,9,5,6,8,5,1,6,8,5,1,9,8,2,9,2,2,3,6,1,9]
184201478
loop: 24.328ms
[2,9,0,1,5,2,6,9,3,9,2,1,2,6,9,8,0,4,3,6,2,3,5,0,6,7,1,1,5,2,1,7,0,7,6,1,6,6,7,7,3,0,6,4,1,9,1,6,9,5,4,7,5,4,9,9,5,9,5,9,5,8,4,5,4,2,5,3,6,9]
399431642
loop: 22.441ms
[5,7,4,1,2,7,1,3,6,8,5,3,8,8,8,8,6,8,1,9,8,2,9,1,3,3,1,5,9,5,1,8,0,1,1,9,8,2,8,9,0,3,5,4,0,0,4,9,0,2,0,1,8,6,9,6,4,7,4,3,4,3,2,6,8,1,3,8,4,4]
4059709080
loop: 11.869ms
Benchmark range: FINISH
time                 77.11 μs   (75.32 μs .. 78.78 μs)
                     0.996 R²   (0.995 R² .. 0.998 R²)
mean                 78.16 μs   (76.81 μs .. 80.03 μs)
std dev              5.283 μs   (3.834 μs .. 7.376 μs)
variance introduced by outliers: 68% (severely inflated)

benchmarking list/tf-random
time                 43.53 μs   (43.31 μs .. 43.78 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 43.89 μs   (43.62 μs .. 44.27 μs)
std dev              1.111 μs   (796.4 ns .. 1.469 μs)
variance introduced by outliers: 24% (moderately inflated)

benchmarking list/splitmix
time                 10.50 μs   (10.44 μs .. 10.59 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 10.60 μs   (10.53 μs .. 10.70 μs)
std dev              283.8 ns   (216.5 ns .. 397.5 ns)
variance introduced by outliers: 30% (moderately inflated)

benchmarking list/splitmix32
time                 53.10 μs   (52.67 μs .. 53.65 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 53.37 μs   (53.02 μs .. 53.92 μs)
std dev              1.496 μs   (1.077 μs .. 2.141 μs)
variance introduced by outliers: 27% (moderately inflated)

benchmarking tree/random
time                 78.27 μs   (77.75 μs .. 79.02 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 78.80 μs   (78.19 μs .. 80.15 μs)
std dev              2.952 μs   (1.674 μs .. 5.519 μs)
variance introduced by outliers: 39% (moderately inflated)

benchmarking tree/tf-random
time                 97.22 μs   (96.57 μs .. 98.01 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 97.81 μs   (97.35 μs .. 98.93 μs)
std dev              2.319 μs   (1.257 μs .. 4.271 μs)
variance introduced by outliers: 20% (moderately inflated)

benchmarking tree/splitmix
time                 31.19 μs   (30.90 μs .. 31.66 μs)
                     0.999 R²   (0.997 R² .. 1.000 R²)
mean                 31.33 μs   (31.07 μs .. 31.76 μs)
std dev              1.113 μs   (776.6 ns .. 1.565 μs)
variance introduced by outliers: 40% (moderately inflated)

benchmarking tree/splitmix32
time                 54.17 μs   (53.95 μs .. 54.40 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 54.11 μs   (53.92 μs .. 54.42 μs)
std dev              814.4 ns   (560.5 ns .. 1.120 μs)

benchmarking list 64/random
time                 690.3 μs   (686.6 μs .. 694.7 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 695.4 μs   (690.1 μs .. 707.7 μs)
std dev              25.56 μs   (15.74 μs .. 45.50 μs)
variance introduced by outliers: 28% (moderately inflated)

benchmarking list 64/tf-random
time                 96.02 μs   (95.22 μs .. 96.70 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 95.70 μs   (95.23 μs .. 96.72 μs)
std dev              2.201 μs   (1.335 μs .. 3.679 μs)
variance introduced by outliers: 19% (moderately inflated)

benchmarking list 64/splitmix
time                 10.52 μs   (10.47 μs .. 10.59 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 10.72 μs   (10.56 μs .. 10.99 μs)
std dev              668.3 ns   (360.0 ns .. 1.091 μs)
variance introduced by outliers: 71% (severely inflated)

benchmarking list 64/splitmix32
time                 47.42 μs   (47.15 μs .. 47.83 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 47.34 μs   (47.16 μs .. 47.66 μs)
std dev              798.2 ns   (588.7 ns .. 1.131 μs)
variance introduced by outliers: 12% (moderately inflated)

benchmarking tree 64/random
time                 393.0 μs   (388.9 μs .. 397.5 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 392.6 μs   (391.2 μs .. 395.1 μs)
std dev              6.418 μs   (4.380 μs .. 9.662 μs)

benchmarking tree 64/tf-random
time                 118.6 μs   (117.8 μs .. 119.5 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 118.6 μs   (118.1 μs .. 119.6 μs)
std dev              2.304 μs   (1.638 μs .. 3.345 μs)
variance introduced by outliers: 13% (moderately inflated)

benchmarking tree 64/splitmix
time                 30.04 μs   (29.92 μs .. 30.22 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 30.27 μs   (30.10 μs .. 30.59 μs)
std dev              748.5 ns   (507.9 ns .. 1.235 μs)
variance introduced by outliers: 24% (moderately inflated)

benchmarking tree 64/splitmix32
time                 50.58 μs   (50.43 μs .. 50.76 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 50.99 μs   (50.72 μs .. 51.53 μs)
std dev              1.311 μs   (644.3 ns .. 2.098 μs)
variance introduced by outliers: 24% (moderately inflated)

Benchmark comparison: FINISH

And this is how fast die harder ran, for comparison.

⇒  time dieharder -a                                                      cabal
#=============================================================================#
#            dieharder version 3.31.1 Copyright 2003 Robert G. Brown          #
#=============================================================================#
   rng_name    |rands/second|   Seed   |
        mt19937|  2.31e+08  |1357289862|
#=============================================================================#
        test_name   |ntup| tsamples |psamples|  p-value |Assessment
#=============================================================================#
   diehard_birthdays|   0|       100|     100|0.32219977|  PASSED  
      diehard_operm5|   0|   1000000|     100|0.99183311|  PASSED  
  diehard_rank_32x32|   0|     40000|     100|0.88829129|  PASSED  
    diehard_rank_6x8|   0|    100000|     100|0.47552993|  PASSED  
   diehard_bitstream|   0|   2097152|     100|0.57207286|  PASSED  
        diehard_opso|   0|   2097152|     100|0.96183858|  PASSED  
        diehard_oqso|   0|   2097152|     100|0.01961944|  PASSED  
         diehard_dna|   0|   2097152|     100|0.72354765|  PASSED  
diehard_count_1s_str|   0|    256000|     100|0.46228067|  PASSED  
diehard_count_1s_byt|   0|    256000|     100|0.05970675|  PASSED  
 diehard_parking_lot|   0|     12000|     100|0.60849546|  PASSED  
    diehard_2dsphere|   2|      8000|     100|0.47403236|  PASSED  
    diehard_3dsphere|   3|      4000|     100|0.17712917|  PASSED  
     diehard_squeeze|   0|    100000|     100|0.76417557|  PASSED  
        diehard_sums|   0|       100|     100|0.00188530|   WEAK   
        diehard_runs|   0|    100000|     100|0.86714451|  PASSED  
        diehard_runs|   0|    100000|     100|0.31468742|  PASSED  
       diehard_craps|   0|    200000|     100|0.27692138|  PASSED  
       diehard_craps|   0|    200000|     100|0.99606867|   WEAK   
 marsaglia_tsang_gcd|   0|  10000000|     100|0.91269254|  PASSED  
 marsaglia_tsang_gcd|   0|  10000000|     100|0.30202499|  PASSED  
         sts_monobit|   1|    100000|     100|0.75039500|  PASSED  
            sts_runs|   2|    100000|     100|0.87121820|  PASSED  
          sts_serial|   1|    100000|     100|0.91276174|  PASSED  
          sts_serial|   2|    100000|     100|0.07785382|  PASSED  
          sts_serial|   3|    100000|     100|0.21391146|  PASSED  
          sts_serial|   3|    100000|     100|0.61636101|  PASSED  
          sts_serial|   4|    100000|     100|0.04852318|  PASSED  
          sts_serial|   4|    100000|     100|0.06890833|  PASSED  
          sts_serial|   5|    100000|     100|0.64640039|  PASSED  
          sts_serial|   5|    100000|     100|0.15566413|  PASSED  
          sts_serial|   6|    100000|     100|0.25925876|  PASSED  
          sts_serial|   6|    100000|     100|0.42188017|  PASSED  
          sts_serial|   7|    100000|     100|0.89712564|  PASSED  
          sts_serial|   7|    100000|     100|0.15752814|  PASSED  
          sts_serial|   8|    100000|     100|0.99200916|  PASSED  
          sts_serial|   8|    100000|     100|0.14754261|  PASSED  
          sts_serial|   9|    100000|     100|0.40224701|  PASSED  
          sts_serial|   9|    100000|     100|0.11783364|  PASSED  
          sts_serial|  10|    100000|     100|0.41104508|  PASSED  
          sts_serial|  10|    100000|     100|0.65270873|  PASSED  
          sts_serial|  11|    100000|     100|0.40068619|  PASSED  
          sts_serial|  11|    100000|     100|0.87154252|  PASSED  
          sts_serial|  12|    100000|     100|0.61520736|  PASSED  
          sts_serial|  12|    100000|     100|0.79457532|  PASSED  
          sts_serial|  13|    100000|     100|0.80324219|  PASSED  
          sts_serial|  13|    100000|     100|0.47835828|  PASSED  
          sts_serial|  14|    100000|     100|0.36973089|  PASSED  
          sts_serial|  14|    100000|     100|0.23513146|  PASSED  
          sts_serial|  15|    100000|     100|0.94750268|  PASSED  
          sts_serial|  15|    100000|     100|0.98515115|  PASSED  
          sts_serial|  16|    100000|     100|0.55321285|  PASSED  
          sts_serial|  16|    100000|     100|0.62540449|  PASSED  
         rgb_bitdist|   1|    100000|     100|0.96320086|  PASSED  
         rgb_bitdist|   2|    100000|     100|0.83821645|  PASSED  
         rgb_bitdist|   3|    100000|     100|0.20321908|  PASSED  
         rgb_bitdist|   4|    100000|     100|0.98524885|  PASSED  
         rgb_bitdist|   5|    100000|     100|0.27739524|  PASSED  
         rgb_bitdist|   6|    100000|     100|0.07489931|  PASSED  
         rgb_bitdist|   7|    100000|     100|0.61751664|  PASSED  
         rgb_bitdist|   8|    100000|     100|0.97131442|  PASSED  
         rgb_bitdist|   9|    100000|     100|0.42223836|  PASSED  
         rgb_bitdist|  10|    100000|     100|0.50178169|  PASSED  
         rgb_bitdist|  11|    100000|     100|0.35154775|  PASSED  
         rgb_bitdist|  12|    100000|     100|0.93804234|  PASSED  
rgb_minimum_distance|   2|     10000|    1000|0.93585747|  PASSED  
rgb_minimum_distance|   3|     10000|    1000|0.19720639|  PASSED  
rgb_minimum_distance|   4|     10000|    1000|0.02352979|  PASSED  
rgb_minimum_distance|   5|     10000|    1000|0.19851905|  PASSED  
    rgb_permutations|   2|    100000|     100|0.85551385|  PASSED  
    rgb_permutations|   3|    100000|     100|0.46500089|  PASSED  
    rgb_permutations|   4|    100000|     100|0.98908052|  PASSED  
    rgb_permutations|   5|    100000|     100|0.19891815|  PASSED  
      rgb_lagged_sum|   0|   1000000|     100|0.16760471|  PASSED  
      rgb_lagged_sum|   1|   1000000|     100|0.42277339|  PASSED  
      rgb_lagged_sum|   2|   1000000|     100|0.56522231|  PASSED  
      rgb_lagged_sum|   3|   1000000|     100|0.81416621|  PASSED  
      rgb_lagged_sum|   4|   1000000|     100|0.48806789|  PASSED  
      rgb_lagged_sum|   5|   1000000|     100|0.75147411|  PASSED  
      rgb_lagged_sum|   6|   1000000|     100|0.99043279|  PASSED  
      rgb_lagged_sum|   7|   1000000|     100|0.99240178|  PASSED  
      rgb_lagged_sum|   8|   1000000|     100|0.45881177|  PASSED  
      rgb_lagged_sum|   9|   1000000|     100|0.76879664|  PASSED  
      rgb_lagged_sum|  10|   1000000|     100|0.32207887|  PASSED  
      rgb_lagged_sum|  11|   1000000|     100|0.57990413|  PASSED  
      rgb_lagged_sum|  12|   1000000|     100|0.98382613|  PASSED  
      rgb_lagged_sum|  13|   1000000|     100|0.92439158|  PASSED  
      rgb_lagged_sum|  14|   1000000|     100|0.22964734|  PASSED  
      rgb_lagged_sum|  15|   1000000|     100|0.71182707|  PASSED  
      rgb_lagged_sum|  16|   1000000|     100|0.87187845|  PASSED  
      rgb_lagged_sum|  17|   1000000|     100|0.95335584|  PASSED  
      rgb_lagged_sum|  18|   1000000|     100|0.93743372|  PASSED  
      rgb_lagged_sum|  19|   1000000|     100|0.12737136|  PASSED  
      rgb_lagged_sum|  20|   1000000|     100|0.81115564|  PASSED  
      rgb_lagged_sum|  21|   1000000|     100|0.10563518|  PASSED  
      rgb_lagged_sum|  22|   1000000|     100|0.27415328|  PASSED  
      rgb_lagged_sum|  23|   1000000|     100|0.72502337|  PASSED  
      rgb_lagged_sum|  24|   1000000|     100|0.66531426|  PASSED  
      rgb_lagged_sum|  25|   1000000|     100|0.31275762|  PASSED  
      rgb_lagged_sum|  26|   1000000|     100|0.72654890|  PASSED  
      rgb_lagged_sum|  27|   1000000|     100|0.96598987|  PASSED  
      rgb_lagged_sum|  28|   1000000|     100|0.98763987|  PASSED  
      rgb_lagged_sum|  29|   1000000|     100|0.00478171|   WEAK   
      rgb_lagged_sum|  30|   1000000|     100|0.91487670|  PASSED  
      rgb_lagged_sum|  31|   1000000|     100|0.75855526|  PASSED  
      rgb_lagged_sum|  32|   1000000|     100|0.00049933|   WEAK   
     rgb_kstest_test|   0|     10000|    1000|0.39034368|  PASSED  
     dab_bytedistrib|   0|  51200000|       1|0.02421839|  PASSED  
             dab_dct| 256|     50000|       1|0.60410352|  PASSED  
Preparing to run test 207.  ntuple = 0
        dab_filltree|  32|  15000000|       1|0.64055173|  PASSED  
        dab_filltree|  32|  15000000|       1|0.28787024|  PASSED  
Preparing to run test 208.  ntuple = 0
       dab_filltree2|   0|   5000000|       1|0.20913741|  PASSED  
       dab_filltree2|   1|   5000000|       1|0.84455594|  PASSED  
Preparing to run test 209.  ntuple = 0
        dab_monobit2|  12|  65000000|       1|0.90696174|  PASSED  
dieharder -a  591.95s user 3.99s system 99% cpu 9:56.29 total

[phadej]

You need to give an argument to dieharder-splitmix wrapper. These tests are not run by default, because they take quite a lot of time to run. Note, that they are slower then built-in dieharder random number generators, as there're no indirection in between. splitmix-dieharder uses -g 200 i.e. "stdin rng" of dieharder.

% time cabal run splitmix-dieharder -- splitmix
Up to date
PASSED  109  98.20%
WEAK      2   1.80%
FAILED    0   0.00%
cabal run splitmix-dieharder -- splitmix  1654,62s user 462,40s system 134% cpu 26:08,72 total

% time cabal run splitmix-dieharder -- tfrandom
Up to date
PASSED  108  97.30%
WEAK      3   2.70%
FAILED    0   0.00%
cabal run splitmix-dieharder -- tfrandom  2348,39s user 185,66s system 164% cpu 25:39,48 total

---

Importantly, the benchmarks in the splitmix don't run dieharder. It's not a benchmark, it's a test.

[jrp2014]

Sure. I think that it's just a matter of aligning the documentation with the cabal (v2) commands that you need to invoke to make things happen, and defocus all the JS stuff. It takes quite a bit of digging to discover that time cabal run splitmix-dieharder -- splitmix is a thing.

This all seems a bit single-threaded. Perhaps that is inherent, but if you are generating lots of randoms ...

It's be good to have some settings that ran for 10m, rather than 30m

[phadej]

dieharder is single threaded. You can select individual dieharder tests via options.

I don't understand, what you mean defocus all the JS stuff. There are no mentions of JavaScripts in README, and in fact, not that much elsewhere either?

[jrp2014]

I'll have a further poke, but is not the cabal-plan in the main README a JavaScript thing? It's not a regular Haskell thing, so far as I am aware.

Are the options documented somewhere (perhaps in a source file)? And the Makefile seems to rely on GHCJS, which is probably a bit niche.

[phadej]

cabal-plan is https://hackage.haskell.org/package/cabal-plan, it is: cabal-plan: Library and utility for processing cabal's plan.json file

plan.json files are generated by cabal's Nix-style local builds and contain detailed information about the build/install plan computed by the cabal solver.

[jrp2014]

Ah. That makes more sense!