`deno bench` ignores `n` and `warmup`options when code being benched is too quick

[albnnc]

Version: Deno 2.2.3

According to this PR, n and warmup options should work again.

However, if I have the following bench code:

let i = 0;

Deno.bench(
  "noop",
  { n: 3, warmup: 1 },
  () => {
    ++i;
    console.log(i);
  },
);

I get unexpected output:

...
test: 67854
test: 67855
test: 67856
test: 67857
test: 67858
test: 67859
test: 67860
test: 67861
test                 7.5 µs       133,900 (  3.2 µs …   4.4 ms)   5.6 µs  44.6 µs  51.2 µs

Do I understand n and warmup options wrong?

Also, in case of missing warmup option and presence of n, should Deno actually warmup benchmark in some "intellectual" way?

[dsherret]

It's because what's being measured is too quick so it couldn't meaningfully give you a more accurate measurement without running it a lot more times. I think it runs it for at least 10ms.

[albnnc]

I tend to disagree.

I wanted to start using these options because my real benchmarks run for too long: single iteration takes up to several seconds while Deno repeats bench for too many times.

That being said, code above if for demonstration purposes only and I used it since it doesn't differ from actual benchmarks in any way related to Deno behaviour.

[dsherret]

What are you disagreeing about?

[dsherret]

I just relooked at the code and it runs the bench for a minimum of 500ms:

deno/cli/js/40_bench.js

Line 478 in 0ef3f6b

const benchTimeInMs = 500;

(warmup is 10ms min)

Maybe that could be reduced for when n is specified, but the result will be less accurate.

[albnnc]

Here's my real case.

Benchmark code

let i = 0;

Deno.bench(
  "entity delete unit (by filter, 10000)",
  { n: 3, warmup: 0 },
  async (b) => {
    ++i;
    console.log("i =", i);
    await bench(b, 10_000);
  },
);

Benchmark output

    CPU | Intel Xeon Processor (Cascadelake)
Runtime | Deno 2.2.3 (x86_64-unknown-linux-gnu)

file:///code/eve/units/entity/delete_bench.ts

benchmark                               time/iter (avg)        iter/s      (min … max)           p75      p99     p995
--------------------------------------- ----------------------------- --------------------- --------------------------
entity delete unit (by filter, 10000): i = 1
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 2
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 3
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 4
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 5
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 6
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 7
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 8
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 9
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 10
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 11
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 12
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 13
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 14
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 15
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 16
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 17
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 18
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 19
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 20
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 21
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 22
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 23
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 24
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000): i = 25
entity delete unit (by filter, 10000): INFO: Configured
entity delete unit (by filter, 10000): INFO(BROKER): Connecting
entity delete unit (by filter, 10000): INFO(DB): Connecting
entity delete unit (by filter, 10000): INFO(STORAGE): Connecting
entity delete unit (by filter, 10000): INFO(TSDB): Connecting
entity delete unit (by filter, 10000): INFO(BROKER_PROVISOR): Wiping broker
entity delete unit (by filter, 10000): INFO(DB_PROVISOR): Wiping DB
entity delete unit (by filter, 10000): INFO(TSDB_PROVISOR): Wiping TSDB
entity delete unit (by filter, 10000): INFO(STORAGE_PROVISOR): Wiping object storage
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Serving known entities (2)
entity delete unit (by filter, 10000): INFO(SYSTEM_PROVISOR): Updating system provision meta
entity delete unit (by filter, 10000): INFO: Bootstrapped
entity delete unit (by filter, 10000)             1.3 s           0.8 (   1.2 s …    1.3 s)    1.3 s    1.3 s    1.3 s

LOG_LEVEL=DEBUG ./make.ts bench eve/units/entity/delete_bench.ts --no-check -  1.71s user 0.52s system 4% cpu 49.806 total

So, benchmark has been ran for 25 times while single run took ~1 second. I can't understand why is it like that, since each run is not "too fast" and I specified n: 3, warmup: 0.

[dsherret]

Ah, right now it's ignoring a warmup of 0 and it probably shouldn't:

deno/cli/js/40_bench.js

Line 254 in 0ef3f6b

let iterations = desc.warmup > 0 ? desc.warmup : 20;

-- It will still do a warmup in that case, but only to figure out if it's a fast running test or not.

[dsherret]

I updated the docs and it should execute much faster now. Also, specifying a warmup of 0 will cause it I don't believe there's a way to get reliable results if a warmup run isn't run or if the number of iterations are lowered too much for fast benchmarks (there's a time precision issue for fast benchmarks where it needs to run for at least a certain number of milliseconds in order to get an accurate result. For example, Windows timer resolution is ~16ms).

That said, I noticed that the warmup will not output to the console and this is probably wrong because that affects the warmup measurement (basically, it thinks the warmup run is really fast, so it runs the test a bunch of times before checking the time which causes it to be slow). I opened #28457 Edit: This was wrong. Seems like sometimes it's fast and sometimes slow for the warmups.