[red-knot] add a large-union-of-string-literals benchmark #17393
Conversation
|
| }, | ||
| |case| { | ||
| let Case { db, .. } = case; | ||
| let result = db.check().unwrap(); |
There was a problem hiding this comment.
I'm not familiar with the existing benchmark setup here, and I see that we do the same thing for benchmark_cold, so this is more of a question: how can we be sure that we are actually re-inferring types in this call here (instead of relying on salsa-cached values)?
There was a problem hiding this comment.
I'm not an expert on this benchmark setup either, but I think the answer is our use of iter_batched_ref, which (per the documentation) treats the input data from the first closure as non-reusable. So it generates a batch of input data by running the first closure (setup) multiple times, then runs the second closure (timed) for each input in the batch. This means that each timed execution of the "check" is running on a brand-new separate Salsa db, with a new memory file system, etc -- they are fully isolated.
There was a problem hiding this comment.
So it generates a batch of input data by running the first closure (setup) multiple times, then runs the second closure (timed) for each input in the batch.
Oh — thanks for the explanation. I was wrongly assuming the "setup" to only run once for some reason (same terminology but different functionality in other benchmarking libraries).
There was a problem hiding this comment.
I think Criterion offers both; if the artifacts from the setup are reusable and only need to be created once, you use iter (this is the more common/basic case), if the setup artifacts are non-reusable and need to be generated once per execution, then you use iter_batched/iter_batched_ref. (I don't find those method names very inherently clear, but the docs are useful.)
* main: (31 commits) [red-knot] Add some knowledge of `__all__` to `*`-import machinery (#17373) Update taiki-e/install-action digest to be7c31b (#17379) Update Rust crate mimalloc to v0.1.46 (#17382) Update PyO3/maturin-action action to v1.49.1 (#17384) Update Rust crate anyhow to v1.0.98 (#17380) dependencies: switch from `chrono` to `jiff` Update Rust crate bstr to v1.12.0 (#17385) [red-knot] Further optimize `*`-import visibility constraints (#17375) [red-knot] Minor 'member_lookup_with_policy' fix (#17407) [red-knot] Initial support for `dataclass`es (#17353) Sync vendored typeshed stubs (#17402) [red-knot] improve function/bound method type display (#17294) [red-knot] Move relation methods from `CallableType` to `Signature` (#17365) [syntax-errors] `await` outside async functions (#17363) [red-knot] optimize is_subtype_of for literals (#17394) [red-knot] add a large-union-of-string-literals benchmark (#17393) Update pre-commit dependencies (#17383) [red-knot] mypy_primer: Fail job on panic or internal errors (#17389) [red-knot] Document limitations of diagnostics-silencing in unreachable code (#17387) [red-knot] detect unreachable attribute assignments (#16852) ...
Summary
Add a benchmark for a large-union case that currently has exponential blow-up in execution time.
Test Plan
cargo bench --bench red_knot