You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The new test test_sort_with_marker uses assert 1==1 which doesn't validate the sorting functionality. Consider asserting the actual result of inner_fn to ensure correctness.
The benchmark fixture always returns the custom no-op implementation, making the marker check redundant. Verify that the real pytest-benchmark fixture is correctly returned when available.
# Check if benchmark fixture is already available (pytest-benchmark is active)if"benchmark"inrequest.fixturenamesandhasattr(request, "_fixturemanager"):
try:
# Try to get the real benchmark fixturereturnrequest.getfixturevalue("benchmark")
except (pytest.FixtureLookupError, AttributeError):
passcustom_benchmark=CustomBenchmark()
ifrequest.node.get_closest_marker("benchmark"):
# Return our custom benchmark for tests marked with @pytest.mark.benchmarkreturncustom_benchmarkreturncustom_benchmark
The PYTEST_PLUGINS assignment is duplicated in multiple test runner functions. Consider extracting it into a shared constant or helper to reduce repetition and simplify maintenance.
Ensure the inner benchmark function is actually executed and its output validated. Call inner_fn() and assert that it returns the expected sorted list instead of a placeholder assertion.
@pytest.mark.benchmark
def test_sort_with_marker(benchmark):
@benchmark
def inner_fn():
return sorter(list(reversed(range(5))))
- assert 1==1+ result = inner_fn()+ assert result == list(range(5))
Suggestion importance[1-10]: 8
__
Why: The test currently asserts a placeholder 1==1 without invoking inner_fn(), so calling it and checking the sorted result fixes test correctness.
Medium
Detect plugin with pluginmanager
Use pytest's plugin manager to detect if the benchmark plugin is active, avoiding reliance on request.fixturenames. Replace the check with request.config.pluginmanager.hasplugin("pytest_benchmark") before retrieving the real fixture.
-if "benchmark" in request.fixturenames and hasattr(request, "_fixturemanager"):+if request.config.pluginmanager.hasplugin("pytest_benchmark"):
try:
- # Try to get the real benchmark fixture
return request.getfixturevalue("benchmark")
except (pytest.FixtureLookupError, AttributeError):
pass
Suggestion importance[1-10]: 6
__
Why: Using pluginmanager.hasplugin("pytest_benchmark") is a more reliable way to detect the real benchmark fixture than inspecting request.fixturenames.
Low
General
Add iteration loop in pedantic
Implement iteration and round loops to more closely mimic pytest-benchmark’s pedantic. Loop over iterations (and optionally rounds) calling target, and return the last result.
def pedantic(
self,
target: Callable,
args: tuple = (),
kwargs: Optional[dict] = None,
iterations: int = 1,
rounds: int = 1,
warmup_rounds: int = 0,
setup: Optional[Callable] = None,
) -> Any:
"""Mimics the pedantic method of pytest-benchmark."""
if setup:
setup()
if kwargs is None:
kwargs = {}
- return target(*args, **kwargs)+ result = None+ for _ in range(iterations):+ result = target(*args, **kwargs)+ return result
Suggestion importance[1-10]: 6
__
Why: Looping over iterations in pedantic better mimics pytest-benchmark behavior and corrects the current single-call implementation.
Low
Support chained no-op calls
Allow chained calls on the custom benchmark by returning self instead of None from no-op methods. This supports patterns like benchmark.timer.start().stop().
def __getattr__(self, name):
- # Return a no-op callable for any attribute- return lambda *args, **kwargs: None+ # Return a no-op callable for any attribute that returns self+ return lambda *args, **kwargs: self
Suggestion importance[1-10]: 4
__
Why: Allowing __getattr__ to return self enables chained no-op calls but only marginally improves usability.
To better emulate pytest-benchmark’s pedantic, run the warmup, iterations, and rounds loops instead of calling the target only once. Aggregate and return the last result.
def pedantic(
self,
target: Callable,
args: tuple = (),
- kwargs: Optional[dict] = None, # noqa: FA100- iterations: int = 1, # noqa: ARG002- rounds: int = 1, # noqa: ARG002- warmup_rounds: int = 0, # noqa: ARG002- setup: Optional[Callable] = None, # noqa: FA100-) -> Any: # noqa: ANN401+ kwargs: Optional[dict] = None,+ iterations: int = 1,+ rounds: int = 1,+ warmup_rounds: int = 0,+ setup: Optional[Callable] = None,+) -> Any:
"""Mimics the pedantic method of pytest-benchmark."""
- if setup:- setup()
if kwargs is None:
kwargs = {}
- return target(*args, **kwargs)+ # warmup+ for _ in range(warmup_rounds):+ if setup:+ setup()+ target(*args, **kwargs)+ # timed loops+ result = None+ for _ in range(rounds):+ if setup:+ setup()+ for _ in range(iterations):+ result = target(*args, **kwargs)+ return result
Suggestion importance[1-10]: 7
__
Why: Adding the warmup, iterations, and rounds loops mirrors pytest-benchmark’s intended behavior and correctly applies to the existing pedantic definition without breaking other logic.
Medium
Use plugin manager to detect benchmark plugin
Rely on pytest’s plugin manager to detect the real benchmark plugin instead of inspecting request.fixturenames, which can cause recursion. This ensures you only override when the pytest-benchmark plugin is not installed.
-if "benchmark" in request.fixturenames and hasattr(request, "_fixturemanager"):- try:- # Try to get the real benchmark fixture- return request.getfixturevalue("benchmark")- except (pytest.FixtureLookupError, AttributeError):- pass+if not request.config.pluginmanager.hasplugin("benchmark"):+ return CustomBenchmark()+return request.getfixturevalue("benchmark")
Suggestion importance[1-10]: 6
__
Why: Relying on pluginmanager.hasplugin("benchmark") is a cleaner way to detect the real plugin, but the proposed snippet drops the existing exception fallback which could lead to uncaught errors if the fixture isn’t registered.
Why: Removing the request argument fixes a real TypeError and restores proper instantiation of DummyBenchmark.
High
General
Simulate iteration loops
Simulate the configured iterations and rounds to better mimic pytest-benchmark's behavior by looping the target call accordingly and returning the last result.
def pedantic(self, target: Callable, args: tuple = (), kwargs: dict = None,
iterations: int = 1, rounds: int = 1, warmup_rounds: int = 0,
setup: Callable = None) -> Any:
"""Mimics the pedantic method of pytest-benchmark."""
if setup:
setup()
if kwargs is None:
kwargs = {}
- return target(*args, **kwargs)+ result = None+ for _ in range(rounds):+ for _ in range(iterations):+ result = target(*args, **kwargs)+ return result
Suggestion importance[1-10]: 5
__
Why: Adding loops for iterations and rounds correctly improves fidelity to pytest-benchmark without introducing errors.
aseembits93
changed the title
Custom Benchmark fixture for running benchmark tests when the pytest-benchmark plugin is diabled
Custom Benchmark fixture for running benchmark tests when the pytest-benchmark plugin is disabled
Jun 9, 2025
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
aseembits93
changed the title
PR Type
Enhancement, Tests
Description
Introduce custom pytest benchmarking plugin
Provide no-op
benchmarkfixture implementationRegister
@pytest.mark.benchmarkpytest markerInclude plugin in test runners’ PYTEST_PLUGINS settings
Changes walkthrough 📝
test_benchmark_bubble_sort.py
Add benchmark-marked test casecode_to_optimize/tests/pytest/benchmarks/test_benchmark_bubble_sort.py
test_sort_with_markerusing@pytest.mark.benchmark@benchmarkcustom_plugin.py
Implement custom benchmark fixture plugincodeflash/benchmarking/plugin/custom_plugin.py
custom_plugin.pydefining a CustomBenchmark class"benchmark"inpytest_configure__call__,pedantic, and dynamic methodsgroup,name,fullname,params,extra_infotest_runner.py
Include custom benchmark plugin in test runnerscodeflash/verification/test_runner.py
PYTEST_PLUGINSto includecustom_pluginin three runnerstests