Skip to content

Fix all failing examples not having reproduction_decorator #4445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions hypothesis-python/RELEASE.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
RELEASE_TYPE: patch

Fixes ``reproduction_decorator`` being missing under :ref:`hypothesis-specific metadata <observability-hypothesis-metadata>` in many :ref:`observability <observability>` observations, when it should have been present.
2 changes: 2 additions & 0 deletions hypothesis-python/docs/reference/integrations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,8 @@ Test case
.. jsonschema:: ./schema_observations.json#/oneOf/0
:hide_key: /additionalProperties, /type

.. _observability-hypothesis-metadata:

Hypothesis metadata
+++++++++++++++++++

Expand Down
4 changes: 3 additions & 1 deletion hypothesis-python/src/hypothesis/internal/observability.py
Original file line number Diff line number Diff line change
Expand Up @@ -306,10 +306,12 @@ def make_testcase(

if status is not None and isinstance(status, Status):
status = status_map[status]
if status is None:
status = status_map[data.status]

return TestCaseObservation(
type="test_case",
status=status if status is not None else status_map[data.status],
status=status,
status_reason=status_reason,
representation=representation,
arguments={
Expand Down
24 changes: 24 additions & 0 deletions hypothesis-python/tests/cover/test_observability.py
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,30 @@ def test_fails(x, y):
assert observation.arguments == {"x": 1, "y": 0}


@pytest.mark.skipif(
PYPY or IN_COVERAGE_TESTS, reason="coverage requires sys.settrace pre-3.12"
)
def test_all_failing_observations_have_reproduction_decorator():
@given(st.integers())
def test_fails(x):
raise AssertionError

with capture_observations() as observations:
# NOTE: For compatibility with Python 3.9's LL(1)
# parser, this is written as a nested with-statement,
# instead of a compound one.
with pytest.raises(AssertionError):
test_fails()

# all failed test case observations should have reprodution_decorator
for observation in [
tc for tc in observations if tc.type == "test_case" and tc.status == "failed"
]:
decorator = observation.metadata.reproduction_decorator
assert decorator is not None
assert decorator.startswith("@reproduce_failure")


@settings(max_examples=20, stateful_step_count=5)
class UltraSimpleMachine(RuleBasedStateMachine):
value = 0
Expand Down