Add observation.metadata.choice_nodes
#4422
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| RELEASE_TYPE: patch | ||
|
|
||
| This release adds the experimental and unstable |OBSERVABILITY_CHOICES| option for :ref:`observability <observability>`, which includes the choice sequence in ``metadata.choice_nodes`` for test case observations if set. | ||
|
|
||
| We are actively working towards a better interface for this. Feel free to use |OBSERVABILITY_CHOICES| to experiment, but don't rely on it yet! | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| { | ||
| "description": "Hypothesis-specific values included in the ``metadata`` key of observations for test cases.", | ||
| "type": "object", | ||
| "properties": { | ||
| "traceback": { | ||
| "type": ["string", "null"], | ||
| "description": "The traceback for failing tests, if and only if ``status == \"failed\"``." | ||
| }, | ||
| "reproduction_decorator": { | ||
| "type": ["string", "null"], | ||
| "description": "The ``@reproduce_failure`` decorator string for failing tests, if and only if ``status == \"failed\"``." | ||
| }, | ||
| "predicates": { | ||
| "type": "object", | ||
| "description": "The number of times each |assume| and |@precondition| predicate was satisfied (``True``) and not satisfied (``False``).", | ||
| "additionalProperties": { | ||
| "type": "object", | ||
| "properties": { | ||
| "satisfied": { | ||
| "type": "integer", | ||
| "minimum": 0, | ||
| "description": "The number of times this predicate was satisfied (``True``)." | ||
| }, | ||
| "unsatisfied": { | ||
| "type": "integer", | ||
| "minimum": 0, | ||
| "description": "The number of times this predicate was not satisfied (``False``)." | ||
| } | ||
| }, | ||
| "required": ["satisfied", "unsatisfied"], | ||
| "additionalProperties": false | ||
| } | ||
| }, | ||
| "backend": { | ||
| "type": "object", | ||
| "description": "Backend-specific observations from |PrimitiveProvider.observe_test_case| and |PrimitiveProvider.observe_information_messages|." | ||
| }, | ||
| "sys.argv": { | ||
| "type": "array", | ||
| "items": {"type": "string"}, | ||
| "description": "The result of ``sys.argv``." | ||
| }, | ||
| "os.getpid()": { | ||
| "type": "integer", | ||
| "description": "The result of ``os.getpid()``." | ||
| }, | ||
| "imported_at": { | ||
| "type": "number", | ||
| "description": "The unix timestamp when Hypothesis was imported." | ||
| }, | ||
| "data_status": { | ||
| "type": "number", | ||
| "enum": [0, 1, 2, 3], | ||
| "description": "The internal status of the ConjectureData for this test case. The values are as follows: ``Status.OVERRUN = 0``, ``Status.INVALID = 1``, ``Status.VALID = 2``, and ``Status.INTERESTING = 3``." | ||
| }, | ||
| "interesting_origin": { | ||
| "type": ["string", "null"], | ||
| "description": "The internal ``InterestingOrigin`` object for failing tests, if and only if ``status == \"failed\"``. The ``traceback`` string value is derived from this object." | ||
| }, | ||
| "choice_nodes": { | ||
| "type": ["array", "null"], | ||
| "description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute may change format or disappear without warning.\n\nThe sequence of choices made during this test case. This includes the choice value, as well as its constraints and whether it was forced or not.\n\nOnly present if |OBSERVABILITY_CHOICES| is ``True``.\n\n.. note::\n\n The choice sequence is a relatively low-level implementation detail of Hypothesis, and is exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence.", | ||
| "items": { | ||
| "type": "object", | ||
| "properties": { | ||
| "type": { | ||
| "type": "string", | ||
| "enum": ["integer", "float", "string", "bytes", "boolean"], | ||
| "description": "The type of choice made. Corresponds to a call to |PrimitiveProvider.draw_integer|, |PrimitiveProvider.draw_float|, |PrimitiveProvider.draw_string|, |PrimitiveProvider.draw_bytes|, or |PrimitiveProvider.draw_boolean|." | ||
| }, | ||
| "value": { | ||
| "description": "The value of the choice. Corresponds to the value returned by a ``PrimitiveProvider.draw_*`` method.\n\n``NaN`` float values are returned as ``[\"float\", <float64_int_value>]``, to distinguish ``NaN`` floats with nonstandard bit patterns. Integers with ``abs(value) >= 2**63`` are returned as ``[\"integer\", str(value)]``, for compatibility with tools with integer size limitations. Bytes are returned as ``[\"bytes\", base64.b64encode(value)]``." | ||
| }, | ||
| "constraints": { | ||
| "type": "object", | ||
| "description": "The constraints for this choice. Corresponds to the constraints passed to a ``PrimitiveProvider.draw_*`` method. ``NaN`` float values, integers with ``abs(value) >= 2**63``, and byte values for constraints are transformed as for the ``value`` attribute." | ||
| }, | ||
| "was_forced": { | ||
| "type": "boolean", | ||
| "description": "Whether this choice was forced. As an implementation detail, Hypothesis occasionally requires that some choices take on a specific value, for instance to end generation of collection elements early for performance. These values are called \"forced\", and have ``was_forced = True``." | ||
| } | ||
| }, | ||
| "required": ["type", "value", "constraints", "was_forced"], | ||
| "additionalProperties": false | ||
| } | ||
| }, | ||
| "spans": { | ||
| "type": "array", | ||
| "items": {"type": "array"}, | ||
| "description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute may change format or disappear without warning.\n\nThe semantically-meaningful spans of the choice sequence of this test case.\n\nEach span has the format ``[label, start, end, discarded]``, where:\n\n* ``label`` is an opaque integer-value string shared by all spans drawn from a particular strategy.\n* ``start`` and ``end`` are indices into the choice sequence for this span, such that ``choices[start:end]`` are the corresponding choices.\n* ``discarded`` is a boolean indicating whether this span was discarded (see |PrimitiveProvider.span_end|).\n\nOnly present if |OBSERVABILITY_CHOICES| is ``True``.\n\n.. note::\n\n Spans are a relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| (and particularly |PrimitiveProvider.span_start| and |PrimitiveProvider.span_end|) for more details about spans." | ||
Zac-HD marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
| }, | ||
| "required": ["traceback", "reproduction_decorator", "predicates", "backend", "sys_argv", "os_getpid", "imported_at", "data_status", "interesting_origin", "choice_nodes"], | ||
| "additionalProperties": false | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.