Add observation.metadata.choice_nodes

[tybug]

Closes (most of) #4387

• depends on Add PrimitiveProvider.on_observation #4416

To do:

• document the choice_nodes format (nonstandard nan patterns, large ints, etc)
• document observability.OBSERVABILITY_CHOICE_NODES
• work out a proper @settings for this? (as a follow-up?)

I'm also combining this with changes to BuildContext (for hypofuzz), so that this PR can be "the entirety of the changes necessary to make HypofuzzProvider work". I've split this out and have dropped it from this PR

[Zac-HD]

Do we have a concrete use-case in mind for the node constraints? If not, how about b64-encoded bytes format for the choice sequence? It'd be a fair bit smaller, and it looks like the human-readable version is already complicated enough that outside parties will want to use our code to deal with that.

Either way, I was thinking that we could have a flat list of [start, end, was_discarded, label] values, derived from the Spans class to track the tree structure of the input.

[Zac-HD]

IIRC we already store this in the top-level status_reason key.

[goes and checks] aaaaaannnd, we mishandle non-failing cases, oops.

[tybug]

yeah, INVALID and OVERRUN get collapsed together (which is probably-good as a library agnostic format)

[tybug]

sorry, I thought this was status, not status_reason. The actual reason I split this out is because hypofuzz wants the InterestingOrigin object itself, not the str(origin) version in status_reason.

[e: and this means the type here is wrong, fixing..]

[e2: nope, it is a string because we're documenting the file format here, not the in-memory format. Hmm, unfortunate that the two are similar but different...]

[tybug]

Do we have a concrete use-case in mind for the node constraints? If not, how about b64-encoded bytes format for the choice sequence? It'd be a fair bit smaller, and it looks like the human-readable version is already complicated enough that outside parties will want to use our code to deal with that.

Concrete use case is passing in-memory to hypofuzz, mostly for ordering corpus elements, cmd+f "choice_nodes" in https://github.com/Zac-HD/hypofuzz/pull/113/files.

I don't have a consumer of the serialized file format in mind, but my thinking is "if you're in a position to be using hypothesis utilities to transform the observations, you should just hook OBSERVABILITY_CALLBACKS and work with it in-memory (or serialize it yourself in your own format to a file). And if you're not in a position to use hypothesis utilities (working in another language etc), you'll greatly benefit from a human-readable format than having to replicate our choices_from_bytes function"

[Zac-HD]

Yeah, OK, fair enough. Feels inelegant but I think moving fast at the cost of inelegant formats is the right call for now.

[tybug]

[start, end, was_discarded, label

I went with [str(span.label), span.start, span.end, span.discarded]

[tybug]

data needs to be frozen to access data.spans, and at that point it seems reasonable to require that all datas passed here are frozen. I don't think freezing more than we used to at any of the callers is harmful, and anyway the whole point of .freeze is a phase-transition at which point it cannot be modified, which we definitely expect if we're reporting an observation for a data.

[Zac-HD]

very nearly done, I think

[Zac-HD]

Let's

• move the information-message schema first
• make this a subheading of the test-case schema
• consider pulling out the choice_nodes and choice_spans to a second subheading and collapsing <details> by default (it's a lot of very-niche-interest text)

[tybug]

I'm not sure sphinx has collapsible elements unless we install some third party extension 🤔

[Zac-HD]

It's a cute trick, the built-in only directive allows for arbitrary html:

.. only:: html

   .. raw:: html

      <details>
      <summary>Click to expand</summary>

Put whatever you want here, and in the HTML build only it'll be inside a details tag!

.. only:: html

   .. raw:: html

      </details>

[tybug]

happy to collapse for now, but TBH I think observability might deserve its own page at this point. (I think ghostwriter does as well). IMO a selling point of big api reference pages is that you don't have to worry about things like niche content or collapsing text

(which also makes cmd+f harder) TIL cmd+f autoexpands <details>

but still, collapsing makes it easy to miss, and this is a pretty important section for the right subset of people

[Zac-HD]

no, good argument, leave it uncollapsed and we'll move it later.