Feature: Hooks #2057
Draft
Feature: Hooks #2057
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…(backslash) in f-strings on Python 3.10 (syntax was added in Python 3.12)".
| @@ -28,7 +27,6 @@ def init_eval_context( | |||
| init_logger(log_level, log_level_transcript) | |||
| init_concurrency() | |||
| init_max_subprocesses(max_subprocesses) | |||
| init_hooks() | |||
There was a problem hiding this comment.
We now init hooks in platform_init().
| @@ -20,6 +20,10 @@ def running_in_notebook() -> bool: | |||
|
|
|||
|
|
|||
| def platform_init() -> None: | |||
| from inspect_ai.hooks._startup import init_hooks | |||
|
|
|||
| init_hooks() | |||
There was a problem hiding this comment.
It is safe to call this multiple times:
- for "new" hooks there is a
_registry_hooks_loadedglobal - for "legacy" hooks, we only try loading them (and display message if loaded) if they're not already loaded
| async def on_run_start(self, data: RunStart) -> None: | ||
| """On run start. | ||
|
|
||
| A "run" is a single invocation of `eval()` or `eval_retry()` which may contain |
There was a problem hiding this comment.
Some of these docstrings might be worth double checking as I've made some claims about when these will be called (users will no doubt have questions otherwise).
| print( | ||
| f"[blue][bold]inspect_ai v{version}[/bold][/blue]\n" | ||
| f"[bright_black]{all_messages}[/bright_black]\n" | ||
| ) |
| Args: | ||
| data: Sample end data. | ||
| """ | ||
| pass |
There was a problem hiding this comment.
I'm unsure about whether I've picked the ideal specifics of the on_sample_* hooks:
on_sample_start: on every try (including retries)
on_sample_end: on a successful completion
on_sample_abort: when a sample errors and has no retries remaining
It feels weird that there could be a mismatch between the number of on_sample_start and on_sample_end events.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This functionality will support integrating observability/logging frameworks like Weights & Biases (#2046).
Requiring careful review:
emit_...functions from. There's quite a bit of nesting, recursion, try/excepts within_eval_async_inner()andtask_run_sample()so could do with a double check.Design decisions:
INSPECT_API_KEY_OVERRIDE) to prevent malicious packages from exfiltrating data? There's many other ways a malicious package can exfiltrate data. They can also just set the env var themselves. We'll print out which hooks are registered at startup (though a malicious hook could defer its registration).override_api_keynot async (but all other hooks are)? Because it is called fromModelAPI.__init__(). To make this async would require a bit of a rework.dataclassesrather than parameters on the hook functions? To protect ourselves against breaking changes when we inevitably want to pass additional data to a hook.Future work: