More --display log improvements

[rasmusfaber]

This PR contains:

• New features
• Changes to dev-tools e.g. CI config / github tooling
• Docs
• Bug fixes
• Code refactor

This further improves upon #1973.

The largest change is about the default log handlers. When using Inspect AI in a cloud setup, you really want to log in JSON format. But Inspect AI always adds log handlers, so you can't easily replace them.

What is the current behavior? (You can also link to an open issue here)

When using --display log, the final results were formatted using the Rich formatter instead of being fully native log-format. Also, the library always adds log handlers to the logging-context, which is impractical when using inspect-ai as a library from other code. And finally, when catching an exception during plan cleanup, the stacktrace was swallowed.

What is the new behavior?

Directly format the task results for display logs. Also log them as they come in instead of waiting until all tasks are complete.

Detect whether log handlers have already been added, and only add them if none have been added.

Add exc_info to the log call when handling plan cleanup exceptions.

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

The largest change here is about not adding log handlers if they have already been added. It should only affect users who use Inspect AI as a library and not as a tool, and I have a hard time imaging why you would add your own log handlers and want Inspect AI to add its own as well. But in principle, users who added their own log handlers and wanted to use Inspect AIs as well, would need to manually add the Inspect AI log handlers as well.

Other information:

You could argue that this should be three PRs. If you prefer that, just let me know, and I will split it out.

[jjallaire]

Thanks for this! Will review over the next few days. It's possible that I'll want to preserve some of our customization of our own logger (as the user can pass log_level to calls to eval which we'll minimally want to forward on to our own logger).

[rasmusfaber]

Thanks for this! Will review over the next few days. It's possible that I'll want to preserve some of our customization of our own logger (as the user can pass log_level to calls to eval which we'll minimally want to forward on to our own logger).

👍 Just let me know if there is something I can do. I guess you are mostly concerned about the change to not add the Inspect log handler. I also considered adding an explicit flag, or moving the the logger setup further out, so it was only done when calling Inspect with the CLI. If you prefer either of these solutions, just say so.

[celiawaggoner]

Hi @jjallaire - do you have an estimate for when you'll be able to review this PR? Thank you!

[jjallaire]

Hi @jjallaire - do you have an estimate for when you'll be able to review this PR? Thank you!

Yes, I should be able to do this by early next week.

[jjallaire]

The early return from init_logger() is my hangup here. The issue is that by returning early we disable a large number of features that people are relying on:

• Our own logger won't respect the --log-level specified for it by the user
• We will not capture any log entries in the transcript
• Trace logging (used to diagnose hangs and crashes) will be disabled
• External file logging (also used for debugging ) will be disabled

I do know that many people configure global loggers and then call eval() so I don't think disabling all of these features is required when the user has configured a global logger (AFAIK this currently works to people's expectations).

I think we need to go for something more surgical that targets exactly the behavior you are trying to prevent. For example, if a root logger is configured it might be enough to pass display_levelno=NOTSET to the LogHandler constructor (which will cause the Inspect logger to never print anything to the console but to still preserve all of the other behaviors detailed above). This would disable our printing to the console but presumably the user has configured logging to go where they want it to (including stderr if desired).