fix(log): correctly handle errors when logging events #1040
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
This PR contains 2 separate but related fixes that will improve our on-call overhead. They would cause some errors to be logged in multiple unstructured lines, generating several unnecessary alerts.
The first problem was that a bug in string formatting would generate an exception during log event processing, triggering the "error when logging event" path. The second problem was with the logging of this error itself, that was using print statements, and therefore generating the multiple unstructured lines.
This finally resolves https://github.com/HathorNetwork/on-call-incidents/issues/153.
Acceptance Criteria
KeyError
to be raised during logging.print()
statements when errors occur during logging.Checklist
master
, confirm this code is production-ready and can be included in future releases as soon as it gets merged