[telemetry] Detect service rechability issues #3426
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add time taken to receive the first response (headers) from the user service - Add Time taken to replay the journal
This was referenced Jun 19, 2025
|
|
pcholakov
approved these changes
Jun 19, 2025
There was a problem hiding this comment.
This is a great observability addition, thank you @muhamadazmy! Left one minor naming comment which you should feel free to ignore.
| pub const INVOKER_TASKS_IN_FLIGHT: &str = "restate.invoker.inflight_tasks"; | ||
| pub const INVOKER_JOURNAL_REPLAY_TIME: &str = "restate.invoker.journal_replay_time.seconds"; | ||
| pub const INVOKER_SERVICE_DOWN_ERRORS: &str = "restate.invoker.service_down_errors.total"; |
There was a problem hiding this comment.
Nitpicky naming observation: Wondering if "unavailable" or "unreachable" might not be more accurate than "down" - since we can't tell authoritatively that it's really down, just that it's not available from our point of view.
There was a problem hiding this comment.
Ah very nice. Thank you. Will apply :)
Summary: Introducing a counter for number of "rechability" issues for a service that can detect a service is down or un-responsive by visualizing `rate(restate.invoker.service_unreachable_errors.total)` coupled with alerts, operator can know when a service is facing connectivity problems
| @@ -21,6 +21,8 @@ pub const INVOKER_TASK_DURATION: &str = "restate.invoker.task_duration.seconds"; | |||
| pub const INVOKER_SERVICE_RESPONSE_TIME: &str = "restate.invoker.service_response_time.seconds"; | |||
| pub const INVOKER_TASKS_IN_FLIGHT: &str = "restate.invoker.inflight_tasks"; | |||
| pub const INVOKER_JOURNAL_REPLAY_TIME: &str = "restate.invoker.journal_replay_time.seconds"; | |||
| pub const INVOKER_SERVICE_UNREACHABLE_ERRORS: &str = | |||
| "restate.invoker.service_unreachable_errors.total"; | |||
There was a problem hiding this comment.
I guess you mean deployment
| @@ -1090,6 +1090,11 @@ where | |||
| .remove_invocation_with_epoch(partition, &invocation_id, invocation_epoch) | |||
| { | |||
| debug_assert_eq!(invocation_epoch, ism.invocation_epoch); | |||
|
|
|||
| if self.is_service_down_error(&error) { | |||
| counter!(INVOKER_SERVICE_UNREACHABLE_ERRORS, "service" => ism.invocation_target.service_name().to_string()).increment(1); | |||
There was a problem hiding this comment.
I think we need to have both the deployment id and the service name. The risk is that it this will be a high cardinality metric.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[telemetry] Detect service rechability issues
Summary:
Introducing a counter for number of "rechability" issues for a service
that can detect a service is down or un-responsive
by visualizing
rate(restate.invoker.service_unreachable_errors.total)coupled with alerts, operatorcan know when a service is facing connectivity problems
Stack created with Sapling. Best reviewed with ReviewStack.