Skip to content

chore(observability): emit component_sent events by source and service #17549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 37 commits into from
Jun 26, 2023
Merged

chore(observability): emit component_sent events by source and service #17549

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
796def9
Add initial version of Cached
StephenWakely May 25, 2023
b205354
Remove the deadlock from writing to the cache after reading
StephenWakely May 26, 2023
d0c36ed
Insert source and service tag into events
StephenWakely May 26, 2023
ec666b0
Add bytesize count grouped by source and service
StephenWakely May 30, 2023
3e06791
Add EventCountTags trait
StephenWakely May 30, 2023
3a5fe02
Merge remote-tracking branch 'origin' into stephen/cached_events
StephenWakely May 31, 2023
15ea497
Fix merge
StephenWakely May 31, 2023
d66654c
Fix compile errors
StephenWakely May 31, 2023
d1c7df6
Merge remote-tracking branch 'origin' into stephen/cached_events
StephenWakely Jun 1, 2023
c895653
Set source and service tags for most other sinks
StephenWakely Jun 1, 2023
d28a809
Register the events with a trait rather than a Fn
StephenWakely Jun 2, 2023
4ade9be
These tests are round trip
StephenWakely Jun 2, 2023
3632fa2
Merge remote-tracking branch 'origin' into stephen/cached_events
StephenWakely Jun 5, 2023
4d9c5df
Clippy
StephenWakely Jun 5, 2023
c9640d9
Add event count tags for loki sink
StephenWakely Jun 5, 2023
a758704
RegisterEvent inherits form RegisterInternalEvent
StephenWakely Jun 6, 2023
5cbfee4
Merge remote-tracking branch 'origin' into stephen/cached_events
StephenWakely Jun 6, 2023
9eab386
Added telemetry options
StephenWakely Jun 7, 2023
ee50bf5
Only collect configured tags
StephenWakely Jun 8, 2023
9fd7854
Added tests and clippy.
StephenWakely Jun 8, 2023
01d8ad1
Little tidy
StephenWakely Jun 9, 2023
87ca474
TaggedEventsSent doesn't need Output
StephenWakely Jun 9, 2023
ebd10c8
Spelling
StephenWakely Jun 9, 2023
b30a4bb
Remove default impl of take_metadata
StephenWakely Jun 9, 2023
8b74470
Outer event should get tags from inner event type
StephenWakely Jun 9, 2023
b11c7a6
Driver should not consume the metadata
StephenWakely Jun 9, 2023
3327904
Merge remote-tracking branch 'origin' into stephen/cached_events
StephenWakely Jun 9, 2023
7a43045
Tags should be an associated type of RegisterEvent
StephenWakely Jun 9, 2023
f43af1b
Feedback from Bruce
StephenWakely Jun 15, 2023
b141f30
Merge remote-tracking branch 'origin' into stephen/cached_events
StephenWakely Jun 19, 2023
8ac7020
Set source tag to be Arc<ComponentKey>
StephenWakely Jun 19, 2023
243ec8b
Replace take_metadata with metadata_mut
StephenWakely Jun 21, 2023
574cbae
Add test for telemetry tags to Kafka sink
StephenWakely Jun 21, 2023
73131e6
Fix datadog integration test
StephenWakely Jun 21, 2023
4dee27c
Use Derivative to replace clone with no bounds
StephenWakely Jun 23, 2023
f1af663
Spelling
StephenWakely Jun 23, 2023
8f0cc1b
Feedback from Bruce
StephenWakely Jun 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion lib/vector-core/src/event/metric/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@ use std::{
};

use chrono::{DateTime, Utc};
use vector_common::EventDataEq;
use vector_common::{
request_metadata::{EventCountTags, GetEventCountTags},
EventDataEq,
};
use vector_config::configurable_component;

use crate::{
Expand Down Expand Up @@ -476,6 +479,15 @@ impl Finalizable for Metric {
}
}

impl GetEventCountTags for Metric {
fn get_tags(&self) -> EventCountTags {
let source = self.metadata().source_id().map(|output| output.to_string());
// Metrics do not contain a service field.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this an intentional choice? I don't see a real-life reason why metrics couldn't be associated with a service.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently only Log events can have schemas. Once we expand it so that a meaning can point to a metric tag then we will be able to extract the service here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to double check that we will be emitting the service tag for logs, metrics, and traces. If we need to follow up on metrics and traces we can do so, but I think we do need to do that before calling this project "finished".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't consider the service tag to be part of the schema - just thought we'd add a metric tag of service: foo


(source, None)
}
}

/// Metric kind.
///
/// Metrics can be either absolute of incremental. Absolute metrics represent a sort of "last write wins" scenario,
Expand Down
9 changes: 5 additions & 4 deletions src/sinks/statsd/service.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use tower::Service;
use vector_common::{
finalization::{EventFinalizers, EventStatus, Finalizable},
internal_event::CountByteSize,
request_metadata::{MetaDescriptive, RequestMetadata},
request_metadata::{MetaDescriptive, RequestCountByteSize, RequestMetadata},
};
use vector_core::stream::DriverResponse;

Expand All @@ -24,8 +24,8 @@ impl Finalizable for StatsdRequest {
}

impl MetaDescriptive for StatsdRequest {
fn get_metadata(&self) -> RequestMetadata {
self.metadata
fn get_metadata(&self) -> &RequestMetadata {
&self.metadata
}
}

Expand All @@ -46,11 +46,12 @@ impl DriverResponse for StatsdResponse {
EventStatus::Delivered
}

fn events_sent(&self) -> CountByteSize {
fn events_sent(&self) -> RequestCountByteSize {
CountByteSize(
self.metadata.event_count(),
self.metadata.events_byte_size(),
)
.into()
}

fn bytes_sent(&self) -> Option<usize> {
Expand Down