vector_component_sent_event_bytes_total not increased on sink shutdown flush

[k24dizzle]

Problem

At 2023-10-28T05:19:56.964090Z I terminated my aggregator pod which has a disk buffer.

According to vector_component_sent_event_bytes_total, there was no s3 file uploaded for ~2 minutes while the pod was restarting. When the pod starts up again, I see it increasing once more.

However in s3, I do see an object uploaded at 05:20:47 UTC.

Because of this I believe vector_component_sent_event_bytes_total does not increment properly when a buffer is flushed during a pod termination.

Configuration

    sources:
      internal_metrics:
        type: internal_metrics
      logs:
        address: 0.0.0.0:6000
        type: vector
        version: "2"
    sinks:
      logs_to_s3:
        batch:
          max_bytes: 1073741824
          timeout_secs: 60
        bucket: bucket_name
        buffer:
          max_size: 1073741824
          type: disk
        compression: gzip
        encoding:
          codec: text
        inputs:
        - logs
        region: us-west-2
        storage_class: INTELLIGENT_TIERING
        type: aws_s3
      prom-exporter:
        address: 0.0.0.0:9090
        inputs:
        - internal_metrics
        type: prometheus_exporter

Version

0.32.1

Debug Output

2023-10-28T05:19:56.964090Z  INFO vector::signal: Signal received. signal="SIGTERM"
2023-10-28T05:19:56.964154Z  INFO vector: Vector has stopped.
2023-10-28T05:19:56.964227Z DEBUG source{component_kind="source" component_id=internal_metrics component_type=internal_metrics component_name=internal_metrics}: vector::topology::builder: Source finished normally.
2023-10-28T05:19:56.964250Z DEBUG source{component_kind="source" component_id=internal_metrics component_type=internal_metrics component_name=internal_metrics}: vector::topology::builder: Source pump finished normally.
2023-10-28T05:19:56.964272Z DEBUG source{component_kind="source" component_id=internal_metrics component_type=internal_metrics component_name=internal_metrics}: vector::topology::builder: Source pump supervisor task finished normally.
2023-10-28T05:19:56.964322Z DEBUG sink{component_kind="sink" component_id=prom-exporter component_type=prometheus_exporter component_name=prom-exporter}: vector::topology::builder: Sink finished normally.
2023-10-28T05:19:56.964941Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="logs_to_s3, logs, get_start_time_prefix" time_remaining="59 seconds left"
2023-10-28T05:19:56.973407Z DEBUG source{component_kind="source" component_id=logs component_type=vector component_name=logs}: vector::topology::builder: Source pump finished normally.
2023-10-28T05:19:56.973407Z DEBUG source{component_kind="source" component_id=logs component_type=vector component_name=logs}: vector::topology::builder: Source finished normally.
2023-10-28T05:19:56.973451Z DEBUG source{component_kind="source" component_id=logs component_type=vector component_name=logs}: vector::topology::builder: Source pump supervisor task finished normally.
2023-10-28T05:19:56.973507Z DEBUG transform{component_kind="transform" component_id=get_start_time_prefix component_type=remap component_name=get_start_time_prefix}: vector_buffers::variants::disk_v2::writer: Writer marked as closed.
2023-10-28T05:19:56.973538Z DEBUG transform{component_kind="transform" component_id=get_start_time_prefix component_type=remap component_name=get_start_time_prefix}: vector::topology::builder: Synchronous transform finished normally.
2023-10-28T05:19:59.343453Z DEBUG sink{component_kind="sink" component_id=logs_to_s3 component_type=aws_s3 component_name=logs_to_s3}: vector::utilization: utilization=0.005912340238561226
,,,
2023-10-28T05:20:41.965050Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="logs_to_s3" time_remaining="14 seconds left"
2023-10-28T05:20:44.343390Z DEBUG sink{component_kind="sink" component_id=logs_to_s3 component_type=aws_s3 component_name=logs_to_s3}: vector::utilization: utilization=0.000000028169736919135196
2023-10-28T05:20:46.965115Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="logs_to_s3" time_remaining="9 seconds left"
2023-10-28T05:20:49.347900Z DEBUG sink{component_kind="sink" component_id=logs_to_s3 component_type=aws_s3 component_name=logs_to_s3}: vector::topology::builder: Sink finished normally.
Stream closed EOF for monitoring/aggregator-0 (vector)

References

#11405