buffer is partially-written/corrupted when multi-stage buffer enabled on high load

[mchtech]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When multi-stage buffers are enabled and the sink reaches its processing capacity limit, disk buffer becomes partially-written/corrupted.

Configuration

data_dir: /var/lib/vector

api:
  enabled: true
  address: 127.0.0.1:8686
  playground: false

sources:
  internal_metrics:
    type: internal_metrics
  input1:
    type: demo_logs
    count: 92233720368547760
    format: apache_common
    interval: 0.0
    lines:
      - 01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
  input2:
    type: demo_logs
    count: 92233720368547760
    format: apache_common
    interval: 0.0
    lines:
      - 01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789

sinks:
  null1:
    type: file
    path: /dev/null
    acknowledgements:
      enabled: true
    inputs:
      - input1
    encoding:
      codec: raw_message
    buffer:
      - type: memory
        max_events: 100
        when_full: overflow
      - type: disk
        max_size: 68719476736
        when_full: block
  null2:
    type: file
    path: /dev/null
    acknowledgements:
      enabled: true
    inputs:
      - input2
    encoding:
      codec: raw_message
    buffer:
      - type: memory
        max_events: 100
        when_full: overflow
      - type: disk
        max_size: 68719476736
        when_full: block
  prom:
    type: prometheus_exporter
    flush_period_secs: 600
    acknowledgements:
      enabled: false
    healthcheck:
      enabled: false
    inputs:
      - internal_metrics
    address: 0.0.0.0:9696

Version

0.46.1

Debug Output

2025-05-17T12:42:04.991427Z ERROR sink{component_kind="sink" component_id=null2 component_type=file}:sink{buffer_type="memory"}: vector_buffers::internal_events: Events dropped. count=32588 intentional=false reason=corrupted_events stage=1
2025-05-17T12:42:05.593998Z ERROR sink{component_kind="sink" component_id=null2 component_type=file}: vector_buffers::internal_events: Error encountered during buffer read. error=The reader detected that a data file contains a partially-written record. error_code="partial_write" error_type="reader_failed" stage="processing" internal_log_rate_limit=true
2025-05-17T12:42:06.990832Z ERROR sink{component_kind="sink" component_id=null1 component_type=file}:sink{buffer_type="memory"}: vector_buffers::internal_events: Events dropped. count=31487 intentional=false reason=corrupted_events stage=1
2025-05-17T12:42:44.992074Z ERROR sink{component_kind="sink" component_id=null2 component_type=file}:sink{buffer_type="memory"}: vector_buffers::internal_events: Events dropped. count=31884 intentional=false reason=corrupted_events stage=1
2025-05-17T12:42:46.711036Z ERROR sink{component_kind="sink" component_id=null2 component_type=file}: vector_buffers::internal_events: Error encountered during buffer read. error=failed to deserialize encoded record from buffer: record length was zero error_code="deser_failed" error_type="reader_failed" stage="processing" internal_log_rate_limit=true
2025-05-17T12:42:46.991007Z ERROR sink{component_kind="sink" component_id=null1 component_type=file}:sink{buffer_type="memory"}: vector_buffers::internal_events: Events dropped. count=33111 intentional=false reason=corrupted_events stage=1
2025-05-17T12:43:06.990452Z ERROR sink{component_kind="sink" component_id=null1 component_type=file}:sink{buffer_type="memory"}: vector_buffers::internal_events: Events dropped. count=32566 intentional=false reason=corrupted_events stage=1
2025-05-17T12:43:06.991820Z ERROR sink{component_kind="sink" component_id=null2 component_type=file}:sink{buffer_type="memory"}: vector_buffers::internal_events: Events dropped. count=18446744073698922162 intentional=false reason=corrupted_events stage=1
2025-05-17T12:43:08.991220Z ERROR sink{component_kind="sink" component_id=null1 component_type=file}:sink{buffer_type="memory"}: vector_buffers::internal_events: Events dropped. count=31685 intentional=false reason=corrupted_events stage=1
2025-05-17T12:43:08.991395Z ERROR sink{component_kind="sink" component_id=null2 component_type=file}:sink{buffer_type="memory"}: vector_buffers::internal_events: Events dropped. count=1559491 intentional=false reason=corrupted_events stage=1
2025-05-17T12:43:10.861617Z ERROR sink{component_kind="sink" component_id=null2 component_type=file}: vector_buffers::internal_events: Error encountered during buffer read. error=failed to deserialize encoded record from buffer: record length was zero error_code="deser_failed" error_type="reader_failed" stage="processing" internal_log_rate_limit=true
2025-05-17T12:43:10.990453Z ERROR sink{component_kind="sink" component_id=null1 component_type=file}:sink{buffer_type="memory"}: vector_buffers::internal_events: Events dropped. count=32841 intentional=false reason=corrupted_events stage=1

Example Data

No response

Additional Context

No response

References

similar: #19759