[filelogreceiver] Duplicate logs when rotating to compressed archive logfile

[zpeczoli]

Component(s)

receiver/filelog

What happened?

Description

Application is writing to uncompressed logfile, with log rotation configured so that when the log file is rotated it's also compressed. The OTel Collector logfile receiver is configured for both the uncompressed log file path and for the compressed "archive" log file paths.

Updates to the live log are processed correctly. However when a "live" log file reaches the size limit and is rotated and compressed, the new compressed logfile is detected as a completely new logfile, so all its contents are re-read and parsed again, resulting in lots of duplication.

Might be an issue with fingerprinting - if the rotated/compressed log file's fingerprint differs from the fingerprint of the original uncompressed file if the fingerprinting is done on the raw bytes of the compressed file, not on its uncompressed contents.

Steps to Reproduce

Set up application and configure log rotation with compression. Set up collector with filelog receiver for both the uncompressed and the rotated-compressed files. Wait for log rotation - observe that the entire rotated/compressed log file is re-read.

Expected Result

Only the last few lines are read from the rotated/compressed log file (whatever wasn't caught just before the rotation)

Actual Result

The entire rotated/compressed log file is re-read.

Collector version

v0.127.0

Environment information

Environment

OS: RHEL9.5

OpenTelemetry Collector configuration

receivers:
  filelog/uncompressed:
    include: [ logs/*.log ]
  filelog/compressed:
    include: [ logs/*.log.gz ]
    compression: gzip

exporters:
  debug:
    verbosity: basic

service:
  pipelines:
    logs:
      receivers: [ filelog/uncompressed, filelog/compressed ]
      processors: [ ]
      exporters: [ debug ]

Log output

2025-06-06T14:19:56.429Z        info    Logs    {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "resource logs": 1, "log records": 1}
2025-06-06T14:19:57.355Z        info    fileconsumer/file.go:267        Started watching file   {"resource": {}, "otelcol.component.id": "filelog/uncompressed", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "component": "fileconsumer", "path": "logs/test.log"}
2025-06-06T14:19:57.430Z        info    Logs    {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "resource logs": 1, "log records": 1}
2025-06-06T14:19:59.429Z        info    Logs    {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "resource logs": 1, "log records": 1}
2025-06-06T14:20:00.223Z        info    fileconsumer/file.go:267        Started watching file   {"resource": {}, "otelcol.component.id": "filelog/compressed", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "component": "fileconsumer", "path": "logs/test-2025-06-06.0.log.gz"}
2025-06-06T14:20:00.429Z        info    Logs    {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "resource logs": 1, "log records": 1}
2025-06-06T14:20:00.568Z        info    Logs    {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "resource logs": 1, "log records": 27}

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/filelog: @andrzej-stencel

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

@zpeczoli Hello!

You have configured two filelog receiver instances and both of them don't communicate with each other. If the new compressed log file matches against another receiver, it will be considered as a new file and all the contents will be read (even if you've set compression as gzip).

Might be an issue with fingerprinting - if the rotated/compressed log file's fingerprint differs from the fingerprint of the original uncompressed file if the fingerprinting is done on the raw bytes of the compressed file, not on its uncompressed contents.

@khushijain21 is currently working to fix this and you can track progress via #40256

[zpeczoli]

Thanks for pointing out the issue with the separate filelog receivers. I actually tried both ways (two filelog receivers; or a single one capturing both rotated and live files), but as the result was the same (because of the fingerprinting) I didn't think it mattered. I understand that with single receiver and Khushi's #40256 merged, it should work then.

[andrzej-stencel]

Thanks @zpeczoli for reporting this and thank you @VihasMakwana for the quick and spot on 🎯 reply!

Agree, Khushi's PR will fix this. Here's the configuration that currently (v0.127.0) causes log duplication, but after is #40256 merged, should work well (I suppose it will require enabling a feature gate, as it will be a breaking change):

receivers:
  filelog/all:
    include:
      - logs/*.log
      - logs/*.log.gz
    compression: auto

exporters:
  debug:
    verbosity: basic

service:
  pipelines:
    logs:
      receivers: [ filelog/all ]
      processors: [ ]
      exporters: [ debug ]

@VihasMakwana @khushijain21 keep me honest here 🙏