[filelog] Can't remove syslog parts from log messages - collector crashes

[driprado]

Component(s)

receiver/filelog

What happened?

Description

When using the remove operator after syslog_parser operator in filelog receiver, the collector panics with a nil pointer dereference. The goal is to remove specific syslog fields (PRI, VERSION, MSGID, STRUCTURED-DATA) from the output, but any attempt to use the remove operator results in a panic.

Steps to Reproduce

  1. Configure filelog receiver with syslog_parser operator
  2. Add remove operator to remove syslog fields
  3. Start the collector
  
  Configuration:
  ```yaml
  receivers:
    filelog/my-app-name:
      include_file_name: false
      resource:
        service.name: my-app-name
      storage: file_storage/filelog
      include:
      - "/var/log/folder/my-app-name/standard.log"
      operators:
      - id: syslog-parser
        type: syslog_parser
        protocol: rfc5424
      - id: remove-fields
        type: remove
        fields:
          - priority
          - version
          - msgid
          - structured_data
      retry_on_failure:
        enabled: true
  ```

Expected Result

  The remove operator should successfully remove specified syslog fields from the log output. For example, given this input:
  
  <139>1 2025-05-30T10:52:12+10:00 localhost my-app-name 1761 - -  #011at java.base/java.lang.Thread.run(Thread.java:840)
  
  It should output only the remaining fields:
  
  2025-05-30T10:52:12+10:00 localhost my-app-name 1761 #011at java.base/java.lang.Thread.run(Thread.java:840)

Actual Result

collector crashes with error:

May 30 11:26:56  stdout-docker_otel[32833]: panic: runtime error: invalid memory address or nil pointer dereference
May 30 11:26:56  stdout-docker_otel[32833]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x3c307ad]
May 30 11:26:56  stdout-docker_otel[32833]: goroutine 106 [running]:
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/entry.(*Entry).Delete(...)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/entry/entry.go:63
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/remove.(*Transformer).Transform(0xc0008c1520, 0x0?)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/operator/transformer/remove/transformer.go:41 +0x8d
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith(0xc0008c1520, {0x5695528, 0xc0005e9cc0}, 0xc000740e40, 0xc000ac5800)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/operator/helper/transformer.go:100 +0x48
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/remove.(*Transformer).Process(0x4e37760?, {0x5695528?, 0xc0005e9cc0?}, 0xc000acd590?)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/operator/transformer/remove/transformer.go:26 +0x35
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write(0xc0004cb400, {0x5695528, 0xc0005e9cc0}, 0xc000740e40)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/operator/helper/writer.go:73 +0x236
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback(0xc0004cb400, {0x5695528, 0xc0005e9cc0}, 0xc000740e40, 0xc000a3e988, 0x50b7620)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/operator/helper/parser.go:125 +0x10d
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/syslog.(*Parser).Process(0xc000f186b0?, {0x5695528?, 0xc0005e9cc0?}, 0x3c3b1e5?)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/operator/parser/syslog/parser.go:58 +0x148
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessBatchWith(0xc000a3ea68?, {0x5695528, 0xc0005e9cc0}, {0xc000042708, 0x64, 0xc000a3ea68?}, 0xc000ac5a68)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/operator/helper/transformer.go:84 +0x58
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/syslog.(*Parser).ProcessBatch(0x0?, {0x5695528?, 0xc0005e9cc0?}, {0xc000042708?, 0x0?, 0x0?})
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/operator/parser/syslog/parser.go:41 +0x3a
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).WriteBatch(0xc000e55a50, {0x5695528, 0xc0005e9cc0}, {0xc000042708, 0x64, 0x64})
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/operator/helper/writer.go:55 +0x294
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/file.(*Input).emitBatch(0xc000e55a40, {0x5695528, 0xc0005e9cc0}, {0xc000ade008?, 0x95?, 0xc000ad1fcc?}, 0x95?, 0x2034?)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/operator/input/file/input.go:49 +0xf2
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).readContents(0xc000e691d0, {0x5695528, 0xc0005e9cc0})
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/fileconsumer/internal/reader/reader.go:235 +0x429
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd(0xc000e691d0, {0x5695528, 0xc0005e9cc0})
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/fileconsumer/internal/reader/reader.go:117 +0x7e9
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1(0xc000e691d0)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/fileconsumer/file.go:169 +0xa5
May 30 11:26:56  stdout-docker_otel[32833]: created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume in goroutine 57
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.125.0/fileconsumer/file.go:166 +0x205
May 30 11:26:56  systemd[1]: docker-otel.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 30 11:26:56  systemd[1]: docker-otel.service: Unit entered failed state.
May 30 11:26:56  systemd[1]: docker-otel.service: Failed with result 'exit-code'.

Collector version

v0.113.0

Environment information

Environment

x86_64 x86_64 x86_64 GNU/Linux

OpenTelemetry Collector configuration

---
receivers:
  filelog/my-app-1:
    include_file_name: false
    resource:
      service.name: my-app-1
    storage: file_storage/filelog
    include:
    - "/var/log/apps/my-app-1/standard.log"
    operators:
    - id: syslog-parser
      type: syslog_parser
      protocol: rfc5424
    - id: remove-fields
      type: remove
      fields:
      - attributes.procid
      - attributes.msgid
      - attributes.structured_data
    retry_on_failure:
      enabled: true
processors:
  batch: {}
  memory_limiter:
    check_interval: 1s
    limit_mib: 100
  resource/loki:
    attributes:
    - action: insert
      key: loki.format
      value: raw
  attributes:
    actions:
    - key: source
      value: on-premise
      action: insert
    - key: env
      value: stg
      action: insert
    - key: host
      value: server-01
      action: insert
    - key: datacenter
      value: dc1
      action: insert
  attributes/loki:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: stream, source, env, host, datacenter, level
  attributes/stream_stdout:
    actions:
    - key: stream
      value: stdout
      action: insert
  deltatocumulative: {}
exporters:
  otlphttp/logs:
    logs_endpoint: https://logs.logging.stg.example.cloud/otlp/v1/logs
    sending_queue:
      storage: file_storage/queue
  loki:
    endpoint: https://logs.logging.stg.example.cloud/loki/api/v1/push
    sending_queue:
      storage: file_storage/queue
service:
  extensions:
  - health_check
  - file_storage/queue
  - file_storage/filelog
  telemetry:
    metrics:
      readers:
      - pull:
          exporter:
            prometheus:
              host: 0.0.0.0
              port: 9777
  pipelines:
    logs/my-app-1:
      receivers:
      - filelog/my-app-1
      processors:
      - memory_limiter
      - filter
      - batch
      - resource/loki
      - attributes/loki
      - attributes/stream_stdout
      - attributes
      exporters:
      - loki
      - count
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  file_storage/queue:
    directory: "/file_storage"
  file_storage/filelog:
    directory: "/file_storage"
    compaction:
      directory: "/file_storage/compact"
connectors:
  count: {}

Log output

Additional context

  - Have tried using both `field` and `fields` in remove operator
  - Have tried with `attributes.` prefix for field names
  - Have tried removing single fields instead of multiple
  - Other receivers in the same config work correctly
  - The syslog_parser operator works correctly on its own

[github-actions]

Pinging code owners:

• receiver/filelog: @andrzej-stencel

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[andrzej-stencel]

The remove operator does not have a fields property, it has and always had a singular field property.

To remove multiple fields, define multiple instances of the operator, like this:

receivers:
  filelog/my-app-1:
    include_file_name: false
    resource:
      service.name: my-app-1
    storage: file_storage/filelog
    include:
    - "/var/log/apps/my-app-1/standard.log"
    operators:
    - id: syslog-parser
      type: syslog_parser
      protocol: rfc5424
    - id: remove-procid
      type: remove
      field: attributes.procid
    - id: remove-msgid
      type: remove
      field: attributes.msgid
    - id: remove-structured-data
      type: remove
      field: attributes.structured_data
    retry_on_failure:
      enabled: true

The collector will log errors like the following when the field to be removed does not exist in a log:

2025-06-02T15:19:06.908+0200    error   helper/transformer.go:118       Failed to process entry {"resource": {}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "operator_id": "remove-procid", "operator_type": "remove", "error": "remove: field does not exist: attributes.procid", "action": "send", "entry.timestamp": "0001-01-01T00:00:00.000Z"}

To prevent this, set the operators' on_error property to send_quiet.

Having said all this, the collector should not panic when misconfigured; it should rather fail to start.

Let me know if this works for you @driprado.

[driprado]

Thank you for your response @andrzej-stencel

I have configured the operator as instructed:

receivers:
  filelog/my-app-1:
    include_file_name: false
    resource:
      service.name: my-app-1
    storage: file_storage/filelog
    include:
    - "/var/log/apps/my-app-1/standard.log"
    operators:
    - id: syslog-parser
      type: syslog_parser
      protocol: rfc5424
    - id: remove-msgid
      type: remove
      field: attributes.msgid
    retry_on_failure:
      enabled: true

And the collector started with no errors, however there were no changes to log output, and the msgid field has not been removed:

<139>1 2025-06-06T11:21:02+10:00 localhost my-app-1 1761 - - #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)