Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modify filter conditions does not work when filter is used as input processor #10168

Open
railsharipov opened this issue Apr 2, 2025 · 0 comments · May be fixed by #10172
Open

Modify filter conditions does not work when filter is used as input processor #10168

railsharipov opened this issue Apr 2, 2025 · 0 comments · May be fixed by #10172
Assignees
@niedbalski
Labels
status: waiting-for-triage

Comments

@railsharipov
Copy link

Bug Report

Describe the bug
The condition key seems to overlap with condition parameter for modify filter. This causes configuration errors when using modify filter as an input processor. I tried replacing modify filter with content_modifier processor but it does not support conditions such as key_exists. I have also noticed that when value field value is an empty string it causes content_modifier processor errors.

To Reproduce
Following input processor causes error messages during fluentbit init:

pipeline:
  inputs:
    - name: tail
      tag: ecs.*
      path: /var/lib/docker/containers/*/*-json.log
      processors:
        logs:
          - name: modify
            condition: key_does_not_exist container
            set: container unknown

This example results in following error logs:

Fluent Bit v4.0.0
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/


[2025/04/02 20:49:15] [error] [processor] condition must be a map
[2025/04/02 20:49:15] [error] failed to set condition for processor 'modify'
[2025/04/02 20:49:15] [error] failed to load 'logs' processors

I have tried replacing modify filter with content_modifier processor with following config:

pipeline:
  inputs:
    - name: tail
      tag: ecs.*
      path: /var/lib/docker/containers/*/*-json.log
      processors:
        logs:
          - name: content_modifier
            condition:
              op: and
              rules:
                - field: container
                  op: not_regex // not sure how else to check if field exists
                  value: ".+"
            key: container
            action: insert
            value: unknown

Unfortunately, last example does not add container key when it does not exist.

Expected behavior
Be able to use filter conditions when using modify filter as input processor.
Be able to add keys when key/field does not exists already.

Your Environment

  • Version used: 4.0.0
  • Configuration: YAML
  • Environment name and version (e.g. Kubernetes? What version?): AWS ECS
  • Operating System and version: Linux
  • Filters and plugins: Modify
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@niedbalski niedbalski

Labels
status: waiting-for-triage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

conditionals: fix handling of empty string values fluent/fluent-bit
2 participants
@niedbalski @railsharipov