Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline Function to Remove Fields Based on Content #22200

Open
kingzacko1 opened this issue Apr 1, 2025 · 2 comments · May be fixed by #22205
Open

Pipeline Function to Remove Fields Based on Content #22200

kingzacko1 opened this issue Apr 1, 2025 · 2 comments · May be fixed by #22205
Assignees

Comments

@kingzacko1
Copy link
Contributor

kingzacko1 commented Apr 1, 2025

What?

As a user I would like to be able to easily drop all fields in a message that are either empty or otherwise not useful in order to reduce storage of unused fields. A common use case is an empty field being denoted by a - value. Ideally, a single rule could be used to drop any fields in a message with this value. For example:

rule "drop fields with - as content"
when
 true
then
 remove_fields_by_content("-");
end

Why?

Currently, this can be accomplished on a field by field basis with many rules that looks like this:

rule "drop field my_field if empty"
when
 to_string($message.my_field) == "-"
then
 remove_single_field("my_field");
end

Supporting this in a single pipeline rule would make this cleanup much more efficient.

See https://github.com/Graylog2/support/issues/202 for more context.

@waab76
Copy link
Contributor

waab76 commented Apr 1, 2025

I think we ought to have the @Graylog2/architecture team weigh in on this. I imagine it's more performant to do this in Java than to do it by having a pipeline rule for each possible field in a message, but I expect it still has the potential to be a performance concern to iterate over all of the fields in a message checking to see if they match a particular value.

@kingzacko1 kingzacko1 linked a pull request Apr 1, 2025 that will close this issue
9 tasks
@kingzacko1 kingzacko1 linked a pull request Apr 1, 2025 that will close this issue
9 tasks
@kingzacko1 kingzacko1 self-assigned this Apr 1, 2025
@kingzacko1
Copy link
Contributor Author

@waab76 agreed. Just made the issue for tracking discussion and wrote the code because it was a quick copy/tweak of the existing remove_multiple_fields rule.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants