[newrelic-logging] cant disable prometheusAPI metric from fluentbit

[KlockiLego]

Helm: newrelic-bundle - nri-bundle-5.0.116

After installing the latest version of new relic, a lot of metrics are collected, while I am only interested in logs (metric data ingest 98%, log data 2% .....)

I cannot disable prometheusAPI metric collection from fluent-bit. Here I see that there has been some change: Updated newrelic-logging to default send Metrics #1619

Metrics that are collected and cause huge data ingest consumption:

fluentbit_filter_drop_records_total
fluentbit_filter_bytes_total
fluentbit_input_storage_chunks_up
fluentbit_input_records_total
fluentbit_output_errors_total
fluentbit_input_files_opened_total
fluentbit_output_chunk_available_capacity_percent
....

[workato-integration]

https://new-relic.atlassian.net/browse/NR-382947

[KlockiLego]

This is documented here:
https://docs.newrelic.com/docs/logs/forward-logs/kubernetes-plugin-log-forwarding/#fluent-bit

Unfortunately the number of data ingest is very large and it should be possible to disable sending these metrics

[Tcohn1]

Thank you for the feedback. We are looking into this issue and will follow up with an update soon

[KlockiLego]

Workaround:
Use: nri-bundle-5.0.112 (with newrelic-logging-1.23.7)

[voorepreethi]

Hey @KlockiLego We have updated the scrape interval of Metrics from 10s to 60s. This will reduce the metrics count by 6. We are doing further analysis on priority and would get back to you .
Thank you

[KlockiLego]

Thanks, but still this is a lot. I have 12 nodes and monthly data ingest was around 120 GB. Now will be 20 GB.

Can you describe exactly what these metrics are and what they are for?
Can you add an option to completely disable it?

I feel like I don't need it, so I use nri-bundle-5.0.112 (with newrelic-logging-1.23.7)

[workato-integration]

Work has been completed on this issue.

[KlockiLego]

@voorepreethi, @Tcohn1 I think work is not completed....
This change/bug/feature caused that each node sends 1.66 GB data to new relics and client must pay for this data ingest.

[Tcohn1]

hey @KlockiLego , can you please provide additional details about your account so we can look into it. According to our estimation the additional GB for the metrics collected from fluent bit should be negligible.

Can you describe exactly what these metrics are and what they are for?

Thank you for the question. we are working on updating our docs to include these details.

Can you add an option to completely disable it?

The new UX is relying on this , therefore at this time disabling is not available.