Profiling for specific process by labels, not all node processes

[lingpeng0314]

I am not sure if this feature is reasonable for this project, may be more proper when migrate this profiler into openTelemetry collector as profiling agent.

We plan to adopt continuous profiling for internal use. And the common use case is to select some specific pods to obtain profiling data instead of all kinds of processes in K8s node.

Invested several agents like Grafana Alloy/Parca/Pixie, all of them support the feature that only profiling specific pods. And as we move eBPF profiler into collector as agent, this should be a very reasonable requirement right?

[florianl]

As a sampling based profiler, eBPF profiler is designed and architected to profile the whole system. Filtering, e.g. on a label, can be applied on post processing stages, the backend or UI.
With the ongoing integration into OTel collector, more processors/filters should become available.

[lingpeng0314]

Look forward to the integration into OTel collector very much.

[lingpeng0314]

@florianl Do we have roadmap for when integrate the profiler into OTel collector?

[vjsamuel]

@florianl , to your comment on limiting it to post processing, it would add overhead on the node given it will profile every single process. is it not possible to filter down to specific pods?

[florianl]

Do we have roadmap for when integrate the profiler into OTel collector?

With the merge of open-telemetry/opentelemetry-collector-releases#908 one big step was achieved to make profiling part of OTel collector. Profiling is still considered not stable in Open Telemetry, as there is still work ongoing. As far as I know, there is no fixed roadmap.

[..] it would add overhead on the node given it will profile every single process. is it not possible to filter down to specific pods?

eBPF profiler is designed as whole system profiler. Benchmarks as real world deployments show, that it's overhead on profiling all processes is minimal and negligible. Adding filtering on the eBPF level would introduce some noticeable complexity and maybe race conditions.