Top N processes by CPU, memory, and disk

[strawgate]

Component(s)

receiver/hostmetrics/internal/scraper/processscraper

Is your feature request related to a problem? Please describe.

Enabling the process scraper with utilization, memory, disk, etc on my Mac results in 5k metrics every scrape interval

Describe the solution you'd like

The ability to have the scraper report metrics only by the top N processes by memory, CPU, or disk.

Describe alternatives you've considered

Using the metrics transform processor would allow combining some of the metrics but I'm not sure how you'd use it to get top N for a process

Additional context

No response

[VihasMakwana]

Sounds like a valid enhancement. I'll let codeowners share their thoughts.

[github-actions]

Pinging code owners for receiver/hostmetrics: @dmitryax @braydonk. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[braydonk]

The concern over the cardinality of these metrics makes a lot of sense, but I could see this feature being confusing for a lot of users depending on how the backend would want to handle something like this. The primary reason for this is that the top N could change over a period of time. If you're tracking timeseries for the top 10 processes and the process in 10th place changes, what it actually looks like from a timeseries perspective is that the series for the old 10th place process stopped and a new one started. Maybe some backends would handle this well, but if I imagine tracking top 10 over a period of time where that top 10 can fluctuate over a period of time I can imagine some confusing scenarios where the reasons for missing data on certain processes is ambiguous. I could be wrong on this, but my instinct is to recommend against this specifically. I am still interested in finding new ways to reign in the volume of metrics from the process scraper though as they can be quite excessive.

P.S. I don't think OTTL or metricstransformprocessor could do this aggregation today. May be a feature request for them if there is a more general use case for it.

[VihasMakwana]

@braydonk That's a great point. I believe this feature would be very valuable to have. For example, users might want to see results similar to htop. Currently, achieving that requires additional processing and results in a lot of redundant data.
We could consider exposing this functionality through configuration. Something like:

hostmetrics:
  collection_interval:  1m
  initial_delay:  1s
  scrappers:
    process:
       ...
       top_n.by: "memory"

I suggest keeping the current behavior as the default and documenting the things you've mentioned about using the new feature. This way, we can introduce the new feature without breaking compatibility for existing users.

[VihasMakwana]

cc: @dmitryax
would appreciate your thoughts on this

[VihasMakwana]

cc: @braydonk