Add a self-healthcheck capability to the otelcol-contrib bundle

[trajano]

Component(s)

No response

Is your feature request related to a problem? Please describe.

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/healthcheckv2extension provides a healthcheck extension that is accessed externally. Can the contrib image include a tool like curl or something fancier like grpcurl that does the healthcheck on the container itself?

Describe the solution you'd like

The Dockerfile that builds the otelcol-contrib image adds HEALTHCHECK and uses grpcurl to check the health of otelcol-contrib and the default config comes with healthcheckv2extension enabled.

The HEALTHCHECK should be disabled by default and enabled only if the environment variable SELF_HEALTHCHECK=1 is set (this ensures backwards compatibility)

Describe alternatives you've considered

Well the only other alternative I can think of is to make a custom otelcol-contrib image that adds grpcurl and the healthcheck

Additional context

No response

[trajano]

Actually instead of grpcurl you can add another command otelcol-contrib healthcheck --if-environment-set that would perform the check

[axw]

Actually instead of grpcurl you can add another command otelcol-contrib healthcheck --if-environment-set that would perform the check

Adding a new command makes sense to me. I could see a couple of different ways to go about this:

1. Enable extensions to define new sub-commands. e.g. you could add a command to the healthcheckv2 extension, and then add some new configuration the collector that tells it to define a new "healthcheck" command that uses the healthcheckv2 extension.
2. Just add a healthcheck command, which would internally use some code shared by the healthcheckv2 extension

Option 1 would be most powerful and flexible, but likely has a longer path to implementation, if others even think it's a good idea. Also, a challenge with that approach would be that the command would only be available in some configurations, which may complicate adding a container HEALTHCHECK - but maybe that's OK.

I'll tag the healthcheckv2 code owners and then we can see if others should be pulled in.

[github-actions]

Pinging code owners for extension/healthcheckv2: @mwear. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[trajano]

The advantage of curl or grpcurl approach is that it "avoids" changing otelcol-contrib too much and shows an example of testing it from outside. But whatever the approach a self-healthcheck would be good.