[cmd/opampsupervisor] Supervisor starts agent with invalid default config

[bacherfl]

Component(s)

cmd/opampsupervisor

What happened?

Description

The OpAMP supervisor currently attempts to start the agent with a config that is missing a pipeline, which leads to the agent process shutting down immediately, due to the invalid config. Subsequently, the supervisor attempts to start the process again periodically, with the same config.

I believe this happens because in the composeMergedConfig() function, there is a check if the message from the server contains at least one config:

opentelemetry-collector-contrib/cmd/opampsupervisor/supervisor/supervisor.go

Line 1113 in e4559ec

configMapIsEmpty := len(config.GetConfig().GetConfigMap()) == 0

If the length of that map is 0, the supervisor creates the noop config containing the minimal config for the collector to run.

However it seems like there is always an empty instance config in the message from the server, which is then skipped later: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/opampsupervisor/supervisor/supervisor.go#L1120-L1126

This will then result in the following config in the produced effective.yaml:

extensions:
    health_check:
        endpoint: localhost:65191
    opamp:
        capabilities:
            reports_available_components: false
        instance_uid: 01933f61-49d0-70f6-acc7-9da8ae71e609
        ppid: 61985
        ppid_poll_interval: 5s
        server:
            ws:
                endpoint: ws://127.0.0.1:65194/v1/opamp
                tls:
                    insecure: true
service:
    extensions:
        - health_check
        - opamp
    telemetry:
        logs:
            encoding: json
        metrics:
            readers:
                - periodic:
                    exporter:
                        otlp:
                            endpoint: http://localhost:4318/v1/metrics
                            protocol: http/protobuf
        resource:
            host.arch: arm64
            host.name: host
            os.description: macOS 15.3.2
            os.type: darwin
            service.instance.id: 01933f61-49d0-70f6-acc7-9da8ae71e609
            service.name: otelcontribcol
            service.version: 0.121.1-dev

Steps to Reproduce

Follow the setup described in the [README}(https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/opampsupervisor/README.md#experimenting-with-the-supervisor)

Expected Result

The collector agent process is either started with a noop pipeline, or not started until a valid configuration has been sent by the OpAMP server

Actual Result

Until a valid configuration is sent by the OpAMP server, the agent process periodically get started and shuts down

Collector version

0.121.1

Environment information

Environment

OS: macOS 15.3.2
Compiler(if manually compiled): go v1.23.7

[github-actions]

Pinging code owners for cmd/opampsupervisor: @evan-bradley @atoulme @tigrannajaryan. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[tpaschalis]

I can reproduce the behavior (hidden below for anyone interested since the instructions have changed since the issue was opened)

$ cd opamp-go/internal/examples/server
$ go run .
2025/04/04 23:51:11.990531 [MAIN] OpAMP Server starting...
2025/04/04 23:51:11.992361 [MAIN] OpAMP Server running...

# From the root of the opentelemetry-collector-contrib repo, run this command, or supply your own otelcontrib collector binary in bin/
$ make otelcontribcol

$ cd opentelemetry-collector-contrib/cmd/opampsupervisor
$ go run . --config examples/supervisor_darwin.yaml
{"level":"info","ts":1743799973.3658051,"logger":"supervisor","caller":"supervisor/supervisor.go:326","msg":"Supervisor starting","id":"0196028e-3bea-7dfa-ace1-bc35841218e7"}

After visiting http://localhost:4321/ I can see the configuration deployed on the collector and the error message

Agent process PID=94599 exited unexpectedly, exit code=1. Will restart in a bit...

The spec states that

The purpose of the "noop" configuration is to make sure the Collector starts and the opamp extension communicates with the Supervisor
The new config is also cached by the Supervisor in a local file, so that subsequent restarts no longer need to start the Collector using the "noop" configuration

My 2c: I don't see an issue with always including a noop configuration, even if there's already one available from the server.

I'd think the behavior of gracefully starting the collector and then subsequently validating or reporting any issues with it is preferrable than crashlooping the collector until the supervisor config is fixed.

[tpaschalis]

TIL of #38809, which might also be relevant

[douglascamata]

I have a similar problem, except that I am giving my Supervisor a valid Collector config via the below:

agent:
  executable: /otelcol-contrib

  # This passes config files to the Collector.
  config_files: 
    - /etc/otel/config.yaml

But the exact same code quoted by @bacherfl makes the Supervisor decide to NOT start my Collector. Effectively a Supervisor connecting to an OpAMP backend that doesn't send remote config is never able to start a Collector.

[douglascamata]

Note that the specification says:

If the remote configuration from the OpAMP Backend contains an empty config map,
the collector will be stopped and will not be run again until a non-empty config map
is received from the OpAMP Backend.

Which is exactly what happens. Unless the backend sends a config, the Supervisor won't start any Collector.

This even makes some configurations of the Supervisor prevent Collector's from starting (besides cases where the backend doesn't send configs that I previously mentioned), i.e. if you set capabilities.accepts_remote_config = false.