[cmd/opampsupervisor] Add support for "base layer" local config for the Collector

[douglascamata]

Component(s)

cmd/opampsupervisor

Is your feature request related to a problem? Please describe.

I want to provide a base configuration, that can be overwritten by any other configuration source (the current local configuration implement is final and not overwrite-able), that the Supervisor can use to start the Collector without having to interact with any OpAMP backend to get remote configuration.

Describe the solution you'd like

We already got local configuration (proposed in #37886, implemented in #38671) support in the Supervisor's agent.config_files. These files are merged last into all other configuration, making it impossible to overwrite any configuration they do.

While enforcement of certain configuration above everything else is a very valid use case, I would like to propose that we also another use case: local configuration that can be used as a "base layer" on which to merge everything else.

It could be configured like this:

agent:
  base_config_files:
    - starting_config.yaml
  config_files: 
    - enforced_config.yaml

And I believe this "base config" should have the lowest priority in merging. In other words, it should be the very first to be merged into the config.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• cmd/opampsupervisor: @evan-bradley @atoulme @tigrannajaryan

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[tigrannajaryan]

I understand the use case, but I wonder if there is a way to design this differently so that it is easier to understand by the end users and harder to misuse. The proposed approach requires the user to read the docs and understand the merging order, which already is not easy and becomes even more complicated.

[douglascamata]

@tigrannajaryan that's already a problem we have today (see #39931, for instance) and one that I don't think will ever be avoided (but we can handle it better). We need very good documentation and logs regarding configuration merging. If possible we could also add this to the specification.

Part of the improvement I believe is better naming. That's why I propose the name base_config_files for this new configuration. I think it communicates well what it does and it's low priority. It could also be named low_priority_config_files.

The name config_files is bad in comparison, because it doesn't give you any clue about the priority of these configuration files. I would rather rename it to enforced_config_files, critical_config_files, top_level_config_files, high_priority_config_files, or something else on these lines that clearly communicates it overrides everything else.

[evan-bradley]

Could we use the order of the config_files key to specify the merge order, and provide special values the user can put to specify where that portion of the config will be merged? For example:

config_files:
- base_config.yaml
- $REMOTE_CONFIG
- security_policy_config.yaml
- $SUPERVISOR_EXTRA_CONFIG
- opampextension_additional_options.yaml

We can then document that config is merged top-to-bottom and we have some special keys that allow modifying the merge order of other config sources. With this the merging order is very clear and we don't have to introduce concepts outside the special values defined by the Supervisor.

[douglascamata]

I like this idea, @evan-bradley!

[douglascamata]

config_files:

• base_config.yaml
• $REMOTE_CONFIG
• security_policy_config.yaml
• $SUPERVISOR_EXTRA_CONFIG
• opampextension_additional_options.yaml

@evan-bradley curious question: what you then dump the configuration indicated by $SUPERVISOR_EXTRA_CONFIG and $REMOTE_CONFIG into a file and merge everything as files or keep those two special ones represented only in-memory?

I'll start working on a PoC for this. :)

[evan-bradley]

For now everything has to be in-memory and merged by the Supervisor. Once the Supervisor supports the confmap.enableMergeAppendOption feature gate, we will instead move to writing all files to disk and having the Collector merge them.