Collector Plug-in system to allow for plug-ins to be dynamically loaded

[markcartertm]

Is your feature request related to a problem? Please describe.

Currently, all collector plug-ins such as producers or exporters need to be statically compiled into the collector. this results in a small number of baseline plug-ins such as Prometheus and Jaeger being built-in to the collector and all other plug-ins to be optional. there is also a collector build that includes all plug-ins, but given the expected growth of the project this will result in an unnecessary large collector binary. unless we add the ability to dynamically add and remove plug-ins through configuration, OTEL customers and vendors will be forced to create custom distributions of OTEL collector with their own specific combination of plug-ins. this will eliminate one of the core value propositions of OTEL which is instrument once and support one or more backends or sources.
One of the main challenges is that the collector is written in Go and Go does not offer native plug-in system.

Describe the solution you'd like
A clear and concise description of what you want to happen.

I suggest implementing a plug-in system using network RPC and dynamic loading similar to HashiCorp implementation of plug-ins in Terraform:
https://www.terraform.io/docs/internals/internal-plugins.html

The expected resulting experience will be:
1.User deploys baseline collector binary together with any set of additional receiver / exporter / processor plug-ins. plug-ins binaries will be placed in /plugins relative to collector binary.
2. When collector process initialize, it will load configuration from file or based on init parameters / environment variable
3. Collector will initialize the plug-ins listed in the configuration file and will communicate with them over network RPC.
4. Optional - if plugin_remote_fetch is set to true in the collector config, collector will do an http get from OpenTelemetry registry for any plug-in listed in the config file that is not available locally
5. optional - if plugin_auto_update is set to true in the collector config, the collector process will compare the version of any plug-in against the latest version available in the OTEL registry and will download newer version if available.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Other options are painful:

1. Only support statically compiled plug-ins resulting in multiple incompatible collector distributions with arbitrary plug-in support or a huge collector with all plug-ins
2. Wait for Go to offer a stable plug-in system - Google indicates that this is not likely in teh 2020 timeline
3. use GRPC for the collector to communicate with plug-ins running as a standalone processes - concern is performance under stress and resulting memory and CPU
4. Use web assembly and TinyGo to dynamically load collector plug-ins - this is an interesting option that requires more input from someone with deeper knowledge.
   see https://github.com/tinygo-org/tinygo

Would appreciate feedback on this proposal so we can drive it to a design and implementation before collector V1 GA.

[markcartertm]

@mtwo can you please have someone from Google review and provide feedback?
@tigrannajaryan would appreciate your feedback

[jrcamp]

Currently, all collector plug-ins such as producers or exporters need to be statically compiled into the collector. this results in a small number of baseline plug-ins such as Prometheus and Jaeger being built-in to the collector and all other plug-ins to be optional. there is also a collector build that includes all plug-ins, but given the expected growth of the project this will result in an unnecessary large collector binary.

A recent build of contrib is 87M (33M gziped). This doesn't seem too unreasonable right now. At what size would a monolithic build be a problem? As you say, it's only going to grow but is that a problem in 2020 or 2025 when go might have a native plugin solution?

The plugin system has some appeal but definitely some drawbacks:

• Development and debugging complexity
• Build complexity
• Size issues due to duplicated libraries/go runtime across all standalone plugins
• Deployment concerns
  • I imagine most production environments don't want arbitrary binaries downloaded at runtime.
  • For firewalled environments you would then have to be able to setup an internal download registry--sounds complicated.
• If there were a plugin system would there still be a need for a monolithic build anyway to satisfy some of the deployment concerns? Otherwise if you provide an uber build with all the plugins there might be even greater bloat due to duplicating libraries across each plugin build.

Overall the cons outweigh the pros IMO.

[mtwo]

I'm going to follow up on #2 within Google, as Go plugins are already available and supported. I have found a few drawbacks with the current implementation of Go plugins:

• It appears that plugins and the main repo need to be from the same repository and versioned together. This probably isn't an issue for us, though it isn't ideal.
• Go plugins don't work on Windows. This is bad, though we could just publish the full Collector on Windows. Windows hosts typically have fewer performance constraints as they're not used to run functions as a service platforms, small devices, or other more constrained environments.
• Plugins can crash the host process (because they run in the same process). This is bad.
• Plugins can't be unloaded once loaded.

I'll find out what's planned for Go plugins, as using an RPC system for our purposes seems odd. In the meantime, I agree with Jay - we don't have that many Collector extensions yet and it seems fine to just embed all of them until we find a good dynamic loading solution.

[tigrannajaryan]

We spent considerable effort to make Collector extensible and customizable without the need to have dynamic plugins system. For constrained environments it is possible to trivially create a custom Collector build that includes only the necessary components. I have a feeling that this addresses some of the use cases for plugins.

That said I do not want to discourage anyone from trying to implement a plugins system if there is a good use case for it. Anyone who wishes to undertake this task and experiment can do so today. I think it is possible to implement "plugin_loader" Receiver, Exporter, Processor and Extension components, which will be statically compiled with the Collector and will be responsible for runtime loading of external plugins. If there is an external contributor who would like to try and experiment with this approach they can do it independently on a custom build of the Collector.

I would not recommend to make such plugins system a standard part of the Collector yet, but we should consider it in the future if:

• The implementation is high-quality and robust, and
• There is a considerable demand for runtime-loadable plugins.

I encourage everyone who is interested in having runtime-loadable plugins to vote for this issue by giving a +1 to Mark's post at the top of this thread.

[mtwo]

@rakyll reached out and suggested that we should not rely on Go's plugin system. In that case my log-term preference is suggestion # 3 (plugins communicating with the Collector over gRPC or some other RPC system). In the short-term, I think that # 1 (all plugins or a subset of plugins are compiled in) will suffice.

Dynamically loading Collector plugins isn't a near-term priority for Google, but we're happy to review or advise on proposals.

[jmacd]

@raykll or @mtwo I would appreciate some insight into why you think that the plugin system should not be used?

I used the plugin mechanism to prototype dynamic loading of the SDK in OTel-Go and it works quite well. The only issue I am aware of is that it is not supported on Windows, and each year the Go developer survey comes out and I ask them for Windows support because I think it would be nice if we could use plugins in OTel.

Here is my branch: jmacd/opentelemetry-go#1

[mtwo]

@rakyll can speak to it on better authority than myself, but her biggest concern is that nobody within the Go community is actively working on the plugin system

…

________________________________ From: Joshua MacDonald <[email protected]> Sent: Friday, June 12, 2020 12:23:09 PM To: open-telemetry/opentelemetry-collector <[email protected]> Cc: Morgan McLean <[email protected]>; Mention <[email protected]> Subject: Re: [open-telemetry/opentelemetry-collector] Collector Plug-in system to allow for plug-ins to be dynamically loaded (#1005) @raykll or @mtwo<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmtwo&data=02%7C01%7C%7C41fe71552e7e43ff58fc08d80f060aeb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637275865904596119&sdata=Nbm86mLXI7fkfQmQorI6nyc%2FX%2FV%2ByxwiqzIzJ1USf1U%3D&reserved=0> I would appreciate some insight into why you think that the plugin system should not be used? I used the plugin mechanism to prototype dynamic loading of the SDK in OTel-Go and it works quite well. The only issue I am aware of is that it is not supported on Windows, and each year the Go developer survey comes out and I ask them for Windows support because I think it would be nice if we could use plugins in OTel. Here is my branch: jmacd/opentelemetry-go#1<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjmacd%2Fopentelemetry-go%2Fpull%2F1&data=02%7C01%7C%7C41fe71552e7e43ff58fc08d80f060aeb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637275865904606116&sdata=xAdt8%2B1v2rU3olS385Ue2XDzohQKUU%2Fy9lPXdxOQZMk%3D&reserved=0> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-telemetry%2Fopentelemetry-collector%2Fissues%2F1005%23issuecomment-643444966&data=02%7C01%7C%7C41fe71552e7e43ff58fc08d80f060aeb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637275865904616115&sdata=AiGrgJaH3E%2B35Wcl5A89zgaHsCAqs%2B4tKiO0O8p3ZGI%3D&reserved=0>, or unsubscribe<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAIXLK5Y5DXXF4GXNFLAVW3RWJ6B3ANCNFSM4NGLD4NQ&data=02%7C01%7C%7C41fe71552e7e43ff58fc08d80f060aeb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637275865904626106&sdata=bmga8%2BAGevIKEbIhy%2FtTUGp5%2FKNKhwq5AfTAN1tQTvI%3D&reserved=0>.

[sargun]

what is the suggestion for people who have proprietary metrics systems / exporters? Right now, it requires forking contrib, and editing the main.go (or dropping a new file into it). Alternatively, I feel like we already have a "plugin" format for metrics (the OT Protobuf format). Is there appetite for being able to point the file exporter to a TCP / Unix domain socket?

[tigrannajaryan]

what is the suggestion for people who have proprietary metrics systems / exporters?

@sargun one way is to implement a proprietary format exporter and add to https://github.com/open-telemetry/opentelemetry-collector-contrib/ repo.

[rakyll]

Is there anything to investigate on this issue? Unfortunately, Russ Cox is not recommending to use the plugins at this point. We can suggest people to build their own custom binaries or implementing a custom exporter as @tigrannajaryan suggests -- depending on the use case and the capabilities needed.

[mtwo]

I don't think that anything is underway, though I could be incorrect. You're likely the leading Go expert in the OT community, so I'm guessing that we'll defer to your judgement.

[sargun]

The code is relatively easy to consume as a module, and as long as it stays that way, you can just copy the bits to register the built in receivers / processors / exporters.

But, I think that the hashicorp grpc style plugin system is kind of ideal compared to the current situation.

[alolita]

@rakyll in my discussions with @jmacd about handling dynamic loading for plug-ins, we evaluated options that may exist in GoLang and did not find any stable implementations. The closest suggestion we had was to extend https://github.com/lightstep/otel-launcher-go which needs further work. Look forward to your suggestions on this.

[rakyll]

(Seeing this quite a while after my first comment and realized that I haven't left a good summary of what's missing in the Go plugin system hence adding some more context for historical records.)

Go plugin system was initially contributed to upstream but has never been finished, and it is not maintained at this point. It's not working on Windows and never been production-ready and it's never been recommend for use.

There are a lot of limitations in the plugin system:

• There is no good cross-platform support.
• We're not sure how Go plugins work with the Go modules, plugins required all packages to rely on the same version of the same dependency. All non standard library dependencies are hard to deal with.
• The version you compile a plugin must exactly match the compiler version used to build the main program, this includes minor versions too. 1.12.1 and 1.12.2 are not same versions.
• It requires CGO which is something we avoid right now and it helps to keep the collector dependencies an almost zero.
• If you need CGO in a plugin, it's a huge headache to deal with compilers and linkers.

Splitting things into separate processes/services is the only way to have some flexibility but it comes with a huge performance penalty and should be secondary choice.

[bogdandrutu]

We created a "builder" that allows users to easily create smaller custom builds.