CLI Plugins Design

[ijc]

I am about to start work on a plugin mechanism for the Docker CLI and wanted to share the intended design here first.

Naming

The name plugin is already in long standing and well established use for engine side plugins and so a new name must be found for CLI plugins.

This document uses cli-plugin(s) throughout (e.g. for command names, packages, paths to search etc), but I expect that may change.

High-Level Overview

The Docker CLI will support a plugin/extension mechanism which should be familiar to users of git or kubectl or many other similar commands.

The top-level docker CLI will search a specific set of paths for binaries with names matching the regexp docker-[a-z][a-z0-9]* (that is, anything with a docker- prefix and one or more alphanumerics (lowercase) in the name, with a leading letter). For any such binary which is found and which is deemed to be acceptable a new top-level command will be exposed by the CLI so a binary named docker-foo would produce a foo command named at the top-level. This would then be callable as docker foo «further subcommands and options».

In this document when we refer to the “plugin name” we mean the unprefixed name unless otherwise specified (so “foo” in the example above).

Top-Level CLI Behaviour and Integration

Discovery and Validation of Installed Plugins

To enumerate all available plugins the following paths (decreasing order of priority) are scanned:

• Unix-like OSes :
  • $HOME/.docker/cli-plugins
  • /usr/local/lib/docker/cli-plugins & /usr/local/libexec/docker/cli-plugins
  • /usr/lib/docker/cli-plugins & /usr/libexec/docker/cli-plugins
• On Windows:
  • %USERPROFILE%\.docker\cli-plugins
  • C:\ProgramData\Docker\cli-plugins

(NB1: unlike git and similar this is not the same as $PATH, for our use cases we don't have the historical precedent of supporting invoking subcommands as docker-foo directly, and we have other preexisting commands, such as docker-compose, which will be in $PATH but are not (yet?) plugins)
(NB2: lib vs libexec represents a split in what distros use for this purpose)

Any files or symbolic links found which match the regex docker-.+ (basically anything at all with the docker- prefix) are considered potential plugin "candidates". Anything found which is not a regular file or symbolic link or which does not match the pattern is ignored and is not considered as a plugin candidate. On Windows files which do not have a .exe suffix are not considered candidates and are ignored.

To be considered a valid plugin a candidate must pass each of these “plugin candidate tests”:

• Must have a name which matches the regex docker-[a-z][a-z0-9]*.
  • (NB: The reason for having a broad candidate list of files matching docker-.+ and then restricting here to just docker-[a-z][a-z0-9]* (instead of having that be the candidate list to start with) is that it causes docker cli-plugins list (see below) to list them and the reason they are not valid, which is useful for the user)
• Must not have a name which conflicts with an existing builtin top-level command.
• Must, where relevant, have appropriate OS “execute” permissions (e.g. Unix x bit set) for the current user.
• Must actually be executed successfully and when executed with the subcommand docker-cli-plugin-metadata must produce a valid JSON document (and nothing else) on its standard output (schema to be discussed later).

A candidate in a higher priority directory path always shadows a candidate in a lower priority directory path, even if the higher priority candidate is found to be lacking in some way (e.g. is not executable).

Running a Plugin

When the Docker CLI is run with an unknown top-level command it will try each of the paths enumerated in “Discovery and Validation of Installed Plugins” looking for the first candidate plugin with the correct name (with the docker- prefix).

If no such candidate is found then this is considered the same as if a builtin subcommand is not found today. That is it will print an error message and exit with code 1.

$ docker unknowncommand
docker: 'unknowncommand' is not a docker command.
See 'docker --help'
$ echo $?
1

The candidate will then be subjected to the same set of “plugin candidate tests” as described above. If it fails any of these tests then this is considered an error and the CLI will exit with an appropriate error message. Note: This will include executing the plugin with the docker-cli-plugin-metadata subcommand in order to retain consistency with enumeration:

$ docker invalidplugin
CLI plugin “invalidplugin” is invalid: some reason
$ echo $?
1

At this point the plugin is executed, all parameters (including global parameters) will be passed in the same order as they appeared on the top-level CLI’s command line. This includes repeating the plugin name, so running docker --global foo --bar results in docker-foo --global foo --bar being executed.

Management of Plugins

No new commands will be added to the CLI binary for the management of CLI plugins. However some modifications will be made to existing top-level commands to accommodate them.

docker help (with no arguments) and docker --help

These existing top-level builtin commands will be modified to integrate valid plugins into the existing lists of builtin commands which they output. Plugins will be listed in the existing “Commands” category and not separated out into “plugins” (a list specifically of plugins can be obtained with “docker plugin ls”). The description will be taken from the plugin metadata.

The existing columnar listing of «command» «description» will be augmented with a new vendor column: «command» «vendor» «description» where the vendor column will either read Builtin or be
taken from the plugin metadata. The vendor will be truncated at 12 characters.

If there are any broken plugins then a new section Invalid plugins will be present in the docker help output which will list them and the reason they are invalid. If there are no invalid plugins then this section is omitted.

docker help «command»

If «command» is not a built-in command then this will be handled by looking up «command» as described in Running a Plugin and the invoking as docker-«command» help «command».

docker info

A list of valid plugins (including vendor, version and short description) will be incorporated into the "pretty" output. A warning will be issued for any invalid plugin, including the reason the plugin was considered invalid.

When using docker info -f «format» an array of all (valid and invalid) plugins will be included as a new CLIPlugin top-level entry. For valid plugins an entry will contain the full set of plugin metadata. For an invalid plugin only Name, Path and Err (the reason the plugin is invalid) are guaranteed be present.

Support for Installing Plugins

Users may install a new plugin by simply dropping the binary into an appropriate path (and, where necessary making it executable).

Requirements Placed on Plugins

A library will be provided which takes care of much of the boring detail of what is required here in order to avoid duplicating code in every plugin. See Plugin Support Framework/Libraries below.

The high-level requirements for a plugin are:

• All plugins must support the docker-cli-plugin-metadata subcommand as described in Discovery and Validation of Installed Plugins. The subcommand should be hidden (omitted from help and similar). The schema is described below in Plugin Metadata.
• All plugins must accept the global Docker CLI options and related environment variables and, where those options are relevant to its operation, honour them.
• All plugins should use the same configuration and authorization mechanisms as the top-level CLI (i.e. read from the same configuration files, honour the same top-level configuration options etc).
• Plugins are responsible for supporting different endpoint versions on servers they interact with (so e.g. are responsible for negotiating Docker API version themselves).

Plugin Metadata

When run with the docker-cli-plugin-metadata subcommand a plugin must produce a JSON document containing metadata about the plugin. This shall contain at least:

• Plugin Metadata Schema Version
• Name
• Version
• Author/Vendor
• URL
• Short description

The full schema will be determined (and documented) as part of the implementation work.

Configuration

The model to be used here is that all plugins (and the main CLI binary) have access to the same set of configuration files and each plugin will be able to read and use that configuration independently. Plugins will search for a config file in the same places as the monolithic CLI.

Plugins are expected to make use of existing global configuration where it makes sense and likewise to consider extending the global configuration where that is sensible. In particular for endpoint configuration they are expected to use the context store.

Where plugins unavoidably require specific configuration the .plugins.«name» key in config.json is reserved for their use.

Plugin Support Framework/Libraries

In order to make it easy to write plugins without reinventing numerous wheels and as an aid to consistency various Go helper packages will be provided. The need for some of these will only become apparent as work progresses to produce various plugins so it is expected that this will iterate.

Overall these helpers are expected to fall into two broad categories.

Plugin Specific

Some library functionality will be specific to the needs of plugins. These will live in github.com/docker/cli/plugin.

Functions will include:

func New(cmd cobra.Command, m Metadata)

This is intended to be call from a plugin's main function will take care of the scaffolding required to register the command, handling supporting the required docker-cli-plugin-metadata subcommand as well as the global arguments.

The cmd argument represents the plugins toplevel command (the name will thus be found in cmd.Use). The m argument will be a struct containing other useful metadata (versioning information etc)

CLI Specific

Some library functionality will not be especially specific to plugins but will be common to the main top-level CLI binary as well. For example:

• configuration file parsing, lookup of options by name.
• credential cache / registry authentication.
• contexts and endpoints (e.g. context store added by fast context switching, see ). In this context it is expected that the various server client libraries will provide integration with the context store, so that e.g. the same endpoint key names are used by all plugins (or builtins) which want to talk to a given service
• Table formatting.

It is expected that much of this code already exists within the current CLI but that it may not be structured in a way which makes it easily or cleanly usable by plugins, which basically amounts to ensuring that relevant bits of the CLI code are arranged as a cleanly consumable packages. Where necessary this refactoring will be done and/or new code written. Most likely ending up in subpackages of github.com/docker/cli details TBD as part of implementation but expect e.g. config and auth and similar.

[thaJeztah]

@ijc discussing this with @tonistiigi and @cpuguy83 and we're wondering what is done with global flags; are those passed to the plugin? (e.g. docker -H foobar <some plugin command>)

[andrewhsu]

Would be good to have a working public example or mock up of what this can help accomplish. To help walk through the process of the interaction between the cli and the plugin. It will help validate the interface provides the needed information.

[thaJeztah]

@ijc also (question from Tonis); what's the purpose of the docker cli-plugin ls and docker cli-plugin inspect commands? (given that there's no actual management commands (yet)?

If you want info about a plugin, docker <plugin> --help could provide information

[cpuguy83]

I'm interested in how is the thing that's plugging in going to benefit from being a plugin? What does the core docker CLI provide to it that makes it beneficial to be a plugin?

[ijc]

we're wondering what is done with global flags; are those passed to the plugin? (e.g. docker -H foobar )

The design states:

All parameters (including global parameters) will be passed in the same order as they appeared on the top-level CLI’s command line. This includes repeating the plugin name, so running docker --global foo --bar results in docker-foo --global foo --bar being executed.

---

Would be good to have a working public example or mock up of what this can help accomplish. To help walk through the process of the interaction between the cli and the plugin. It will help validate the interface provides the needed information.

The PR will have both real and mocked example plugins (needed for unit and e2e testing also). I think it would be counterproductive to spend a lot of time trying to nail down the specifics in the abstract at this phase, since there is lots we can and will learn during the implementation about the exact interactions. We should focus on getting the basics in, making use of them in real plugins and then iterating where we find issues.

---

what's the purpose of the docker cli-plugin ls and docker cli-plugin inspect commands? (given that there's no actual management commands (yet)?

I think ls is useful/necessary even in the absence of management commands, but in particular docker cli-plugin ls is also (uniquely) valuable in that it is the place which will include invalid plugins to help diagnose why an expected plugin is not showing up (e.g it will be where it says things like "not executable" or "invalid metadata").

While bunch of the metadata exposed by docker cli-plugin inspect could be exposed by docker <plugin> --help I think it is best to reserve the latter primarily for information on actual usage of the plugin rather than metainformation about the plugin.

---

I'm interested in how is the thing that's plugging in going to benefit from being a plugin? What does the core docker CLI provide to it that makes it beneficial to be a plugin?

For something like docker/app (which we intend to be one early consumer of this functionality) it means that it integrates into the CLI UX more nicely but without tying the development cycles of app and cli together. It's basically the same as 3rd party plugins to any similar system such as git or kubectl.

(edit: added breaks between the different questions, mainly because when rendered my quote of the document ran into my quote of Andrew's question in an unhelpfully confusing way)

[thaJeztah]

The design states

Ah. sorry for that; I was multi-tasking, and typing that during the maintainers meeting, and couldn't find it at a glance

[tonistiigi]

I think ls is useful/necessary even in the absence of management commands, but in particular docker cli-plugin ls is also (uniquely) valuable in that it is the place which will include invalid plugins to help diagnose why an expected plugin is not showing up (e.g it will be where it says things like "not executable" or "invalid metadata").

Presumably, we would list them in docker --help and plain docker anyway. And could show the errors there the same way.

While bunch of the metadata exposed by docker cli-plugin inspect could be exposed by docker --help I think it is best to reserve the latter primarily for information on actual usage of the plugin rather than metainformation about the plugin.

We could just put the metainformation in the beginning of <plugin> --help (if this info is needed at all).

It might we worth considering using the new dial-stdio command for providing the connection to the daemon in plugins to reduce the duplicate code and make sure all plugins always work with all connection types and are forward compatible in this regard.

[ijc]

We could just put the metainformation in the beginning of <plugin> --help

When I said "exposed by docker --help" I actually meant docker plugin --help, so I think my preference to keep that for the plugin usage and metadata elsewhere still applies, the metadata will mostly be a distraction to people who just want the help info.

(if this info is needed at all).

It has version and vendor info etc, so I think it is. It also lists the actual path to the plugin, plus any other paths which were shadowed by it, which is useful for diagnosing issues where a user isn't using the plugin they think they are.

It might we worth considering using the new dial-stdio command for providing the connection to the daemon in plugins to reduce the duplicate code and make sure all plugins always work with all connection types and are forward compatible in this regard.

Interesting, yes that seems like a useful thing to pursue. It would mostly just mean a wrapper in some library to help plugins with calling it consistently.

[tonistiigi]

the metadata will mostly be a distraction to people who just want the help info.

Or we could just make a convention that docker <plugin> --version shows the metadata if we want to keep it separate from --help. Doesn't seem that much different from docker cli-plugin inspect <plugin> from user's point of view.

[ijc]

I suppose we could, but inspect and ls are available for most other objects the CLI deals with and I'm not sure why CLI plugins should be special. Also by forcing it into some other output (like --help or --version) you would lose useful things which inspect gives you like machine readable JSON output, --format etc.

[tonistiigi]

I suppose we could, but inspect and ls are available for most other objects the CLI deals with and I'm not sure why CLI plugins should be special.

I guess the issue here is about why cli plugins are considered objects instead of like any other command that is just loaded differently. My assumption is that users would not make this distinction and would want to use them using same logic they use for other commands.

Also by forcing it into some other output (like --help or --version) you would lose useful things which inspect gives you like machine readable JSON output, --format etc.

docker version supports format as well and can provide json output so isn't a unique concept to inspect.

[ijc]

I guess the issue here is about why cli plugins are considered objects instead of like any other command that is just loaded differently.

The main difference is that the set of commands available as plugins is not fixed and in particular unlike a builtin command their installation/availability can be broken in subtle ways (e.g. by just failing to set the x permission bit). Treating them as objects allows users to inspect and diagnose this (and I don't expect users will interact with these commands very much at all unless something is not working for them).

My assumption is that users would not make this distinction and would want to use them using same logic they use for other commands.

For using them this is true a plugin foo is used with docker foo and it has help docker foo --help and docker help foo which tell you about how to use them, like any other command.

But aside from actually using them there are other operations on CLI plugins which I think are useful but not related to using them which shouldn't be conflated -- i.e. inspecting their properties and validating their availability for which I think it makes sense to treat them as objects.

Right now what I have is something like:

$ docker cli-plugin list
NAME                VENDOR              DESCRIPTION                            STATUS
badmeta                                                                        invalid metadata: invalid character 'i' looking for beginning of object key string
helloworld          Docker Inc.         A basic Hello World plugin for tests   ok
unexec                                                                         unable to execute: exit status 127
$ docker cli-plugin inspect helloworld
[
    {
        "Version": "0.1.0",
        "Vendor": "Docker Inc.",
        "ShortDescription": "A basic Hello World plugin for tests",
        "SchemaVersion": "",
        "Name": "helloworld",
        "Path": "/home/ijc/development/.../docker-helloworld"
        // If there were other `docker-helloworld`'s in the plugin search path, then
        // they would be listed here as being shadowed.
    }
]
$ docker help 
Usage:	docker [OPTIONS] COMMAND

A self-sufficient runtime for containers
... (nb: badmeta doesn't appear here because it is invalid) ...
  exec                      Run a command in a running container
  export                    Export a container's filesystem as a tar archive
  helloworld  (Docker Inc.) A basic Hello World plugin for tests
  history                   Show the history of an image
  images                    List images
... (nb: neither does unexec appear) ...
$ docker helloworld
Hello World!
$ docker help helloworld
A basic Hello World plugin for tests

Usage:
  docker helloworld [flags]

Flags:
  -h, --help   help for helloworld

This all seems quite straight forward and natural to me and is analogous to how other things work.
What are the downsides of treating them as objects? What problems or issues are we trying to avoid by not doing so?