Adding custom plugins to ROBOT

[gouttegd]

This PR adds the possibility of using pluggable commands, or commands that are not part of the built-in set of commands but are instead provided by third-party plugins.

Rationale

Reasons to enable the use of plugins include:

• fast prototyping and testing of possible future built-in commands (a new command could be first distributed as a plugin, without requiring a ROBOT release, before being ultimately co-opted to become a built-in command);
• allowing the creation and use of “niche” commands that someone may need, but that have a scope too narrow to warrant being included a built-in commands;
• lowering the bar for using Java to create specialized OBO manipulation tools (by delegating command line parsing and I/O operations to ROBOT, and focusing instead solely on the task at hand);
• simplifying and improving complex pipelines that currently require alternating between ROBOT and other tools, if those other tools could be replaced by ROBOT (pluggable) commands.

Proposed behaviour

This PR implements the following behaviour:

At startup, after the built-in commands have been registered to the org.obolibrary.robot.CommandManager, ROBOT looks for Jar files in a pre-defined set of directories:

• the directory specified by the robot.pluginsdir Java system property, if any;
• the directory specified by the ROBOT_PLUGINS_DIRECTORY environment variable, if any;
• the calling user’s ~/.robot/plugins directory.

In each Jar file found in those directories, ROBOT looks for a service provider for the org.obolibrary.robot.Command interface – that is, it looks for a resource file named META-INF/services/org.obolibrary.robot.Command, containing a list of classes implementing that interface. If such a service is found in a Jar file, the corresponding classes (i.e. the commands) are added to the set of commands recognized by the CommandManager.

To avoid clashes between command names (e.g. a plugin providing a command with the same name as a built-in command, or two plugins providing two commands with the same name), the basename of the Jar file that provides a command is prepended to the actual command name (as returned by the org.obolibrary.robot.Command#getName() method) when the command is added to the CommandManager. For example, if the file foo.jar provides a command named bar, that command will be registered under the name foo:bar – that’s how it will be listed in ROBOT’s help message, and how it will need to be invoked on the command line.

How to create a plugin

Developers wishing to create a plugin have to:

1. Write a class implementing the org.obolibrary.robot.Command interface, for each command they wish to create, the same way the built-in commands are implemented.

2. Package the compiled Java classes in a Jar file.

3. Add to that Jar file a META-INF/services/org.obolibrary.robot.Command file, containing a list of all the fully qualified class names representing the commands they wish to “expose” to ROBOT.

Examples

My robot-plugins repository contains an example of a plugin providing two pluggable commands: hello, a pure “proof-of-concept” command (it does nothing useful except injecting a silly annotation into the current ontology), and merge-species, a re-implementation of owltools’s --merge-species-ontology command.

Another example is available in my ocelliless repository, where a ROBOT command version of the ocelliless tool exists in the as-robot-plugin branch.

• docs/ have been added/updated
• tests have been added/updated
• mvn verify says all tests pass
• mvn site says all JavaDocs correct
• CHANGELOG.md has been updated

[jamesamcl]

I think this will be an extremely valuable addition and help the community consolidate on robot for I/O instead of having ad hoc scripts everywhere using different methods of loading/saving ontologies.

The main concern for us at EBI would be the addition of plugins breaking existing scripts, but I think the prefixing foo:bar comprehensively solves this.

[balhoff]

@gouttegd how do plugins deal with dependencies that aren't already in ROBOT?

[gouttegd]

@balhoff Such dependencies would need to be provided in the same Jar file as the one containing the plugin itself.

The best way to deal with that, I believe, would be to use the Maven plugin shade to selectively include or exclude the dependencies from the Jar to be built (excluding ROBOT itself and everything that comes with it – OWL API, ELK, etc. —, but including everything else).

The interesting question is: what would happen if two different plugins (in two different Jar files) both depends on the same libraries that are not already in ROBOT? My understanding is that the Java virtual machine would use the class files from whatever Jar file is loaded first, but I would need to do more testing to be sure. Thanks for raising this point.

[gouttegd]

The interesting question is: what would happen if two different plugins (in two different Jar files) both depends on the same libraries that are not already in ROBOT? My understanding is that the Java virtual machine would use the class files from whatever Jar file is loaded first.

So that’s not what actually happens. Because each plugin file is loaded by a specific class loader (not shared with the default class loader or the class loaders of the other Jar files in the plugin directory), the classes that are found in one plugin file are only visible from the plugins of that very Jar file.

This has importance consequences:

A) Every plugin must come with its own dependencies (unless those dependencies are already part of ROBOT, e.g. the OWL API). If two plugins A and B both depend on a library C, the author of B cannot hope to save space by deciding not to bundle C and to rely instead on the version of C that is bundled with A, because that version is only visible from A.

B) Two plugins can depend on different versions of the same dependency without breaking each other. If A depends on C-v0.1.0 and B depends on C-v0.2.0, they can each bundle the version they need.

[balhoff]

@gouttegd that sounds reassuring, although I guess it might be possible for a plugin to break if a later version of ROBOT drops or changes a dependency that the plugin was using. I suppose if a plugin bundles a dependency that's already used by ROBOT, there would be a clash?

[gouttegd]

it might be possible for a plugin to break if a later version of ROBOT drops or changes a dependency that the plugin was using

Yes. More generally, any change in ROBOT could break a plugin, if the change touches something that the plugin was dependent on. A change in the dependencies bundled with ROBOT (e.g. if ROBOT no longer depends on a given library and therefore no longer bundles it, or if it bundles a newer version, etc.) is just one possibility among others.

I suppose if a plugin bundles a dependency that's already used by ROBOT, there would be a clash?

No “clash” per se, but the plugin would not see the version of the dependency that it is bundled with – the version bundled with ROBOT would always take precedence. Whether this would lead to problems for the plugin would depend on how much the version provided by ROBOT and the version expected by the plugin differs. Best case scenario, the two versions are compatible and nothing bad happens; worst case scenario, the plugin could crash.

In any case, because ROBOT-provided dependencies always take precedence, breakage can only happen in one direction: ROBOT can break plugins, but plugins cannot break ROBOT. This I believe is the most important – we don’t want ROBOT to start behaving differently depending on whether some plugins are present or not.

[jamesaoverton]

This is a good idea @gouttegd, and well explained. I haven't had time to look at it in detail yet, but it's on my to do list.

[matentzn]

@gouttegd I think we can finalise this PR. Do you want to do it? I will deal with getting it merged when James is back. I can also finalsie it (CHANGELOG etc), just let me know how you want to play it. (for now, the "docs" update should probably be a single section in the global robot main docs, with a link to your own docs, we can work on folding this in more detail later).

[gouttegd]

I can take care of updating the CHANGELOG and the doc (can probably do that tomorrow).

Not sure about the tests, though. Ideally we’d want to test that invoking a plugin works, but obviously that requires having a plugin to test. Either we test by downloading one of my already available plugins (e.g. the KGCL or SSSOM one), but I don’t like the idea of the test suite being dependent on something external to the repository, or we add a “mock” plugin in this very repository (such a plugin could also act as an example of how to create a plugin).

[gouttegd]

@matentzn Documentation and changelog updated.

The only thing that is missing is some tests – for now all we have are the existing tests to show that the addition of the plugin feature does not break the CommandManager, but we have no tests to show that the plugin feature is working as expected.

Adding such tests would be relatively “invasive”, since ideally I think we should at a minimum:

• add an entire new top-level Maven module (e.g. robot-plugin), containing a “mock” plugin;
• build that module along with robot-core and robot-command;
• add a new test fixture in robot-command that tries to launch ROBOT after setting either the robot.pluginsdir property or the ROBOT_PLUGINS_DIRECTORY env variable and check that it loads the plugin correctly.

[matentzn]

Does not sound like a very enjoyable task :D I am a bit on the fence if we need this or not. I guess it would be better if there were proper tests in place for this. I think your suggestion sounds great!

[gouttegd]

I’ll work on that on a separate branch. No promises as to when I’ll find the time to do it, though. Your call (and James’) on whether it is worth waiting for or if you’re happy with the tests being added a posteriori.

[matentzn]

No promises at all needed for anything - the earliest I can even talk with James about this is last week of August :D

[gouttegd]

There is a rudimentary test on a new PR (#1142).

I am not entirely happy with it as it only tests the bare minimum (trying to load one plugin from the ROBOT_PLUGINS_DIRECTORY environment variable), but I am not sure testing more conditions would be worth the added complexity.

[jamesaoverton]

Closing in favour of #1142