Skip to content

Restructure to decouple native libraries from core plugin #2772

Open
@jmazanec15

Description

@jmazanec15

Description

A general hindrance to development for the k-NN plugin is setting up cmake and building the native libraries. It takes time and effort to configure the dependencies and also to actually build the libraries. If someone is not working on anything to do with the C++ side, this is, in general, a painful, unnecessary experience.

In order to make this process smoother, I propose to move the C++ libraries into their own gradle subproject, native, that the core plugin will take a dependency on. Then, for testing, we can explicitly annotate whether a test needs the native libraries built or not. So the experience will be something like

## Same goes for run/integTest/build
./gradlew :test --require-native-engines=true/false

To ease the development experience, --require-native-engines should be false by default. For Github CI, we should always run everything

This will immediately improve the development experience. In the long term, it is also a step that will make it easier to:

  1. Make the engines extendible ([META] Supporting New Vector Engine in OpenSearch #2605)
  2. Ship library artifacts separately ([BUG] KNN plugin on maven is missing the lib/ directory that contains necessary files for the plugin to work #2643)
  3. Move k-NN functionality to core (without needing to bring along native code)

Metadata

Metadata

Assignees

No one assigned

    Labels

    InfrastructureChanges to infrastructure, testing, CI/CD, pipelines, etc.

    Type

    No type

    Projects

    Status

    Backlog (Hot)

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions