[BUG] Missing `lib` folder after plugin installation using `opensearch-plugin` CLI tool

[AlexRuiz7]

What is the bug?
The plugins/opensearch-knn/lib folder and all its contents are missing.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Uninstall the opensearch-knn plugin.
2. Reinstall the opensearch-knn plugin.
3. List the plugins/opensearch-knn/lib folder. It doesn't exist.

What is the expected behavior?
The plugin and all its dependencies are installed properly.

What is your host/environment?

• OS: Docker
• Version 2.11.1

Do you have any additional context?

Using OpenSearch's Docker compose, I tried to remove and reinstall the opensearch-knn plugin to check whether the lib folder is installed.

OpenSearch version:

[opensearch@ae29f5538322 ~]$ bin/opensearch --version
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.11.1.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Version: 2.11.1, Build: tar/6b1986e964d440be9137eba1413015c31c5a7752/2023-11-29T21:43:10.135035992Z, JVM: 17.0.8

opensearch-knn/lib contents:

[opensearch@ae29f5538322 ~]$ ls plugins/opensearch-knn/lib/
libgomp.so.1  libopensearchknn_common.so  libopensearchknn_faiss.so  libopensearchknn_nmslib.so

opensearch-knn plugin removal:

[opensearch@ae29f5538322 ~]$ bin/opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-sql
[opensearch@ae29f5538322 ~]$ bin/opensearch-plugin remove opensearch-knn
ERROR: plugin [opensearch-knn] cannot be removed because it is extended by other plugins: [opensearch-neural-search]
[opensearch@ae29f5538322 ~]$ bin/opensearch-plugin remove opensearch-neural-search
-> removing [opensearch-neural-search]...
[opensearch@ae29f5538322 ~]$ bin/opensearch-plugin remove opensearch-knn
-> removing [opensearch-knn]...
[opensearch@ae29f5538322 ~]$ ls plugins/opensearch-knn
ls: cannot access 'plugins/opensearch-knn': No such file or directory

opensearch-knn plugin installation:

[opensearch@ae29f5538322 ~]$ bin/opensearch-plugin install --batch --verbose org.opensearch.plugin:opensearch-knn:2.11.1.0
-> Installing org.opensearch.plugin:opensearch-knn:2.11.1.0
Checking if url exists: https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-knn/2.11.1.0/opensearch-knn-linux-x64-2.11.1.0.zip
-> Downloading org.opensearch.plugin:opensearch-knn:2.11.1.0 from maven central
Retrieving zip from https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-knn/2.11.1.0/opensearch-knn-2.11.1.0.zip
- Plugin information:
Name: opensearch-knn
Description: OpenSearch k-NN plugin
Version: 2.11.1.0
OpenSearch Version: 2.11.1
Java Version: 11
Native Controller: false
Extended Plugins: [lang-painless]
 * Classname: org.opensearch.knn.plugin.KNNPlugin
Folder name: 
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission loadLibrary.opensearchknn_faiss
* java.lang.RuntimePermission loadLibrary.opensearchknn_nmslib
* java.net.SocketPermission * connect,resolve
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed opensearch-knn with folder name opensearch-knn

lib folder is not present:

[opensearch@ae29f5538322 ~]$ ls plugins/opensearch-knn/
LICENSE.txt  commons-lang-2.6.jar     guava-32.0.1-jre.jar         plugin-descriptor.properties
NOTICE.txt   failureaccess-1.0.1.jar  opensearch-knn-2.11.1.0.jar  plugin-security.policy

[navneet1v]

@AlexRuiz7 in the zip folder of k-NN the lib is empty because the content of the lib are platform dependent.

Can you please add some details how this is impacting your project/repo?

cc: @vamshin , @peterzhuamazon

[peterzhuamazon]

Hi, please use the manifest.yml file in the opensearch bundle to get the knn zip link.
You can then download the zip and install with the plugin cli, which contains lib.

Thanks.

[AlexRuiz7]

@AlexRuiz7 in the zip folder of k-NN the lib is empty because the content of the lib are platform dependent.

Can you please add some details how this is impacting your project/repo?

cc: @vamshin , @peterzhuamazon

I see. Makes sense. I'm not aware of the impact if these libraries are missing, but I assume the plugin won't work properly.

[AlexRuiz7]

We'll try using the manifest.yml file. I'll let you know of my results!

Thanks to both of you for the replies.

[AlexRuiz7]

Is there any option to download the manifest.yml file without downloading the whole bundle?

[navneet1v]

@AlexRuiz7 please create an issue in https://github.com/opensearch-project/opensearch-build repo. The maintainers of that repo can ans this question. Right now I am unable to transfer this issue to opensearch-build.

[jmazanec15]

Closing will track in #2197