[FEATURE] Allow downloading ML models from Cloud providers buckets

[IanMenendez]

Is your feature request related to a problem?
There is no solution to upload ML models from Google Cloud buckets (or other cloud providers) to OpenSearch. The only way is to make bucket objects public and use the public URL, therefore exposing ML models to anyone.

This makes the process of uploading a model quite tedious as we need to upload the model by chunks from our local because we do not want to expose ML models to the public internet

What solution would you like?
ModelHelper.downloadAndSplit currently uses ai.djl.training.util.DownloadUtils to download ML models. The solution would be to add (or check if there is an already coded solution) a helper function that allows gcloud (gs://bucket/model.zip) or AWS (s3://bucket/model.zip) type links

What alternatives have you considered?
Making bucket publicly accessible to anyone or uploading model from local making the process tedious

Do you have any additional context?
No

[ylwu-amzn]

hi, @IanMenendez thanks, this is a good suggestion.

Actually ml-commons supports uploading model from local file now. You can refer to https://opensearch-project.github.io/opensearch-py-ml/examples/demo_deploy_cliptextmodel.html and https://opensearch-project.github.io/opensearch-py-ml/examples/demo_transformer_model_train_save_upload_to_openSearch.html#Step-5:-Upload-the-model-to-OpenSearch-cluster

[IanMenendez]

hi, @IanMenendez thanks, this is a good suggestion.

Actually ml-commons supports uploading model from local file now. You can refer to https://opensearch-project.github.io/opensearch-py-ml/examples/demo_deploy_cliptextmodel.html and https://opensearch-project.github.io/opensearch-py-ml/examples/demo_transformer_model_train_save_upload_to_openSearch.html#Step-5:-Upload-the-model-to-OpenSearch-cluster

@ylwu-amzn I am aware that uploading models from local is supported but by going down this path we are sacrificing simplicity as:

• We would need to manually manage and upload each chunk of the model. While by uploading by URL we just need to specify the URL
• Increases steps of sharing ML models through our company as we would have to download the model from the bucket and then upload it instead of just specifying the model bucket URL

[ylwu-amzn]

Make sense. Agree that supporting ML models from cloud provider is a good feature. Will add this to dev plan. But appreciate if you prefer to contribute.

[rishabh1815769]

@ylwu-amzn I would like to contribute to this feature as part of the OSCI-2023 program.

[austintlee]

@ylwu-amzn can we assign this one to @rishabh1815769 ? @rishabh1815769 Have you made any progress on this?

[ylwu-amzn]

Thanks @austintlee Sure, assign to @rishabh1815769, one suggestion: share your design on this issue first so others can help review.

[khoaisohd]

Hi experts, do we have any update for this feature, the current project I'm working on needs this feature to allow customers to upload their model from OCI(Oracle cloud infrastructure) object storage
I did a design and a simple POC to verify the design with OCI object storage(Oracle Cloud Infrastructure). Would it be possible for me to contribute to this feature?

[ylwu-amzn]

@rishabh1815769 any update ?

@khoaisohd Thanks, I think you can post your design to this issue first, so community can discuss together.