Pickable storage driver clients

[antoinejeannot]

Hi !

This PR aims at adding a way to pickle the ModelLibrary which, for the moment, is impossible due to storage drivers (especially boto3 and gcs).

It introduces a MODELKIT_LAZY_DRIVER environment variable which, if set, will prevent the Storage Providers from storing the drivers (boto3, gcs, azure), instead, the configuration settings will be stored allowing the corresponding Storage Provider to build it on the fly.

This will make way easier the use of python libraries which use pickle such as Apache Spark and multi-processing, leveraging modelkit.

Thanks for reviewing, as always!

[ldeflandre]

Nicely done.

It would be nice to document the new usage it enables !

[antoinejeannot]

@ldeflandre sorry for the missing documentation 🤦
Please take a look at the latest commit!

[llin17]

Good job ! I'm wondering the use-case differences between MODELKIT_LAZT_LOADING and MODELKIT_LAZY_DRIVER. Dose setting MODELKIT_LAZY_DRIVER=True cover all the use-case of MODELKIT_LAZT_LOADING=True ? If it's the case, should we remove MODELKIT_LAZT_LOADING to avoid confusion ?

[antoinejeannot]

I would like to wait for @tgenin's approval before merging

[antoinejeannot]

@llin17 that's a good remark thanks! Actually the two mechanisms have different goals, and you might want to allow the MODELKIT_LAZY_DRIVER to access your cloud provider at any time and disable MODELKIT_LAZY_LOADING to build all your models before forking / sending models to workers etc. (with the assumption that your models are pickable of course :))

[llin17]

Thanks ! What do you think if we set the default value of MODELKIT_LAZY_DRIVER to True ?

[tgenin]

Very nice feature, thank you @antoinejeannot

[antoinejeannot]

Thanks for your pretty relevant remarks as always @tgenin, please take a look at the latest commit 🙏