Easier CustomDataset Creation

[lancechua]

Description

IO read write functions typically follow this signature:

def load(path: Union[Path, str], **kwars) -> obj:
    ...

def save(obj, path: Union[Path, str], **kwars) -> None:
    ...

Creating custom datasets should ideally be as easy as supplying load / save function(s) that follow this function signature.

Context

kedro supports an extensive range of datasets, but it is not exhaustive. Popular libraries used by relatively niche communities like xarray and arviz aren't currently supported.

Beyond these examples, unofficially adding support for more obscure datasets would be easier.

Initially, I was looking to implement something like this and asked in the Discord chat if this pattern made sense.
Then, @datajoely suggested I open a feature request.

Possible Implementation

We can consider a Dataset class factory. Maybe GenericDataset with a class method .create_custom_dataset(name: str, load: callable, save: callable).

Usage would look something like xarrayNetCDF = GenericDataset("xarrayNetCDF", xarray.open_dataset, lambda x, path, **kwargs: x.to_netcdf(path, **kwargs)).
Entries can be added to the data catalog yaml just as with any other custom dataset implementation.

Possible Alternatives

• LambdaDataset is very similar but the load, and save are hard coded in the implementation, and cannot be parameterized in the data catalog, as far as I'm aware
• Subclassing AbstractDataset is an option, but this feature request seeks to reduce boilerplate when defining new datasets
• Adding xarray support #1346 officially requires implementing nuances like cloud file storage, partitioned datasets, lazy loading, etc.

[datajoely]

Hi @lancechua thank you for raising this! We have discussed this as a team and have had a good conversation regarding ways we can proceed.

You will see that this issue has now been referenced in the #1778 super issue which is tracking all of the ways we plan to improve the DataCatalog at a low level.

Your last point:

Adding xarray support kedro-org/kedro-plugins#165 officially requires implementing nuances like cloud file storage, partitioned datasets, lazy loading, etc.

Is the most interesting, a lot of the paint-points to mention have emerged organically. Versioning, fsspec etc. have all arrived in Kedro way later than the original AbstractDataSet class definition, implementing a new DataSet today requires (probably too much) copy and pasting.

The team are going to have a go at prototyping what this could look like, so you may be interested in subscribing to parent issue to contribute to these design decisions. The initial thinking reflected on the rather flat structure we have today where everything inherits from AbstractDataSet (or AbstractVersionedDataSet) and we could utilise OO concepts much better and make implementation/testing/overriding much simpler.

[deepyaman]

Is the most interesting, a lot of the paint-points to mention have emerged organically. Versioning, fsspec etc. have all arrived in Kedro way later than the original AbstractDataSet class definition, implementing a new DataSet today requires (probably too much) copy and pasting.

I feel like too much copy-pasting has been an issue for a really long time. 😅

I would be keen to break down the points of duplication (e.g. default argument handling, fsspec-related boilerplate, etc.) and add mixins for each (as in #15), because that way we can make progress on resolving these pain points without having to limit ourselves to a universal solution that works for every possible dataset.

[astrojuanlu]

Collecting some thoughts from our latest Tech Design session about this topic:

Recent developments

• Linked issues: Re-design io.core and io.data_catalog #1778, Move AbstractDataSet to Kedro-Plugins #2409
• Some recent experiences trying to add "weird" datasets
  • My own Kaggle dataset https://github.com/astrojuanlu/kedro-kaggle-dataset
  • FITS dataset using Astropy, because SunPy and Astropy https://github.com/b4rlw/sunspot-segmentation/blob/master/src/sunspots/extras/datasets/sunpy_datasets.py
    • "It is also worth noting that a HDU cannot be selected within the node logic, because even though the FITS file metadata might be loaded, the data inside a particular HDU cannot be loaded unless the corresponding file is still open, and by the time a dataset arrives at a node, it is not."
  • HDF dataset, recently reopened Add catalog dataset for HDF formats not supported by pandas kedro-plugins#240

General thoughts

• There are 3 dimensions to take into account
  • Where data is
  • What format is
  • What memory representation
• Ideally a design would decouple the three, but this is not practical, and since Kedro offloads data loading to other libraries, the dataset is subject to the abstractions by those other libraries and how they solve the decoupling
• The old design was very painful and using fsspec greatly improved it https://docs.kedro.org/en/0.14.3/kedro.io.html#data-sets
• There is some sort of decoupling with pandas.GenericDataSet https://docs.kedro.org/en/latest/kedro_datasets.pandas.GenericDataSet.html but it cannot be easily achieved with everything

Current issues

• Lots of copy-pasting
  • @deepyaman's comment right above Easier CustomDataset Creation #1936 (comment)
  • "Relatedly, why is _describe an abstract method at all? People just have to write way too much code when writing a custom dataset"
  • There's a suggestion to add a fsspec mixin [kedro-datasets] fsspec mixin #4314
    • But multiple inheritance brings some problems
    • AbstractDataSet base class contains lots of logic, can't be easily turned into a protocol Move AbstractDataSet to Kedro-Plugins #2409 (comment)
• Unclear how to support "lazy" datasets
  • Case in point: HDF, FITS, others
  • Datasets are eager, so if the data has to be lazily loaded, users have to jump through some hoops
• Somewhat unclear separation of "format" and "transport"
  • Not really an issue of the current design, but we could do a better job at communicating it, or just assume that people will do whatever works for them
• Unclear if full reproducibility is encouraged
  • Case in point: Kaggle
  • Another: "My experience loading COVID data https://github.com/deepyaman/inauditus/blob/develop/refresh-data I just write a script to download the data from an external source, and then treat it as a pandas.CSVDataSet. There are other problems with doing the “expensive” data download in a load function, e.g. that we’re loading from a remote source each time when we really should just have it in a downloaded source."
• Unclear interaction with versioning
  • "And it is more subtle about versioning - it actually works only if you use a specific function. The newly ManagedTableDataset which use the built-in versioning actually doesn’t work like this."
• Creating a custom dataset requires a full Kedro installation, which some users consider too heavy
  • The topic of Move AbstractDataSet to Kedro-Plugins #2409

Other links and ideas

• Any redesign should be grounded on user research and user needs
• @antonymilne suggested making dataset parameters keyword-only, which enables non-breaking refactoring post-0.19.0
  • If we think this is worth doing, we should open an issue and prioritize it sooner rather than later
• GetInData "pipeline dataset" https://github.com/getindata/kedro-azureml/blob/develop/kedro_azureml/datasets/pipeline_dataset.py

[astrojuanlu]

A user that quite likely got confused when defining their own dataset, most likely because they aren't copy-pasting enough https://www.linen.dev/s/kedro/t/12646523/hi-folks-i-m-trying-to-create-a-custom-kedro-dataset-inherit#ac22d4cb-dd72-4072-905a-f9c5cb352205