Change in newest version of datashader leads to bug in anemoi-training

[sahahner]

What happened?

When installing newest version of dask (2025.1.0), the import of the datashader fails and one is not able to start a training.

What are the steps to reproduce the bug?

• set up a new environment
• install anemoi-training
• start a training

Version

develop

Platform (OS and architecture)

leonardo/atos

Relevant log output

>>> import datashader as dsh
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/leonardo/home/userexternal/shahner0/2501_anemoi-core-waves/env-wave-core/lib/python3.11/site-packages/datashader/__init__.py", line 14, in <module>
    from . import data_libraries                             # noqa (API import)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/leonardo/home/userexternal/shahner0/2501_anemoi-core-waves/env-wave-core/lib/python3.11/site-packages/datashader/data_libraries/__init__.py", line 5, in <module>
    from . import dask    # noqa (API import)
    ^^^^^^^^^^^^^^^^^^
  File "/leonardo/home/userexternal/shahner0/2501_anemoi-core-waves/env-wave-core/lib/python3.11/site-packages/datashader/data_libraries/dask.py", line 55, in <module>
    bypixel.pipeline.register(dd.core.DataFrame)(dask_pipeline)
                              ^^^^^^^^^^^^^^^^^
AttributeError: module 'dask.dataframe.core' has no attribute 'DataFrame'

Accompanying data

No response

Organisation

ECMWF

[anaprietonem]

The issue is known by datashader community holoviz/datashader#1381 . They should do a release removing the deprecated dask dataframe functionality. Mean time we might need to pin Dask to or below (2024.12.1) to avoid this issue

Update: the datashader developers have now sorted the issue. The version in main already includes the fix, and a pypi release it's expected by end of the week. Once that's done, this issue could be closed

[HCookie]

The pyproject.toml has been updated to pin dask<=2024.12.1.
Once datashader updates we will remove it and close this issue.