speed to write files to s3 storage

[tinaok]

https://github.com/pangeo-data/pangeo-eosc/blob/main/notebooks/SwiftXarrayTest.ipynb
@sebastian-luna-valero @guillaumeeb
I've tried to write the Zarr file following the example and I confirm it is very slow.

Is there any way to make it faster?

[sebastian-luna-valero]

I can check with CESNET after @guillaumeeb confirms issues.

[guillaumeeb]

@tinaok could you point towards the code cell that you feel is slow? I have to admit I didn't go into details about performances, but this seemed reasonable to me when I did those tests.

If I remember correctly, this was between 0.5s and 2s per chunk, depending on chunk size.

[tinaok]

Thank you @guillaumeeb and @sebastian-luna-valero
Here is the example I was making to teach everyone to write their Zarr file close to https://pangeo-clivar.vm.fedcloud.eu/jupyterhub/

https://github.com/pangeo-data/clivar-2022/blob/main/tutorial/examples/notebooks/save_to_cesnet_bucket_zarr_s3fs.ipynb

[tinaok]

@guillaumeeb If you have better example than that, please place them at https://github.com/pangeo-data/clivar-2022/blob/main/tutorial/examples/notebooks/ and make pull request?

[guillaumeeb]

So, I've looked at the notebook and tried to reproduce. This is something I've experienced during my tests: when writing a Zarr file to S3 on CESNET, there is an non compressible time before the writes of chunks actually start. This time is about 30s.

From your example, we can see that it is not related to Dask. So this must be on Zarr or Xarray side. I'm not sure what is happening between the Python command and task graphs begin in Dask. Maybe it's the time it takes to create Metadata? But this seems really high. You can verify this by using a LocalCluster and watching the tasks stream.

Maybe it comes from CESNET, I don't know. Did you try the same code using another Cloud infrastructure?

What I can say, is that once the writes of chunks are started, the bandwidth seems reasonable, so bandwidth and latency of CESNET storage looks good when writing the chunks of data.

We can try simple writes from command line or with s3fs to see if it comes from CESNET, or from something internal to Xarray or Zarr, or maybe from a conjunction of both...

[guillaumeeb]

Hm, trying

ds.to_zarr("/tmp/myfile", mode='w', consolidated=True)

takes less than a second, so there is definitely something going on with Zarr and the object store.

[guillaumeeb]

I've also tried to sync the dataset from local storage to CESNET object store using aws s3 cli:

 time aws s3 --endpoint https://object-store.cloud.muni.cz --profile pangeoswift sync /tmp/myfile s3://tmp/guillaumeeb/
upload: ../../tmp/myfile/.zattrs to s3://tmp/guillaumeeb/.zattrs   
upload: ../../tmp/myfile/lon/.zarray to s3://tmp/guillaumeeb/lon/.zarray
upload: ../../tmp/myfile/Tair/.zattrs to s3://tmp/guillaumeeb/Tair/.zattrs
upload: ../../tmp/myfile/Tair/.zarray to s3://tmp/guillaumeeb/Tair/.zarray
upload: ../../tmp/myfile/lat/0 to s3://tmp/guillaumeeb/lat/0       
upload: ../../tmp/myfile/lat/.zarray to s3://tmp/guillaumeeb/lat/.zarray
upload: ../../tmp/myfile/.zgroup to s3://tmp/guillaumeeb/.zgroup  
upload: ../../tmp/myfile/.zmetadata to s3://tmp/guillaumeeb/.zmetadata
upload: ../../tmp/myfile/lat/.zattrs to s3://tmp/guillaumeeb/lat/.zattrs
upload: ../../tmp/myfile/Tair/0.0.0 to s3://tmp/guillaumeeb/Tair/0.0.0
upload: ../../tmp/myfile/lon/0 to s3://tmp/guillaumeeb/lon/0  
upload: ../../tmp/myfile/time/0 to s3://tmp/guillaumeeb/time/0
upload: ../../tmp/myfile/time/.zattrs to s3://tmp/guillaumeeb/time/.zattrs
upload: ../../tmp/myfile/time/.zarray to s3://tmp/guillaumeeb/time/.zarray
upload: ../../tmp/myfile/lon/.zattrs to s3://tmp/guillaumeeb/lon/.zattrs

real    0m1.989s
user    0m0.733s
sys     0m0.184s

Less than 2 seconds, so the problem does not come from CESNET per se. We're in the last situation, conjunction between using Zarr/Xarray and the object store. We should try the notebook on another system with S3 storage if possible. Else, maybe ask some help on Pangeo community side.

[tinaok]

Thank you very much @guillaumeeb !! I would like to start letting students use the object storage for writing their Zarr file.
I'll may be then warn student if it is small file, they should write it in netcdf format. And use Zarr only for big Zarr tmp files, until we have definitive solution.

Do you have a way for avoiding one students deleting other student's data?

[tinaok]

Did you try the same code using another Cloud infrastructure?

I have access to https://us-central1-b.gcp.pangeo.io/ but I do not have access to it's s3 storage so I can not bench. May be @rabernat can help us?

[rabernat]

Everyone on that hub has access to the scratch bucket where you can write a much data as you want.

I tried this example on the main cluster in the following way

import xarray as xr
import os
ds = xr.tutorial.open_dataset("air_temperature.nc").load()
SCRATCH_BUCKET = os.environ['SCRATCH_BUCKET'] 
%time ds.to_zarr(f'{SCRATCH_BUCKET}/air_temperature.zarr')

It took 5s. Definitely not very fast! But not 30s either. This is using gcsfs.

Just a thought...rather than using an s3fs.S3Map try using a zarr.storage.FSStore. That's the approach recommended in the Zarr tutorial: https://github.com/zarr-developers/tutorials/blob/main/zarr_cloud_native_geospatial_2022.ipynb

There are some optimizations available in FSStore that are not there in S3Map, and this could be making a difference.

See zarr-developers/zarr-python#911 for some relevant discussion.

[tinaok]

Thank you @rabernat but I do not understand yet what I should change in this notebook to use Zarr.storage.FSStore ...

Also, we have one more question that we want to ask how US Pangeo is doing #17 (comment)

[rabernat]

Thank you @rabernat but I do not understand yet what I should change in this notebook to use Zarr.storage.FSStore ...

You want to construct your store as follows

uri = f"s3://{s3_prefix}/{zarr_file_name}"
store = zarr.storage.FSStore(uri, **storage_kwargs)

where storage_kwargs has all of the customization options. (See https://github.com/zarr-developers/tutorials/blob/main/zarr_cloud_native_geospatial_2022.ipynb)

Also, we have one more question that we want to ask how US Pangeo is doing #17 (comment)

2i2c-org/infrastructure#1322

[rabernat]

To profile this, here is what I am doing. From my notebook, I run

%prun -D zarr.prof ds.to_zarr(f'{SCRATCH_BUCKET}/air_temperature.zarr')

Then I download thezarr.prof file and visualize it locally with python -m snakeviz zarr.prof. This produces a view like this:

zarr.prof.gz (gzipped because github doesn't allow just .prof files.)

What I have learned from this is that an awful lot of time is being spent on calling __contains__ on the store. In other words, Zarr is checking, maybe too much, what data are already stored. If the list operation on your object store is slow, this will take even more time.

I would be very interested to see the same profiling for your object store. Feel free to run the same command and post your results here.

This seems like a good target for optimization within Zarr. I'm pinging @martindurant, and expert on all of these things, to see if he can see any low-hanging fruit.

[guillaumeeb]

I've tried switching from S3Map to FSStore with no luck, the time taken is about the same, between 40 and 45s!

So I profiled the call, an here is what I get:

The profiling file.
zarr.prof.gz

I'm not sure how to analyze this, it seems that every steps take longer time. I guess this is probably due to a higher latency of some commands?