Compare the performance of reading ICON in zarr vs. reading ICON in NetCDF4

[sbAsma]

Description

This issue aims to compare the performance of the two data readers developed in relation to Issue #429 and Issue #352.

The performance comparison will focus on the following metrics:

• Memory usage — measured as the difference in Resident Set Size (RSS) before and after key operations
• Execution time — measured using high-resolution timers (time.perf_counter)
• Data throughput — computed as memory change divided by duration (KiB/s)

Tools

The main tool for system-level profiling will be psutil, which will be used to monitor CPU and memory usage over time.

Methodology

• The number of workers, batch size, and system configuration will be kept constant across tests
• Both readers will be run under the same conditions to ensure fair comparison
• Metrics will be collected during:
  • Initialization
  • Iteration through the dataset

Results

The results will be collected in a hedge doc

[sbAsma]

assign me

[github-actions]

Issue assigned to @sbAsma.

[jpolz]

Are you planning to use dummy data? Currently the two data readers are designed for differently organized simulations at different resolutions.

[sbAsma]

Are you planning to use dummy data? Currently the two data readers are designed for differently organized simulations at different resolutions.

Yes. Now that you're mentioning it, I wonder if I should create a zarr store like you did, and compare the performances.