Improve RAIDZ expansion performance #15680
Labels
Type: Feature
Feature request or new feature
Comments
behlendorf
pushed a commit
to behlendorf/zfs
that referenced
this issue
Dec 6, 2024
- Instead of copying one ashift-sized block per ZIO, copy as much as we have contiguous data up to 16MB per old vdev. To avoid data moves use gang ABDs, so that read ZIOs can directly fill buffers for write ZIOs. ABDs have much smaller overhead than ZIOs in both memory usage and processing time, plus big I/Os do not depend on I/O aggregation and scheduling to reach decent performance on HDDs. - Reduce raidz_expand_max_copy_bytes to 16MB on 32bit platforms. - Use 32bit range tree when possible (practically always now) to slightly reduce memory usage. - Use ZIO_PRIORITY_REMOVAL for early stages of expansion, same as for main ones. - Fix rate overflows in `zpool status` reporting. With these changes expanding RAIDZ1 from 4 to 5 children I am able to reach 6-12GB/s rate on SSDs and ~500MB/s on HDDs, both are limited by devices instead of CPU. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#15680 Closes openzfs#16819
lundman
pushed a commit
to openzfsonwindows/openzfs
that referenced
this issue
Jan 26, 2025
- Instead of copying one ashift-sized block per ZIO, copy as much as we have contiguous data up to 16MB per old vdev. To avoid data moves use gang ABDs, so that read ZIOs can directly fill buffers for write ZIOs. ABDs have much smaller overhead than ZIOs in both memory usage and processing time, plus big I/Os do not depend on I/O aggregation and scheduling to reach decent performance on HDDs. - Reduce raidz_expand_max_copy_bytes to 16MB on 32bit platforms. - Use 32bit range tree when possible (practically always now) to slightly reduce memory usage. - Use ZIO_PRIORITY_REMOVAL for early stages of expansion, same as for main ones. - Fix rate overflows in `zpool status` reporting. With these changes expanding RAIDZ1 from 4 to 5 children I am able to reach 6-12GB/s rate on SSDs and ~500MB/s on HDDs, both are limited by devices instead of CPU. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#15680 Closes openzfs#16819
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the feature would like to see added to OpenZFS
Performance of RAIDZ expansion could be improved.
How will this feature improve OpenZFS?
Currently the RAIDZ expansion performance is very slow and it can take several days to add a single large drive.
Improving expansion performance would result in a shorter period of degraded FS performance for users.
Expanding by mote than a single drive is not currently supported, so to add a few new drives to a pool can take weeks.
Additional context
There was discussion on this matter during development of the RAIDZ expansion feature where it was noted that performance is very slow, and the opinion of the developer was that it wouldn't be difficult to improve after the feature has landed.
Maybe it could be considered an item for some near-term follow-up on the landing of the new feature?
The text was updated successfully, but these errors were encountered: