Introduce zfs rewrite subcommand #17246
Conversation
github-actions
bot
added
the
Status: Work in Progress
|
I've tried to find some kernel APIs to wire this to, but found that plenty of Linux file systems each implement their own IOCTL's for similar purposes. I did the same, except the IOCTL number I took almost arbitrary, since ZFS seems quite rough in this area. I am open to any better ideas before this is committed. |
|
This looks amazing! Not having to sift through half a dozen shell scripts every time this comes up to see what currently handles the most edge cases correctly is very much appreciated. Especially with RaidZ expansion, being able to direct users to run a built-in command instead of debating what script to send them to would be very nice. Also being able to reliably rewrite a live dataset while it's in use without having to worry about skipped files or mtime conflicts would make the whole process much less of a hassle. With the only thing to really worry about being snapshots/space usage this seems as close to perfect as reasonably possible (without diving deep into internals and messing with snapshot immutability). Bravo! |
amotin
added
the
Status: Design Review Needed
|
thank you. Fixes one of the biggest problems with ZFS. Is there a way to suspend the process? It might be nice to have it run only during off hours. |
It does one file at a time, and should be killable in between. Signal handling within one huge file can probably be added. Though the question of the process restart is on the user. I didn't plan to go that deep into the area within this PR. |
|
I couldn't find documentation in the files changed, so I have to guess how it actually works. Is it a file at a time? I guess you could feed it with a "find" command. For a system with a billion files, do you have a sense how long this is gong to take? We can do scrubs in a day or two, but rsync is impractically slow. If this is happening at the file system level, that migth be the case here as well. |
This will likely be a good use case for GNU Parallel. |
It can take a directory as an argument and there are some recursive functions and iterators in the code so piping find into it should not be necessary. That avoids some userspace file handling overhead, but it still has to go through the contents of each directory one file at a time. I also don't see any parallel execution or threading (though I'm not too familiar with ZFS internals, maybe some of the primitives used here run asynchronously?). Whether doing parallelism in userspace by just calling it for many files/directories at once or not it should have the required locking to just run in the background and be significantly more elegant than the CP + mtime (or potentially userspace hash) check to make sure files didn't change during the copy process avoiding one of the potential pitfalls of existing solutions. |
|
I haven't benchmarked it deep yet, but unless the files are tiny, I don't expect there is a major need for parallelism. The code in kernel should handle up to 16MB at a time, plus allows ZFS to do read-ahead and write-back on top of that, so there will be quite a lot in the pipeline to saturate the disks and/or the system, especially if there is some compression/checksuming/encryption. And without need to copy data to/from user-space, the only thread will not be doing too much, I think mostly a decompression from ARC. Bunch of small files on a wide HDD pool I suspect may indeed suffer from read latency, but that in user-space we can optimize/parallelize all day long. |
|
I gave this a quick test. It's very fast and does exactly what it says 👍 I can already see people writing scripts that go though every dataset, setting the optimal compression, recordsize, etc, and zfs rewrite-ing them. |
|
Cool! Though the recordsize is one of things it can't change, since it would requite real byte-level copy, not just marking existing blocks dirty. I am not sure it can be done under the load in general. At least it would be much more complicated. |
|
Umm this is basically same as doing send | recv, isn't it? I mean, in a way, this is already possible to do without any changes, isn't it? Recv will even respect a lower recordsize, if I'm not mistaken - at least when receiving into a pool without large blocks support, it has to do that. I'm thinking whether we can do better, in the original sense of ZFS "better", meaning "automagic" - what do you think of using snapshots, send|recv, in a loop with ever decreasing delta size and then when the delta isn't decreasing anymore, we could swap those datasets and use (perhaps slightly modified) It'd be even cooler if it could coalesce smaller blocks into larger ones, but that potentially implies performance problems with write amplification, I would say if the app writes in smaler chunks that it gets onto disk in such smaller chunks, it's probably for the best to leave them that way. For any practical use-case I could think of though, I would definitely appreciate the ability to split the blocks of a dataset using smaller If there's a way how to make |
|
send recv has the huge downside of requiring 2x the space, even if you do the delta size thing since it has to send the entire dataset at least once and old data can't be deleted until the new dataset is complete.
Isn't this exactly what rewrite does? Change the options, run it and all the blocks are changed in the background. Without an application even seeing a change to the file. And unlike send recv it only needs a few MB of extra space. Edit: with the only real exception being record size, but recv also solves that only partially at best and it doesn't look like there's a reasonable way to work around that in a wholly transparent fashion. |
|
amotin
force-pushed
the
rewrite
branch
4 times, most recently
from
d23a371 to
c5f4413
Compare
|
Which release is this game changing enhancement likely to land in? |
|
@stuartthebruce So far it haven't landed even in master, so anybody who want to speed it up is welcome to test and comment. In general though, when completed, there is no reason why aside of 2.4.0 it can't be ported back to some 2.3.x of the time. |
Good to know there are no obvious blockers from including in a future 2.3.x. Once this hits master I will help by setting up a test system with 1/2PB of 10^9 small files to see if I can break it. Is there any reason to think the code will be sensitive to Linux vs FreeBSD? |
IOCTL interface of the kernels is obviously slightly different, requiring OS-specific shims, as with most of other VFS-related code. But seems like not a big problem, as Tony confirmed it works on Linux too from the first try. |
Since this introduces new IOCTL API, I'd appreciate some feedback before it hit master in case some desired functionality might require API changes aside of the |
OK, I will see if I can find some time this next week to stress test. |
This allows to rewrite content of specified file(s) as-is without modifications, but at a different location, compression, checksum, dedup, copies and other parameter values. It is faster than read plus write, since it does not require data copying to user-space. It is also faster for sync=always datasets, since without data modification it does not require ZIL writing. Also since it is protected by normal range range locks, it can be done under any other load. Also it does not affect file's modification time or other properties. Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc.
This allows to rewrite content of specified file(s) as-is without modifications, but at a different location, compression, checksum, dedup, copies and other parameter values. It is faster than read plus write, since it does not require data copying to user-space. It is also faster for sync=always datasets, since without data modification it does not require ZIL writing. Also since it is protected by normal range range locks, it can be done under any other load. Also it does not affect file's modification time or other properties. Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Rob Norris <[email protected]>
This allows to rewrite content of specified file(s) as-is without modifications, but at a different location, compression, checksum, dedup, copies and other parameter values. It is faster than read plus write, since it does not require data copying to user-space. It is also faster for sync=always datasets, since without data modification it does not require ZIL writing. Also since it is protected by normal range range locks, it can be done under any other load. Also it does not affect file's modification time or other properties. Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Rob Norris <[email protected]>
re:naming I guess "zfs anneal" could be a possibility if "rewrite" is to be used for something else. That's what I've been calling this kind of functionality myself. It seems to me like a fitting description for the task this tool is meant to accomplish. |
Neat idea! As an added bonus, it's metaphor from metallurgy, just like scrub and resilver. |
|
FYI, I found another use case for this feature: to heal a remote snapshot backup. For example, I have a large (146TB) snapshot with two (out of 55.7M) files that return EIO CKSUM errors on a remote backup but are OK on the primary production instance (long story). I will use |
|
Could this be the basis of a somewhat naive defrag? Similar to a |
|
I just came across this, it seems we can use this to rewrite files to special vdevs, right? |
|
from what i read in this thread, its as if you did a |
This allows to rewrite content of specified file(s) as-is without modifications, but at a different location, compression, checksum, dedup, copies and other parameter values. It is faster than read plus write, since it does not require data copying to user-space. It is also faster for sync=always datasets, since without data modification it does not require ZIL writing. Also since it is protected by normal range range locks, it can be done under any other load. Also it does not affect file's modification time or other properties. Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Rob Norris <[email protected]>
I believe you are all thinking about the metaphor |
This does raise a significant semantics question for anyone implementing this feature, I think: How do you handle corrupted files? Do you overwrite the file and mark the whole thing correct with valid checksums, with the corrupted bits set to whatever undefined value you read back (or just zeros)? Or do you leave the checksum errors on that file in place after? I suppose this is highly implementation-dependant at the moment. This particular use case would suggest clearing the errors and setting the file contents to "something, anything" which seems like an "ok" thing to so, so long as the user understands and accepts that this is going to happen. Maybe warn before executing zfs rewrite on a filesystem with checksum errors? (should it stop if it hits one at runtime?) |
Not exactly. Generally when people think of tempering they think of softening material, usually in the case of steel which has been made glass hard and now needs to be pulled back to something more reasonable to use. I suggested annealing because annealing removes all sorts of discontinuities in the metal in the process of that normalization, in the same way that ZFS rewrite could remove inconsistencies in compression algorithm, raidz stripe length, etc etc. One of the things that "annealing" covers but "tempering" does not is the softening of copper back to a workable state (after it's been work-hardened from lots of hammering) so that further hammering and shaping can be done. The latter is the analogy I had in mind when I suggested the term. |
I believe you missunderstood. The files you rewrite aren't the corrupted ones. The corrupted ones are in the offsite snapshot. So anything that's read and written out again is valid data, with passing checksums. This is just about forcing zfs send to retransmit that data without having to change it from userspace. Usually this would be handled by doing a replication from scratch or with the standard cp, rm, mv cycle on the live version to force replication to transmit the blocks again and have a new, uncorrupted snapshot in the offsite location. |
Yes, not exactly. As all metaphors... Tempering is the final operation to leave an alloy in its toughest, most optimized, and inexorable form. Annealing leaves metal in its softest, most malleable form... the exact opposite goal for user data. Anyway, the merge was performed with "rewrite" which is far and away the lowest obfuscation, however boring it may be, so is honestly the best choice for the masses.
👀 I feel like Mugatu in Zoolander, but nothing a pint at the pub won't solve... 🤣 |
Annealing allows one to do further work to a piece of copper, removing the internal dislocations accumulated from previous work. I'm sure that users would like to continue to work with and shape their arrays over time and this gives them a way to remove accumulated inconsistencies at various points along the way.
Really my biggest issue with "rewrite" is that it could be confused with block pointer rewrite, the once promised feature never delivered, which it certainly isn't except in the most roundabout way imaginable. Yes, technically rewriting the data rewrites the block pointers, but it's a very different kind of operation. |
|
How does this interact with snapshots? If I rewrite everything and have a snapshot, am I now using twice the space? |
Yes, if you rewrite a block that's in a snapshot, then the snapshot keeps one, and you get a new one, so it will use more space on the pool. This is mentioned in It may not be twice the space; it depends on the properties at time of rewrite. |
I'm not really sure the distinction between "rewrite the data" and "rewrite the block pointer" really means anything here, since the BP describes how to interpret the data, so changing the data layout or transforms necessarily requires the the block pointer to change. In any case, I doubt there's going to be much confusion. "Block pointer rewrite" is pretty inside-baseball at this point; most people who just use OpenZFS have likely never heard of BPR or if they have, not with any particular idea of what it is or should be. Hell, I've been working on OpenZFS for three years now and I only know it in wish form ("gee, it'd be nice to just upgrade all the block pointers"). Which is what |
|
How does zfs rewrite interact with an array that has data errors? |
AFAIU the biggest difference between "zfs rewrite" and the mythical block-pointer-rewrite is that data rewritten by "zfs rewrite" count as newly-written data, with some unpleasant consequences:
At VFS layer, |
Kernel should return error to user-space, same as read, but user-space there is made to log errors and continue with next file. |
@maxximino Look here: #17565 . :) |
Motivation and Context
For years users were asking for an ability to re-balance pool after vdev addition, de-fragment randomly written files, change some properties for already written files, etc. The closest option would be to either copy and rename a file or send/receive/rename the dataset. Unfortunately all of those options have some downsides.
Description
This change introduces new
zfs rewritesubcommand, that allows to rewrite content of specified file(s) as-is without modifications, but at a different location, compression, checksum, dedup, copies and other parameter values. It is faster than read plus write, since it does not require data copying to user-space. It is also faster for sync=always datasets, since without data modification it does not require ZIL writing. Also since it is protected by normal range range locks, it can be done under any other load. Also it does not affect file's modification time or other properties.How Has This Been Tested?
Manually tested it on FreeBSD. Linux-specific code is not yet tested.
Types of changes
Checklist:
Signed-off-by.