HDDS-13494. DirectoryDeletingService incorrectly waits for all the deleted directories processing #8852
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…leted directories processing Change-Id: I3b291f36b12af7393ba7299618abbbd8b9f02550
Change-Id: Ia529ff3102679eb32109920f181bb38c23634cb7 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java
Change-Id: Ibe26d77423cdec1da37f8df07a0aa56cb2a5c407
jojochuang
reviewed
Jul 23, 2025
| @@ -76,6 +87,7 @@ private OzoneConfiguration createConfAndInitValues() throws IOException { | |||
| ServerUtils.setOzoneMetaDirPath(conf, newFolder.toString()); | |||
| conf.setTimeDuration(OZONE_DIR_DELETING_SERVICE_INTERVAL, 3000, | |||
| TimeUnit.MILLISECONDS); | |||
| conf.setInt(OZONE_THREAD_NUMBER_DIR_DELETION, threadCount); | |||
There was a problem hiding this comment.
Unrelated: ozone.thread.number.dir.deletion should be renamed to ozone.om.thread.number.dir.deletion.
Along with a bunch other configuration property names in OMConfigKeys
There was a problem hiding this comment.
Lets do this as a separate jira
| @@ -76,6 +87,7 @@ private OzoneConfiguration createConfAndInitValues() throws IOException { | |||
| ServerUtils.setOzoneMetaDirPath(conf, newFolder.toString()); | |||
| conf.setTimeDuration(OZONE_DIR_DELETING_SERVICE_INTERVAL, 3000, | |||
| TimeUnit.MILLISECONDS); | |||
| conf.setInt(OZONE_THREAD_NUMBER_DIR_DELETION, threadCount); | |||
There was a problem hiding this comment.
isn't the default value of the property 10 already?
There was a problem hiding this comment.
The conf gets initialized in the base class. I wanted to explicitly override the value here for the test case.
...ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java
Show resolved
Hide resolved
...ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java
Show resolved
Hide resolved
| @@ -555,7 +555,7 @@ private void processDeletedDirsForStore(SnapshotInfo currentSnapshotInfo, KeyMan | |||
| return false; | |||
| } | |||
| }, isThreadPoolActive(deletionThreadPool) ? deletionThreadPool : ForkJoinPool.commonPool()); | |||
| processedAllDeletedDirs = future.thenCombine(future, (a, b) -> a && b); | |||
| processedAllDeletedDirs = processedAllDeletedDirs.thenCombine(future, (a, b) -> a && b); | |||
...e-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestDirectoryDeletingService.java
Outdated
Show resolved
Hide resolved
Change-Id: Ia1965b32e3dc27f3f83e5648a4b86939fa2c8e14
|
@jojochuang Can we merge this particular change? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently while parallely processing deleted directory table, the main thread doesn't wait for all the threads to finish it's set of deleted directory which could lead to main thread prematurely rocksdb iterator and also leading to incorrectly marking the deep cleaned directories.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13494
How was this patch tested?
Added unit tests for the same.