[feat](param-refactor)Rename FS v2 to v1, remove legacy FS, and integrate new Storage/Azure features #51539
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rate new Storage/Azure features This pull request includes the following updates: Rename FS v2 to v1 The current FS v2 implementation has been renamed to FS v1, making it the default and primary file system implementation. Remove Legacy FS All code and dependencies related to the legacy FS (the old v1 implementation) have been completely removed. Repository Cleanup Dropped all compatibility logic for the old FS. The Repository layer now exclusively uses the new FS and Storage interfaces. Deprecated configuration entries, injection code, and transitional hooks have been removed. New Storage and Azure Feature Integration Added support for new Azure file system features Improved integration between the Storage layer and Repository for better scalability.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
TPC-H: Total hot run time: 34009 ms |
TPC-DS: Total hot run time: 193317 ms |
ClickBench: Total hot run time: 29.27 s |
|
run buildall |
TPC-H: Total hot run time: 34311 ms |
TPC-DS: Total hot run time: 188044 ms |
ClickBench: Total hot run time: 29.32 s |
FE UT Coverage ReportIncrement line coverage |
|
run p0 |
|
run buildall |
TPC-H: Total hot run time: 33888 ms |
TPC-DS: Total hot run time: 192685 ms |
ClickBench: Total hot run time: 29.17 s |
|
run external |
|
run buildall |
TPC-H: Total hot run time: 33810 ms |
TPC-DS: Total hot run time: 192632 ms |
ClickBench: Total hot run time: 29.84 s |
morningman
reviewed
Jun 7, 2025
|
|
||
| super(StorageBackend.StorageType.S3.name(), StorageBackend.StorageType.S3, | ||
| new S3ObjStorage(s3Properties)); | ||
| this.s3Properties = s3Properties; |
There was a problem hiding this comment.
Why need 2 properties? s3Properties and storageProperties?
There was a problem hiding this comment.
Each file system has its own parameter class with specific methods. Using the concrete type improves clarity and avoids unnecessary abstraction.
Since StorageProperties was somewhat redundant, I refactored it into an abstract base class. This allows each subclass to simply return its own concrete instance while still sharing a common structure.
fe/fe-core/src/main/java/org/apache/doris/fs/remote/S3FileSystem.java
Outdated
Show resolved
Hide resolved
| public DFSFileSystem(HdfsCompatibleProperties hdfsProperties) { | ||
| super(StorageBackend.StorageType.HDFS.name(), StorageBackend.StorageType.HDFS); | ||
| this.properties.putAll(hdfsProperties.getOrigProps()); | ||
| this.storageProperties = hdfsProperties; |
There was a problem hiding this comment.
Why need both storageProperties and hdfsProperties
| authenticator = HadoopAuthenticator.getHadoopAuthenticator(authConfig); | ||
| Configuration conf = hdfsProperties.getHadoopConfiguration(); | ||
| // TODO: Temporarily disable the HDFS file system cache to prevent instances from being closed by | ||
| // each other in V1. This line can be removed once V1 and V2 are unified. |
There was a problem hiding this comment.
There is anything else we need to do to unify v1 and v2?
|
run buildall |
TPC-H: Total hot run time: 33655 ms |
TPC-DS: Total hot run time: 192566 ms |
ClickBench: Total hot run time: 29.43 s |
FE UT Coverage ReportIncrement line coverage |
|
run p0 |
|
run buildall |
TPC-H: Total hot run time: 33704 ms |
TPC-DS: Total hot run time: 193207 ms |
ClickBench: Total hot run time: 29.27 s |
CalvinKirs
force-pushed
the
master-fsv2-fsv1
branch
from
c9ac2b0 to
5442be3
Compare
|
run buildall |
CalvinKirs
commented
Jun 8, 2025
Comment on lines
103
to
+104
| "storage_resource" = "${resource_table_use}", | ||
| "cooldown_datetime" = "2025-06-08 00:00:00" | ||
| "cooldown_datetime" = "2035-06-08 00:00:00" |
There was a problem hiding this comment.
This kind of absolute time writing should be prohibited
TPC-H: Total hot run time: 33541 ms |
TPC-DS: Total hot run time: 192480 ms |
ClickBench: Total hot run time: 28.79 s |
FE UT Coverage ReportIncrement line coverage |
morningman
approved these changes
Jun 8, 2025
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
approved
|
PR approved by anyone and no changes requested. |
wyxxxcat
approved these changes
Jun 9, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#50238
What problem does this PR solve?
Rename FS v2 to v1
The current FS v2 implementation has been renamed to FS v1, making it the default and primary file system implementation.
Remove Legacy FS
All code and dependencies related to the legacy FS (the old v1 implementation) have been completely removed.
Repository Cleanup
Dropped all compatibility logic for the old FS.
The Repository layer now exclusively uses the new FS and Storage interfaces.
Deprecated configuration entries, injection code, and transitional hooks have been removed.
New Storage and Azure Feature Integration
Added support for new Azure file system features
Improved integration between the Storage layer and Repository for better scalability.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)