HADOOP-19518. ABFS: [FNSOverBlob] WASB to ABFS Migration Config Support Script #7564
+237
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
anujmodi2021
reviewed
Apr 3, 2025
| ## Introduction | ||
|
|
||
| ABFS driver has now built support for | ||
| FNS accounts (over BlobEndpoint that WASB Driver uses) using the ABFS scheme. |
There was a problem hiding this comment.
Adda link here to fns_blob.md file
| ABFS driver has now built support for | ||
| FNS accounts (over BlobEndpoint that WASB Driver uses) using the ABFS scheme. | ||
|
|
||
| The legacy WASB driver has been **deprecated** and is no longer recommended for |
There was a problem hiding this comment.
Add a link to deprecated_wasb.md here.
May be first documentation PR needs to be merged
| contactTeamMsg="For any queries or support, kindly reach out to us at '[email protected]'." | ||
| endpoint=".dfs." | ||
| printf "Select 'HNS' if you're migrating to ABFS driver with Hierarchical Namespace enabled account, | ||
| or 'Non-HNS' if you're migrating with Non-Hierarchical Namespace (FNS) account. \n" |
There was a problem hiding this comment.
Nit: Statement construction should be same for both the options.
Select 'HNS' if you're migrating to ABFS driver for Hierarchical Namespace enabled account, or 'Non-HNS' if you're migrating to ABFS driver for Non-Hierarchical Namespace (FNS) account. \n
| # Stop the script if any unsupported config is found | ||
| for key in "${unsupported_configs_list[@]}"; do | ||
| if grep -q "$key" "$OUTPUT_FILE"; then | ||
| echo "FAILURE: Remove the following configuration from file and rerun: '$key'" |
There was a problem hiding this comment.
Tell user why to remove?
"Unsupported Config found"
| ) | ||
|
|
||
| # Configurations not required in ABFS Driver and can be removed | ||
| obsolete_configs_list=( |
There was a problem hiding this comment.
Where is the real mapping for supported configs defined?
There was a problem hiding this comment.
The configs supported and present in both ABFS, WASB remain as it is (except for endpoint change if required)
Script only makes changes for renames, deleting the obsoletes and error for unsupported ones
| # Remove the property block if any name tag is empty | ||
| xmlstarlet ed -L -d "//property[not(name) or name='']" "$OUTPUT_FILE" | ||
|
|
||
| echo "Updated file: $OUTPUT_FILE" |
There was a problem hiding this comment.
Is this script tested?
There was a problem hiding this comment.
Yes, tested with a sample config file to test the correct working and that the config values stay intact
manika137
force-pushed
the
HADOOP-19518_wasbScript
branch
from
093e7fb to
82c7c02
Compare
| @@ -0,0 +1,157 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
Better to keep file name in camel case or all characters in small case just like we have for other test scripts.
| exit 1 | ||
| fi | ||
|
|
||
| if [[ "$1" != *.xml ]]; then |
There was a problem hiding this comment.
Since we have already assigned $1 to FILE, it would be better to use $FILE. if [[ "$FILE" != *.xml ]]; then
| done | ||
|
|
||
| # Mapping for renaming configurations | ||
| declare -A rename_configs_map=( |
There was a problem hiding this comment.
variable naming should be as per java (use camel casing instead of snake casing)
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
| do | ||
| case $namespaceType in | ||
| HNS) | ||
| xmlstarlet ed -L -i '//configuration/property[1]' -t elem -n property -v '' \ |
There was a problem hiding this comment.
if ! command -v xmlstarlet &> /dev/null; then
echo "Error: 'xmlstarlet' is not installed. Please install it to run this script."
exit 1
fi
nit : If xmlstarlet is not installed, we should manually throw error
| ) | ||
|
|
||
| # Configurations not required in ABFS Driver and can be removed | ||
| obseleteConfigsList=( |
There was a problem hiding this comment.
typo: obsoleteConfigsList
|
|
||
| FILE=$1 | ||
|
|
||
| if [ ! -f "$FILE" ]; then |
There was a problem hiding this comment.
One overall suggestion would be to break down the different operations into functions, as we have lot going into the script. Some suggestion would be like
check_dependencies() { ... }
validate_input_file() { ... }
prompt_namespace_type() { ... }
rename_configurations() { ... }
remove_obsolete_configs() { ... }
handle_defaultFS_endpoint() { ... }
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
The legacy WASB driver has been deprecated and is no longer recommended for use. To support customer onboard for migration from WASB to ABFS driver, we've introduced a script to help with the configuration changes required for the same.
The script requires the configuration file (in XML format) used for WASB and would generate configuration file required for ABFS driver respectively.
JIRA ticket: https://issues.apache.org/jira/browse/HADOOP-19518
How was this patch tested?
No production code change, no testing needed.