[Enhancement] support alter hive/hudi catalog's properties #56212

haiboself · 2025-02-24T07:13:03Z

Why I'm doing:

When the Hive metastore address changes, we need to modify the Hive catalog's properties accordingly.

What I'm doing:

This PR adds support for modifying Hive/Hudi catalog properties, but requires restarting all FE nodes for the changes to take effect.

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

CLAassistant · 2025-02-24T07:13:09Z

All committers have signed the CLA.

DorianZheng · 2025-02-25T09:23:22Z

fe/fe-core/src/main/java/com/starrocks/server/CatalogMgr.java

-                catalog.getConfig().put("ranger.plugin.hive.service.name", serviceName);
+                catalog.getConfig().putAll(properties);
+                if (serviceName == null && !properties.isEmpty()) {
+                    LOG.info("Altering catalog properties (excluding `ranger.plugin.hive.service.name`) " +


The notion of restarting FE to enable newly added properties to take effect is not viable. Could you make these newly added properties take effect without restarting FE? I'm interested to talk about the design with you.

These properties changes, if applied directly, would affect the in-memory state of the catalog's metadata cache or require re-establishing the connection with the metastore. On one hand, implementing such changes is prone to causing issues and complex, and on the other hand, they would immediately impact business queries with potential irrecoverable consequences. Therefore, i think it is preferable to apply these changes through a restart. This approach simplifies implementation and minimizes the impact on users.

I agree that changing the state of the original catalog in place is error-prone. However, I think we can treat the catalog as immutable. Instead of modifying the state of the original catalog, we can create a new catalog with new properties and replace it atomically

could we reuse the create catalog logic with alter properties, and repalce the connector object atomically?

Solid suggestion. I'll carefully evaluate the feasibility and implications, then circle back with a detailed analysis.

@DorianZheng @Youngwb please re review

fe/fe-core/src/main/java/com/starrocks/connector/ConnectorMgr.java

fe/fe-core/src/main/java/com/starrocks/server/CatalogMgr.java

Youngwb · 2025-03-04T07:03:11Z

fe/fe-core/src/main/java/com/starrocks/server/CatalogMgr.java

+            // recreate catalog
+            reCreatCatalog(catalog, alterProperties, isReplay);
+        } else {
+            String catalogName = catalog.getName();


Why do we need to handle the logic for Ranger separately? Can't we reuse the logic reCreatCatalog?

Rebuilding a catalog is a heavyweight operation. Therefore, we handle it in two scenarios: we avoid rebuilding when unnecessary to prevent any impact on queries.

The operation of alter catalog is rarely performed. I don't think it's worth increasing the complexity of the code logic for this.

fe/fe-core/src/main/java/com/starrocks/sql/analyzer/CatalogAnalyzer.java

fe/fe-core/src/main/java/com/starrocks/server/CatalogMgr.java

Youngwb · 2025-03-05T02:26:10Z

fe/fe-core/src/main/java/com/starrocks/server/CatalogMgr.java

+            newConnector = connectorMgr.createHiddenConnector(
+                    new ConnectorContext(catalogName, type, newProperties), isReplay);
+            if (null == newConnector) {
+                throw new DdlException("create hidden connector failed");


Suggested change

throw new DdlException("create hidden connector failed");

throw new DdlException("alter catalog failed", e);

user should not known the hidden connector, and they need error message to change their alter command

I will catch this exception at line 209 and throw a new DdlException with the message 'ALTER operation failed: Connector creation error'. This will explicitly inform users that:

Their ALTER command failed

The failure reason is connector creation issue."

Can this solution resolve the problem?

line 209: throw new DdlException(String.format("Alter catalog failed, msg: [%s]", e.getMessage()), e);

Youngwb · 2025-03-05T03:42:58Z

fe/fe-core/src/main/java/com/starrocks/server/CatalogMgr.java

+            // replace old connector with new connector
+            connectorMgr.addConnector(catalogName, newConnector);
+
+            String serviceName = newProperties.get("ranger.plugin.hive.service.name");


If user first alter ranger.plugin.hive.service.name, then alter other properties, the AccessControl will be reset to RangerStarRocksAccessController or NativeAccessController

Youngwb · 2025-03-07T02:33:36Z

@haiboself Do you have a need to cp to branch-3.2? I think branch 3.3 is sufficient

Signed-off-by: haiboself <[email protected]>

haiboself · 2025-03-07T05:52:11Z

@haiboself Do you have a need to cp to branch-3.2? I think branch 3.3 is sufficien
@Youngwb Agreed, branch-3.3 should be sufficient for our current needs.

sonarqubecloud · 2025-03-07T05:54:00Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2025-03-07T07:28:22Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2025-03-07T07:29:10Z

[FE Incremental Coverage Report]

✅ pass : 46 / 51 (90.20%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/server/CatalogMgr.java	37	42	88.10%	[181, 193, 194, 196, 208]
🔵	com/starrocks/sql/analyzer/CatalogAnalyzer.java	3	3	100.00%	[]
🔵	com/starrocks/connector/ConnectorMgr.java	6	6	100.00%	[]

github-actions · 2025-03-07T07:29:12Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

haiboself · 2025-03-10T01:36:00Z

@DorianZheng @Youngwb please review and merge

github-actions · 2025-03-13T01:58:46Z

@Mergifyio backport branch-3.3

github-actions · 2025-03-13T01:58:46Z

@Mergifyio backport branch-3.4

mergify · 2025-03-13T01:59:12Z

backport branch-3.3

✅ Backports have been created

#56853 [Enhancement] support alter hive/hudi catalog's properties (backport #56212) has been created for branch branch-3.3 but encountered conflicts

mergify · 2025-03-13T01:59:13Z

backport branch-3.4

✅ Backports have been created

#56854 [Enhancement] support alter hive/hudi catalog's properties (backport #56212) has been created for branch branch-3.4

Signed-off-by: haiboself <[email protected]> (cherry picked from commit 1acf89f) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/server/CatalogMgr.java

Signed-off-by: haiboself <[email protected]> (cherry picked from commit 1acf89f)

…56212) (#56854) Co-authored-by: bo.hai <[email protected]>

…#56212) Signed-off-by: haiboself <[email protected]>

…56212) (#56870) Signed-off-by: haiboself <[email protected]>

…#56212) Signed-off-by: haiboself <[email protected]> Signed-off-by: crossoverJie <[email protected]>

haiboself requested a review from a team as a code owner February 24, 2025 07:13

github-actions bot added behavior_changed title needs [type] labels Feb 24, 2025

mergify bot assigned haiboself Feb 24, 2025

haiboself force-pushed the feature/support_alter_catalog_property branch from 3da5910 to 9b405ab Compare February 24, 2025 07:19

haiboself changed the title ~~[feature] support alter hive/hudi catalog's properties~~ [Enhancement] support alter hive/hudi catalog's properties Feb 24, 2025

github-actions bot added 3.4 3.2 3.3 and removed title needs [type] labels Feb 24, 2025

haiboself force-pushed the feature/support_alter_catalog_property branch from a1e07bb to 1bafcbe Compare February 25, 2025 02:20

DorianZheng reviewed Feb 25, 2025

View reviewed changes

haiboself requested a review from a team as a code owner March 4, 2025 05:57

haiboself requested review from DorianZheng and Youngwb March 4, 2025 05:58

Youngwb reviewed Mar 4, 2025

View reviewed changes

haiboself force-pushed the feature/support_alter_catalog_property branch from eadceeb to 3ad7e8a Compare March 4, 2025 08:23

haiboself requested a review from Youngwb March 4, 2025 09:58

Youngwb reviewed Mar 5, 2025

View reviewed changes

haiboself requested a review from Youngwb March 7, 2025 02:19

Youngwb approved these changes Mar 7, 2025

View reviewed changes

haiboself added 7 commits March 7, 2025 13:45

[feature] support alter hive/hudi catalog's properties

9f299e6

Signed-off-by: haiboself <[email protected]>

[fix] fix ut

ad43cb8

Signed-off-by: haiboself <[email protected]>

[fix] recreate connector when alter catalog

3596a89

Signed-off-by: haiboself <[email protected]>

[fix] fix by review

f4fa505

Signed-off-by: haiboself <[email protected]>

[fix] reCreatCatalog in all situation

40f4a3a

Signed-off-by: haiboself <[email protected]>

[fix] add unit test

1c182ac

Signed-off-by: haiboself <[email protected]>

[fix] fix

277874a

Signed-off-by: haiboself <[email protected]>

github-actions bot removed the 3.2 label Mar 7, 2025

HangyuanLiu self-assigned this Mar 11, 2025

HangyuanLiu approved these changes Mar 12, 2025

View reviewed changes

DorianZheng approved these changes Mar 12, 2025

View reviewed changes

DorianZheng enabled auto-merge (squash) March 12, 2025 09:36

stephen-shelby approved these changes Mar 13, 2025

View reviewed changes

DorianZheng merged commit 1acf89f into StarRocks:main Mar 13, 2025
67 checks passed

github-actions bot removed 3.3 3.4 labels Mar 13, 2025

mergify bot mentioned this pull request Mar 13, 2025

[Enhancement] support alter hive/hudi catalog's properties (backport #56212) #56853

Closed

18 tasks

mergify bot pushed a commit that referenced this pull request Mar 13, 2025

[Enhancement] support alter hive/hudi catalog's properties (#56212)

ebffd88

Signed-off-by: haiboself <[email protected]> (cherry picked from commit 1acf89f)

mergify bot mentioned this pull request Mar 13, 2025

[Enhancement] support alter hive/hudi catalog's properties (backport #56212) #56854

Merged

18 tasks

wanpengfei-git pushed a commit that referenced this pull request Mar 13, 2025

[Enhancement] support alter hive/hudi catalog's properties (backport #…

c83137c

…56212) (#56854) Co-authored-by: bo.hai <[email protected]>

github-actions bot added the 3.4-merged label Mar 13, 2025

haiboself added a commit to haiboself/starrocks that referenced this pull request Mar 13, 2025

[Enhancement] support alter hive/hudi catalog's properties (StarRocks…

ab02d61

…#56212) Signed-off-by: haiboself <[email protected]>

haiboself added a commit to haiboself/starrocks that referenced this pull request Mar 13, 2025

[Enhancement] support alter hive/hudi catalog's properties (StarRocks…

07c0b76

…#56212) Signed-off-by: haiboself <[email protected]>

dirtysalt pushed a commit that referenced this pull request Mar 13, 2025

[Enhancement] support alter hive/hudi catalog's properties (backport #…

608c1c6

…56212) (#56870) Signed-off-by: haiboself <[email protected]>

github-actions bot added the 3.3-merged label Mar 13, 2025

crossoverJie pushed a commit to crossoverJie/starrocks that referenced this pull request Mar 26, 2025

[Enhancement] support alter hive/hudi catalog's properties (StarRocks…

8e9fff6

…#56212) Signed-off-by: haiboself <[email protected]> Signed-off-by: crossoverJie <[email protected]>

	throw new DdlException("create hidden connector failed");
	throw new DdlException("alter catalog failed", e);

[Enhancement] support alter hive/hudi catalog's properties #56212

[Enhancement] support alter hive/hudi catalog's properties #56212

Uh oh!

Conversation

haiboself commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

CLAassistant commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DorianZheng Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DorianZheng Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haiboself Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Youngwb commented Mar 7, 2025

Uh oh!

haiboself commented Mar 7, 2025

Uh oh!

sonarqubecloud bot commented Mar 7, 2025

Quality Gate passed

Uh oh!

github-actions bot commented Mar 7, 2025

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions bot commented Mar 7, 2025

[FE Incremental Coverage Report]

file detail

Uh oh!

github-actions bot commented Mar 7, 2025

[BE Incremental Coverage Report]

Uh oh!

haiboself commented Mar 10, 2025

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2025

Uh oh!

github-actions bot commented Mar 13, 2025

Uh oh!

haiboself commented Feb 24, 2025 •

edited

Loading

CLAassistant commented Feb 24, 2025 •

edited

Loading

DorianZheng Feb 25, 2025 •

edited

Loading

DorianZheng Feb 25, 2025 •

edited

Loading

haiboself Mar 7, 2025 •

edited

Loading

mergify bot commented Mar 13, 2025 •

edited

Loading

mergify bot commented Mar 13, 2025 •

edited

Loading