ddl: support refresh meta ddl #60837

River2000i · 2025-04-25T06:55:30Z

What problem does this PR solve?

Issue Number: close #60924

Problem Summary:
When BR uses Tablemode, which is expected to be set to RestoreMode when full restore batch crate tables, including tables created during log backup. However, if an exchange partition occurs during log backup, the table will still be a partition during full restore. In log restoring, the table ID will exchange with partition ID, which exchange by meta kv directly instead of infoschema, so the table mode cannot be modified for the table(since the new table ID is not store in infoschema). In addition to exchange partition, this problem may also occur in operations such as drop table\schema. The root cause is that br will directly restore meta kv, which leads to inconsistency between meta kv and infoschema.

What changed and how does it work?

Implement RefreshMeta() DDL to update schema version by schema ID and new table ID. So infoschema can be consistent with meta kv when br restore meta kv.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2025-04-25T06:55:40Z

Hi @River2000i. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2025-04-25T06:55:47Z

Hi @River2000i. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lance6716 · 2025-04-29T03:11:48Z

/ok-to-test

codecov · 2025-04-29T03:30:05Z

Codecov Report

Attention: Patch coverage is 73.19588% with 26 lines in your changes missing coverage. Please review.

Project coverage is 74.8062%. Comparing base (61e061e) to head (bfeca3c).
Report is 16 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #60837        +/-   ##
================================================
+ Coverage   73.1447%   74.8062%   +1.6615%     
================================================
  Files          1724       1746        +22     
  Lines        477109     497778     +20669     
================================================
+ Hits         348980     372369     +23389     
+ Misses       106725     102247      -4478     
- Partials      21404      23162      +1758

Flag	Coverage Δ
integration	`48.8299% <0.0000%> (?)`
unit	`72.3764% <70.3296%> (+0.0099%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7804% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`62.1079% <ø> (+14.5596%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…Meta

River2000i · 2025-04-29T07:04:35Z

/retest

River2000i · 2025-04-30T09:47:57Z

/retest

Tristan1900

rest LGTM!

Tristan1900 · 2025-05-12T16:59:03Z

pkg/ddl/table.go

@@ -1699,3 +1699,25 @@ func onAlterNoCacheTable(jobCtx *jobContext, job *model.Job) (ver int64, err err
 	}
 	return ver, err
 }
+
+func onRefreshMeta(jobCtx *jobContext, job *model.Job) (ver int64, err error) {


could we add check here or at RefreshMeta to verify that the info schema doesn't have tableInfo with same table id or name as the job specified but without restoreMode? so to be safe that refresh doesn't override anything unexpected.

maybe we can skip refresh when tableinfo is exist in info schema and the tablemode isn't restoreMode?

Tristan1900 · 2025-05-12T20:48:06Z

pkg/ddl/table.go

+	if err != nil {
+		job.State = model.JobStateCancelled
+		return 0, errors.Trace(err)
+	}


if underlying table is removed by BR, can we update that to info schema as well. Right now it's assuming underlying TiKV also contain the specified table meta KV.

if underlying table is removed by BR

now skip table validate in ddl

…Meta

…chema

Tristan1900 · 2025-05-15T00:20:01Z

LGTM, but maybe please have DDL team to take another look of the change?
@River2000i
THanks

River2000i · 2025-05-15T01:13:43Z

/retest

GMHDBJD · 2025-05-15T12:10:18Z

pkg/infoschema/builder.go

+		}
+		return applyDropSchema(b, schemaDiff), nil
+	}
+	// Schema exists in kv but not in infoschema, drop it from infoschema.


drop -> create?

fix in bfeca3c

ti-chi-bot · 2025-05-15T12:11:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Benjamin2037, GMHDBJD, Leavrth, wjhuang2016

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/ddl/OWNERS~~ [Benjamin2037,GMHDBJD,wjhuang2016]
~~pkg/infoschema/OWNERS~~ [Benjamin2037,GMHDBJD,wjhuang2016]
~~pkg/meta/OWNERS~~ [Benjamin2037,GMHDBJD,wjhuang2016]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-05-15T12:11:05Z

[LGTM Timeline notifier]

Timeline:

2025-05-12 02:52:48.198177789 +0000 UTC m=+327234.346139018: ☑️ agreed by Leavrth.
2025-05-12 08:25:35.962075165 +0000 UTC m=+347202.110036379: ☑️ agreed by Benjamin2037.
2025-05-12 16:59:32.279696243 +0000 UTC m=+378038.427657447: ✖️🔁 reset by Tristan1900.
2025-05-15 08:38:29.839487928 +0000 UTC m=+15476.095385325: ☑️ agreed by wjhuang2016.
2025-05-15 12:11:04.402322863 +0000 UTC m=+28230.658220256: ☑️ agreed by GMHDBJD.

River2000i · 2025-05-15T12:23:46Z

/unhold

River2000i · 2025-05-15T13:09:18Z

/retest

River2000i · 2025-05-15T13:37:50Z

/retest

close pingcap#60924

implement RefreshMeta internal ddl

a4aa412

River2000i changed the title ~~ddl: support refresh meta ddl~~ [WIP]ddl: support refresh meta ddl Apr 25, 2025

ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 25, 2025

add test

2b591ad

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 28, 2025

update comment

beb2ad6

River2000i changed the title ~~[WIP]ddl: support refresh meta ddl~~ ddl: support refresh meta ddl Apr 29, 2025

ti-chi-bot bot removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels Apr 29, 2025

ti-chi-bot bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Apr 29, 2025

River2000i added 5 commits April 29, 2025 12:03

fix comment

f9a1cab

add ActionRefreshMeta to BDRActionMap

209ee26

fix comment

37264a9

fix ci

e7f869f

Merge remote-tracking branch 'upstream/master' into TableMode-Refresh…

f8efafe

…Meta

update comment

1f6545d

Tristan1900 suggested changes May 12, 2025

View reviewed changes

ti-chi-bot bot removed the lgtm label May 12, 2025

Tristan1900 reviewed May 12, 2025

View reviewed changes

River2000i added 4 commits May 13, 2025 11:27

Merge remote-tracking branch 'upstream/master' into TableMode-Refresh…

74e8530

…Meta

add ut

7638778

add test for drop table

b2a7b31

refresh meta support few scenarios of meta kv inconsistent with infos…

70507ff

…chema

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 14, 2025

River2000i added 3 commits May 14, 2025 16:45

update comment

04bbe0d

contruct new schemaDiff before apply

67263a7

fix comment

ff5f58d

wjhuang2016 approved these changes May 15, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label May 15, 2025

GMHDBJD approved these changes May 15, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels May 15, 2025

update comment

bfeca3c

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 15, 2025

ti-chi-bot bot merged commit 3a767d3 into pingcap:master May 15, 2025
27 checks passed

Tristan1900 mentioned this pull request May 7, 2025

br: add table filter for Pitr restore #57613

Open

17 tasks

River2000i added a commit to River2000i/tidb that referenced this pull request Jun 4, 2025

ddl: support refresh meta ddl (pingcap#60837)

de3ac1c

close pingcap#60924

River2000i mentioned this pull request Jun 4, 2025

tidb: release 8.5 20250114 v8.5.0 | tidb-test=release-8.5.1 #61493

Merged

13 tasks

ddl: support refresh meta ddl #60837

ddl: support refresh meta ddl #60837

Uh oh!

Conversation

River2000i commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

ti-chi-bot bot commented Apr 25, 2025

Uh oh!

tiprow bot commented Apr 25, 2025

Uh oh!

lance6716 commented Apr 29, 2025

Uh oh!

codecov bot commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

River2000i commented Apr 29, 2025

Uh oh!

River2000i commented Apr 30, 2025

Uh oh!

Tristan1900 left a comment

Choose a reason for hiding this comment

Uh oh!

Tristan1900 May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

River2000i May 13, 2025

Choose a reason for hiding this comment

Uh oh!

Tristan1900 May 12, 2025

Choose a reason for hiding this comment

Uh oh!

River2000i May 14, 2025

Choose a reason for hiding this comment

Uh oh!

Tristan1900 commented May 15, 2025

Uh oh!

River2000i commented May 15, 2025

Uh oh!

GMHDBJD May 15, 2025

Choose a reason for hiding this comment

Uh oh!

River2000i May 15, 2025

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented May 15, 2025

Uh oh!

ti-chi-bot bot commented May 15, 2025

[LGTM Timeline notifier]

Uh oh!

River2000i commented May 15, 2025

Uh oh!

River2000i commented May 15, 2025

Uh oh!

River2000i commented May 15, 2025

Uh oh!

Uh oh!

Uh oh!

River2000i commented Apr 25, 2025 •

edited

Loading

codecov bot commented Apr 29, 2025 •

edited

Loading

Tristan1900 May 12, 2025 •

edited

Loading