[Enhancement] remove rouitine load plan lock #50784

hoffermei · 2024-09-06T03:47:45Z

Why I'm doing:

As trace below, routine load job plan takes much time, especially in table with many tablets case. In case routine load job with many concurrent tasks, conflict on lock will be outstanding.

Affect(class count: 3 , method count: 170) cost in 1839 ms, listenerId: 26
`---ts=2024-08-16 19:58:37;thread_name=pool-17-thread-5;id=1aa54a8;is_daemon=false;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@63947c6b
    `---[153.629655ms] com.starrocks.load.routineload.RoutineLoadJob:plan()
        +---[0.00% 0.001884ms ] com.starrocks.server.GlobalStateMgr:getCurrentState() #746
        +---[0.00% 0.001393ms ] com.starrocks.server.GlobalStateMgr:getDb() #746
        +---[0.00% 0.001403ms ] com.starrocks.catalog.Database:getTable() #750
        +---[0.00% 0.001974ms ] com.starrocks.load.LoadUtils:buildIcebergTableSink() #754
        +---[6.02% 9.242233ms ] com.starrocks.catalog.Database:readLock() #755
        +---[0.05% 0.072716ms ] com.starrocks.load.streamload.StreamLoadInfo:fromRoutineLoadJob() #758
        |   +---[3.42% 0.002485ms ] com.starrocks.load.routineload.RoutineLoadJob:getFormat()
        |   +---[2.12% min=6.52E-4ms,max=8.92E-4ms,total=0.001544ms,count=2] com.starrocks.load.routineload.RoutineLoadJob:getColumnDescs()
        |   +---[1.01% 7.32E-4ms ] com.starrocks.load.routineload.RoutineLoadJob:getWhereExpr()
        |   +---[1.24% 9.02E-4ms ] com.starrocks.load.routineload.RoutineLoadJob:getMergeCondition()
        |   +---[0.98% 7.11E-4ms ] com.starrocks.load.routineload.RoutineLoadJob:getColumnSeparator()
        |   +---[1.02% 7.42E-4ms ] com.starrocks.load.routineload.RoutineLoadJob:getRowDelimiter()
        |   +---[0.91% 6.61E-4ms ] com.starrocks.load.routineload.RoutineLoadJob:getPartitions()
        |   +---[3.28% 0.002385ms ] com.starrocks.load.routineload.RoutineLoadJob:isStrictMode()
        |   +---[1.61% 0.001172ms ] com.starrocks.load.routineload.RoutineLoadJob:getTimezone()
        |   +---[1.69% 0.001232ms ] com.starrocks.load.routineload.RoutineLoadJob:getJsonPaths()
        |   +---[1.39% 0.001011ms ] com.starrocks.load.routineload.RoutineLoadJob:getJsonRoot()
        |   +---[1.50% 0.001092ms ] com.starrocks.load.routineload.RoutineLoadJob:isStripOuterArray()
        |   +---[1.87% 0.001363ms ] com.starrocks.load.routineload.RoutineLoadJob:isPartialUpdate()
        |   +---[3.47% min=6.01E-4ms,max=6.81E-4ms,total=0.002524ms,count=4] com.starrocks.load.routineload.RoutineLoadJob:getSessionVariables()
        |   +---[1.79% 0.001303ms ] com.starrocks.load.routineload.RoutineLoadJob:getSinkWithBrokerName()
        |   +---[2.66% 0.001934ms ] com.starrocks.load.routineload.RoutineLoadJob:isDiscardUnknownFields()
        |   +---[1.60% 0.001162ms ] com.starrocks.load.routineload.RoutineLoadJob:isAutoFillMissingFields()
        |   `---[1.64% 0.001192ms ] com.starrocks.load.routineload.RoutineLoadJob:isEnableDeltaLHWrite()
        +---[0.01% 0.012053ms ] com.starrocks.planner.StreamLoadPlanner:<init>() #758
        +---[93.88% 144.230367ms ] com.starrocks.planner.StreamLoadPlanner:plan() #759
        +---[0.00% 0.002204ms ] com.starrocks.thrift.TQueryOptions:setLoad_job_type() #760
        +---[0.00% 0.001683ms ] com.starrocks.server.GlobalStateMgr:getCurrentGlobalTransactionMgr() #763
        +---[0.00% 0.001853ms ] com.starrocks.catalog.Database:getId() #763
        +---[0.00% 0.003156ms ] com.starrocks.transaction.GlobalTransactionMgr:getTransactionState() #763
        +---[0.00% 0.001232ms ] com.starrocks.planner.StreamLoadPlanner:getDestTable() #767
        +---[0.00% 0.004238ms ] com.starrocks.transaction.TransactionState:addTableIndexes() #769
        `---[0.01% 0.013325ms ] com.starrocks.catalog.Database:readUnlock() #773
        

`---ts=2024-08-16 20:00:56;thread_name=pool-17-thread-47;id=1aa54d7;is_daemon=false;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@63947c6b
    `---[226.964576ms] com.starrocks.planner.StreamLoadPlanner:plan()
        +---[0.00% 0.001283ms ] com.starrocks.catalog.OlapTable:getKeysType() #153
        +---[0.00% 0.007694ms ] com.starrocks.planner.StreamLoadPlanner:resetAnalyzer() #155
        +---[0.00% 0.001864ms ] com.starrocks.analysis.DescriptorTable:createTupleDescriptor() #157
        +---[0.00% 0.001322ms ] com.starrocks.load.streamload.StreamLoadInfo:getNegative() #158
        +---[0.00% 0.001333ms ] com.starrocks.load.streamload.StreamLoadInfo:isPartialUpdate() #164
        +---[0.00% 0.001092ms ] com.google.common.collect.Lists:newArrayList() #168
        +---[0.00% 6.81E-4ms ] com.starrocks.load.streamload.StreamLoadInfo:isPartialUpdate() #170
        +---[0.00% 0.001282ms ] com.starrocks.catalog.Table:getFullSchema() #173
        +---[0.11% min=6.71E-4ms,max=0.018104ms,total=0.242874ms,count=265] com.starrocks.analysis.DescriptorTable:addSlotDescriptor() #176
        +---[0.09% min=5.51E-4ms,max=0.018615ms,total=0.199948ms,count=265] com.starrocks.analysis.SlotDescriptor:setIsMaterialized() #177
        +---[0.11% min=6.41E-4ms,max=0.016832ms,total=0.240397ms,count=265] com.starrocks.analysis.SlotDescriptor:setColumn() #178
        +---[0.06% min=5.21E-4ms,max=0.001242ms,total=0.144655ms,count=265] com.starrocks.catalog.Column:isAllowNull() #179
        +---[0.07% min=5.31E-4ms,max=0.021229ms,total=0.169557ms,count=265] com.starrocks.analysis.SlotDescriptor:setIsNullable() #179
        +---[0.08% min=5.51E-4ms,max=0.017352ms,total=0.178227ms,count=265] com.starrocks.catalog.Column:getType() #184
        +---[0.07% min=5.2E-4ms,max=0.008285ms,total=0.160288ms,count=265] com.starrocks.catalog.Type:isVarchar() #184
        +---[0.01% min=6.31E-4ms,max=0.001382ms,total=0.026249ms,count=39] com.starrocks.sql.optimizer.statistics.IDictManager:getInstance() #185
        +---[0.01% min=5.1E-4ms,max=0.009478ms,total=0.030535ms,count=39] com.starrocks.catalog.Table:getId() #185
        +---[0.01% min=5.71E-4ms,max=0.001122ms,total=0.024064ms,count=39] com.starrocks.catalog.Column:getName() #186
        +---[0.02% min=6.41E-4ms,max=0.015489ms,total=0.052346ms,count=39] com.starrocks.sql.optimizer.statistics.IDictManager:hasGlobalDict() #185
        +---[0.00% 0.001202ms ] com.starrocks.planner.PlanNodeId:<init>() #200
        +---[0.00% 0.003376ms ] com.starrocks.planner.StreamLoadScanNode:<init>() #200
        +---[0.00% 0.001753ms ] com.starrocks.planner.StreamLoadScanNode:setUseVectorizedLoad() #202
        +---[0.22% 0.507622ms ] com.starrocks.planner.StreamLoadScanNode:init() #203
        +---[0.11% 0.243186ms ] com.starrocks.planner.StreamLoadScanNode:finalizeStats() #204
        +---[0.01% 0.020398ms ] com.starrocks.analysis.DescriptorTable:computeMemLayout() #206
        +---[0.00% 0.001613ms ] com.starrocks.catalog.OlapTable:writeQuorum() #213
        +---[0.01% 0.029415ms ] com.starrocks.planner.StreamLoadPlanner:getAllPartitionIds() #214
        +---[0.00% 0.001593ms ] com.starrocks.catalog.OlapTable:getExternalTable() #223
        +---[0.00% 0.001322ms ] com.starrocks.load.streamload.StreamLoadInfo:isEnableDeltaLHWrite() #224
        +---[0.00% 0.001502ms ] com.starrocks.catalog.OlapTable:enableReplicatedStorage() #253
        +---[0.00% 0.001854ms ] com.starrocks.planner.OlapTableSink:<init>() #253
        +---[0.00% 0.001323ms ] com.starrocks.load.streamload.StreamLoadInfo:getTxnId() #254
        +---[0.00% 0.001132ms ] com.starrocks.catalog.Database:getId() #254
        +---[0.00% 0.001253ms ] com.starrocks.load.streamload.StreamLoadInfo:getTimeout() #254
        +---[0.00% 0.005821ms ] com.starrocks.planner.OlapTableSink:init() #254
        +---[0.00% 0.001272ms ] com.starrocks.load.streamload.StreamLoadInfo:getMergeConditionStr() #255
        +---[0.00% 0.001322ms ] com.starrocks.load.Load:checkMergeCondition() #255
        +---[0.00% 8.51E-4ms ] com.starrocks.load.streamload.StreamLoadInfo:getMergeConditionStr() #256
        +---[98.15% 222.766132ms ] com.starrocks.planner.OlapTableSink:complete() #256
        +---[0.00% 0.001883ms ] com.starrocks.planner.PlanFragmentId:<init>() #282
        +---[0.00% 0.003367ms ] com.starrocks.planner.PlanFragment:<init>() #282
        +---[0.00% 0.002054ms ] com.starrocks.planner.PlanFragment:setSink() #283
        +---[0.00% 0.001042ms ] com.starrocks.planner.PlanFragment:setPipelineDop() #288
        +---[0.00% 7.52E-4ms ] com.starrocks.planner.PlanFragment:setSink() #289
        +---[0.00% 0.001613ms ] com.starrocks.planner.PlanFragment:setLoadGlobalDicts() #297
        +---[0.00% 0.001392ms ] com.starrocks.planner.PlanFragment:createDataSink() #299
        +---[0.00% 0.001322ms ] com.starrocks.thrift.TExecPlanFragmentParams:<init>() #301
        
        
`---ts=2024-08-28 15:14:31;thread_name=pool-17-thread-1;id=4dd7b8;is_daemon=false;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@63947c6b
    `---[314.711223ms] com.starrocks.planner.OlapTableSink:createLocation()
        `---[100.00% 314.701295ms ] com.starrocks.planner.OlapTableSink:createLocation() #391
            `---[100.00% 314.689062ms ] com.starrocks.planner.DataSink:createLocation()
                +---[0.00% 8.31E-4ms ] com.starrocks.thrift.TOlapTableLocationParam:<init>() #117
                +---[0.00% 9.21E-4ms ] com.google.common.collect.HashMultimap:create() #119
                +---[0.00% 6.11E-4ms ] com.starrocks.server.GlobalStateMgr:getCurrentState() #121
                +---[0.00% 0.001683ms ] com.starrocks.server.GlobalStateMgr:getOrCreateSystemInfo() #122
                +---[0.01% min=7.62E-4ms,max=0.002765ms,total=0.025407ms,count=11] com.starrocks.catalog.OlapTable:getPartition() #124
                +---[0.00% min=4.11E-4ms,max=7.31E-4ms,total=0.00511ms,count=11] com.starrocks.catalog.OlapTable:getPartitionInfo() #125
                +---[0.00% min=3.2E-4ms,max=5.41E-4ms,total=0.004088ms,count=11] com.starrocks.catalog.Partition:getId() #125
                +---[0.00% min=4.11E-4ms,max=6.91E-4ms,total=0.005551ms,count=11] com.starrocks.catalog.OlapTable:writeQuorum() #125
                +---[0.00% min=7.11E-4ms,max=0.001092ms,total=0.009489ms,count=11] com.starrocks.catalog.PartitionInfo:getQuorumNum() #125
                +---[0.01% min=9.12E-4ms,max=0.002354ms,total=0.0177ms,count=11] com.starrocks.catalog.Partition:getMaterializedIndices() #126
                +---[0.00% min=3.6E-4ms,max=6.52E-4ms,total=0.004668ms,count=11] com.starrocks.catalog.MaterializedIndex:getTablets() #127
                +---[2.14% min=2.8E-4ms,max=0.045635ms,total=6.740494ms,count=22528] com.starrocks.catalog.OlapTable:isLakeTable() #128
                +---[2.19% min=2.8E-4ms,max=0.011712ms,total=6.882584ms,count=22528] com.starrocks.catalog.OlapTable:getClusterId() #136
                +---[14.36% min=8.21E-4ms,max=0.581831ms,total=45.190484ms,count=22528] com.starrocks.catalog.LocalTablet:getNormalReplicaBackendPathMap() #136
                +---[2.33% min=3.0E-4ms,max=0.009458ms,total=7.339949ms,count=22528] com.google.common.collect.Multimap:keySet() #137
                +---[2.32% min=3.0E-4ms,max=0.01069ms,total=7.302429ms,count=22528] com.google.common.collect.Multimap:keySet() #144
                +---[3.71% min=4.7E-4ms,max=0.01608ms,total=11.668303ms,count=22528] com.google.common.collect.Lists:newArrayList() #144
                +---[2.09% min=2.7E-4ms,max=0.014968ms,total=6.578761ms,count=22528] com.starrocks.catalog.Tablet:getId() #176
                +---[2.37% min=3.0E-4ms,max=0.009588ms,total=7.444809ms,count=22528] com.starrocks.thrift.TTabletLocation:<init>() #177
                +---[2.84% min=3.7E-4ms,max=0.027272ms,total=8.947718ms,count=22528] com.starrocks.thrift.TOlapTableLocationParam:addToTablets() #175
                +---[2.42% min=3.2E-4ms,max=0.013275ms,total=7.630027ms,count=22528] com.google.common.collect.Multimap:entries() #178
                +---[6.51% min=2.8E-4ms,max=0.018976ms,total=20.48902ms,count=67584] com.starrocks.catalog.Replica:getBackendId() #179
                +---[9.44% min=3.6E-4ms,max=0.019376ms,total=29.716088ms,count=67584] com.google.common.collect.Multimap:put() #179
                +---[0.00% 7.91E-4ms ] com.starrocks.server.GlobalStateMgr:getCurrentSystemInfo() #188
                +---[0.02% 0.063088ms ] com.starrocks.system.SystemInfoService:checkExceedDiskCapacityLimit() #188
                `---[0.00% 8.71E-4ms ] com.starrocks.common.Status:ok() #189

What I'm doing:

move routine load plan out of read lock (table lock in 3.3 and db lock in 2.5), by only make a shadow copy of table in read lock, then generate plan by use table's shadow copy as parameter without read lock.

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Signed-off-by: hoffermei <[email protected]>

sonarqubecloud · 2024-09-06T03:55:26Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-09-06T05:13:37Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-09-06T05:14:11Z

[FE Incremental Coverage Report]

✅ pass : 30 / 35 (85.71%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/load/routineload/RoutineLoadJob.java	24	29	82.76%	[910, 912, 913, 914, 916]
🔵	com/starrocks/catalog/OlapTable.java	6	6	100.00%	[]

github-actions · 2024-09-06T05:14:19Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

hoffermei requested a review from a team as a code owner September 6, 2024 03:47

github-actions bot added 3.3 3.2 3.1 3.0 2.5 labels Sep 6, 2024

mergify bot assigned hoffermei Sep 6, 2024

remove rouitine load plan lock

66908f3

Signed-off-by: hoffermei <[email protected]>

hoffermei force-pushed the remove_routine_load_plan_lock branch from 13e3acd to 66908f3 Compare September 6, 2024 03:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Enhancement] remove rouitine load plan lock #50784

[Enhancement] remove rouitine load plan lock #50784

hoffermei commented Sep 6, 2024

Uh oh!

sonarqubecloud bot commented Sep 6, 2024

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

Uh oh!

[Enhancement] remove rouitine load plan lock #50784

Are you sure you want to change the base?

[Enhancement] remove rouitine load plan lock #50784

Conversation

hoffermei commented Sep 6, 2024

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

sonarqubecloud bot commented Sep 6, 2024

Quality Gate passed

Uh oh!

github-actions bot commented Sep 6, 2024

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions bot commented Sep 6, 2024

[FE Incremental Coverage Report]

file detail

Uh oh!

github-actions bot commented Sep 6, 2024

[BE Incremental Coverage Report]

Uh oh!

Uh oh!