Skip to content

[Enhancement] Add source progress to result of show routine load (branch-3.1) (backport #35816) #38298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 3, 2024

Conversation

rickif
Copy link
Contributor

@rickif rickif commented Jan 3, 2024

This is a backport of #35816 to branch-3.1

Why I'm doing:
We need a easy way to check the lag of routine load job.
What I'm doing:
This PR adds SourceProgress to the result of show routine load, which denotes the producer's progress. In this way, we can check the lag of routine load job easily. Meanwhile, little overhead is added to the show routine load.

MySQL [db0]> show routine load\G;
*************************** 1. row ***************************
                  Id: 15008
                Name: rl_simple
          CreateTime: 2023-11-27 16:31:18
           PauseTime: NULL
             EndTime: NULL
              DbName: db0
           TableName: tbl_simple
               State: RUNNING
      DataSourceType: KAFKA
      CurrentTaskNum: 1
       JobProperties: {"partitions":"*","partial_update":"false","columnToColumnExpr":"key1,key2","maxBatchIntervalS":"10","partial_update_mode":"null","whereExpr":"*","dataFormat":"json","timezone":"GMT+08:00","format":"json","log_rejected_record_num":"0","taskTimeoutSecond":"60","json_root":"","maxFilterRatio":"1.0","strict_mode":"false","jsonpaths":"","taskConsumeSecond":"15","desireTaskConcurrentNum":"5","maxErrorNum":"0","strip_outer_array":"false","currentTaskConcurrentNum":"1","maxBatchRows":"200000"}
DataSourceProperties: {"topic":"topic-simple-json-continuous1","currentKafkaPartitions":"0","brokerList":"127.0.0.1:49092"}
    CustomProperties: {"group.id":"rl_simple_6dd18de6-a46d-4d43-8f36-4c71433cd5c5"}
           Statistic: {"receivedBytes":0,"errorRows":0,"committedTaskNum":0,"loadedRows":0,"loadRowsRate":0,"abortedTaskNum":13,"totalRows":0,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":1}
            Progress: {"0":"1007861"}
ReasonOfStateChanged: 
        ErrorLogUrls: 
         TrackingSQL: 
            OtherMsg: 
LatestSourcePosition: {"0":"2053532"}


Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.2
    • 3.1
    • 3.0
    • 2.5

…tarRocks#35816)

Signed-off-by: Alex Zhu <[email protected]>
Signed-off-by: ricky <[email protected]>
Co-authored-by: Alex Zhu <[email protected]>
Co-authored-by: wyb <[email protected]>
Signed-off-by: ricky <[email protected]>
@rickif rickif force-pushed the cherry-pick/branch-3.1-16e55e6 branch from e8a999e to f78f400 Compare January 3, 2024 06:05
Copy link

sonarqubecloud bot commented Jan 3, 2024

Copy link

github-actions bot commented Jan 3, 2024

[FE Incremental Coverage Report]

pass : 14 / 15 (93.33%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/load/routineload/PulsarRoutineLoadJob.java 0 1 00.00% [133]
🔵 com/starrocks/load/routineload/RoutineLoadJob.java 1 1 100.00% []
🔵 com/starrocks/load/routineload/KafkaTaskInfo.java 3 3 100.00% []
🔵 com/starrocks/load/routineload/KafkaRoutineLoadJob.java 9 9 100.00% []
🔵 com/starrocks/sql/ast/ShowRoutineLoadStmt.java 1 1 100.00% []

Copy link

github-actions bot commented Jan 3, 2024

[BE Incremental Coverage Report]

pass : 0 / 0 (0%)

@wyb wyb enabled auto-merge (squash) January 3, 2024 06:29
@wyb wyb merged commit 3f1548c into StarRocks:branch-3.1 Jan 3, 2024
@rickif rickif deleted the cherry-pick/branch-3.1-16e55e6 branch January 3, 2024 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants