feat(hash agg): do state cleaning by watermarks in hash agg #6330
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## main #6330 +/- ##
==========================================
- Coverage 73.23% 73.23% -0.01%
==========================================
Files 1023 1023
Lines 163534 163543 +9
==========================================
+ Hits 119760 119765 +5
- Misses 43774 43778 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
Is this a sequel of #6299? |
st1page
reviewed
Nov 15, 2022
jon-chuang
reviewed
Nov 17, 2022
soundOfDestiny
force-pushed
the
zl_clnstate
branch
from
c6b80ed to
9bb5b1d
Compare
2 tasks
2 tasks
soundOfDestiny
force-pushed
the
zl_clnstate
branch
3 times, most recently
from
4664a38 to
f55f8d1
Compare
2 tasks
soundOfDestiny
force-pushed
the
zl_clnstate
branch
from
f55f8d1 to
5f34a84
Compare
st1page
reviewed
Dec 5, 2022
| @@ -495,8 +505,14 @@ impl<K: HashKey, S: StateStore> HashAggExecutor<K, S> { | |||
| // Nothing to flush. | |||
| // Call commit on state table to increment the epoch. | |||
| iter_table_storage(storages).for_each(|state_table| { | |||
There was a problem hiding this comment.
I think this can only apply on MaterializedInput but not on AggStateStorage::Table?
There was a problem hiding this comment.
I think this can only apply on
MaterializedInputbut not onAggStateStorage::Table?
why? register state table also starts with group key
| futures::future::try_join_all( | ||
| iter_table_storage(storages).map(|state_table| state_table.commit(epoch)), | ||
| ) | ||
| futures::future::try_join_all(iter_table_storage(storages).map(|state_table| async { |
st1page
approved these changes
Dec 5, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.
What's changed and what's your intention?
do state cleaning by watermarks in hash agg
leverage
delete_rangeinterface in storageenumerate vnodes that belong to the table
need to enumerate too many vnodes
Checklist
./risedev check(or alias,./risedev c)Documentation
If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.
Types of user-facing changes
Please keep the types that apply to your changes, and remove those that do not apply.
Release note
Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.
Refer to a related PR or issue link (optional)
#6112