Skip to content

feat(hash agg): do state cleaning by watermarks in hash agg #6330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 5, 2022

Conversation

soundOfDestiny
Copy link
Contributor

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

  • Summarize your change (mandatory)
    do state cleaning by watermarks in hash agg
  • How does this PR work? Need a brief introduction for the changed logic (optional)
    leverage delete_range interface in storage
  • Describe clearly one logical change and avoid lazy messages (optional)
    enumerate vnodes that belong to the table
  • Describe any limitations of the current code (optional)
    need to enumerate too many vnodes

Checklist

  • I have written necessary rustdoc comments
  • All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

Please keep the types that apply to your changes, and remove those that do not apply.

  • Installation and deployment
  • Connector (sources & sinks)
  • SQL commands, functions, and operators
  • RisingWave cluster configuration changes
  • Other (please specify in the release note below)

Release note

Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.

Refer to a related PR or issue link (optional)

#6112

@github-actions github-actions bot added the type/feature Type: New feature. label Nov 11, 2022
@codecov
Copy link

codecov bot commented Nov 11, 2022

Codecov Report

Merging #6330 (db1c4cc) into main (f036974) will decrease coverage by 0.00%.
The diff coverage is 54.54%.

@@            Coverage Diff             @@
##             main    #6330      +/-   ##
==========================================
- Coverage   73.23%   73.23%   -0.01%     
==========================================
  Files        1023     1023              
  Lines      163534   163543       +9     
==========================================
+ Hits       119760   119765       +5     
- Misses      43774    43778       +4     
Flag Coverage Δ
rust 73.23% <54.54%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/stream/src/executor/hash_agg.rs 95.92% <54.54%> (-0.56%) ⬇️
src/stream/src/executor/aggregation/minput.rs 96.39% <0.00%> (-0.11%) ⬇️
src/common/src/types/ordered_float.rs 32.23% <0.00%> (+0.19%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@wcy-fdu wcy-fdu removed their request for review November 13, 2022 05:55
@BugenZhao
Copy link
Member

Is this a sequel of #6299?

@BugenZhao BugenZhao marked this pull request as draft November 14, 2022 06:08
@soundOfDestiny soundOfDestiny marked this pull request as ready for review November 20, 2022 10:41
Copy link
Contributor

@st1page st1page left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

others LGTM

@@ -495,8 +505,14 @@ impl<K: HashKey, S: StateStore> HashAggExecutor<K, S> {
// Nothing to flush.
// Call commit on state table to increment the epoch.
iter_table_storage(storages).for_each(|state_table| {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can only apply on MaterializedInput but not on AggStateStorage::Table?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can only apply on MaterializedInput but not on AggStateStorage::Table?

why? register state table also starts with group key

futures::future::try_join_all(
iter_table_storage(storages).map(|state_table| state_table.commit(epoch)),
)
futures::future::try_join_all(iter_table_storage(storages).map(|state_table| async {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@mergify mergify bot merged commit 14a55f4 into main Dec 5, 2022
@mergify mergify bot deleted the zl_clnstate branch December 5, 2022 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/feature Type: New feature.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants