Skip to content

feat(watermark): clean state for top n by watermark #8107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from

Conversation

yuhao-su
Copy link
Contributor

@yuhao-su yuhao-su commented Feb 21, 2023

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

  • clean state for top n by watermark

#7810
#8106

TODO:
add tests.

Checklist For Contributors

  • I have demonstrated that backward compatibility is not broken by breaking changes and created issues to track deprecated features to be removed in the future. (Please refer to the issue)
  • All checks passed in ./risedev check (or alias, ./risedev c)

Checklist For Reviewers

  • I have requested macro/micro-benchmarks as this PR can affect performance substantially, and the results are shown.

Documentation

  • My PR DOES NOT contain user-facing changes.
Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

  • Installation and deployment
  • Connector (sources & sinks)
  • SQL commands, functions, and operators
  • RisingWave cluster configuration changes
  • Other (please specify in the release note below)

Release note

@github-actions github-actions bot added the type/feature Type: New feature. label Feb 21, 2023
@yuhao-su yuhao-su marked this pull request as ready for review February 22, 2023 05:52
@codecov
Copy link

codecov bot commented Feb 22, 2023

Codecov Report

Merging #8107 (59798ed) into main (6c5f68f) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

❗ Current head 59798ed differs from pull request most recent head 4535013. Consider uploading reports for the commit 4535013 to get more accurate results

@@            Coverage Diff             @@
##             main    #8107      +/-   ##
==========================================
- Coverage   71.21%   71.19%   -0.03%     
==========================================
  Files        1126     1126              
  Lines      181215   181265      +50     
==========================================
- Hits       129053   129051       -2     
- Misses      52162    52214      +52     
Flag Coverage Δ
rust 71.19% <0.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/common/src/util/ordered/serde.rs 95.67% <0.00%> (-3.37%) ⬇️
...src/optimizer/plan_node/stream_watermark_filter.rs 58.41% <0.00%> (ø)
src/stream/src/executor/top_n/group_top_n.rs 92.35% <0.00%> (-0.61%) ⬇️
...tream/src/executor/top_n/group_top_n_appendonly.rs 0.00% <0.00%> (ø)
src/stream/src/executor/top_n/top_n_appendonly.rs 87.74% <0.00%> (-1.44%) ⬇️
src/stream/src/executor/top_n/top_n_plain.rs 94.46% <0.00%> (-1.99%) ⬇️
src/stream/src/executor/top_n/utils.rs 77.08% <ø> (ø)
src/stream/src/from_proto/top_n.rs 0.00% <0.00%> (ø)
src/batch/src/task/task_execution.rs 51.62% <0.00%> (-0.51%) ⬇️
src/storage/src/hummock/compactor/iterator.rs 96.40% <0.00%> (-0.28%) ⬇️
... and 6 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Contributor

@st1page st1page left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the logic of watermark derivation in optimizer too.

} else {
// TODO(yuhao): handle watermark in descending case
}
Ok(Some(watermark))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we emit the watermark in descending case? it might retract the lower values from the current topN result.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh not like this. Will fix.

watermark: Watermark,
) -> StreamExecutorResult<Option<Watermark>> {
Ok(if watermark.col_idx == self.storage_key_indices[0] {
Some(watermark)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto. descending case. I think we need to compare the last key with the watermark here.

@yuhao-su yuhao-su marked this pull request as draft February 23, 2023 04:36
@github-actions
Copy link
Contributor

This PR has been open for 60 days with no activity. Could you please update the status? Feel free to ping a reviewer if you are waiting for review.

@st1page st1page closed this May 11, 2023
@xxchan xxchan deleted the yuhao/watermark_top_n branch March 11, 2025 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants