Skip to content

fix(core): Fix head-of-line blocking in concurrent tasks #5941

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 3, 2025

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Apr 2, 2025

Which issue does this PR close?

Close #5938

Rationale for this change

Our current ConcurrentTasks will poll the first task first before adding new tasks. In the worst cases, we will keep polling the first task without making any progress.

This PR removed this poll and only await on the tasks while our queue is full.

What changes are included in this PR?

  • Remove the busy poll on task.
  • Only await while task queue is full.
  • Avoid waiting for the first task to finish.

Are there any user-facing changes?

After this change, opendal can better use concurrent reading/writing.

Before

image

After

image

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Apr 2, 2025
@Xuanwo Xuanwo changed the title Optimize concurrent task feat(core): Reduce extra poll on concurrent tasks Apr 2, 2025
@dosubot dosubot bot added core releases-note/refactor The PR does a refactor on code or has a title that begins with "refactor" labels Apr 2, 2025
@github-actions github-actions bot added the releases-note/feat The PR implements a new feature or has a title that begins with "feat" label Apr 2, 2025
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Apr 2, 2025
@Xuanwo Xuanwo changed the title feat(core): Reduce extra poll on concurrent tasks fix(core): Fix head-of-line blocking in concurrent tasks Apr 2, 2025
@github-actions github-actions bot added the releases-note/fix The PR fixes a bug or has a title that begins with "fix" label Apr 2, 2025
@Xuanwo Xuanwo removed releases-note/feat The PR implements a new feature or has a title that begins with "feat" releases-note/refactor The PR does a refactor on code or has a title that begins with "refactor" labels Apr 2, 2025
@Xuanwo
Copy link
Member Author

Xuanwo commented Apr 2, 2025

Test failed for

image

I'm working on this.

Signed-off-by: Xuanwo <[email protected]>
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Apr 2, 2025
@Xuanwo Xuanwo requested review from meteorgan and tisonkun April 2, 2025 12:02
Signed-off-by: Xuanwo <[email protected]>
@Xuanwo Xuanwo requested a review from meteorgan April 2, 2025 14:16
Signed-off-by: Xuanwo <[email protected]>
@Xuanwo Xuanwo requested review from meteorgan and yihong0618 April 2, 2025 16:57
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Apr 3, 2025
@Xuanwo Xuanwo merged commit ad50cc6 into main Apr 3, 2025
279 checks passed
@Xuanwo Xuanwo deleted the optimize-concurrent-task branch April 3, 2025 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core lgtm This PR has been approved by a maintainer releases-note/fix The PR fixes a bug or has a title that begins with "fix" size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug: Recheck task scheduling
4 participants