Implement rebalance mechanism in Pulsar Functions #7388
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
srkukarni
reviewed
Jun 30, 2020
| @@ -93,6 +94,8 @@ | |||
|
|
|||
| private final IScheduler scheduler; | |||
|
|
|||
| private final IRebalance rebalancer; | |||
jerrypeng
changed the title
merlimat
reviewed
Jul 1, 2020
| * @param workers | ||
| * @return | ||
| */ | ||
| List<Function.Assignment> rebalance(List<Function.Assignment> currentAssignments, Set<String> workers); |
There was a problem hiding this comment.
Should we have a default implementation that just returns the current assignment?
There was a problem hiding this comment.
Sure we can do it for BC reasons. It would return a empty list because there will be no change
conf/functions_worker.yml
Outdated
| @@ -78,6 +78,7 @@ numHttpServerThreads: 8 | |||
| # function assignment and scheduler | |||
|
|
|||
| schedulerClassName: "org.apache.pulsar.functions.worker.scheduler.RoundRobinScheduler" | |||
| rebalancerClassName: "org.apache.pulsar.functions.worker.scheduler.RoundRobinScheduler" | |||
There was a problem hiding this comment.
This shouldn't be needed anymore, no?
merlimat
approved these changes
Jul 1, 2020
jerrypeng
force-pushed
the
functions_rebalance
branch
from
22ee6d6 to
0ac832c
Compare
sijie
approved these changes
Jul 2, 2020
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Aug 24, 2020
Co-authored-by: Jerry Peng <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Currently there no mechanism for rebalancing functions scheduler on workers. Workload for functions may become skewed.
Modifications
Will add mechanism to run the rebalance periodically automatically in subsequent PR.