Make the broker round robin selection in task execution configurable

[allenxwang]

Summary

1. Why:

We found that the rebalance operation significantly slows down towards the latter half of the execution. It happens when most of the tasks in the proposal involve a single broker. The current behavior is that the ExecutionTaskPlanner would sort the brokers according to the ReplicaMoveStrategy, continuously pick a task from the first broker until it reaches the broker's max concurrency. However, it would skip the broker in the same round if the broker is already involved in a picked task. The effect is that almost all tasks towards the end of the execution are related to the broker that has the most tasks, thus the execution is bottlenecked by the single broker's max concurrency.

2. What:

We will make this round robin behavior configurable and true by default. When it is set to false, the ExecutionTaskPlanner can keep picking the tasks from same broker that has the highest priority when it is asked to find the next set of tasks to execute. The ExecutionTaskPlanner would compare the broker to pick in the following order:

• use the ReplicaMoveStrategy to compare brokers by comparing the priority of its first set of tasks.
• If the above comparison is equal, it will depend on the total number of tasks involving that broker. This is exact what we need to prioritize the broker with large number of tasks.
• Fallback to the broker id comparison

Expected Behavior

…

Actual Behavior

…

Steps to Reproduce

1. …
2. …

Known Workarounds

Additional evidence

Categorization

• documentation
• bugfix
• new feature
• refactor
• security/CVE
• other

This PR resolves # if any.