Hope lws supports a pause feature similar to deployment

[loda13]

What would you like to be added:

1. It is hoped that lws can support a pause function similar to deployment, enabling the suspension of updates. After the pause is triggered, it should ensure that the current group of leaderworkers undergoing updates completes the update before pausing.
2. support resume

Why is this needed:

In the current PD separate deployment setup, the PD is deployed by two LWS instances, and updating the PD requires both LWS instances to jointly complete this update.

During a complete update process, it is necessary to ensure that the old and new versions of P can always find their corresponding versions of D nodes, in order to avoid the error of being unable to decode after the prefill request is completed.

1. D nodes are updated first, followed by P nodes, to ensure that the updated P nodes can always connect to the corresponding updated D nodes.

2. After all P nodes are fully updated, the remaining D nodes are updated to ensure that any remaining old P nodes can still work with old D nodes during the transition.

Completion requirements:

Shell           D (Decode)         P (Prefill)
  |                |                   |
  | -- Trigger D update -------------> |
  | <----------- D reports ready ----- |
  |                | -- Trigger P update -------------> |
  |                | <----------- P reports ready ----- |
  |                | -- Continue P batch update -------> |
  |                | <------ P batch update done ------- |
  | -- Trigger D batch update -------> |
  | <------ D batch update done ------ |

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[loda13]

Here’s an additional explanation:

1. The pause function is also needed to control the update speed of the two LWS for both P and D. This is to avoid a scenario where one (either P or D) finishes updating but the other just starts, resulting in too few available P-D pairs to handle online traffic—even during low-traffic periods.

2. After a request is routed, there must be a paired P and D to process it. While they don’t strictly need to be the same version, pause and update sequencing must be enforced to ensure this requirement.

[Luonon17]

I have similar troubles.

Whether LWS/rolloutStrategy can support manually triggering the update of Pods, with the aim of using the StatefulSet/UpdateStrategy/OnDelete.

[panpan0000]

Hi, @loda13 , by the way

How do you implement Disaggregated Prefill(P/D) with LWS?
who is the leader and who is the worker?
As we know, there're lmcache-server and router (take vllm-production-stack as example) which a shared basic, I believe they should not be managed by LWS.
if we put router as leader, but I can not separate two workers as P or D.
if we take 2 LWS instances ,one for Prefill, another for Decode , then who is leader for each instance ? (considering the model is small, we don't need to do multi-node inference, so one pod for P, one pod for D is enough for that small model )

[loda13]

Hi, @loda13 , by the way

How do you implement Disaggregated Prefill(P/D) with LWS? who is the leader and who is the worker? As we know, there're lmcache-server and router (take vllm-production-stack as example) which a shared basic, I believe they should not be managed by LWS. if we put router as leader, but I can not separate two workers as P or D. if we take 2 LWS instances , then who is leader for each instance ?

PD consists of two separate LWS , each with its own leader and worker. Specifically, P has a leader and a worker, and D also has a leader and a worker.

[kerthcet]

Is this similar to #511 @loda13

[kerthcet]

The updating progress is really interesting, what exactly pattern do you prefer? A parallel update of both P & D or D first then P, I think P first is harmful since no corresponding D there. Glad to hear more opinions. I'm thinking of a new orchestration patten build on top of lws/deployment to cover the complex PD disaggregating process.