[PD] Handle P/D failure and reconnect without affecting other instances

[ShangmingCai]

Motivation

This PR enables Mooncake to handle both prefill and decode failures and reconnect without affecting other healthy instances. You can restart the dead/killed prefill instance or decode instance when it fails, and continue to pair it with on-the-fly instances.

This PR requires mooncake-transfer-engine >= 0.3.2.post1

CC: @whybeyoung @ByronHsu

Modifications

• Add a health route in the bootstrap server, and add a heartbeat and clean thread on the decode side to handle prefill failures and abort requests.
• Identify failures with failed mooncake sessions when the transfer sync timeout occurs, which usually means the decode instance has been killed, and abort requests from the same session.
• Add a clear func for kv sender and receiver to make sure we clean the request status, which can prevent future failures caused by an identical bootstrap room. This should be helpful if users are using their own lb that might have a higher chance of generating the same bootstrap room for different requests after a period of time.
• Record what makes the room fail in the prefill side and sync failure to the decode node.
• Add failure exception impl to clear status and report the root cause.
• Optimize the logging level of all logs related to PD with mooncake.
• Remove the indices assertion to prevent a potential page size mismatch (only happens occasionally when PD+ chunked prefill + page size > 1, which is a bug that will be fixed) kills the transfer thread.
• Reduce the thread pool size for kv transfer to prevent consuming too many CPU resources.
• Add multiple env vars to enable user-specific configurations.

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[whybeyoung]

LGTM, thank you. Will test it .
It's a implement of feature request #6215