[fix][broker] Fix broker shutdown delay by resolving hanging health checks

[lhotari]

Fixes #23413

Motivation

When a broker is shutting down while there are ongoing health checks, the process can be significantly delayed. This happens because health check operations continue attempting to reconnect and retry, as evidenced in the logs where the internal Pulsar client keeps performing lookups without proper termination (example logs in https://gist.github.com/lhotari/1caca1bac6fbcff75e9ff6339c1626b5).

Modifications

• Extracted health check logic into a dedicated HealthChecker class for better separation of concerns
• Implemented proper handling of pending health check operations during shutdown:
  • Added tracking of pending CompletableFuture operations
  • Added logic to complete these futures with exceptions when the broker is shutting down
• Created a separate PulsarClient instance for health checks that can be properly closed during shutdown
  • This dedicated client allows health check operations to be properly terminated
  • Reused existing thread pools where appropriate (except for lookup and scheduled executors)
• Added helper methods to PulsarService for creating customized PulsarClient instances
• Refactored the heartbeat topic deletion process
• Added comprehensive javadocs to the new HealthChecker class
• Moved related tests to the broker group

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 77.95918% with 54 lines in your changes missing coverage. Please review.

Project coverage is 74.21%. Comparing base (bbc6224) to head (8bf6787).
Report is 1041 commits behind head on master.

Files with missing lines	Patch %	Lines
...rg/apache/pulsar/broker/service/HealthChecker.java	72.26%	35 Missing and 3 partials ⚠️
...n/java/org/apache/pulsar/broker/PulsarService.java	87.00%	8 Missing and 5 partials ⚠️
...rg/apache/pulsar/broker/service/BrokerService.java	33.33%	1 Missing and 1 partial ⚠️
...g/apache/pulsar/broker/admin/impl/BrokersBase.java	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24210      +/-   ##
============================================
+ Coverage     73.57%   74.21%   +0.64%     
+ Complexity    32624    32155     -469     
============================================
  Files          1877     1866      -11     
  Lines        139502   144728    +5226     
  Branches      15299    16530    +1231     
============================================
+ Hits         102638   107410    +4772     
+ Misses        28908    28835      -73     
- Partials       7956     8483     +527     

Flag	Coverage Δ
inttests	26.71% <47.75%> (+2.13%)	⬆️
systests	23.28% <12.24%> (-1.04%)	⬇️
unittests	73.70% <77.14%> (+0.86%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...g/apache/pulsar/broker/admin/impl/BrokersBase.java	88.51% <80.00%> (+16.05%)	⬆️
...rg/apache/pulsar/broker/service/BrokerService.java	83.74% <33.33%> (+2.96%)	⬆️
...n/java/org/apache/pulsar/broker/PulsarService.java	84.71% <87.00%> (+2.34%)	⬆️
...rg/apache/pulsar/broker/service/HealthChecker.java	72.26% <72.26%> (ø)

... and 1078 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.