[fix][meta] Fix ephemeral handling of ZK nodes and fix MockZooKeeper ephemeral and ZK stat handling

[lhotari]

Motivation

This PR is a follow-up to #23984 to fix ephemeral status handling in ZKMetadataStore and to improve the ZooKeeper testing infrastructure.

A key addition is enabling pulsar-metadata tests to run against MockZooKeeper through a proper MetadataStoreProvider implementation, ensuring MockZooKeeper correctly implements all features that Pulsar depends on.

Modifications

1. Fixed Production Code Issues:

• Fixed ephemeral status handling in ZKMetadataStore to properly detect ephemeral nodes by using the correct comparison value (0L) for non-ephemeral (== persistent) nodes

2. MockZooKeeper improvements and fixes:

• Fixed ephemeral node handling and cleanup on session termination
• Improved ZK stat handling with proper timestamps and version numbers
• Added proper session ID management and consistent thread safety
• Fixed persistent watcher cleanup on session termination
• Fixed getChildren implementation and removed code duplication
• Fixed ordering issues by migrating to use a single threaded executor model
  • This allowed to remove locks and synchronization in most cases

3. Test Framework Improvements::

• Implemented MockZooKeeperMetadataStoreProvider to allow pulsar-metadata tests to use MockZooKeeper
  • This ensures that MockZooKeeper correctly implements all MetadataStore features that Pulsar depends on
• Enhanced PulsarTestContext to support toggling between MockZooKeeper and real ZK (TestZKServer) implementations
• Added ability to run MockedPulsarServiceBaseTest tests with both MockZooKeeper and real ZK (TestZKServer)

There are some unrelated NPE fixes and improvements for better exception messages. I faced those issues while debugging BrokerServiceTest and noticed those exceptions in the execution logs. Some of the exceptions were due to invalid tests, however the improvements will be useful also later.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[lhotari]

cherry-picking depends on #23686

[lhotari]

Some Pulsar tests are using MockZooKeeper's delay method. After #23988, MockZooKeeper is correctly single threaded and a delay will block the execution of this code in ZKSessionWatcher:

pulsar/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKSessionWatcher.java

Lines 108 to 115 in 6e8c349

	try {
	zkClientState = future.get(tickTimeMillis, TimeUnit.MILLISECONDS);
	} catch (TimeoutException e) {
	// Consider zk disconnection if zk operation takes more than TICK_TIME
	zkClientState = Watcher.Event.KeeperState.Disconnected;
	}
	checkState(zkClientState);

This will trigger a ConnectionLost session event. That's why a delay should be kept under 2000 ms.
Currently MockZooKeeper's session timeout is hard coded to 30000ms and the check interval is 1/15 of this, therefore 2000ms. It would be a good improvement to use the configured metadataStoreSessionTimeoutMillis instead.

[lhotari]

Some Pulsar tests are using MockZooKeeper's delay method. After #23988, MockZooKeeper is correctly single threaded and a delay will block the execution of this code in ZKSessionWatcher:

pulsar/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKSessionWatcher.java

Lines 108 to 115 in 6e8c349

	try {
	zkClientState = future.get(tickTimeMillis, TimeUnit.MILLISECONDS);
	} catch (TimeoutException e) {
	// Consider zk disconnection if zk operation takes more than TICK_TIME
	zkClientState = Watcher.Event.KeeperState.Disconnected;
	}
	checkState(zkClientState);

This will trigger a ConnectionLost session event. That's why a delay should be kept under 2000 ms. Currently MockZooKeeper's session timeout is hard coded to 30000ms and the check interval is 1/15 of this, therefore 2000ms. It would be a good improvement to use the configured metadataStoreSessionTimeoutMillis instead.

This is handled in #24171