Regression since PR #10417 was created (3 days ago) - Sink open produced uncaught exception

[devinbost]

I created a build from master and upgraded my brokers to it, and my function refuses to start due to the following exception. I tried restarting the brokers, but the exception did not go away.
I rolled back the brokers to a version I created 2-3 days ago (immediately after PR #10417 was created since I used the branch rdhabalia:batch_permit (https://github.com/rdhabalia/pulsar/tree/batch_permit) for the build), and the exception disappeared.

03:52:29.849 [myTenant/myNamespace/function-filter-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - Sink open produced uncaught exception:
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: Disconnected from server at server08.xyz.com/10.20.69.37:6650
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_282]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_282]
	at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957) ~[?:1.8.0_282]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940) ~[?:1.8.0_282]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_282]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_282]
	at org.apache.pulsar.client.impl.ClientCnx.lambda$channelInactive$2(ClientCnx.java:257) ~[java-instance.jar:?]
	at org.apache.pulsar.common.util.collections.ConcurrentLongHashMap$Section.forEach(ConcurrentLongHashMap.java:437) ~[java-instance.jar:?]
	at org.apache.pulsar.common.util.collections.ConcurrentLongHashMap.forEach(ConcurrentLongHashMap.java:163) ~[java-instance.jar:?]
	at org.apache.pulsar.client.impl.ClientCnx.channelInactive(ClientCnx.java:257) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[java-instance.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) ~[java-instance.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[java-instance.jar:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[java-instance.jar:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819) ~[java-instance.jar:?]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[java-instance.jar:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[java-instance.jar:?]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[java-instance.jar:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[java-instance.jar:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[java-instance.jar:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[java-instance.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.apache.pulsar.client.api.PulsarClientException: Disconnected from server at server08.xyz.com/10.20.69.37:6650
	at org.apache.pulsar.client.impl.ClientCnx.channelInactive(ClientCnx.java:254) ~[java-instance.jar:?]
	... 20 more
03:52:29.850 [myTenant/myNamespace/function-filter-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - [myTenant/myNamespace/function-filter:0] Uncaught exception in Java Instance
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: Disconnected from server at server08.xyz.com/10.20.69.37:6650
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_282]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_282]
	at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957) ~[?:1.8.0_282]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940) ~[?:1.8.0_282]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_282]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_282]
	at org.apache.pulsar.client.impl.ClientCnx.lambda$channelInactive$2(ClientCnx.java:257) ~[java-instance.jar:?]
	at org.apache.pulsar.common.util.collections.ConcurrentLongHashMap$Section.forEach(ConcurrentLongHashMap.java:437) ~[java-instance.jar:?]
	at org.apache.pulsar.common.util.collections.ConcurrentLongHashMap.forEach(ConcurrentLongHashMap.java:163) ~[java-instance.jar:?]
	at org.apache.pulsar.client.impl.ClientCnx.channelInactive(ClientCnx.java:257) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[java-instance.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) ~[java-instance.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[java-instance.jar:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[java-instance.jar:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[java-instance.jar:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819) ~[java-instance.jar:?]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[java-instance.jar:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[java-instance.jar:?]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[java-instance.jar:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[java-instance.jar:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[java-instance.jar:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[java-instance.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.apache.pulsar.client.api.PulsarClientException: Disconnected from server at server08.xyz.com/10.20.69.37:6650
	at org.apache.pulsar.client.impl.ClientCnx.channelInactive(ClientCnx.java:254) ~[java-instance.jar:?]
	... 20 more

[devinbost]

FYI @eolivelli @lhotari

[devinbost]

I created an even newer build from master (since writing this issue), and to my great surprise, the error was gone....
So, I'm closing this issue.
It disappeared as quickly as it appeared.
If anyone else sees it again, please re-open this issue.