[Bug]: CassandraToBigtableIT highly flaky

[Abacn]

Related Template(s)

cassandra-to-bigtable

Template Version

N/A

What happened?

It's been failing on both main branch and PR runs.

https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/2287/checks?check_run_id=39722028625.

I checked that there isn't recent change on Template code However

Update: it's not permared but highly flaky. We probably need retry or wait on Cassandra resource manager spin up

Relevant log output

com.datastax.oss.driver.api.core.AllNodesFailedException: Could not reach any contact point, make sure you've provided valid addresses (showing first 1 nodes, use getAllErrors() for more): Node(endPoint=/10.128.0.106:47906, hostId=null, hashCode=1b5c3f95): [com.datastax.oss.driver.api.core.connection.ConnectionInitException: [s0|control|connecting...] Protocol initialization request, step 1 (OPTIONS): failed to send request (io.netty.channel.StacklessClosedChannelException)]
	at com.datastax.oss.driver.api.core.AllNodesFailedException.copy(AllNodesFailedException.java:141)
	at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
	at com.datastax.oss.driver.api.core.session.SessionBuilder.build(SessionBuilder.java:835)
	at org.apache.beam.it.cassandra.CassandraResourceManager.<init>(CassandraResourceManager.java:97)
	Suppressed: com.datastax.oss.driver.api.core.connection.ConnectionInitException: [s0|control|connecting...] Protocol initialization request, step 1 (OPTIONS): failed to send request (io.netty.channel.StacklessClosedChannelException)
		at com.datastax.oss.driver.internal.core.channel.ProtocolInitHandler$InitRequest.fail(ProtocolInitHandler.java:356)
		at com.datastax.oss.driver.internal.core.channel.ChannelHandlerRequest.writeListener(ChannelHandlerRequest.java:87)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
		Suppressed: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.128.0.106:47906
		Caused by: java.net.ConnectException: Connection refused
			at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	Caused by: io.netty.channel.StacklessClosedChannelException
		at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0()(Unknown Source)

[Abacn]

cc: @georgecma could you please take a look?