Transactional stream transactions in limbo for a long time before failing with invalid epoch

[leviramsey]

Versions used

Alpakka Kafka 6.0.0
Akka version: 2.9.6

Observed Behavior

After network connectivity issues between consuming service and brokers (with some instances reporting ProducerFencincExceptions and failing), it is possible for a transactional stream to fail to complete a transaction but not abort the transaction until many tens of minutes later when a new message is consumed. At this point the production in the transaction fails with an invalid epoch.

[leviramsey]

Possibly related to #1829 (perhaps the difference in manifestation is due to a transaction being in flight at the time of disruption?)

[leviramsey]

The transaction timeout should be 10 seconds, so it's a bit suprising that the transaction isn't aborted shortly after 10 seconds has passed.