[Bug]: FileBasedDeadLetterQueueReconsumer could result in duplicates

[damccorm]

Related Template(s)

Anything using FileBasedDeadLetterQueueReconsumer (as best I can tell, looks like just spanner changestreams + datastreams templates - https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20FileBasedDeadLetterQueueReconsumer&type=code )

Template Version

latest - seen in 2023-07-18-00_rc00, but also just based on code inspection

What happened?

The way

DataflowTemplates/v2/common/src/main/java/com/google/cloud/teleport/v2/cdc/dlq/FileBasedDeadLetterQueueReconsumer.java

Line 91 in 613b9aa

return in.getPipeline()

is written, it could result in duplicates if Dataflow experiences any sort of backlog or slowdown. Specifically, the following scenario could happen:

1. Generate sequence fires, and everything before the reshuffle happens
2. Generate sequence fires again, and everything before the reshuffle happens
3. We remove the file after the reshuffle
4. We try to remove the file after the reshuffle again, which logs but succeeds -

   DataflowTemplates/v2/common/src/main/java/com/google/cloud/teleport/v2/cdc/dlq/FileBasedDeadLetterQueueReconsumer.java

   Line 143 in 613b9aa

   LOG.error("Unable to delete file {}. Exception: {}", file, e);

Relevant log output

[damccorm]

Tagging some folks who may be interested in taking a look at this. Low likelihood bug which can lead to duplicates.

@manitgupta @Deep1998 @dhercher