Sync job fails/retries itself after successfully transferring all the data.

[gui0506]

Enviroment

• Airbyte version: 0.29.13-alpha
• OS Version / Instance: AWS EKS
• Deployment: Kubernetes
• Source Connector and version: Postgres 0.3.11
• Destination Connector and version: BigQuery 0.1.1
• Severity: Very Low / Low / Medium / High / Critical
• Step where error happened: Deploy / Sync job / Setup new connection / Update connector / Upgrade Airbyte

Current Behavior

Sync job fails after successfully transferring all the data. The kubernetes pods are terminated gracefully with completed status.

Expected Behavior

Sync job should not fail after successfully transferring all data.

Logs

logs-185-0.txt

2021-09-06 23:56:25 ERROR () DefaultReplicationWorker(run):148 - Sync worker failed.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Cannot find pod while trying to retrieve exit code. This probably means the Pod was not correctly created.
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:140) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:52) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:146) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Suppressed: java.lang.RuntimeException: Cannot find pod while trying to retrieve exit code. This probably means the Pod was not correctly created.
at io.airbyte.workers.process.KubePodProcess.getReturnCode(KubePodProcess.java:548) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at io.airbyte.workers.process.KubePodProcess.exitValue(KubePodProcess.java:573) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at java.lang.Process.hasExited(Process.java:333) ~[?:?]
at java.lang.Process.isAlive(Process.java:323) ~[?:?]
at io.airbyte.workers.WorkerUtils.gentleCloseWithHeartbeat(WorkerUtils.java:111) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at io.airbyte.workers.WorkerUtils.gentleCloseWithHeartbeat(WorkerUtils.java:95) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at io.airbyte.workers.protocols.airbyte.DefaultAirbyteSource.close(DefaultAirbyteSource.java:126) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:121) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:52) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:146) ~[io.airbyte-airbyte-workers-0.29.12-alpha.jar:?]
at java.lang.Thread.run(Thread.java:832) [?:?]

Steps to Reproduce

1. Set up a connection with Postgres as the source (CDC) and BigQuery as the target
2. Sync a large table (In my case, 10.18 GB | 37,656,941 records)
3. Sometimes the sync will fail and retry even though everything was successful. (Not always. About 20% of the time)

Are you willing to submit a PR?

[danieldiamond]

@sherifnada also experiencing this randomly now on subsequent syncs
CDC configuration
Source: MySQL 0.4.4
Destination: Snowflake 0.3.14
EC2 Docker Airbyte instance 0.29.17-alpha

[sherifnada]

Reading the logs, I can't see anything wrong with the sync itself which makes it look like a worker coordination issue.

@danieldiamond can you share logs from your failing instance?

cc @jrhizor do you have any ideas about what might be going on here?

[danieldiamond]

logs-23-0.txt

looking back at these logs, they actually start with the error

The connector is trying to read binlog starting at binlog file 'mysql-bin-changelog.162732', pos=834769, skipping 2 events plus 1 rows, but this is no longer available on the server. Reconfigure the connector to use a snapshot when needed.

but the job continues ahead and appears to "succeed" whilst still causing a retry

[sherifnada]

@danieldiamond what's the sync frequency on this one? this sounds like the sync waited "too long" and now the data is no longer in the binlog.

Alternatively, are you re-using the same CDC source in multiple connections, potentially having overlapping consumers of that binlog?

[danieldiamond]

attached sync timestamps

1. see sync frequency. it is manual, so i ran it once (full refresh) and then again 2 days later. i doubt the sync waited too long as that "successful but retried" sync took 8 minutes.
2. only one CDC source for one connection. (in a seperate airbyte instance i have the same CDC source, with the same account but that connection is disabled and this new instance, with full-refresh sync on 09/10 worked fine)

separately. on point 2. I do have multiple connections with the same CDC source to the same destination. is this not allowed? I run these two multiple connections at the same time and they appeared to work as expected although I run into the "hanging" issue, where it doesnt actually COPY the data after reading it.

[sherifnada]

I do have multiple connections with the same CDC source to the same destination. is this not allowed?

They shouldn't send into the same schema otherwise you'll get overwrites but it shouldn't cause the source connector to fail in either case.