Description
Connector Name
destination-snowflake
Connector Version
3.2.2
What step the error happened?
During the sync
Relevant information
For some reason, the connector seems to get stuck on flushing the queues to snowflake tables. I have a connection from SQL Server to Snowflake with 20 tables. The biggest table contains about 600K rows. When launching the sync, it seems that a first set of 6 tables is being fetched from the source and put on queues, the queues are flushed to the internal airbyte tables on snowflake, but then the sync gets stuck in a loop where those 6 queues are continuously being flushed again, even though they are empty. This also results in copy statements in snowflake that are importing 0 rows to the internal airbyte tables.
I've tried many things like resetting the connection, deleting and re-creating the connection,... but the issue persists. Below are the set of logs that are being emitted every 30 seconds or so.
Relevant log output
2023-10-14 16:32:28 destination > INFO i.a.i.d.FlushWorkers(printWorkerInfo):137 WORKER INFO
Pool queue size: 0, Active threads: 0
2023-10-14 16:32:37 destination > INFO i.a.i.d.DetectStreamToFlush(getNextStreamToFlush):123 flushing: trigger info: EXACT_200_V2_RAW - orsrg, time trigger: true , size trigger: false current threshold b: 10 MB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes
2023-10-14 16:32:37 destination > INFO i.a.i.d.FlushWorkers(lambda$flush$1):143 Flush Worker (be919) -- Worker picked up work.
2023-10-14 16:32:37 destination > INFO i.a.i.d.FlushWorkers(lambda$flush$1):145 Flush Worker (be919) -- Attempting to read from queue namespace: EXACT_200_V2_RAW, stream: orsrg.
2023-10-14 16:32:37 destination > INFO i.a.i.d.FlushWorkers(lambda$flush$1):158 Flush Worker (be919) -- Batch contains: 0 records, 0 bytes bytes.
2023-10-14 16:32:37 destination > WARN i.a.i.d.s.c.CsvSerializedBuffer(flushWriter):85 Trying to flush but no printer is initialized.
2023-10-14 16:32:37 destination > WARN i.a.i.d.s.c.CsvSerializedBuffer(closeWriter):95 Trying to close but no printer is initialized.
2023-10-14 16:32:37 destination > INFO i.a.i.d.r.BaseSerializedBuffer(flush):172 Finished writing data to 634fd0a4-e681-48e2-924e-abe25f440e5111479372499380664447.csv.gz (0 bytes)
2023-10-14 16:32:37 destination > INFO i.a.i.d.s.AsyncFlush(flush):98 Flushing CSV buffer for stream orsrg (0 bytes) to staging
2023-10-14 16:32:38 destination > INFO i.a.d.j.DefaultJdbcDatabase(lambda$unsafeQuery$1):115 closing connection
2023-10-14 16:32:38 destination > INFO i.a.i.d.s.SnowflakeInternalStagingSqlOperations(uploadRecordsToStage):96 Successfully loaded records to stage 2023/10/14/16/6C7F9768-568A-446E-A5A2-52822B9C72B6/ with 0 re-attempt(s)
2023-10-14 16:32:38 destination > INFO i.a.i.d.b.BufferManager(printQueueInfo):108 START OF QUEUE INFO
This represents an estimation of the size of the elements contain in the in memory buffer.
Global Mem Manager -- max: 742.41 MB, allocated: 84.97 MB (84.97187900543213 MB), % used: 0.11445469243481197
Queue name: frhsrg, num records: 0, num bytes: 0 bytes
Queue name: kstdr, num records: 0, num bytes: 0 bytes
Queue name: cicmpy, num records: 0, num bytes: 0 bytes
Queue name: amutak, num records: 0, num bytes: 0 bytes
Queue name: Items, num records: 0, num bytes: 0 bytes
Queue name: ItemClasses, num records: 0, num bytes: 0 bytes
Queue name: gbkmut, num records: 0, num bytes: 0 bytes
Queue name: grtbk, num records: 0, num bytes: 0 bytes
Queue name: orsrg, num records: 0, num bytes: 0 bytes
Queue name: humres, num records: 0, num bytes: 0 bytes
State Manager memory usage: Allocated: 9 MB, Used: -120335 bytes, percentage Used -0.011609
END OF QUEUE INFOContribute
- Yes, I want to contribute