Destination CDK: Skip flush of zero byte records.

[gisripa]

What

Related: #31425
Fixes: https://github.com/airbytehq/airbyte-internal-issues/issues/6728

In the old code, the below trace was the culprit in AsyncFlush.kt, fixing it for good in new flow.

2024-03-18 23:37:22 destination > WARN pool-4-thread-1 i.a.c.i.d.s.c.CsvSerializedBuffer(flushWriter):85 Trying to flush but no printer is initialized.
2024-03-18 23:37:22 destination > WARN pool-4-thread-1 i.a.c.i.d.s.c.CsvSerializedBuffer(closeWriter):95 Trying to close but no printer is initialized.
2024-03-18 23:37:22 destination > INFO pool-4-thread-1 i.a.c.i.d.r.BaseSerializedBuffer(flush):172 Finished writing data to c062ec97-5fca-4b9c-815e-dd35430f525b6128245000758765392.csv.gz (0 bytes)

This seems to happen when sources are slow and Async framework issues a flush irrespective of the number of records, leading to the Flush function just initializing empty buffer and doing UPLOAD/COPY with 0 byte files.

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jun 5, 2024 5:01pm

[gisripa]

• Destination Snowflake: Sync Id, generation_id and Meta #39107
• Destination CDK: Skip flush of zero byte records.  #39106 👈
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @gisripa and the rest of your teammates on Graphite

[edgao]

🤦 so it really was that easy

does it.flush need to be before the byteCount check? Or can we skip that too?

[gisripa]

🤦 so it really was that easy

does it.flush need to be before the byteCount check? Or can we skip that too?

we still need to do flush because weirdly that call does close of the file stream and all. otherwise we'll end up with open fds and open byte streams 🤦‍♂️