Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc #57578

edgao · 2025-04-11T16:22:12Z

in very broad strokes:

CollectionUtils, ConnectorExceptionUtils - just copying code from the old CDK to the bulk CDK
AirbyteType - adding some utility methods to make our life easier
toolkits/load-db - the actual core logic
- there are a few classes, which represent fundamental functionality
- and subpackages for different load styles:
  - direct_load_table - this is the skeleton for how we'll implement direct-load tables
  - legacy_typing_deduping - a full implementation of T+D
- notably: the intended usage is for the destination to build a Factory, which returns the appropriate Writer. See e.g. BigqueryWriter.
and then there's just a ton of implementation code in destination-bigquery

vercel · 2025-04-11T16:22:17Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 24, 2025 4:42pm

edgao · 2025-04-17T16:49:44Z

airbyte-cdk/bulk/toolkits/load-db/src/main/kotlin/io/airbyte/cdk/load/orchestration/Sql.kt

+ *
+ * Callers are encouraged to use the static factory methods instead of the public constructor.
+ */
+data class Sql(val transactions: List<List<String>>) {


this is a copy of the existing Sql class

edgao · 2025-04-17T16:50:17Z

...ulk/toolkits/load-db/src/main/kotlin/io/airbyte/cdk/load/orchestration/DestinationHandler.kt

+
+package io.airbyte.cdk.load.orchestration
+
+interface DestinationHandler {


this is similar to the existing DestinationHandler interface, but trimmed down. The "initial status" stuff has ben moved to DestinationInitialStatusGatherer.

What a name

Oh, this refers to the Destination not the Destination. I was confused

This is really a DatabaseDestinationHandler?

also FYI the other db stuff has db in the namespace (cdk.load.db.orchestration)

Or StructuredDataStoreHandler if you want to get really pedantic, since it will also cover warehouses.

I'll go with "database", I feel like context is sufficiently clear that it also refers to warehouses :P

renamed in (as yet unpushed) bafeb80

edgao · 2025-04-17T16:51:18Z

...byte/cdk/load/orchestration/legacy_typing_deduping/TypingDedupingDestinationInitialStatus.kt

+
+import io.airbyte.cdk.load.orchestration.DestinationInitialStatus
+import java.time.Instant
+


this is mostly identical to the old DestinationInitialStatus class, but cleaned up for kotlin style.

frifriSF59

Some nits / tweaks but I think this looks good

frifriSF59 · 2025-04-23T15:52:49Z

...c/main/kotlin/io/airbyte/cdk/load/orchestration/legacy_typing_deduping/TypingDedupingUtil.kt

+object TypingDedupingUtil {
+    // copied wholesale from old CDK's StreamId
+    fun concatenateRawTableName(namespace: String, name: String): String {
+        val plainConcat = namespace + name


Thoughts about this:

val plain = namespace + name // Find all underscore‐runs, get the max length (or 1 if none) val longestRun = Regex("_+") .findAll(plain) .map { it.value.length } .maxOrNull() .let { max(it ?: 0, 1) } // Build: namespace + "_raw" + "_" * (longestRun + 1) + "stream_" + name val underscores = "_".repeat(longestRun + 1) return buildString { append(namespace) append("_raw") append(underscores) append("stream_") append(name) }

done in 7477f85 (I used a string template instead of buildstring, IMO it's easier to read)

...main/kotlin/io/airbyte/cdk/load/orchestration/legacy_typing_deduping/TypingDedupingWriter.kt

frifriSF59 · 2025-04-23T15:59:43Z

...y/src/main/kotlin/io/airbyte/integrations/destination/bigquery/BigQuerySQLNameTransformer.kt

+ *
+ * (I have no explanation for the method names.)
+ */
+class BigQuerySQLNameTransformer {


Should we mark this as deprecated?

mm. I think what we should do is actually rename it + the methods for clarity. Not as part of this PR though, the diff would be really annoying

frifriSF59 · 2025-04-23T16:01:14Z

...ation-bigquery/src/main/kotlin/io/airbyte/integrations/destination/bigquery/BigQueryUtils.kt

+                        """{
+                      "format_type": "CSV",
+                      "flattening": "No flattening"
+                    }"""


Should we have a .trimIndent() here?

.../kotlin/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.kt

...tlin/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQueryDatabaseHandler.kt

...e/integrations/destination/bigquery/typing_deduping/BigqueryDatabaseInitialStatusGatherer.kt

# Conflicts: # airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/message/DestinationMessage.kt # airbyte-integrations/connectors/destination-bigquery/src/main/kotlin/io/airbyte/integrations/destination/bigquery/formatter/BigQueryRecordFormatter.kt # airbyte-integrations/connectors/destination-bigquery/src/main/kotlin/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.kt # airbyte-integrations/connectors/destination-bigquery/src/test-integration/kotlin/io/airbyte/integrations/destination/bigquery/BigQueryDestinationTestUtils.kt # airbyte-integrations/connectors/destination-bigquery/src/test-integration/kotlin/io/airbyte/integrations/destination/bigquery/BigqueryWriteTest.kt # airbyte-integrations/connectors/destination-bigquery/src/test/kotlin/io/airbyte/integrations/destination/bigquery/typing_deduping/BigqueryDestinationHandlerTest.kt

edgao changed the base branch from master to destination_bigquery_bulk_cdk April 11, 2025 16:22

octavia-squidington-iii added the connectors/destination/bigquery label Apr 11, 2025

vercel bot deployed to Preview April 11, 2025 16:28 View deployment

vercel bot deployed to Preview April 14, 2025 17:45 View deployment

edgao force-pushed the destination_bigquery_bulk_cdk branch from 9d266d5 to 8acd8fb Compare April 14, 2025 20:02

edgao force-pushed the edgao/bigquery_call_old_code branch from f7306f6 to d9fe774 Compare April 14, 2025 20:02

vercel bot deployed to Preview April 14, 2025 20:08 View deployment

edgao changed the title ~~Edgao/bigquery call old code~~ Edgao/bigquery port interfaces Apr 14, 2025

vercel bot deployed to Preview April 14, 2025 23:31 View deployment

vercel bot deployed to Preview April 15, 2025 00:10 View deployment

vercel bot deployed to Preview April 15, 2025 16:38 View deployment

vercel bot deployed to Preview April 15, 2025 22:00 View deployment

edgao force-pushed the edgao/bigquery_call_old_code branch from bfef4d8 to e2e04af Compare April 16, 2025 23:06

vercel bot deployed to Preview April 16, 2025 23:13 View deployment

edgao force-pushed the destination_bigquery_bulk_cdk branch from 5e17a90 to 27e0aec Compare April 16, 2025 23:26

edgao force-pushed the edgao/bigquery_call_old_code branch from e2e04af to 47b9096 Compare April 16, 2025 23:26

vercel bot deployed to Preview April 16, 2025 23:33 View deployment

vercel bot deployed to Preview April 16, 2025 23:45 View deployment

edgao force-pushed the destination_bigquery_bulk_cdk branch from 27e0aec to a72b07f Compare April 16, 2025 23:54

edgao force-pushed the edgao/bigquery_call_old_code branch from 765cf5b to 29483a6 Compare April 16, 2025 23:56

vercel bot deployed to Preview April 17, 2025 00:02 View deployment

edgao force-pushed the destination_bigquery_bulk_cdk branch from a72b07f to ff01262 Compare April 17, 2025 15:13

edgao force-pushed the edgao/bigquery_call_old_code branch from 29483a6 to 73e6ae3 Compare April 17, 2025 16:43

edgao commented Apr 17, 2025

View reviewed changes

vercel bot deployed to Preview April 17, 2025 16:56 View deployment

vercel bot deployed to Preview April 17, 2025 18:48 View deployment

vercel bot deployed to Preview April 17, 2025 19:07 View deployment

frifriSF59 approved these changes Apr 23, 2025

View reviewed changes

frifriSF59 reviewed Apr 23, 2025

View reviewed changes

...e/integrations/destination/bigquery/typing_deduping/BigqueryDatabaseInitialStatusGatherer.kt Outdated Show resolved Hide resolved

edgao added 8 commits April 24, 2025 09:23

simplify raw table name gen

7477f85

update thread count

3498570

trimIndent

22078b9

simplify impl

522bd19

use kotlin stuff

c6c1f3a

better string munge

b1bacbf

use QueryJobConfiguration.of

1db68c5

merge filters

2f8e95e

edgao marked this pull request as ready for review April 24, 2025 16:38

edgao requested review from a team as code owners April 24, 2025 16:38

edgao merged commit dfbdfb6 into destination_bigquery_bulk_cdk Apr 24, 2025
14 of 18 checks passed

edgao deleted the edgao/bigquery_call_old_code branch April 24, 2025 16:38

vercel bot deployed to Preview April 24, 2025 16:42 View deployment

edgao added a commit that referenced this pull request Apr 25, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

27c381e

edgao added a commit that referenced this pull request Apr 28, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

b63a44f

edgao added a commit that referenced this pull request May 1, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

6b0d509

edgao added a commit that referenced this pull request May 1, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

6be885d

edgao added a commit that referenced this pull request May 2, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

9664e25

edgao added a commit that referenced this pull request May 6, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

ec485f1

frifriSF59 pushed a commit that referenced this pull request May 6, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

fa51aae

edgao added a commit that referenced this pull request May 6, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

c2895ad

edgao added a commit that referenced this pull request May 6, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

fc93f0e

edgao added a commit that referenced this pull request May 7, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

3618f89

edgao added a commit that referenced this pull request May 7, 2025

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc (#57578)

ab8a791

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc #57578

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc #57578

edgao commented Apr 11, 2025 •

edited

Loading

vercel bot commented Apr 11, 2025 •

edited

Loading

edgao Apr 17, 2025

edgao Apr 17, 2025

frifriSF59 Apr 21, 2025

johnny-schmidt Apr 22, 2025

johnny-schmidt Apr 22, 2025

johnny-schmidt Apr 22, 2025

edgao Apr 23, 2025

edgao Apr 23, 2025

edgao Apr 17, 2025

frifriSF59 left a comment

frifriSF59 Apr 23, 2025

edgao Apr 24, 2025

frifriSF59 Apr 23, 2025

edgao Apr 24, 2025

frifriSF59 Apr 23, 2025


		package io.airbyte.cdk.load.orchestration

		interface DestinationHandler {


		import io.airbyte.cdk.load.orchestration.DestinationInitialStatus
		import java.time.Instant

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc #57578

Bigquery / Bulk Load CDK: implement T+D, truncate refresh, etc #57578

Conversation

edgao commented Apr 11, 2025 • edited Loading

vercel bot commented Apr 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frifriSF59 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edgao commented Apr 11, 2025 •

edited

Loading

vercel bot commented Apr 11, 2025 •

edited

Loading