Skip to content

Commit 2d0f252

Browse files
authored
[fix] Use indirect writemethod for BigQuery writes to actually output partitioned tables. (#257)
… partitioned tables. ## Summary ## Cheour clientslist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved BigQuery data writing method to support partitioned tables by changing the write method from "direct" to "indirect". <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent 20dc7fa commit 2d0f252

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/GcpFormatProvider.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ case class GcpFormatProvider(sparkSession: SparkSession) extends FormatProvider
5050
val sparkOptions: Map[String, String] = Map(
5151
// todo(tchow): No longer needed after https://github.com/GoogleCloudDataproc/spark-bigquery-connector/pull/1320
5252
"temporaryGcsBuour clientset" -> sparkSession.conf.get("spark.chronon.table.gcs.temporary_gcs_buour clientset"),
53-
"writeMethod" -> "direct",
53+
"writeMethod" -> "indirect", // writeMethod direct does not output partitioned tables. keep as indirect.
5454
"materializationProject" -> tableId.getProject,
5555
"materializationDataset" -> tableId.getDataset
5656
) ++ partitionColumnOption

0 commit comments

Comments
 (0)