Using tableUtils.partitionColumn rather than hardcoded ds (#579)

varant-zlai · ezvz · web-flow · commit f3acb8e15caf · 2025-04-02T00:53:06.000-07:00
## Summary
Using tableUtils.partitionColumn rather than hardcoded ds

## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update

Co-authored-by: ezvz &lt;vzanoyan@gmail.com&gt;
diff --git a/spark/src/main/scala/ai/chronon/spark/MergeJob.scala b/spark/src/main/scala/ai/chronon/spark/MergeJob.scala
@@ -97,7 +97,7 @@ class MergeJob(node: JoinMergeNode, range: DateRange, joinParts: Seq[JoinPart])(
         } else {
           dayStep
         }
-      val wheres = effectiveRange.whereClauses("ds")
+      val wheres = effectiveRange.whereClauses(tableUtils.partitionColumn)
       val sql = QueryUtils.build(null, partTable, wheres)
       logger.info(s"Pulling data from joinPart table with: $sql")
       (joinPart, tableUtils.scanDfBase(null, partTable, List.empty, wheres, None))

Original file line number	Diff line number	Diff line change
`@@ -97,7 +97,7 @@ class MergeJob(node: JoinMergeNode, range: DateRange, joinParts: Seq[JoinPart])(`
`97`	`97`	`} else {`
`98`	`98`	`dayStep`
`99`	`99`	`}`
`100`		`- val wheres = effectiveRange.whereClauses("ds")`
	`100`	`+ val wheres = effectiveRange.whereClauses(tableUtils.partitionColumn)`
`101`	`101`	`val sql = QueryUtils.build(null, partTable, wheres)`
`102`	`102`	`logger.info(s"Pulling data from joinPart table with: $sql")`
`103`	`103`	`(joinPart, tableUtils.scanDfBase(null, partTable, List.empty, wheres, None))`