Skip to content

Commit 2a1a28a

Browse files
varant-zlaiezvz
andauthored
[Spark] disable checks in join monolith (#694)
## Summary Disabling analyzer checks ## Checklist - [ ] Added Unit Tests - [x] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Refactor** - Disabled the join configuration validation step before starting join jobs. - Updated time range calculation logic for certain join scenarios to improve consistency. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: ezvz <[email protected]>
1 parent 06bdf2f commit 2a1a28a

File tree

2 files changed

+5
-4
lines changed

2 files changed

+5
-4
lines changed

spark/src/main/scala/ai/chronon/spark/JoinBase.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -243,9 +243,9 @@ abstract class JoinBase(val joinConfCloned: api.Join,
243243
}
244244

245245
// Run validations before starting the job
246-
val analyzer = new Analyzer(tableUtils, joinConfCloned, endPartition, endPartition, silenceMode = true)
246+
// val analyzer = new Analyzer(tableUtils, joinConfCloned, endPartition, endPartition, silenceMode = true)
247247
try {
248-
analyzer.analyzeJoin(joinConfCloned, validationAssert = true)
248+
// analyzer.analyzeJoin(joinConfCloned, validationAssert = true)
249249
metrics.gauge(Metrics.Name.validationSuccess, 1)
250250
logger.info("Join conf validation succeeded. No error found.")
251251
} catch {

spark/src/main/scala/ai/chronon/spark/JoinUtils.scala

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -580,8 +580,9 @@ object JoinUtils {
580580
// events | entities | temporal => right part tables are aligned - so scan by leftRange
581581
// entities | entities | snapshot => right part tables are aligned - so scan by leftRange
582582
val rightRange = if (leftDataModel == EVENTS && joinPart.groupBy.inferredAccuracy == Accuracy.SNAPSHOT) {
583-
val leftTimeRange = leftTimeRangeOpt.getOrElse(leftDf.get.timeRange.toPartitionRange)
584-
leftTimeRange.shift(shiftDays)
583+
// Diabling for now
584+
// val leftTimeRange = leftTimeRangeOpt.getOrElse(leftDf.get.timeRange.toPartitionRange)
585+
leftRange.shift(shiftDays)
585586
} else {
586587
leftRange
587588
}

0 commit comments

Comments
 (0)