Logging fix -- make root cause more clear if label job has misaligned dates

[varant-zlai]

Summary

Improve logging when labelDs - Window does not overlap with join data

Checklist

• Added Unit Tests
• Covered by existing CI
• Integration tested
• Documentation update

Summary by CodeRabbit

• New Features

  • Enhanced flexibility in date handling for label join operations, allowing for an optional start date override.
  • Introduction of new JSON configuration files for the quickstart.purchases.v1 dataset, facilitating structured data processing and analysis.

• Bug Fixes

  • Improved diagnostic messaging to clearly indicate when required data elements are missing during processing.
  • Enhanced error detection to ensure necessary computation windows are present before proceeding.

[coderabbitai]

Walkthrough

The changes update the LabelJoinV2 class to utilize a labelDateRange parameter instead of a string, enhancing how partitions are defined. The compute method now iterates over days in this range, invoking a new computeDay method for label computation. Logging has been improved for clarity, and error handling now ensures at least one computable window exists, throwing detailed errors when none are found.

Changes

File(s)	Change Summary
spark/.../LabelJoinV2.scala	Updated constructor to accept labelDateRange; modified compute method to call computeDay for each day; enhanced logging and error handling.
spark/.../Driver.scala	Introduced logic for handling startPartitionOverride and updated LabelJoinV2 constructor to accept a DateRange.
spark/.../LabelJoinV2Test.scala	Updated test cases to create DateRange objects for LabelJoinV2 instances instead of single dates.
api/python/.../parse_teams.py	Added functionality to handle Join objects in update_metadata; defined a nested function for metadata updates.
spark/.../BUILD.bazel	Modified streaming_test suite configuration; added new submission_test suite with specific attributes.
spark/.../JobSubmitter.scala	Changed logic for retrieving executionInfo from metadata; improved handling of mode configurations.
spark/.../JobSubmitterTest.scala	Introduced new test class for JobSubmitter with multiple test cases validating its methods.
spark/src/test/resources/group_bys/team/...	Added new JSON configuration files for quickstart.purchases.v1 dataset with metadata and aggregation specs.

Possibly related PRs

• Set the outputNamespace for join parts and label parts #612: Enhancements to label join functionality and metadata handling related to LabelJoinV2.
• chore: move modular architecture pieces into dedicated module #590: Updates in the Driver class for handling DateRange, directly linked to LabelJoinV2 changes.
• feat: support providing additional confs as yaml file for Driver.scala #164: Modifications in Driver.scala related to date range handling and LabelJoinV2 constructor updates.

Suggested reviewers

• nikhil-zlai
• piyush-zlai

Poem

In code's embrace, the dates now align,
With logs that illuminate, our joins brightly shine.
Errors now speak, with messages clear,
As partitions dance, bringing data near.
Cheers to the changes, let the joins flow free!
🚀✨

Warning

Review ran into problems

🔥 Problems

GitHub Actions and Pipeline Checks: Resource not accessible by integration - https://docs.github.com/rest/actions/workflow-runs#list-workflow-runs-for-a-repository.

Please grant the required permissions to the CodeRabbit GitHub App under the organization or repository settings.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro (Legacy)

📥 Commits

Reviewing files that changed from the base of the PR and between 21ad549 and 366dc22.

📒 Files selected for processing (10)

• api/python/ai/chronon/cli/compile/parse_configs.py (0 hunks)
• api/python/ai/chronon/cli/compile/parse_teams.py (2 hunks)
• spark/BUILD.bazel (1 hunks)
• spark/src/main/scala/ai/chronon/spark/Driver.scala (4 hunks)
• spark/src/main/scala/ai/chronon/spark/batch/LabelJoinV2.scala (7 hunks)
• spark/src/main/scala/ai/chronon/spark/submission/JobSubmitter.scala (1 hunks)
• spark/src/test/resources/group_bys/team/purchases.v1 (1 hunks)
• spark/src/test/resources/group_bys/team/purchases_only_conf_common.v1 (1 hunks)
• spark/src/test/scala/ai/chronon/spark/test/batch/LabelJoinV2Test.scala (3 hunks)
• spark/src/test/scala/ai/chronon/spark/test/submission/JobSubmitterTest.scala (1 hunks)

💤 Files with no reviewable changes (1)

• api/python/ai/chronon/cli/compile/parse_configs.py

✅ Files skipped from review due to trivial changes (1)

• spark/src/test/resources/group_bys/team/purchases_only_conf_common.v1

🚧 Files skipped from review as they are similar to previous changes (2)

• spark/src/main/scala/ai/chronon/spark/Driver.scala
• spark/src/test/scala/ai/chronon/spark/test/batch/LabelJoinV2Test.scala

🧰 Additional context used

🧬 Code Graph Analysis (2)

spark/src/main/scala/ai/chronon/spark/batch/LabelJoinV2.scala (4)

api/src/main/scala/ai/chronon/api/Extensions.scala (1)

• toPartitionRange (1256-1260)

spark/src/main/scala/ai/chronon/spark/LabelJoin.scala (1)

• compute (122-147)

api/src/main/scala/ai/chronon/api/DataRange.scala (1)

• steps (82-87)

spark/src/main/scala/ai/chronon/spark/TableUtils.scala (1)

• scanDf (593-613)

api/python/ai/chronon/cli/compile/parse_teams.py (2)

api/src/main/scala/ai/chronon/api/ParametricMacro.scala (1)

• api (66-66)

api/src/main/scala/ai/chronon/api/Builders.scala (1)

• MetaData (261-315)

⏰ Context from checks skipped due to timeout of 90000ms (18)

• GitHub Check: streaming_tests
• GitHub Check: streaming_tests
• GitHub Check: join_tests
• GitHub Check: groupby_tests
• GitHub Check: analyzer_tests
• GitHub Check: analyzer_tests
• GitHub Check: spark_tests
• GitHub Check: spark_tests
• GitHub Check: groupby_tests
• GitHub Check: join_tests
• GitHub Check: fetcher_tests
• GitHub Check: fetcher_tests
• GitHub Check: scala_compile_fmt_fix
• GitHub Check: batch_tests
• GitHub Check: batch_tests
• GitHub Check: non_spark_tests
• GitHub Check: non_spark_tests
• GitHub Check: enforce_triggered_workflows

🔇 Additional comments (21)

spark/src/main/scala/ai/chronon/spark/batch/LabelJoinV2.scala (12)

24-24: API change to use DateRange instead of String

Constructor now accepts api.DateRange instead of String, improving type safety.

---

36-36: Proper conversion from DateRange to PartitionRange

Correctly uses the toPartitionRange extension method.

---

102-108: Improved handling of date ranges

Splits computation across multiple days using steps(1) and unions results.

---

110-110: Good refactoring to support day-by-day processing

Extracted core logic to computeDay method for reuse.

---

131-144: Better error logging

Created reusable baseLogString for consistent error reporting.

---

145-154: More informative error handling

Clearer requirement check for computable windows with detailed error message.

---

157-163: Improved warning message

Reuses baseLogString for consistent logging structure.

---

174-179: Updated method call with correct parameters

Passes labelDsAsPartitionRange to support multi-day processing.

---

199-200: Updated method signature

Added parameters needed for multi-day processing.

---

202-202: More informative logging

References labelDsAsPartitionRange.start for date context.

---

209-209: Proper date range shifting

Correctly shifts the label date range by window length.

---

237-237: Consistent range usage

Uses labelDsAsPartitionRange for snapshot data fetching.

api/python/ai/chronon/cli/compile/parse_teams.py (3)

14-14: Added necessary imports

Imported Join and MetaData types for new functionality.

---

91-101: Added Join metadata handling

New helper function properly sets outputNamespace for join parts.

---

102-109: Applied metadata updates to all join parts

Ensures consistent outputNamespace across join and label parts.

spark/src/main/scala/ai/chronon/spark/submission/JobSubmitter.scala (2)

46-55: Improved metadata handling

Extracts metadata first, then executionInfo for better structure.

---

56-73: More robust null checking

Added explicit null handling for executionInfo and better mode config extraction.

spark/src/test/resources/group_bys/team/purchases.v1 (1)

1-148: New test configuration

Added complete test resource for GroupBy operation with necessary metadata.

spark/src/test/scala/ai/chronon/spark/test/submission/JobSubmitterTest.scala (1)

1-68: Well-structured test class

Tests are comprehensive and follow testing best practices.

spark/BUILD.bazel (2)

209-214: Path correction for resources

Fixed resource path reference.

---

216-223: Good addition of submission_test suite

New test suite properly configured to match other test suites.

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai plan to trigger planning for file edits and PR creation.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.