Lineage DAG (outdated) #234
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
## Summary ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Style** - Streamlined command construction for better readability. - Improved code formatting and readability across multiple functions. - Enhanced dictionary and string formatting. - Consistent error message presentation. - **Bug Fixes** - Updated conditional logic for downloading the dataproc jar to ensure it only occurs in the appropriate mode. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary Migrated `service_commons` module to Bazel from sbt. ## Checklist - [ ] Added Unit Tests - [X] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes - **New Features** - Enhanced build configuration to support Java 17 - Added new JVM rulesets for improved testing and development - **Dependencies** - Updated Netty to version 4.1.111.Final - Added JUnit 5 testing components - Integrated Vert.x 4.5.10 libraries - Added logback and configuration libraries - **Build Improvements** - Refined Bazel build rules for Java libraries and test suites - Streamlined dependency management <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: nikhil-zlai <[email protected]> Co-authored-by: Kumar Teja Chippala <[email protected]>
## Summary ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update Tested: ``` (dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/check_for_java11) $ asdf local java corretto-17.0.13.11.1 (dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/check_for_java11) $ asdf reshim java corretto-17.0.13.11.1 j(dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/check_for_java11) $ javac --version javac 17.0.13 (dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/check_for_java11) $ python api/py/ai/chronon/repo/run.py --mode upload --conf production/group_bys/quickstart/purchases.v1 --ds 2023-12-01 --dataproc (dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/check_for_java11) $ (dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/check_for_java11) $ bash distribution/build_and_upload_gcp_artifacts.sh canary Working in /Users/davidhan/zipline/chronon Building wheel Failed major version of 17. Expecting java version of at least 11. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added commands for generating Python code from Thrift files in the build script. - Introduced a clean command to prepare the build environment before assembly tasks. - Added validation for the existence of the Zipline wheel file after building. - **Chores** - Updated build configuration to target Java 11 compiler settings. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary ^^^ ### Testing This testing assumes the gbu job (this one is just the job that generates the data to bigquery) has run. Tested with this command: ``` (dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (main) $ python api/py/ai/chronon/repo/run.py --mode upload-to-kv --conf production/group_bys/quickstart/purchases.v1 --partition-string=2023-11-30 --dataproc --repo=/Users/davidhan/zipline/chronon/api/py/test/sample ``` and led to successful run of this job: https://console.cloud.google.com/dataproc/jobs/2f6b0b81-7b34-4a92-840d-cb90059f3d42/monitoring?region=us-central1&project=canary-443022 ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added support for a new upload mode to Google Cloud Platform. - Enhanced configuration handling for GCP Dataproc clusters. - Introduced new GCP-related options in the command-line interface. - Updated JSON configuration with new GCP parameters. - **Bug Fixes** - Improved error handling and argument processing in various components. - **Refactor** - Updated environment variable naming conventions. - Restructured configuration management across multiple files. - Enhanced clarity and organization in code structure. - **Chores** - Added support for multiple Java versions. - Updated build and deployment scripts for improved reliability. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
… repartitioning (#238) ## Summary ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Performance Optimization** - Updated DataFrame coalescing logic to improve resource utilization and performance. - Introduced more flexible configuration for DataFrame partition management. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…#244) ## Summary - Removed directory nesting for components (except shadcn-svelte components in `ui`) - Rework page layout structure and data loading strategy - Removes unnecessary data loading (ex. data distributions) - Improves performance by no longer rendering other/offscreen tab content - Provides better page component organization (and smaller files) via use of layouts (including nested) - Data loading can be handled more granularly and consistently via `load()` function. This also enables browser caching via `setHeaders({ 'cache-control': ... })` ([example](https://github.com/techniq/strava-analysis/blob/main/src/routes/analysis/%2Blayout.server.ts#L73C3-L75C6)) - Persist tab state for drift/distributions as route params ## Checklist - [ ] Added Unit Tests - [x] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **Refactor** - Simplified import paths across multiple frontend components - Removed redundant subdirectories in component imports - **New Features** - Added new layout for joins detail page with tab navigation - Introduced new observability navigation tabs - Created new ModelTable component for displaying model details - Added new distributions page for feature distributions - Implemented zoom functionality in distribution charts - Added placeholder content for lineage page - **Changes** - Updated page routing for joins detail pages - Modified page load and redirect behaviors - Restructured observability and job tracker views - Added loading state handling in distribution components - Implemented redirects for specific observability routes <!-- end of auto-generated comment: release notes by coderabbit.ai --> <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":"main"} ``` --> --------- Co-authored-by: Sean Lynch <[email protected]>
## Summary
## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update
<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"main","parentHead":"","trunk":"main"}
```
-->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Removed unused `fade` transition import from Svelte in distribution
and drift observability pages
- No functional changes to the components
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Sean Lynch <[email protected]>
## Summary ^^^ ## Checklist - [ ] Added Unit Tests - [x] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a new Async KV Store writer test to the workflow - Enhanced test suite with tagging capabilities - **Chores** - Updated project dependencies in build configuration - Refined GitHub Actions workflow configuration <!-- end of auto-generated comment: release notes by coderabbit.ai -->
sean-zlai
force-pushed
the
sean/lineage-dag2
branch
from
97f40c5 to
f8d1bac
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Checklist