-
Notifications
You must be signed in to change notification settings - Fork 0
refactor tableutils into its own module catalog #679
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Co-authored-by: Thomas Chow <[email protected]>
WalkthroughThis change reorganizes the codebase by moving Changes
Sequence Diagram(s)sequenceDiagram
participant UserCode
participant CatalogLib
participant TableUtils
UserCode->>CatalogLib: import ai.chronon.spark.catalog.TableUtils
CatalogLib->>TableUtils: Provide table utility methods
UserCode->>TableUtils: Call utility methods as before
Possibly related PRs
Suggested reviewers
Poem
Warning Review ran into problems🔥 ProblemsGitHub Actions and Pipeline Checks: Resource not accessible by integration - https://docs.github.com/rest/actions/workflow-runs#list-workflow-runs-for-a-repository. Please grant the required permissions to the CodeRabbit GitHub App under the organization or repository settings. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
@@ -14,30 +14,28 @@ | |||
* limitations under the License. | |||
*/ | |||
|
|||
package ai.chronon.spark | |||
package ai.chronon.spark.catalog |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
adding a new catalog
module.
## Summary - Creates a `catalog` module to replace the `format` one we had. - Add a new bazel lib target for the `catalog` module - Move TableUtils there, it should live alongside the formatprovider stuff. - Rename imports - Followup is to remove the dependency on the main `spark` lib. This requires decoupling with `Extensions`. Then we can drastically slim the deps down in cloud gcp. ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":""} ``` --> Co-authored-by: Thomas Chow <[email protected]>
## Summary - Creates a `catalog` module to replace the `format` one we had. - Add a new bazel lib target for the `catalog` module - Move TableUtils there, it should live alongside the formatprovider stuff. - Rename imports - Followup is to remove the dependency on the main `spark` lib. This requires decoupling with `Extensions`. Then we can drastically slim the deps down in cloud gcp. ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":""} ``` --> Co-authored-by: Thomas Chow <[email protected]>
## Summary - Creates a `catalog` module to replace the `format` one we had. - Add a new bazel lib target for the `catalog` module - Move TableUtils there, it should live alongside the formatprovider stuff. - Rename imports - Followup is to remove the dependency on the main `spark` lib. This requires decoupling with `Extensions`. Then we can drastically slim the deps down in cloud gcp. ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":""} ``` --> Co-authored-by: Thomas Chow <[email protected]>
## Summary - Creates a `catalog` module to replace the `format` one we had. - Add a new bazel lib target for the `catalog` module - Move TableUtils there, it should live alongside the formatprovider stuff. - Rename imports - Followup is to remove the dependency on the main `spark` lib. This requires decoupling with `Extensions`. Then we can drastically slim the deps down in cloud gcp. ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":""} ``` --> Co-authored-by: Thomas Chow <[email protected]>
## Summary - Creates a `catalog` module to replace the `format` one we had. - Add a new bazel lib target for the `catalog` module - Move TableUtils there, it should live alongside the formatprovider stuff. - Rename imports - Followup is to remove the dependency on the main `spark` lib. This requires decoupling with `Extensions`. Then we can drastically slim the deps down in cloud gcp. ## Cheour clientslist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- av pr metadata This information is embedded by the av CLI when creating PRs to traour clients the status of staour clientss when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":""} ``` --> Co-authored-by: Thomas Chow <[email protected]>
Summary
catalog
module to replace theformat
one we had.catalog
modulespark
lib. This requires decoupling withExtensions
. Then we can drastically slim the deps down in cloud gcp.Checklist