-
Notifications
You must be signed in to change notification settings - Fork 0
Fix missing slfj4 implementation causing error logs at dataproc job startup #634
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
WalkthroughThis update in the Changes
Possibly related PRs
Suggested reviewers
Poem
Warning Review ran into problems🔥 ProblemsGitHub Actions and Pipeline Checks: Resource not accessible by integration - https://docs.github.com/rest/actions/workflow-runs#list-workflow-runs-for-a-repository. Please grant the required permissions to the CodeRabbit GitHub App under the organization or repository settings. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
✅ Files skipped from review due to trivial changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (20)
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
api/BUILD.bazel
Outdated
deps = [ | ||
maven_artifact("javax.annotation:javax.annotation.api"), | ||
maven_artifact("org.slf4j:slf4j-api"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was actually thinking - this lib probably doesn't need the log4j stuff does it? Shouldn't it just be the scala lib below?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
basically, move these lgoger deps directly to the lib
target below.
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tools/build_rules/cloud_gcp/BUILD (1)
12-12
: Typo in comment."pull g some" appears to be a typo. Should be "pull out some" or similar.
- # pull g some slf4j-impl dependencies - these can be included at the application deploy target level if needed + # pull out some slf4j-impl dependencies - these can be included at the application deploy target level if needed
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro (Legacy)
📒 Files selected for processing (1)
tools/build_rules/cloud_gcp/BUILD
(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: non_spark_tests
- GitHub Check: non_spark_tests
- GitHub Check: enforce_triggered_workflows
🔇 Additional comments (1)
tools/build_rules/cloud_gcp/BUILD (1)
12-13
: Fixed SLF4J implementation issue.Adding log4j-slf4j2-impl resolves the multiple SLF4J providers error during Dataproc job startup.
…tartup (#634) ## Summary Fix for #549 (comment) Tested on integration tests and I don't see the error logs anymore: ``` SLF4J(W): Class path contains multiple SLF4J providers. SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5db6b9cd] SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features. Array(group-by-backfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance) Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9 Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022 <----------------------------------------------------------------------------------- ------------------------------------------------------------------------------------ DATAPROC LOGS ------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------> Waiting for job output... 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse 25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml 25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. ``` used to show up like this: ``` ERROR StatusLogger Unrecognized format specifier [thread] ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [level] ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [logger] ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [msg] ERROR StatusLogger Unrecognized conversion specif ``` ex: https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292 ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Chores** - Improved dependency management for logging integrations by adjusting library configurations. - Enhanced consistency across the build process, contributing to a more stable and maintainable system. These behind-the-scenes improvements provide a more robust foundation for future updates, ensuring reliable and consistent behavior without altering the end-user experience. <!-- end of auto-generated comment: release notes by coderabbit.ai --> <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":"main"} ``` --> --------- Co-authored-by: tchow-zlai <[email protected]> Co-authored-by: Thomas Chow <[email protected]>
…tartup (#634) ## Summary Fix for #549 (comment) Tested on integration tests and I don't see the error logs anymore: ``` SLF4J(W): Class path contains multiple SLF4J providers. SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5db6b9cd] SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features. Array(group-by-backfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance) Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9 Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022 <----------------------------------------------------------------------------------- ------------------------------------------------------------------------------------ DATAPROC LOGS ------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------> Waiting for job output... 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse 25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml 25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. ``` used to show up like this: ``` ERROR StatusLogger Unrecognized format specifier [thread] ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [level] ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [logger] ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [msg] ERROR StatusLogger Unrecognized conversion specif ``` ex: https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292 ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Chores** - Improved dependency management for logging integrations by adjusting library configurations. - Enhanced consistency across the build process, contributing to a more stable and maintainable system. These behind-the-scenes improvements provide a more robust foundation for future updates, ensuring reliable and consistent behavior without altering the end-user experience. <!-- end of auto-generated comment: release notes by coderabbit.ai --> <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":"main"} ``` --> --------- Co-authored-by: tchow-zlai <[email protected]> Co-authored-by: Thomas Chow <[email protected]>
…tartup (#634) ## Summary Fix for #549 (comment) Tested on integration tests and I don't see the error logs anymore: ``` SLF4J(W): Class path contains multiple SLF4J providers. SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5db6b9cd] SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features. Array(group-by-backfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance) Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9 Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022 <----------------------------------------------------------------------------------- ------------------------------------------------------------------------------------ DATAPROC LOGS ------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------> Waiting for job output... 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse 25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml 25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. ``` used to show up like this: ``` ERROR StatusLogger Unrecognized format specifier [thread] ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [level] ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [logger] ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [msg] ERROR StatusLogger Unrecognized conversion specif ``` ex: https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292 ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Chores** - Improved dependency management for logging integrations by adjusting library configurations. - Enhanced consistency across the build process, contributing to a more stable and maintainable system. These behind-the-scenes improvements provide a more robust foundation for future updates, ensuring reliable and consistent behavior without altering the end-user experience. <!-- end of auto-generated comment: release notes by coderabbit.ai --> <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":"main"} ``` --> --------- Co-authored-by: tchow-zlai <[email protected]> Co-authored-by: Thomas Chow <[email protected]>
…tartup (#634) ## Summary Fix for #549 (comment) Tested on integration tests and I don't see the error logs anymore: ``` SLF4J(W): Class path contains multiple SLF4J providers. SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5db6b9cd] SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features. Array(group-by-backfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance) Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9 Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022 <----------------------------------------------------------------------------------- ------------------------------------------------------------------------------------ DATAPROC LOGS ------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------> Waiting for job output... 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse 25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml 25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. ``` used to show up like this: ``` ERROR StatusLogger Unrecognized format specifier [thread] ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [level] ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [logger] ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [msg] ERROR StatusLogger Unrecognized conversion specif ``` ex: https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292 ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Chores** - Improved dependency management for logging integrations by adjusting library configurations. - Enhanced consistency across the build process, contributing to a more stable and maintainable system. These behind-the-scenes improvements provide a more robust foundation for future updates, ensuring reliable and consistent behavior without altering the end-user experience. <!-- end of auto-generated comment: release notes by coderabbit.ai --> <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":"main"} ``` --> --------- Co-authored-by: tchow-zlai <[email protected]> Co-authored-by: Thomas Chow <[email protected]>
…tartup (#634) ## Summary Fix for #549 (comment) Tested on integration tests and I don't see the error logs anymore: ``` SLF4J(W): Class path contains multiple SLF4J providers. SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] SLF4J(W): Found provider [ch.qos.logbaour clients.classic.spi.Logbaour clientsServiceProvider@5db6b9cd] SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features. Array(group-by-baour clientsfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance) Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9 Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022 <----------------------------------------------------------------------------------- ------------------------------------------------------------------------------------ DATAPROC LOGS ------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------> Waiting for job output... 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse 25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml 25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. ``` used to show up like this: ``` ERROR StatusLogger Unrecognized format specifier [thread] ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [level] ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [logger] ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [msg] ERROR StatusLogger Unrecognized conversion specif ``` ex: https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292 ## Cheour clientslist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Chores** - Improved dependency management for logging integrations by adjusting library configurations. - Enhanced consistency across the build process, contributing to a more stable and maintainable system. These behind-the-scenes improvements provide a more robust foundation for future updates, ensuring reliable and consistent behavior without altering the end-user experience. <!-- end of auto-generated comment: release notes by coderabbit.ai --> <!-- av pr metadata This information is embedded by the av CLI when creating PRs to traour clients the status of staour clientss when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":"main"} ``` --> --------- Co-authored-by: tchow-zlai <[email protected]> Co-authored-by: Thomas Chow <[email protected]>
Summary
Fix for #549 (comment)
Tested on integration tests and I don't see the error logs anymore:
used to show up like this:
ex: https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292
Checklist
Summary by CodeRabbit
These behind-the-scenes improvements provide a more robust foundation for future updates, ensuring reliable and consistent behavior without altering the end-user experience.