Skip to content

Commit 989308a

Browse files
david-zlaitchow-zlaithomaschow
authored
Fix missing slfj4 implementation causing error logs at dataproc job startup (#634)
## Summary Fix for #549 (comment) Tested on integration tests and I don't see the error logs anymore: ``` SLF4J(W): Class path contains multiple SLF4J providers. SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5db6b9cd] SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82] WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features. Array(group-by-backfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance) Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9 Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022 <----------------------------------------------------------------------------------- ------------------------------------------------------------------------------------ DATAPROC LOGS ------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------> Waiting for job output... 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. 25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse 25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml 25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead. ``` used to show up like this: ``` ERROR StatusLogger Unrecognized format specifier [thread] ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [level] ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [logger] ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern. ERROR StatusLogger Unrecognized format specifier [msg] ERROR StatusLogger Unrecognized conversion specif ``` ex: https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292 ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Chores** - Improved dependency management for logging integrations by adjusting library configurations. - Enhanced consistency across the build process, contributing to a more stable and maintainable system. These behind-the-scenes improvements provide a more robust foundation for future updates, ensuring reliable and consistent behavior without altering the end-user experience. <!-- end of auto-generated comment: release notes by coderabbit.ai --> <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":"main"} ``` --> --------- Co-authored-by: tchow-zlai <[email protected]> Co-authored-by: Thomas Chow <[email protected]>
1 parent 538d170 commit 989308a

File tree

2 files changed

+3
-2
lines changed

2 files changed

+3
-2
lines changed

api/BUILD.bazel

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,8 +33,9 @@ scala_library(
3333
"//tools/build_rules/spark:spark-exec",
3434
maven_artifact("com.fasterxml.jackson.core:jackson-core"),
3535
maven_artifact("com.fasterxml.jackson.core:jackson-databind"),
36-
maven_artifact("org.slf4j:slf4j-api"),
3736
maven_artifact("org.apache.commons:commons-lang3"),
37+
maven_artifact("org.apache.logging.log4j:log4j-slf4j2-impl"),
38+
maven_artifact("org.slf4j:slf4j-api"),
3839
maven_artifact("com.google.code.gson:gson"),
3940
maven_artifact_with_suffix("org.scala-lang.modules:scala-collection-compat"),
4041
maven_artifact_with_suffix("org.scala-lang.modules:scala-parser-combinators"),

tools/build_rules/cloud_gcp/BUILD

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,6 @@ java_binary(
99
# Remove commons text as without this exclusion, Flink's JM isn't able to load the execution graph as our repo version (1.11.0) is
1010
# higher than Flink's version (1.10.0) and this results in the Flink UI not loading
1111
maven_artifact("org.apache.commons:commons-text"),
12-
# pull out some slf4j-impl dependencies - these can be included at the application deploy target level if needed
1312
],
1413
)
14+

0 commit comments

Comments
 (0)