Skip to content

Fix missing slfj4 implementation causing error logs at dataproc job startup #634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 17, 2025

Conversation

david-zlai
Copy link
Contributor

@david-zlai david-zlai commented Apr 11, 2025

Summary

Fix for #549 (comment)

Tested on integration tests and I don't see the error logs anymore:

SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5db6b9cd]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features.
Array(group-by-backfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance)
Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9
Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022

                <-----------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------
                                                  DATAPROC LOGS
                ------------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------>
                
Waiting for job output...
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse
25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml
25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.

used to show up like this:

ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specif

ex: https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292

Checklist

  • Added Unit Tests
  • Covered by existing CI
  • Integration tested
  • Documentation update

Summary by CodeRabbit

  • Chores
    • Improved dependency management for logging integrations by adjusting library configurations.
    • Enhanced consistency across the build process, contributing to a more stable and maintainable system.

These behind-the-scenes improvements provide a more robust foundation for future updates, ensuring reliable and consistent behavior without altering the end-user experience.

Copy link
Contributor

coderabbitai bot commented Apr 11, 2025

Walkthrough

This update in the api/BUILD.bazel file modifies the deps list of the scala_library definition for lib. It removes the direct inclusion of the Maven artifact org.slf4j:slf4j-api and adds org.apache.logging.log4j:log4j-slf4j2-impl. The org.slf4j:slf4j-api artifact is reintroduced later in the deps list, indicating a change in the order of dependencies rather than a complete removal. Additionally, a commented line about excluding some slf4j-impl dependencies is removed from the java_binary rule in tools/build_rules/cloud_gcp/BUILD.

Changes

File Change Summary
api/BUILD.bazel Modified scala_library (lib) deps: removed direct org.slf4j:slf4j-api, added org.apache.logging.log4j:log4j-slf4j2-impl, reintroduced org.slf4j:slf4j-api later.
tools/build_rules/cloud_gcp/BUILD Removed a commented line about excluding slf4j-impl dependencies in java_binary (cloud_gcp).

Possibly related PRs

Suggested reviewers

  • piyush-zlai

Poem

In the Bazel realm, changes unfold,
Dependencies shift, a story retold.
Log4j now shines, while SLF4J stays,
In the library's heart, they dance and play.
A build refined, in code we behold!

Warning

Review ran into problems

🔥 Problems

GitHub Actions and Pipeline Checks: Resource not accessible by integration - https://docs.github.com/rest/actions/workflow-runs#list-workflow-runs-for-a-repository.

Please grant the required permissions to the CodeRabbit GitHub App under the organization or repository settings.


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro (Legacy)

📥 Commits

Reviewing files that changed from the base of the PR and between e2cb0a3 and f1d4222.

📒 Files selected for processing (1)
  • tools/build_rules/cloud_gcp/BUILD (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • tools/build_rules/cloud_gcp/BUILD
⏰ Context from checks skipped due to timeout of 90000ms (20)
  • GitHub Check: service_tests
  • GitHub Check: cloud_gcp_tests
  • GitHub Check: online_tests
  • GitHub Check: cloud_aws_tests
  • GitHub Check: service_tests
  • GitHub Check: service_commons_tests
  • GitHub Check: online_tests
  • GitHub Check: cloud_gcp_tests
  • GitHub Check: hub_tests
  • GitHub Check: cloud_aws_tests
  • GitHub Check: hub_tests
  • GitHub Check: aggregator_tests
  • GitHub Check: flink_tests
  • GitHub Check: api_tests
  • GitHub Check: aggregator_tests
  • GitHub Check: flink_tests
  • GitHub Check: api_tests
  • GitHub Check: orchestration_tests
  • GitHub Check: orchestration_tests
  • GitHub Check: enforce_triggered_workflows

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@david-zlai david-zlai requested a review from tchow-zlai April 11, 2025 22:08
api/BUILD.bazel Outdated
deps = [
maven_artifact("javax.annotation:javax.annotation.api"),
maven_artifact("org.slf4j:slf4j-api"),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was actually thinking - this lib probably doesn't need the log4j stuff does it? Shouldn't it just be the scala lib below?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

basically, move these lgoger deps directly to the lib target below.

tchow-zlai and others added 7 commits April 14, 2025 09:54
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
tools/build_rules/cloud_gcp/BUILD (1)

12-12: Typo in comment.

"pull g some" appears to be a typo. Should be "pull out some" or similar.

-        # pull g some slf4j-impl dependencies - these can be included at the application deploy target level if needed
+        # pull out some slf4j-impl dependencies - these can be included at the application deploy target level if needed
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro (Legacy)

📥 Commits

Reviewing files that changed from the base of the PR and between f6d3e11 and e2cb0a3.

📒 Files selected for processing (1)
  • tools/build_rules/cloud_gcp/BUILD (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (3)
  • GitHub Check: non_spark_tests
  • GitHub Check: non_spark_tests
  • GitHub Check: enforce_triggered_workflows
🔇 Additional comments (1)
tools/build_rules/cloud_gcp/BUILD (1)

12-13: Fixed SLF4J implementation issue.

Adding log4j-slf4j2-impl resolves the multiple SLF4J providers error during Dataproc job startup.

@david-zlai david-zlai requested a review from tchow-zlai April 17, 2025 14:13
@david-zlai david-zlai merged commit 9be2b09 into main Apr 17, 2025
24 checks passed
@david-zlai david-zlai deleted the davidhan/fix_slfj4_missing branch April 17, 2025 19:44
@coderabbitai coderabbitai bot mentioned this pull request Apr 18, 2025
4 tasks
kumar-zlai pushed a commit that referenced this pull request Apr 25, 2025
…tartup (#634)

## Summary

Fix for
#549 (comment)

Tested on integration tests and I don't see the error logs anymore:

```
SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5db6b9cd]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features.
Array(group-by-backfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance)
Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9
Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022

                <-----------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------
                                                  DATAPROC LOGS
                ------------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------>
                
Waiting for job output...
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse
25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml
25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.

```

used to show up like this:
```
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specif
```
ex:
https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292


## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [x] Integration tested
- [ ] Documentation update



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Chores**
- Improved dependency management for logging integrations by adjusting
library configurations.
- Enhanced consistency across the build process, contributing to a more
stable and maintainable system.

These behind-the-scenes improvements provide a more robust foundation
for future updates, ensuring reliable and consistent behavior without
altering the end-user experience.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"main","parentHead":"","trunk":"main"}
```
-->

---------

Co-authored-by: tchow-zlai <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
kumar-zlai pushed a commit that referenced this pull request Apr 29, 2025
…tartup (#634)

## Summary

Fix for
#549 (comment)

Tested on integration tests and I don't see the error logs anymore:

```
SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5db6b9cd]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features.
Array(group-by-backfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance)
Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9
Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022

                <-----------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------
                                                  DATAPROC LOGS
                ------------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------>
                
Waiting for job output...
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse
25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml
25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.

```

used to show up like this:
```
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specif
```
ex:
https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292


## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [x] Integration tested
- [ ] Documentation update



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Chores**
- Improved dependency management for logging integrations by adjusting
library configurations.
- Enhanced consistency across the build process, contributing to a more
stable and maintainable system.

These behind-the-scenes improvements provide a more robust foundation
for future updates, ensuring reliable and consistent behavior without
altering the end-user experience.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"main","parentHead":"","trunk":"main"}
```
-->

---------

Co-authored-by: tchow-zlai <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
chewy-zlai pushed a commit that referenced this pull request May 15, 2025
…tartup (#634)

## Summary

Fix for
#549 (comment)

Tested on integration tests and I don't see the error logs anymore:

```
SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5db6b9cd]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features.
Array(group-by-backfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance)
Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9
Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022

                <-----------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------
                                                  DATAPROC LOGS
                ------------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------>
                
Waiting for job output...
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse
25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml
25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.

```

used to show up like this:
```
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specif
```
ex:
https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292


## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [x] Integration tested
- [ ] Documentation update



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Chores**
- Improved dependency management for logging integrations by adjusting
library configurations.
- Enhanced consistency across the build process, contributing to a more
stable and maintainable system.

These behind-the-scenes improvements provide a more robust foundation
for future updates, ensuring reliable and consistent behavior without
altering the end-user experience.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"main","parentHead":"","trunk":"main"}
```
-->

---------

Co-authored-by: tchow-zlai <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
chewy-zlai pushed a commit that referenced this pull request May 15, 2025
…tartup (#634)

## Summary

Fix for
#549 (comment)

Tested on integration tests and I don't see the error logs anymore:

```
SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5db6b9cd]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features.
Array(group-by-backfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance)
Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9
Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022

                <-----------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------
                                                  DATAPROC LOGS
                ------------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------>
                
Waiting for job output...
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse
25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml
25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.

```

used to show up like this:
```
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specif
```
ex:
https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292


## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [x] Integration tested
- [ ] Documentation update



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Chores**
- Improved dependency management for logging integrations by adjusting
library configurations.
- Enhanced consistency across the build process, contributing to a more
stable and maintainable system.

These behind-the-scenes improvements provide a more robust foundation
for future updates, ensuring reliable and consistent behavior without
altering the end-user experience.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"main","parentHead":"","trunk":"main"}
```
-->

---------

Co-authored-by: tchow-zlai <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
chewy-zlai pushed a commit that referenced this pull request May 16, 2025
…tartup (#634)

## Summary

Fix for
#549 (comment)

Tested on integration tests and I don't see the error logs anymore:

```
SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
SLF4J(W): Found provider [ch.qos.logbaour clients.classic.spi.Logbaour clientsServiceProvider@5db6b9cd]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@6f43c82]
WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features.
Array(group-by-baour clientsfill, --conf-path=purchases.v1_dev, --end-date=2025-04-11, --conf-type=group_bys, --is-gcp, --gcp-project-id=canary-443022, --gcp-bigtable-instance-id=zipline-canary-instance)
Dataproc submitter job id: 673a141c-f205-44a0-b6ef-4050263d9fb9
Safe to exit. Follow the job status at: https://console.cloud.google.com/dataproc/jobs/673a141c-f205-44a0-b6ef-4050263d9fb9/configuration?region=us-central1&project=canary-443022

                <-----------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------
                                                  DATAPROC LOGS
                ------------------------------------------------------------------------------------
                ------------------------------------------------------------------------------------>
                
Waiting for job output...
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
25/04/11 22:07:43 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.
Using warehouse dir: /tmp/673a141c-f205-44a0-b6ef-4050263d9fb9/local_warehouse
25/04/11 22:07:45 INFO HiveConf: Found configuration file file:/etc/hive/conf.dist/hive-site.xml
25/04/11 22:07:45 WARN SparkConf: The configuration key 'spark.yarn.executor.failuresValidityInterval' has been deprecated as of Spark 3.5 and may be removed in the future. Please use the new key 'spark.executor.failuresValidityInterval' instead.

```

used to show up like this:
```
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specif
```
ex:
https://github.com/zipline-ai/chronon/actions/runs/14407847807/job/40409703201#step:5:292


## Cheour clientslist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [x] Integration tested
- [ ] Documentation update



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Chores**
- Improved dependency management for logging integrations by adjusting
library configurations.
- Enhanced consistency across the build process, contributing to a more
stable and maintainable system.

These behind-the-scenes improvements provide a more robust foundation
for future updates, ensuring reliable and consistent behavior without
altering the end-user experience.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to traour clients
the status of staour clientss when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"main","parentHead":"","trunk":"main"}
```
-->

---------

Co-authored-by: tchow-zlai <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants