Skip to content

Bazel builds enabled along with sbt #893

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 32 commits into from
Jan 23, 2025
Merged

Conversation

krisnaru
Copy link
Contributor

@krisnaru krisnaru commented Dec 31, 2024

Summary

  • Enabled bazel builds for sources, tests
  • Sbt works as is
  • Supported scala versions: 2.12 and 2.13
  • Supported spark versions 3.1, 3.2,3.5

Why / Goal

Goal is to improve the build environment(by sandboxing) and provide multiple spark versions

Test Plan

Unit testing green
quick start steps green https://chronon.ai/getting_started/Tutorial.html

Build commands

  • bazel build //service
  • bazel build //online
  • bazel build //api/...
  • Uber jar: bazel build //spark:spark-assembly_deploy.jar
  • Build for scala version and spark version
    bazel build --config scala_2.12 --define spark_version=3.2 //spark/...

Checklist

  • Readme updates
  • Publish maven artifacts

Reviewers

@nikhilsimha @piyush-zlai

Copy link
Collaborator

@nikhil-zlai nikhil-zlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! This is massively impactful Krish! Thanks for putting this up.

@krisnaru krisnaru changed the title Bazel builds enabled for all java scala sources Bazel builds enabled along with sbt Jan 2, 2025
Copy link
Collaborator

@piyush-zlai piyush-zlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking the time to make this PR. Swapping over to bazel would be great for chronon!
Couple of comments / qs:

  • Lets also add some notes on how devs can use build / add deps etc using bazel in the devnotes
  • While we're keeping sbt & bazel around in the transitionary period, is there anything we need to call out for folks to take care of while updating deps / such?

Copy link
Collaborator

@mears-stripe mears-stripe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!! Excited about the project being bazel-fied!

@krisnaru krisnaru force-pushed the bazel-phase1 branch 5 times, most recently from 983a914 to 7e8a4b2 Compare January 14, 2025 05:34
Copy link
Collaborator

@hzding621 hzding621 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Left a comment about maintaining bazel/SBT consistency as other comments also mentioned.

🙏 Thanks for contributing - this is huge!

@pengyu-hou
Copy link
Collaborator

we are getting really close but the last commit seems breaking the build. Could you reproduce it with the sbt build?

[error] IO error while decoding /Users/pengyu_hou/workspace/chronon/spark/src/main/spark-default/ai/chronon/spark/EncoderUtil.scala with UTF-8: /Users/pengyu_hou/workspace/chronon/spark/src/main/spark-default/ai/chronon/spark/EncoderUtil.scala (No such file or directory)

Copy link
Collaborator

@pengyu-hou pengyu-hou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the change and being patient with our review process. Looking forward to phase 2.

@pengyu-hou pengyu-hou merged commit 0a1751e into airbnb:main Jan 23, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants