Skip to content

Bazel migration for Flink module #237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Jan 21, 2025
Merged

Bazel migration for Flink module #237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
cc9ab7f
Build config for aggregator module
kumar-zlai Jan 9, 2025
c22461a
Minor fix to change naming convention for api module targets
kumar-zlai Jan 9, 2025
0d8c084
Initial partial commit with CatalystHelper renaming
kumar-zlai Jan 9, 2025
6d6dd6a
Initial partial commit with CatalystHelper renaming
kumar-zlai Jan 9, 2025
7160500
Minor fix to update the naming convention for thrift targets
kumar-zlai Jan 9, 2025
b8d54e8
Partial commit with build config to merge changes from main
kumar-zlai Jan 9, 2025
fd4163a
Merge branch 'kumarteja/bazel-build' into kumarteja/online-bazel-migr…
kumar-zlai Jan 9, 2025
6f3def4
working bazel build config for online module
kumar-zlai Jan 9, 2025
d4e34ee
Merge branch 'main' into kumarteja/online-bazel-migration
kumar-zlai Jan 9, 2025
e13ac8a
Minor changes to cleanup unused imports
kumar-zlai Jan 9, 2025
cb2b3fd
minor scalafixAll changes
kumar-zlai Jan 9, 2025
add315d
All spark targets working
kumar-zlai Jan 10, 2025
5fea6d9
Initial working version with all spark targets
kumar-zlai Jan 10, 2025
76c6f62
Merged with main branch
kumar-zlai Jan 10, 2025
cae2d80
Minor clean up of unused dependencies
kumar-zlai Jan 10, 2025
0c8a955
Modified logging implementation dependency to match out sbt config
kumar-zlai Jan 10, 2025
da2ff7f
Merge branch 'main' into kumarteja/spark-bazel-migration
kumar-zlai Jan 15, 2025
13182dd
Initial working version for src lib, tests pending
kumar-zlai Jan 16, 2025
9dd1e09
test-lib build still not working
kumar-zlai Jan 16, 2025
3076062
unit tests build working
kumar-zlai Jan 17, 2025
f449cbf
Merged with latest main branch
kumar-zlai Jan 17, 2025
2e8fa95
Merged with latest main branch
kumar-zlai Jan 17, 2025
6a08127
Minor refactoring to remove duplicate dependency declaration
kumar-zlai Jan 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions flink/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
flink_deps = [
scala_artifact("org.apache.flink:flink-streaming-scala"),
scala_artifact("org.apache.flink:flink-scala"),
maven_artifact("org.apache.flink:flink-streaming-java"),
maven_artifact("org.apache.flink:flink-core"),
maven_artifact("org.apache.flink:flink-metrics-dropwizard"),
maven_artifact("org.apache.flink:flink-metrics-core"),
maven_artifact("org.apache.flink:flink-clients"),
maven_artifact("org.apache.flink:flink-yarn"),
]

flink_test_deps = [
# Libraries
scala_artifact("org.apache.flink:flink-streaming-scala"),
scala_artifact("org.apache.flink:flink-scala"),
maven_artifact("org.apache.flink:flink-streaming-java"),
maven_artifact("org.apache.flink:flink-core"),
maven_artifact("org.apache.flink:flink-runtime"),
# Testing
maven_artifact("org.apache.flink:flink-test-utils"),
maven_artifact("org.apache.flink:flink-runtime-tests"),
]

scala_library(
name = "lib",
srcs = glob(["src/main/**/*.scala"]),
visibility = ["//visibility:public"],
deps = flink_deps + [
"//api:lib",
"//api:thrift",
"//aggregator:lib",
"//online:lib",
"//tools/build_rules/spark:spark-exec",
scala_artifact("org.scala-lang.modules:scala-java8-compat"),
scala_artifact("org.scala-lang.modules:scala-collection-compat"),
scala_artifact("org.rogach:scallop"),
maven_artifact("io.dropwizard.metrics:metrics-core"),
],
)

test_deps = flink_test_deps + [
":lib",
"//online:lib",
"//api:thrift",
"//api:lib",
"//aggregator:lib",
"//tools/build_rules/spark:spark-exec",
# Libraries
scala_artifact("org.scala-lang.modules:scala-java8-compat"),
scala_artifact("org.scala-lang.modules:scala-collection-compat"),
# Testing
scala_artifact("org.scalatest:scalatest-matchers-core"),
scala_artifact("org.scalatest:scalatest-core"),
scala_artifact("org.scalatest:scalatest"),
scala_artifact("org.scalatest:scalatest-flatspec"),
scala_artifact("org.scalatest:scalatest-funsuite"),
scala_artifact("org.scalatest:scalatest-shouldmatchers"),
scala_artifact("org.scalactic:scalactic"),
scala_artifact("org.scalatestplus:mockito-3-4"),
scala_artifact("org.mockito:mockito-scala"),
maven_artifact("org.mockito:mockito-core"),
maven_artifact("org.scalatest:scalatest-compatible"),
maven_artifact("junit:junit"),
maven_artifact("com.novocode:junit-interface"),
]

scala_library(
name = "test-lib",
srcs = glob(["src/test/**/*.scala"]),
visibility = ["//visibility:public"],
deps = test_deps,
)

scala_test_suite(
name = "test",
srcs = glob(["src/test/**/*.scala"]),
visibility = ["//visibility:public"],
deps = test_deps + [":test-lib",],
)
9 changes: 9 additions & 0 deletions tools/build_rules/dependencies/maven_repository.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,15 @@ maven_repository = repository(
# Kafka
"org.apache.kafka:kafka-clients:3.9.0",

# Flink
"org.apache.flink:flink-streaming-scala_2.12:1.17.0",
"org.apache.flink:flink-metrics-dropwizard:1.17.0",
"org.apache.flink:flink-clients:1.17.0",
"org.apache.flink:flink-yarn:1.17.0",
"org.apache.flink:flink-runtime:1.17.0",
"org.apache.flink:flink-runtime:1.17.0:tests",
"org.apache.flink:flink-test-utils:1.17.0",

# Vertx
"io.vertx:vertx-core:4.5.10",
"io.vertx:vertx-web:4.5.10",
Expand Down
Loading