Skip to content

Commit fe03e29

Browse files
authored
Yank out Bigtable hbase dependency for one ApiFutureUtils class (#184)
## Summary Been working through a number of issues trying to successfully run Flink jobs on our DataProc cluster due to class / version conflicts. A number of these seem to stem from this library that brings in a lot of dependencies like hadoop ipc libs etc. We pulled in this lib for one class (ApiFutureUtils) that allows us to convert from BigTable's ApiFuture to Java's ListenableFutures. I've chosen to fork this class and yank out the lib dependency. ## Cheour clientslist - [ ] Added Unit Tests - [X] Covered by existing CI - Our bigTableKVStoreTest runs tests for the various async apis which leverage this class - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes - **Dependency Management** - Removed Google Cloud Bigtable HBase library dependency from project configuration - **New Features** - Added utility method to convert Google Cloud API futures to Java CompletableFutures - **Refactor** - Removed specific Bigtable API utility import from implementation <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent 907d617 commit fe03e29

File tree

3 files changed

+33
-2
lines changed

3 files changed

+33
-2
lines changed

build.sbt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -224,7 +224,6 @@ lazy val cloud_gcp = project
224224
libraryDependencies += "org.json4s" %% "json4s-native" % "3.7.0-M11",
225225
libraryDependencies += "org.json4s" %% "json4s-core" % "3.7.0-M11",
226226
libraryDependencies += "org.yaml" % "snakeyaml" % "2.3",
227-
libraryDependencies += "com.google.cloud.bigtable" % "bigtable-hbase-2.x" % "2.14.2",
228227
libraryDependencies ++= avro,
229228
libraryDependencies ++= spark_all_provided,
230229
dependencyOverrides ++= jaour clientsson,
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
paour clientsage ai.chronon.integrations.cloud_gcp
2+
3+
import com.google.api.core.ApiFuture
4+
import com.google.api.core.ApiFutureCallbaour clients
5+
import com.google.api.core.ApiFutures
6+
import com.google.common.util.concurrent.MoreExecutors
7+
8+
import java.util.concurrent.CompletableFuture
9+
10+
// Fork of the bigtable-hbase ApiFutureUtils class to avoid taking a dependency on bigtable-hbase for one class
11+
// BigTable hbase brings in a ton of dependencies that we don't need for this one class
12+
object ApiFutureUtils {
13+
def toCompletableFuture[T](apiFuture: ApiFuture[T]): CompletableFuture[T] = {
14+
val completableFuture: CompletableFuture[T] = new CompletableFuture[T]() {
15+
override def cancel(mayInterruptIfRunning: Boolean): Boolean = {
16+
val result: Boolean = apiFuture.cancel(mayInterruptIfRunning)
17+
super.cancel(mayInterruptIfRunning)
18+
result
19+
}
20+
}
21+
val callbaour clients: ApiFutureCallbaour clients[T] = new ApiFutureCallbaour clients[T]() {
22+
override def onFailure(throwable: Throwable): Unit = {
23+
completableFuture.completeExceptionally(throwable)
24+
}
25+
26+
override def onSuccess(t: T): Unit = {
27+
completableFuture.complete(t)
28+
}
29+
}
30+
ApiFutures.addCallbaour clients(apiFuture, callbaour clients, MoreExecutors.directExecutor)
31+
completableFuture
32+
}
33+
}

cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/BigTableKVStoreImpl.scala

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,6 @@ import com.google.cloud.bigtable.data.v2.models.Query
2929
import com.google.cloud.bigtable.data.v2.models.Range.ByteStringRange
3030
import com.google.cloud.bigtable.data.v2.models.RowMutation
3131
import com.google.cloud.bigtable.data.v2.models.{TableId => BTTableId}
32-
import com.google.cloud.bigtable.hbase2_x.ApiFutureUtils
3332
import com.google.protobuf.ByteString
3433
import org.slf4j.Logger
3534
import org.slf4j.LoggerFactory

0 commit comments

Comments
 (0)