Skip to content

Commit a9e13ae

Browse files
authored
Yank out Bigtable hbase dependency for one ApiFutureUtils class (#184)
## Summary Been working through a number of issues trying to successfully run Flink jobs on our DataProc cluster due to class / version conflicts. A number of these seem to stem from this library that brings in a lot of dependencies like hadoop ipc libs etc. We pulled in this lib for one class (ApiFutureUtils) that allows us to convert from BigTable's ApiFuture to Java's ListenableFutures. I've chosen to fork this class and yank out the lib dependency. ## Checklist - [ ] Added Unit Tests - [X] Covered by existing CI - Our bigTableKVStoreTest runs tests for the various async apis which leverage this class - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes - **Dependency Management** - Removed Google Cloud Bigtable HBase library dependency from project configuration - **New Features** - Added utility method to convert Google Cloud API futures to Java CompletableFutures - **Refactor** - Removed specific Bigtable API utility import from implementation <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent 4c356fb commit a9e13ae

File tree

3 files changed

+33
-2
lines changed

3 files changed

+33
-2
lines changed

build.sbt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -224,7 +224,6 @@ lazy val cloud_gcp = project
224224
libraryDependencies += "org.json4s" %% "json4s-native" % "3.7.0-M11",
225225
libraryDependencies += "org.json4s" %% "json4s-core" % "3.7.0-M11",
226226
libraryDependencies += "org.yaml" % "snakeyaml" % "2.3",
227-
libraryDependencies += "com.google.cloud.bigtable" % "bigtable-hbase-2.x" % "2.14.2",
228227
libraryDependencies ++= avro,
229228
libraryDependencies ++= spark_all_provided,
230229
dependencyOverrides ++= jackson,
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
package ai.chronon.integrations.cloud_gcp
2+
3+
import com.google.api.core.ApiFuture
4+
import com.google.api.core.ApiFutureCallback
5+
import com.google.api.core.ApiFutures
6+
import com.google.common.util.concurrent.MoreExecutors
7+
8+
import java.util.concurrent.CompletableFuture
9+
10+
// Fork of the bigtable-hbase ApiFutureUtils class to avoid taking a dependency on bigtable-hbase for one class
11+
// BigTable hbase brings in a ton of dependencies that we don't need for this one class
12+
object ApiFutureUtils {
13+
def toCompletableFuture[T](apiFuture: ApiFuture[T]): CompletableFuture[T] = {
14+
val completableFuture: CompletableFuture[T] = new CompletableFuture[T]() {
15+
override def cancel(mayInterruptIfRunning: Boolean): Boolean = {
16+
val result: Boolean = apiFuture.cancel(mayInterruptIfRunning)
17+
super.cancel(mayInterruptIfRunning)
18+
result
19+
}
20+
}
21+
val callback: ApiFutureCallback[T] = new ApiFutureCallback[T]() {
22+
override def onFailure(throwable: Throwable): Unit = {
23+
completableFuture.completeExceptionally(throwable)
24+
}
25+
26+
override def onSuccess(t: T): Unit = {
27+
completableFuture.complete(t)
28+
}
29+
}
30+
ApiFutures.addCallback(apiFuture, callback, MoreExecutors.directExecutor)
31+
completableFuture
32+
}
33+
}

cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/BigTableKVStoreImpl.scala

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,6 @@ import com.google.cloud.bigtable.data.v2.models.Query
2929
import com.google.cloud.bigtable.data.v2.models.Range.ByteStringRange
3030
import com.google.cloud.bigtable.data.v2.models.RowMutation
3131
import com.google.cloud.bigtable.data.v2.models.{TableId => BTTableId}
32-
import com.google.cloud.bigtable.hbase2_x.ApiFutureUtils
3332
import com.google.protobuf.ByteString
3433
import org.slf4j.Logger
3534
import org.slf4j.LoggerFactory

0 commit comments

Comments
 (0)