Skip to content

Commit 679ce18

Browse files
tulirentuliren
authored
🎉 Destination databricks: move driver downloading to gradle (#9995)
* Download jdbc driver in dockerfile * Download jdbc driver in dockerfile * Remove driver downloading from ci * Update document * Update public doc * Download jdbc driver in gradle * Clean up gradle file * Clean up dockerfile * Fix typo
1 parent a6d89ce commit 679ce18

File tree

9 files changed

+41
-48
lines changed

9 files changed

+41
-48
lines changed

‎.github/workflows/publish-command.yml

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -67,10 +67,6 @@ jobs:
6767
uses: actions/checkout@v2
6868
with:
6969
repository: ${{github.event.pull_request.head.repo.full_name}} # always use the branch's repository
70-
- name: Install Unzip for Databricks
71-
if: github.event.inputs.connector == 'connectors/destination-databricks'
72-
run: |
73-
apt-get update && apt-get install -y unzip
7470
- name: Install Java
7571
uses: actions/setup-java@v1
7672
with:

‎.github/workflows/test-command.yml

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -62,10 +62,6 @@ jobs:
6262
uses: actions/checkout@v2
6363
with:
6464
repository: ${{ github.event.inputs.repo }}
65-
- name: Install Unzip for Databricks
66-
if: github.event.inputs.connector == 'connectors/destination-databricks'
67-
run: |
68-
apt-get update && apt-get install -y unzip
6965
- name: Install Java
7066
uses: actions/setup-java@v1
7167
with:

‎airbyte-integrations/connectors/destination-databricks/.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@
33
# https://databricks.com/spark/jdbc-drivers-download
44
# By downloading this driver, you agree to the terms & conditions:
55
# https://databricks.com/jdbc-odbc-driver-license
6-
lib/SparkJDBC42.jar
6+
lib/*

‎airbyte-integrations/connectors/destination-databricks/README.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,16 @@ This is the repository for the Databricks destination connector in Java.
44
For information about how to use this connector within Airbyte, see [the User Documentation](https://docs.airbyte.io/integrations/destinations/databricks).
55

66
## Databricks JDBC Driver
7-
This connector requires a JDBC driver to connect to Databricks cluster. The driver is developed by Simba. Before downloading and using this driver, you must agree to the [JDBC ODBC driver license](https://databricks.com/jdbc-odbc-driver-license). This means that you can only use this driver to connector third party applications to Apache Spark SQL within a Databricks offering using the ODBC and/or JDBC protocols. The driver can be downloaded from [here](https://databricks.com/spark/jdbc-drivers-download).
7+
This connector requires a JDBC driver to connect to Databricks cluster. The driver is developed by Simba.
88

9-
The CI downloads the JDBC driver in [this script](https://github.com/airbytehq/airbyte/blob/master/tools/lib/databricks.sh).
9+
{% hint style="warning" %}
10+
Before building, or using this connector, you must agree to the [JDBC ODBC driver license](https://databricks.com/jdbc-odbc-driver-license). This means that you can only use this driver to connector third party applications to Apache Spark SQL within a Databricks offering using the ODBC and/or JDBC protocols.
11+
{% endhint %}
1012

11-
This is currently a private connector that is only available in Airbyte Cloud. To build and publish this connector, first download the driver and put it under the `lib` directory. Please do not publish this connector publicly. We are working on a solution to publicize it.
13+
This is currently a private connector that is only available on Airbyte Cloud. We are working on a solution to publicize it (issue [\#6043](https://github.com/airbytehq/airbyte/issues/6043)).
14+
15+
- If you want to use this connector now, you can build the connector locally, and publish it to your own docker registry. See the "[Build](#build)" or "[Building via Gradle](#building-via-gradle)" sections below for details. Please do not publish this connector publicly.
16+
- If you want to work on the connector code, first build the connector. The building process will automatically download the driver to the [`lib`](./lib) directory. You can also manually download the driver from [here](https://databricks.com/spark/jdbc-drivers-download).
1217

1318
## Local development
1419

‎airbyte-integrations/connectors/destination-databricks/build.gradle

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,22 @@
1+
// #############################################################################################
2+
// # #
3+
// # _______________ _____ ______ _ _ _ #
4+
// # |_ | _ | ___ / __ \ | _ \ (_) | | (_) #
5+
// # | | | | | |_/ | / \/ | | | |_ __ ___ _____ _ __ | | _ ___ ___ _ __ ___ ___ #
6+
// # | | | | | ___ | | | | | | '__| \ \ / / _ | '__| | | | |/ __/ _ | '_ \/ __|/ _ \ #
7+
// # /\__/ | |/ /| |_/ | \__/\ | |/ /| | | |\ V | __| | | |___| | (_| __| | | \__ | __/ #
8+
// # \____/|___/ \____/ \____/ |___/ |_| |_| \_/ \___|_| \_____|_|\___\___|_| |_|___/\___| #
9+
// # #
10+
// # By building this connector, you agree to the JDBC ODBC driver license: #
11+
// # https://databricks.com/jdbc-odbc-driver-license #
12+
// # #
13+
// #############################################################################################
14+
115
plugins {
216
id 'application'
317
id 'airbyte-docker'
418
id 'airbyte-integration-test-java'
19+
id "de.undercouch.download" version "5.0.1"
520
}
621

722
application {
@@ -33,3 +48,19 @@ dependencies {
3348
integrationTestJavaImplementation project(':airbyte-integrations:bases:standard-destination-test')
3449
integrationTestJavaImplementation project(':airbyte-integrations:connectors:destination-databricks')
3550
}
51+
52+
task downloadJdbcDriverZip(type: Download) {
53+
src 'https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/2.6.21/SimbaSparkJDBC42-2.6.21.1039.zip'
54+
dest new File(buildDir, 'SimbaSparkJDBC42-2.6.21.1039.zip')
55+
overwrite false
56+
}
57+
58+
task extractJdbcDriverFile(dependsOn: downloadJdbcDriverZip, type: Copy) {
59+
from {
60+
zipTree(downloadJdbcDriverZip.dest)
61+
}
62+
into 'lib/'
63+
include 'SparkJDBC42.jar'
64+
}
65+
66+
compileJava.dependsOn tasks.extractJdbcDriverFile

‎docs/integrations/destinations/databricks.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ This destination syncs data to Databricks cluster. Each stream is written to its
66

77
This connector requires a JDBC driver to connect to Databricks cluster. The driver is developed by Simba. Before using the driver and the connector, you must agree to the [JDBC ODBC driver license](https://databricks.com/jdbc-odbc-driver-license). This means that you can only use this connector to connector third party applications to Apache Spark SQL within a Databricks offering using the ODBC and/or JDBC protocols.
88

9-
Due to legal reasons, this is currently a private connector that is only available in Airbyte Cloud. We are working on publicizing it. Please follow [this issue](https://github.com/airbytehq/airbyte/issues/6043) for progress.
9+
Due to legal reasons, this is currently a private connector that is only available on Airbyte Cloud. We are working on publicizing it. Please follow [this issue](https://github.com/airbytehq/airbyte/issues/6043) for progress. In the interim, you can build the connector locally, publish it to your own image registry, and use it privately. See the [developer doc](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/destination-databricks/README.md) for details. Please do not publish this connector publicly.
1010

1111
## Sync Mode
1212

‎tools/bin/ci_integration_test.sh

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,13 @@
33
set -e
44

55
. tools/lib/lib.sh
6-
. tools/lib/databricks.sh
76

87
# runs integration tests for an integration name
98

109
connector="$1"
1110
all_integration_tests=$(./gradlew integrationTest --dry-run | grep 'integrationTest SKIPPED' | cut -d: -f 4)
1211
run() {
1312
if [[ "$connector" == "all" ]] ; then
14-
_get_databricks_jdbc_driver
1513
echo "Running: ./gradlew --no-daemon --scan integrationTest"
1614
./gradlew --no-daemon --scan integrationTest
1715
else
@@ -36,10 +34,6 @@ else
3634
integrationTestCommand=":airbyte-integrations:connectors:$connector:integrationTest"
3735
fi
3836
if [ -n "$selected_integration_test" ] ; then
39-
if [[ "$selected_integration_test" == *"databricks"* ]] ; then
40-
_get_databricks_jdbc_driver
41-
fi
42-
4337
echo "Running: ./gradlew --no-daemon --scan $integrationTestCommand"
4438
./gradlew --no-daemon --scan "$integrationTestCommand"
4539
else

‎tools/integrations/manage.sh

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@ set -e
44
set -x
55

66
. tools/lib/lib.sh
7-
. tools/lib/databricks.sh
87

98
USAGE="
109
Usage: $(basename "$0") <cmd>
@@ -40,10 +39,6 @@ cmd_build() {
4039

4140
local run_tests=$1; shift || run_tests=true
4241

43-
if [[ "airbyte-integrations/connectors/destination-databricks" == "${path}" ]]; then
44-
_get_databricks_jdbc_driver
45-
fi
46-
4742
echo "Building $path"
4843
./gradlew --no-daemon "$(_to_gradle_path "$path" clean)"
4944
./gradlew --no-daemon "$(_to_gradle_path "$path" build)"

‎tools/lib/databricks.sh

Lines changed: 0 additions & 24 deletions
This file was deleted.

0 commit comments

Comments
 (0)