🎉 New Destination: Yellowbrick (#35775)

Co-authored-by: Marcos Marx <[email protected]>

Author: markcusack

airbyte-integrations/connectors/destination-yellowbrick/BOOTSTRAP.md

@@ -0,0 +1,7 @@
+# Yellowbrick Destination Connector Bootstrap
+
+Yellowbrick is a highly efficient and elastically scalable data warehouse that runs on Kubernetes in all major public clouds and on-premises.
+
+Yellowbrick connector produces the standard Airbyte outputs using `_airbyte_raw_*` tables storing the JSON blob data first. Afterward, these are transformed and normalized into separate tables, potentially "exploding" nested streams into their own tables if [basic normalization](https://docs.airbyte.io/understanding-airbyte/basic-normalization) is configured.
+
+See [this](https://docs.airbyte.io/integrations/destinations/yellowbrick) link for more information about the connector.

airbyte-integrations/connectors/destination-yellowbrick/README.md

@@ -0,0 +1,68 @@
+# Destination Yellowbrick
+
+This is the repository for the Yellowbrick destination connector in Java.
+For information about how to use this connector within Airbyte, see [the User Documentation](https://docs.airbyte.com/integrations/destinations/yellowbrick).
+
+## Local development
+
+#### Building via Gradle
+From the Airbyte repository root, run:
+```
+./gradlew :airbyte-integrations:connectors:destination-yellowbrick:build
+```
+
+#### Create credentials
+**If you are a community contributor**, generate the necessary credentials and place them in `secrets/config.json` conforming to the spec file in `src/main/resources/spec.json`.
+Note that the `secrets` directory is git-ignored by default, so there is no danger of accidentally checking in sensitive information.
+
+**If you are an Airbyte core member**, follow the [instructions](https://docs.airbyte.com/connector-development#using-credentials-in-ci) to set up the credentials.
+
+### Locally running the connector docker image
+
+#### Build
+Build the connector image via Gradle:
+```
+./gradlew :airbyte-integrations:connectors:destination-yellowbrick:airbyteDocker
+```
+When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
+the Dockerfile.
+
+#### Run
+Then run any of the connector commands as follows:
+```
+docker run --rm airbyte/destination-yellowbrick:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-yellowbrick:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-yellowbrick:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-yellowbrick:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+
+## Testing
+We use `JUnit` for Java tests.
+
+### Unit and Integration Tests
+Place unit tests under `src/test/io/airbyte/integrations/destinations/yellowbrick`.
+
+#### Acceptance Tests
+Airbyte has a standard test suite that all destination connectors must pass. Implement the `TODO`s in
+`src/test-integration/java/io/airbyte/integrations/destinations/yellowbrickDestinationAcceptanceTest.java`.
+
+### Using gradle to run tests
+All commands should be run from airbyte project root.
+To run unit tests:
+```
+./gradlew :airbyte-integrations:connectors:destination-yellowbrick:unitTest
+```
+To run acceptance and custom integration tests:
+```
+./gradlew :airbyte-integrations:connectors:destination-yellowbrick:integrationTest
+```
+
+## Dependency Management
+
+### Publishing a new version of the connector
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+1. Make sure your changes are passing unit and integration tests.
+1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
+1. Create a Pull Request.
+1. Pat yourself on the back for being an awesome contributor.
+1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

airbyte-integrations/connectors/destination-yellowbrick/build.gradle

@@ -0,0 +1,15 @@
+plugins {
+    id 'airbyte-java-connector'
+    id 'org.jetbrains.kotlin.jvm' version '1.9.23'
+}
+
+airbyteJavaConnector {
+    cdkVersionRequired = '0.27.7'
+    features = ['db-destinations', 'datastore-postgres', 'typing-deduping']
+    useLocalCdk = false
+}
+
+application {
+    mainClass = 'io.airbyte.integrations.destination.yellowbrick.YellowbrickDestination'
+    applicationDefaultJvmArgs = ['-XX:+ExitOnOutOfMemoryError', '-XX:MaxRAMPercentage=75.0']
+}

airbyte-integrations/connectors/destination-yellowbrick/icon.svg

@@ -0,0 +1,26 @@
+data:
+  ab_internal:
+    ql: 200
+    sl: 100
+  connectorSubtype: database
+  connectorType: destination
+  definitionId: 1f7bac7e-53ff-4e0b-b6df-b74aa85cf703
+  dockerImageTag: 0.0.1
+  dockerRepository: airbyte/destination-yellowbrick
+  documentationUrl: https://docs.airbyte.com/integrations/destinations/yellowbrick
+  githubIssueLabel: destination-yellowbrick
+  icon: yellowbrick.svg
+  license: ELv2
+  name: Yellowbrick
+  registries:
+    cloud:
+      dockerRepository: airbyte/destination-yellowbrick
+      enabled: true
+    oss:
+      enabled: true
+  releaseStage: alpha
+  supportLevel: community
+  supportsDbt: false
+  tags:
+    - language:java
+metadataSpecVersion: "1.0"

airbyte-integrations/connectors/destination-yellowbrick/metadata.yaml

@@ -0,0 +1,143 @@
+/*
+ * Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+ */
+
+package io.airbyte.integrations.destination.yellowbrick;
+
+import static io.airbyte.cdk.integrations.util.PostgresSslConnectionUtils.DISABLE;
+import static io.airbyte.cdk.integrations.util.PostgresSslConnectionUtils.PARAM_MODE;
+import static io.airbyte.cdk.integrations.util.PostgresSslConnectionUtils.PARAM_SSL;
+import static io.airbyte.cdk.integrations.util.PostgresSslConnectionUtils.PARAM_SSL_MODE;
+import static io.airbyte.cdk.integrations.util.PostgresSslConnectionUtils.obtainConnectionOptions;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.collect.ImmutableMap;
+import io.airbyte.cdk.db.factory.DatabaseDriver;
+import io.airbyte.cdk.db.jdbc.JdbcDatabase;
+import io.airbyte.cdk.db.jdbc.JdbcUtils;
+import io.airbyte.cdk.integrations.base.*;
+import io.airbyte.cdk.integrations.base.ssh.SshWrappedDestination;
+import io.airbyte.cdk.integrations.destination.async.deser.StreamAwareDataTransformer;
+import io.airbyte.cdk.integrations.destination.jdbc.AbstractJdbcDestination;
+import io.airbyte.cdk.integrations.destination.jdbc.typing_deduping.JdbcDestinationHandler;
+import io.airbyte.cdk.integrations.destination.jdbc.typing_deduping.JdbcSqlGenerator;
+import io.airbyte.commons.json.Jsons;
+import io.airbyte.integrations.base.destination.typing_deduping.DestinationHandler;
+import io.airbyte.integrations.base.destination.typing_deduping.ParsedCatalog;
+import io.airbyte.integrations.base.destination.typing_deduping.SqlGenerator;
+import io.airbyte.integrations.base.destination.typing_deduping.migrators.Migration;
+import io.airbyte.integrations.destination.yellowbrick.typing_deduping.YellowbrickDataTransformer;
+import io.airbyte.integrations.destination.yellowbrick.typing_deduping.YellowbrickDestinationHandler;
+import io.airbyte.integrations.destination.yellowbrick.typing_deduping.YellowbrickSqlGenerator;
+import io.airbyte.integrations.destination.yellowbrick.typing_deduping.YellowbrickState;
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class YellowbrickDestination extends AbstractJdbcDestination<YellowbrickState> implements Destination {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(YellowbrickDestination.class);
+
+  public static final String DRIVER_CLASS = DatabaseDriver.POSTGRESQL.getDriverClassName();
+
+  public static Destination sshWrappedDestination() {
+    return new SshWrappedDestination(new YellowbrickDestination(), JdbcUtils.HOST_LIST_KEY, JdbcUtils.PORT_LIST_KEY);
+  }
+
+  public YellowbrickDestination() {
+    super(DRIVER_CLASS, new YellowbrickSQLNameTransformer(), new YellowbrickSqlOperations());
+  }
+
+  @Override
+  protected Map<String, String> getDefaultConnectionProperties(final JsonNode config) {
+    final Map<String, String> additionalParameters = new HashMap<>();
+    if (!config.has(PARAM_SSL) || config.get(PARAM_SSL).asBoolean()) {
+      if (config.has(PARAM_SSL_MODE)) {
+        if (DISABLE.equals(config.get(PARAM_SSL_MODE).get(PARAM_MODE).asText())) {
+          additionalParameters.put("sslmode", DISABLE);
+        } else {
+          additionalParameters.putAll(obtainConnectionOptions(config.get(PARAM_SSL_MODE)));
+        }
+      } else {
+        additionalParameters.put(JdbcUtils.SSL_KEY, "true");
+        additionalParameters.put("sslmode", "require");
+      }
+    }
+    return additionalParameters;
+  }
+
+  @Override
+  public JsonNode toJdbcConfig(final JsonNode config) {
+    final String schema = Optional.ofNullable(config.get(JdbcUtils.SCHEMA_KEY)).map(JsonNode::asText).orElse("public");
+
+    String encodedDatabase = config.get(JdbcUtils.DATABASE_KEY).asText();
+    if (encodedDatabase != null) {
+      try {
+        encodedDatabase = URLEncoder.encode(encodedDatabase, "UTF-8");
+      } catch (final UnsupportedEncodingException e) {
+        // Should never happen
+        e.printStackTrace();
+      }
+    }
+    final String jdbcUrl = String.format("jdbc:postgresql://%s:%s/%s?",
+        config.get(JdbcUtils.HOST_KEY).asText(),
+        config.get(JdbcUtils.PORT_KEY).asText(),
+        encodedDatabase);
+
+    final ImmutableMap.Builder<Object, Object> configBuilder = ImmutableMap.builder()
+        .put(JdbcUtils.USERNAME_KEY, config.get(JdbcUtils.USERNAME_KEY).asText())
+        .put(JdbcUtils.JDBC_URL_KEY, jdbcUrl)
+        .put(JdbcUtils.SCHEMA_KEY, schema);
+
+    if (config.has(JdbcUtils.PASSWORD_KEY)) {
+      configBuilder.put(JdbcUtils.PASSWORD_KEY, config.get(JdbcUtils.PASSWORD_KEY).asText());
+    }
+
+    if (config.has(JdbcUtils.JDBC_URL_PARAMS_KEY)) {
+      configBuilder.put(JdbcUtils.JDBC_URL_PARAMS_KEY, config.get(JdbcUtils.JDBC_URL_PARAMS_KEY).asText());
+    }
+
+    return Jsons.jsonNode(configBuilder.build());
+  }
+
+  @Override
+  protected JdbcDestinationHandler<YellowbrickState> getDestinationHandler(String databaseName, JdbcDatabase database, String rawTableSchema) {
+    return new YellowbrickDestinationHandler(databaseName, database, rawTableSchema);
+  }
+
+  @Override
+  protected JdbcSqlGenerator getSqlGenerator() {
+    return new YellowbrickSqlGenerator(new YellowbrickSQLNameTransformer());
+  }
+
+  @Override
+  protected StreamAwareDataTransformer getDataTransformer(ParsedCatalog parsedCatalog, String defaultNamespace) {
+    return new YellowbrickDataTransformer();
+  }
+
+  @Override
+  public boolean isV2Destination() {
+    return true;
+  }
+
+  @Override
+  protected List<Migration<YellowbrickState>> getMigrations(JdbcDatabase database,
+                                                            String databaseName,
+                                                            SqlGenerator sqlGenerator,
+                                                            DestinationHandler<YellowbrickState> destinationHandler) {
+    return List.of();
+  }
+
+  public static void main(final String[] args) throws Exception {
+    final Destination destination = YellowbrickDestination.sshWrappedDestination();
+    LOGGER.info("starting destination: {}", YellowbrickDestination.class);
+    new IntegrationRunner(destination).run(args);
+    LOGGER.info("completed destination: {}", YellowbrickDestination.class);
+  }
+
+}

airbyte-integrations/connectors/destination-yellowbrick/src/main/java/io/airbyte/integrations/destination/yellowbrick/YellowbrickDestination.java

@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+ */
+
+package io.airbyte.integrations.destination.yellowbrick;
+
+import io.airbyte.cdk.integrations.destination.StandardNameTransformer;
+
+public class YellowbrickSQLNameTransformer extends StandardNameTransformer {
+
+  @Override
+  public String applyDefaultCase(final String input) {
+    return input.toLowerCase();
+  }
+
+}