Skip to content

✨Destination Iceberg: Bump Iceberg from 1.1.0 to 1.3.0 and add REST catalog support #28158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 19, 2023
Merged

✨Destination Iceberg: Bump Iceberg from 1.1.0 to 1.3.0 and add REST catalog support #28158

merged 2 commits into from
Jul 19, 2023

Conversation

nastra
Copy link
Contributor

@nastra nastra commented Jul 11, 2023 •
edited
Loading

What

This updates the Iceberg dependency from 1.1.0 to 1.3.0 and also adds REST catalog support

Pre-merge Actions

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.

@octavia-squidington-iii octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jul 11, 2023
@github-actions
Copy link
Contributor

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • The connector tests are passing in CI
  • You've updated the connector's metadata.yaml file (new!)
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@nastra nastra changed the title Destination Iceberg: Bump Iceberg from 1.1.0 to 1.2.1 and add REST catalog support ✨Destination Iceberg: Bump Iceberg from 1.1.0 to 1.2.1 and add REST catalog support Jul 11, 2023
@nastra
Copy link
Contributor Author

nastra commented Jul 11, 2023

image

@nastra nastra changed the title ✨Destination Iceberg: Bump Iceberg from 1.1.0 to 1.2.1 and add REST catalog support ✨Destination Iceberg: Bump Iceberg from 1.1.0 to 1.3.0 and add REST catalog support Jul 12, 2023
@nastra nastra force-pushed the rest-catalog-integration branch from df32ec5 to 35103a1 Compare July 12, 2023 05:56
nastra
nastra commented Jul 12, 2023
View reviewed changes
airbyte-integrations/connectors/destination-iceberg/build.gradle
implementation "software.amazon.awssdk:sts:2.17.257"
implementation "software.amazon.awssdk:sdk-core:2.17.257"
implementation "software.amazon.awssdk:aws-core:2.17.257"
implementation "software.amazon.awssdk:utils:2.20.18"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is to match the AWS SDK version being used by Iceberg: https://github.com/apache/iceberg/blob/apache-iceberg-1.3.0/versions.props#L24

nastra
nastra commented Jul 12, 2023
View reviewed changes
...tion-iceberg/src/main/java/io/airbyte/integrations/destination/iceberg/IcebergConstants.java
@@ -27,6 +27,9 @@ public class IcebergConstants {
public static final String JDBC_PASSWORD_CONFIG_KEY = "password";
public static final String JDBC_SSL_CONFIG_KEY = "ssl";
public static final String JDBC_CATALOG_SCHEMA_CONFIG_KEY = "catalog_schema";
public static final String REST_CATALOG_URI_CONFIG_KEY = "rest_uri";
public static final String REST_CATALOG_CREDENTIAL_CONFIG_KEY = "rest_credential";
Copy link
Contributor Author

@nastra nastra Jul 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REST catalog uses either a token or a credential (both are optional) to interact with the REST server: https://github.com/apache/iceberg/blob/apache-iceberg-1.3.0/core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Properties.java#L24-L28

nastra
nastra commented Jul 12, 2023
View reviewed changes
...tions/connectors/destination-iceberg/src/test-integration/resources/rest-catalog-compose.yml
@@ -0,0 +1,42 @@
version: "3"

services:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a slimmed down version of Iceberg's quickstart example: https://iceberg.apache.org/spark-quickstart/

@marcosmarxm marcosmarxm added the team/tse Technical Support Engineers label Jul 12, 2023
@marcosmarxm
Copy link
Member

Hello @nastra I added this our team's backlog and I'm going to review during the week.

@nastra nastra force-pushed the rest-catalog-integration branch from 35103a1 to f976b19 Compare July 19, 2023 07:00
marcosmarxm
marcosmarxm approved these changes Jul 19, 2023
View reviewed changes
Copy link
Member

@marcosmarxm marcosmarxm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nastra

@marcosmarxm
Copy link
Member

/approve-and-merge reason="Connector still uses http requests which is going to be fix in next contribution"

@octavia-approvington
Copy link
Contributor

This looks fine!
Merged!
Imagine it being fine

@octavia-approvington octavia-approvington merged commit 588c7d6 into airbytehq:master Jul 19, 2023
@nastra nastra deleted the rest-catalog-integration branch July 20, 2023 06:21
efimmatytsin pushed a commit to scentbird/airbyte that referenced this pull request Jul 27, 2023
@nastra @marcosmarxm @efimmatytsin
Destination Iceberg: Bump Iceberg from 1.1.0 to 1.3.0 and add REST ca…
7d756dc 
…talog support (airbytehq#28158)

Co-authored-by: Marcos Marx <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marcosmarxm marcosmarxm marcosmarxm approved these changes

@octavia-approvington octavia-approvington octavia-approvington approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation community connectors/destination/iceberg team/tse Technical Support Engineers
Projects
No open projects
Community Contributed PRs
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@nastra @marcosmarxm @octavia-approvington @octavia-squidington-iii