Skip to content

Latest commit

 

History

History
123 lines (100 loc) · 6.35 KB

firebolt.md

File metadata and controls

123 lines (100 loc) · 6.35 KB

Firebolt

This page guides you through the process of setting up the Firebolt destination connector.

Prerequisites

This Firebolt destination connector has two replication strategies:

  1. SQL: Replicates data via SQL INSERT queries. This leverages Firebolt SDK to execute queries directly on Firebolt Engines. Not recommended for production workloads as this does not scale well.

  2. S3: Replicates data by first uploading data to an S3 bucket, creating an External Table and writing into a final Fact Table. This is the recommended loading approach. Requires an S3 bucket and credentials in addition to Firebolt credentials.

For SQL strategy:

  • Host
  • Username
  • Password
  • Database
  • Account
  • Engine

Airbyte automatically picks an approach depending on the given configuration - if S3 configuration is present, Airbyte will use the S3 strategy.

For S3 strategy:

  • Username
  • Password
  • Database
  • Account
  • S3 Bucket Name
    • See this to create an S3 bucket.
  • S3 Bucket Region
    • Create the S3 bucket on the same region as the Firebolt database.
  • Access Key Id
  • Secret Access Key
    • Corresponding key to the above key id.
  • Host (optional)
    • Firebolt backend URL. Can be left blank for most usecases.
  • Engine (optional)
    • If connecting to a non-default engine you should specify its name or url here.

Setup guide

  1. Sign up to Firebolt following the guide
  2. Follow the getting started tutorial to setup a database.
  3. Create a service account.
  4. Create an engine as described in here
  5. (Optional) Create a staging S3 bucket (for the S3 strategy).
  6. (Optional) Create an IAM with programmatic access to read, write and delete objects from an S3 bucket.

Supported sync modes

The Firebolt destination connector supports the following sync modes:

  • Full Refresh
  • Incremental - Append Sync

Connector-specific features & highlights

Output schema

Each stream will be output into its own raw Fact table in Firebolt. Each table will contain 3 columns:

  • _airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed. The column type in Firebolt is VARCHAR.
  • _airbyte_emitted_at: a timestamp representing when the event was pulled from the data source. The column type in Firebolt is TIMESTAMP.
  • _airbyte_data: a json blob representing the event data. The column type in Firebolt is VARCHAR but can be be parsed with JSON functions.

Changelog

Expand to review
Version Date Pull Request Subject
0.2.18 2024-09-14 45562 Update dependencies
0.2.17 2024-09-07 45245 Update dependencies
0.2.16 2024-08-31 44991 Update dependencies
0.2.15 2024-08-24 44698 Update dependencies
0.2.14 2024-08-22 44530 Update test dependencies
0.2.13 2024-08-17 44239 Update dependencies
0.2.12 2024-08-10 43682 Update dependencies
0.2.11 2024-08-03 43143 Update dependencies
0.2.10 2024-07-27 42703 Update dependencies
0.2.9 2024-07-20 42211 Update dependencies
0.2.8 2024-07-13 41789 Update dependencies
0.2.7 2024-07-10 41602 Update dependencies
0.2.6 2024-07-09 41118 Update dependencies
0.2.5 2024-07-06 40854 Update dependencies
0.2.4 2024-06-27 40578 Replaced deprecated AirbyteLogger with logging.Logger
0.2.3 2024-06-25 40494 Update dependencies
0.2.2 2024-06-22 40078 Update dependencies
0.2.1 2024-06-06 39157 [autopull] Upgrade base image to v1.2.2
0.2.0 2024-05-08 36443 Service account authentication support
0.1.1 2024-03-05 35838 Un-archive connector
0.1.0 2022-05-18 13118 New Destination: Firebolt