You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| Connection Name | A custom name for your connection |
32
-
|[Replication frequency](/using-airbyte/core-concepts/sync-schedules.md)| How often data syncs (can be scheduled, cron, API-triggered or manual) |
33
-
|[Destination namespace](/using-airbyte/core-concepts/namespaces.md)| Where the replicated data is written to in the destination |
34
-
| Destination stream prefix| A prefix added to each table name in the destination |
32
+
|[Schedule Type](/using-airbyte/core-concepts/sync-schedules.md)| How often data syncs (can be scheduled, cron, API-triggered or manual) |
33
+
|[Destination Namespace](/using-airbyte/core-concepts/namespaces.md)| Where the replicated data is written to in the destination |
34
+
| Destination Stream Prefix| A prefix added to each table name in the destination |
35
35
|[Detect and propagate schema changes](/cloud/managing-airbyte-cloud/manage-schema-changes.md)| How Airbyte handles schema changes in the source |
36
36
|[Connection Data Residency](/cloud/managing-airbyte-cloud/manage-data-residency.md)| Where data will be processed (Cloud only) |
37
37
38
-
## Modify streams in your connection
38
+
## Modify Streams
39
39
40
-
In the **Activate the streams you want to sync** table, you choose which streams to sync and how they are loaded to the destination.
40
+
On the "Schema" tab, you choose which streams to sync and how they are loaded to the destination.
41
41
42
42
:::info
43
43
A connection's schema consists of one or many streams. Each stream is most commonly associated with a database table or an API endpoint. Within a stream, there can be one or many fields or columns.
44
44
:::
45
45
46
-
To modify streams:
46
+
To modify streams, click **Connections** and then click the connection you want to change. Click the **Schema** tab to see all the streams Airbyte can sync. To modify an individual stream:
47
47
48
-
1. In the Airbyte UI, click **Connections** and then click the connection you want to change.
49
-
50
-
2. Click the **Replication** tab.
51
-
52
-
3. Scroll down to the **Activate the streams you want to sync** table.
53
-
54
-
Modify an individual stream:
55
-
56
-
1. In the **Activate the streams you want to sync** table, toggle **Sync** on or off for your selected stream. To select or deselect all streams, click the checkbox in the table header. To deselect an individual stream, deselect its checkbox in the table.
48
+
1. Toggle **Sync** on or off for your selected stream. To select or deselect all streams at once, use "Hide disabled streams" in the table header. To deselect an individual stream, use the toggle in its row.
57
49
58
50
2. Click the **Sync mode** dropdown and select the sync mode you want to apply. Depending on the sync mode you select, you may need to choose a cursor or primary key.
| Destination default | All streams will be replicated to the single default namespace defined in the Destination's settings. |
14
-
| Mirror source structure | Some sources (for example, databases) provide namespace information for a stream. If a source provides namespace information, the destination will mirror the same namespace when this configuration is set. For sources or streams where the source namespace is not known, the behavior will default to the "Destination default" option. |
15
-
| Custom format | All streams will be replicated to a single user-defined namespace. See<ahref="/understanding-airbyte/namespaces#--custom-format"> Custom format</a> for more details |
13
+
| Custom | All streams will be replicated to a single user-defined namespace. See<ahref="/understanding-airbyte/namespaces#--custom-format"> Custom format</a> for more details |
14
+
| Destination-defined | All streams will be replicated to the single default namespace defined in the Destination's settings. |
15
+
| Source-defined | Some sources (for example, databases) provide namespace information for a stream. If a source provides namespace information, the destination will mirror the same namespace when this configuration is set. For sources or streams where the source namespace is not known, the behavior will default to the "Destination default" option. |
16
+
16
17
17
18
Most of our destinations support this feature. To learn if your connector supports this, head to the individual connector page to learn more. If your desired destination doesn't support it, you can ignore this feature.
18
19
@@ -26,7 +27,19 @@ In a source, the namespace is the location from where the data is replicated to
26
27
27
28
Airbyte supports namespaces and allows Sources to define namespaces, and Destinations to write to various namespaces. In Airbyte, the following options are available and are set on each individual connection.
28
29
29
-
### Destination default
30
+
### Custom
31
+
32
+
When replicating multiple sources into the same destination, you may create table conflicts where tables are overwritten by different syncs. This is where using a custom namespace will ensure data is synced accurately.
33
+
34
+
For example, a Github source can be replicated into a `github` schema. However, you may have multiple connections writing from different GitHub repositories \(common in multi-tenant scenarios\).
35
+
36
+
:::tip
37
+
To write more than 1 table with the same name to your destination, Airbyte recommends writing the connections to unique namespaces to avoid mixing data from the different GitHub repositories.
38
+
:::
39
+
40
+
You can enter plain text (most common) or additionally add a dynamic parameter `${SOURCE_NAMESPACE}`, which uses the namespace provided by the source if available.
41
+
42
+
### Destination-defined
30
43
31
44
All streams will be replicated and stored in the default namespace defined on the destination settings page, which is typically defined when the destination was set up. Depending on your destination, the namespace refers to:
32
45
@@ -45,21 +58,9 @@ All streams will be replicated and stored in the default namespace defined on th
45
58
If you prefer to replicate multiple sources into the same namespace, use the `Stream Prefix` configuration to differentiate data from these sources to ensure no streams collide when writing to the destination.
46
59
:::
47
60
48
-
### Mirror source structure
61
+
### Source-Defined
49
62
50
-
Some sources \(such as databases based on JDBC\) provide namespace information from which a stream has been extracted. Whenever a source is able to fill this field in the catalog.json file, the destination will try to write to exactly the same namespace when this configuration is set. For sources or streams where the source namespace is not known, the behavior will fall back to the "Destination default". Most APIs do not provide namespace information.
51
-
52
-
### Custom format
53
-
54
-
When replicating multiple sources into the same destination, you may create table conflicts where tables are overwritten by different syncs. This is where using a custom namespace will ensure data is synced accurately.
55
-
56
-
For example, a Github source can be replicated into a `github` schema. However, you may have multiple connections writing from different GitHub repositories \(common in multi-tenant scenarios\).
57
-
58
-
:::tip
59
-
To write more than 1 table with the same name to your destination, Airbyte recommends writing the connections to unique namespaces to avoid mixing data from the different GitHub repositories.
60
-
:::
61
-
62
-
You can enter plain text (most common) or additionally add a dynamic parameter `${SOURCE_NAMESPACE}`, which uses the namespace provided by the source if available.
63
+
Some sources \(such as databases based on JDBC\) provide namespace information from which a stream has been extracted. Whenever a source is able to fill this field in the catalog.json file, the destination will try to write to exactly the same namespace when this configuration is set. For sources or streams where the source namespace is not known, the behavior will fall back to the default namespace defined in the destination configuration. Most APIs do not provide namespace information.
|[Replication Frequency](/using-airbyte/core-concepts/sync-schedules.md)| When should a data sync be triggered? |
30
-
|[Destination Namespace and Stream Prefix](/using-airbyte/core-concepts/namespaces.md)| Where should the replicated data be written? |
29
+
|[Catalog Selection](/cloud/managing-airbyte-cloud/configuring-connections.md#modify-streams-in-your-connection)| What data should be replicated from the source to the destination? |
31
30
|[Sync Mode](/using-airbyte/core-concepts/sync-modes/README.md)| How should the streams be replicated (read and written)? |
31
+
|[Sync Schedule](/using-airbyte/core-concepts/sync-schedules.md)| When should a data sync be triggered? |
32
+
|[Destination Namespace and Stream Prefix](/using-airbyte/core-concepts/namespaces.md)| Where should the replicated data be written? |
32
33
|[Schema Propagation](/cloud/managing-airbyte-cloud/manage-schema-changes.md)| How should Airbyte handle schema drift in sources? |
33
-
|[Catalog Selection](/cloud/managing-airbyte-cloud/configuring-connections.md#modify-streams-in-your-connection)| What data should be replicated from the source to the destination? |
34
34
35
35
## Stream
36
36
@@ -51,7 +51,7 @@ Examples of fields:
51
51
- A column in the table in a relational database
52
52
- A field in an API response
53
53
54
-
## Sync Schedules
54
+
## Sync Schedule
55
55
56
56
There are three options for scheduling a sync to run:
Copy file name to clipboardExpand all lines: docs/using-airbyte/core-concepts/sync-schedules.md
+13-4
Original file line number
Diff line number
Diff line change
@@ -4,13 +4,13 @@ products: all
4
4
5
5
# Sync Schedules
6
6
7
-
For each connection, you can select between three options that allow a sync to run. The three options for `Replication Frequency` are:
7
+
For each connection, you can select between three options that allow a sync to run. The three options for `Schedule Type` are:
8
8
9
9
- Scheduled (e.g. every 24 hours, every 2 hours)
10
-
- Cron scheduling
10
+
- Cron
11
11
- Manual
12
12
13
-
## Sync Limitations
13
+
## Sync Considerations
14
14
15
15
* Only one sync per connection can run at a time.
16
16
* If a sync is scheduled to run before the previous sync finishes, the scheduled sync will start after the completion of the previous sync.
@@ -21,6 +21,15 @@ For Scheduled or cron scheduled syncs, Airbyte guarantees syncs will initiate wi
21
21
:::
22
22
23
23
## Scheduled syncs
24
+
You can choose between the following scheduled options:
25
+
- Every 24 hours (most common)
26
+
- Every 12 hours
27
+
- Every 8 hours
28
+
- Every 6 hours
29
+
- Every 3 hours
30
+
- Every 2 hours
31
+
- Every 1 hour
32
+
24
33
When a scheduled connection is first created, a sync is executed immediately after creation. After that, a sync is run once the time since the last sync \(whether it was triggered manually or due to a schedule\) has exceeded the schedule interval. For example:
25
34
26
35
-**October 1st, 2pm**, a user sets up a connection to sync data every 24 hours.
@@ -30,7 +39,7 @@ When a scheduled connection is first created, a sync is executed immediately aft
30
39
-**October 3rd, 2:01pm:** since the last sync was less than 24 hours ago, no sync is run
31
40
-**October 3rd, 5:01pm:** It has been more than 24 hours since the last sync, so a sync is run
32
41
33
-
## Cron Scheduling
42
+
## Cron Syncs
34
43
If you prefer more precision in scheduling your sync, you can also use CRON scheduling to set a specific time of day or month.
35
44
36
45
Airbyte uses the CRON scheduler from [Quartz](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html). We recommend reading their [documentation](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) to understand the required formatting. You can also refer to these examples:
Copy file name to clipboardExpand all lines: docs/using-airbyte/getting-started/add-a-destination.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ Destinations are the data warehouses, data lakes, databases and analytics tools
11
11
12
12
Once you've signed up for Airbyte Cloud or logged in to your Airbyte Open Source deployment, click on the **Destinations** tab in the navigation bar found on the left side of the dashboard. This will take you to the list of available destinations.
Copy file name to clipboardExpand all lines: docs/using-airbyte/getting-started/add-a-source.md
+1-2
Original file line number
Diff line number
Diff line change
@@ -6,8 +6,7 @@ products: all
6
6
7
7
Setting up a new source in Airbyte is a quick and simple process! When viewing the Airbyte UI, you'll see the main navigation bar on the left side of your screen. Click the **Sources** tab to bring up a list of all available sources.
8
8
9
-
<Arcadeid="4V0TGOX02P0rwVNwz4MR"title="Getting Started (Source)" />
10
-
9
+
<Arcadeid="WjbChISa931Hc55yx4cb"title="Getting Started (Source)" />
11
10
12
11
You can use the provided search bar, or simply scroll down the list to find the source you want to replicate data from. Let's use a demo source, Faker, as an example. Clicking on the **Sample Data (Faker)** card will bring us to its setup page.
0 commit comments