You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Fix more typos in the docs
* fix another typo
* add "to"
* make "example" plural
* remove extra words
* add period
* remove extra "the"
* use `source_defined_cursor` instead of `cursor_field` for `AirbyteStream`
* Use ConfiguredAirbyteCatalog instead of AirbyteCatalog
1.`connectionStatus` - an [AirbyteConnectionStatus](https://github.com/airbytehq/airbyte/blob/922bfd08a9182443599b78dbb273d70cb9f63d30/airbyte-protocol/models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml#L99-L112) wrapped in an `AirbyteMessage` of type `connection_status`.
83
83
* The `check` command attempts to connect to the underlying data source in order to verify that the provided credentials are usable.
84
-
* e.g. If the given the credentials, it can connect to the Postgres database, it will return a success response. If it fails \(perhaps the password is incorrect\), it will return a failed response and \(when possible\) a helpful error message.
84
+
* e.g. If given the credentials, it can connect to the Postgres database, it will return a success response. If it fails \(perhaps the password is incorrect\), it will return a failed response and \(when possible\) a helpful error message.
Copy file name to clipboardExpand all lines: docs/understanding-airbyte/beginners-guide-to-catalog.md
+5-5
Original file line number
Diff line number
Diff line change
@@ -31,7 +31,7 @@ CREATE TABLE "airlines" (
31
31
CREATETABLE "pilots" (
32
32
"id"INTEGER,
33
33
"airline_id"INTEGER,
34
-
"name"INTEGER
34
+
"name"VARCHAR
35
35
);
36
36
```
37
37
@@ -93,7 +93,7 @@ Let's walk through what each field in a stream means.
93
93
94
94
*`name` - The name of the stream.
95
95
*`supported_sync_modes` - This field lists the type of data replication that this source supports. The possible values in this array include `FULL_REFRESH`\([docs](connections/full-refresh-overwrite.md)\) and `INCREMENTAL`\([docs](connections/incremental-append.md)\).
96
-
*`source_defined_cursor` - If the stream supports `INCREMENTAL` replication, then this field signal whether the source can figure out how to detect new records on its own or not.
96
+
*`source_defined_cursor` - If the stream supports `INCREMENTAL` replication, then this field signals whether the source can figure out how to detect new records on its own or not.
97
97
*`json_schema` - This field is a [JsonSchema](https://json-schema.org/understanding-json-schema) object that describes the structure of the data. Notice that each key in the `properties` object corresponds to a column name in our database table.
98
98
99
99
Now we understand _what_ data is available from this source. Next we will configure _how_ we want to replicate that data.
@@ -136,7 +136,7 @@ Just as with the `AirbyteCatalog` the `ConfiguredAirbyteCatalog` contains a list
136
136
Let's walk through each field in the `ConfiguredAirbyteStream`:
137
137
138
138
*`sync_mode` - This field must be one of the values that was in `supported_sync_modes` in the `AirbyteStream` - Configures which sync mode will be used when data is replicated.
139
-
*`stream` - Hopefully this one looks familiar! This field contains an `AirbyteStream`. It should be _identical_ the one we saw in the `AirbyteCatalog`.
139
+
*`stream` - Hopefully this one looks familiar! This field contains an `AirbyteStream`. It should be _identical_to the one we saw in the `AirbyteCatalog`.
140
140
*`cursor_field` - When `sync_mode` is `INCREMENTAL` and `source_defined_cursor = false`, this field configures which field in the stream will be used to determine if a record should be replicated or not. Read more about this concept in our [documentation of incremental replication](connections/incremental-append.md).
141
141
142
142
### Summary of the Postgres Example
@@ -145,7 +145,7 @@ When thinking about `AirbyteCatalog` and `ConfiguredAirbyteCatalog`, remember th
145
145
146
146
## API Examples
147
147
148
-
The `AirbyteCatalog` offers the flexibility in how to model the data for an API. In the next two example, we will model data from the same API--a stock ticker--in two different ways. In the first, the source will return a single stream called `ticker`, and in the second, the source with return a stream for each stock symbol it is configured to retrieve data for. Each stream's name will be a stock symbol.
148
+
The `AirbyteCatalog` offers the flexibility in how to model the data for an API. In the next two examples, we will model data from the same API--a stock ticker--in two different ways. In the first, the source will return a single stream called `ticker`, and in the second, the source with return a stream for each stock symbol it is configured to retrieve data for. Each stream's name will be a stock symbol.
149
149
150
150
### Static Streams Example
151
151
@@ -251,7 +251,7 @@ This example provides another way of thinking about exposing data in a source. A
251
251
252
252
## Nested Schema Example
253
253
254
-
Often, a data source contains "nested" data. In other words this is data that where each record contains other objects nested inside it. Cases like this are cannot be easily modeled just as tables / columns. This is why Airbyte uses JsonSchema to model the schema of its streams.
254
+
Often, a data source contains "nested" data. In other words this is data where each record contains other objects nested inside it. Cases like this cannot be easily modeled just as tables / columns. This is why Airbyte uses JsonSchema to model the schema of its streams.
255
255
256
256
Let's imagine we are modeling a flight object. A flight object might look like this:
Copy file name to clipboardExpand all lines: docs/understanding-airbyte/catalog.md
+2-2
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,7 @@ This section will document the meaning of each field in an `AirbyteStream`
16
16
17
17
*`json_schema` - This field contains a [JsonSchema](https://json-schema.org/understanding-json-schema) representation of the schema of the stream.
18
18
*`supported_sync_modes` - The sync modes that the stream supports. By default, all sources support `FULL_REFRESH`. Even if this array is empty, it can be assumed that a source supports `FULL_REFRESH`. The allowed sync modes are `FULL_REFRESH` and `INCREMENTAL`.
19
-
*`source_defined_cursor` - If a source supports the `INCREMENTAL` sync mode, and it sets this field to true, it is responsible for determining internally how it tracks which records in a source are new or updated since the last sync It is an array of keys to a field in the schema.
19
+
*`source_defined_cursor` - If a source supports the `INCREMENTAL` sync mode, and it sets this field to true, it is responsible for determining internally how it tracks which records in a source are new or updated since the last sync. It is an array of keys to a field in the schema.
20
20
*`default_cursor_field` - If a source supports the `INCREMENTAL` sync mode, it may, optionally, set this field. If this field is set, and the user does not override it with the `cursor_field` attribute in the `ConfiguredAirbyteStream`\(described below\), this field will be used as the cursor.
21
21
22
22
## ConfiguredAirbyteStream
@@ -26,7 +26,7 @@ This section will document the meaning of each field in an `ConfiguredAirbyteStr
26
26
*`stream` - This field contains the `AirbyteStream` that it is configured.
27
27
*`sync_mode` - The sync mode that will be used to sync that stream. The value in this field MUST be present in the `supported_sync_modes` array for the discovered `AirbyteStream` of this stream.
28
28
*`cursor_field` - This field is an array of keys to a field in the schema that in the `INCREMENTAL` sync mode will be used to determine if a record is new or updated since the last sync.
29
-
* If an `AirbyteStream`defines a `cursor_field`, then the `cursor_field` attribute in `ConfiguredAirbyteStream` will be ignored.
29
+
* If an `AirbyteStream`has `source_defined_cursor` set to `true`, then the `cursor_field` attribute in `ConfiguredAirbyteStream` will be ignored.
30
30
* If an `AirbyteStream` defines a `default_cursor_field`, then the `cursor_field` attribute in `ConfiguredAirbyteStream` is not required, but if it is set, it will override the default value.
31
31
* If an `AirbyteStream` does not define a `cursor_field` or a `default_cursor_field`, then `ConfiguredAirbyteStream` must define a `cursor_field`.
0 commit comments