Skip to content

Commit 2569071

Browse files
authored
Update low-code documentation (#15748)
* newlines * remove dots for consistency * a few more edits * more edits
1 parent a460c17 commit 2569071

File tree

11 files changed

+42
-37
lines changed

11 files changed

+42
-37
lines changed

docs/connector-development/config-based/authentication.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ authenticator:
2727
token: "hello"
2828
```
2929

30-
More information on bearer authentication can be found [here](https://swagger.io/docs/specification/authentication/bearer-authentication/)
30+
More information on bearer authentication can be found [here](https://swagger.io/docs/specification/authentication/bearer-authentication/).
3131

3232
### BasicHttpAuthenticator
3333

docs/connector-development/config-based/error-handling.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ requester:
3636
### From error message
3737
3838
Errors can also be defined by parsing the error message.
39-
For instance, this error handler will ignores responses if the error message contains the string "ignorethisresponse"
39+
For instance, this error handler will ignore responses if the error message contains the string "ignorethisresponse"
4040
4141
```yaml
4242
requester:
@@ -72,8 +72,8 @@ requester:
7272
response_filters:
7373
- http_codes: [ 404 ]
7474
action: IGNORE
75-
- http_codes: [ 429 ]
76-
action: RETRY
75+
- http_codes: [ 429 ]
76+
action: RETRY
7777
```
7878
7979
## Backoff Strategies
@@ -148,8 +148,8 @@ requester:
148148
backoff_strategies:
149149
- type: "WaitTimeFromHeaderBackoffStrategy"
150150
header: "wait_time"
151-
- type: "ConstantBackoffStrategy"
152-
backoff_time_in_seconds: 5
151+
- type: "ConstantBackoffStrategy"
152+
backoff_time_in_seconds: 5
153153
154154
```
155155

docs/connector-development/config-based/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
## From scratch
44

55
- [Overview](overview.md)
6-
- [Yaml structure](overview.md)
6+
- [Yaml structure](yaml-structure.md)
77
- [Reference docs](https://airbyte-cdk.readthedocs.io/en/latest/api/airbyte_cdk.sources.declarative.html)
88

99
## Concepts

docs/connector-development/config-based/overview.md

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -51,12 +51,12 @@ A stream generally corresponds to a resource within the API. They are analogous
5151
A stream is defined by:
5252

5353
1. A name
54-
2. Primary key (Optional): Used to uniquely identify records, enabling deduplication. Can be a string for single primary keys, a list of strings for composite primary keys, or a list of list of strings for composite primary keys consisting of nested fields.
54+
2. Primary key (Optional): Used to uniquely identify records, enabling deduplication. Can be a string for single primary keys, a list of strings for composite primary keys, or a list of list of strings for composite primary keys consisting of nested fields
5555
3. [Schema](../cdk-python/schemas.md): Describes the data to sync
5656
4. [Data retriever](overview.md#data-retriever): Describes how to retrieve the data from the API
5757
5. [Cursor field](../cdk-python/incremental-stream.md) (Optional): Field to use as stream cursor. Can either be a string, or a list of strings if the cursor is a nested field.
5858
6. [Transformations](./record-selector.md#transformations) (Optional): A set of transformations to be applied on the records read from the source before emitting them to the destination
59-
7. [Checkpoint interval](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#state--checkpointing) (Optional): Defines the interval, in number of records, at which incremental syncs should be checkpointed.
59+
7. [Checkpoint interval](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#state--checkpointing) (Optional): Defines the interval, in number of records, at which incremental syncs should be checkpointed
6060

6161
More details on streams and sources can be found in the [basic concepts section](../cdk-python/basic-concepts.md).
6262

@@ -87,9 +87,11 @@ The `SimpleRetriever`'s data flow can be described as follows:
8787
2. Select the records from the response
8888
3. Repeat for as long as the paginator points to a next page
8989

90-
More details on the record selector can be found in the [record selector section](record-selector.md)
91-
More details on the stream slicers can be found in the [stream slicers section](stream-slicers.md)
92-
More details on the paginator can be found in the [pagination section](pagination.md)
90+
More details on the record selector can be found in the [record selector section](record-selector.md).
91+
92+
More details on the stream slicers can be found in the [stream slicers section](stream-slicers.md).
93+
94+
More details on the paginator can be found in the [pagination section](pagination.md).
9395

9496
## Requester
9597

@@ -104,7 +106,8 @@ There is currently only one implementation, the `HttpRequester`, which is define
104106
6. An error handler: Defines how to handle errors
105107

106108
More details on authentication can be found in the [authentication section](authentication.md).
107-
More details on error handling can be found in the [error handling section](error-handling.md)
109+
110+
More details on error handling can be found in the [error handling section](error-handling.md).
108111

109112
## Connection Checker
110113

docs/connector-development/config-based/pagination.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -100,9 +100,10 @@ paginator:
100100
```
101101

102102
Assuming the endpoint to fetch data from is `https://cloud.airbyte.com/api/get_data`,
103-
the first request will be sent as `https://cloud.airbyte.com/api/get_data`
103+
the first request will be sent as `https://cloud.airbyte.com/api/get_data`.
104+
104105
Assuming the id of the last record fetched is 1000,
105-
the next request will be sent as `https://cloud.airbyte.com/api/get_data?from=1000`
106+
the next request will be sent as `https://cloud.airbyte.com/api/get_data?from=1000`.
106107

107108
#### Cursor paginator in path
108109

@@ -121,5 +122,6 @@ paginator:
121122

122123
Assuming the endpoint to fetch data from is `https://cloud.airbyte.com/api/get_data`,
123124
the first request will be sent as `https://cloud.airbyte.com/api/get_data`
125+
124126
Assuming the response's next url is `https://cloud.airbyte.com/api/get_data?page=1&page_size=100`,
125127
the next request will be sent as `https://cloud.airbyte.com/api/get_data?page=1&page_size=100`

docs/connector-development/config-based/request-options.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ requester:
4040
It is also possible for authenticators to set request parameters or headers as needed.
4141
For instance, the `BearerAuthenticator` will always set the `Authorization` header.
4242

43-
More details on the various authenticators can be found in the [authentication section](authentication.md)
43+
More details on the various authenticators can be found in the [authentication section](authentication.md).
4444

4545
## Paginators
4646

@@ -63,7 +63,7 @@ paginator:
6363
field_name: "page"
6464
```
6565

66-
More details on paginators can be found in the [pagination section](pagination.md)
66+
More details on paginators can be found in the [pagination section](pagination.md).
6767

6868
## Stream slicers
6969

@@ -85,4 +85,4 @@ stream_slicer:
8585
inject_into: "request_parameter"
8686
```
8787

88-
More details on the stream slicers can be found in the [stream-slicers section](stream-slicers.md)
88+
More details on the stream slicers can be found in the [stream-slicers section](stream-slicers.md).

docs/connector-development/config-based/stream-slicers.md

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ When a stream is read incrementally, a state message will be output by the conne
99
At the beginning of a `read` operation, the `StreamSlicer` will compute the slices to sync given the connection config and the stream's current state,
1010
As the `Retriever` reads data from the `Source`, the `StreamSlicer` keeps track of the `Stream`'s state, which will be emitted after reading each stream slice.
1111

12-
More information of stream slicing can be found in the [stream-slices section](../cdk-python/stream-slices.md)
12+
More information of stream slicing can be found in the [stream-slices section](../cdk-python/stream-slices.md).
1313

1414
## Implementations
1515

@@ -56,10 +56,9 @@ If the `cursor_field` is `created`, and the record is `{"id": 1234, "created": "
5656

5757
When reading data from the source, the cursor value will be updated to the max datetime between
5858

59-
- the last record's cursor field
60-
- the start of the stream slice
61-
- the current cursor value
62-
This ensures that the cursor will be updated even if a stream slice does not contain any data.
59+
- The last record's cursor field
60+
- The start of the stream slice
61+
- The current cursor value. This ensures that the cursor will be updated even if a stream slice does not contain any data
6362

6463
#### Stream slicer on dates
6564

@@ -164,7 +163,7 @@ retriever:
164163
stream_slice_field: "repository"
165164
```
166165
167-
[^1] This is a slight oversimplification. See update cursor section for more details on how the cursor is updated
166+
[^1] This is a slight oversimplification. See [update cursor section](#cursor-update) for more details on how the cursor is updated.
168167
169168
## More readings
170169

docs/connector-development/config-based/tutorial/0-getting-started.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Throughout this tutorial, we'll walk you through the creation an Airbyte source
88

99
We'll build a connector reading data from the Exchange Rates API, but the steps will apply to other HTTP APIs you might be interested in integrating with.
1010

11-
The API documentations can be found [here](https://exchangeratesapi.io/documentation/).
11+
The API documentations can be found [here](https://apilayer.com/marketplace/exchangerates_data-api).
1212
In this tutorial, we will read data from the following endpoints:
1313

1414
- `Latest Rates Endpoint`

docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -11,13 +11,13 @@ Over the course of this tutorial, we'll be editing a few files that were generat
1111

1212
We'll also be creating the following files:
1313

14-
- `source-exchange-rates-tutorial/secrets/config.json`: This is the configuration file we'll be using to test the connector. It's schema should match the schema defined in the spec file.
15-
- `source-exchange-rates-tutorial/secrets/invalid_config.json`: This is an invalid configuration file we'll be using to test the connector. It's schema should match the schema defined in the spec file.
14+
- `source-exchange-rates-tutorial/secrets/config.json`: This is the configuration file we'll be using to test the connector. Its schema should match the schema defined in the spec file.
15+
- `source-exchange-rates-tutorial/secrets/invalid_config.json`: This is an invalid configuration file we'll be using to test the connector. Its schema should match the schema defined in the spec file.
1616
- `source_exchange_rates_tutorial/schemas/rates.json`: This is the [schema definition](../../cdk-python/schemas.md) for the stream we'll implement.
1717

1818
## Updating the connector spec and config
1919

20-
Let's populate the specification (`spec.yaml`) the configuration (`secrets/config.json), so the connector can access the access key and base currency.
20+
Let's populate the specification (`spec.yaml`) and the configuration (`secrets/config.json`) so the connector can access the access key and base currency.
2121

2222
1. We'll add these properties to the connector spec in `source-exchange-rates-tutorial/source_exchange_rates_tutorial/spec.yaml`
2323

@@ -61,9 +61,9 @@ $ echo '{"access_key": "<your_access_key>", "base": "USD"}' > secrets/config.js
6161
Next, we'll update the connector definition (`source-exchange-rates-tutorial/source_exchange_rates_tutorial/exchange_rates_tutorial.yaml`). It was generated by the code generation script.
6262
More details on the connector definition file can be found in the [overview](../overview.md) and [connection definition](../yaml-structure.md) sections.
6363

64-
Let's fill this out these TODOs with the information found in the [Exchange Rates API docs](https://exchangeratesapi.io/documentation/)
64+
Let's fill this out these TODOs with the information found in the [Exchange Rates API docs](https://apilayer.com/marketplace/exchangerates_data-api).
6565

66-
1. First, let's rename the stream from `customers` to `rates`, and update the primary key to `date`
66+
1. First, let's rename the stream from `customers` to `rates`, and update the primary key to `date`.
6767

6868
```yaml
6969
streams:
@@ -84,7 +84,7 @@ check:
8484
Adding the reference in the `check` tells the `check` operation to use that stream to test the connection.
8585

8686
2. Next we'll set the base url.
87-
According to the API documentation, the base url is `"https://api.exchangeratesapi.io/v1/"`.
87+
According to the API documentation, the base url is `"https://api.apilayer.com"`.
8888

8989
```yaml
9090
definitions:
@@ -141,7 +141,7 @@ definitions:
141141
base: "{{ config['base'] }}"
142142
```
143143

144-
The full connection definition should now look like
144+
The full connector definition should now look like
145145

146146
```yaml
147147
version: "0.1.0"

docs/connector-development/config-based/tutorial/5-incremental-reads.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,10 +87,11 @@ For example:
8787
> "historical": true, "base": "USD", "date": "2022-07-18"
8888

8989
The connector will now always read data for the start date, which is not exactly what we want.
90-
Instead, we would like to iterate over all the dates between the start_date and today and read data for each day.
90+
Instead, we would like to iterate over all the dates between the `start_date` and today and read data for each day.
9191

9292
We can do this by adding a `DatetimeStreamSlicer` to the connector definition, and update the `path` to point to the stream_slice's `start_date`:
93-
More details on the stream slicers can be found [here](../stream-slicers.md) <FIXME: need to fix links>
93+
94+
More details on the stream slicers can be found [here](../stream-slicers.md).
9495

9596
Let's first define a stream slicer at the top level of the connector definition:
9697

docs/connector-development/config-based/yaml-structure.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ In this example, outer.inner.k2 will evaluate to "MyKey is MyValue"
135135
## References
136136
137137
Strings can contain references to previously defined values.
138-
The parser will dereference these values to produce a complete ConnectionDefinition
138+
The parser will dereference these values to produce a complete object definition.
139139
140140
References can be defined using a "*ref({arg})" string.
141141
@@ -230,7 +230,7 @@ nested.path: "uh oh"
230230
value: "uh oh"
231231
```
232232

233-
To resolve the ambiguity, we try looking for the reference key at the top level, and then traverse the structs downward
233+
To resolve the ambiguity, we try looking for the reference key at the top-level, and then traverse the structs downward
234234
until we find a key with the given path, or until there is nothing to traverse.
235235

236236
More details on referencing values can be found [here](https://airbyte-cdk.readthedocs.io/en/latest/api/airbyte_cdk.sources.declarative.parsers.html?highlight=yamlparser#airbyte_cdk.sources.declarative.parsers.yaml_parser.YamlParser).
@@ -264,7 +264,7 @@ This means that both these string templates will evaluate to the same string:
264264
1. `"{{ options.name }}"`
265265
2. `"{{ options['name'] }}"`
266266

267-
In additional to passing additional values through the kwargs argument, macros can be called from within the string interpolation.
267+
In additional to passing additional values through the $options argument, macros can be called from within the string interpolation.
268268
For example,
269269
`"{{ max(2, 3) }}" -> 3`
270270

0 commit comments

Comments
 (0)