Skip to content

Commit 4e7ba06

Browse files
girardaAndybrianjlai
authored
Document guidelines on low-code support (#16387)
* Guide for using lowcode or not * Use headers * Update docs/connector-development/config-based/overview.md Co-authored-by: Andy <[email protected]> * Update docs/connector-development/config-based/overview.md Co-authored-by: Andy <[email protected]> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <[email protected]> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <[email protected]> * an -> a Co-authored-by: Andy <[email protected]> Co-authored-by: Brian Lai <[email protected]>
1 parent 1d9608c commit 4e7ba06

File tree

1 file changed

+70
-14
lines changed

1 file changed

+70
-14
lines changed

‎docs/connector-development/config-based/overview.md

Lines changed: 70 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -15,22 +15,78 @@ The process then submits HTTP requests to the API endpoint, and extracts records
1515

1616
See the [connector definition section](yaml-structure.md) for more information on the YAML file describing the connector.
1717

18+
## Does this framework support the connector I want to build?
19+
20+
Not all APIs are can be built using this framework because its featureset is still limited.
21+
This section describes guidelines for determining whether a connector for a given API can be built using the config-based framework. Please let us know through the #lowcode-earlyaccess Slack channel if you'd like to build something that falls outside what we currently support and we'd be happy to discuss and prioritize in the coming months!
22+
23+
Refer to the API's documentation to answer the following questions:
24+
25+
### Is this a HTTP REST API returning data as JSON?
26+
27+
The API documentation should show which HTTP method must be used to retrieve data from the API.
28+
For example, the [documentation for the Exchange Rates Data API](https://apilayer.com/marketplace/exchangerates_data-api#documentation-tab) says the GET method should be used, and that the response is a JSON object.
29+
30+
Other API types such as SOAP or GraphQL are not supported.
31+
32+
Other encoding schemes such as CSV or Protobuf are not supported.
33+
34+
Integrations that require the use of an SDK are not supported.
35+
36+
### Do queries return the data synchronously or do they trigger a bulk workflow?
37+
38+
Some APIs return the data of interest as part of the response. This is the case for the [Exchange Rates Data API](https://apilayer.com/marketplace/exchangerates_data-api#documentation-tab) - each request results in a response containing the data we're interested in.
39+
40+
Other APIs use bulk workflows, which means a query will trigger an asynchronous process on the integration's side. [Zendesk bulk queries](https://developer.zendesk.com/api-reference/ticketing/tickets/tickets/#bulk-mark-tickets-as-spam) are an example of such integrations.
41+
42+
An initial request will trigger the workflow and return an ID and a job status. The actual data then needs to be fetched when the asynchronous job is completed.
43+
44+
Asynchronous bulk workflows are not supported.
45+
46+
### What is the pagination mechanism?
47+
48+
The only pagination mechanisms supported are
49+
50+
* Offset count passed either by query params or request header such as [Sendgrid](https://docs.sendgrid.com/api-reference/bounces-api/retrieve-all-bounces)
51+
* Page count passed either by query params or request header such as [Greenhouse](https://developers.greenhouse.io/harvest.html#get-list-applications)
52+
* Cursor field pointing to the URL of the next page of records such as [Sentry](https://docs.sentry.io/api/pagination/)
53+
54+
### What is the authorization mechanism?
55+
56+
Endpoints that require authenticating using a query param or a HTTP header, as is the case for the [Exchange Rates Data API](https://apilayer.com/marketplace/exchangerates_data-api#authentication), are supported.
57+
58+
Endpoints that require authenticating using Basic Auth over HTTPS, as is the case for [Greenhouse](https://developers.greenhouse.io/harvest.html#authentication), are supported.
59+
60+
Endpoints that require authenticating using OAuth 2.0, as is the case for [Strava](https://developers.strava.com/docs/authentication/#introduction), are supported.
61+
62+
Other authentication schemes such as GWT are not supported.
63+
64+
### Is the schema static or dynamic?
65+
66+
Only static schemas are supported.
67+
68+
Dynamically deriving the schema from querying an endpoint is not supported.
69+
70+
### Does the endpoint have a strict rate limit
71+
72+
Throttling is not supported, but the connector can use exponential backoff to avoid API bans in case it gets rate limited. This can work for APIs with high rate limits, but not for those that have strict limits on a small time-window, such as the [Reddit Ads API](https://ads-api.reddit.com/docs/#section/Rate-Limits), which limits to 1 request per second.
73+
1874
## Supported features
1975

20-
| Feature | Support |
21-
|--------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
22-
| Transport protocol | HTTP |
23-
| HTTP methods | GET, POST |
24-
| Data format | Json |
25-
| Resource type | Collections<br/>Sub-collection |
26-
| [Pagination](./pagination.md) | [Page limit](./pagination.md#page-increment)<br/>[Offset](./pagination.md#offset-increment)<br/>[Cursor](./pagination.md#cursor) |
27-
| [Authentication](./authentication.md) | [Header based](./authentication.md#ApiKeyAuthenticator)<br/>[Bearer](./authentication.md#BearerAuthenticator)<br/>[Basic](./authentication.md#BasicHttpAuthenticator)<br/>[OAuth](./authentication.md#OAuth) |
28-
| Sync mode | Full refresh<br/>Incremental |
29-
| Schema discovery | Only static schemas |
30-
| [Stream slicing](./stream-slicers.md) | [Datetime](./stream-slicers.md#Datetime), [lists](./stream-slicers.md#list-stream-slicer), [parent-resource id](./stream-slicers.md#Substream-slicer) |
76+
| Feature | Support |
77+
|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
78+
| Transport protocol | HTTP |
79+
| HTTP methods | GET, POST |
80+
| Data format | JSON |
81+
| Resource type | Collections<br/>Sub-collection |
82+
| [Pagination](./pagination.md) | [Page limit](./pagination.md#page-increment)<br/>[Offset](./pagination.md#offset-increment)<br/>[Cursor](./pagination.md#cursor) |
83+
| [Authentication](./authentication.md) | [Header based](./authentication.md#ApiKeyAuthenticator)<br/>[Bearer](./authentication.md#BearerAuthenticator)<br/>[Basic](./authentication.md#BasicHttpAuthenticator)<br/>[OAuth](./authentication.md#OAuth) |
84+
| Sync mode | Full refresh<br/>Incremental |
85+
| Schema discovery | Only static schemas |
86+
| [Stream slicing](./stream-slicers.md) | [Datetime](./stream-slicers.md#Datetime), [lists](./stream-slicers.md#list-stream-slicer), [parent-resource id](./stream-slicers.md#Substream-slicer) |
3187
| [Record transformation](./record-selector.md) | [Field selection](./record-selector.md#selecting-a-field)<br/>[Adding fields](./record-selector.md#adding-fields)<br/>[Removing fields](./record-selector.md#removing-fields)<br/>[Filtering records](./record-selector.md#filtering-records) |
32-
| [Error detection](./error-handling.md) | [From HTTP status code](./error-handling.md#from-status-code)<br/>[From error message](./error-handling.md#from-error-message) |
33-
| [Backoff strategies](./error-handling.md#Backoff-Strategies) | [Exponential](./error-handling.md#Exponential-backoff)<br/>[Constant](./error-handling.md#Constant-Backoff)<br/>[Derived from headers](./error-handling.md#Wait-time-defined-in-header) |
88+
| [Error detection](./error-handling.md) | [From HTTP status code](./error-handling.md#from-status-code)<br/>[From error message](./error-handling.md#from-error-message) |
89+
| [Backoff strategies](./error-handling.md#Backoff-Strategies) | [Exponential](./error-handling.md#Exponential-backoff)<br/>[Constant](./error-handling.md#Constant-Backoff)<br/>[Derived from headers](./error-handling.md#Wait-time-defined-in-header) |
3490

3591
If a feature you require is not supported, you can [request the feature](../../contributing-to-airbyte/README.md#requesting-new-features) and use the [Python CDK](../cdk-python/README.md).
3692

@@ -67,7 +123,7 @@ There is currently only one implementation, the `SimpleRetriever`, which is defi
67123

68124
1. Requester: Describes how to submit requests to the API source
69125
2. Paginator: Describes how to navigate through the API's pages
70-
3. Record selector: Describes how to extract records from an HTTP response
126+
3. Record selector: Describes how to extract records from a HTTP response
71127
4. Stream Slicer: Describes how to partition the stream, enabling incremental syncs and checkpointing
72128

73129
Each of those components (and their subcomponents) are defined by an explicit interface and one or many implementations.

0 commit comments

Comments
 (0)