|
1 | 1 | # Heartbeats
|
2 | 2 |
|
3 |
| -Many transient issues can occur when moving data, especially for long jobs. Often the fix is a simple restart. |
| 3 | +Many transient issues can occur when moving data. One class of issues is an unresponsive Source or Destination. In this case, the fix is often a simple restart. |
4 | 4 |
|
5 |
| -Airbyte aims to make restarts as automated as possible and uses a heartbeating mechanism to do so. This is performed on 2 different components: the source and the destination. |
| 5 | +Airbyte aims to make restarts as automated as possible and uses a heartbeating mechanism to do so. |
6 | 6 |
|
7 |
| -Heartbeat errors are expected to be transient and should automatically resolve. If they do not, it is likely a sign of a more serious issue. |
| 7 | +Airbyte monitors for responses from the Sources and Destination, interpreting these as 'heartbeats'. If the Source or Destination does not heartbeat within |
| 8 | +a certain time frame, Airbyte triggers a heartbeat error and automatically restarts the job. |
8 | 9 |
|
9 |
| -## Known Causes |
| 10 | +Heartbeats are a final catch-all mechanism. Errors are expected to be transient and should automatically resolve. If they do not, it is likely a sign of a more serious issue. |
| 11 | +In these cases, Airbyte takes the more conservative approach. Airbyte restarts the job to avoid a seemingly endless job, and highlight to users the existence of a potential issue. |
10 | 12 |
|
11 |
| -Possible reasons for this issue: |
| 13 | +## Known Heartbeat Error Causes |
| 14 | + |
| 15 | +Possible reasons for a heartbeat error: |
12 | 16 | 1. Certain API sources take an unknown amount of time to generate asynchronous responses (e.g., Salesforce, Facebook, Amplitude). No workaround currently exists.
|
13 | 17 | 2. Certain API sources can be rate-limited for a time period longer than their configured threshold. Although Airbyte tries its best to handle this on a per-connector basis, rate limits are not always predictable.
|
14 | 18 | 3. Database sources can be slow to respond to a query. This can be due to a variety of reasons, including the size of the database, the complexity of the query, and the number of other queries being made to the database at the same time.
|
|
0 commit comments