Skip to content

Commit c8d06f4

Browse files
authored
Docs: create overview architecture diagram + reorder docs (#33896)
1 parent 3601ac6 commit c8d06f4

File tree

2 files changed

+38
-19
lines changed

2 files changed

+38
-19
lines changed

docs/understanding-airbyte/high-level-view.md

+32-13
Original file line numberDiff line numberDiff line change
@@ -11,16 +11,35 @@ The platform provides all the horizontal services required to configure and run
1111
Connectors are independent modules which push/pull data to/from sources and destinations. Connectors are built in accordance with the [Airbyte Specification](./airbyte-protocol.md), which describes the interface with which data can be moved between a source and a destination using Airbyte. Connectors are packaged as Docker images, which allows total flexibility over the technologies used to implement them.
1212

1313
A more concrete diagram can be seen below:
14-
15-
![3.048-Kilometer view](../.gitbook/assets/understanding_airbyte_high_level_architecture.png)
16-
17-
* `UI`: An easy-to-use graphical interface for interacting with the Airbyte API.
18-
* `WebApp Server`: Handles connection between UI and API.
19-
* `Config Store`: Stores all the connections information \(credentials, frequency...\).
20-
* `Scheduler Store`: Stores statuses and job information for the scheduler bookkeeping.
21-
* `Config API`: Airbyte's main control plane. All operations in Airbyte such as creating sources, destinations, connections, managing configurations, etc.. are configured and invoked from the API.
22-
* `Scheduler`: The scheduler takes work requests from the API and sends them to the Temporal service to parallelize. It is responsible for tracking success/failure and for triggering syncs based on the configured frequency.
23-
* `Temporal Service`: Manages the task queue and workflows for the Scheduler.
24-
* `Worker`: The worker connects to a source connector, pulls the data and writes it to a destination.
25-
* `Temporary Storage`: A storage that workers can use whenever they need to spill data on a disk.
26-
14+
```mermaid
15+
---
16+
title: Architecture Overview
17+
config:
18+
theme: neutral
19+
---
20+
flowchart LR
21+
W[fa:fa-display WebApp/UI]
22+
S[fa:fa-server Server/Config API]
23+
D[(fa:fa-table Config & Jobs)]
24+
T(fa:fa-calendar Temporal)
25+
W2[1..n Airbyte Workers]
26+
W -->|sends API requests| S
27+
S -->|store data| D
28+
S -->|create workflow| T
29+
T -->|launch task| W2
30+
W2 -->|return job| T
31+
W2 -->|launches| Source
32+
W2 -->|launches| Destination
33+
```
34+
35+
* **Web App/UI** [`airbyte-webapp`, `airbyte-proxy`]: An easy-to-use graphical interface for interacting with the Airbyte API.
36+
* **Server/Config API** [`airbyte-server`, `airbyte-server-api`]: Handles connection between UI and API. Airbyte's main control plane. All operations in Airbyte such as creating sources, destinations, connections, managing configurations, etc.. are configured and invoked from the API.
37+
* **Database Config & Jobs** [`airbyte-db`]: Stores all the connections information \(credentials, frequency...\).
38+
* **Temporal Service** [`airbyte-temporal`]: Manages the task queue and workflows.
39+
* **Worker** [`airbyte-worker`]: The worker connects to a source connector, pulls the data and writes it to a destination.
40+
41+
The diagram shows the steady-state operation of Airbyte, there are components not described you'll see in your deployment:
42+
* **Cron** [`airbyte-cron`]: Clean the server and sync logs (when using local logs)
43+
* **Bootloader** [`airbyte-bootloader`]: Upgrade and Migrate the Database tables and confirm the enviroment is ready to work.
44+
45+
This is a holistic high-level description of each component. For Airbyte deployed in Kubernetes the structure is very similar with a few changes.

docusaurus/sidebars.js

+6-6
Original file line numberDiff line numberDiff line change
@@ -381,18 +381,18 @@ const understandingAirbyte = {
381381
type: "category",
382382
label: "Understand Airbyte",
383383
items: [
384-
"understanding-airbyte/beginners-guide-to-catalog",
384+
"understanding-airbyte/high-level-view",
385385
"understanding-airbyte/airbyte-protocol",
386386
"understanding-airbyte/airbyte-protocol-docker",
387-
"understanding-airbyte/operations",
388-
"understanding-airbyte/high-level-view",
389387
"understanding-airbyte/jobs",
390-
"understanding-airbyte/tech-stack",
391-
"understanding-airbyte/cdc",
388+
"understanding-airbyte/database-data-catalog",
389+
"understanding-airbyte/beginners-guide-to-catalog",
392390
"understanding-airbyte/supported-data-types",
391+
"understanding-airbyte/operations",
392+
"understanding-airbyte/cdc",
393393
"understanding-airbyte/json-avro-conversion",
394-
"understanding-airbyte/database-data-catalog",
395394
"understanding-airbyte/schemaless-sources-and-destinations",
395+
"understanding-airbyte/tech-stack",
396396
],
397397
};
398398

0 commit comments

Comments
 (0)