Skip to content

Source Notion: Migrate to Low-Code #35974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 55 commits into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
970a518
add manifest from builder
ChristoGrab Mar 4, 2024
4456c7d
replace python source with declarative
ChristoGrab Mar 4, 2024
73892b8
implement user transformation
ChristoGrab Mar 4, 2024
087ed69
implement properties transformation for pages/databases streams
ChristoGrab Mar 4, 2024
bb4ab93
add blocks stream error handling
ChristoGrab Mar 4, 2024
8e09068
update CDK version
ChristoGrab Mar 6, 2024
8d42448
add test catalogs
ChristoGrab Mar 6, 2024
5fc31b0
update CDK to 0.68.3
ChristoGrab Mar 7, 2024
53081e9
upgrade CDK to 0.68.4
ChristoGrab Mar 7, 2024
6d44e5a
remove page schema from substream manifests
ChristoGrab Mar 7, 2024
de58e46
uncomment pages incremental logic
ChristoGrab Mar 7, 2024
a690589
implement semi-incremental sync for comments stream
ChristoGrab Mar 8, 2024
c912c83
implement semi-incremental for blocks stream
ChristoGrab Mar 9, 2024
6366b06
fix get_filter_date logic
ChristoGrab Mar 9, 2024
be1b554
remove unused source unit tests
ChristoGrab Mar 10, 2024
6b7cf22
remove schemas from manifest
ChristoGrab Mar 10, 2024
fbdddcf
implement bearer authenticator
ChristoGrab Mar 10, 2024
0e3db77
implement SelectiveAuthenticator
ChristoGrab Mar 11, 2024
b562095
add unit tests for transformations
ChristoGrab Mar 11, 2024
5e210ed
add naive code for custom retriever
ChristoGrab Mar 14, 2024
c9390b2
implement hybrid Blocks stream
ChristoGrab Mar 18, 2024
212b0e0
implement and test custom authenticator
ChristoGrab Mar 18, 2024
f847c7f
remove user transformation component
ChristoGrab Mar 19, 2024
c6c69fd
chore: cleanup
ChristoGrab Mar 19, 2024
8d03eb4
fix keyerror in record_filter
ChristoGrab Mar 19, 2024
9a57897
consolidate tests
ChristoGrab Mar 19, 2024
8a1dd80
use jinja template for multiple authenticators
ChristoGrab Mar 20, 2024
c2571d5
chore: cleanup auth and stream_slicing
ChristoGrab Mar 20, 2024
0f567a2
refactor/cleanup tests
ChristoGrab Mar 20, 2024
97323f2
update changelog, add migration guide draft
ChristoGrab Mar 20, 2024
6585aa7
finish test cleanup
ChristoGrab Mar 21, 2024
9cb8419
update metadata and migration guide
ChristoGrab Mar 21, 2024
4918927
reimplement get_updated_state unit test
ChristoGrab Mar 21, 2024
7d567df
chore: format
ChristoGrab Mar 21, 2024
4507ad2
remove poetry.toml, update metadata message
ChristoGrab Mar 21, 2024
cb8c54b
chore: format
ChristoGrab Mar 21, 2024
de19d5b
add user transformation as custom component
ChristoGrab Mar 28, 2024
d8077f6
implement filter component for data feed streams
ChristoGrab Mar 29, 2024
6138929
format
ChristoGrab Mar 29, 2024
0256964
fix pagination conditions
ChristoGrab Mar 29, 2024
1401272
private methods and fixed cursor comparisons
ChristoGrab Mar 29, 2024
41c63a7
revert pagination logic in manifest
ChristoGrab Mar 29, 2024
b398300
update method invocation, revert pagination changes
ChristoGrab Mar 29, 2024
ce668a5
update release metadata
ChristoGrab Apr 1, 2024
d651e4d
fix semi-incremental cursor
ChristoGrab Apr 1, 2024
1bf3c6c
fix unit tests
ChristoGrab Apr 1, 2024
8247a54
add test for custom close_slice
ChristoGrab Apr 1, 2024
f7bfdb5
update deps, cleanup test files
ChristoGrab Apr 1, 2024
fb19026
update cdk, revert custom incremental and data feed
ChristoGrab Apr 1, 2024
0c773d9
remove unused test
ChristoGrab Apr 1, 2024
02596f4
remove custom filter and revert to data_feed implementation
ChristoGrab Apr 3, 2024
a7ddd44
re-implement and fix data feed filter
ChristoGrab Apr 3, 2024
de80046
update abnormal state comments
ChristoGrab Apr 3, 2024
6992880
Update upgrade deadline
tolik0 Apr 12, 2024
ffc800f
Merge branch 'master' into christo/notion-low-code
tolik0 Apr 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions airbyte-integrations/connectors/source-notion/.coveragerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[run]
omit =
source_notion/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,57 @@
"name": "blocks"
}
}
},
{
"type": "STREAM",
"stream": {
"stream_descriptor": { "name": "comments", "namespace": null },
"stream_state": {
"states": [
{
"partition": {
"id": "9cca331a-1edb-4120-9a86-cf40214fac37",
"parent_slice": {}
},
"cursor": { "last_edited_time": "2099-10-10T13:51:00.000Z" }
},
{
"partition": {
"id": "8fbb79d0-f858-4fe7-9e88-6b4019c72365",
"parent_slice": {}
},
"cursor": { "last_edited_time": "2099-10-10T13:53:00.000Z" }
},
{
"partition": {
"id": "18fe94d2-60ec-4a7e-be86-b43e5c9198af",
"parent_slice": {}
},
"cursor": { "last_edited_time": "2099-10-10T13:47:00.000Z" }
},
{
"partition": {
"id": "249f3796-7e81-47b0-9075-00ed2d06439d",
"parent_slice": {}
},
"cursor": { "last_edited_time": "2099-09-12T20:55:00.000Z" }
},
{
"partition": {
"id": "29299296-ef3f-4aff-aef5-02d651a59be3",
"parent_slice": {}
},
"cursor": { "last_edited_time": "2099-09-12T20:56:00.000Z" }
},
{
"partition": {
"id": "a55d276e-4bc2-4fcc-9fb3-e60b867c86e7",
"parent_slice": {}
},
"cursor": { "last_edited_time": "2099-10-10T13:52:00.000Z" }
}
]
}
}
}
]

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"default_primary_key": [["id"]]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite",
"destination_sync_mode": "append",
"primary_key": [["id"]]
},
{
Expand All @@ -22,7 +22,7 @@
},
"primary_key": [["id"]],
"cursor_field": ["last_edited_time"],
"sync_mode": "incremental",
"sync_mode": "full_refresh",
"destination_sync_mode": "append"
},
{
Expand All @@ -36,7 +36,7 @@
},
"primary_key": [["id"]],
"cursor_field": ["last_edited_time"],
"sync_mode": "incremental",
"sync_mode": "full_refresh",
"destination_sync_mode": "append"
},
{
Expand All @@ -50,7 +50,7 @@
},
"primary_key": [["id"]],
"cursor_field": ["last_edited_time"],
"sync_mode": "incremental",
"sync_mode": "full_refresh",
"destination_sync_mode": "append"
},
{
Expand All @@ -60,11 +60,11 @@
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_primary_key": [["id"]],
"default_cursor_field": ["page_last_edited_time"]
"default_cursor_field": ["last_edited_time"]
},
"primary_key": [["id"]],
"cursor_field": ["page_last_edited_time"],
"sync_mode": "incremental",
"cursor_field": ["last_edited_time"],
"sync_mode": "full_refresh",
"destination_sync_mode": "append"
}
]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,20 @@
"cursor_field": ["last_edited_time"],
"sync_mode": "incremental",
"destination_sync_mode": "append"
},
{
"stream": {
"name": "comments",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_primary_key": [["id"]],
"default_cursor_field": ["last_edited_time"]
},
"primary_key": [["id"]],
"cursor_field": ["last_edited_time"],
"sync_mode": "incremental",
"destination_sync_mode": "append"
}
]
}
16 changes: 14 additions & 2 deletions airbyte-integrations/connectors/source-notion/metadata.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: 6e00b415-b02e-4160-bf02-58176a0ae687
dockerImageTag: 2.2.0
dockerImageTag: 3.0.0
dockerRepository: airbyte/source-notion
documentationUrl: https://docs.airbyte.com/integrations/sources/notion
githubIssueLabel: source-notion
Expand All @@ -30,6 +30,18 @@ data:
releaseStage: generally_available
releases:
breakingChanges:
3.0.0:
message:
The source Notion connector is being migrated from the Python CDK to our declarative low-code CDK.
Due to changes in the handling of state format between these CDKs, this migration constitutes a breaking change
for users syncing the `Comments` stream.
To ensure a smooth migration, please reset your data for this stream upon updating. This will facilitate a fresh first sync.
If you are not syncing the `Comments` stream, you can upgrade without any further action.
For more information, see our migration documentation for source Notion.
upgradeDeadline: "2024-04-29"
scopedImpact:
- scopeType: stream
impactedScopes: ["comments"]
2.0.0:
message:
Version 2.0.0 introduces schema changes to multiple properties shared
Expand All @@ -46,5 +58,5 @@ data:
supportLevel: certified
tags:
- language:python
- cdk:python
- cdk:low-code
metadataSpecVersion: "1.0"
90 changes: 45 additions & 45 deletions airbyte-integrations/connectors/source-notion/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading