Skip to content

✨ Source Harvest: Migrate to Low Code #35863

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 61 commits into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
ef1099d
delete expected records
pnilan Feb 21, 2024
2af2052
added records
pnilan Feb 22, 2024
329df04
Merge branch 'master' of github.com:airbytehq/airbyte
pnilan Feb 22, 2024
6961735
Remove spec.json, streams.py, schemas/, add manifest. Replace source.…
pnilan Feb 23, 2024
1fff83e
Merge branch 'master' into pnilan/source-harvest-low-code-migration
pnilan Mar 6, 2024
6a5cd5f
Removes availability strategy
pnilan Mar 6, 2024
69d1973
Merge conflict resoution
pnilan Mar 6, 2024
ab9787b
Updated for incremental substream fix
pnilan Mar 6, 2024
5caa451
updates manifest parent id
pnilan Mar 6, 2024
dac3d51
Updates CDK to 0.68.4
pnilan Mar 7, 2024
369d3de
Merge branch 'master' into pnilan/source-harvest-low-code-migration
pnilan Mar 7, 2024
db634ed
Updated report streams to use proper datetime format
pnilan Mar 7, 2024
8071da0
Updates report request range to 1 year
pnilan Mar 7, 2024
ce2dbfa
Updates substreams to include parent_id field
pnilan Mar 8, 2024
241dd02
chore: format code
pnilan Mar 8, 2024
cb5ba00
Update configured catalog, acceptance-test_config config refs, aand s…
pnilan Mar 8, 2024
85a0c51
Updated tag to low-code
pnilan Mar 8, 2024
7c93e4f
Deleted unneeded unit tests, modified config builder for updated conf…
pnilan Mar 8, 2024
4fb0207
Updated for multiple authenticators
pnilan Mar 8, 2024
b820dd1
Merge branch 'master' into pnilan/source-harvest-low-code-migration
pnilan Mar 8, 2024
7a89883
Bumps major version number and updates changelog, harvest-migrations,…
pnilan Mar 8, 2024
ff17ecd
Add breakingChange release to metadata
pnilan Mar 8, 2024
3f21b15
Updates config names, skips spec check, adds schemas and spec back, a…
pnilan Mar 9, 2024
8ae909f
chore: format code
pnilan Mar 9, 2024
e5cc020
Updates ConfigBuilder for new format, updates invoice integration tes…
pnilan Mar 10, 2024
38e5174
Remove conftest and test_source as tests are no longer relevant for l…
pnilan Mar 10, 2024
9a50ce8
Revert chargebee expected records
pnilan Mar 10, 2024
074d11e
Add `per_page` query params
pnilan Mar 10, 2024
a35d1fe
Update http_mocker requests to include `per_page` query params
pnilan Mar 10, 2024
5f95b6c
chore: format code
pnilan Mar 11, 2024
e25e822
Update config to more closely align with old format
pnilan Mar 12, 2024
ee8806f
Update sample/invalid config files
pnilan Mar 12, 2024
1afc971
Remove spec.yaml, return to manifest.yaml
pnilan Mar 12, 2024
79843dd
Merge branch 'master' into pnilan/source-harvest-low-code-migration
pnilan Mar 20, 2024
81ea2af
Update CDK to latest
pnilan Mar 20, 2024
b95dd9b
Updates integration tests to reflect updated credentials object in co…
pnilan Mar 20, 2024
337b789
Adds test files back, updates manifest to include 401, 403, 404 error…
pnilan Mar 21, 2024
20ad600
Update source tests
pnilan Mar 22, 2024
99472b5
Updates unit tests and adds freezegun as dependency
pnilan Mar 25, 2024
c1f0137
Removes errant imports
pnilan Mar 25, 2024
c4a9e06
chore: format code
pnilan Mar 25, 2024
317f4ae
Update sync_mode for stream slice test
pnilan Mar 25, 2024
7205e59
Add `config_migrations` to add `auth_type` to config for backwards co…
pnilan Mar 25, 2024
aa93e8e
chore: format code
pnilan Mar 25, 2024
d5dc5b7
Adds unit tests for config_migrations.py
pnilan Mar 25, 2024
c44bf73
Chore: format code
pnilan Mar 25, 2024
1b6c57d
Moves `freezegun` to dev dependency
pnilan Mar 25, 2024
9db4699
Updates `harvest-migrations.md`, `harvest.md`, and `metadata.yaml` to…
pnilan Mar 25, 2024
41185be
Updates to latest cdk with updated record counts datatype
pnilan Mar 27, 2024
3d86541
Update migration steps language
pnilan Apr 1, 2024
13f2e1c
Merge branch 'master' into pnilan/source-harvest-low-code-migration
pnilan Apr 1, 2024
f84f11c
Merge branch 'pnilan/source-harvest-low-code-migration' of github.com…
pnilan Apr 1, 2024
09d0c9f
Merge branch 'master' into pnilan/source-harvest-low-code-migration
pnilan Apr 9, 2024
80ce449
Resolve merge conflict w/ poetry.lock
pnilan Apr 9, 2024
d91f274
Updates state format for incremental substreams in abnormal_state.json
pnilan Apr 9, 2024
4f7f57f
chore: format code
pnilan Apr 9, 2024
46063c9
Fixes configured catalog error
pnilan Apr 10, 2024
3aafa84
chore: format code
pnilan Apr 10, 2024
a213d06
Updates date in changelog and metadata
pnilan Apr 10, 2024
efa7e18
Remove spec bypass
pnilan Apr 10, 2024
a08499b
Updates changelog wording
pnilan Apr 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -3,34 +3,32 @@ test_strictness_level: "high"
acceptance_tests:
spec:
tests:
- spec_path: "source_harvest/spec.json"
- spec_path: "source_harvest/spec.yaml"
backward_compatibility_tests_config:
disable_for_version: "1.0.0"
connection:
tests:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "secrets/old_config.json"
status: "succeed"
- config_path: "secrets/config_oauth.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"
discovery:
tests:
- config_path: "secrets/config.json"
- config_path: "secrets/config_oauth.json"
basic_read:
tests:
- config_path: "secrets/config.json"
- config_path: "secrets/config_oauth.json"
expect_records:
path: "integration_tests/expected_records.jsonl"
fail_on_extra_columns: false
incremental:
tests:
- config_path: "secrets/config_with_date_range.json"
- config_path: "secrets/config_oauth.json"
configured_catalog_path: "integration_tests/incremental_catalog.json"
future_state:
future_state_path: "integration_tests/abnormal_state.json"
timeout_seconds: 2400
full_refresh:
tests:
- config_path: "secrets/config.json"
- config_path: "secrets/config_oauth.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,6 @@
"stream_descriptor": { "name": "invoices" }
}
},
{
"type": "STREAM",
"stream": {
"stream_state": { "updated_at": "2217-06-26T21:20:07Z" },
"stream_descriptor": { "name": "invoice_messages" }
}
},
{
"type": "STREAM",
"stream": {
"stream_state": { "updated_at": "2217-06-26T21:20:07Z" },
"stream_descriptor": { "name": "invoice_payments" }
}
},
{
"type": "STREAM",
"stream": {
Expand All @@ -48,13 +34,6 @@
"stream_descriptor": { "name": "estimates" }
}
},
{
"type": "STREAM",
"stream": {
"stream_state": { "updated_at": "2217-06-26T21:20:07Z" },
"stream_descriptor": { "name": "estimate_messages" }
}
},
{
"type": "STREAM",
"stream": {
Expand Down Expand Up @@ -125,13 +104,6 @@
"stream_descriptor": { "name": "users" }
}
},
{
"type": "STREAM",
"stream": {
"stream_state": { "updated_at": "2217-06-26T21:20:07Z" },
"stream_descriptor": { "name": "project_assignments" }
}
},
{
"type": "STREAM",
"stream": {
Expand Down Expand Up @@ -194,5 +166,81 @@
"stream_state": { "to": "22170626" },
"stream_descriptor": { "name": "time_team" }
}
},
{
"type": "STREAM",
"stream": {
"stream_state": {
"states": [
{
"partition": { "id": 2695071, "parent_slice": {} },
"cursor": { "updated_at": "2199-05-27T18:12:42Z" }
}
]
},
"stream_descriptor": { "name": "estimate_messages" }
}
},
{
"type": "STREAM",
"stream": {
"stream_state": {
"states": [
{
"partition": { "id": 3758384, "parent_slice": {} },
"cursor": { "updated_at": "2199-05-05T08:19:35Z" }
},
{
"partition": { "id": 3758383, "parent_slice": {} },
"cursor": { "updated_at": "2199-05-05T08:19:35Z" }
},
{
"partition": { "id": 3758382, "parent_slice": {} },
"cursor": { "updated_at": "2199-05-05T08:19:35Z" }
},
{
"partition": { "id": 3758381, "parent_slice": {} },
"cursor": { "updated_at": "2199-05-05T08:19:35Z" }
},
{
"partition": { "id": 3758380, "parent_slice": {} },
"cursor": { "updated_at": "2199-05-05T12:52:20Z" }
}
]
},
"stream_descriptor": { "name": "project_assignments" }
}
},
{
"type": "STREAM",
"stream": {
"stream_state": {
"states": [
{
"partition": { "id": 28174531, "parent_slice": {} },
"cursor": { "updated_at": "2199-05-25T16:46:28Z" }
}
]
},
"stream_descriptor": { "name": "invoice_messages" }
}
},
{
"type": "STREAM",
"stream": {
"stream_state": {
"states": [
{
"partition": { "id": 28174545, "parent_slice": {} },
"cursor": { "updated_at": "2199-05-26T09:07:06Z" }
},
{
"partition": { "id": 28174531, "parent_slice": {} },
"cursor": { "updated_at": "2199-05-26T09:06:37Z" }
}
]
},
"stream_descriptor": { "name": "invoice_payments" }
}
}
]
Loading
Loading