Skip to content

Add SubscriptionUsage stream to Orb Source #21951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Feb 7, 2023
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
be918f2
wip
pmossman Jan 25, 2023
d80e395
mvp
pmossman Jan 26, 2023
fe72599
big rework to make incremental possibly work
pmossman Jan 26, 2023
287e1b4
incremental might be working
pmossman Jan 27, 2023
17f91cb
add external_customer_id to subscriptions stream
pmossman Jan 27, 2023
b258b0f
keep state indexed by subscription_id
pmossman Jan 27, 2023
a16b924
add a source defined primary key
pmossman Jan 27, 2023
6d12415
move super call after assignments
pmossman Jan 27, 2023
0ffcf55
start work on integration tests
pmossman Jan 27, 2023
cc9802f
format and more test setup
pmossman Jan 27, 2023
f2fd4e5
tests and cleanup
pmossman Jan 31, 2023
6b7ac9b
fix yield from
pmossman Jan 31, 2023
656e78a
format fixes
pmossman Jan 31, 2023
ab44f5a
fix type of new subscriptions property
pmossman Jan 31, 2023
f00616a
add subscription with actual usage for integration test config
pmossman Feb 2, 2023
aadeae5
fix start date parsing and use correct keys in integration test
pmossman Feb 2, 2023
c6bd835
update docs
pmossman Feb 2, 2023
6688b59
bump version from 0.1.4 to 0.1.5 and update changelog
pmossman Feb 2, 2023
cfee0c7
make start_date a required field and set version 0.2.0 instead of 0.1.5
pmossman Feb 3, 2023
0c5114d
major version bump from 0.1.4 to 1.0.0 to reflect backwards-incompati…
pmossman Feb 3, 2023
e6ec1bb
pass start_date in unit test
pmossman Feb 3, 2023
c587cfd
add sample start_date
pmossman Feb 3, 2023
69778ef
start_date already converted
pmossman Feb 4, 2023
88b0591
add start_date to sample_config
pmossman Feb 4, 2023
e4c7988
remove accidental char
pmossman Feb 7, 2023
e22d6c9
Merge branch 'master' into parker/orb-source
pmossman Feb 7, 2023
a698506
auto-bump connector version
octavia-squidington-iii Feb 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-orb/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY source_orb ./source_orb
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.4
LABEL io.airbyte.version=1.0.0
LABEL io.airbyte.name=airbyte/source-orb
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ connector_image: airbyte/source-orb:dev
tests:
spec:
- spec_path: "source_orb/spec.json"
backward_compatibility_tests_config:
disable_for_version: "0.1.4"
connection:
- config_path: "secrets/config.json"
status: "succeed"
Expand All @@ -23,6 +25,10 @@ tests:
# This points to a specific customer's credit ledger entries in the state,
# and this customer is in the integration test account.
credits_ledger_entries: ["hHQF5BT5jtyj9r7V", "created_at"]

# This points to a specific subscription's usage entries in the state,
# and this subscription is in the integration test account.
subscription_usage: ["FDWRvxuBUiFfZech", "timeframe_start"]
full_refresh:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,10 @@
"hHQF5BT5jtyj9r7V": {
"created_at": "2122-01-01T00:00:00Z"
}
},
"subscription_usage": {
"FDWRvxuBUiFfZech": {
"timeframe_start": "2122-01-01T00:00:00Z"
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,22 @@
},
"sync_mode": "incremental",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "subscription_usage",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["timeframe_start"],
"source_defined_primary_key": [
["subscription_id"],
["billable_metric_id"],
["timeframe_start"]
]
},
"sync_mode": "incremental",
"destination_sync_mode": "overwrite"
}
]
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
{
"api_key": "<sample_api_key>"
"api_key": "<sample_api_key>",
"start_date": "2023-01-25T00:00:00Z"
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,10 @@
"7c507794-7413-4467-8f1d-d3785a6c65ca": {
"created_at": "2022-01-01T00:00:00Z"
}
},
"subscription_usage": {
"someId": {
"timeframe_start": "2022-01-01T00:00:00Z"
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"quantity": {
"type": "number"
},
"timeframe_start": {
"type": "string",
"format": "date-time"
},
"timeframe_end": {
"type": "string",
"format": "date-time"
},
"billable_metric_name": {
"type": "string"
},
"billable_metric_id": {
"type": "string"
},
"subscription_id": {
"type": "string"
}
},
"required": [
"quantity",
"timeframe_start",
"timeframe_end",
"billable_metric_name",
"billable_metric_id",
"subscription_id"
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@
"customer_id": {
"type": "string"
},
"external_customer_id": {
"type": ["null", "string"]
},
"start_date": {
"type": ["null", "string"],
"format": "date-time"
Expand Down
Loading