Skip to content

🎉 Amazon SP extra endpoint support #5248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 54 commits into from
Sep 17, 2021
Merged

🎉 Amazon SP extra endpoint support #5248

Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
edb86b4
Add GET_FBA_INVENTORY_AGED_DATA data
htrueman Jul 16, 2021
31f3d32
Add GET_MERCHANT_LISTINGS_ALL_DATA stream support
htrueman Jul 19, 2021
9777d16
Update schemas
htrueman Jul 20, 2021
53524e7
Update configured_catalog.json
htrueman Jul 20, 2021
539dd07
Update connector to airbyte-cdk
htrueman Jul 22, 2021
59af900
Add amazon seller partner test creds
htrueman Jul 22, 2021
b726bca
Update state sample files
htrueman Jul 22, 2021
4780a80
Apply code format
htrueman Jul 22, 2021
ee9b91f
Merge remote-tracking branch 'origin/master' into htrueman/source-ama…
htrueman Jul 23, 2021
0eca787
Update acceptance-test-config.yml
htrueman Jul 23, 2021
26ff528
Add dummy integration test
htrueman Jul 23, 2021
2b08caa
Refactor auth signature.
htrueman Jul 29, 2021
17cbbd0
Merge remote-tracking branch 'origin/master' into htrueman/source-ama…
htrueman Jul 29, 2021
ea341a2
Remove print_function import from auth.py
htrueman Jul 30, 2021
a87d7c5
Refactor source class.
htrueman Jul 30, 2021
de789a7
Add dummy integration test
htrueman Jul 30, 2021
fc1ccd4
Merge remote-tracking branch 'origin/master' into htrueman/source-ama…
htrueman Jul 30, 2021
1a7d817
Typing added.
htrueman Aug 1, 2021
de328df
Add extra streams and schemas
htrueman Aug 3, 2021
359c8f4
Update docs and spec
htrueman Aug 6, 2021
6ef8446
Merge remote-tracking branch 'origin/master' into htrueman/source-ama…
htrueman Aug 6, 2021
900fcb8
Post merge code fixes
htrueman Aug 6, 2021
aa0a5c8
Merge branch 'htrueman/source-amazon-sp-expand-endpoint-support' into…
htrueman Aug 6, 2021
62bc962
Merge remote-tracking branch 'origin/master' into htrueman/source-ama…
htrueman Aug 6, 2021
e831b72
Fix test setup
htrueman Aug 9, 2021
f9c3428
Fix test setup
htrueman Aug 9, 2021
e3469e2
Add sample_state.json
htrueman Aug 9, 2021
bcd4efd
Update reports streams logics.
htrueman Aug 10, 2021
2961a08
Update tests config.
htrueman Aug 11, 2021
a518992
Merge fixes
htrueman Aug 13, 2021
46cff71
Add reports stream slices.
htrueman Aug 17, 2021
4a56620
Post review fixes.
htrueman Aug 17, 2021
fb5c029
Merge remote-tracking branch 'origin/htrueman/source-amazon-sp-change…
htrueman Aug 19, 2021
781912d
Streams update
htrueman Aug 19, 2021
1711c7a
Merge remote-tracking branch 'origin/master' into htrueman/source-ama…
htrueman Aug 25, 2021
54f1cf0
Add reports document retrieval and decrypting.
htrueman Aug 25, 2021
f7f31e0
Add CVS parsing into result rows
htrueman Aug 26, 2021
eabc55e
Update ReportsAmazonSPStream class to be the child of Stream class.
htrueman Sep 1, 2021
c3c06e6
Schema updates
htrueman Sep 2, 2021
60b8258
Source check method updated
htrueman Sep 5, 2021
ff12577
Update ReportsAmazonSPStream retry report logics
htrueman Sep 5, 2021
034e09d
Update check_connection source method
htrueman Sep 10, 2021
1f3cb57
Update reports read_records method.
htrueman Sep 10, 2021
d59e101
Update streams.py
htrueman Sep 10, 2021
4a3daa0
Merge remote-tracking branch 'origin/master' into htrueman/source-ama…
htrueman Sep 14, 2021
3573b18
Update acceptance tests config.
htrueman Sep 14, 2021
aeab6a8
Update report read_records logics
htrueman Sep 14, 2021
b49e49c
Add reports streams rate limit handling logics.
htrueman Sep 15, 2021
ba88b73
Merge remote-tracking branch 'origin/master' into htrueman/source-ama…
htrueman Sep 17, 2021
092d54d
Source Amazon SP: Update reports streams logics. (#5311)
htrueman Sep 17, 2021
c8e8248
Merge branch 'htrueman/source-amazon-sp-extra-endpoint-support' of gi…
htrueman Sep 17, 2021
434ccc2
Bump source version.
htrueman Sep 17, 2021
00b90e1
Mock time.sleep in test_reports_stream_send_request_backoff_exception…
htrueman Sep 17, 2021
44a4329
Acceptance test basic_read test disabled
htrueman Sep 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 33 additions & 17 deletions airbyte-integrations/connectors/source-amazon-seller-partner/acceptance-test-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,38 @@ tests:
spec:
- spec_path: "source_amazon_seller_partner/spec.json"
connection:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"
- config_path: "secrets/config.json"
status: "succeed"
timeout_seconds: 60
- config_path: "integration_tests/invalid_config.json"
status: "failed"
timeout_seconds: 60
discovery:
- config_path: "secrets/config.json"
- config_path: "secrets/config.json"
basic_read:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog_no_orders.json"
empty_streams: []
incremental:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog_no_orders.json"
future_state_path: "integration_tests/abnormal_state.json"
cursor_paths:
GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL: ["createdTime"]
full_refresh:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
empty_streams:
[
"Orders",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really bad from a testing standpoint. We basically have zero guarantees about the output here. But I understand why we did it. Do we have an issue to solve this problem? If not could you create one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we definitely need a way on how to fill those streams with data. There is an issue on that #5340

"GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL",
"GET_MERCHANT_LISTINGS_ALL_DATA",
"GET_FBA_INVENTORY_AGED_DATA",
"GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL",
"GET_FLAT_FILE_OPEN_LISTINGS_DATA",
"GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA",
"GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA",
"GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT",
"VendorDirectFulfillmentShipping",
]
timeout_seconds: 600
# TODO: uncomment when Orders (or any other incremental) stream is filled with data
# incremental:
# - config_path: "secrets/config.json"
# configured_catalog_path: "integration_tests/configured_catalog.json"
# future_state_path: "integration_tests/future_state.json"
# cursor_paths:
# Orders: ["LastUpdateDate"]
# full_refresh:
# - config_path: "secrets/config.json"
# configured_catalog_path: "integration_tests/configured_catalog.json"
14 changes: 0 additions & 14 deletions ...ntegrations/connectors/source-amazon-seller-partner/integration_tests/abnormal_state.json
View file
Open in desktop

This file was deleted.

81 changes: 63 additions & 18 deletions ...rations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,37 +16,82 @@
"stream": {
"name": "GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["createdTime"]
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "incremental",
"destination_sync_mode": "append",
"cursor_field": ["createdTime"]
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_MERCHANT_LISTINGS_ALL_DATA",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["createdTime"]
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "incremental",
"destination_sync_mode": "append",
"cursor_field": ["createdTime"]
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_FBA_INVENTORY_AGED_DATA",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["createdTime"]
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "incremental",
"destination_sync_mode": "append",
"cursor_field": ["createdTime"]
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_FLAT_FILE_OPEN_LISTINGS_DATA",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "VendorDirectFulfillmentShipping",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}
49 changes: 49 additions & 0 deletions ...s/source-amazon-seller-partner/integration_tests/configured_catalog_no_empty_streams.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
{
"streams": [
{
"stream": {
"name": "GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_FBA_INVENTORY_AGED_DATA",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}
40 changes: 0 additions & 40 deletions ...nnectors/source-amazon-seller-partner/integration_tests/configured_catalog_no_orders.json
View file
Open in desktop

This file was deleted.

5 changes: 5 additions & 0 deletions ...-integrations/connectors/source-amazon-seller-partner/integration_tests/future_state.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"Orders": {
"LastUpdateDate": "2121-07-01T00:00:00Z"
}
}
15 changes: 15 additions & 0 deletions ...-integrations/connectors/source-amazon-seller-partner/integration_tests/sample_state.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,20 @@
},
"GET_FBA_INVENTORY_AGED_DATA": {
"createdTime": "2021-07-01T00:00:00Z"
},
"GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL": {
"createdTime": "2021-07-01T00:00:00Z"
},
"GET_FLAT_FILE_OPEN_LISTINGS_DATA": {
"createdTime": "2021-07-01T00:00:00Z"
},
"GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA": {
"createdTime": "2021-07-01T00:00:00Z"
},
"GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA": {
"createdTime": "2021-07-01T00:00:00Z"
},
"GET_VENDOR_INVENTORY_HEALTH_REPORT": {
"createdTime": "2021-07-01T00:00:00Z"
}
}
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-amazon-seller-partner/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@

from setuptools import find_packages, setup

MAIN_REQUIREMENTS = ["airbyte-cdk~=0.1", "boto3~=1.16", "pendulum~=2.1"]
MAIN_REQUIREMENTS = ["airbyte-cdk~=0.1", "boto3~=1.16", "pendulum~=2.1", "pycryptodome~=3.10"]

TEST_REQUIREMENTS = [
"pytest~=6.1",
Expand Down
2 changes: 1 addition & 1 deletion ...integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/auth.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,12 @@
import hmac
import urllib.parse
from typing import Any, Mapping
from urllib.parse import urlparse

import pendulum
import requests
from airbyte_cdk.sources.streams.http.auth import Oauth2Authenticator
from requests.auth import AuthBase
from requests.compat import urlparse


class AWSAuthenticator(Oauth2Authenticator):
Expand Down
Loading