Skip to content

🐛 Source Shopify: fix customer journey summary field schema type #43326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Aug 6, 2024
Merged

🐛 Source Shopify: fix customer journey summary field schema type #43326

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
ef41bf9
test
bazarnov Jul 11, 2024
edd3ee8
Merge remote-tracking branch 'origin/master'
bazarnov Jul 11, 2024
cb299d0
Merge remote-tracking branch 'origin/master'
bazarnov Jul 13, 2024
adcae54
Merge branch 'master' of https://github.com/airbytehq/airbyte
bazarnov Jul 16, 2024
5057187
Merge remote-tracking branch 'origin/master'
bazarnov Jul 16, 2024
7203d14
Merge remote-tracking branch 'origin/master'
bazarnov Jul 16, 2024
2a8143d
Merge remote-tracking branch 'origin/master'
bazarnov Jul 17, 2024
f71eb18
Merge remote-tracking branch 'origin/master'
bazarnov Jul 17, 2024
7c05236
Merge remote-tracking branch 'origin/master'
bazarnov Jul 26, 2024
41e1e73
Merge remote-tracking branch 'origin/master'
bazarnov Jul 29, 2024
41d30ba
Merge branch 'master' of https://github.com/airbytehq/airbyte
bazarnov Jul 29, 2024
844dbb8
Merge remote-tracking branch 'origin/master'
bazarnov Jul 30, 2024
6690dd5
Merge remote-tracking branch 'origin/master'
bazarnov Jul 31, 2024
68ed8cd
Merge remote-tracking branch 'origin/master'
bazarnov Aug 2, 2024
993f697
Merge remote-tracking branch 'origin/master'
bazarnov Aug 2, 2024
1a1bf21
Merge remote-tracking branch 'origin/master'
bazarnov Aug 2, 2024
381172d
reverted
bazarnov Aug 2, 2024
76ef43b
Merge remote-tracking branch 'origin/master'
bazarnov Aug 3, 2024
456188a
Merge remote-tracking branch 'origin/master'
bazarnov Aug 4, 2024
a0ec5e5
Merge remote-tracking branch 'origin/master'
bazarnov Aug 5, 2024
edfcc85
Merge remote-tracking branch 'origin/master'
bazarnov Aug 5, 2024
b1a2976
Merge remote-tracking branch 'origin/master'
bazarnov Aug 6, 2024
fd6892e
Merge remote-tracking branch 'origin/master'
bazarnov Aug 6, 2024
34f9d52
fixed
bazarnov Aug 6, 2024
c843654
updated changelog
bazarnov Aug 6, 2024
be949bc
updated description
bazarnov Aug 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions airbyte-integrations/connectors/source-shopify/acceptance-test-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@ acceptance_tests:
discovery:
tests:
- config_path: "secrets/config.json"
backward_compatibility_tests_config:
# specified the Type for `customer_journey_summary` field,
# for `customer_journey_summary` stream.
disable_for_version: 2.14.17
basic_read:
tests:
- config_path: "secrets/config_transactions_with_user_id.json"
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-shopify/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: 9da77001-af33-4bcd-be46-6252bf9342b9
dockerImageTag: 2.4.17
dockerImageTag: 2.4.18
dockerRepository: airbyte/source-shopify
documentationUrl: https://docs.airbyte.com/integrations/sources/shopify
githubIssueLabel: source-shopify
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-shopify/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
version = "2.4.17"
version = "2.4.18"
name = "source-shopify"
description = "Source CDK implementation for Shopify."
authors = [ "Airbyte <[email protected]>",]
Expand Down
330 changes: 167 additions & 163 deletions ...tegrations/connectors/source-shopify/source_shopify/schemas/customer_journey_summary.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,175 +17,179 @@
"format": "date-time"
},
"customer_journey_summary": {
"ready": {
"description": "Whether the attributed sessions for the order have been created yet.",
"type": ["null", "boolean"]
},
"moments_count": {
"description": "The total number of customer moments associated with this order. Returns null if the order is still in the process of being attributed.",
"type": ["null", "object"],
"properties": {
"count": {
"description": "Count of elements.",
"type": ["null", "integer"]
},
"precision": {
"description": "Precision of count, how exact is the value.",
"type": ["null", "string"]
"description": "Represents a customer's visiting activities on a shop's online store.",
"type": ["null", "object"],
"properties": {
"ready": {
"description": "Whether the attributed sessions for the order have been created yet.",
"type": ["null", "boolean"]
},
"moments_count": {
"description": "The total number of customer moments associated with this order. Returns null if the order is still in the process of being attributed.",
"type": ["null", "object"],
"properties": {
"count": {
"description": "Count of elements.",
"type": ["null", "integer"]
},
"precision": {
"description": "Precision of count, how exact is the value.",
"type": ["null", "string"]
}
}
}
},
"customer_order_index": {
"description": "The position of the current order within the customer's order history. Test orders aren't included.",
"type": ["null", "integer"]
},
"days_to_conversion": {
"description": "The number of days between the first session and the order creation date. The first session represents the first session since the last order, or the first session within the 30 day attribution window, if more than 30 days have passed since the last order.",
"type": ["null", "integer"]
},
"first_visit": {
"description": "The customer's first session going into the shop.",
"type": ["null", "object"],
"properties": {
"id": {
"description": "A globally-unique ID.",
"type": ["null", "integer"]
},
"landing_page": {
"description": "URL of the first page the customer landed on for the session.",
"type": ["null", "string"]
},
"landing_page_html": {
"description": "Landing page information with URL linked in HTML. For example, the first page the customer visited was",
"type": ["null", "string"]
},
"occurred_at": {
"description": "The date and time when the customer's session occurred.",
"type": ["null", "string"],
"format": "date-time"
},
"referral_code": {
"description": "Marketing referral code from the link that the customer clicked to visit the store.",
"type": ["null", "string"]
},
"referrer_url": {
"description": "Webpage where the customer clicked a link that sent them to the online store.",
"type": ["null", "string"]
},
"source": {
"description": "Source from which the customer visited the store, such as a platform (Facebook, Google), email, direct, a website domain, QR code, or unknown.",
"type": ["null", "string"]
},
"source_type": {
"description": "Type of marketing tactic.",
"type": ["null", "string"]
},
"source_description": {
"description": "Describes the source explicitly for first or last session.",
"type": ["null", "string"]
},
"utm_parameters": {
"description": "A set of UTM parameters gathered from the URL parameters of the referrer.",
"type": ["null", "object"],
"properties": {
"campaign": {
"description": "The name of a marketing campaign.",
"type": ["null", "string"]
},
"content": {
"description": "Identifies specific content in a marketing campaign. Used to differentiate between similar content or links in a marketing campaign to determine which is the most effective.",
"type": ["null", "string"]
},
"medium": {
"description": "The medium of a marketing campaign, such as a banner or email newsletter.",
"type": ["null", "string"]
},
"source": {
"description": "The source of traffic to the merchant's store, such as Google or an email newsletter.",
"type": ["null", "string"]
},
"term": {
"description": "Paid search terms used by a marketing campaign.",
"type": ["null", "string"]
},
"customer_order_index": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does that mean that the customer_order_index was not nested enough? Does that also mean that those fields are empty in the destination right now as we populate customer_journey_summary. customer_order_index instead of customer_order_index?

The same question applies to the other fields like first_visit

Copy link
Collaborator Author

@bazarnov bazarnov Aug 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was not nested under the properties - yes, but originally it was nested correctly under the customer_journey_summary object, as expected.

Screenshot 2024-08-06 at 19 22 45

I don't think so; no one has asked questions yet, we have had this for 2 months, and I assume the destination_v2 handles the Mapping without type declared correctly for us.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh! I got confused by the indentation. So we assume here the if there are fields under a property that does not have properties, than the platform use those fields as properties? Or does the platform ignore nested fields?

Like, on master we have this:

    "customer_journey_summary": {
      "ready": {
        "description": "Whether the attributed sessions for the order have been created yet.",
        "type": ["null", "boolean"]
      },

Either the platform assume customer_journey_summary to be an object and does not clean the nested fields in customer_journey_summary or the platform just doesn't even consider the nested fields as customer_journey_summary is an object. Does that make sense to you?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we assume here the if there are fields under a property that does not have properties, than the platform use those fields as properties? Or does the platform ignore nested fields?

This should be done by the destination v2, which should treat the non-typed object as an object that has something, thus it's pushed to the destination as an json-string, I believe, instead of the serialized object.

"description": "The position of the current order within the customer's order history. Test orders aren't included.",
"type": ["null", "integer"]
},
"days_to_conversion": {
"description": "The number of days between the first session and the order creation date. The first session represents the first session since the last order, or the first session within the 30 day attribution window, if more than 30 days have passed since the last order.",
"type": ["null", "integer"]
},
"first_visit": {
"description": "The customer's first session going into the shop.",
"type": ["null", "object"],
"properties": {
"id": {
"description": "A globally-unique ID.",
"type": ["null", "integer"]
},
"landing_page": {
"description": "URL of the first page the customer landed on for the session.",
"type": ["null", "string"]
},
"landing_page_html": {
"description": "Landing page information with URL linked in HTML. For example, the first page the customer visited was",
"type": ["null", "string"]
},
"occurred_at": {
"description": "The date and time when the customer's session occurred.",
"type": ["null", "string"],
"format": "date-time"
},
"referral_code": {
"description": "Marketing referral code from the link that the customer clicked to visit the store.",
"type": ["null", "string"]
},
"referrer_url": {
"description": "Webpage where the customer clicked a link that sent them to the online store.",
"type": ["null", "string"]
},
"source": {
"description": "Source from which the customer visited the store, such as a platform (Facebook, Google), email, direct, a website domain, QR code, or unknown.",
"type": ["null", "string"]
},
"source_type": {
"description": "Type of marketing tactic.",
"type": ["null", "string"]
},
"source_description": {
"description": "Describes the source explicitly for first or last session.",
"type": ["null", "string"]
},
"utm_parameters": {
"description": "A set of UTM parameters gathered from the URL parameters of the referrer.",
"type": ["null", "object"],
"properties": {
"campaign": {
"description": "The name of a marketing campaign.",
"type": ["null", "string"]
},
"content": {
"description": "Identifies specific content in a marketing campaign. Used to differentiate between similar content or links in a marketing campaign to determine which is the most effective.",
"type": ["null", "string"]
},
"medium": {
"description": "The medium of a marketing campaign, such as a banner or email newsletter.",
"type": ["null", "string"]
},
"source": {
"description": "The source of traffic to the merchant's store, such as Google or an email newsletter.",
"type": ["null", "string"]
},
"term": {
"description": "Paid search terms used by a marketing campaign.",
"type": ["null", "string"]
}
}
},
"admin_graphql_api_id": {
"description": "Unique identifier for the customer in the Admin GraphQL API.",
"type": ["null", "string"]
}
},
"admin_graphql_api_id": {
"description": "Unique identifier for the customer in the Admin GraphQL API.",
"type": ["null", "string"]
}
}
},
"last_visit": {
"description": "The last session before an order is made.",
"type": ["null", "object"],
"properties": {
"id": {
"description": "A globally-unique ID.",
"type": ["null", "integer"]
},
"landing_page": {
"description": "URL of the first page the customer landed on for the session.",
"type": ["null", "string"]
},
"landing_page_html": {
"description": "Landing page information with URL linked in HTML. For example, the first page the customer visited was",
"type": ["null", "string"]
},
"occurred_at": {
"description": "The date and time when the customer's session occurred.",
"type": ["null", "string"],
"format": "date-time"
},
"referral_code": {
"description": "Marketing referral code from the link that the customer clicked to visit the store.",
"type": ["null", "string"]
},
"referrer_url": {
"description": "Webpage where the customer clicked a link that sent them to the online store.",
"type": ["null", "string"]
},
"source": {
"description": "Source from which the customer visited the store, such as a platform (Facebook, Google), email, direct, a website domain, QR code, or unknown.",
"type": ["null", "string"]
},
"source_type": {
"description": "Type of marketing tactic.",
"type": ["null", "string"]
},
"source_description": {
"description": "Describes the source explicitly for first or last session.",
"type": ["null", "string"]
},
"utm_parameters": {
"description": "A set of UTM parameters gathered from the URL parameters of the referrer.",
"type": ["null", "object"],
"properties": {
"campaign": {
"description": "The name of a marketing campaign.",
"type": ["null", "string"]
},
"content": {
"description": "Identifies specific content in a marketing campaign. Used to differentiate between similar content or links in a marketing campaign to determine which is the most effective.",
"type": ["null", "string"]
},
"medium": {
"description": "The medium of a marketing campaign, such as a banner or email newsletter.",
"type": ["null", "string"]
},
"source": {
"description": "The source of traffic to the merchant's store, such as Google or an email newsletter.",
"type": ["null", "string"]
},
"term": {
"description": "Paid search terms used by a marketing campaign.",
"type": ["null", "string"]
},
"last_visit": {
"description": "The last session before an order is made.",
"type": ["null", "object"],
"properties": {
"id": {
"description": "A globally-unique ID.",
"type": ["null", "integer"]
},
"landing_page": {
"description": "URL of the first page the customer landed on for the session.",
"type": ["null", "string"]
},
"landing_page_html": {
"description": "Landing page information with URL linked in HTML. For example, the first page the customer visited was",
"type": ["null", "string"]
},
"occurred_at": {
"description": "The date and time when the customer's session occurred.",
"type": ["null", "string"],
"format": "date-time"
},
"referral_code": {
"description": "Marketing referral code from the link that the customer clicked to visit the store.",
"type": ["null", "string"]
},
"referrer_url": {
"description": "Webpage where the customer clicked a link that sent them to the online store.",
"type": ["null", "string"]
},
"source": {
"description": "Source from which the customer visited the store, such as a platform (Facebook, Google), email, direct, a website domain, QR code, or unknown.",
"type": ["null", "string"]
},
"source_type": {
"description": "Type of marketing tactic.",
"type": ["null", "string"]
},
"source_description": {
"description": "Describes the source explicitly for first or last session.",
"type": ["null", "string"]
},
"utm_parameters": {
"description": "A set of UTM parameters gathered from the URL parameters of the referrer.",
"type": ["null", "object"],
"properties": {
"campaign": {
"description": "The name of a marketing campaign.",
"type": ["null", "string"]
},
"content": {
"description": "Identifies specific content in a marketing campaign. Used to differentiate between similar content or links in a marketing campaign to determine which is the most effective.",
"type": ["null", "string"]
},
"medium": {
"description": "The medium of a marketing campaign, such as a banner or email newsletter.",
"type": ["null", "string"]
},
"source": {
"description": "The source of traffic to the merchant's store, such as Google or an email newsletter.",
"type": ["null", "string"]
},
"term": {
"description": "Paid search terms used by a marketing campaign.",
"type": ["null", "string"]
}
}
},
"admin_graphql_api_id": {
"description": "Unique identifier for the customer in the Admin GraphQL API.",
"type": ["null", "string"]
}
},
"admin_graphql_api_id": {
"description": "Unique identifier for the customer in the Admin GraphQL API.",
"type": ["null", "string"]
}
}
}
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/shopify.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,7 @@ For all `Shopify GraphQL BULK` api requests these limitations are applied: https

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.4.18 | 2024-08-06 | [43326](https://github.com/airbytehq/airbyte/pull/43326) | Added missing `type` type for `customer_journey_summary` field for `Customer Journey Summary` stream schema |
| 2.4.17 | 2024-08-02 | [42973](https://github.com/airbytehq/airbyte/pull/42973) | Fixed `FAILED` Job handling for `no-checkpointing` BULK Streams, fixed STATE collision for REST Streams with `Deleted Events` |
| 2.4.16 | 2024-07-21 | [42095](https://github.com/airbytehq/airbyte/pull/42095) | Added the `Checkpointing` for the `BULK` streams, fixed the `store` redirection |
| 2.4.15 | 2024-07-27 | [42806](https://github.com/airbytehq/airbyte/pull/42806) | Update dependencies |
Expand Down
Loading