Skip to content

🚨🚨 Source Linkedin Ads: Update pivotValues pk type for Ad Analytics streams #37531

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 24, 2024
Merged

🚨🚨 Source Linkedin Ads: Update pivotValues pk type for Ad Analytics streams #37531

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8515a79
added string_of_pivot_values to ad_analytics schema and set as pk
darynaishchenko Apr 24, 2024
7709847
updated unit tests
darynaishchenko Apr 24, 2024
598f311
bump version, migration docs
darynaishchenko Apr 24, 2024
bfab168
format fix
darynaishchenko Apr 24, 2024
3ce46e3
updated changelog
darynaishchenko Apr 24, 2024
1f88eab
updated breaking change docs
darynaishchenko Apr 24, 2024
5f8ccdd
updated expected records
darynaishchenko Apr 24, 2024
44e5e2e
updated pyproject.toml
darynaishchenko Apr 24, 2024
a2e5144
Merge branch 'master' into daryna/source-linkedin-ads/change-pk-pivot…
darynaishchenko Apr 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion airbyte-integrations/connectors/source-linkedin-ads/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: 137ece28-5434-455c-8f34-69dc3782f451
dockerImageTag: 1.0.1
dockerImageTag: 2.0.0
dockerRepository: airbyte/source-linkedin-ads
documentationUrl: https://docs.airbyte.com/integrations/sources/linkedin-ads
githubIssueLabel: source-linkedin-ads
Expand Down Expand Up @@ -48,6 +48,23 @@ data:
- "ad_member_seniority_analytics"
- "ad_member_region_analytics"
- "ad_member_company_analytics"
2.0.0:
message: This upgrade changes primary key for *-analytics streams from pivotValues[array of strings] to string_of_pivot_values[string].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This upgrade changes primary key for *-analytics streams from pivotValues[array of strings] to string_of_pivot_values[string] so that it is compatible with more destination types.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

upgradeDeadline: "2024-05-14"
scopedImpact:
- scopeType: stream
impactedScopes:
- "ad_campaign_analytics"
- "ad_creative_analytics"
- "ad_impression_device_analytics"
- "ad_member_company_size_analytics"
- "ad_member_country_analytics"
- "ad_member_job_function_analytics"
- "ad_member_job_title_analytics"
- "ad_member_industry_analytics"
- "ad_member_seniority_analytics"
- "ad_member_region_analytics"
- "ad_member_company_analytics"
suggestedStreams:
streams:
- accounts
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-linkedin-ads/source_linkedin_ads/analytics_streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ class LinkedInAdsAnalyticsStream(IncrementalLinkedinAdsStream, ABC):

endpoint = "adAnalytics"
# For Analytics streams, the primary_key is the entity of the pivot [Campaign URN, Creative URN, etc.] + `end_date`
primary_key = ["pivotValues", "end_date"]
primary_key = ["string_of_pivot_values", "end_date"]
cursor_field = "end_date"
records_limit = 15000
FIELDS_CHUNK_SIZE = 18
Expand Down
3 changes: 3 additions & 0 deletions ...integrations/connectors/source-linkedin-ads/source_linkedin_ads/schemas/ad_analytics.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,9 @@
"type": ["null", "string"]
}
},
"string_of_pivot_values": {
"type": ["null", "string"]
},
"postClickJobApplications": {
"type": ["null", "number"]
},
Expand Down
9 changes: 9 additions & 0 deletions airbyte-integrations/connectors/source-linkedin-ads/source_linkedin_ads/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,12 @@ def transform_col_names(record: Dict, dict_keys: list = []) -> Mapping[str, Any]
return record


def transform_pivot_values(record: Dict) -> Mapping[str, Any]:
pivot_values = record.get("pivotValues", [])
record["string_of_pivot_values"] = ",".join(pivot_values)
return record


def transform_data(records: List) -> Iterable[Mapping]:
"""
We need to transform the nested complex data structures into simple key:value pair,
Expand All @@ -323,6 +329,9 @@ def transform_data(records: List) -> Iterable[Mapping]:
if "variables" in record:
record = transform_variables(record)

if "pivotValues" in record:
record = transform_pivot_values(record)
Comment on lines +332 to +333
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This 'if' implies that one of the primary keys for the stream can be null. Is it the case that at least the other PK, end_date, will always be non-null?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is common for all streams, so non ad-analytics streams doesn't have pivotValues field. ad-analytics streams always have pivotValues and end_date.


record = transform_col_names(record, DESTINATION_RESERVED_KEYWORDS)

yield record
3 changes: 3 additions & 0 deletions ...s/connectors/source-linkedin-ads/unit_tests/utils_tests/samples/test_data_for_tranform.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@
}
},
"pivot": "TEST_PIVOT_VALUE",
"pivotValues": ["TEST_PIVOT_VALUE_1", "TEST_PIVOT_VALUE_2"]
}
]

Expand Down Expand Up @@ -142,5 +143,7 @@
"start_date": "2021-08-13",
"end_date": "2021-08-13",
"_pivot": "TEST_PIVOT_VALUE",
"string_of_pivot_values": "TEST_PIVOT_VALUE_1,TEST_PIVOT_VALUE_2",
"pivotValues": ["TEST_PIVOT_VALUE_1", "TEST_PIVOT_VALUE_2"]
}
]
2 changes: 1 addition & 1 deletion ...integrations/connectors/source-linkedin-ads/unit_tests/utils_tests/test_transform_data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from source_linkedin_ads.utils import transform_data


def test_transfrom_data():
def test_transform_data():
"""
As far as we transform the data within the generator object,
we use list() to have the actual output for the test assertion.
Expand Down
31 changes: 31 additions & 0 deletions docs/integrations/sources/linkedin-ads-migrations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,36 @@
# LinkedIn Ads Migration Guide

## Upgrading to 2.0.0

Version 2.0.0 introduces changes in the primary key selected for all *-analytics streams (including custom ones).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add this also: This upgrade changes primary key for *-analytics streams from pivotValues[array of strings] to string_of_pivot_values[string].

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated


- "ad_campaign_analytics"
- "ad_creative_analytics"
- "ad_impression_device_analytics"
- "ad_member_company_size_analytics"
- "ad_member_country_analytics"
- "ad_member_job_function_analytics"
- "ad_member_job_title_analytics"
- "ad_member_industry_analytics"
- "ad_member_seniority_analytics"
- "ad_member_region_analytics"
- "ad_member_company_analytics"

## Migration Steps

Clearing your data is required for the affected streams in order to continue syncing successfully. To clear your data for the affected streams, follow the steps below:

1. Select **Connections** in the main navbar and select the connection(s) affected by the update.
2. Select the **Schema** tab.
1. Select **Refresh source schema** to bring in any schema changes. Any detected schema changes will be listed for your review.
2. Select **OK** to approve changes.
3. Select **Save changes** at the bottom of the page.
1. Ensure the **Clear affected streams** option is checked to ensure your streams continue syncing successfully with the new schema.
4. Select **Save connection**.

This will clear the data in your destination for the subset of streams with schema changes. After the clear succeeds, trigger a sync by clicking **Sync Now**. For more information on clearing your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/reset).


## Upgrading to 1.0.0

Version 1.0.0 introduces changes in the primary key selected for all *-analytics streams (including custom ones).
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/linkedin-ads.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ After 5 unsuccessful attempts - the connector will stop the sync operation. In s

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------|
| 2.0.0 | 2024-04-24 | [37531](https://github.com/airbytehq/airbyte/pull/37531) | Change primary key for Analytics Streams |
| 1.0.1 | 2024-03-28 | [34152](https://github.com/airbytehq/airbyte/pull/34152) | Proceed pagination if return less than expected |
| 1.0.0 | 2024-04-10 | [36927](https://github.com/airbytehq/airbyte/pull/36927) | Update primary key for Analytics Streams |
| 0.8.0 | 2024-03-19 | [36267](https://github.com/airbytehq/airbyte/pull/36267) | Pin airbyte-cdk version to `^0` |
Expand Down
Loading