Source Google Ads: make streams incremental #10315

augan-rymkhan · 2022-02-14T12:19:35Z

What

Resolves #103
In the logs provided here I see that when reading ad_group_ads stream page token has expired, because currently filtering by segments.date is not implemented for this stream.

How

Make streams: ad_group_ads, ad_groups, accounts, campaigns support incremental.

One thing to note, when filtering by segments.date records returned from stream ad_group_ad might be duplicated, this is discussed in the resources below:
link1
link2

Recommended reading order

source_google_ads/streams.py
source_google_ads/source.py

augan-rymkhan · 2022-02-14T12:22:10Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/1840989553
❌ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/1840989553
🐛 https://gradle.com/s/ez25y55z3bmcm
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
============= 3 failed, 18 passed, 1 skipped in 760.61s (0:12:40) ==============

augan-rymkhan · 2022-02-14T14:23:46Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/1841573498
✅ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/1841573498
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/tests/test_incremental.py        69     38    45%
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/__init__.py                  2      0   100%
source_google_ads/custom_query_stream.py      75     50    33%
source_google_ads/google_ads.py               68     10    85%
source_google_ads/source.py                   73     24    67%
source_google_ads/streams.py                 129     25    81%
--------------------------------------------------------------
TOTAL                                        347    109    69%
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/__init__.py                  2      0   100%
source_google_ads/custom_query_stream.py      75      6    92%
source_google_ads/google_ads.py               68      7    90%
source_google_ads/source.py                   73     19    74%
source_google_ads/streams.py                 129     11    91%
--------------------------------------------------------------
TOTAL                                        347     43    88%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
================== 21 passed, 1 skipped in 1382.83s (0:23:02) ==================

keu · 2022-02-15T21:34:33Z

@augan-rymkhan the links you posted don't explain why duplication will happen. They explain why this is not a duplicate, this is expected behavior (it is not filtering but grouping) and we see different records this means all records carry useful information for us.
Actually, this exposes another issue, GoogleAds implements incremental sync incorrectly and may lose data in case of failure because of a premature advancing of the cursor.

Unfortunately, the fact that this problem and similar in other connectors went unnoticed so far shows how badly we test this with SAT.

Also the PK definition is wrong in all streams (how destination is OK with this??? @sherifnada)
PK is date + id, not just id.
The streams show metrics for entities for each day, so it is MxN not N

codecov · 2022-02-16T06:07:03Z

Codecov Report

❗ No coverage uploaded for pull request base (master@9360508). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 407a3cc differs from pull request most recent head 9bf1c0e. Consider uploading reports for the commit 9bf1c0e to get more accurate results

@@            Coverage Diff            @@
##             master   #10315   +/-   ##
=========================================
  Coverage          ?   69.82%           
=========================================
  Files             ?        5           
  Lines             ?      348           
  Branches          ?        0           
=========================================
  Hits              ?      243           
  Misses            ?      105           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9360508...9bf1c0e. Read the comment docs.

keu · 2022-02-16T06:14:05Z

airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py

    """
    Accounts stream: https://developers.google.com/google-ads/api/fields/v8/customer
    """

-    primary_key = "customer.id"
+    primary_key = ["customer.id", "segments.date"]


all incremental streams must be updated

keu · 2022-02-16T06:14:23Z

airbyte-integrations/connectors/source-google-ads/integration_tests/configured_catalog.json

-        "source_defined_primary_key": [["ad_group_ad.ad.id"]]
+        "supported_sync_modes": ["full_refresh", "incremental"],
+        "source_defined_cursor": true,
+        "source_defined_primary_key": [["ad_group_ad.ad.id"]],


and here as well

keu

lgtm after my comments addressed

augan-rymkhan · 2022-02-16T08:03:07Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/1851590321
✅ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/1851590321
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/tests/test_incremental.py        69     38    45%
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/__init__.py                  2      0   100%
source_google_ads/custom_query_stream.py      75     50    33%
source_google_ads/google_ads.py               68     10    85%
source_google_ads/source.py                   73     24    67%
source_google_ads/streams.py                 130     24    82%
--------------------------------------------------------------
TOTAL                                        348    108    69%
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/__init__.py                  2      0   100%
source_google_ads/custom_query_stream.py      75      6    92%
source_google_ads/google_ads.py               68      7    90%
source_google_ads/source.py                   73     19    74%
source_google_ads/streams.py                 130     11    92%
--------------------------------------------------------------
TOTAL                                        348     43    88%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
================== 21 passed, 1 skipped in 1464.81s (0:24:24) ==================

augan-rymkhan · 2022-02-16T14:44:04Z

/publish connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/1853423371
✅ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/1853423371

make streams as incremental

a6c0e7c

github-actions bot added the area/connectors Connector related issues label Feb 14, 2022

augan-rymkhan changed the title ~~make streams as incremental~~ Source Google Ads: make streams incremental Feb 14, 2022

augan-rymkhan temporarily deployed to more-secrets February 14, 2022 12:20 Inactive

octavia-squidington-iii temporarily deployed to more-secrets February 14, 2022 12:24 Inactive

Augan93 added 2 commits February 14, 2022 20:08

refactor Source

0b8b985

fix typing

063979d

augan-rymkhan temporarily deployed to more-secrets February 14, 2022 14:24 Inactive

octavia-squidington-iii temporarily deployed to more-secrets February 14, 2022 14:26 Inactive

augan-rymkhan marked this pull request as ready for review February 14, 2022 16:15

augan-rymkhan requested review from keu, sergei-solonitcyn and Zirochkaa February 14, 2022 16:15

fix primary_key

b7fba06

augan-rymkhan temporarily deployed to more-secrets February 16, 2022 06:05 Inactive

keu reviewed Feb 16, 2022

View reviewed changes

keu approved these changes Feb 16, 2022

View reviewed changes

fix primary key in configured_catalog

9a22900

augan-rymkhan temporarily deployed to more-secrets February 16, 2022 07:14 Inactive

sergei-solonitcyn approved these changes Feb 16, 2022

View reviewed changes

octavia-squidington-iii temporarily deployed to more-secrets February 16, 2022 08:05 Inactive

bump the version

9fd4123

github-actions bot added the area/documentation Improvements or additions to documentation label Feb 16, 2022

augan-rymkhan temporarily deployed to more-secrets February 16, 2022 14:22 Inactive

fix primary key in catalog

407a3cc

augan-rymkhan temporarily deployed to more-secrets February 16, 2022 14:41 Inactive

octavia-squidington-iii temporarily deployed to more-secrets February 16, 2022 14:46 Inactive

updated spec and def yaml

9bf1c0e

augan-rymkhan temporarily deployed to more-secrets February 16, 2022 15:26 Inactive

augan-rymkhan merged commit 741e672 into master Feb 16, 2022

augan-rymkhan deleted the arymkhan/google-ads-enable-incremental-stream branch February 16, 2022 15:27

octavia-squidington-iii mentioned this pull request Feb 17, 2022

Bump Airbyte version from 0.35.30-alpha to 0.35.31-alpha #10446

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Google Ads: make streams incremental #10315

Source Google Ads: make streams incremental #10315

augan-rymkhan commented Feb 14, 2022 •

edited

Loading

augan-rymkhan commented Feb 14, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Feb 14, 2022 •

edited by github-actions bot

Loading

keu commented Feb 15, 2022 •

edited

Loading

codecov bot commented Feb 16, 2022 •

edited

Loading

keu Feb 16, 2022

keu Feb 16, 2022

keu left a comment

augan-rymkhan commented Feb 16, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Feb 16, 2022 •

edited by github-actions bot

Loading

Source Google Ads: make streams incremental #10315

Source Google Ads: make streams incremental #10315

Conversation

augan-rymkhan commented Feb 14, 2022 • edited Loading

What

How

Recommended reading order

augan-rymkhan commented Feb 14, 2022 • edited by github-actions bot Loading

augan-rymkhan commented Feb 14, 2022 • edited by github-actions bot Loading

keu commented Feb 15, 2022 • edited Loading

codecov bot commented Feb 16, 2022 • edited Loading

Codecov Report

keu Feb 16, 2022

Choose a reason for hiding this comment

keu Feb 16, 2022

Choose a reason for hiding this comment

keu left a comment

Choose a reason for hiding this comment

augan-rymkhan commented Feb 16, 2022 • edited by github-actions bot Loading

augan-rymkhan commented Feb 16, 2022 • edited by github-actions bot Loading

augan-rymkhan commented Feb 14, 2022 •

edited

Loading

augan-rymkhan commented Feb 14, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Feb 14, 2022 •

edited by github-actions bot

Loading

keu commented Feb 15, 2022 •

edited

Loading

codecov bot commented Feb 16, 2022 •

edited

Loading

augan-rymkhan commented Feb 16, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Feb 16, 2022 •

edited by github-actions bot

Loading