Source Google Ads: add TypeTransformer to force proper type casting #19208

arsenlosenko · 2022-11-09T10:52:40Z

What

Resolving:
https://github.com/airbytehq/oncall/issues/947 - Wrong schema typing for campaigns stream

How

Add a TypeTransformer to campaigns stream class to force type conversion to expected values

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

arsenlosenko · 2022-11-09T10:53:14Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3427514077
✅ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3427514077
Python tests coverage:

Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/models.py                   18      0   100%
source_google_ads/__init__.py                  2      0   100%
source_google_ads/google_ads.py               68     10    85%
source_google_ads/streams.py                 165     26    84%
source_google_ads/source.py                   86     24    72%
source_google_ads/custom_query_stream.py      75     46    39%
--------------------------------------------------------------
TOTAL                                        414    106    74%
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/models.py                   18      0   100%
source_google_ads/__init__.py                  2      0   100%
source_google_ads/streams.py                 165      8    95%
source_google_ads/source.py                   86      5    94%
source_google_ads/custom_query_stream.py      75      6    92%
source_google_ads/google_ads.py               68     12    82%
--------------------------------------------------------------
TOTAL                                        414     31    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              345    110    68%   53, 64-72, 77-84, 88-89, 93-94, 178, 216-233, 242-250, 254-259, 265, 298-303, 341-348, 391-393, 396, 461-469, 481-484, 489, 545-546, 552, 555, 591-601, 614-639
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1508    348    77%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
============ 28 passed, 1 skipped, 27 warnings in 717.69s (0:11:57) ============

CLAassistant · 2022-11-09T10:56:26Z

All committers have signed the CLA.

arsenlosenko · 2022-11-09T11:27:29Z

currently testing these changes locally, creating this PR for preliminary review

alafanechere · 2022-11-09T13:04:32Z

In the scope of the OC issue I'd suggest only enabling the transformer on the Campaign stream. @grubberr @darynaishchenko what do you think? I'm worried about potential regressions if we enable it on all streams.

arsenlosenko · 2022-11-09T13:08:56Z

@alafanechere I agree, I'll specify the TypeTransformer only to the Campaigns stream, one sec

arsenlosenko · 2022-11-09T13:11:51Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3428435524
✅ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3428435524
Python tests coverage:

Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/models.py                   18      0   100%
source_google_ads/__init__.py                  2      0   100%
source_google_ads/google_ads.py               68     10    85%
source_google_ads/streams.py                 165     26    84%
source_google_ads/source.py                   86     24    72%
source_google_ads/custom_query_stream.py      75     46    39%
--------------------------------------------------------------
TOTAL                                        414    106    74%
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/models.py                   18      0   100%
source_google_ads/__init__.py                  2      0   100%
source_google_ads/streams.py                 165      8    95%
source_google_ads/source.py                   86      5    94%
source_google_ads/custom_query_stream.py      75      6    92%
source_google_ads/google_ads.py               68     12    82%
--------------------------------------------------------------
TOTAL                                        414     31    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              345    110    68%   53, 64-72, 77-84, 88-89, 93-94, 178, 216-233, 242-250, 254-259, 265, 298-303, 341-348, 391-393, 396, 461-469, 481-484, 489, 545-546, 552, 555, 591-601, 614-639
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1508    348    77%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
============ 28 passed, 1 skipped, 27 warnings in 924.20s (0:15:24) ============

…-value-bigint

lazebnyi · 2022-11-10T13:21:59Z

/publish connector=connectors/source-google-ads

🕑 Publishing the following connectors:
connectors/source-google-ads
https://github.com/airbytehq/airbyte/actions/runs/3437156354

Connector	Did it publish?	Were definitions generated?
connectors/source-google-ads	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…19208) * Source Google Ads: add TypeTransformer to force proper type casting * Update changelog * Speciy TypeTransformer only to Campaigns stream * auto-bump connector version Co-authored-by: Serhii Lazebnyi <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]>

Source Google Ads: add TypeTransformer to force proper type casting

678cc7e

arsenlosenko self-assigned this Nov 9, 2022

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Nov 9, 2022

octavia-squidington-iv added the connectors/source/google-ads label Nov 9, 2022

Update changelog

799b7f1

arsenlosenko requested review from grubberr and darynaishchenko November 9, 2022 11:25

Speciy TypeTransformer only to Campaigns stream

136bc1f

Merge branch 'master' into arsenlosenko/947-source-google-ads-invalid…

9860736

…-value-bigint

lazebnyi self-requested a review November 10, 2022 13:18

lazebnyi approved these changes Nov 10, 2022

View reviewed changes

auto-bump connector version

bd1832e

octavia-squidington-iii temporarily deployed to more-secrets November 10, 2022 14:03 Inactive

lazebnyi merged commit 244d6d9 into master Nov 10, 2022

lazebnyi deleted the arsenlosenko/947-source-google-ads-invalid-value-bigint branch November 10, 2022 14:39

Moandco mentioned this pull request Nov 10, 2022

Google Ads: Fix wrong type for metrics.conversations #19030

Closed

14 tasks

github-actions bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Google Ads: add TypeTransformer to force proper type casting #19208

Source Google Ads: add TypeTransformer to force proper type casting #19208

arsenlosenko commented Nov 9, 2022 •

edited by lazebnyi

Loading

arsenlosenko commented Nov 9, 2022 •

edited by github-actions bot

Loading

CLAassistant commented Nov 9, 2022 •

edited

Loading

arsenlosenko commented Nov 9, 2022

alafanechere commented Nov 9, 2022

arsenlosenko commented Nov 9, 2022

arsenlosenko commented Nov 9, 2022 •

edited by github-actions bot

Loading

lazebnyi commented Nov 10, 2022 •

edited by github-actions bot

Loading

Source Google Ads: add TypeTransformer to force proper type casting #19208

Source Google Ads: add TypeTransformer to force proper type casting #19208

Conversation

arsenlosenko commented Nov 9, 2022 • edited by lazebnyi Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

arsenlosenko commented Nov 9, 2022 • edited by github-actions bot Loading

Build Passed

CLAassistant commented Nov 9, 2022 • edited Loading

arsenlosenko commented Nov 9, 2022

alafanechere commented Nov 9, 2022

arsenlosenko commented Nov 9, 2022

arsenlosenko commented Nov 9, 2022 • edited by github-actions bot Loading

Build Passed

lazebnyi commented Nov 10, 2022 • edited by github-actions bot Loading

arsenlosenko commented Nov 9, 2022 •

edited by lazebnyi

Loading

arsenlosenko commented Nov 9, 2022 •

edited by github-actions bot

Loading

CLAassistant commented Nov 9, 2022 •

edited

Loading

arsenlosenko commented Nov 9, 2022 •

edited by github-actions bot

Loading

lazebnyi commented Nov 10, 2022 •

edited by github-actions bot

Loading