Source Bing Ads: add fields to schema #20005

LuliRib · 2022-12-01T22:48:11Z

added field keyword to bing ads keyword_performance_report_daily

added field segments.device to google ads keyword_report

CLAassistant · 2022-12-01T22:48:18Z

All committers have signed the CLA.

airbyte-integrations/connectors/source-google-ads/source_google_ads/schemas/keyword_report.json

marcosmarxm · 2022-12-02T17:34:21Z

/test connector=connectors/source-bing-ads

🕑 connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/3603835643
✅ connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/3603835643
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_bing_ads/__init__.py       2      0   100%
source_bing_ads/source.py       260      9    97%
source_bing_ads/cache.py         18      1    94%
source_bing_ads/reports.py      133     17    87%
source_bing_ads/client.py        93     14    85%
-------------------------------------------------
TOTAL                           506     41    92%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     199     93    53%   35, 41-43, 48, 53, 59, 65, 71, 77-79, 98, 103-105, 111-113, 119-120, 125-126, 131, 137, 146-155, 161-166, 181, 205, 236, 242, 250-255, 263-268, 276-289, 294-300, 307-318, 325-341
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1569    350    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
============ 29 passed, 1 skipped, 29 warnings in 652.23s (0:10:52) ============

marcosmarxm · 2022-12-02T17:58:15Z

@LuliRib can you bump the dockerfile version and documentation changelog too?

LuliRib · 2022-12-02T18:07:05Z

Hi, sure. From where should I extract the changelog? Did the tests pass correctly for both google ads and bing ads? If I´m not wrong, The source-google-ads Dockerfile version LABEL io.airbyte.version=0.2.5 LABEL io.airbyte.version=0.1.16 is the source-bing-ads Dockerfile version

…

________________________________ From: Marcos Marx ***@***.***> Sent: Friday, December 2, 2022 05:58 PM To: airbytehq/airbyte ***@***.***> Cc: Laila R. ***@***.***>; Mention ***@***.***> Subject: Re: [airbytehq/airbyte] Add fields to schema (PR #20005) @LuliRib<https://github.com/LuliRib> can you bump the dockerfile version and documentation changelog too? — Reply to this email directly, view it on GitHub<#20005 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARUVBWUQUIRCJEZNHQHCFFTWLI2EDANCNFSM6AAAAAASRJOSS4>. You are receiving this because you were mentioned.Message ID: ***@***.***>

marcosmarxm · 2022-12-02T20:18:01Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3604819499
❌ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3604819499
🐛 https://gradle.com/s/2oe5fplnzz3cw

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
============= 2 failed, 28 passed, 1 skipped in 111.68s (0:01:51) ==============

LuliRib · 2022-12-05T12:29:21Z

Hi @marcosmarxm, first of all, I´m grateful for your help.
I saw bing-ads test passed.
I talked to google-ads api team and we figured out the problem with google ads. The segments.device is not compatible with
"metrics.historical_quality_score". Can we just delete it from the schema?
If you can please help me a bit more with it, I need it urgently..
And again, thank you

deleted "metrics.historical_quality_score" which is not compatible with segments.device This was verified by the google ads team

sh4sh · 2022-12-06T15:51:10Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3631123416
✅ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3631123416
Python tests coverage:

Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/models.py                   18      0   100%
source_google_ads/__init__.py                  2      0   100%
source_google_ads/google_ads.py               68     10    85%
source_google_ads/streams.py                 165     26    84%
source_google_ads/source.py                   86     24    72%
source_google_ads/custom_query_stream.py      75     46    39%
--------------------------------------------------------------
TOTAL                                        414    106    74%
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/models.py                   18      0   100%
source_google_ads/__init__.py                  2      0   100%
source_google_ads/streams.py                 165      8    95%
source_google_ads/source.py                   86      5    94%
source_google_ads/custom_query_stream.py      75      6    92%
source_google_ads/google_ads.py               68     12    82%
--------------------------------------------------------------
TOTAL                                        414     31    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================== 30 passed, 1 skipped in 1048.72s (0:17:28) ==================

marcosmarxm · 2022-12-06T23:59:08Z

@LuliRib can you split the contribution into 2 PRs one for Bing Ads and other for Google Ads? I'll ask the connector team to check your proposal of changes.

LuliRib · 2022-12-07T00:07:12Z

@marcosmarxm hi, not sure how to do it, keeping the data, tests of each source

marcosmarxm · 2022-12-07T00:49:17Z

@LuliRib please open a new branch/PR to make the changes to Google Ads.

marcosmarxm · 2022-12-07T00:50:44Z

@LuliRib because this connector is GA I'd requested to our connector team to make the final review before publishing the changes

LuliRib · 2022-12-07T00:53:06Z

Thank you so much!! Do you have an estimation on how much it would take? And regarding the google ads pr. I´ll create a new one, but the tests that already ran wont appear.. and the conversation, ok?

…

________________________________ From: Marcos Marx ***@***.***> Sent: Wednesday, December 7, 2022 12:50 AM To: airbytehq/airbyte ***@***.***> Cc: Laila R. ***@***.***>; Mention ***@***.***> Subject: Re: [airbytehq/airbyte] Source Bing Ads: add fields to schema (PR #20005) @LuliRib<https://github.com/LuliRib> because this connector is GA I'd requested to our connector team to make the final review before publishing the changes — Reply to this email directly, view it on GitHub<#20005 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARUVBWV3E5OKC54YADWPMTDWL7NPBANCNFSM6AAAAAASRJOSS4>. You are receiving this because you were mentioned.Message ID: ***@***.***>

LuliRib · 2022-12-07T01:07:08Z

hi, just created a new pr #20175 for google ads source

…

________________________________ From: Laila R. ***@***.***> Sent: Wednesday, December 7, 2022 12:53 AM To: airbytehq/airbyte ***@***.***> Subject: Re: [airbytehq/airbyte] Source Bing Ads: add fields to schema (PR #20005) Thank you so much!! Do you have an estimation on how much it would take? And regarding the google ads pr. I´ll create a new one, but the tests that already ran wont appear.. and the conversation, ok?

________________________________ From: Marcos Marx ***@***.***> Sent: Wednesday, December 7, 2022 12:50 AM To: airbytehq/airbyte ***@***.***> Cc: Laila R. ***@***.***>; Mention ***@***.***> Subject: Re: [airbytehq/airbyte] Source Bing Ads: add fields to schema (PR #20005) @LuliRib<https://github.com/LuliRib> because this connector is GA I'd requested to our connector team to make the final review before publishing the changes — Reply to this email directly, view it on GitHub<#20005 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARUVBWV3E5OKC54YADWPMTDWL7NPBANCNFSM6AAAAAASRJOSS4>. You are receiving this because you were mentioned.Message ID: ***@***.***>

...egrations/connectors/source-bing-ads/source_bing_ads/schemas/keyword_performance_report.json

added property additionalProperties: true to keyword field

I think now it´s what you asked. Thank you

added "additionalProperties": true to the root of schema

LuliRib · 2022-12-07T10:55:14Z

@bazarnov, thank you so much! What should I do next?

bazarnov · 2022-12-07T10:56:44Z

@marcosmarxm could you please assist here?

marcosmarxm · 2022-12-07T15:51:24Z

/publish connector=connectors/source-bing-ads

🕑 Publishing the following connectors:
connectors/source-bing-ads
https://github.com/airbytehq/airbyte/actions/runs/3640521127

Connector	Did it publish?	Were definitions generated?
connectors/source-bing-ads	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

LuliRib · 2022-12-07T18:43:18Z

@bazarnov , @marcosmarxm I receive ONLY null values in the new keyword field. Is there a way you could check it?

tomasx · 2022-12-13T10:28:59Z

Same here Keyword field in keyword_performance_report_daily table always null

LuliRib · 2022-12-13T10:30:57Z

Hi @tomasx , I found the problem. Hopefully when it will be approved and merged to the code it will fix the problem

LuliRib · 2022-12-13T10:31:43Z

it´s pr #20300

LuliRib added 2 commits December 1, 2022 23:44

Update keyword_performance_report.json

50c7704

Update keyword_report.json

f721eca

octavia-squidington-iv added area/connectors Connector related issues bounty community connectors/source/bing-ads connectors/source/google-ads labels Dec 1, 2022

LuliRib added 3 commits December 2, 2022 00:00

Merge branch 'master' into add-fields-to-schema

cc89531

Merge branch 'master' into add-fields-to-schema

7d77d91

Merge branch 'master' into add-fields-to-schema

f6c3c7f

marcosmarxm requested changes Dec 2, 2022

View reviewed changes

airbyte-integrations/connectors/source-google-ads/source_google_ads/schemas/keyword_report.json Outdated Show resolved Hide resolved

marcosmarxm changed the title ~~Add fields to schema~~ Source Bing Ads: add fields to schema Dec 2, 2022

LuliRib added 2 commits December 5, 2022 13:33

Update keyword_report.json

7d6c2c2

deleted "metrics.historical_quality_score" which is not compatible with segments.device This was verified by the google ads team

Merge branch 'master' into add-fields-to-schema

72c821c

LuliRib requested a review from marcosmarxm December 5, 2022 12:36

sajarin added internal and removed bounty labels Dec 5, 2022

LuliRib added 3 commits December 5, 2022 18:10

Merge branch 'master' into add-fields-to-schema

eb664cc

Merge branch 'master' into add-fields-to-schema

1100e99

Merge branch 'master' into add-fields-to-schema

d91d2ae

rollback google ads change

3a617c1

bump connector version

a63c630

octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Dec 7, 2022

marcosmarxm approved these changes Dec 7, 2022

View reviewed changes

Merge branch 'master' into add-fields-to-schema

5f36a49

bazarnov requested changes Dec 7, 2022

View reviewed changes

...egrations/connectors/source-bing-ads/source_bing_ads/schemas/keyword_performance_report.json Show resolved Hide resolved

LuliRib added 3 commits December 7, 2022 07:48

Update keyword_performance_report.json

85ada7f

added property additionalProperties: true to keyword field

Update keyword_performance_report.json

dc33120

Update keyword_performance_report.json

942014f

I think now it´s what you asked. Thank you

LuliRib requested a review from bazarnov December 7, 2022 07:04

LuliRib added 2 commits December 7, 2022 08:14

Merge branch 'master' into add-fields-to-schema

8a19a0f

Update keyword_performance_report.json

2c4c010

added "additionalProperties": true to the root of schema

LuliRib mentioned this pull request Dec 7, 2022

Source Google Ads: add segments.device and remove metrics.historical_quality_score from keyword_report #20175

Closed

bazarnov approved these changes Dec 7, 2022

View reviewed changes

Merge branch 'master' into add-fields-to-schema

4a9a5d2

auto-bump connector version

668bc9d

marcosmarxm merged commit f328706 into airbytehq:master Dec 7, 2022

LuliRib deleted the add-fields-to-schema branch December 7, 2022 16:30

octavia-squidington-iii mentioned this pull request Dec 7, 2022

Bump Airbyte version from 0.40.23 to 0.40.24 #20206

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Bing Ads: add fields to schema #20005

Source Bing Ads: add fields to schema #20005

LuliRib commented Dec 1, 2022

CLAassistant commented Dec 1, 2022 •

edited

Loading

marcosmarxm commented Dec 2, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 2, 2022

LuliRib commented Dec 2, 2022 via email

marcosmarxm commented Dec 2, 2022 •

edited by github-actions bot

Loading

LuliRib commented Dec 5, 2022

sh4sh commented Dec 6, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 6, 2022

LuliRib commented Dec 7, 2022

marcosmarxm commented Dec 7, 2022

marcosmarxm commented Dec 7, 2022

LuliRib commented Dec 7, 2022 via email

LuliRib commented Dec 7, 2022 via email

LuliRib commented Dec 7, 2022

bazarnov commented Dec 7, 2022

marcosmarxm commented Dec 7, 2022 •

edited by github-actions bot

Loading

LuliRib commented Dec 7, 2022

tomasx commented Dec 13, 2022

LuliRib commented Dec 13, 2022

LuliRib commented Dec 13, 2022

Source Bing Ads: add fields to schema #20005

Source Bing Ads: add fields to schema #20005

Conversation

LuliRib commented Dec 1, 2022

CLAassistant commented Dec 1, 2022 • edited Loading

marcosmarxm commented Dec 2, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Dec 2, 2022

LuliRib commented Dec 2, 2022 via email

marcosmarxm commented Dec 2, 2022 • edited by github-actions bot Loading

Build Failed

LuliRib commented Dec 5, 2022

sh4sh commented Dec 6, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Dec 6, 2022

LuliRib commented Dec 7, 2022

marcosmarxm commented Dec 7, 2022

marcosmarxm commented Dec 7, 2022

LuliRib commented Dec 7, 2022 via email

LuliRib commented Dec 7, 2022 via email

LuliRib commented Dec 7, 2022

bazarnov commented Dec 7, 2022

marcosmarxm commented Dec 7, 2022 • edited by github-actions bot Loading

LuliRib commented Dec 7, 2022

tomasx commented Dec 13, 2022

LuliRib commented Dec 13, 2022

LuliRib commented Dec 13, 2022

CLAassistant commented Dec 1, 2022 •

edited

Loading

marcosmarxm commented Dec 2, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 2, 2022 •

edited by github-actions bot

Loading

sh4sh commented Dec 6, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 7, 2022 •

edited by github-actions bot

Loading