🎉 New Source: Google Search Console #5350

gaart · 2021-08-11T17:24:56Z

What

Move Google Search Console from singer to native CDK

How

Rewrite the connector using Python CDK.

GSC has several notable features:

Authorization. GSC implements 2 ways of authorization: authorization through a service account or personal.

In spec.json, we have optional authorization fields, we provide the choice for the user.
We implement ServiceAccountAuthenticator, which is based on Oauth2Authenticator.
The connector will automatically choose the first correct authorization type (if both are correct, it will choose personal authentication).
Getting personal credentials is not a trivial task, so we have a credentials directory where we have several scripts to make this process more user-friendly.

Pagination.

A user can have more than 1 website to download data, so we have to download data from each one.
The same functionality is used in the searchType field (only for analytic streams). We have 4 searchType and we have to load data from each one.
These conditions are converted to a nested loop, but the CDK does not have the ability to directly implement these loops. So, we use a little trick with next_page_token - we pretend our websites and searchTypes are just different pages, and this iterative functionality is implemented in the next_page_token method.

Pre-merge Checklist

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

sherifnada · 2021-08-12T05:16:01Z

airbyte-integrations/connectors/source-google-search-console/Dockerfile

@@ -0,0 +1,16 @@
+FROM python:3.7-slim


we should use alpine, see source-zendesk-support for an example

tried an image from the zendesk connector, got timezone detection error during testing, rolled back to python:3.7-slim

TZ issue should be fixed, see latest templates

sherifnada · 2021-08-12T05:30:05Z

airbyte-integrations/connectors/source-google-search-console/credentials/credentials.json

@@ -0,0 +1,5 @@
+{


this file should be in a dir called secrets/ otherwise creds will get checked into source control by accident

…-source-google-search-console # Conflicts: # .github/workflows/publish-command.yml # .github/workflows/test-command.yml

sherifnada · 2021-09-03T04:18:51Z

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

@@ -306,6 +306,11 @@
  dockerRepository: airbyte/source-pokeapi
  dockerImageTag: 0.1.1
  documentationUrl: https://docs.airbyte.io/integrations/sources/pokeapi
+- sourceDefinitionId: eb4c9e00-db83-4d63-a386-39cfa91012a8
+  name: Google Search Console (native)


@gaart reminder about this comment

sherifnada · 2021-09-03T04:25:38Z

...integrations/connectors/source-google-search-console/source_google_search_console/streams.py

+
+class Sitemaps(GoogleSearchConsole):
+    """
+    API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3/sitemaps


thanks for including API docs links, makes it much easier to see what's going on!

sherifnada · 2021-09-03T04:27:08Z

...integrations/connectors/source-google-search-console/source_google_search_console/streams.py

+
+    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
+        records = response.json().get(self.data_field) or []
+        for record in sorted(records, key=lambda x: x["keys"][0]):


just so that I'm on the same page, so the sorting was not needed?

sherifnada

Looks great! a few more points here:

I think state updates are not functioning as intended due to interaction with stream slices
we should remove the singer connector from the connector index
need to go through the checklist to create needed docs
can you add a bootstrap doc like we discussed in retro?

sherifnada · 2021-09-03T04:34:13Z

...integrations/connectors/source-google-search-console/source_google_search_console/streams.py

+
+            yield record
+
+    def get_updated_state(


I don't this is correct? it needs to be scoped by searchType. Currently if you sync only 1 search Type then the connector fails, the state output will just say date: September 1st which means that we will skip syncing data from previous dates for all other search types. I could be reading this wrong though?

let me know if I can help explain stream slices more clearly, it's definitely not an obvious concept at first

also can you add a unit test for testing state?

@sherifnada The functionality of state updates has been updated, but now we are faced with the problem of the acceptance test failing due to nesting, we cannot tell the test to get a value from a nested structure with dynamic fields.
Any suggestions?

sherifnada · 2021-09-03T04:36:44Z

...integrations/connectors/source-google-search-console/source_google_search_console/streams.py

+            yield record
+
+
+class SearchAnalyticsAllFields(SearchAnalytics):


we should put a warning in the docs that this will take a very long time to sync because it's so granular

gaart · 2021-09-07T11:45:20Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1209277304
❌ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1209277304

…airbytehq/airbyte into gaart/5235-source-google-search-console

gaart · 2021-09-07T12:27:15Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1209407271
❌ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1209407271

gaart · 2021-09-08T13:06:57Z

/test connector=connectors/source-google-search-console

gaart · 2021-09-08T13:55:26Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1213601944
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1213601944

sherifnada · 2021-09-08T20:25:38Z

airbyte-integrations/builds.md

@@ -29,6 +29,7 @@
 | Google Ads   | [![source-google-ads](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-google-ads%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-google-ads) |
 | Google Adwords   | [![source-google-adwords-singer](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-google-adwords-singer%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-google-adwords-singer) |
 | Google Analytics   | [![source-googleanalytics-singer](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-googleanalytics-singer%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-googleanalytics-singer) |
+| Google Search Console   | [![source-google-search-console](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-google-search-console%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-google-search-console) |


could you insert it in alphabetical order?

sherifnada · 2021-09-08T20:28:40Z

airbyte-integrations/connectors/source-google-search-console/BOOTSTRAP.md

@@ -0,0 +1,55 @@
+# Google Search Console
+
+The API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3/parameters.


Suggested change

The API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3/parameters.

From [the docs](https://support.google.com/webmasters/answer/9128668?hl=en):

Google Search Console is a free service offered by Google that helps you monitor, maintain, and troubleshoot your site's presence in Google Search results.

Search Console offers tools and reports for the following actions:

* Confirm that Google can find and crawl your site.

* Fix indexing problems and request re-indexing of new or updated content.

* View Google Search traffic data for your site: how often your site appears in Google Search, which search queries show your site, how often searchers click through for those queries, and more.

* Receive alerts when Google encounters indexing, spam, or other issues on your site.

* Show you which sites link to your website.

* Troubleshoot issues for AMP, mobile usability, and other Search features.

The API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3/parameters.

sherifnada · 2021-09-08T20:34:20Z

airbyte-integrations/connectors/source-google-search-console/BOOTSTRAP.md

+we have to deal with a large dataset. 
+
+In order to reduce the amount of data, and to retrieve a specific dataset (for example, to get country specific data) 
+we can use SearchAnalyticsByCountry.


Suggested change

we can use SearchAnalyticsByCountry.

we can use SearchAnalyticsByCountry. So each of the SearchAnalytics streams groups data by certain dimensions like date, country, page, etc...

sherifnada · 2021-09-08T20:35:25Z

airbyte-integrations/connectors/source-google-search-console/BOOTSTRAP.md

+To chose one we use an authorization field with the `oneOf` parameter  in the `spec.json` file.
+
+
+## Analytics


I think this section should be in the source code since if the code changes this can go out of date, so putting it next to the code reduces the chance it goes out of date. . The bootstrap should be more focused on higher level details of how the API works

sherifnada · 2021-09-08T20:36:10Z

docs/integrations/sources/google-search-console.md

@@ -82,6 +80,15 @@ If you lose this key pair, you will need to generate a new one!
 ### Note
 You can return to the [API Console/Credentials](https://console.cloud.google.com/apis/credentials) at any time to view the email address, public key fingerprints, and other information, or to generate additional public/private key pairs. For more details about service account credentials in the API Console, see [Service accounts](https://cloud.google.com/iam/docs/understanding-service-accounts) in the API Console help file.

+### Using the existing User Account 


no need to add this section to the external docs as it will be automated via oauth in the UI. This is an excellent README however to place in the credentials/ directory.

sherifnada · 2021-09-08T20:37:37Z

docs/integrations/sources/google-search-console.md


 ## How to create the client credentials for Google Search Console, to use with Airbyte?

 You can either:
 * Use the existing `Service Account` for your Google Project with granted Admin Permissions
+* Use the existing `User Account` for your Google Project with granted Admin Permissions


Suggested change

* Use the existing `User Account` for your Google Project with granted Admin Permissions

* Use your personal Google User Account with oauth. If you choose this option, your account must have permissions to view the Google Search Console project you choose.

sherifnada · 2021-09-08T20:37:54Z

docs/integrations/README.md

@@ -42,7 +42,7 @@ Airbyte uses a grading system for connectors to help users understand what to ex
 |[Google Analytics](./sources/googleanalytics.md)| Beta |
 |[Google Analytics v4](./sources/google-analytics-v4.md)| Beta |
 |[Google Directory](./sources/google-directory.md)| Certified |
-|[Google Search Console](./sources/google-search-console.md)| Beta |
+|[Google Search Console](./sources/google-search-console.md)| Alpha |


should still be beta since we are committed to supporting it

Suggested change

|[Google Search Console](./sources/google-search-console.md)| Alpha |

|[Google Search Console](./sources/google-search-console.md)| Beta |

sherifnada · 2021-09-08T20:40:25Z

...integrations/connectors/source-google-search-console/source_google_search_console/streams.py

+            current_stream_state = {site_url: {search_type: {self.cursor_field: latest_benchmark}}}
+
+        # this line is required to pass the acceptance test successfully
+        current_stream_state[self.cursor_field] = current_stream_state[site_url][search_type][self.cursor_field]


is there any way to avoid having this here i.e; adding information only to use it in tests? Is the problem that we want the max over all search types and the yaml format doesn't support that? if so let's add this comment to help the next dev understand

Suggested change

current_stream_state[self.cursor_field] = current_stream_state[site_url][search_type][self.cursor_field]

# we need to get the max date over all searchTypes but the current acctest YAML format doesn't

# support that

current_stream_state[self.cursor_field] = current_stream_state[site_url][search_type][self.cursor_field]

gaart · 2021-09-10T05:56:05Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1220120844
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1220120844

gaart · 2021-09-10T06:26:50Z

/publish connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1220195151
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1220195151

gaart added 6 commits August 11, 2021 20:07

Generate Google Search Console connector

b58d80e

Add schema

73f717e

Upd schema

87a3447

Upd authenticator

1802891

Add creds retrieving script

01c8fac

Merge branch 'master' into gaart/5235-source-google-search-console

256464d

github-actions bot added the area/connectors Connector related issues label Aug 11, 2021

gaart changed the title ~~Gaart/5235 source google search console~~ Source: Google Search Console Aug 11, 2021

gaart changed the title ~~Source: Google Search Console~~ New Source: Google Search Console Aug 11, 2021

gaart changed the title ~~New Source: Google Search Console~~ Source: Google Search Console Aug 11, 2021

gaart linked an issue Aug 11, 2021 that may be closed by this pull request

New Source: Airbyte native Google Search Console #5235

Closed

Remove legacy dep

326c8be

sherifnada reviewed Aug 12, 2021

View reviewed changes

gaart added 4 commits August 16, 2021 17:02

Upd dockerfile base image

e7a9c56

Add sample config

e8a8965

Upd source definitions, add ci_credentials injection

70253a5

Upd schema

d19a5c4

jrhizor temporarily deployed to more-secrets August 17, 2021 06:50 Inactive

Upd GSC creds injection

61f6403

jrhizor temporarily deployed to more-secrets August 17, 2021 08:49 Inactive

jrhizor temporarily deployed to more-secrets August 17, 2021 09:18 Inactive

jrhizor temporarily deployed to more-secrets August 17, 2021 10:09 Inactive

Cleanup

9d490ed

jrhizor temporarily deployed to more-secrets August 17, 2021 13:30 Inactive

jrhizor temporarily deployed to more-secrets August 18, 2021 10:11 Inactive

Add tzdata

eaedff9

jrhizor temporarily deployed to more-secrets August 18, 2021 10:20 Inactive

Upd tzdata installing

f8ca7ff

jrhizor temporarily deployed to more-secrets August 18, 2021 10:28 Inactive

Merge branch 'master' of github.com:airbytehq/airbyte into gaart/5235…

0ade7af

…-source-google-search-console # Conflicts: # .github/workflows/publish-command.yml # .github/workflows/test-command.yml

gaart requested a review from sherifnada September 2, 2021 13:45

sherifnada reviewed Sep 3, 2021

View reviewed changes

gaart added 4 commits September 7, 2021 13:19

Remove source-google-search-console-singer from source definitions

d6453c0

Upd creds retrieving base image

bb6e70c

Upd schema

af0cfd0

Upd docs

5a7b4d4

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 7, 2021

gaart and others added 2 commits September 7, 2021 14:44

Add badge

ae6f5da

Merge branch 'master' into gaart/5235-source-google-search-console

820500f

jrhizor temporarily deployed to more-secrets September 7, 2021 11:47 Inactive

gaart added 3 commits September 7, 2021 15:23

Upd schema

7f0d726

Upd docs

1c66136

Merge branch 'gaart/5235-source-google-search-console' of github.com:…

c6b4ec3

…airbytehq/airbyte into gaart/5235-source-google-search-console

jrhizor temporarily deployed to more-secrets September 7, 2021 12:30 Inactive

Move the cursor field to the top

c5fe6f8

jrhizor temporarily deployed to more-secrets September 8, 2021 13:58 Inactive

sherifnada approved these changes Sep 8, 2021

View reviewed changes

Upd docs

2336322

jrhizor temporarily deployed to more-secrets September 10, 2021 05:58 Inactive

jrhizor temporarily deployed to more-secrets September 10, 2021 06:29 Inactive

gaart merged commit c0d4652 into master Sep 10, 2021

gaart deleted the gaart/5235-source-google-search-console branch September 10, 2021 08:47

jrhizor mentioned this pull request Sep 16, 2021

Bump Airbyte version from 0.29.17-alpha to 0.29.18-alpha #6125

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Google Search Console #5350

🎉 New Source: Google Search Console #5350

gaart commented Aug 11, 2021 •

edited

Loading

sherifnada Aug 12, 2021

gaart Aug 18, 2021

sherifnada Sep 1, 2021

sherifnada Aug 12, 2021

sherifnada Sep 3, 2021

sherifnada Sep 3, 2021

sherifnada Sep 3, 2021

sherifnada left a comment

sherifnada Sep 3, 2021

gaart Sep 7, 2021

gaart Sep 8, 2021

sherifnada Sep 3, 2021

gaart commented Sep 7, 2021 •

edited by github-actions bot

Loading

gaart commented Sep 7, 2021 •

edited by github-actions bot

Loading

gaart commented Sep 8, 2021

gaart commented Sep 8, 2021 •

edited by github-actions bot

Loading

sherifnada Sep 8, 2021

sherifnada Sep 8, 2021

sherifnada Sep 8, 2021

sherifnada Sep 8, 2021

sherifnada Sep 8, 2021

sherifnada Sep 8, 2021

sherifnada Sep 8, 2021

sherifnada Sep 8, 2021

gaart commented Sep 10, 2021 •

edited by github-actions bot

Loading

gaart commented Sep 10, 2021 •

edited by github-actions bot

Loading

		yield record


		class SearchAnalyticsAllFields(SearchAnalytics):

		@@ -0,0 +1,55 @@
		# Google Search Console

		The API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3/parameters.

-The API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3/parameters.
+From [the docs](https://support.google.com/webmasters/answer/9128668?hl=en):
+Google Search Console is a free service offered by Google that helps you monitor, maintain, and troubleshoot your site's presence in Google Search results.
+Search Console offers tools and reports for the following actions:
+* Confirm that Google can find and crawl your site.
+* Fix indexing problems and request re-indexing of new or updated content.
+* View Google Search traffic data for your site: how often your site appears in Google Search, which search queries show your site, how often searchers click through for those queries, and more.
+* Receive alerts when Google encounters indexing, spam, or other issues on your site.
+* Show you which sites link to your website.
+* Troubleshoot issues for AMP, mobile usability, and other Search features.
+The API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3/parameters.

	we can use SearchAnalyticsByCountry.
	we can use SearchAnalyticsByCountry. So each of the SearchAnalytics streams groups data by certain dimensions like date, country, page, etc...

		To chose one we use an authorization field with the `oneOf` parameter in the `spec.json` file.


		## Analytics

	* Use the existing `User Account` for your Google Project with granted Admin Permissions
	* Use your personal Google User Account with oauth. If you choose this option, your account must have permissions to view the Google Search Console project you choose.

	\|[Google Search Console](./sources/google-search-console.md)\| Alpha \|
	\|[Google Search Console](./sources/google-search-console.md)\| Beta \|

🎉 New Source: Google Search Console #5350

🎉 New Source: Google Search Console #5350

Conversation

gaart commented Aug 11, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sherifnada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaart commented Sep 7, 2021 • edited by github-actions bot Loading

gaart commented Sep 7, 2021 • edited by github-actions bot Loading

gaart commented Sep 8, 2021

gaart commented Sep 8, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaart commented Sep 10, 2021 • edited by github-actions bot Loading

gaart commented Sep 10, 2021 • edited by github-actions bot Loading

gaart commented Aug 11, 2021 •

edited

Loading

gaart commented Sep 7, 2021 •

edited by github-actions bot

Loading

gaart commented Sep 7, 2021 •

edited by github-actions bot

Loading

gaart commented Sep 8, 2021 •

edited by github-actions bot

Loading

gaart commented Sep 10, 2021 •

edited by github-actions bot

Loading

gaart commented Sep 10, 2021 •

edited by github-actions bot

Loading