Google Ads: Support user-specified queries in GAQL (Google Ads Query Language)

[zestyping]

Tell us about the problem you're trying to solve

We have an existing data pipeline that pulls various reports from the Google AdWords API, and we're trying to replace it with an Airbyte connection. Because the Google AdWords API is being deprecated soon, and Google is encouraging everyone to use the new Google Ads API instead, we're using Airbyte's new Google Ads API connector.

However, the Google Ads API connector doesn't support all the performance reports that we were using from the AdWords API. For example, we use the Age Range Performance report, the Gender Performance Report, and other reports. Moreover, we might want to use or create other reports in the future, and it would be difficult to do that if we have to wait for Airbyte to add a specific type of report and release a new version of the connector every time we want another one.

Describe the solution you’d like

One of the big advantages of the Google Ads API, and something I was really excited about when I learned the Airbyte was going to support it, is that you can issue arbitrary queries in GAQL. The various performance reports are just GAQL queries (here's one example).

If the Google Ads source connector allowed us to specify an arbitrary GAQL query, that would open up vast possibilities for users with a single change.

Describe the alternative you’ve considered or used

The current version of the Google Ads source (0.1.5) supports a few reports: #4962 added account_performance_report, ad_group_ad_report, display_keyword_performance_report, display_topics_performance_report, and shopping_performance_report. These are somewhat useful but don't include all the reports we need. The workaround would be to keep requesting new reports every time we want one, and waiting for support to be added to the connector. It seems like it would be better to add GAQL support so anyone can construct any kind of report they want.

[sherifnada]

great suggestion @zestyping . One tricky thing will be running incremental sync based on an arbitrary query. Some ideas about this:

1. The user optionally inputs the name of the field to use as a cursor (this would need to be a date or number field, but without validating that the field is a "sane" cursor type this could be a big footgun)
2. we don't support incremental sync for custom reports (which seems undesirable)
3. there is probably another idea I'm missing

[zestyping]

Hey @sherifnada, I'm sorry I never got around to responding to your comment!

It turns out we are running into exactly the issue you predicted now, because we want to write a custom GAQL query on a table that happens not to have a segments.date field to use as a cursor. Because insert_segments_date_expr always inserts some expressions involving segments.date, the GAQL query won't run in this case.

So, my teammate @jd-sanders is working on a small change now to let us toggle off the query editing performed by insert_segments_date_expr when the stream has no such field. Coming soon to a GitHub PR near you.