🐛 Source Zendesk Support/Chat: got stuck after 100k records

[marcosmarxm]

Is this your first time deploying Airbyte: No
OS Version / Instance: Ubuntu 20.04
Memory / Disk: 8Gb / 100 GB ssd
Deployment: Docker
Airbyte Version: 0.35.61-alpha
Source name/version: airbyte/source-zendesk-chat:0.1.6 (latest)
Destination name/version: airbyte/destination-bigquery: 1.0.5 (latest)
Step: normal sync.
Description: everytime the process starts to sync it freezes after some hours, the same happens with source-zendesk-support:0.2.3 (latest).

See discussion in Discourse: https://discuss.airbyte.io/t/zendesk-chat-and-zendesk-support-freezes-after-some-hours/455

[RobertoBonnet]

Hello guys. I have a solution for Zendesk Chat

[RobertoBonnet]

TL DR;

• Zendesk API changed the rate limit for offset pagination. Rate limit down to 10 request/minute after page 1000 -https://support.zendesk.com/hc/en-us/articles/4408824785946-Announcing-API-rate-limits-for-Offset-based-Pagination
• Sleep to asynchronous requests with 8 works (default) is a problem -https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py#L275

I started to debug the connector to Zendesk suport. The problem only affect clients with to much data, indeed.

Some routes requests are mades using asynchronous requests. This work to small data, but the process is stucking everything when the error start for large data.

Searching I have found some changed in Zendesk API that affect this code. The rate limit change for pagination greater than 1000. Here at Hurb we are making some changes:

SatisfactionRatings - Process request by request using cursor-based pagination
TicketMetrics - The cursor-based pagination aren't working here. There is a error in Zendesk DB. To avoid the problem, we are processing request by request applying the rule of 10 request per minute after pagination 1000 using offset-based pagination

[RobertoBonnet]

There is a way to see the error. Just make a request to https://{account}.zendesk.com/api/v2/ticket_metrics.json?page=999999 11 times in a minute

[marcosmarxm]

Thanks @RobertoBonnet to dive into this! Really helpful analysis

[marcosmarxm]

There is a way to see the error. Just make a request to https://{account}.zendesk.com/api/v2/ticket_metrics.json?page=999999 11 times in a minute

Should we add this new request limits to your PR?

[RobertoBonnet]

@marcosmarxm It won't be needed by the tests I performed. The way the data is fetched does not go through this rate limit. I made some changes and I'm processing the full sync. Waiting to see what will happen. 3 days have passed.