Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update max rate-limiting metadata #36734

Merged
merged 5 commits into from
Apr 3, 2024
Merged

Conversation

ChristoGrab
Copy link
Contributor

@ChristoGrab ChristoGrab commented Apr 1, 2024

What

Updating max rate-limit metadata for sources Github, Sentry, Zendesk Support

Github: Per-hour window listed in docs.
Sentry: Docs do not explicitly list retry windows, and point to checking retry headers to determine these values. All endpoints returned a 1-minute retry window in tests.
Zendesk Support: Per-minute values listed in docs.

Copy link

vercel bot commented Apr 1, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Apr 3, 2024 5:30pm

@ChristoGrab ChristoGrab marked this pull request as ready for review April 1, 2024 20:55
@ChristoGrab ChristoGrab requested a review from maxi297 April 1, 2024 20:55
@octavia-squidington-iv octavia-squidington-iv requested review from a team April 1, 2024 20:57
Copy link
Contributor

@maxi297 maxi297 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one question about Sentry but I'll provide conditional approval so that I'm not blocking the release

@@ -16,7 +16,7 @@ data:
githubIssueLabel: source-sentry
icon: sentry.svg
license: MIT
maxSecondsBetweenMessages: 64800
maxSecondsBetweenMessages: 60
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking the Sentry API doc, it seems like the rate limit window is a second: We restrict both how frequently a request is made (requests per second rate limit) <...>. What is the reasoning for having 60 seconds here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was originally a bit confused by this as well, but when testing and viewing the response headers, the x-sentry-rate-limit-reset was being set to the next full minute from the request (so a request sent at 12:00:02 will return a reset time of 12:01:00, as will a request from 12:00:36). Worth noting that I ran some more tests and some of the streams do seem to have smaller windows, but the events stream in particular consistently sets this value to the next full minute.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document this somewhere as it would be interesting to know where this comes from. I'm also curious is HTTP requests to sentry can take more than a minute, maxSecondsBetweenMessages should be increased. I don't know how big the responses can be with big datasets but there is that to consider

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing, just added a comment for clarity 👍 I agree this value may actually be higher given it's not as set in stone as their docs would suggest, but my understanding is also that at the moment we don't actually do anything with this value unless it's at least 5400 seconds (or 90 minutes). Is that incorrect?

@ChristoGrab ChristoGrab merged commit 8c4095c into master Apr 3, 2024
32 checks passed
@ChristoGrab ChristoGrab deleted the christo/maxsecondsmessages branch April 3, 2024 21:26
nurikk pushed a commit to nurikk/airbyte that referenced this pull request Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants