Deal with GitHub's secondary rate limit #2355

fthomas · 2021-11-15T14:24:22Z

My public Scala Steward instance has repeatedly hit GitHub's secondary rate limit recently when creating PRs:

org.scalasteward.core.util.UnexpectedResponse: uri: https://api.github.com/repos/***/***/pulls
method: POST
status: 403 Forbidden
headers: Headers(
  access-control-allow-origin: *,
  access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
  content-security-policy: default-src 'none'
  content-type: application/json; charset=utf-8
  date: Wed, 10 Nov 2021 04:26:48 GMT
  referrer-policy: origin-when-cross-origin
  strict-origin-when-cross-origin
  server: GitHub.com
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  vary: Accept-Encoding, Accept, X-Requested-With
  x-accepted-oauth-scopes: 
  x-content-type-options: nosniff
  x-frame-options: deny
  x-github-media-type: github.v3
  x-github-request-id: B1C8:AD73:2D8A937:2E660AE:618B4A03
  x-oauth-scopes: delete_repo
  notifications, repo, workflow
  x-ratelimit-limit: 5000
  x-ratelimit-remaining: 4290
  x-ratelimit-reset: 1636520514
  x-ratelimit-resource: core
  x-ratelimit-used: 710
  x-xss-protection: 0
  )
body: {
  "message":"You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later.",
  "documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#secondary-rate-limits"
}

GitHub docs about the secondary rate limit are here:

The response doesn't include a Retry-After header, so we don't know how long Scala Steward should wait before creating the next PR.

The text was updated successfully, but these errors were encountered:

Instead of `(Boolean, List[Update.Single])` where the first element designate if the second is empty, we use `Option[Nel[UpdateState.WithUpdate]]` instead. This is a little bit nicer since we don't need to interpret the `Boolean` but just traverse the `Option`. Having the `UpdateState`s available in `StewardAlg` will probably also come in handy when dealing with #2355. I imagine that some kind of rate limiter in `StewardAlg` will solve that issue.

d-g-n · 2022-01-24T15:55:19Z

Hello, we're trying to run a local version of scala-steward against approx 30-35 repos, but given how out of date they are, we're frequently getting body: {"message":"You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later.","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#secondary-rate-limits"},

i acknowledge that github isn't returning the correct header but is there any config or setting we can tweak to hopefully mitigate this until a full fix is in place?

htmldoug · 2022-03-03T19:58:27Z

Looks like github returns Retry-After now!

When you have been limited, use the Retry-After response header to slow down. The value of the Retry-After header will always be an integer, representing the number of seconds you should wait before making requests again. For example, Retry-After: 30 means you should wait 30 seconds before sending more requests.

https://docs.github.com/en/rest/guides/best-practices-for-integrators#dealing-with-secondary-rate-limits

Actual response I received:

org.scalasteward.core.util.UnexpectedResponse: uri: https://api.github.com/repos/***/***/pulls?head=****/***%3Aupdate/google-api-services-sheets-v4-rev20220221-1.32.1&base=main&state=all
method: GET
status: 403 Forbidden
headers:
  access-control-allow-origin: *
  access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
  content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubusercontent.com www.githubstatus.com collector.githubapp.com collector.github.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events translator.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src render.githubusercontent.com viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com collector.github.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com objects-origin.githubusercontent.com secured-user-images.githubusercontent.com/; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/
  content-type: application/json; charset=utf-8
  date: Thu, 03 Mar 2022 16:14:51 GMT
  expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
  referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
  retry-after: 60
  server: GitHub.com
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  vary: Accept-Encoding, Accept, X-Requested-With
  x-content-type-options: nosniff
  x-frame-options: deny
  x-github-media-type: github.v3; format=json
  x-github-request-id: AEC9:5D27:43638DF:7FCD679:6220E961
  x-xss-protection: 0
body: {
  "documentation_url": "https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits",
  "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again."
}

fthomas · 2022-03-04T05:23:10Z

You got this header in response to a GET for listing PRs. I posted a response to a POST for creating a PR. The documentation you cited also mentions that there is no RetryAfter for creating PRs. So I doubt that GitHub has changed anything here.

htmldoug · 2022-03-04T20:56:33Z

Ah, good point. So #2540 would only fix a subset of these issues. The majority of the failures I'm seeing are actually from POST .../forks, so I can at least take care of those. I'll update my PR so it keeps this issue open.

org.scalasteward.core.util.UnexpectedResponse: uri: https://api.github.com/repos/***/***/forks
method: POST
status: 403 Forbidden
headers:
  access-control-allow-origin: *
  access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
  content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubusercontent.com www.githubstatus.com collector.githubapp.com collector.github.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events translator.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src render.githubusercontent.com viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com collector.github.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com objects-origin.githubusercontent.com secured-user-images.githubusercontent.com/; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/
  content-type: application/json; charset=utf-8
  date: Thu, 03 Mar 2022 16:14:51 GMT
  expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
  referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
  retry-after: 60
  server: GitHub.com
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  vary: Accept-Encoding, Accept, X-Requested-With
  x-content-type-options: nosniff
  x-frame-options: deny
  x-github-media-type: github.v3; format=json
  x-github-request-id: AEC9:5D27:4363908:7FCD6C5:6220E97B
  x-xss-protection: 0
body: {
  "documentation_url": "https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits",
  "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again."
}

exoego · 2022-04-19T00:51:11Z

Regarding #2540, code LGTM 👍
However, secondary rate limit for POST does not return Retry-After header, so I am afraid that adding Retry-After is going to be an overhead.

From the document,

If you're making a large number of POST, PATCH, PUT, or DELETE requests for a single user or client ID, wait at least one second between each request.

I wonder adding sleep(1.second) probably mitigate secondary rate limit.

htmldoug · 2022-04-20T20:44:24Z

Thanks for the review of #2540!

The document also says:

When you have been limited, use the Retry-After response header to slow down. The value of the Retry-After header will always be an integer, representing the number of seconds you should wait before making requests again. For example, Retry-After: 30 means you should wait 30 seconds before sending more requests.

Implementing both of the github best practices seems reasonable to me--probably best as separate PRs.

I am afraid that adding Retry-After is going to be an overhead.

It's true that not all requests would return Retry-After. I haven't benchmarked it, although I expect that checking for the presence of the header should have negligible CPU/memory cost. Additionally, I'd be surprised if this code path were hot enough to optimize.

Did you mean this as a blocker for #2540? Is there anything else I can do to address concerns so that this can be merged?

We have a long list of repos we run scala-steward against and it's currently failing for us at the end of that list. I'd love to see a fix merged one way or another.

exoego · 2022-06-15T01:48:02Z

I've merged scala-steward-org/scala-steward-action#335
It does not cover all cases but helps some cases definitely.

I keep this issue open until we have solutions for other cases.

rossabaker · 2023-01-25T17:46:43Z

We are still struggling with this. I wonder if the Retry-After from the other endpoints gives us some empirical guess as to what would be an appropriate time to sleep? Like, is it always a fixed duration, or is it (more likely) some token bucket and varies wildly from run to run?

rossabaker · 2023-01-25T17:51:56Z

Or if we know whether we're talking on the order of resting for a couple seconds vs "a few minutes" like the message. A couple seconds seems worthwhile. "A few minutes" could really add up on runners where we pay by the minute.

fthomas · 2023-01-27T07:06:40Z

I don't maintain an instance that works with GitHub anymore, but still have logs of @scala-steward from March and April 2022. I looked at four occasions where it hit the secondary rate limit and it was able to create PRs again after ~ 40, 30, 8, and 40 minutes.

fthomas · 2023-01-27T07:12:48Z

I don't know if waiting for a few seconds or minutes between PRs would have meant that it would have never hit the secondary rate limit.

fthomas mentioned this issue Dec 12, 2021

Let Scala Steward send upgrades to Scala.js 1.8.0. #2402

Merged

fthomas mentioned this issue Dec 24, 2021

Change return type of PruningAlg#needsAttention #2427

Merged

mzuehlke mentioned this issue Jan 28, 2022

Parallelize UpdateAlg.findUpdates #2487

Merged

fthomas mentioned this issue Feb 6, 2022

Add SecondaryRateLimitExceeded exception #2509

Merged

exoego added the enhancement New feature or request label Feb 9, 2022

htmldoug mentioned this issue Mar 3, 2022

Retry on Retry-After response header #2540

Merged

exoego mentioned this issue Mar 20, 2022

Need to handle rate limits scala-steward-org/scala-steward-action#335

Closed

fthomas mentioned this issue Oct 14, 2022

Is scala steward still running #2733

Closed

fthomas added the cat:forge label Jan 15, 2023

fthomas mentioned this issue Jan 19, 2023

More strategies for (org-level) throttling of PRs #2934

Open

fthomas linked a pull request Feb 11, 2023 that will close this issue

Add simple instance-wide PR throttle #2970

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deal with GitHub's secondary rate limit #2355

Deal with GitHub's secondary rate limit #2355

fthomas commented Nov 15, 2021 •

edited

Loading

d-g-n commented Jan 24, 2022 •

edited

Loading

htmldoug commented Mar 3, 2022 •

edited

Loading

fthomas commented Mar 4, 2022

htmldoug commented Mar 4, 2022

exoego commented Apr 19, 2022

htmldoug commented Apr 20, 2022 •

edited

Loading

exoego commented Jun 15, 2022 •

edited

Loading

rossabaker commented Jan 25, 2023

rossabaker commented Jan 25, 2023

fthomas commented Jan 27, 2023

fthomas commented Jan 27, 2023

Deal with GitHub's secondary rate limit #2355

Deal with GitHub's secondary rate limit #2355

Comments

fthomas commented Nov 15, 2021 • edited Loading

d-g-n commented Jan 24, 2022 • edited Loading

htmldoug commented Mar 3, 2022 • edited Loading

fthomas commented Mar 4, 2022

htmldoug commented Mar 4, 2022

exoego commented Apr 19, 2022

htmldoug commented Apr 20, 2022 • edited Loading

exoego commented Jun 15, 2022 • edited Loading

rossabaker commented Jan 25, 2023

rossabaker commented Jan 25, 2023

fthomas commented Jan 27, 2023

fthomas commented Jan 27, 2023

fthomas commented Nov 15, 2021 •

edited

Loading

d-g-n commented Jan 24, 2022 •

edited

Loading

htmldoug commented Mar 3, 2022 •

edited

Loading

htmldoug commented Apr 20, 2022 •

edited

Loading

exoego commented Jun 15, 2022 •

edited

Loading