-
-
Notifications
You must be signed in to change notification settings - Fork 197
Description
lychee v0.10.3
I'm seeing many [429] ... | Failed: Network error: Too Many Requests errors.
The issue is that the links we want to check are actually queries to an API which then return URLs again.
We are doing well over 1000 requests.
From their docs:
Rate Limits and Good Citizenship
We kindly ask that you insert 50 – 100 milliseconds of delay between the requests you send to the server
Submitting excessive requests to the server may result in a
HTTP 429 Too Many Requestsstatus code. Continuing to overload the API after this point may result in a temporary or permanent ban of your IP address.
Is there any default delay? Could not find information on that.
The lychee docs list various mitigations to circumvent rate limits: https://lychee.cli.rs/#/troubleshooting/rate-limits?id=rate-limits
But none really help here in order to prevent getting banned.