Skip to content

Add custom delay inbetween requests (prevent ban) #989

@tooomm

Description

@tooomm
lychee v0.10.3

I'm seeing many [429] ... | Failed: Network error: Too Many Requests errors.

The issue is that the links we want to check are actually queries to an API which then return URLs again.
We are doing well over 1000 requests.

From their docs:

Rate Limits and Good Citizenship

We kindly ask that you insert 50 – 100 milliseconds of delay between the requests you send to the server

Submitting excessive requests to the server may result in a HTTP 429 Too Many Requests status code. Continuing to overload the API after this point may result in a temporary or permanent ban of your IP address.

Is there any default delay? Could not find information on that.

The lychee docs list various mitigations to circumvent rate limits: https://lychee.cli.rs/#/troubleshooting/rate-limits?id=rate-limits
But none really help here in order to prevent getting banned.

Metadata

Metadata

Assignees

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions