Originally posted by @jcharaoui in #27

I'm having a similar issue where a website I want to check has a lot of download links, but lychee is struggling to complete checking all the links and timing out on many of them.

Example URL: https://dist.torproject.org/torbrowser/16.0a2/tor-browser-linux-x86_64-16.0a2.tar.xz

When checking this link using the GET method, it downloads the whole file, while using the HEAD method the check completes in a matter of milliseconds.

The inconsistency observed is likely due to variations in the configurations of the various HTTP daemons out there.

It looks like the entire response body is cached and this causes the entire body to be read, even when it's not needed (i.e., when fragment checking is disabled).