Add option to prevent downloading the entirety of a file #76
Description
In one of our projects, we include various links to the latest binary release artefacts of our project within the documentation, as well as many links to other fairly large binaries. This can lead to the plugin bringing total mkdocs build time to 15+ minutes.
To validate a URL, you needn't download the entire file, just a few bytes will do. It would be great if a configuration parameter were added that limited the download to the first N bytes of a file.
I implemented the following quick hack locally to see the how much difference it would make:
diff --git a/htmlproofer/plugin.py b/htmlproofer/plugin.py
index 59d62fa..3f98e87 100644
--- a/htmlproofer/plugin.py
+++ b/htmlproofer/plugin.py
@@ -146,7 +146,8 @@ class HtmlProoferPlugin(BasePlugin):
@lru_cache(maxsize=1000)
def resolve_web_scheme(self, url: str) -> int:
try:
- response = self._session.get(url, timeout=URL_TIMEOUT)
+ headers = {"Range": "bytes=0-100"}
+ response = self._session.get(url, timeout=URL_TIMEOUT, headers=headers)
return response.status_code
except requests.exceptions.Timeout:
return 504The build time went from:
INFO - Documentation built in 982.92 seconds
to:
INFO - Documentation built in 39.47 seconds