The previous implementation removed HEAD request validation before downloading models from Hugging Face, resulting in expensive failed downloads instead of lightweight pre-flight checks, particularly for 401 Unauthorized errors.

Changes

• Added validate_url_with_head() function - Performs HEAD requests before downloads, raises HTTPError for 4xx/5xx responses to enable candidate retry
• Refactored file_downloader() - Uses shared validation function to eliminate duplication
• Validates URLs before both download paths:
  • DiffusionPipeline.download() - checks model_index.json accessibility
  • hf_hub_download() - checks file URL accessibility
• Uses hf_hub_url() - Ensures consistent URL construction and respects revision parameter
• Case-insensitive header handling - Prevents duplicate Authorization headers across different casings

Implementation

def validate_url_with_head(
    url: str, 
    token: Optional[str] = None, 
    headers: Optional[Dict] = None,
    timeout: int = 2
) -> None:
    """Validates URL accessibility via HEAD request. Raises HTTPError for 4xx/5xx."""
    if headers is None:
        headers = {}
    else:
        headers = dict(headers)  # Avoid mutating caller's dict
    
    if token and not any(k.lower() == "authorization" for k in headers):
        headers["Authorization"] = f"Bearer {token}"
    
    try:
        response = requests.head(url, headers=headers, allow_redirects=True, timeout=timeout)
        response.raise_for_status()  # Raises for 401 and other errors
    except requests.HTTPError:
        raise
    except requests.exceptions.RequestException as e:
        logger.info(f"HEAD check failed (continuing to download): {url} -> {e}")

Non-HTTPError request failures (timeouts, network errors) are logged but don't block downloads, as the full download may still succeed.

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.