Skip to content

Fix: Inability to get file size from HuggingFace downloader#45

Merged
djx-trans-n merged 2 commits intomainfrom
fix/HuggingFace-whisper-download
Jul 4, 2025
Merged

Fix: Inability to get file size from HuggingFace downloader#45
djx-trans-n merged 2 commits intomainfrom
fix/HuggingFace-whisper-download

Conversation

@djx-trans-n
Copy link
Collaborator

Fix: Inability to get file size from HuggingFace downloader

1. What does this PR do?

This PR fixes the file size retrieval logic in HuggingFaceDownloader. The key changes include:

  • In the getWhisperKitFileSize method, a fallback mechanism is now triggered when the Content-Length from a HEAD request is invalid (e.g., less than or equal to 0).
  • A new private method, getFileSizeWithRangeRequest, has been added. It sends a GET request with a Range: bytes=0-0 header to retrieve the file's Content-Range information, from which the accurate total file size is parsed.

2. Why is this PR needed?

When running the swama pull whisper-tiny command, the program would crash due to a division-by-zero error. The root cause is:

  • The HuggingFace server does not return a Content-Length header for HEAD requests on files with content compression enabled (like gzip or br), causing httpResponse.expectedContentLength to return -1.
  • This invalid value was treated as 0 and passed as the total parameter to the ProgressBar.
  • In the ProgressBar, a division-by-zero error occurred when calculating the download percentage (percent = downloaded / total), leading to a crash.

This PR resolves the issue by implementing a more reliable mechanism for fetching the file size.

3. Related Discussions

  • Fixes the runtime crash Fatal error: Double value cannot be converted to Int because it is either infinite or NaN that occurred during swama pull.

4. How was this tested?

  • Manual Testing:
    • Set the environment variable: export SWAMA_REGISTRY=HUGGING_FACE.
    • Run the command: swama pull whisper-tiny.
    • Verified that the download process no longer crashes and now displays the correct percentage progress bar.

5. Impact Analysis

  • Scope: This change only affects the HuggingFaceDownloader and has no impact on ModelScopeDownloader or other modules.
  • Risk: The risk is low. This change adds a more robust fallback for file size retrieval, improving the stability of model downloads from HuggingFace.
  • Reviewer Focus:
    • The implementation logic of the getFileSizeWithRangeRequest method.
    • The condition for triggering the fallback mechanism

@djx-trans-n djx-trans-n merged commit 4327a06 into main Jul 4, 2025
2 checks passed
@djx-trans-n djx-trans-n deleted the fix/HuggingFace-whisper-download branch July 4, 2025 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants