Skip to content

Add rustls-tls feature#1732

Merged
ArthurZucker merged 1 commit intohuggingface:mainfrom
torymur:rustls-tls-feature
Feb 11, 2025
Merged

Add rustls-tls feature#1732
ArthurZucker merged 1 commit intohuggingface:mainfrom
torymur:rustls-tls-feature

Conversation

@torymur
Copy link
Copy Markdown
Contributor

@torymur torymur commented Feb 10, 2025

Allow opting out of openssl in favor of rustls via features = ["http", "rustls-tls"].

Tokenizers with features = ["http"] depends on hf-hub:

[[package]]
name = "tokenizers"
version = "0.21.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ecededfed68a69bc657e486510089e255e53c3d38cc7d4d59c8742668ca2cae"
dependencies = [
 "aho-corasick",
 "derive_builder",
 "esaxx-rs",
 "getrandom 0.2.15",
 "hf-hub 0.3.2",
...
]

And default features of hf-hub bring openssl with it via native-tls:

[[package]]
name = "hf-hub"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b780635574b3d92f036890d8373433d6f9fc7abb320ee42a5c25897fc8ed732"
dependencies = [
 "dirs",
 "indicatif",
 "log",
 "native-tls",
 "rand",
 "serde",
 "serde_json",
 "thiserror 1.0.69",
 "ureq",
]

However, hf-hub provides a way out of openssl via rustls-tls feature: https://github.com/huggingface/hf-hub/blob/v0.4.1/Cargo.toml#L47

This change will allow to use tokenizers without openssl with features = ["http", "rustls-tls"].

Copy link
Copy Markdown
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for both the bump and adding rustls-tls

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker ArthurZucker merged commit 4f1a810 into huggingface:main Feb 11, 2025
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants