README.md contains non-functional code #1633
Description
The README.md (and the corresponding landing page of the documentation) contains non-functional code. Specifically the following:
Loading a pretrained tokenizer from the Hub
use tokenizers::tokenizer::{Result, Tokenizer};
fn main() -> Result<()> {
# #[cfg(feature = "http")]
# {
let tokenizer = Tokenizer::from_pretrained("bert-base-cased", None)?;
let encoding = tokenizer.encode("Hey there!", false)?;
println!("{:?}", encoding.get_tokens());
# }
Ok(())
}Here, the function to load the tokenizer from a pretrained model is not available in 'Tokenizer':
error[E0599]: no function or associated item named `from_pretrained` found for struct `Tokenizer` in the current scope
--> src/main.rs:30:32
|
30 | let tokenizer = Tokenizer::from_pretrained("bert-base-cased", None);
| ^^^^^^^^^^^^^^^ function or associated item not found in `Tokenizer`