Fix bos/eos tokens in bpe tokenizer in c++ #4112
Description
Those tokens are hard-coded for llama and doesn't work for other models. They should be set dynamically based on the config of the tokenizer to support HF models.
{{ message }}
Those tokens are hard-coded for llama and doesn't work for other models. They should be set dynamically based on the config of the tokenizer to support HF models.