🚀 The feature, motivation and pitch
Currently /tokenize return array of token ids
"tokens": [
50258,
50363,
16216
],
Can it also return the token strings, e.g.
"token_strs": [
"this",
" is",
" a"
],
Alternatives
No response
Additional context
No response
Before submitting a new issue...
🚀 The feature, motivation and pitch
Currently /tokenize return array of token ids
Can it also return the token strings, e.g.
Alternatives
No response
Additional context
No response
Before submitting a new issue...