The OpenAI API allows you to submit a list of tokens (rather than a string) as your prompt. This can be useful in some scenarios, e.g. for benchmarking where you want to be sure you know exactly how many tokens are in the prompt. Also pretty easy to implement (just skip the tokenizer).
The OpenAI API allows you to submit a list of tokens (rather than a string) as your prompt. This can be useful in some scenarios, e.g. for benchmarking where you want to be sure you know exactly how many tokens are in the prompt. Also pretty easy to implement (just skip the tokenizer).