Usage: xcat [OPTIONS] <COMMAND> [ARGS]...
Arguments:
<COMMAND>
[ARGS]...
Options:
--tiktoken <TIKTOKEN> Divide stdin by chunks up to N tokens long
-h, --help Print help
-V, --version Print versionxcat reads from its stdin. For each line read, it spawns <COMMAND> with
[ARGS], and puts that single line on the spawned processes stdin.
When the --tiktoken <N> option is present, instead of dividing stdin by
newlines, it divides it by chunks up to N tokens long. It currently
uses the
cl100k_base
encoding suitable for the ChatGPT models and text-embedding-ada-002.
$ cat Cargo.toml | xcat -- wc -c
10
14
18
17
1
15
52$ echo "This is an example of using the tiktoken option." | xcat --tiktoken 5 -- cat
This is an example of
using the tiktoken option
.