Skip to content

llama : add benchmark example#2626

Merged
slaren merged 19 commits intomasterfrom
llama-benchmark
Aug 18, 2023
Merged

llama : add benchmark example#2626
slaren merged 19 commits intomasterfrom
llama-benchmark

Conversation

@slaren
Copy link
Member

@slaren slaren commented Aug 15, 2023

Adds an example for running performance benchmarks. Multiple values can be specified for each option, and it will run the matrix of all of them. Supports output to csv, json or markdown.

Example markdown output:

model backend n_gpu_layers test t/s
LLaMA 7B mostly Q4_0 CUDA 99 pp 512 2242.06 ± 24.26
LLaMA 7B mostly Q4_0 CUDA 99 tg 128 43.09 ± 0.41

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

high priority Very important issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants