Skip to content

Introduce LocalScore CLI#734

Merged
cjpais merged 1 commit intomozilla-ai:mainfrom
cjpais:LocalScore
Apr 2, 2025
Merged

Introduce LocalScore CLI#734
cjpais merged 1 commit intomozilla-ai:mainfrom
cjpais:LocalScore

Conversation

@cjpais
Copy link
Copy Markdown
Collaborator

@cjpais cjpais commented Apr 1, 2025

This PR introduces LocalScore, an open-source benchmarking tool that measures LLM performance on local hardware. Key features include:

  • Measures prompt processing speed, generation speed, and time-to-first-token
  • Supports CPU/GPU benchmarking across Windows/macOS/Linux
  • Provides a unified scoring system via geometric mean of metrics
  • Optional submission to public leaderboard at localscore.ai
  • Multiple execution options (standalone binary, llamafile integration)

The tool collects anonymous hardware specs and performance metrics to help users compare different setups. Includes CLI interface with options for:

  • GPU backend selection
  • Multiple test repetitions
  • Result submission control

Built as part of the Mozilla Builders program.

Usage Examples:

# Basic benchmark
./localscore -m model.gguf

# CPU-only test with extended repetitions
./localscore -m model.gguf --cpu --extended

# Automatic result submission
./localscore -m model.gguf -y

Further details can be found in the README.

@cjpais cjpais merged commit 11fe18e into mozilla-ai:main Apr 2, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant