Skip to content

feature/probabilistic metrics#77

Merged
pantonante merged 13 commits intomainfrom
feature/probabilistic-metrics
Jan 2, 2025
Merged

feature/probabilistic metrics#77
pantonante merged 13 commits intomainfrom
feature/probabilistic-metrics

Conversation

@pantonante
Copy link
Contributor

@pantonante pantonante commented Jan 2, 2025

  • Adds probabilistic metrics
  • Refactor all metrics to use Jinja templates for prompts
  • Improve custom metrics (e.g., LLM-as-a-Judge) creation
  • Refactor LLMFactory
  • Improve parallelism with CPU and IO bound processing
  • Add support for model names in TokenCount metric
  • Fix issue with memory leakage when using any matching strategy in deterministic metric
  • Fix issues with Deberta models, runner and NLTK
  • Update docs (incl. update Starlight version)

@ellipsis-dev
Copy link
Contributor

ellipsis-dev bot commented Jan 2, 2025

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at help@ellipsis.dev


Generated with ❤️ by ellipsis.dev

@pantonante
Copy link
Contributor Author

  • Adds probabilistic metrics
  • Refactor all metrics to use Jinja templates for prompts
  • Improve custom metrics (e.g., LLM-as-a-Judge) creation
  • Refactor LLMFactory
  • Improve parallelism with CPU and IO bound processing
  • Add support for model names in TokenCount metric
  • Fix issue with memory leakage when using any matching strategy in deterministic metric
  • Fix issues with Deberta models, runner and NLTK
  • Update docs (incl. update Starlight version)

@pantonante pantonante merged commit 31e305e into main Jan 2, 2025
@pantonante pantonante deleted the feature/probabilistic-metrics branch January 2, 2025 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants