Merged
Conversation
Fix issue relari-ai#69 where LLMs other than OpenAI APIs are not being called
Contributor
There was a problem hiding this comment.
👍 Looks good to me! Reviewed everything up to 52de4da in 35 seconds
More details
- Looked at
14lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. continuous_eval/metrics/generation/text/llm_based.py:39
- Draft comment:
The change correctly passes the model instance toLLMBasedContextCoverage, ensuring that the same LLM is used for both faithfulness and context coverage metrics. This is a necessary fix to maintain consistency and functionality across different LLMs. - Reason this comment was not posted:
Confidence changes required:0%
The PR aims to fix an issue where LLMs other than OpenAI APIs are not being called by passing the model instance to the LLMBasedContextCoverage class. The change is made in the LLMBasedFaithfulness class where the model is now explicitly passed to the LLMBasedContextCoverage instance. This is a logical fix as it ensures that the same model used in the LLMBasedFaithfulness class is also used for context coverage, maintaining consistency across the evaluation metrics.
Workflow ID: wflow_h7pGXOOU7Qo3Vq4h
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
Contributor
|
@kelvinchanwh Thanks for contributing! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix issue #69 where LLMs other than OpenAI APIs are not being called
Summary:
Fixes model passing in
LLMBasedFaithfulnessto ensureLLMBasedContextCoveragecorrectly utilizes the specified LLM model.Key points:
LLMBasedFaithfulnessclass incontinuous_eval/metrics/generation/text/llm_based.py.LLMBasedContextCoveragein__call__method.Generated with ❤️ by ellipsis.dev