fixed double counting issue for precision / average precision#55
Merged
pantonante merged 1 commit intomainfrom Apr 2, 2024
Merged
fixed double counting issue for precision / average precision#55pantonante merged 1 commit intomainfrom
pantonante merged 1 commit intomainfrom
Conversation
Contributor
There was a problem hiding this comment.
👍 Looks good to me!
- Reviewed the entire pull request up to acded6d
- Looked at
50lines of code in3files - Took 1 minute and 11 seconds to review
More info
- Skipped
0files when reviewing. - Skipped posting
3additional comments because they didn't meet confidence threshold of50%.
1. continuous_eval/metrics/retrieval/precision_recall_f1.py:33:
- Assessed confidence :
50% - Comment:
The conversion of 'gt_components' to a set might lead to loss of information if the ground truth context is supposed to have duplicates. Please ensure that this is not the case. - Reasoning:
The changes in precision_recall_f1.py seem to be addressing the issue of double counting in precision calculation. The author has introduced a flag 'ret_component_matched' to ensure that a retrieved component is only counted once even if it matches with multiple ground truth components. This seems to be a logical fix. However, the conversion of 'gt_components' to a set might lead to loss of information if the ground truth context is supposed to have duplicates. I need to check the rest of the codebase to understand if this is the case.
2. continuous_eval/metrics/retrieval/ranked.py:37:
- Assessed confidence :
0% - Comment:
The 'break' statement ensures that a retrieved chunk is only counted once even if it matches with multiple ground truth chunks. This seems to be a logical fix for the double counting issue. - Reasoning:
The changes in ranked.py seem to be addressing the issue of double counting in the calculation of average precision. The author has introduced a 'break' statement to ensure that a retrieved chunk is only counted once even if it matches with multiple ground truth chunks. This seems to be a logical fix.
3. pyproject.toml:24:
- Assessed confidence :
50% - Comment:
The protobuf version has been downgraded from 4.25.1 to 4.23.4. Please ensure that this doesn't affect other parts of the codebase that rely on features available in the newer version of protobuf. - Reasoning:
The change in pyproject.toml seems to be a downgrade of the protobuf version from 4.25.1 to 4.23.4. This might be due to compatibility issues with other dependencies. However, it's important to ensure that this downgrade doesn't affect other parts of the codebase that rely on features available in the newer version of protobuf.
Workflow ID: wflow_goALjd5d5StLHuB0
Not what you expected? You can customize the content of the reviews using rules. Learn more here.
pantonante
approved these changes
Apr 2, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
This PR fixes a double counting issue in precision and average precision calculations and downgrades the
protobufversion.Key points:
precision_recall_f1.pyto prevent double counting of retrieved components.ranked.pyto break the loop after finding the first relevant match.protobufversion inpyproject.toml.Generated with ❤️ by ellipsis.dev