Conversation
There was a problem hiding this comment.
❌ Changes requested. Reviewed everything up to abedd8b in 36 seconds
More details
- Looked at
88lines of code in3files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_arh5EbQPeJG41abu
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
|
|
||
| from continuous_eval.metrics.base import Metric | ||
|
|
||
| _CHARACTERS_PER_TOKEN = 4.0 |
There was a problem hiding this comment.
Consider using a more dynamic or configurable approach for _CHARACTERS_PER_TOKEN instead of a fixed value to better accommodate different languages or text formats.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on d39208a in 16 seconds
More details
- Looked at
72lines of code in2files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. docs/src/content/docs/metrics/overview.md:96
- Draft comment:
The documentation for theTokenCountmetric is accurate and aligns with the detailed documentation intoken_count.md. It correctly lists the metric under the Deterministic category for Retrieval metrics and provides a brief definition and the required input. - Reason this comment was not posted:
Confidence changes required:0%
The documentation for the TokenCount metric in the overview.md file is accurate and aligns with the detailed documentation in token_count.md. It correctly lists the metric under the Deterministic category for Retrieval metrics and provides a brief definition and the required input. No issues or inconsistencies are found in this documentation snippet.
Workflow ID: wflow_OHoTzzysnHKRafsY
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on dd610a6 in 34 seconds
More details
- Looked at
86lines of code in2files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. README.md:18
- Draft comment:
The PR description mentions the addition of theTokenCountmetric, but the diff does not include changes to any Python files such ascontinuous_eval/metrics/retrieval/tokens.py. Please ensure that all relevant changes are included in the PR. - Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_dUTdZ3I9ja94hUpX
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
Summary:
Added
TokenCountmetric to count tokens in retrieved context usingtiktokenencoder or an approximation, with tests and documentation.Key points:
TokenCountmetric incontinuous_eval/metrics/retrieval/tokens.py.continuous_eval/metrics/retrieval/__init__.pyto includeTokenCount.TokenCountclass to count tokens usingtiktokenencoder or an approximation.TokenCountintests/retrieval_metrics_test.py.TokenCountindocs/src/content/docs/metrics/Retrieval/Deterministic/token_count.md.docs/src/content/docs/metrics/overview.mdto includeTokenCount.Generated with ❤️ by ellipsis.dev