Removed token count of user message#836
Closed
tdechanterac wants to merge 1 commit intomicrosoft:mainfrom
Closed
Conversation
Contributor
|
@tdechanterac Thanks for figuring this out! The code in ChatPlugin.cs is quite "dense" (to use an euphemism). I had to spend quite some time reading it to fully understand the impact of that one-line change. I think your fix would make the code better. But the code would still not be guaranteed to include the user prompt because a big enough portion of the token budget could be taken by memories. So I came up with bigger change that I think is "safer" and more closely respects the original intent of the code: #839 Please take a look at it and feel free to comment on it. This new PR would make your change no longer needed. While I was at it, I made a few related changes to make ChatPlugin.cs more readable too. |
Author
|
Thank you ! |
github-merge-queue bot
pushed a commit
that referenced
this pull request
Mar 9, 2024
### Motivation and Context The user message is not added to the GPT completion request when the token count of the prompt exceeds 50% of the configuration. This fix is inspired by #836 ### Description Fix budget calculation and clarify / simplify code. ### Contribution Checklist - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [Contribution Guidelines](https://github.com/microsoft/chat-copilot/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/chat-copilot/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
teamleader-dev
pushed a commit
to vlink-group/chat-copilot
that referenced
this pull request
Oct 7, 2024
### Motivation and Context The user message is not added to the GPT completion request when the token count of the prompt exceeds 50% of the configuration. This fix is inspired by microsoft#836 ### Description Fix budget calculation and clarify / simplify code. ### Contribution Checklist - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [Contribution Guidelines](https://github.com/microsoft/chat-copilot/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/chat-copilot/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
kb0039
pushed a commit
to aaronba/chat-copilot
that referenced
this pull request
Jan 8, 2025
### Motivation and Context The user message is not added to the GPT completion request when the token count of the prompt exceeds 50% of the configuration. This fix is inspired by microsoft#836 ### Description Fix budget calculation and clarify / simplify code. ### Contribution Checklist - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [Contribution Guidelines](https://github.com/microsoft/chat-copilot/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/chat-copilot/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
The user message is not added to the GPT completion request when the token count of the prompt exceeds 50% of the configuration.
We can now use large prompts in ChatCopilot.
Description
The method
GetChatContextTokenLimitfromChatPluginwas used to count the user message, indicating that it needed to be added. However, it was never added directly; instead, it was added by theGetAllowedChatHistoryAsyncmethod.For example, if you have a configuration with:
CompletionTokenLimit: 4096ResponseTokenLimit: 1024and the following prompt:
You end up with a completion request that only contains the user intent, and the bot responds with:
Note: We could also address this issue by adding the user message directly into the chat history and ensuring that the message is not added a second time in
GetAllowedChatHistoryAsync.