Suport for tool functions returning ToolResult image values#88
Conversation
…r ToolResult values from tool functions.
|
Found 1 changed notebook. Review the changes at https://app.gitnotebooks.com/AnswerDotAI/claudette/pull/88 |
jph00
left a comment
There was a problem hiding this comment.
Thanks! Couple of minor suggestions to consider.
|
@austinvhuang could you add a markdown cell above this describing what it is and what it's for? Quite a few folks outside AAI use Claudette too so I want to ensure we keep things well documented. |
ToolResult image values
|
Updated this PR so it's no longer only handles the no-op case of It now supports images and supports When mk_funcres / mk_toolres get a tool function return value of ToolResult, if the result_type is an image then it inserts the image tool result content here: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/implement-tool-use#example-of-tool-result-with-images. |
This PR is a prerequisite for AnswerDotAI/dialoghelper#21
This PR adds the definition of a special
ToolResulttype which carries a mime-type-likeresult_typemetadata. In the case ofresult_typebeing str, it simply unwraps the.dataand makes a tool result with.dataas its string value.In the future
ToolResultwill allow for other return types to be specially handled (eg media types need their own handling to be represented in the message history correctly), but for now this implementation is just a no-op on the current behavior with thisToolResulttype instead of rawstringificaton for every tool function return value.Eventually these responsibilities will all get get migrated to the litellm wrapper, but we implement this here now so that we can move ahead with upstream changes without breaking anything.