feat (core/middleware): Custom Tool Parser for Open Source Models#5866
feat (core/middleware): Custom Tool Parser for Open Source Models#5866
Conversation
…anced function call handling
…pdate tool-call middleware exports
…rove error handling
|
@lgrammel I'm ready for review, I've reviewed the license for relaxed-json and it looks fine. |
…into tool-call-middleware
|
I've reviewed your PR at a high level and have 2 fundamental concerns:
That said, it is very valuable to have this middleware today, and I realize that the prompt modifications are fundamental to its functionality. My suggestion would therefore be to publish the middleware as a standalone npm package in a repository that you own. We are happy to include a link to your package from the middleware page, please file a PR for the docs change when ready. |
I think the more powerful function calls you mentioned in #2 are the direction that many companies, including OpenAI, are heading in, but there are still many API providers that do not support native function calls. How about writing documentation to strongly recommend using it only for models that "do not support function calls", or adding a bypass in the middleware that does not perform transformation for models that support native function calls? If you give me feedback, I can reflect it and make modifications. I already opened the package in @ai-sdk-tool/parser, but it was rejected by another project that used it because it was an external dependency. But if you can't help it, a document suggestion sounds great enough, please consider it and let me know. |
|
I'd much prefer having this live outside the AI SDK. You could publish a package & github repo under your own account, and we'd be more than happy to link. We can also reference the middleware for providers that have less sophisticated tool calling support. Again, I think your middleware is very useful, but we are pretty strict about avoiding specific prompts or prompt modifications in the AI SDK itself. |
|
I understand perfectly, I will move the above v5 implementation out and post a new PR with proper documentation, thanks for reviewing |
Background
#3521
Summary
Transform the user's function call schema and input into a prompt, and parse the LLM's response into JSON to implement the function call as a middleware. This unlocks the function call ability for models like ollama's model or openrouter's gemma that do not support function calls.
Future Work
Change
createToolMiddleware,hermesToolMiddleware,gemmaToolMiddleware)