Skip to content

feat (core/middleware): Custom Tool Parser for Open Source Models#5866

Closed
minpeter wants to merge 12 commits intovercel:v5from
minpeter:tool-call-middleware
Closed

feat (core/middleware): Custom Tool Parser for Open Source Models#5866
minpeter wants to merge 12 commits intovercel:v5from
minpeter:tool-call-middleware

Conversation

@minpeter
Copy link
Copy Markdown
Contributor

@minpeter minpeter commented Apr 21, 2025

Background

#3521

Summary

Transform the user's function call schema and input into a prompt, and parse the LLM's response into JSON to implement the function call as a middleware. This unlocks the function call ability for models like ollama's model or openrouter's gemma that do not support function calls.

Future Work

  • Provide basic middleware for more model types
  • Could add documentation on how to use the middleware
  • Add tests for tool call middleware and relaxed-json

Change

  • Added 2 middleware examples (I used Openrouter for convenience, but it looks like you could replace it with gemma from the google provider)
  • Exposes three middlewares for function call functionality. (createToolMiddleware, hermesToolMiddleware, gemmaToolMiddleware)
  • Port the implementation of relaxed-json to TypeScript and add it to the utils. (No alternative found as of now)

Comment thread packages/ai/core/middleware/tool-call-middleware.ts Outdated
@minpeter minpeter requested a review from lgrammel April 21, 2025 11:46
@minpeter minpeter marked this pull request as ready for review April 21, 2025 11:46
@minpeter
Copy link
Copy Markdown
Contributor Author

@lgrammel I'm ready for review, I've reviewed the license for relaxed-json and it looks fine.

@lgrammel
Copy link
Copy Markdown
Collaborator

I've reviewed your PR at a high level and have 2 fundamental concerns:

  1. The PR includes prompt modifications, something we try to avoid having in the AI SDK (with very minor exceptions).
  2. Longer term, the AI providers will move towards stronger tool call support, rendering this middleware unnecessary or even counter-productive (see e.g. https://cookbook.openai.com/examples/gpt4-1_prompting_guide#tool-calls ).

That said, it is very valuable to have this middleware today, and I realize that the prompt modifications are fundamental to its functionality. My suggestion would therefore be to publish the middleware as a standalone npm package in a repository that you own. We are happy to include a link to your package from the middleware page, please file a PR for the docs change when ready.

@minpeter
Copy link
Copy Markdown
Contributor Author

minpeter commented Apr 22, 2025

I've reviewed your PR at a high level and have 2 fundamental concerns:

  1. The PR includes prompt modifications, something we try to avoid having in the AI SDK (with very minor exceptions).
  2. Longer term, the AI providers will move towards stronger tool call support, rendering this middleware unnecessary or even counter-productive (see e.g. https://cookbook.openai.com/examples/gpt4-1_prompting_guide#tool-calls ).

That said, it is very valuable to have this middleware today, and I realize that the prompt modifications are fundamental to its functionality. My suggestion would therefore be to publish the middleware as a standalone npm package in a repository that you own. We are happy to include a link to your package from the middleware page, please file a PR for the docs change when ready.

I think the more powerful function calls you mentioned in #2 are the direction that many companies, including OpenAI, are heading in, but there are still many API providers that do not support native function calls. How about writing documentation to strongly recommend using it only for models that "do not support function calls", or adding a bypass in the middleware that does not perform transformation for models that support native function calls?

If you give me feedback, I can reflect it and make modifications. I already opened the package in @ai-sdk-tool/parser, but it was rejected by another project that used it because it was an external dependency.

But if you can't help it, a document suggestion sounds great enough, please consider it and let me know.

@lgrammel
Copy link
Copy Markdown
Collaborator

lgrammel commented Apr 22, 2025

I'd much prefer having this live outside the AI SDK. You could publish a package & github repo under your own account, and we'd be more than happy to link. We can also reference the middleware for providers that have less sophisticated tool calling support. Again, I think your middleware is very useful, but we are pretty strict about avoiding specific prompts or prompt modifications in the AI SDK itself.

@minpeter
Copy link
Copy Markdown
Contributor Author

I understand perfectly, I will move the above v5 implementation out and post a new PR with proper documentation, thanks for reviewing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants