Skip to content

fix: Nvidia - fix structured output syntax#3058

Merged
anakin87 merged 10 commits intodeepset-ai:mainfrom
Akash504-ai:fix/nvidia-guided-json
Mar 30, 2026
Merged

fix: Nvidia - fix structured output syntax#3058
anakin87 merged 10 commits intodeepset-ai:mainfrom
Akash504-ai:fix/nvidia-guided-json

Conversation

@Akash504-ai
Copy link
Copy Markdown
Contributor

@Akash504-ai Akash504-ai commented Mar 26, 2026

Related Issues

Proposed Changes

The guided_json schema passed via generation_kwargs["extra_body"] was incorrectly nested when sent to the NVIDIA API.

This resulted in payloads like:

{
  "extra_body": {
    "extra_body": {
      "guided_json": {...}
    }
  }
}

Because of this, the model ignored the schema and returned non-JSON output, causing JSONDecodeError in tests.

Fix

  • Flatten nested extra_body before making API calls
  • Apply fix in both run and run_async
  • Use explicit parent class calls instead of super() to avoid issues with the @component decorator

This ensures guided_json is correctly forwarded to the NVIDIA API.

How did you test it?

  • Ran all NVIDIA integration tests:
pytest integrations/nvidia/tests -vv

  • Result:

    • 180 tests passed
    • 17 skipped (expected live tests)

Notes for the reviewer

  • The fix is scoped only to the NVIDIA integration
  • No changes were made to core Haystack classes
  • This improves correctness of structured generation without altering existing APIs

Checklist

  • I have read the contributors guidelines and code of conduct
  • I have updated the related issue with new insights and changes
  • I added unit tests and updated the docstrings (no new tests needed, existing tests now pass)
  • I've used a conventional commit type (fix:)

@Akash504-ai Akash504-ai requested a review from a team as a code owner March 26, 2026 20:19
@Akash504-ai Akash504-ai requested review from davidsbatista and removed request for a team March 26, 2026 20:19
@github-actions github-actions Bot added the type:documentation Improvements or additions to documentation label Mar 26, 2026
@anakin87 anakin87 requested review from anakin87 and removed request for davidsbatista March 27, 2026 06:56
anakin87
anakin87 requested changes Mar 27, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@anakin87 anakin87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I run the test locally with your changes, the integration test using NVIDIA_API_KEY still fails. In this PR, it does not run for security reasons.

Are you sure that it works for you? Please investigate more...

@anakin87
Copy link
Copy Markdown
Member

anakin87 commented Mar 27, 2026

This seems to work instead

import json

from haystack.dataclasses import ChatMessage

from haystack_integrations.components.generators.nvidia import NvidiaChatGenerator

json_schema = {
    "type": "object",
    "properties": {"title": {"type": "string"}, "rating": {"type": "number"}},
    "required": ["title", "rating"],
}

prompt = (
    "Return the title and the rating based on the following movie review.\n"
    "Review: Inception is a really well made film. I rate it four stars out of five."
)

component = NvidiaChatGenerator(
    model="meta/llama-3.1-70b-instruct",
    generation_kwargs={
        "response_format": {
            "type": "json_schema",
            "json_schema": {"name": "movie_review", "schema": json_schema},
        },
    },
)

results = component.run([ChatMessage.from_user(prompt)])
text = results["replies"][0].text
print(repr(text))
output = json.loads(text)
print(f"Parsed: title={output['title']}, rating={output['rating']}")

Feel free to experiment more and maybe we can steer this PR to just show this new syntax for NVIDIA.

Let me know...

@Akash504-ai
Copy link
Copy Markdown
Contributor Author

Akash504-ai commented Mar 27, 2026

Thanks for checking this!
You are right, my current fix ensures that the schema is passed correctly but it seems the NVIDIA API expects the newer response_format structure instead of guided_json.
I'll update the implementation to align with the new syntax and verify it against the live API behavior.

Will push an updated version shortly.

@Akash504-ai
Copy link
Copy Markdown
Contributor Author

Akash504-ai commented Mar 27, 2026

I have updated the implementation to follow the newer NVIDIA API syntax.This change is now applied consistently in both run and run_async.
All NVIDIA integration tests pass locally (180 passed, 17 skipped).

Please let me know if this aligns better with the expected approach or if further adjustments are needed!

anakin87
anakin87 requested changes Mar 27, 2026
View reviewed changes
Comment thread ...rations/nvidia/src/haystack_integrations/components/generators/nvidia/chat/chat_generator.py Outdated
@Akash504-ai
Copy link
Copy Markdown
Contributor Author

Akash504-ai commented Mar 28, 2026

Thanks for the feedback!
All CI checks are now passing. All NVIDIA integration tests pass locally (180 passed, 17 skipped). Please let me know if anything else needs adjustment.

@anakin87 anakin87 changed the title fix(nvidia): correct guided_json handling in extra_body for structured output fix: Nvidia - fix structured output syntax Mar 30, 2026
anakin87
anakin87 approved these changes Mar 30, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@anakin87 anakin87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

I applied some minor changes

@anakin87 anakin87 merged commit 8aa8655 into deepset-ai:main Mar 30, 2026
9 checks passed
@Akash504-ai
Copy link
Copy Markdown
Contributor Author

Akash504-ai commented Mar 30, 2026

Thanks a lot for the review and for applying the final changes! Really appreciate the guidance on aligning with the newer NVIDIA response_format syntax.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anakin87 anakin87 anakin87 approved these changes

Assignees

No one assigned

Labels

integration:nvidia type:documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Nvidia guided JSON test fails

2 participants

@Akash504-ai @anakin87