Skip to content

feat(image-gen): support Codex reference-image edits on raw SSE path#33644

Open
xiaoyaner0201 wants to merge 1 commit into
NousResearch:mainfrom
xiaoyaner0201:feat/codex-image-ref-sse
Open

feat(image-gen): support Codex reference-image edits on raw SSE path#33644
xiaoyaner0201 wants to merge 1 commit into
NousResearch:mainfrom
xiaoyaner0201:feat/codex-image-ref-sse

Conversation

@xiaoyaner0201

@xiaoyaner0201 xiaoyaner0201 commented May 28, 2026

Copy link
Copy Markdown

Summary

  • extend the image_generate tool schema/dispatch path to accept reference_images
  • pass references through the current openai-codex provider as Responses input_image content
  • keep the current raw SSE / httpx Codex Responses path from main instead of reverting to SDK stream parsing
  • set the Codex image_generation tool to action: edit whenever references are present
  • add bounded validation for reference inputs: max 8 refs, 20 MB local files, 30 MB data URLs, base64 data-URL validation, and basic PNG/JPEG/WebP/GIF MIME detection

Relationship to prior PRs

This overlaps with earlier reference-image work:

This PR is intentionally narrower and updated for current main / Hermes 0.15:

Follow-ups can add provider-level capability advertisement / shared normalization helpers if maintainers prefer that broader direction.

Hermes 0.15 compatibility

  • rebased on current main (0.15.1) instead of the older 0.14-era base
  • preserves the current plugin-provider dispatch behavior, including explicit image_gen.provider = "fal" routing through the FAL plugin
  • keeps image_gen.model forwarding unchanged
  • preserves plain text-to-image behavior: no action is sent when reference_images is absent
  • allows existing local-path callers to continue using normal paths; local references are validated for existence, size, and image MIME before being embedded as data URLs

Testing

  • python -m pytest tests/plugins/image_gen/test_openai_codex_provider.py tests/tools/test_image_generation_plugin_dispatch.py tests/tools/test_image_generation.py -q -o 'addopts='
  • python -m py_compile plugins/image_gen/openai-codex/__init__.py tools/image_generation_tool.py tests/plugins/image_gen/test_openai_codex_provider.py tests/tools/test_image_generation_plugin_dispatch.py tests/tools/test_image_generation.py
  • git diff --check
  • compatibility smoke: imported hermes_cli.__version__ == "0.15.1" and verified image_generate exposes reference_images

@alt-glitch alt-glitch added type/feature New feature or request comp/plugins Plugin system and bundled plugins tool/vision Vision analysis and image generation P3 Low — cosmetic, nice to have duplicate This issue or pull request already exists labels May 28, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #21463 — both add reference_images support to image_generate for the openai-codex provider. Also overlaps with #15308 and closed #18805.

@xiaoyaner0201 xiaoyaner0201 force-pushed the feat/codex-image-ref-sse branch from aa77375 to 790b9dc Compare May 29, 2026 16:44
@xiaoyaner0201 xiaoyaner0201 changed the title feat(image-gen): support Codex reference images feat(image-gen): support Codex reference-image edits on raw SSE path May 29, 2026
@xiaoyaner0201

Copy link
Copy Markdown
Author

Thanks — agreed that this overlaps with #21463, #15308, and the closed #18805.

I updated this PR so it is no longer presented as an independent competing direction. The intended positioning is a small current-main refresh for the Codex image path:

I did not add the broader provider capability contract from #15308 in this PR, to keep the patch focused on the current Codex provider path. The schema now documents that reference_images is currently honored by openai-codex, while other backends may reject or ignore it.

Happy to align with maintainer preference here: if maintainers would rather move #21463 forward, I can close this PR and extract only the raw-SSE + action: edit pieces there; otherwise this PR can serve as the current-main/Hermes-0.15 successor that incorporates the relevant prior art.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins duplicate This issue or pull request already exists P3 Low — cosmetic, nice to have tool/vision Vision analysis and image generation type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants