Skip to content

feat: support multiple images per column in image context#257

Merged
nabinchha merged 4 commits into
mainfrom
nmulepati/feat/255-handle-multiple-image-contexts-per-column
Jan 28, 2026
Merged

feat: support multiple images per column in image context#257
nabinchha merged 4 commits into
mainfrom
nmulepati/feat/255-handle-multiple-image-contexts-per-column

Conversation

@nabinchha

@nabinchha nabinchha commented Jan 28, 2026

Copy link
Copy Markdown
Contributor

📋 Summary

Adds support for handling multiple images per column in image context configurations. Previously, each image context column could only reference a single image. This change enables columns to contain multiple images as a list or JSON-serialized array.

🔄 Changes

✨ Added

  • Support for list-of-strings and JSON-serialized list input formats in ImageContext.get_contexts() (models.py)
  • Comprehensive test coverage for various input formats: single string, list, JSON-serialized list, invalid JSON, and empty list (test_models.py)

🔧 Changed

  • Breaking API change: Renamed ModalityContext.get_context()get_contexts() to return list instead of single dict (models.py) though this is mostly internal and not user facing.
  • Updated ImageContext.get_contexts() to parse and normalize various input formats to list of contexts (models.py)
  • Modified LLM completion generator to flatten multiple contexts using extend() instead of append() (llm_completion.py)
  • Reordered message construction to pack multimodal context before user text messages for better model processing (utils.py - commit 998d880)

🔍 Attention Areas

⚠️ Reviewers: Please pay special attention to the following:

  • packages/data-designer-config/src/data_designer/config/models.py - Breaking API change: get_context()get_contexts() with list return type
  • The JSON parsing logic handles edge cases (non-list JSON objects, invalid JSON strings) by treating them as single values
  • Message ordering change (multimodal context now precedes text). This is based on feedback from folks working on VLM training.

🤖 Generated with AI

@nabinchha nabinchha requested a review from a team as a code owner January 28, 2026 19:43
@nabinchha nabinchha requested a review from eric-tramel January 28, 2026 19:43
Comment on lines +56 to +64
{
"type": "image_url",
"image_url": {"url": "data:image/png;base64,image1base64", "format": "png"},
},
{
"type": "image_url",
"image_url": {"url": "data:image/png;base64,image2base64", "format": "png"},
},
{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe a dumb question, but is there any case in which you'd like to identify a particular context out of the group of contexts? Here I guess you have index but then you need to know the order.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you'd need to call out by the order... like "what are the difference between the scene in the 1st and the 3rd picture?"

johnnygreco
johnnygreco previously approved these changes Jan 28, 2026
eric-tramel
eric-tramel previously approved these changes Jan 28, 2026

@eric-tramel eric-tramel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@nabinchha

Copy link
Copy Markdown
Contributor Author

I'll merge this shortly once I verify a test workflow I have looks gtg.

@nabinchha nabinchha merged commit 3d86a38 into main Jan 28, 2026
46 checks passed
@nabinchha nabinchha deleted the nmulepati/feat/255-handle-multiple-image-contexts-per-column branch January 28, 2026 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants