feat(ios): add inline Adaptive Card rendering in chat#42350
feat(ios): add inline Adaptive Card rendering in chat#42350VikrantSingh01 wants to merge 3 commits intoopenclaw:mainfrom
Conversation
Greptile SummaryThis PR adds native Adaptive Card rendering to the iOS/macOS chat UI. When an assistant message contains
Confidence Score: 3/5
Last reviewed commit: 72ceae6 |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 72ceae6993
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
New files in apps/shared/OpenClawKit/Sources/OpenClawChatUI/AdaptiveCard/: - AdaptiveCardModels.swift — Codable models for AC v1.5 (TextBlock, FactSet, ColumnSet, Container, Image, Actions) - AdaptiveCardParser.swift — marker extraction from message text - AdaptiveCardView.swift — SwiftUI renderer with dark/light mode support Modified: - ChatMessageViews.swift — card detection in ChatAssistantTextBody before markdown render; inline AdaptiveCardView when card found Element mapping: TextBlock->Text, FactSet->label-value rows, ColumnSet->HStack, Container->VStack, Image->AsyncImage, Action.Submit->Button, Action.OpenUrl->Link macOS gets this for free (reuses OpenClawChatView from shared kit).
72ceae6 to
c55a554
Compare
- Update version comment from v1.5 to v1.6 - Fix AnyCodableAction to handle nested dicts, arrays, Bool, Int, Double - Add Table, RichTextBlock, CodeBlock, ImageSet, ActionSet elements - Add Action.Execute alongside Action.Submit - Extract template data from adaptive-card-data markers in parser - Remove redundant extraLarge case in textBlockFont (already lowercased) - Disable Action.Submit/Execute buttons with help text (no server routing) - Add renderers for all new element types
3e4f307 to
3c88350
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3c88350cbe
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| } | ||
| } | ||
| } | ||
| .frame(maxWidth: .infinity, alignment: .leading) |
There was a problem hiding this comment.
Honor Column.width when laying out ColumnSet
The renderColumnSet path forces every column to .frame(maxWidth: .infinity) and never reads cs.columns[i].width, so cards that specify auto, stretch, or weighted widths all render as equal columns. This breaks Adaptive Card layout semantics (for example, icon+content two-column cards) and can cause content crowding or incorrect alignment.
Useful? React with 👍 / 👎.
| if let url = URL(string: img.url) { | ||
| AsyncImage(url: url) { phase in |
There was a problem hiding this comment.
Show fallback when image URL cannot be parsed
renderImage only renders when URL(string: img.url) succeeds, but there is no else branch for invalid URL strings. In that case the image element silently disappears (including any altText), which drops card content instead of showing the placeholder label used in the AsyncImage failure case.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 80a0674033
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| } | ||
|
|
||
| struct Column: Codable { | ||
| let width: String? |
There was a problem hiding this comment.
Accept numeric Column.width values during decoding
CardElement.Column.width is declared as String?, but Adaptive Card payloads commonly send weighted widths as numbers (for example "width": 2). Swift Codable treats that as a type mismatch, so decoding a ColumnSet throws and AdaptiveCardParser.parseAdaptiveCardMarkers returns nil, which prevents inline rendering for otherwise valid cards and falls back to raw marker text. Please decode width as a string-or-number union (and normalize) so valid weighted layouts do not break parsing.
Useful? React with 👍 / 👎.
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
BradGroux
left a comment
There was a problem hiding this comment.
Review: iOS/macOS Adaptive Card Renderer (SwiftUI)
853 lines adding a SwiftUI renderer in the shared OpenClawChatUI package. Includes a full Codable model layer with discriminated unions (manual init(from:) keyed on "type"), marker parsing, and SwiftUI rendering. The Codable approach with typed enums is excellent and is the right pattern for Swift. SF Symbols integration for icons is a nice touch.
Blockers
1. Strict CodeBlock decoding fails entire card parse for valid variant payloads
struct CodeBlock: Codable {
let codeSnippet: String
let language: String?
}The model expects a codeSnippet key, but card producers commonly use code or text as the key name for code block content. When a card uses one of these variants, the Codable decode fails for the CodeBlock element, and because the element is decoded as part of the body array, the entire AdaptiveCard parse returns nil. The user sees nothing instead of a degraded card.
Fix options:
- Add
CodingKeysmappingcodeSnippet,code, andtextas alternatives - Use a custom
init(from:)that tries multiple keys - Make
codeSnippetoptional and fall back tocodeortextif absent
2. Merge conflict with #41735 in docs/plugins/community.md
This PR modifies community.md to add the Adaptive Cards entry. PR #41735 is dedicated to exactly this change. Both PRs touch the same lines. Merge #41735 first, then rebase this PR and remove the community.md change from it to avoid conflicts and duplicate entries.
Suggestions
3. Invalid image URLs render blank space with no fallback
private func renderImage(_ img: CardElement.ImageElement) -> some View {
if let url = URL(string: img.url) {
AsyncImage(url: url) { ... }
}
}When URL(string:) returns nil (malformed URL string), the if let falls through and nothing is rendered. The user sees blank space with no indication that an image was intended. Add an else branch that renders a placeholder icon or descriptive text like "[Image unavailable]".
4. ColumnSet ignores width semantics
ForEach(cs.columns.indices, id: \.self) { ... }
.frame(maxWidth: .infinity, alignment: .leading)All columns are rendered with equal width regardless of the width property specified in the card JSON (e.g., "auto", "stretch", "50px"). This means cards designed with intentional column sizing will render incorrectly. At minimum, handle "auto" (size to content) vs "stretch" (fill remaining), which covers the majority of real-world cards.
5. ImageSet grid uses height value as column width minimum
GridItem(.adaptive(minimum: self.imageMaxHeight(size)))The imageMaxHeight function returns a height-based value, but it is being used as the minimum column width for the LazyVGrid. These are different dimensions. On devices with different aspect ratios, this produces unexpected grid density. Use a width-based calculation instead, or derive the grid minimum from the image size property directly.
Nits
- Unknown elements and unknown actions are silently ignored with no debug logging. For development and troubleshooting, consider a
#if DEBUGlog statement when an unrecognized type is encountered. This makes it much easier to diagnose schema drift between server and client.
Summary
The Swift Codable model layer with discriminated unions is genuinely well-designed and is the cleanest typed approach across all the platform PRs. The main concerns are the strict CodeBlock decoding (which will drop entire cards in practice) and the community.md conflict. Fix those, handle image fallbacks, and address the column/grid sizing issues for a solid merge.
|
This pull request has been automatically marked as stale due to inactivity. |
|
Codex review: found issues before merge. Summary Reproducibility: yes. Source inspection gives high-confidence paths: a schema-valid card with numeric Next step before merge Security Review findings
Review detailsBest possible solution: Define the native rich-message contract and client URL/image policy, then land a rebased Swift implementation that preserves thinking visibility, supports schema-valid layouts, adds focused Swift tests, and includes the required changelog entry. Do we have a high-confidence way to reproduce the issue? Yes. Source inspection gives high-confidence paths: a schema-valid card with numeric Is this the best way to solve the issue? No. Native iOS/macOS rendering may be a valid direction, but this branch is not the best merge shape until schema compatibility, thinking visibility, URL/image policy, docs policy, tests, and changelog are aligned. Full review comments:
Overall correctness: patch is incorrect Security concerns:
Acceptance criteria:
What I checked:
Likely related people:
Remaining risk / open question:
Codex review notes: model gpt-5.5, reasoning high; reviewed against a7c5a0425988. |
Summary
Adds native Adaptive Card rendering to the iOS and macOS apps. When a chat message contains
<!--adaptive-card-->markers, the card is parsed and rendered inline in the chat bubble using SwiftUI.Replaces closed #42306 (closed due to force push during review fixes).
New files (in
apps/shared/OpenClawKit/Sources/OpenClawChatUI/AdaptiveCard/)AdaptiveCardModels.swift— Codable models with recursive AnyCodableAction for complex data payloadsAdaptiveCardParser.swift— marker extraction + JSON decodeAdaptiveCardView.swift— SwiftUI renderer with dark/light mode, column width support, image fallbackModified
ChatMessageViews.swift— cachedparsedCardcomputed property (not inbody), think-block filtering before card parsingReview feedback addressed
body(performance)extralargenotextraLarge)<think>don't leak)macOS gets this for free (reuses
OpenClawChatView).Ecosystem Context
This PR is part of the Adaptive Cards feature set powered by:
adaptive_cardtool, MCP bridge, channel-aware prompt guidance, fallback text generation, action routingThe plugin emits
<!--adaptive-card-->markers in tool result text. This PR adds the iOS/macOS client-side parser and SwiftUI renderer that consumes those markers.Element support
TextBlock, FactSet, ColumnSet, Container, Image, Action.Submit (disabled with help text), Action.OpenUrl
Performance target
<100ms parse + render for typical cards.
Related PRs