feat(ios): add inline Adaptive Card rendering in chat#42306
feat(ios): add inline Adaptive Card rendering in chat#42306VikrantSingh01 wants to merge 0 commit intoopenclaw:mainfrom
Conversation
Greptile SummaryThis PR adds native Adaptive Card rendering to the iOS/macOS chat UI, introducing three new Swift files (models, parser, SwiftUI renderer) and wiring them into
Confidence Score: 2/5
Last reviewed commit: 762ab48 |
| font: font, | ||
| textColor: OpenClawChatTheme.assistantText) | ||
| // Check for adaptive card markers before standard markdown rendering | ||
| if let parsed = AdaptiveCardParser.parseAdaptiveCardMarkers(from: self.text) { |
There was a problem hiding this comment.
JSON decoding runs on every SwiftUI render
AdaptiveCardParser.parseAdaptiveCardMarkers(from:) performs string range searches and a full JSONDecoder.decode(...) call inside the body computed property, which SwiftUI can call many times — and very frequently during message streaming. This will cause repeated JSON decoding of the same content on every render pass, directly contradicting the stated <50ms performance target under streaming load.
The parsed result should be computed once and cached. A common pattern is to move the parsing into a let/@State calculated outside body, or hoist the parsed card up to a parent view, for example:
private struct ChatAssistantTextBody: View {
let text: String
let markdownVariant: ChatMarkdownVariant
let includesThinking: Bool
// Parsed once per unique text value
private var parsed: ParsedAdaptiveCard? {
AdaptiveCardParser.parseAdaptiveCardMarkers(from: text)
}
...
}But even that recomputes on each render. Ideally, the ParsedAdaptiveCard result should be lifted into the model layer (e.g., alongside primaryText) and passed down so parsing only happens when the message content changes, not on every view invalidation.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/shared/OpenClawKit/Sources/OpenClawChatUI/ChatMessageViews.swift
Line: 623
Comment:
**JSON decoding runs on every SwiftUI render**
`AdaptiveCardParser.parseAdaptiveCardMarkers(from:)` performs string range searches and a full `JSONDecoder.decode(...)` call inside the `body` computed property, which SwiftUI can call many times — and very frequently during message streaming. This will cause repeated JSON decoding of the same content on every render pass, directly contradicting the stated <50ms performance target under streaming load.
The parsed result should be computed once and cached. A common pattern is to move the parsing into a `let`/`@State` calculated outside `body`, or hoist the parsed card up to a parent view, for example:
```swift
private struct ChatAssistantTextBody: View {
let text: String
let markdownVariant: ChatMarkdownVariant
let includesThinking: Bool
// Parsed once per unique text value
private var parsed: ParsedAdaptiveCard? {
AdaptiveCardParser.parseAdaptiveCardMarkers(from: text)
}
...
}
```
But even that recomputes on each render. Ideally, the `ParsedAdaptiveCard` result should be lifted into the model layer (e.g., alongside `primaryText`) and passed down so parsing only happens when the message content changes, not on every view invalidation.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in 0a2fd15. Moved parsing to a parsedCard computed property on the struct (outside body). Also added filteredText that strips think blocks before card parsing.
| case .submit(let a): | ||
| Button { | ||
| // Submit action tap (logged for debugging) | ||
| print("[AdaptiveCard] Action.Submit tapped: \(a.title ?? "untitled")") | ||
| } label: { | ||
| Text(a.title ?? "Submit") | ||
| .font(.caption) | ||
| } | ||
| .buttonStyle(.bordered) | ||
| .controlSize(.small) | ||
| case .unknown: |
There was a problem hiding this comment.
Action.Submit button silently does nothing
The Action.Submit button is rendered with full interactive styling (.bordered, .controlSize(.small)) making it appear functional to users, but its handler only print(...)s to the console. There is no callback, delegate, or environment key to propagate the submit event and its data payload to the host application.
A user tapping this button will receive no feedback and no action will be taken. This is a UX regression for any card that relies on submit actions. Either:
- Expose an action handler (e.g., via an
@Environmentor a closure onAdaptiveCardView) so the host can respond to submit taps. - Disable the button (
.disabled(true)) and add a visual affordance indicating it is not yet supported, so users aren't misled.
Without a real handler, Action.Submit cards will silently appear broken to end users.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/shared/OpenClawKit/Sources/OpenClawChatUI/AdaptiveCard/AdaptiveCardView.swift
Line: 179-189
Comment:
**`Action.Submit` button silently does nothing**
The `Action.Submit` button is rendered with full interactive styling (`.bordered`, `.controlSize(.small)`) making it appear functional to users, but its handler only `print(...)`s to the console. There is no callback, delegate, or environment key to propagate the submit event and its `data` payload to the host application.
A user tapping this button will receive no feedback and no action will be taken. This is a UX regression for any card that relies on submit actions. Either:
1. Expose an action handler (e.g., via an `@Environment` or a closure on `AdaptiveCardView`) so the host can respond to submit taps.
2. Disable the button (`.disabled(true)`) and add a visual affordance indicating it is not yet supported, so users aren't misled.
Without a real handler, `Action.Submit` cards will silently appear broken to end users.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in 0a2fd15. Button is now .disabled(true) with .help("Card actions not yet supported"). No non-functional interactive buttons ship.
| struct AnyCodableAction: Codable { | ||
| let value: Any | ||
|
|
||
| init(from decoder: Decoder) throws { | ||
| let container = try decoder.singleValueContainer() | ||
| if let dict = try? container.decode([String: String].self) { | ||
| self.value = dict | ||
| } else if let str = try? container.decode(String.self) { | ||
| self.value = str | ||
| } else { | ||
| self.value = "" | ||
| } | ||
| } | ||
|
|
||
| func encode(to encoder: Encoder) throws { | ||
| var container = encoder.singleValueContainer() | ||
| if let dict = self.value as? [String: String] { | ||
| try container.encode(dict) | ||
| } else if let str = self.value as? String { | ||
| try container.encode(str) | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
AnyCodableAction silently drops non-[String: String] data
The decoder only attempts [String: String] and then String. In real-world Adaptive Card Action.Submit payloads, data frequently contains mixed-type dictionaries — numbers, booleans, nested objects, or arrays. Any of these will fall through to self.value = "", silently discarding the payload data.
For example, a card with:
{ "type": "Action.Submit", "data": { "amount": 42, "confirmed": true } }...will decode data as "" instead of the intended dictionary.
Since this data is what the host application would act upon in a real submit handler, data loss here would cause broken submit flows once a real handler is wired up. Consider broadening the type to handle [String: Any] or using a JSONValue enum pattern that covers all JSON primitive types.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/shared/OpenClawKit/Sources/OpenClawChatUI/AdaptiveCard/AdaptiveCardModels.swift
Line: 124-146
Comment:
**`AnyCodableAction` silently drops non-`[String: String]` data**
The decoder only attempts `[String: String]` and then `String`. In real-world Adaptive Card `Action.Submit` payloads, `data` frequently contains mixed-type dictionaries — numbers, booleans, nested objects, or arrays. Any of these will fall through to `self.value = ""`, silently discarding the payload data.
For example, a card with:
```json
{ "type": "Action.Submit", "data": { "amount": 42, "confirmed": true } }
```
...will decode `data` as `""` instead of the intended dictionary.
Since this `data` is what the host application would act upon in a real submit handler, data loss here would cause broken submit flows once a real handler is wired up. Consider broadening the type to handle `[String: Any]` or using a `JSONValue` enum pattern that covers all JSON primitive types.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in 0a2fd15. AnyCodableAction now decodes [String: AnyCodableAction] (recursive), [AnyCodableAction], String, Double, Bool, Int — no more silent data loss on complex payloads.
| @ViewBuilder | ||
| private func renderColumnSet(_ cs: CardElement.ColumnSet) -> some View { | ||
| HStack(alignment: .top, spacing: 8) { | ||
| ForEach(cs.columns.indices, id: \.self) { i in | ||
| VStack(alignment: .leading, spacing: 6) { | ||
| if let items = cs.columns[i].items { | ||
| ForEach(items.indices, id: \.self) { j in | ||
| self.renderElement(items[j]) | ||
| } | ||
| } | ||
| } | ||
| .frame(maxWidth: .infinity, alignment: .leading) | ||
| } | ||
| } |
There was a problem hiding this comment.
Column.width is parsed but never used in layout
CardElement.Column.width is decoded from the JSON but renderColumnSet always applies .frame(maxWidth: .infinity) to every column regardless. This means adaptive cards with explicit column widths (e.g., "width": "auto", "width": "1", "width": "200px") will all render at equal width, breaking the intended layout.
The width field should be applied during rendering. The Adaptive Cards spec supports "auto", "stretch", and pixel/weight values, so at minimum distinguishing between "auto" and weighted numeric values would improve fidelity.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/shared/OpenClawKit/Sources/OpenClawChatUI/AdaptiveCard/AdaptiveCardView.swift
Line: 91-104
Comment:
**`Column.width` is parsed but never used in layout**
`CardElement.Column.width` is decoded from the JSON but `renderColumnSet` always applies `.frame(maxWidth: .infinity)` to every column regardless. This means adaptive cards with explicit column widths (e.g., `"width": "auto"`, `"width": "1"`, `"width": "200px"`) will all render at equal width, breaking the intended layout.
The `width` field should be applied during rendering. The Adaptive Cards spec supports `"auto"`, `"stretch"`, and pixel/weight values, so at minimum distinguishing between `"auto"` and weighted numeric values would improve fidelity.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in 0a2fd15. Column width is now read and applied: "auto" gets maxWidth: nil, others stretch with .infinity.
| private func textBlockFont(_ tb: CardElement.TextBlock) -> Font { | ||
| switch tb.size?.lowercased() { | ||
| case "extralarge", "extraLarge": | ||
| return .title | ||
| case "large": | ||
| return .title2 | ||
| case "medium": | ||
| return .body | ||
| case "small": | ||
| return .caption | ||
| default: | ||
| return .subheadline | ||
| } | ||
| } |
There was a problem hiding this comment.
Unreachable "extraLarge" case in font switch
tb.size?.lowercased() is called before the switch, so the input is already fully lowercase. The "extraLarge" pattern in the first case will never match because a lowercased string cannot contain an uppercase L.
| private func textBlockFont(_ tb: CardElement.TextBlock) -> Font { | |
| switch tb.size?.lowercased() { | |
| case "extralarge", "extraLarge": | |
| return .title | |
| case "large": | |
| return .title2 | |
| case "medium": | |
| return .body | |
| case "small": | |
| return .caption | |
| default: | |
| return .subheadline | |
| } | |
| } | |
| private func textBlockFont(_ tb: CardElement.TextBlock) -> Font { | |
| switch tb.size?.lowercased() { | |
| case "extralarge": | |
| return .title | |
| case "large": | |
| return .title2 | |
| case "medium": | |
| return .body | |
| case "small": | |
| return .caption | |
| default: | |
| return .subheadline | |
| } | |
| } |
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/shared/OpenClawKit/Sources/OpenClawChatUI/AdaptiveCard/AdaptiveCardView.swift
Line: 59-72
Comment:
**Unreachable `"extraLarge"` case in font switch**
`tb.size?.lowercased()` is called before the `switch`, so the input is already fully lowercase. The `"extraLarge"` pattern in the first `case` will never match because a lowercased string cannot contain an uppercase `L`.
```suggestion
private func textBlockFont(_ tb: CardElement.TextBlock) -> Font {
switch tb.size?.lowercased() {
case "extralarge":
return .title
case "large":
return .title2
case "medium":
return .body
case "small":
return .caption
default:
return .subheadline
}
}
```
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in 0a2fd15. Changed "extraLarge" to "extralarge" since lowercased() is already applied.
| private func renderImage(_ img: CardElement.ImageElement) -> some View { | ||
| if let url = URL(string: img.url) { | ||
| AsyncImage(url: url) { phase in | ||
| switch phase { | ||
| case .success(let image): | ||
| image | ||
| .resizable() | ||
| .scaledToFit() | ||
| .frame(maxHeight: self.imageMaxHeight(img.size)) | ||
| .clipShape(RoundedRectangle(cornerRadius: 6, style: .continuous)) | ||
| case .failure: | ||
| Label(img.altText ?? "Image", systemImage: "photo") | ||
| .font(.caption) | ||
| .foregroundStyle(.secondary) | ||
| default: | ||
| ProgressView() | ||
| .frame(height: 40) | ||
| } | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
No fallback shown for invalid image URLs
When URL(string: img.url) returns nil (malformed URL), renderImage silently produces an EmptyView — no error label, no placeholder. By contrast, the AsyncImage .failure phase correctly shows a Label with img.altText. An invalid URL silently vanishes from the layout, which can be confusing.
Consider rendering the same Label(img.altText ?? "Image", systemImage: "photo") fallback for the nil-URL case:
} else {
Label(img.altText ?? "Image", systemImage: "photo")
.font(.caption)
.foregroundStyle(.secondary)
}Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/shared/OpenClawKit/Sources/OpenClawChatUI/AdaptiveCard/AdaptiveCardView.swift
Line: 123-143
Comment:
**No fallback shown for invalid image URLs**
When `URL(string: img.url)` returns `nil` (malformed URL), `renderImage` silently produces an `EmptyView` — no error label, no placeholder. By contrast, the `AsyncImage` `.failure` phase correctly shows a `Label` with `img.altText`. An invalid URL silently vanishes from the layout, which can be confusing.
Consider rendering the same `Label(img.altText ?? "Image", systemImage: "photo")` fallback for the nil-URL case:
```swift
} else {
Label(img.altText ?? "Image", systemImage: "photo")
.font(.caption)
.foregroundStyle(.secondary)
}
```
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in 0a2fd15. Added Label(img.altText ?? "Image", systemImage: "photo") fallback for invalid image URLs.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 762ab48bce
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| font: font, | ||
| textColor: OpenClawChatTheme.assistantText) | ||
| // Check for adaptive card markers before standard markdown rendering | ||
| if let parsed = AdaptiveCardParser.parseAdaptiveCardMarkers(from: self.text) { |
There was a problem hiding this comment.
Parse adaptive cards after filtering think blocks
This parses adaptive-card markers from self.text before applying includesThinking, so cards embedded inside <think>...</think> are still rendered even when assistant trace is hidden. In the normal showsAssistantTrace == false path, that leaks content that was previously suppressed by AssistantTextParser (for example, a model/tool that emits card JSON in a thinking segment will now surface it in the UI). Parse against the visible response text (or strip thinking blocks first) to preserve the existing confidentiality boundary.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0a2fd15. Card parsing now runs against filteredText (after AssistantTextParser.segments strips think blocks), not raw self.text. Cards inside <think> blocks no longer leak.
762ab48 to
936607c
Compare
Summary
Adds native Adaptive Card rendering to the iOS and macOS apps. When a chat message contains
<!--adaptive-card-->markers, the card is parsed and rendered inline in the chat bubble using SwiftUI.New files (in
apps/shared/OpenClawKit/Sources/OpenClawChatUI/AdaptiveCard/)AdaptiveCardModels.swift(179 lines) — Codable models for AC v1.5 with manual decoding dispatching ontypefieldAdaptiveCardParser.swift(44 lines) — marker extraction + JSON decodeAdaptiveCardView.swift(213 lines) — SwiftUI renderer with dark/light mode via@Environment(\.colorScheme)Modified
ChatMessageViews.swift—ChatAssistantTextBodychecks for markers before markdown render; rendersAdaptiveCardViewinline when foundElement support
TextBlock (with weight/size/subtle), FactSet (label-value rows), ColumnSet (HStack), Container (VStack with background), Image (AsyncImage), Action.Submit (Button), Action.OpenUrl (openURL)
macOS
Gets this for free — reuses
OpenClawChatViewfrom the sharedOpenClawChatUImodule.Performance target
<50ms first render, <2MB memory per card.
Part of the Adaptive Cards first-class GenUI strategy. Works with the standalone openclaw-adaptive-cards plugin.