extensions/whatsapp/src/auto-reply/monitor/on-message.ts (1)

279-299: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Fix group voice-note spoken-mention gating when nativeIngestion is enabled

With tools.media.audio.nativeIngestion === true, runAudioPreflightOnce() sets preflightAudioTranscript = null, and the second applyGroupGating call only supplies mentionText when typeof preflightAudioTranscript === "string"—so no mentionText is provided. Since mention detection then runs against the original msg.body (e.g., "<media:audio>"), the retry pass still results in shouldProcess: false, and the handler returns before processMessage runs, silently dropping group voice notes meant to activate via spoken mentions.

Split the “audio STT skip” behavior from “mention gating needs STT”: either run a lightweight STT/transcription only to produce mentionText when needsMentionText is true (even under native ingestion), or explicitly document/guard this limitation for spoken-mention activation in groups with native ingestion.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@extensions/whatsapp/src/auto-reply/monitor/on-message.ts` around lines 279 -
299, The second applyGroupGating call is missing mentionText when
runAudioPreflightOnce() sets preflightAudioTranscript = null under
tools.media.audio.nativeIngestion, causing spoken-mention gating to fail; modify
the flow around runAudioPreflightOnce(), preflightAudioTranscript and
applyGroupGating so that when nativeIngestion is enabled but the gating/mention
logic requires a spoken mention, you still obtain a lightweight
STT/transcription or a dedicated mentionText (e.g., call a minimal transcription
helper or preserve the preflight result) and pass it into applyGroupGating via
the mentionText property (referencing runAudioPreflightOnce,
preflightAudioTranscript, and applyGroupGating) so the retry pass supplies
mentionText even with native ingestion enabled.

src/agents/agent-hooks/context-pruning/pruner.ts (1)

150-165: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Consider adding character estimation for audio content.

The function now accepts AudioContent but doesn't estimate any characters for audio blocks (they silently contribute 0). If audio is sent inline to multimodal models, it likely consumes context window quota. Consider adding an AUDIO_CHAR_ESTIMATE constant similar to IMAGE_CHAR_ESTIMATE (line 14) to account for audio in context pruning calculations.

💡 Suggested addition

 const IMAGE_CHAR_ESTIMATE = 8_000;
+const AUDIO_CHAR_ESTIMATE = 12_000; // Rough estimate for audio content
 const PRUNED_CONTEXT_IMAGE_MARKER = "[image removed during context pruning]";

 function estimateTextAndImageChars(
   content: ReadonlyArray<TextContent | ImageContent | AudioContent>,
 ): number {
   let chars = 0;
   for (const block of content) {
     const text = coerceTextBlock(block);
     if (text !== null) {
       chars += estimateWeightedTextChars(text);
       continue;
     }
     if (isImageBlock(block)) {
       chars += IMAGE_CHAR_ESTIMATE;
+      continue;
     }
+    if (block.type === "audio") {
+      chars += AUDIO_CHAR_ESTIMATE;
+    }
   }
   return chars;
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/agents/agent-hooks/context-pruning/pruner.ts` around lines 150 - 165, The
function estimateTextAndImageChars ignores AudioContent so audio blocks
contribute 0; add a constant AUDIO_CHAR_ESTIMATE (similar to
IMAGE_CHAR_ESTIMATE) and update estimateTextAndImageChars to detect audio blocks
(e.g., via an existing isAudioBlock helper or a new check for AudioContent) and
add AUDIO_CHAR_ESTIMATE to chars when encountered; keep existing handling for
text via coerceTextBlock and images via isImageBlock, and ensure
AUDIO_CHAR_ESTIMATE is exported/defined alongside IMAGE_CHAR_ESTIMATE so pruning
uses a conservative audio token budget.

src/agents/embedded-agent-runner/run/attempt.ts (1)

3991-4012: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Thread media options through the runtime-only submission path.

Line 3991 still submits the runtime-only branch with only preflightResult. With the new audioCount skip logic, an audio-only turn can now reach this branch and still lose its loaded audio payload before the model call.

Proposed fix

-            try {
-              if (promptSubmission.runtimeOnly) {
-                await promptActiveSession(promptForSession, {
-                  preflightResult: armModelPromptTransform,
-                });
-              } else {
-                const cleanupRuntimeContextMessage = installRuntimeContextMessageForPrompt({
-                  session: activeSession,
-                  message: runtimeContextMessageForCurrentTurn,
-                });
-                try {
-                  // Only attach images/audio options when present so models
-                  // that don't expect those parameters aren't handed empties.
-                  const promptOptions: Parameters<typeof promptActiveSession>[1] = {
-                    preflightResult: armModelPromptTransform,
-                  };
-                  if (imageResult.images.length > 0) {
-                    promptOptions.images = imageResult.images;
-                  }
-                  if (audioResult.audio.length > 0) {
-                    promptOptions.audio = audioResult.audio;
-                  }
-                  await promptActiveSession(promptForSession, promptOptions);
-                } finally {
-                  cleanupRuntimeContextMessage();
-                }
-              }
+            try {
+              // Only attach images/audio options when present so models
+              // that don't expect those parameters aren't handed empties.
+              const promptOptions: Parameters<typeof promptActiveSession>[1] = {
+                preflightResult: armModelPromptTransform,
+              };
+              if (imageResult.images.length > 0) {
+                promptOptions.images = imageResult.images;
+              }
+              if (audioResult.audio.length > 0) {
+                promptOptions.audio = audioResult.audio;
+              }
+              if (promptSubmission.runtimeOnly) {
+                await promptActiveSession(promptForSession, promptOptions);
+              } else {
+                const cleanupRuntimeContextMessage = installRuntimeContextMessageForPrompt({
+                  session: activeSession,
+                  message: runtimeContextMessageForCurrentTurn,
+                });
+                try {
+                  await promptActiveSession(promptForSession, promptOptions);
+                } finally {
+                  cleanupRuntimeContextMessage();
+                }
+              }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/agents/embedded-agent-runner/run/attempt.ts` around lines 3991 - 4012,
The runtime-only branch currently calls promptActiveSession(promptForSession, {
preflightResult: armModelPromptTransform }) and therefore drops image/audio
payloads for audio-only or image-containing turns; modify the runtime-only path
in the promptSubmission.runtimeOnly branch to build the same promptOptions
object used in the non-runtime path (include preflightResult and conditionally
add images from imageResult.images and audio from audioResult.audio) and pass
that promptOptions to promptActiveSession so media is preserved; reference
promptSubmission, promptActiveSession, promptForSession, promptOptions,
imageResult, and audioResult to locate and update the code.

src/agents/embedded-agent-runner/run/images.ts (1)

778-795: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Verify inbound audio claim-check URIs always carry an audio MIME or extension.

A claim-check URI is classified as audio only when the bracket contains (audio/...) or the id has an audio extension. Unlike detectImageReferences (which accepts any media:// URI), this requires a discriminator. Since on-message.ts skips the STT preflight when nativeIngestion is enabled, an inbound voice-note URI that lacks both signals would be detected as neither image nor audio — and with STT already skipped, the audio cue is lost entirely (the exact failure this PR aims to prevent).

Confirm the Gateway/store format for inbound audio attachments retains an audio extension on the media id or emits the (audio/...) annotation.

#!/bin/bash
# Inspect how inbound media claim-check entries are formatted, and whether
# audio attachments retain an extension or MIME annotation in the bracket.
rg -nP -C4 'media attached' --type=ts -g '!**/*.test.ts'
rg -nP -C3 'media://inbound' --type=ts -g '!**/*.test.ts'
# Filename sanitization / extension retention in store
fd -i 'store.ts' --exec rg -nP -C3 'sanitizeFilename|extname|\.ogg|audio/'

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/agents/embedded-agent-runner/run/images.ts` around lines 778 - 795, The
current block using MEDIA_URI_REGEX only adds a media claim-check when the
bracket contains an (audio/...) MIME or the id has an audio extension, causing
inbound audio URIs without those signals to be skipped; update the logic in the
MEDIA_URI_REGEX handling so that any matched media://inbound/<id> is normalized
(use normalizeRefForDedupe) and added to refs (and seen) as type "media-uri"
regardless of isAudio, instead of continuing early when !isAudio; keep isAudio
detection (via isAudioExtension and the /\(audio\// check) for downstream STT
decision but do not prevent adding the reference, and ensure this change aligns
with detectImageReferences behavior that accepts any media:// URIs.