Skip to content

Make the main mascot react emotionally to messages and conversation state #1144

@senamakel

Description

@senamakel

[Feature] Make the main mascot react emotionally to messages and conversation state

Summary

Upgrade the main mascot so it reacts to what is happening in the conversation, not just the raw chat transport lifecycle.

The mascot should show emotions based on the message flow and meaning of the interaction, for example:

  • curiosity / attention when the user asks something interesting
  • concern / caution when something goes wrong
  • delight / satisfaction when a task succeeds
  • confusion / uncertainty when the agent is struggling or the request is ambiguous
  • calm / neutral when nothing notable is happening

Problem

Today the mascot is mostly driven by coarse runtime phases instead of the emotional or conversational meaning of the exchange.

What exists today:

  • app/src/features/human/useHumanMascot.ts already reacts to chat transport events like inference_start, text_delta, chat_done, and chat_error
  • app/src/features/human/Mascot/Ghosty.tsx already supports a small MascotFace state model
  • app/src/services/chatService.ts already exposes the main chat event stream, including text, thinking, tool, subagent, done, and error events

What is missing:

  • the mascot does not meaningfully react to what the user or agent is actually saying
  • current reactions are mostly limited to generic thinking, speaking, and normal state changes
  • there is no emotional layer that maps conversation outcomes to visible expressions
  • the mascot therefore feels mechanically animated rather than emotionally engaged with the chat

This matters because the mascot is supposed to feel like the visible face of the agent. If it only changes state based on transport phases, it will feel disconnected from the actual interaction. The mascot should feel aware of what is happening in the conversation.

Solution (optional)

Add a message-aware emotional reaction layer on top of the existing mascot lifecycle.

High-level approach:

  • keep the current low-level activity states (thinking, speaking, listening, etc.)
  • add a higher-level emotion/reaction model driven by message meaning and conversation outcome
  • combine those two layers so the mascot can be both:
    • doing something
    • feeling something about it

Examples of signals the system can use:

  • user message intent/tone
  • assistant response tone
  • success/failure of tool calls
  • subagent success/failure
  • chat errors
  • special outcomes like clarification, hesitation, completion, excitement, warning, or apology

Possible reaction states:

  • curious
  • happy
  • confused
  • concerned
  • proud / satisfied
  • apologetic
  • cautious
  • neutral

Scope expectations:

  • this should not rely only on random expression changes
  • reactions should be driven by actual conversation/runtime signals
  • the system should remain bounded and legible rather than becoming noisy or over-animated
  • fallback behavior should remain simple if no strong emotional cue is present

Implementation note:

  • this can begin as a deterministic heuristic layer if needed, but it should be structured so the mascot can react consistently to conversation meaning and runtime outcomes rather than only transport phases.

Acceptance criteria

  • The mascot can express more than the current basic transport states.

  • The mascot reacts to message/conversation meaning, not only to thinking / speaking transport phases.

  • Success, error, ambiguity, and other notable runtime outcomes can produce distinct visible reactions.

  • The emotional reaction layer remains understandable and does not become visually noisy.

  • Default/fallback behavior stays calm when there is no strong cue.

  • The implementation reuses the existing chat event surface where possible.

  • Tests cover key mascot reaction state transitions.

  • Diff coverage ≥ 80% — the implementing PR meets the changed-lines coverage gate (Vitest + cargo-llvm-cov, enforced by .github/workflows/coverage.yml).

  • The feature is documented so contributors understand how message/runtime signals map to mascot emotions.

Related

  • app/src/features/human/useHumanMascot.ts
  • app/src/features/human/Mascot/Ghosty.tsx
  • app/src/features/human/Mascot/**
  • app/src/services/chatService.ts

Metadata

Metadata

Assignees

Labels

agentBuilt-in agents, prompts, orchestration, and agent runtime in src/openhuman/agent/.enhancementfeatureNet-new user-facing capability or product behavior.feature-requestreact-uiReact app work in app/src: pages, components, providers, store, and UX.
No fields configured for Feature.

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions