Skip to content

feat(matrix): harden rendering, media, E2EE and diagnostics#18507

Draft
nepenth wants to merge 14 commits into
NousResearch:mainfrom
nepenth:feature/matrix-media-e2ee-diagnostics
Draft

feat(matrix): harden rendering, media, E2EE and diagnostics#18507
nepenth wants to merge 14 commits into
NousResearch:mainfrom
nepenth:feature/matrix-media-e2ee-diagnostics

Conversation

@nepenth

@nepenth nepenth commented May 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the Matrix rendering, media, E2EE, diagnostics, and opt-in Synapse integration layer.

Depends On

Scope

  • Matrix formatted HTML sanitization and URL-scheme allowlisting.
  • Structured mentions and room-wide mention controls.
  • Media upload/download hardening, streaming size caps, and signed-URL redaction.
  • E2EE off|optional|required behavior and recovery-key safety.
  • Diagnostics, capability reporting, docs, and opt-in Synapse integration coverage.
  • Test-only browser path-discovery isolation fix for the negative agent-browser CLI path.

Review Notes

This remains a draft stacked preview until #18506 lands. It was rebuilt after #18505 merged, so the remaining stack is now only #18506 -> #18507.

For incremental review, compare against #18506 or filter to the four commits from feat(matrix): harden rendering media and e2ee through test(browser): isolate agent-browser missing-cli path.

Validation

  • Local Matrix/browser subset: 375 passed, 1 skipped.
  • GitHub CI is green.

@alt-glitch alt-glitch added type/feature New feature or request P2 Medium — degraded but workaround exists platform/matrix Matrix adapter (E2EE) comp/gateway Gateway runner, session dispatch, delivery labels May 1, 2026
@nepenth nepenth marked this pull request as ready for review May 1, 2026 19:35
@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch 3 times, most recently from 3641ef1 to 9fcd4b4 Compare May 7, 2026 19:59
@nepenth

nepenth commented May 7, 2026

Copy link
Copy Markdown
Contributor Author

May 7, 2026 Update

This PR has been refreshed on top of the updated PR 2 stack after the v2026.5.7 release.

Branch Refresh

  • Rebased onto refreshed PR 2 / feature/matrix-tools-interactions.
  • Current head: 9fcd4b420.
  • Base head observed during refresh: 04193cf71.

Matrix Long-Reply Fix

  • Fixed split Matrix response metadata for long replies.
  • Long responses now keep every chunk in the Matrix thread.
  • Only the first chunk carries the explicit reply fallback.

Why this matters:

  • Some Matrix clients render m.in_reply_to as an inline quote.
  • Repeating that fallback on every split chunk can make the original user message reappear mid-answer.
  • In practice this looked like the agent suddenly quoted the user in the middle of a long response and sometimes made the response appear duplicated.

Local Validation

Passed:

scripts/run_tests.sh \
  tests/gateway/test_matrix.py \
  tests/gateway/test_matrix_project_context_isolation.py \
  tests/gateway/test_matrix_exec_approval.py \
  tests/gateway/test_matrix_mention.py \
  tests/gateway/test_matrix_voice.py \
  tests/tools/test_matrix_tools.py \
  tests/gateway/test_display_config.py \
  -q -o 'addopts='

Result:

338 passed, 1 skipped

Recovery-key focused validation also passed:

scripts/run_tests.sh \
  tests/e2e/matrix_xsign_bootstrap/test_bootstrap.py \
  tests/gateway/test_matrix.py \
  -k recovery_key \
  -q -o 'addopts='

Result:

5 passed, 1 skipped

Also passed:

python -m py_compile \
  gateway/platforms/matrix.py \
  hermes_cli/config.py \
  gateway/config.py \
  gateway/display_config.py

git diff --check

@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch 2 times, most recently from 709e4bc to 5014ea3 Compare May 11, 2026 15:33
@nepenth nepenth changed the title feat(matrix): harden rendering media e2ee diagnostics and docs feat(matrix): harden rendering, media, E2EE and diagnostics May 11, 2026
@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch from 5014ea3 to be63efd Compare May 14, 2026 12:30
@nepenth nepenth requested a review from a team May 14, 2026 12:30
@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch from be63efd to bf86ee4 Compare May 14, 2026 12:35
@nepenth

nepenth commented May 14, 2026

Copy link
Copy Markdown
Contributor Author

Update: long Matrix response fallback

This PR continues to include the long-message split fallback fix.

Why:

  • Some Matrix clients can render split fallback replies as quoted user-message blocks in the middle of a long agent response.
  • That can make the response appear to restart from the beginning after the quote.

Behavior after this update:

  • Split-message fallback avoids repeated reply/thread fallback behavior that can duplicate visible response content.
  • This works with the PR 2 default of final-answer-only Matrix output, reducing edited/interim room events while preserving long final responses.

Validation:

  • Focused Matrix/gateway test set passes locally: 297 passed.

@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch from bf86ee4 to b55768a Compare May 14, 2026 12:54
@nepenth

nepenth commented May 14, 2026

Copy link
Copy Markdown
Contributor Author

History cleanup

Rebased and cleaned commit history for reviewability.

  • No scope change.
  • Final tree is equivalent to the previous branch state.
  • Focused Matrix validation still passes locally: 297 passed.

@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch 3 times, most recently from c00b872 to 44ba2e1 Compare May 16, 2026 19:04
@nepenth

nepenth commented May 16, 2026

Copy link
Copy Markdown
Contributor Author

v2026.5.16 refresh

Rebased this Matrix PR stack onto current upstream main after the v2026.5.16 / v0.14.0 release.

Notes:

  • The v2026.5.16 release did not include this Matrix parity work; these PRs remain open and unmerged.
  • Checked Matrix/gateway-adjacent upstream changes from the release window.
  • Focused Matrix validation passed locally: 323 passed.
  • py_compile and git diff --check passed.

@nepenth nepenth marked this pull request as draft May 16, 2026 19:45
@nepenth

nepenth commented May 16, 2026

Copy link
Copy Markdown
Contributor Author

Draft stacked preview

This PR is intentionally marked draft while #18505 is the active review target and #18506 is the next interaction-layer preview. It remains open for full-stack testing and external review.

Depends on

Scope

  • Matrix formatted HTML sanitization and URL-scheme allowlisting.
  • Structured mentions and room-wide mention controls.
  • Media upload/download hardening, streaming size caps, and signed-URL redaction.
  • E2EE off|optional|required behavior and recovery-key safety.
  • Diagnostics, capability reporting, docs, and opt-in Synapse integration coverage.

Review note
This is the largest PR in the stack. If maintainers want it split after #18505/#18506 direction is settled, the clean split is rendering/media first, then E2EE/diagnostics/docs.

@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch 3 times, most recently from 8d3bf71 to b59a0b4 Compare May 31, 2026 11:16
@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch 3 times, most recently from 19d39bc to d6f223f Compare June 6, 2026 13:20
@nepenth

nepenth commented Jun 6, 2026

Copy link
Copy Markdown
Contributor Author

Rebased this PR onto current main (56236b16e) after the v0.16.0 / v2026.6.5 release.

Notes:

The custom Synapse/AgentFirst metadata PR (#23815) has been retired from the active stack to keep this series focused on upstreamable core Matrix functionality.

@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch from d6f223f to 936cb8b Compare June 10, 2026 13:44
@nepenth

nepenth commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

Jun 10, 2026 stacked preview refresh

Rebased onto current upstream main via the refreshed #18505 + #18506 stack.

Item Value
Base (main) a72bb037
Branch head 6b7c1badd
Commits ahead of main 21 (4 incremental atop #18506)
Status draft stacked preview — do not merge before #18505 and #18506

Incremental scope (4 commits atop #18506)

  1. feat(matrix): harden rendering media and e2ee — formatted HTML sanitization, media batching/size caps, signed-URL log redaction, MATRIX_E2EE_MODE=off|optional|required fail-closed behavior, recovery-key handling (never logged; 0600 output file opt-in)
  2. test(matrix): add opt-in Synapse integration harnesstests/e2e/matrix_synapse_gateway/ (integration/matrix_synapse marker; no default CI cost)
  3. test(matrix): cover media e2ee and long replies
  4. docs(matrix): explain inbound dispatch via handle_sync loop + docs/env-var updates

Local validation

.venv/bin/python -m pytest \
  tests/gateway/test_matrix.py \
  tests/tools/test_matrix_tools.py \
  tests/e2e/matrix_xsign_bootstrap/test_bootstrap.py -k recovery_key -q

Result: 529 passed, 1 skipped (full Matrix suite); recovery-key subset 5 passed, 1 skipped

Review note

Until lower stack PRs merge, Files changed includes foundation + tools commits. For incremental review, filter to commits from 2ee7655f5 through 6b7c1badd.

Synapse e2e tests are opt-in (pytest -m integration); they do not expand default CI surface area.

Leaving as draft until maintainers land #18505 / #18506.

@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch 2 times, most recently from e9430f5 to 2321b47 Compare June 11, 2026 02:46
@nepenth

nepenth commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

Jun 11 stacked refresh and CI fix

Restacked this PR onto the updated #18506 branch.

  • The incremental range now contains the 3 Matrix media/E2EE/diagnostics commits plus one test-only CI stabilization commit.
  • tests/tools/test_browser_homebrew_paths.py::TestFindAgentBrowser::test_raises_when_not_found now mocks ensure_dependency("browser") to return False, so the negative path-discovery test does not spend the 30s per-test timeout trying to install agent-browser.
  • The previous test (3) timeout was reproduced as a test-isolation issue, fixed, and verified locally.

Validation:

  • Local tests/tools/test_browser_homebrew_paths.py: 22 passed.
  • Local top-stack Matrix suite: 354 passed, 1 skipped.
  • GitHub CI is green on this PR.

@nepenth nepenth force-pushed the feature/matrix-media-e2ee-diagnostics branch from fcf6b60 to 437a2d5 Compare June 11, 2026 11:58
@nepenth

nepenth commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

Jun 11 update: remaining stack is now #18506 -> #18507

#18505 has landed in main, so this draft preview was rebuilt on top of the cleaned #18506 branch.

Validation:

  • Local Matrix/browser subset: 375 passed, 1 skipped.
  • GitHub CI is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/matrix Matrix adapter (E2EE) type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants