Skip to content

fix(unified): append-only message feed so windowing can't evict a seen message (#3719)#3738

Merged
Yeraze merged 1 commit into
mainfrom
fix/unified-feed-accumulate-3719
Jun 25, 2026
Merged

fix(unified): append-only message feed so windowing can't evict a seen message (#3719)#3738
Yeraze merged 1 commit into
mainfrom
fix/unified-feed-accumulate-3719

Conversation

@Yeraze

@Yeraze Yeraze commented Jun 25, 2026

Copy link
Copy Markdown
Owner

Summary

Structural fix for the #3719 / #3720 disappearing-message bug, complementing the server-side mitigations in #3736.

Root cause (read side): UnifiedMessagesPage rendered allMessages straight off data.pages — i.e. the latest poll of a capped newest-N server window. A message exclusive to a high-traffic source (e.g. a continental MQTT bridge) gets pushed past that window within ~60s as newer traffic arrives, so the next 10s poll's page 1 no longer contains it and it vanishes from the view — even though the row is still in the DB.

#3736 raises the server fetch window (limit×2 → limit×5), which buys ~167s of headroom at ~3 msg/s. That's a mitigation: a burstier source or longer poll gap re-opens the same hole, and it fetches more rows every poll.

This PR fixes it structurally on the client: the live feed becomes append-only. Every poll's pages are folded into a persistent map keyed by dedupKey; we render from that map instead of the latest poll. Once a message has been shown it stays shown, regardless of how the server window shifts — so the disappearance can't happen at any source traffic rate.

Changes

  • New src/utils/unifiedMessageAccumulator.ts — pure foldUnifiedMessagePages(acc, pages, cap):
  • UnifiedMessagesPage.tsx — replaced the page-flatten useMemo with a useRef accumulator folded via the helper; reset synchronously on channel change so feeds never blend.

Tradeoff

A message deleted server-side stays visible until channel switch / refresh (rare; the alternative — re-introducing windowing eviction — is worse). Memory is bounded by the 5000 cap.

Tests

  • unifiedMessageAccumulator.test.ts — sort, intra/cross-page dedup, upsert-wins, undefined-pages, cap-drops-oldest, and the core regression: a message present in poll 1 but absent from poll 2 stays in the feed.
  • Full suite: 7528 passed, 0 failed.

Refs #3719, #3720, #3736

🤖 Generated with Claude Code

…seen message (#3719)

The unified feed rendered straight off the latest poll (data.pages), which is
a capped newest-N window. A message exclusive to a high-traffic source gets
pushed past the window within ~60s as newer traffic arrives, so the next poll
drops it from the view even though it's still in the DB — the #3719/#3720
disappearance.

Structural fix: fold every poll's pages into a persistent map keyed by dedupKey
and render from that, making the live feed append-only. Once a message is shown
it stays shown regardless of how the server window shifts; upserts prefer the
newest (most-complete) merged copy; sort stays by createdAt (#3122). Reset on
channel change; capped at 5000 (oldest dropped, still re-fetchable on scroll).

This decouples display persistence from server window size, so it holds at any
source traffic rate — unlike raising fetchLimit, which only buys proportional
headroom. Complements the server-side fixes in #3736.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SJPe6J5vKrcbwzt6vCdtrt
@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown

Claude finished @Yeraze's task —— View job


PR Review: fix(unified): append-only message feed

Tasks:

  • Read changed files
  • Analyze code quality and best practices
  • Check for potential bugs
  • Review performance considerations
  • Check security concerns
  • Evaluate test coverage
  • Post review feedback

Overall Assessment

This is a well-designed structural fix for a real disappearing-message bug. The core idea — making the live feed append-only via a persistent Map keyed by dedupKey — is sound, and the implementation is clean with appropriate guardrails (upsert semantics, cap, reset on channel change). The PR is production-ready with only minor observations worth noting.


src/utils/unifiedMessageAccumulator.ts

Positives:

  • Pure function with no side-effects on the outside world — easy to test and reason about in isolation.
  • Generic <T extends AccumulableMessage> keeps the utility reusable for other paginated feeds without requiring changes.
  • The mutation-in-place design (acc is mutated) is intentional and documented, which is important since the caller relies on the ref surviving re-renders.

Observation — sort stability on equal createdAt:
The sort a.createdAt - b.createdAt is unstable when two messages share the same millisecond timestamp (e.g. two MQTT bridged messages batched in the same poll). Different JS engines may produce different orderings for equal elements. This is pre-existing behaviour, but since the accumulator now persists order across polls, a transient inconsistency on one poll becomes permanent. Worth a follow-up if message ordering with identical createdAt turns out to matter.

Observation — cap threshold triggers re-rebuild every render above 5000:
When out.length > cap, the function clears and re-populates acc in O(cap). That's fine. However, if the caller passes a cap lower than the current acc.size, every subsequent call will do a clear+rebuild even if no new messages were added (because acc will be at cap after the rebuild, then get 0 new messages but out.length will equal cap, not exceed it — actually this is fine). No real issue here; just confirmed the logic is correct.


src/pages/UnifiedMessagesPage.tsx (lines 287–303)

Concern — ref mutation inside useMemo violates React rules:

// lines 294–303
const accumulatorRef = useRef<Map<string, UnifiedMessage>>(new Map());
const accChannelRef = useRef<string>(selectedChannel);
const allMessages = useMemo<UnifiedMessage[]>(() => {
  // Reset synchronously when the channel changes so feeds never blend.
  if (accChannelRef.current !== selectedChannel) {
    accChannelRef.current = selectedChannel;
    accumulatorRef.current = new Map();   // ← ref mutation inside useMemo
  }
  return foldUnifiedMessagePages(accumulatorRef.current, data?.pages);
}, [data?.pages, selectedChannel]);

React's rules of hooks state that useMemo must be a pure computation — it must not produce observable side-effects. Mutating a useRef value is technically a side-effect (it modifies external mutable state), and React is free to discard and recompute useMemo callbacks multiple times in Strict Mode / Concurrent Mode. If React re-runs this memo before committing, acc gets doubled-up entries because foldUnifiedMessagePages mutates acc in place.

A safer pattern would be to use useEffect for the reset and a separate useMemo that only reads from acc after the effect has run, or — more pragmatically for a ref-based accumulator — handle the full fold in a useEffect that sets a useState snapshot:

const [allMessages, setAllMessages] = useState<UnifiedMessage[]>([]);
useEffect(() => {
  if (accChannelRef.current !== selectedChannel) {
    accChannelRef.current = selectedChannel;
    accumulatorRef.current = new Map();
  }
  setAllMessages(foldUnifiedMessagePages(accumulatorRef.current, data?.pages));
}, [data?.pages, selectedChannel]);

This keeps the mutation entirely inside an effect (no Strict Mode double-run hazard) and gives React a proper state-driven re-render signal. The current implementation may work fine in production builds, but will produce subtle bugs in React 18+ Strict Mode dev where effects/memos intentionally double-invoke.

Minor — refetch() on manual refresh button doesn't reset the accumulator:
Line 494 calls refetch() which re-fetches only page 1. The accumulator retains all prior messages, which is the desired behaviour. However if the user specifically wants to see a "clean" feed (e.g. they suspect stale data), there's no way to flush the accumulator without switching channels. A reset-on-explicit-refresh option might be worth a future ticket.


src/utils/unifiedMessageAccumulator.test.ts

Positives:

  • The core regression test ("retains a message that is gone from a later poll") directly captures the bug scenario — excellent.
  • Good coverage of: sort order, intra/cross-page dedup, upsert-wins, undefined pages, cap behaviour (both snapshot length and acc.size trimmed).

Gap — concurrent channel blending (the reset path):
The test suite doesn't cover the channel-reset guard. Specifically: if selectedChannel changes between two foldUnifiedMessagePages calls, does the test confirm that old-channel messages don't bleed into the new channel's feed? The reset happens in UnifiedMessagesPage.tsx, not in foldUnifiedMessagePages itself, so the accumulator utility doesn't know about channels. A test (or comment) clarifying this responsibility split would be helpful for future maintainers.

Gap — empty page in multi-page response:
No test covers pages = [[], [msg('a', 100)]] (an empty first page). This is an edge case the server might return under some query conditions.


Security

No concerns. The accumulator is a pure client-side data structure; there's no XSS risk beyond what already exists in renderMessageWithLinks. The dedupKey is used only as a Map key, not rendered directly.


Summary

Area Status
Core fix correctness ✅ Sound
API / types ✅ Well-defined
Ref mutation in useMemo ⚠️ Technically violates React rules; harmless in prod but risky in Strict Mode
Test coverage of the bug ✅ Direct regression test present
Test coverage of channel reset ⚠️ Not covered in unit tests (responsibility split between util and page)
Security ✅ No concerns
Performance ✅ O(N) per poll; cap prevents unbounded growth

The useMemo-with-ref-mutation pattern is the one item I'd flag for a fix before merging to main, given that React Strict Mode double-invocation will mutate accumulatorRef twice per poll in development and could surface confusing duplicate-entry bugs during dev testing.

@Yeraze Yeraze merged commit 53926b0 into main Jun 25, 2026
19 checks passed
@Yeraze Yeraze deleted the fix/unified-feed-accumulate-3719 branch June 25, 2026 12:54
Yeraze added a commit that referenced this pull request Jun 25, 2026
…the window-widening

The limit*5 widening in this PR was a mitigation for the #3719 disappearance:
it kept a single-source-exclusive message inside the server window for ~167s.
The client-side append-only accumulator merged in #3738 now makes a displayed
message persist regardless of how the server window moves, at any traffic rate,
so the extra fetch headroom is redundant and only adds per-poll cost. Restore
the original 2x (same-packet dedup headroom). The _dbchan/_radio dedup-key fix
in this PR is a separate correctness bug and is kept.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SJPe6J5vKrcbwzt6vCdtrt
Yeraze added a commit that referenced this pull request Jun 25, 2026
…x region discovery timeout (#3719, #3734) (#3736)

* fix: prevent messages from vanishing in multi-source unified feed (#3719)

Two compounding bugs caused messages to disappear from the unified channel
view ~60 seconds after arrival when multiple sources are connected.

1. fetchLimit starvation: the per-source window was limit×2 = 200 rows.
   A high-traffic MQTT bridge (e.g. a continental Canadaverse bridge at
   ~3 msg/s) pushes a message exclusive to that source beyond position 200
   in ~60 s, making it invisible to the unified endpoint on the next poll.
   Raised to limit×5 = 500, giving ~167 s of headroom at 3 msg/s — well
   above the 10 s poll interval in all realistic deployments.

2. _dbchan/_radio suffix breaks dedup key: meshtasticManager inserts
   server-decrypted copies of a message with IDs ending in `_dbchan` or
   `_radio`. extractPacketIdFromRowId returned null for these because the
   last underscore-segment was not numeric, so those copies got a
   text+timestamp dedupKey instead of the correct packetId-based key.
   They appeared as duplicate entries or were sorted separately rather than
   merging with the originals receptions array. Fixed by stripping known
   server-copy suffixes before extracting the numeric packet ID.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014DhSciCtki5JMrpUrKxwnj

* fix(meshcore): drop tag guard in request_regions — ANON_REQ BinaryResponse tag is unmatched (#3734)

The #3702 fix assumed sendToRadioFrame fires Sent with no expectedAckCrc
(tag = null → skip matching). In practice the firmware DOES populate
expectedAckCrc (CRC of the outgoing ANON_REQ packet), but the repeater's
BinaryResponse (0x8C) carries tag = 0 — a different scheme than SendBinaryReq
(50), which explicitly echoes the CRC. With tag = non-zero CRC and
BinaryResponse.tag = 0, the old guard:

  if (tag !== null && tag !== 0 && r?.tag !== tag) return;

evaluates to true and silently drops every response, causing the 15-second
inner timeout and "No regions found" for every user attempt (#3734).

Fix: remove the tag-matching guard entirely. CMD_SEND_ANON_REQ (57) does
not use CRC-echoing tag correlation; operations are serialized by
runExclusiveRadioOp so the first BinaryResponse after sentReceived is
guaranteed to be ours. The tag variable is also removed since it is no
longer used.

Updates: correct the stale runExclusiveRadioOp comment that claimed
sendToRadioFrame fires Sent with no expectedAckCrc (it does).

Test: add a regression test simulating the exact #3734 scenario —
expectedAckCrc = 0xcafe1234 in Sent, tag = 0 in BinaryResponse — and
verify the regions are resolved rather than timing out.

Fixes #3734

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014DhSciCtki5JMrpUrKxwnj

* revert fetchLimit to 2x — client append-only feed (#3738) supersedes the window-widening

The limit*5 widening in this PR was a mitigation for the #3719 disappearance:
it kept a single-source-exclusive message inside the server window for ~167s.
The client-side append-only accumulator merged in #3738 now makes a displayed
message persist regardless of how the server window moves, at any traffic rate,
so the extra fetch headroom is redundant and only adds per-poll cost. Restore
the original 2x (same-packet dedup headroom). The _dbchan/_radio dedup-key fix
in this PR is a separate correctness bug and is kept.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SJPe6J5vKrcbwzt6vCdtrt

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant