fix(unified): append-only message feed so windowing can't evict a seen message (#3719)#3738
Conversation
…seen message (#3719) The unified feed rendered straight off the latest poll (data.pages), which is a capped newest-N window. A message exclusive to a high-traffic source gets pushed past the window within ~60s as newer traffic arrives, so the next poll drops it from the view even though it's still in the DB — the #3719/#3720 disappearance. Structural fix: fold every poll's pages into a persistent map keyed by dedupKey and render from that, making the live feed append-only. Once a message is shown it stays shown regardless of how the server window shifts; upserts prefer the newest (most-complete) merged copy; sort stays by createdAt (#3122). Reset on channel change; capped at 5000 (oldest dropped, still re-fetchable on scroll). This decouples display persistence from server window size, so it holds at any source traffic rate — unlike raising fetchLimit, which only buys proportional headroom. Complements the server-side fixes in #3736. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SJPe6J5vKrcbwzt6vCdtrt
|
Claude finished @Yeraze's task —— View job PR Review: fix(unified): append-only message feedTasks:
Overall AssessmentThis is a well-designed structural fix for a real disappearing-message bug. The core idea — making the live feed append-only via a persistent
|
| Area | Status |
|---|---|
| Core fix correctness | ✅ Sound |
| API / types | ✅ Well-defined |
Ref mutation in useMemo |
|
| Test coverage of the bug | ✅ Direct regression test present |
| Test coverage of channel reset | |
| Security | ✅ No concerns |
| Performance | ✅ O(N) per poll; cap prevents unbounded growth |
The useMemo-with-ref-mutation pattern is the one item I'd flag for a fix before merging to main, given that React Strict Mode double-invocation will mutate accumulatorRef twice per poll in development and could surface confusing duplicate-entry bugs during dev testing.
…the window-widening The limit*5 widening in this PR was a mitigation for the #3719 disappearance: it kept a single-source-exclusive message inside the server window for ~167s. The client-side append-only accumulator merged in #3738 now makes a displayed message persist regardless of how the server window moves, at any traffic rate, so the extra fetch headroom is redundant and only adds per-poll cost. Restore the original 2x (same-packet dedup headroom). The _dbchan/_radio dedup-key fix in this PR is a separate correctness bug and is kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SJPe6J5vKrcbwzt6vCdtrt
…x region discovery timeout (#3719, #3734) (#3736) * fix: prevent messages from vanishing in multi-source unified feed (#3719) Two compounding bugs caused messages to disappear from the unified channel view ~60 seconds after arrival when multiple sources are connected. 1. fetchLimit starvation: the per-source window was limit×2 = 200 rows. A high-traffic MQTT bridge (e.g. a continental Canadaverse bridge at ~3 msg/s) pushes a message exclusive to that source beyond position 200 in ~60 s, making it invisible to the unified endpoint on the next poll. Raised to limit×5 = 500, giving ~167 s of headroom at 3 msg/s — well above the 10 s poll interval in all realistic deployments. 2. _dbchan/_radio suffix breaks dedup key: meshtasticManager inserts server-decrypted copies of a message with IDs ending in `_dbchan` or `_radio`. extractPacketIdFromRowId returned null for these because the last underscore-segment was not numeric, so those copies got a text+timestamp dedupKey instead of the correct packetId-based key. They appeared as duplicate entries or were sorted separately rather than merging with the originals receptions array. Fixed by stripping known server-copy suffixes before extracting the numeric packet ID. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014DhSciCtki5JMrpUrKxwnj * fix(meshcore): drop tag guard in request_regions — ANON_REQ BinaryResponse tag is unmatched (#3734) The #3702 fix assumed sendToRadioFrame fires Sent with no expectedAckCrc (tag = null → skip matching). In practice the firmware DOES populate expectedAckCrc (CRC of the outgoing ANON_REQ packet), but the repeater's BinaryResponse (0x8C) carries tag = 0 — a different scheme than SendBinaryReq (50), which explicitly echoes the CRC. With tag = non-zero CRC and BinaryResponse.tag = 0, the old guard: if (tag !== null && tag !== 0 && r?.tag !== tag) return; evaluates to true and silently drops every response, causing the 15-second inner timeout and "No regions found" for every user attempt (#3734). Fix: remove the tag-matching guard entirely. CMD_SEND_ANON_REQ (57) does not use CRC-echoing tag correlation; operations are serialized by runExclusiveRadioOp so the first BinaryResponse after sentReceived is guaranteed to be ours. The tag variable is also removed since it is no longer used. Updates: correct the stale runExclusiveRadioOp comment that claimed sendToRadioFrame fires Sent with no expectedAckCrc (it does). Test: add a regression test simulating the exact #3734 scenario — expectedAckCrc = 0xcafe1234 in Sent, tag = 0 in BinaryResponse — and verify the regions are resolved rather than timing out. Fixes #3734 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014DhSciCtki5JMrpUrKxwnj * revert fetchLimit to 2x — client append-only feed (#3738) supersedes the window-widening The limit*5 widening in this PR was a mitigation for the #3719 disappearance: it kept a single-source-exclusive message inside the server window for ~167s. The client-side append-only accumulator merged in #3738 now makes a displayed message persist regardless of how the server window moves, at any traffic rate, so the extra fetch headroom is redundant and only adds per-poll cost. Restore the original 2x (same-packet dedup headroom). The _dbchan/_radio dedup-key fix in this PR is a separate correctness bug and is kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SJPe6J5vKrcbwzt6vCdtrt --------- Co-authored-by: Claude <noreply@anthropic.com>
Summary
Structural fix for the #3719 / #3720 disappearing-message bug, complementing the server-side mitigations in #3736.
Root cause (read side):
UnifiedMessagesPagerenderedallMessagesstraight offdata.pages— i.e. the latest poll of a capped newest-N server window. A message exclusive to a high-traffic source (e.g. a continental MQTT bridge) gets pushed past that window within ~60s as newer traffic arrives, so the next 10s poll's page 1 no longer contains it and it vanishes from the view — even though the row is still in the DB.#3736 raises the server fetch window (
limit×2 → limit×5), which buys ~167s of headroom at ~3 msg/s. That's a mitigation: a burstier source or longer poll gap re-opens the same hole, and it fetches more rows every poll.This PR fixes it structurally on the client: the live feed becomes append-only. Every poll's pages are folded into a persistent map keyed by
dedupKey; we render from that map instead of the latest poll. Once a message has been shown it stays shown, regardless of how the server window shifts — so the disappearance can't happen at any source traffic rate.Changes
src/utils/unifiedMessageAccumulator.ts— purefoldUnifiedMessagePages(acc, pages, cap):dedupKeyso a later, more-complete merged copy (extrareceptions, ack state, upgraded names) wins.createdAtascending (chat layout; clock-skew-safe, [BUG] Large Meshtastic TCP node repeatedly disconnects during/after full config sync; passive per-source mode improves stability #3122).UnifiedMessagesPage.tsx— replaced the page-flattenuseMemowith auseRefaccumulator folded via the helper; reset synchronously on channel change so feeds never blend.Tradeoff
A message deleted server-side stays visible until channel switch / refresh (rare; the alternative — re-introducing windowing eviction — is worse). Memory is bounded by the 5000 cap.
Tests
unifiedMessageAccumulator.test.ts— sort, intra/cross-page dedup, upsert-wins, undefined-pages, cap-drops-oldest, and the core regression: a message present in poll 1 but absent from poll 2 stays in the feed.Refs #3719, #3720, #3736
🤖 Generated with Claude Code