feat(mqtt): full source-scoped ingest with channel decryption and unified-view fixes#3089
Conversation
…fied-view fixes
MQTT bridges were connecting to upstream brokers but contributing almost
nothing to the unified views. This commit fixes the entire ingest path so
MQTT-relayed traffic lands properly attributed and dedups across TCP +
MQTT receptions of the same mesh packet.
Ingest pipeline (src/server/mqttIngestion.ts):
- ingestServiceEnvelope is now async, dispatches four additional portnums
with the same storage semantics as the TCP path: TRACEROUTE_APP (incl.
route segments + hop telemetry), NEIGHBORINFO_APP, PAXCOUNTER_APP, and
STORE_FORWARD_APP (ROUTER_HEARTBEAT + ROUTER_TEXT_*).
- Server-side channel decryption via channelDecryptionService, mirroring
meshtasticManager.processMeshPacket — encrypted packets get a synthetic
decoded field populated from any matching channel_database PSK.
- New bootstrapMqttChannelDatabase auto-seeds the well-known LongFast key
(`AQ==`, expanded via expandShorthandPsk) so MQTT decryption works out
of the box. Idempotent across multiple MQTT sources.
- Auto-records each (channel_name, packet.channel) pair seen on incoming
envelopes into the per-source channels table, so the unified channels
picker surfaces MQTT-side channel labels.
- Message row IDs now use the TCP convention `${sourceId}_${fromNum}_${packetId}`.
This is load-bearing — extractPacketIdFromRowId in unifiedRoutes.ts splits
on `_` and reads the trailing segment to build the cross-source dedup key.
Previously diverged with hyphens, which made the same packet appear N
times in the unified view (one per receiving source).
- Messages now go through databaseService.messages.insertMessage(msg, sourceId)
(repo) instead of databaseService.insertMessage(msg) (facade) — the facade
drops the sourceId, leaving rows orphaned with sourceId=NULL and invisible
to source-scoped queries.
Geo-membership seed (src/server/mqttPacketFilter.ts, mqttBridgeManager.ts):
- New seedMembership(positions) marks nodes whose stored position is inside
the bbox as 'in' on bridge start, so non-POSITION traffic from already-
known nodes flows without waiting for a fresh POSITION_APP broadcast.
- New seedTrustedNodes(nodeNums) bypasses the bbox check for any node
already heard directly by a non-MQTT source — fixes the case where the
operator's own station (GPS off, no stored position) was geo-dropped on
MQTT-relayed copies of its packets.
Unified routes (src/server/routes/unifiedRoutes.ts):
- Telemetry endpoint: getLatestTelemetryByNode now takes a sourceId so
cross-source telemetry no longer misattributes to whichever source happens
to be iterating in the outer loop. Telemetry repo signature widened.
- Channels endpoint: multi-slot match (filter, not find) so MQTT sources
with multiple slots sharing one channel name all resolve.
- unifiedChannelDisplayName now accepts a preset hint; slot 0 with empty
name resolves to the firmware-derived label (MediumFast, LongFast, etc.)
from the persisted `lora.preset.<sourceId>` setting. This collapses
TCP empty-name slot 0 with MQTT-side "MediumFast" rows into one picker
entry. meshtasticManager persists the preset on each LoRa config response.
- New MODEM_PRESET_CHANNEL_NAMES constant + modemPresetChannelName helper
in src/server/constants/meshtastic.ts (firmware-spec pascal-case names,
no spaces — distinct from the human-friendly "Long Fast" labels).
- Expanded jsdoc on extractPacketIdFromRowId documenting the row-ID
contract so future ingest paths don't diverge.
Misc:
- docker-compose.dev.yml: remove the standalone LN4CY mqtt-proxy sidecar
service. The feature has been integrated.
Tests (10064 → 10064 passing locally):
- mqttPacketFilter: seedMembership (7 cases) + seedTrustedNodes (4 cases)
- mqttIngestion: per-portnum handler tests (TRACEROUTE/NEIGHBORINFO/
PAXCOUNTER/STORE_FORWARD), plus updated mocks for the new repo paths.
- unifiedRoutes: preset-derived slot 0 name, fallback to "Primary" when
preset absent, and TCP+MQTT collapse into one picker entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Claude finished @Yeraze's task —— View job Code Review Results✅ Overall AssessmentThis is a well-implemented and comprehensive PR that successfully addresses a significant architectural gap in MQTT ingestion. The code quality is high, follows established patterns, and demonstrates deep understanding of the MeshMonitor codebase. 🏆 StrengthsExcellent Architecture & Design
Code Quality & Best Practices
Security & Reliability
Test Coverage
|
Patch release covering the MQTT-ingest rework (#3089) and follow-on UX fixes (#3093 channel display-name, #3091 Channel Database move to Global) plus the usual dependency bumps and minor fixes. Version bumps across all five canonical files: package.json, package-lock.json (regenerated), helm/meshmonitor/Chart.yaml, desktop/package.json, desktop/src-tauri/tauri.conf.json. CHANGELOG: organized under [4.6.2] with the user-visible MQTT/channel items called out as the headline changes. CLAUDE.md: - Migration count 62+ → 63+ (latest: 063_drop_source_id_from_channel_database) - "Every row carries a sourceId" rule gains an exception note for channel_database — it's intentionally global (decryption service tries every enabled row regardless of source) and migration 063 dropped the dead sourceId column. Blog post + news.json: "MeshMonitor v4.6.2 — MQTT in the Unified views & cleaner channel names" — three things users will notice immediately (slot-0 channel renamed from "Primary" to the modem-preset label, Channel Database moved to Global Settings, MQTT sources now participate in Unified Messages and Unified Telemetry) plus action items after upgrade. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
MQTT bridges were connecting to upstream brokers but contributing almost nothing to the unified views. This PR fixes the entire ingest path so MQTT-relayed traffic lands properly attributed, decrypts via the channel database, and dedups across TCP + MQTT receptions of the same mesh packet.
Before: ~99% packet drop rate on Florida MQTT, zero messages in DB from MQTT sources, no MQTT entries in the unified channels picker or source filter.
After: ~25–35% ingest rate, full nodes/messages/telemetry/traceroutes/neighbors populated per MQTT source, picker entries from TCP and MQTT collapse to one logical channel, and a single mesh packet heard by TCP + Florida MQTT + Yeraze Broker appears as one unified entry with three receptions.
What's in here
Ingest pipeline (
src/server/mqttIngestion.ts)ingestServiceEnvelopeis async now and dispatches four additional portnums with the same storage semantics as the TCP path:TRACEROUTE_APP(incl. route segments + hop telemetry),NEIGHBORINFO_APP,PAXCOUNTER_APP, andSTORE_FORWARD_APP(ROUTER_HEARTBEAT+ROUTER_TEXT_*).channelDecryptionService, mirroringmeshtasticManager.processMeshPacket— encrypted packets get a syntheticdecodedfield populated from any matchingchannel_databasePSK.bootstrapMqttChannelDatabaseauto-seeds the well-known LongFast key (AQ==, expanded byexpandShorthandPsk) so MQTT decryption works out of the box for new MQTT sources. Idempotent across multiple sources.(envelope.channelId, packet.channel)pair seen into the per-sourcechannelstable so the unified channels picker surfaces MQTT-side labels.${sourceId}_${fromNum}_${packetId}. This is load-bearing —extractPacketIdFromRowIdinunifiedRoutes.tssplits on_and reads the trailing segment for the cross-source dedup key. Previously diverged with hyphens, which made the same packet appear N times in the unified view.databaseService.messages.insertMessage(msg, sourceId)(repo) instead ofdatabaseService.insertMessage(msg)(facade). The facade drops thesourceId, leaving rows orphaned withsourceId=NULLand invisible to source-scoped queries.Geo-membership seed (
src/server/mqttPacketFilter.ts,mqttBridgeManager.ts)seedMembership(positions)marks nodes whose stored position is inside the bbox as'in'on bridge start, so non-POSITION traffic from already-known nodes flows without waiting for a freshPOSITION_APPbroadcast (which can be hours and is reset on every restart).seedTrustedNodes(nodeNums)bypasses the bbox check for any node already heard directly by a non-MQTT source — fixes the case where the operator's own station (GPS off, no stored position) was geo-dropped on MQTT-relayed copies of its own packets.Unified routes (
src/server/routes/unifiedRoutes.ts)/telemetry:getLatestTelemetryByNodenow takes asourceIdso cross-source telemetry no longer misattributes to whichever source happens to be iterating in the outer loop. Telemetry repo signatures widened./channels+/messages: multi-slot match (filter, notfind) so MQTT sources with multiple slots sharing one channel name all resolve.unifiedChannelDisplayNameaccepts a preset hint; slot 0 with empty name resolves to the firmware-derived label (MediumFast,LongFast, etc.) from the persistedlora.preset.<sourceId>setting. Collapses TCP empty-name slot 0 with MQTT-side"MediumFast"rows into one picker entry.meshtasticManagerpersists the preset on each LoRa config response.MODEM_PRESET_CHANNEL_NAMESconstant +modemPresetChannelNamehelper insrc/server/constants/meshtastic.ts(firmware-spec pascal-case names — distinct from the human-friendly"Long Fast"labels).extractPacketIdFromRowIddocumenting the row-ID contract so future ingest paths don't diverge.Misc
docker-compose.dev.yml: removed the standalone LN4CYmqtt-proxysidecar service. The feature has been integrated.Test plan
npx vitest run— 10064 tests pass (4 new in mqttPacketFilter, 10 new in mqttIngestion, 3 new in unifiedRoutes)npx tsc --noEmit -p tsconfig.json— cleanchannel_databaseMediumFastshows in the unified picker (wasPrimary)🤖 Generated with Claude Code