Skip to content

feat(traceroutes): add packetId column for cross-source trace correlation (#3623)#3643

Merged
Yeraze merged 1 commit into
mainfrom
feat/3623-traceroute-packet-id
Jun 22, 2026
Merged

feat(traceroutes): add packetId column for cross-source trace correlation (#3623)#3643
Yeraze merged 1 commit into
mainfrom
feat/3623-traceroute-packet-id

Conversation

@Yeraze

@Yeraze Yeraze commented Jun 22, 2026

Copy link
Copy Markdown
Owner

Summary

Closes #3623.

The traceroutes table stored no reference to the originating Meshtastic packet, so the same physical traceroute heard on multiple sources (e.g. a direct TCP listener and an MQTT feed) couldn't be correlated, and individual traces couldn't be grouped/displayed per-packet (all flood-routing branches of one packet) instead of only the collapsed resulting path.

This adds a nullable packetId column to the traceroutes table and populates it on every ingestion path.

Changes

  • Migration 097 (097_add_packet_id_to_traceroutes.ts) — adds packetId across SQLite / PostgreSQL / MySQL, idempotent. BIGINT on PG/MySQL since Meshtastic packet ids are unsigned 32-bit and overflow a signed INTEGER; SQLite INTEGER is already 64-bit. NULL = pre-migration rows.
  • Schema (db/schema/traceroutes.ts, all three backends) + types (db/types.ts and the database.ts facade interface).
  • Repository (db/repositories/traceroutes.ts) — threads packetId through insertTraceroute, upsertTracerouteSync (insert + pending-update branches), updateTracerouteResponse, and the row-normalization helpers.
  • PopulationmeshPacket.id on the TCP path (meshtasticManager.ts) and packet.id on the MQTT path (mqttIngestion.ts). For traceroutes we initiate, the pending row's packetId is filled in when the response arrives via updateTracerouteResponse.

Note on routeBack

The issue discussion suggested return-path persistence was also missing. It turns out routeBack is already persisted and returned end-to-end (column, insert/upsert, select, normalization all present), so no change was needed there.

Scope

This delivers the storage/correlation foundation. The per-packet branch visualization shown in the issue is a separate frontend feature that this unblocks (the packetId is now stored and returned by the traceroute queries).

Testing

  • 2 new repository round-trip tests — insert-with-packetId, and pending→response fill — run across SQLite/PostgreSQL/MySQL.
  • Migration count (96→97) and last-migration assertions updated.
  • Full Vitest suite green (7127 passed, 0 failed).
  • tsc -p tsconfig.server.json --noEmit clean.

🤖 Generated with Claude Code

…tion (#3623)

The traceroutes table stored no reference to the originating Meshtastic
packet, so the same physical traceroute heard on multiple sources (e.g. a
direct TCP listener and an MQTT feed) could not be correlated, and individual
traces couldn't be grouped/displayed per-packet (all flood-routing branches of
one packet) rather than only as the collapsed resulting path.

Adds a nullable `packetId` column to the traceroutes table:
- Migration 097 (SQLite/PostgreSQL/MySQL, idempotent). BIGINT on PG/MySQL
  since Meshtastic packet ids are unsigned 32-bit and overflow signed INTEGER;
  SQLite INTEGER is already 64-bit. NULL = pre-migration rows.
- Schema (all three backends), DbTraceroute types (db/types + database facade),
  and the repository insert/upsert/update + normalization paths.
- Populated from `meshPacket.id` on the TCP path (meshtasticManager) and
  `packet.id` on the MQTT path (mqttIngestion). For traceroutes we initiated,
  the pending row's packetId is filled in when the response arrives via
  updateTracerouteResponse.

`routeBack` (the other gap flagged in the issue discussion) was already
persisted end-to-end, so no change was needed there.

The packetId is now stored and returned by the traceroute queries; the
per-packet branch visualization in the issue is a separate frontend feature
this unblocks.

Tests: 2 new repository round-trip tests (insert + response-fill, across all
three backends); migration count/last-name assertions updated. Full Vitest
suite green (7127 passed), server tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011JEaCGwY9Wz8jeV4e22GW4
@github-actions

github-actions Bot commented Jun 22, 2026

Copy link
Copy Markdown

Claude finished @Yeraze's task —— View job


PR Review: feat(traceroutes): add packetId column for cross-source trace correlation (#3623)

  • Read changed files
  • Analyze code quality and best practices
  • Check for potential bugs
  • Review performance considerations
  • Check security concerns
  • Evaluate test coverage
  • Post final review

Overall this is a clean, well-scoped change. The implementation follows existing project conventions closely and the migration is properly idempotent across all three backends. A few observations below, ranging from a genuine bug to minor style points.

Bug: Inconsistent packetId handling in meshtasticManager.ts

In src/server/meshtasticManager.ts:7185, the TCP ingestion path stores packetId using Number():

packetId: meshPacket.id != null ? Number(meshPacket.id) : null,

But at line 7205, the sibling telemetry call for the same packet uses the safe unsigned conversion:

packetId: meshPacket.id ? Number(meshPacket.id) : undefined,

Neither of these applies the >>> 0 unsigned coercion that the MQTT path (mqttIngestion.ts:587) uses correctly:

packetId: typeof packet.id === 'number' ? packet.id >>> 0 : null,

Meshtastic packet IDs are unsigned 32-bit values. If meshPacket.id arrives as a negative-valued signed integer (possible depending on how the protobuf decoder represents the value), Number() will store the wrong value. The MQTT path protects against this with >>> 0; the TCP path should too. This inconsistency existed before this PR but the new packetId column makes it observable. Recommend aligning to:

packetId: meshPacket.id != null ? (Number(meshPacket.id) >>> 0) : null,

Minor: DbTraceroute type uses number | null but schema uses BIGINT

src/db/types.ts:196:

packetId?: number | null;

JS number is a 64-bit float, so it safely covers the full unsigned 32-bit range (max ~4.29 billion). No precision issue here, just worth noting this is consistent with how packetId is typed on DbTelemetry (types.ts:171) and DbPacketLog (types.ts:311), so this is fine and conventional.

Minor: normalizeTracerouteRow vs getAllTraceroutesSync

normalizeTracerouteRow (traceroutes.ts:864-875) converts null to undefined for packetId, which is correct. getAllTraceroutesSync (traceroutes.ts:881-897) does the same inline. The duplication is minor but worth noting — if a new nullable column is added later, there are two places to update instead of one. Not blocking.

SQLite migration error handling is slightly different from other migrations

migration 097 SQLite catches any error and logs a warning if it isn't duplicate column:

} else {
  logger.warn('Could not add packetId to traceroutes:', e.message);
}

This matches migrations 019 (add_channel_to_traceroutes) and others exactly, so it's consistent. Fine.

Test coverage: good

The two new round-trip tests (packetId (#3623) - persists on insert... and packetId (#3623) - populated when a pending traceroute gets its response) cover the most important correctness points. Using 4_000_000_001 as the test value to explicitly exercise the signed-32-bit overflow boundary is a nice touch.

One gap: there is no test that packetId round-trips through upsertTracerouteSync (the SQLite sync path used by the main insertTraceroute facade for SQLite). The async repository path is tested, but the sync path in upsertTracerouteSync also sets packetId at lines 551 and 566. A test exercising the SQLite sync upsert (update branch and insert branch) with a packetId would round out coverage.

No security or performance concerns

The column is nullable, indexed only via the primary key, and the migration adds no indexes. For the intended cross-source correlation use case a future index on (packetId) or (packetId, sourceId) may be desirable, but that's an optimization for the visualization feature this PR explicitly defers.


Summary: One real bug (unsigned coercion inconsistency on the TCP path), one coverage gap (SQLite sync upsert path not tested), and the rest is solid. The migration, schema, repository threading, and ingestion population all look correct.

@Yeraze Yeraze merged commit e53bda9 into main Jun 22, 2026
27 of 28 checks passed
@Yeraze Yeraze deleted the feat/3623-traceroute-packet-id branch June 22, 2026 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEAT] database is lacking 'packet id' column in 'traceroutes' table

1 participant