Skip to content

fix timing issues#166

Merged
dannym-arx merged 6 commits intomasterfrom
fix/message-timing-issues-157
Feb 6, 2026
Merged

fix timing issues#166
dannym-arx merged 6 commits intomasterfrom
fix/message-timing-issues-157

Conversation

@dannym-arx
Copy link
Contributor

@dannym-arx dannym-arx commented Feb 2, 2026

This PR fixes message timing and ordering inconsistencies by recording when messages are processed locally (processed_at) and using that timestamp as a secondary sort key so ordering is deterministic even when created_at ties or devices have clock skew. It updates message construction, group last-message bookkeeping, in-memory and SQLite storage, storage traits, and UniFFI bindings so group.last_message_id aligns with messages()[0].id under the canonical display order.

What changed:

  • Messages now include processed_at: Timestamp set when the client processes/receives the message, and Groups now include last_message_processed_at: Option.
  • Message display ordering is standardized to created_at DESC, processed_at DESC, id DESC across storage implementations (memory and SQLite) to preserve reception order and ensure deterministic ordering.
  • Added Group::update_last_message_if_newer to atomically decide and update last_message_at, last_message_processed_at, and last_message_id using the canonical display-order comparison.
  • SQLite migration V002 adds messages.processed_at (backfilled from created_at), adds groups.last_message_processed_at (backfilled from last_message_at), and creates composite index idx_messages_sorting to support ORDER BY created_at DESC, processed_at DESC, id DESC.
  • Propagated the new fields and behavior across crates: mdk-core, mdk-memory-storage, mdk-sqlite-storage, mdk-storage-traits, and mdk-uniffi, plus changelogs, docs, and test updates.

Security impact:

  • None identified; no changes to cryptography, key handling, SQLCipher configuration, credential validation, or sensitive error messages.

Protocol changes:

  • None to MLS protocol or signed-event semantics; created_at remains the sender-derived event timestamp and processed_at is client-local metadata (Nostr semantics unchanged).

API surface:

  • Breaking changes: public Message now includes processed_at: Timestamp and public Group includes last_message_processed_at: Option, requiring callers that construct these types to provide or handle the new fields.
  • Behavioral change: GroupStorage::messages() ordering semantics changed to created_at DESC, processed_at DESC, id DESC while preserving the same API signatures.
  • Storage schema change: SQLite migrations add processed_at to messages, last_message_processed_at to groups, and create a composite index idx_messages_sorting to support the new ordering.
  • UniFFI/FFI change: Message.processed_at and Group.last_message_processed_at are exposed on FFI structs (u64 / Option) and are populated from the internal Timestamp fields.

Testing:

  • Tests were updated across crates to initialize and assert on processed_at and last_message_processed_at, and many tests were standardized to use shared deterministic timestamps to reflect the new ordering and backfill semantics.

@coderabbitai
Copy link

coderabbitai bot commented Feb 2, 2026

📝 Walkthrough

Walkthrough

Adds processed_at to Message and last_message_processed_at to Group across core, storage-traits, storage implementations, and UniFFI; updates message sorting to created_at DESC, processed_at DESC, id DESC; adds SQLite migration to store/backfill processed_at and index; tests updated.

Changes

Cohort / File(s) Summary
Core message handling
crates/mdk-core/src/messages/create.rs, crates/mdk-core/src/messages/application.rs, crates/mdk-core/src/messages/process.rs, crates/mdk-core/src/groups.rs, crates/mdk-core/src/welcomes.rs, crates/mdk-core/src/epoch_snapshots.rs
Populate Message.processed_at at creation/processing; replace ad-hoc last-message assignments with Group::update_last_message_if_newer; initialize Group.last_message_processed_at at call sites and tests.
Public API / Storage traits
crates/mdk-storage-traits/src/messages/types.rs, crates/mdk-storage-traits/src/groups/types.rs, crates/mdk-storage-traits/src/groups/mod.rs, crates/mdk-storage-traits/src/test_utils.rs, crates/mdk-storage-traits/tests/...
Add processed_at: Timestamp to Message and last_message_processed_at: Option<Timestamp> to Group; add Message::display_order_cmp and Group::update_last_message_if_newer; update docs and tests to reflect composite ordering.
Memory storage
crates/mdk-memory-storage/src/groups.rs, crates/mdk-memory-storage/src/messages.rs, crates/mdk-memory-storage/src/lib.rs
Change in-memory message sorting to created_at DESC, processed_at DESC, id DESC; add last_message_processed_at to Group and adapt tests to use shared timestamps.
SQLite schema & migration
crates/mdk-sqlite-storage/migrations/V002__add_processed_at_to_messages.sql
Add processed_at (messages) and last_message_processed_at (groups); backfill existing rows; create composite index idx_messages_sorting on (mls_group_id, created_at DESC, processed_at DESC, id DESC).
SQLite storage implementation
crates/mdk-sqlite-storage/src/db.rs, crates/mdk-sqlite-storage/src/groups.rs, crates/mdk-sqlite-storage/src/messages.rs, crates/mdk-sqlite-storage/src/lib.rs
Read/write processed_at and last_message_processed_at in row mapping and persistence; include processed_at in INSERT/UPDATE and ORDER BY; add backward-compatible fallbacks for legacy rows; tests updated.
UniFFI bindings
crates/mdk-uniffi/src/lib.rs, crates/mdk-uniffi/CHANGELOG.md
Expose processed_at (Message) and last_message_processed_at (Group) in UniFFI structs and map from internal timestamps; update conversion implementations and docs.
Changelogs & docs
crates/mdk-core/CHANGELOG.md, crates/mdk-memory-storage/CHANGELOG.md, crates/mdk-sqlite-storage/CHANGELOG.md, crates/mdk-storage-traits/CHANGELOG.md, crates/mdk-uniffi/CHANGELOG.md
Document new fields, composite deterministic sorting change, SQLite migration and index, and test updates.

Sequence Diagram(s)

sequenceDiagram
    participant Core as Core (message creation)
    participant Storage as Storage (GroupStorage)
    participant DB as DB (SQLite / Memory)

    Core->>Core: now = Timestamp::now()\nconstruct Message(processed_at = now)
    Core->>Storage: save_message(message)
    Storage->>DB: INSERT/UPDATE message (created_at, processed_at, id)
    DB-->>Storage: write ok
    Storage->>Storage: group.update_last_message_if_newer(message)
    Storage->>DB: INSERT/UPDATE group (last_message_at, last_message_processed_at, last_message_id)
    DB-->>Storage: group write ok
    Storage-->>Core: ack saved
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested labels

breaking-change, storage

Suggested reviewers

  • erskingardner
  • jgmontoya
  • mubarakcoded
🚥 Pre-merge checks | ✅ 3 | ❌ 1
❌ Failed checks (1 inconclusive)
Check name Status Explanation Resolution
Title check ❓ Inconclusive The title 'fix timing issues' is vague and generic, failing to specify which timing issues or what aspect of the codebase is being addressed. Use a more specific title that describes the actual changes, such as 'Add message processing timestamps for consistent ordering' or 'Track processed_at timestamp to fix message ordering consistency'.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
No Sensitive Identifier Leakage ✅ Passed PR introduces timestamp fields and comparison methods without exposing sensitive identifiers or adding new logging that would leak sensitive data.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix/message-timing-issues-157

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link

github-actions bot commented Feb 2, 2026

✅ Coverage: 90.77% → 90.83% (+0.06%)

@dannym-arx dannym-arx mentioned this pull request Feb 2, 2026
@github-actions
Copy link

github-actions bot commented Feb 2, 2026

⚠️ Coverage: 90.79% → 90.79% (0.00%)

@github-actions
Copy link

github-actions bot commented Feb 4, 2026

✅ Coverage: 90.79% → 90.86% (+0.07%)

@github-actions
Copy link

github-actions bot commented Feb 6, 2026

✅ Coverage: 90.79% → 90.86% (+0.07%)

@github-actions
Copy link

github-actions bot commented Feb 6, 2026

✅ Coverage: 90.79% → 90.86% (+0.07%)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@crates/mdk-storage-traits/src/messages/types.rs`:
- Around line 91-106: The rustdoc links in the compare_display_keys function
comment reference Group fields as [Group::last_message_at],
[Group::last_message_processed_at], and [Group::last_message_id] which won't
resolve because Group isn't in scope; update those links to use fully qualified
paths like crate::groups::types::Group::last_message_at,
crate::groups::types::Group::last_message_processed_at, and
crate::groups::types::Group::last_message_id so cargo doc can resolve them.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
crates/mdk-storage-traits/src/groups/types.rs (1)

80-106: ⚠️ Potential issue | 🔴 Critical

Redact sensitive identifiers from Group Debug output.

Group derives Debug while containing mls_group_id, nostr_group_id, image_key, and image_nonce, which must not be exposed in Debug output. Replace the derive with a redacted Debug impl.

🔒 Proposed redacted Debug implementation
-#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
+#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
 pub struct Group {
     ...
 }
+
+impl fmt::Debug for Group {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.debug_struct("Group")
+            .field("mls_group_id", &"[REDACTED]")
+            .field("nostr_group_id", &"[REDACTED]")
+            .field("name", &self.name)
+            .field("description", &self.description)
+            .field("image_hash", &self.image_hash)
+            .field("image_key", &"[REDACTED]")
+            .field("image_nonce", &"[REDACTED]")
+            .field("admin_pubkeys", &self.admin_pubkeys)
+            .field("last_message_id", &self.last_message_id)
+            .field("last_message_at", &self.last_message_at)
+            .field("last_message_processed_at", &self.last_message_processed_at)
+            .field("epoch", &self.epoch)
+            .field("state", &self.state)
+            .finish()
+    }
+}
crates/mdk-storage-traits/src/messages/types.rs (1)

38-66: ⚠️ Potential issue | 🔴 Critical

Implement a custom Debug impl for Message to redact sensitive identifiers.

The Message struct at line 38 derives Debug while containing mls_group_id and other sensitive fields (content, tags, event) that must not be exposed in debug output. Remove the Debug derive and implement a custom Debug trait that redacts these fields.

🔒 Proposed redacted Debug implementation
-#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
+#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
 pub struct Message {
     ...
 }
+
+impl fmt::Debug for Message {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.debug_struct("Message")
+            .field("id", &self.id)
+            .field("pubkey", &self.pubkey)
+            .field("kind", &self.kind)
+            .field("mls_group_id", &"[REDACTED]")
+            .field("created_at", &self.created_at)
+            .field("processed_at", &self.processed_at)
+            .field("content", &"[REDACTED]")
+            .field("tags", &"[REDACTED]")
+            .field("event", &"[REDACTED]")
+            .field("wrapper_event_id", &self.wrapper_event_id)
+            .field("epoch", &self.epoch)
+            .field("state", &self.state)
+            .finish()
+    }
+}

This addresses the security requirement that sensitive identifiers (mls_group_id, group-related content, event data) must never be exposed in Debug trait implementations.

@github-actions
Copy link

github-actions bot commented Feb 6, 2026

✅ Coverage: 90.79% → 90.87% (+0.08%)

@dannym-arx dannym-arx merged commit 04ac262 into master Feb 6, 2026
12 checks passed
@dannym-arx dannym-arx deleted the fix/message-timing-issues-157 branch February 6, 2026 16:02
erskingardner added a commit to alltheseas/mdk that referenced this pull request Feb 18, 2026
…store

The snapshot_groups_table() and restore_group_from_snapshot() functions
were missing the last_message_processed_at column added in V002 (PR marmot-protocol#166).
After a MIP-03 commit race rollback, this column would reset to NULL,
undoing the message ordering fix for that group.
erskingardner added a commit to alltheseas/mdk that referenced this pull request Feb 18, 2026
…store

The snapshot_groups_table() and restore_group_from_snapshot() functions
were missing the last_message_processed_at column added in V002 (PR marmot-protocol#166).
After a MIP-03 commit race rollback, this column would reset to NULL,
undoing the message ordering fix for that group.
erskingardner added a commit that referenced this pull request Feb 18, 2026
* feat: add self-update tracking to Group struct (MIP-02)

Adds two fields to the Group struct:
- `needs_self_update: bool` — set by accept_welcome(), cleared when a
  pure self-update commit is merged
- `last_self_update_at: Option<Timestamp>` — updated every time a pure
  self-update commit is merged, enabling periodic rotation tracking

The merge_pending_commit() method inspects the StagedCommit before
merging to detect pure self-updates (update proposals only, no
add/remove/psk), matching the validation logic in
is_pure_self_update_commit().

Includes SQLite migration, UniFFI bindings, serde backward-compat
defaults, and test updates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add groups_needing_self_update() query

Default trait method on GroupStorage that returns active groups needing
a self-update: either post-join (needs_self_update=true) or stale
rotation (last_self_update_at older than threshold). Exposed via MDK
public API and UniFFI bindings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: replace two self-update fields with SelfUpdateState enum

Replace needs_self_update (bool) and last_self_update_at (Option<Timestamp>)
with a single self_update_state: SelfUpdateState field on Group. The enum has
three variants: NotRequired, Required (MIP-02 post-join), and
CompletedAt(Timestamp) (MIP-00 periodic rotation).

This eliminates impossible state combinations, reduces the SQLite migration
to a single column (last_self_update_at INTEGER), and fixes the snapshot/
restore mechanism to preserve self-update state across MIP-03 rollbacks.

Adds changelogs for mdk-storage-traits, mdk-core, mdk-sqlite-storage, and
mdk-uniffi.

* fix: consolidate changelog headers and prefix UniFFI self_update_state

- Remove duplicate '### Breaking changes' and '### Added' headers in
  mdk-storage-traits and mdk-uniffi changelogs, merging PR #184 entries
  into the existing sections.
- Prefix the CompletedAt timestamp in UniFFI with 'completed_at:' so
  FFI clients can reliably distinguish state variants by string prefix
  rather than parsing an ambiguous bare integer.

* style: fix import placement, derive order, comment clarity, and add serde tests

- Move in-function imports to module scope in epoch_snapshots.rs tests
  and cross_storage.rs (per AGENTS.md import placement rules)
- Fix SelfUpdateState derive order: Default at end per project convention
- Clarify is_self_update comment: all() is vacuously true on empty
  iterators, which is expected for path-based self-updates
- Rename local variable all_updates to no_non_update_proposals for
  clarity
- Add serde roundtrip tests for SelfUpdateState (NotRequired/Required/
  CompletedAt) and verify #[serde(default)] behavior on Group

* fix(sqlite-storage): include last_message_processed_at in snapshot/restore

The snapshot_groups_table() and restore_group_from_snapshot() functions
were missing the last_message_processed_at column added in V002 (PR #166).
After a MIP-03 commit race rollback, this column would reset to NULL,
undoing the message ordering fix for that group.

* fix(core): track self-update state in OwnCommitPending path

The OwnCommitPending handler in process.rs called OpenMLS
merge_pending_commit() directly, bypassing the self-update detection
in MDK::merge_pending_commit(). After a rollback/reprocess of a
self-update commit, self_update_state would remain Required instead
of transitioning to CompletedAt, causing a redundant self-update.

* refactor: remove NotRequired variant from SelfUpdateState

Group creators now start with CompletedAt(now) since creating a group
with a fresh key is effectively the first rotation. This ensures every
group is always on the periodic rotation clock and eliminates the blind
spot where creators were never flagged for staleness.

The enum now has two variants: Required (post-join, MIP-02) and
CompletedAt(Timestamp) (last rotation, MIP-00). The SQLite column
becomes NOT NULL DEFAULT 0.

---------

Co-authored-by: alltheseas <alltheseas@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Jeff Gardner <202880+erskingardner@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants