Skip to content

Activities can be delivered out of order, causing federation issues #536

@dahlia

Description

@dahlia

Problem

ActivityPub activities can be delivered out of order when they relate to the same object, causing federation issues. Specifically, when a post is created and quickly deleted, the Delete activity can arrive at remote instances before the Create activity, resulting in “zombie posts” that should have been deleted.

This was observed in production: hackers-pub/hackerspub#162 (comment).

Scenario

  1. User creates a post → Create(Note) activity is enqueued
  2. User immediately deletes the post → Delete(Note) activity is enqueued
  3. Both activities are sent to remote instances independently
  4. Due to parallel workers and lack of ordering guarantees, Delete(Note) may arrive first
  5. Remote instance ignores the delete (no post exists yet)
  6. Remote instance later receives and persists Create(Note)
  7. Result: “zombie post” exists on remote instance but not on origin

Root cause

The MessageQueue interface doesn't provide:

  1. Ordering guarantees — messages for the same object can be processed out of order
  2. Cancellation mechanism — can't cancel a queued Create when a Delete is enqueued shortly after
  3. Dependency tracking — no way to express that one activity depends on or supersedes another

While PostgresMessageQueue uses ORDER BY created, parallel workers can still process messages out of order. This issue affects all MessageQueue implementations (Postgres, Redis, SQLite, AMQP, Deno KV, etc.).

Proposed solutions

Option A: Add message cancellation API

interface MessageQueue {
  enqueue(message: any, options?: MessageQueueEnqueueOptions & { key?: string }): Promise<void>;
  cancel(key: string): Promise<boolean>; // Returns true if cancelled
}

Users could assign keys to messages and cancel pending ones:

await queue.enqueue(createActivity, { key: `create:${noteId}` });
// Later, if deleted quickly:
if (await queue.cancel(`create:${noteId}`)) {
  // Create was cancelled, no need to send Delete
} else {
  // Create already sent, send Delete
  await queue.enqueue(deleteActivity);
}

Pros: Flexible, allows application-level logic
Cons: Requires checking if cancellation succeeded; complex to implement in distributed queues

Option B: Per-object ordering

interface MessageQueueEnqueueOptions {
  delay?: Temporal.Duration;
  orderingKey?: string; // Messages with same key are processed in order
}

Activities for the same object would be guaranteed to arrive in order:

await queue.enqueue(createActivity, { orderingKey: noteId });
await queue.enqueue(deleteActivity, { orderingKey: noteId });
// Delete always arrives after Create

Pros: Matches ActivityPub's causal ordering semantics; works well with existing queue backends
Cons: May reduce parallelism; requires implementation changes in all queue backends

Option C: Activity batching

Allow grouping related activities so they're sent atomically or cancelled together:

interface MessageQueue {
  enqueueGroup(messages: any[], options?: { atomicCancel?: boolean }): Promise<string>;
  cancelGroup(groupId: string): Promise<boolean>;
}

Pros: Handles complex multi-message scenarios
Cons: Most complex to implement; may not fit all use cases; batching semantics unclear for ActivityPub

Additional context

This is a framework-level issue that affects any Fedify application with rapid Create/Delete operations. A proper fix would benefit the entire ecosystem.

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions