What

Makes tags a coherent cross-store link that an AI agent can use end to end. Two pieces:

1. A tags filter on /search (and the MCP search-content tool): pull every item carrying given tags, e.g. ?tags=person:ada,project:atlas.
2. The same filter now also covers memories (content_type=memory), so the AI can retrieve memories by tag. Before this, the AI could add tags to a memory (update-memory) but had no way to get them back by tag.

The model (how the AI links a person to memories / a timeframe)

One string namespace spans three stores. The same tag on different items connects them.

flowchart LR
  subgraph "one tag namespace"
    direction LR
    T["person:ada"]
  end
  F["screen frame<br/>vision_tags"] --- T
  A["audio chunk<br/>audio_tags"] --- T
  M["memory<br/>memories.tags"] --- T
  T --> Q1["GET /search?tags=person:ada<br/>(screen + audio)"]
  T --> Q2["GET /search?content_type=memory&tags=person:ada<br/>(facts)"]

• Add a tag to a frame/audio: POST /tags/vision/{id} or /tags/audio/{id}. To a memory: tags field on POST /memories / PUT /memories/{id}.
• Link a person to a memory: tag the memory person:ada.
• Link a person to a timeframe: tag captures person:ada and query with start_time/end_time, or rely on the memory's created_at + frame_id provenance.
• Everything about a person: one call for captures (content_type=all&tags=person:ada) + one for facts (content_type=memory&tags=person:ada).
• Frames are pruned by retention, so durable links belong on memories; tag frames/audio for short-term recall. The MCP tool descriptions now teach exactly this.

Behavior

• tags is comma-separated. Multiple tags AND together (intersection). Matching is exact membership, not substring (person:ada does not match person:adam).
• Applies to screen (OCR), audio, and memory. input and accessibility have no tag table and return nothing when a tag filter is set.
• content_type=all unions tagged screen + audio (it never includes memory, tagged or not, same as today).
• count_search_results agrees with the filtered set, so pagination total stays correct.
• Empty / absent tags behaves exactly as before. Strictly opt-in.

Implementation

• Public search() / count_search_results() signatures kept stable (~60 callers untouched) via search_with_tags() / count_search_results_with_tags() siblings; the route handler calls those.
• Screen/audio matching: json_each + HAVING COUNT(DISTINCT) = json_array_length over the junction tables. Memory matching: the same exact-AND shape over the memories.tags JSON array, threaded through new list_memories / count_memories tags_all params (the public GET /memories?tags= keeps its existing single-substring filter).
• MCP add-tags, update-memory, search-content descriptions updated to teach namespaced tags and retrieval. screenpipe-api SKILL.md (shipped + repo copies) gains a concise Tags section.

Tests

cargo test -p screenpipe-db --test db — 37 pass, including:

• test_search_filter_by_tags — single / shared / AND / unknown / no-filter / count.
• test_tag_filter_audio_and_cross_modal — audio filter, content_type=all cross-modal union, exact-not-substring, input gated, count.
• test_memory_filter_by_tags — memory exact AND, no-substring, FTS compose, all-excludes-memory, count.

cargo clippy clean (no new warnings).

Performance

Benchmarked on the search hot path with tests/tag_filter_bench.rs (ignored; 200k frames / 200k vision_tags / 60k audio / 50k memories, in-memory). Adversarial case: tags are rare and on the oldest rows, fighting ORDER BY timestamp DESC LIMIT.

EXPLAIN QUERY PLAN confirms screen/audio filtering drives off the tag indexes (tags.name UNIQUE + idx_vision_tags_tag_id) and PK-looks-up frames, so it never scans all frames:

The tag filter is faster than unfiltered search because it narrows to the tagged set instead of scanning everything. The one linear path is memory: memories.tags is unindexed JSON, so the filter is a full scan + correlated json_each (~0.3 us/row, ~16 ms @ 50k → ~160 ms @ 500k). Fine at realistic memory counts; if memories ever reach millions, add a memory_tags junction table mirroring vision_tags. No-tag queries are unaffected (the json_array_length(?)=0 OR ... guard short-circuits).

Not in this PR

A graph-traversal endpoint (co-occurring tags for a given tag). Note /connections is already taken by integrations, so it would need a name like /graph or /related. Natural follow-up.