Skip to content

fix(media): retain inbound media with recursive cleanup TTL#38292

Merged
vincentkoc merged 7 commits intomainfrom
vincentkoc-code/media-retention-sweep
Mar 7, 2026
Merged

fix(media): retain inbound media with recursive cleanup TTL#38292
vincentkoc merged 7 commits intomainfrom
vincentkoc-code/media-retention-sweep

Conversation

@vincentkoc
Copy link
Copy Markdown
Member

Summary

  • add media.ttlHours as the persisted inbound media retention knob (default 24 hours)
  • recurse media TTL cleanup through nested directories like remote cache trees
  • run startup and hourly gateway-owned media cleanup so buffer-backed inbound writes are collected too

Testing

  • pnpm vitest run src/media/store.test.ts src/gateway/server-maintenance.test.ts
  • pnpm check (fails in pre-existing extensions/feishu/src/media.ts with unrelated timeout type errors)

Fixes #33078
Related #24519
Related #29211

@aisle-research-bot
Copy link
Copy Markdown

aisle-research-bot Bot commented Mar 6, 2026

🔒 Aisle Security Analysis

We found 1 potential security issue(s) in this PR:

# Severity Title
1 🟡 Medium Potential DoS from unbounded recursive media cleanup on gateway maintenance timer

1. 🟡 Potential DoS from unbounded recursive media cleanup on gateway maintenance timer

Property Value
Severity Medium
CWE CWE-400
Location src/gateway/server-maintenance.ts:139-160

Description

A new hourly gateway maintenance interval runs a recursive filesystem cleanup over the entire media directory tree:

  • startGatewayMaintenanceTimers() calls cleanOldMedia() with { recursive: true, pruneEmptyDirs: true }.
  • cleanOldMedia() performs readdir/lstat over every entry and descends into subdirectories.
  • Inbound media files can be created via normal remote interactions (attachments downloaded and stored under ~/.openclaw/media/...), so an attacker who can send many small attachments can inflate the number of files/directories.
  • The cleanup has no cap/time budget/backpressure, so the periodic scan can become very expensive (CPU + disk IO) and degrade gateway responsiveness or cause an availability issue.

Vulnerable code (trigger):

mediaCleanupInFlight = cleanOldMedia(params.mediaCleanupTtlMs, {
  recursive: true,
  pruneEmptyDirs: true,
});
...
const mediaCleanup = setInterval(() => {
  void runMediaCleanup();
}, 60 * 60_000);

Note: media.ttlHours is loaded from local config (operator-controlled), not directly from a remote request, but once enabled, remote users can still amplify the cost by causing many inbound media files to exist within the retention window.

Recommendation

Mitigate worst-case traversal costs and attacker-amplification:

  • Apply quotas to inbound media creation (per remote sender/session/day): max file count, max total bytes, and/or eviction on write.
  • Make cleanup incremental with a time budget / max-entries-per-run (store a cursor or maintain an index), rather than scanning the full tree each time.
  • Consider using a background worker/process (or worker_threads) so heavy IO doesn’t block the gateway event loop.
  • Add jitter to the schedule to avoid synchronized cleanup across multiple instances.

Example (time/entry budget):

async function cleanOldMediaBudgeted(ttlMs: number, budget: { maxEntries: number; maxMs: number }) {
  const start = Date.now();
  let seen = 0;// walk directories but stop when (seen >= maxEntries || Date.now() - start >= maxMs)// persist cursor for next run
}

setInterval(() => void cleanOldMediaBudgeted(ttlMs, { maxEntries: 10_000, maxMs: 2_000 }), 60*60_000);

This prevents a single run from monopolizing IO/CPU when the media tree is large.


Analyzed PR: #38292 at commit d58111f

Last updated on: 2026-03-07T03:39:00Z

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Mar 6, 2026

Greptile Summary

This PR introduces a configurable inbound media retention TTL (media.ttlHours, default 24 h) and upgrades the media cleanup logic to recurse through nested subdirectories (e.g. remote cache trees), addressing the gap where buffer-backed writes via saveMediaBuffer had no automatic cleanup. A startup sweep and an hourly gateway-managed interval are added alongside the existing per-save cleanup.

Key observations:

  • The recursive removeExpiredFilesRecursively function correctly uses fs.lstat to avoid following symlinks when traversing subdirectories, which is an improvement over the previous fs.stat-based approach.
  • saveMediaSource (line 275 of store.ts) still calls cleanOldMedia() with no arguments, meaning it uses the hardcoded 2-minute DEFAULT_TTL_MS, not media.ttlHours. Users who configure a longer retention window may be surprised to see files written via that code-path evicted much sooner during active use.
  • After recursive deletion, empty directory trees (e.g. remote-cache/session-abc/images/) are never pruned. For long-running servers with many sessions, empty directories will accumulate indefinitely in the media tree.
  • The schema, types, labels, and help-text are all updated consistently.
  • New tests cover both the recursive store cleanup and the maintenance timer wiring.

Confidence Score: 4/5

  • Safe to merge; changes are well-tested and the core logic is correct, with two minor style concerns that don't affect correctness.
  • The implementation is logically sound: the recursive cleanup correctly handles symlinks with lstat, the hourly interval follows established patterns, and the new tests provide adequate coverage. The two deductions are: (1) saveMediaSource still uses the 2-minute default TTL rather than the user-configured ttlHours, creating a subtle inconsistency with user expectations; and (2) empty directories from cleaned-up session trees are never removed, which could accumulate over time. Neither is a runtime error, but both are worth addressing before the feature is widely relied upon.
  • src/media/store.ts — the saveMediaSource inline cleanOldMedia() call and the lack of empty-directory pruning after recursive deletion.

Comments Outside Diff (1)

  1. src/media/store.ts, line 275 (link)

    saveMediaSource cleanup ignores user-configured TTL

    saveMediaSource still calls cleanOldMedia() with no arguments, which uses DEFAULT_TTL_MS (2 minutes). Since media.ttlHours is now a user-visible retention knob, callers setting a longer retention window (e.g. ttlHours: 48) may be surprised that files written via saveMediaSource are still evicted after 2 minutes during active-save activity, while files written via saveMediaBuffer are only collected by the hourly gateway cleanup.

    If the 2-minute sweep here is intentional as a "pre-save space reclamation" rather than a retention guarantee, a short comment clarifying the distinction would help avoid future confusion.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: src/media/store.ts
    Line: 275
    
    Comment:
    **`saveMediaSource` cleanup ignores user-configured TTL**
    
    `saveMediaSource` still calls `cleanOldMedia()` with no arguments, which uses `DEFAULT_TTL_MS` (2 minutes). Since `media.ttlHours` is now a user-visible retention knob, callers setting a longer retention window (e.g. `ttlHours: 48`) may be surprised that files written via `saveMediaSource` are still evicted after 2 minutes during active-save activity, while files written via `saveMediaBuffer` are only collected by the hourly gateway cleanup.
    
    If the 2-minute sweep here is intentional as a "pre-save space reclamation" rather than a retention guarantee, a short comment clarifying the distinction would help avoid future confusion.
    
    How can I resolve this? If you propose a fix, please make it concise.

Last reviewed commit: a97cd56

Comment thread src/media/store.ts Outdated
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a97cd56a2a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/media/store.ts Outdated
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8d2a17c8fd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/media/store.ts
@vincentkoc vincentkoc merged commit ba9eaf2 into main Mar 7, 2026
29 of 31 checks passed
@vincentkoc vincentkoc deleted the vincentkoc-code/media-retention-sweep branch March 7, 2026 03:06
joshavant pushed a commit that referenced this pull request Mar 7, 2026
* Config: add media retention TTL setting

* Media: recurse persisted media cleanup

* Gateway: add persisted media cleanup timer

* Media: harden retention cleanup sweep

* Media: make recursive retention cleanup opt-in

* Media: retry writes after empty-dir cleanup race
vincentkoc added a commit to BryanTegomoh/openclaw-upstream that referenced this pull request Mar 8, 2026
…#38292)

* Config: add media retention TTL setting

* Media: recurse persisted media cleanup

* Gateway: add persisted media cleanup timer

* Media: harden retention cleanup sweep

* Media: make recursive retention cleanup opt-in

* Media: retry writes after empty-dir cleanup race
jenawant pushed a commit to jenawant/openclaw that referenced this pull request Mar 10, 2026
…#38292)

* Config: add media retention TTL setting

* Media: recurse persisted media cleanup

* Gateway: add persisted media cleanup timer

* Media: harden retention cleanup sweep

* Media: make recursive retention cleanup opt-in

* Media: retry writes after empty-dir cleanup race
V-Gutierrez pushed a commit to V-Gutierrez/openclaw-vendor that referenced this pull request Mar 17, 2026
…#38292)

* Config: add media retention TTL setting

* Media: recurse persisted media cleanup

* Gateway: add persisted media cleanup timer

* Media: harden retention cleanup sweep

* Media: make recursive retention cleanup opt-in

* Media: retry writes after empty-dir cleanup race
alexey-pelykh pushed a commit to remoteclaw/remoteclaw that referenced this pull request Mar 20, 2026
…#38292)

* Config: add media retention TTL setting

* Media: recurse persisted media cleanup

* Gateway: add persisted media cleanup timer

* Media: harden retention cleanup sweep

* Media: make recursive retention cleanup opt-in

* Media: retry writes after empty-dir cleanup race

(cherry picked from commit ba9eaf2)
alexey-pelykh pushed a commit to remoteclaw/remoteclaw that referenced this pull request Mar 20, 2026
…#38292)

* Config: add media retention TTL setting

* Media: recurse persisted media cleanup

* Gateway: add persisted media cleanup timer

* Media: harden retention cleanup sweep

* Media: make recursive retention cleanup opt-in

* Media: retry writes after empty-dir cleanup race

(cherry picked from commit ba9eaf2)
lovewanwan pushed a commit to lovewanwan/openclaw that referenced this pull request Apr 28, 2026
…#38292)

* Config: add media retention TTL setting

* Media: recurse persisted media cleanup

* Gateway: add persisted media cleanup timer

* Media: harden retention cleanup sweep

* Media: make recursive retention cleanup opt-in

* Media: retry writes after empty-dir cleanup race
ogt-redknie pushed a commit to ogt-redknie/OPENX that referenced this pull request May 2, 2026
…#38292)

* Config: add media retention TTL setting

* Media: recurse persisted media cleanup

* Gateway: add persisted media cleanup timer

* Media: harden retention cleanup sweep

* Media: make recursive retention cleanup opt-in

* Media: retry writes after empty-dir cleanup race
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
…#38292)

* Config: add media retention TTL setting

* Media: recurse persisted media cleanup

* Gateway: add persisted media cleanup timer

* Media: harden retention cleanup sweep

* Media: make recursive retention cleanup opt-in

* Media: retry writes after empty-dir cleanup race
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gateway Gateway runtime maintainer Maintainer-authored PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(media): periodic cleanup for all inbound media files

1 participant