Skip to content

Remote server creates 95GB log after file watcher loses sync on broad workspace #57042

@Clouder0

Description

@Clouder0

Reproduction steps

I do not have a minimal clean-install reproduction yet, but this happened on a Linux remote server when Zed opened a very broad workspace rooted around my home directory.

Observed setup:

  1. Use Zed remote development on Linux.
  2. Open a broad workspace, likely /home/<user> or a similarly high-level directory.
  3. The workspace contains many repositories, package caches, container storage, language caches, build outputs, browser profiles, and symlinks.
  4. Zed remote server recursively scans/watches the tree.
  5. The remote server reaches the OS file watch limit.
  6. After that, the log repeatedly emits filesystem watcher lost sync ... scheduling rescan.

This produced two very large remote-server logs:

  • server-setup-5.log: 101,680,369,572 bytes, 390,397,662 lines
  • server-workspace-12.log: 3,612,958,235 bytes, 14,257,405 lines

Counts from the affected logs:

  • server-setup-5.log: 390,365,712 occurrences of filesystem watcher lost sync
  • server-workspace-12.log: 14,253,969 occurrences of filesystem watcher lost sync

Current vs. Expected behavior

Current behavior:

The remote server generated about 99 GiB of logs under ~/.local/share/zed/logs.

The dominant repeated message is:

{"level":2,"module_path":"fs::fs_watcher","file":"crates/fs/src/fs_watcher.rs","line":117,"message":"(remote server) filesystem watcher lost sync for \"...\"; scheduling rescan"}

Before the log storm, the same log reports the OS watcher limit being reached:

{"level":1,"module_path":"crates/worktree/src/worktree.rs","file":"crates/worktree/src/worktree.rs","line":4936,"message":"(remote server) OS file watch limit reached. about [\"/home/clouder/micromamba/pkgs/pyarrow-core-19.0.1-py310hac404ae_0_cpu/lib/python3.10/site-packages/ pyarrow/include/arrow/vendored/double-conversion\"]"}

The lost-sync messages then repeat hundreds of millions of times, including paths under package caches, container overlays, Go module cache, micromamba packages, browser profiles, build directories, and many repositories.

Expected behavior:

Zed should not allow remote-server logs to grow without bound. After watch-limit exhaustion or watcher resync failure, it should throttle repeated watcher messages, avoid repeated rescan storms, and/or rotate/cap remote-server logs.

Zed version and system specs

Zed remote server binary currently installed:

zed-remote-server-stable-1.2.6+stable.280.20b7f31e7dbe8233a198728ccf3c8aa1180c13e4

System:
OS: Arch Linux
Kernel: Linux 7.0.5-arch1-1 x86_64
Memory: 45 GiB
Architecture: x86_64

Attach Zed log file

Relevant redacted log excerpts
  {"level":1,"module_path":"crates/remote_server/src/server.rs","file":"crates/remote_server/src/server.rs","line":1144,"message":"(remote server) data did not match any variant of untagged enum FormatterList at line 52 column 1"}
  {"level":1,"module_path":"crates/worktree/src/worktree.rs","file":"crates/worktree/src/worktree.rs","line":4936,"message":"(remote server) OS file watch limit reached. about [\"/home/clouder/micromamba/pkgs/pyarrow-core-19.0.1-py310hac404ae_0_cpu/lib/python3.10/site-packages/pyarrow/include/arrow/vendored/double-conversion\"]"}
  {"level":2,"module_path":"fs::fs_watcher","file":"crates/fs/src/fs_watcher.rs","line":117,"message":"(remote server) filesystem watcher lost sync for \"/home/clouder/.bun/install/cache/hono@4.11.0@@@1/dist/types/helper/html\"; scheduling rescan"}
  {"level":2,"module_path":"fs::fs_watcher","file":"crates/fs/src/fs_watcher.rs","line":117,"message":"(remote server) filesystem watcher lost sync for \"/home/clouder/.local/share/containers/storage/overlay/l/XHNSIVK4U5ALBYB6YBJLURXDNZ/app/node_modules/@testing-library/ react\"; scheduling rescan"}
  {"level":2,"module_path":"fs::fs_watcher","file":"crates/fs/src/fs_watcher.rs","line":117,"message":"(remote server) filesystem watcher lost sync for \"/home/clouder/go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v1.38.0\"; scheduling rescan"}

Full affected log sizes:

95G server-setup-5.log
3.4G server-workspace-12.log

The full files are too large to attach directly.

Relevant Zed settings

settings.json

API keys and unrelated private values redacted.

{
  "telemetry": {
    "diagnostics": false,
    "metrics": false
  },
  "features": {
    "edit_prediction_provider": "zed"
  },
  "languages": {
    "Python": {
      "format_on_save": "on",
      "formatter": [
        {
          "code_actions": {
            "source.fixAll.ruff": true,
            "source.organizeImports": true
          }
        },
        {
          "language_server": { "name": "ruff" }
        }
      ]
    }
  },
  "lsp": {
    "wakatime": {
      "initialization_options": {
        "api-key": "<redacted>"
      }
    },
    "clangd": {
      "binary": {
        "arguments": [
          "--background-index",
          "--compile-commands-dir=build"
        ]
      }
    },
    "pyright": {
      "settings": {
        "python.analysis": {
          "typeCheckingMode": "basic",
          "diagnosticMode": "workspace"
        }
      }
    }
  }
}

Note: the log also reports data did not match any variant of untagged enum FormatterList at line 52 column 1, likely due to my Python formatter configuration. That is a separate configuration issue, but it may have contributed extra noise.

Relevant Keymap

None

(for AI issues) Model provider details

Not an AI issue.

If you are using WSL on Windows, what flavor of Linux are you using?

Arch Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:performanceFeedback for performance issues, speed, memory usage, etcfrequency:uncommonBugs that happen for a small subset of users, special configurations, rare circumstances, etcpriority:P2Average run-of-the-mill bugsstate:needs reproNeeds reproduction steps and/or someone to reproduce

    Type

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions