Skip to content

Windows support fixes (psmux-compatible)#207

Draft
marcfargas wants to merge 6 commits into
awslabs:mainfrom
marcfargas:psmux-support
Draft

Windows support fixes (psmux-compatible)#207
marcfargas wants to merge 6 commits into
awslabs:mainfrom
marcfargas:psmux-support

Conversation

@marcfargas

@marcfargas marcfargas commented Apr 25, 2026

Copy link
Copy Markdown

Summary

Independent set of Windows-portability fixes that lets cao-server run on Windows
when psmux (a tmux-compatible Rust multiplexer,
https://github.com/psmux/psmux) is installed as tmux on PATH. No
architectural changes — vanilla CAO works on Windows with a drop-in tmux
replacement.

Thanks to @psmux for the very responsive turnaround on 3.3.4, which retired
five of the original CAO-side workarounds.

Why drop libtmux for session/window operations

clients/tmux.py previously used libtmux for Server.new_session(),
Session.from_session_id(), and friends. libtmux's session lookups depend on
-F formatting and =NAME exact-match prefix being honoured by the
underlying binary, which earlier psmux releases didn't do consistently —
returning empty session lists and throwing TmuxObjectDoesNotExist
(manifesting as the misleading zip() argument 2 is shorter).

The fix replaces the libtmux session-management layer with direct
subprocess.run(["tmux", ...]) calls using argument forms that both tmux and
psmux honour. libtmux is still imported for back-compat with mocks in
existing tests; only the lookup paths are bypassed. Linux/macOS behaviour
with real tmux is bit-for-bit unchanged.

What's in here (4 commits)

  1. fix: Windows portability across api/utils/providers/tests — defer
    Unix-only imports in api/main.py (fcntl/pty/termios); os.path.isabs
    for Windows absolute paths; Path.home() RuntimeError guard;
    FileHandler(delay=True) to avoid Windows file-locks during cleanup; URI
    handling and skill-injection path fixes; POSIX-permission and symlink tests
    skipped on Windows.
  2. feat(clients/tmux): replace libtmux with subprocess client for psmux 3.3.4
    the libtmux→subprocess refactor, env propagation via -e KEY=VAL, argv-form
    command building in providers, deduplicated pwsh_join.
  3. fix(clients/tmux,providers): Windows-specific pwsh and psmux quoting fixes
    POSIX-name env filter, send-keys -l fallback for the broken paste-buffer,
    UTF-8 subprocess decoding, pwsh_join &-prefix and trailing-\ strip,
    forced window rename, PSReadLine history suppression so CAO-injected commands
    don't pollute the user's shell history.
  4. docs(readme): require psmux >= 3.3.4 on Windows.

psmux compatibility workarounds (CAO-side, pending upstream fixes)

Every site is grep-able as # psmux-workaround: ... in clients/tmux.py.
Three remain after psmux 3.3.4; each is linked to an open upstream issue:

Site psmux behaviour CAO workaround Upstream
paste-buffer (send_keys, send_keys_via_paste) Buffer stored but never delivered to pane Use send-keys -l (literal) on Windows psmux/psmux#264
new-session -e KEY=VAL with trailing-\ value Argv parser drops subsequent -e flags after a value ending in \ followed by spaces Strip trailing backslashes from env values (lossless on Windows paths) psmux/psmux#265
new-session -n NAME / new-window -n NAME Auto-rename overrides the explicit name Disable automatic-rename and re-assert the name post-creation psmux/psmux#266

Each comment in the source links the issue so the workaround is removable
once the fix lands and a binary is published.

Test results

  • Linux (real tmux): unchanged from origin/main baseline.
  • Windows (psmux 3.3.4 on PATH as tmux): 1524 passed / 0 failed / 30
    skipped / 78 platform-deselected.

Scope

Intentionally narrow. No multiplexer abstraction, no new modules, no new
dependencies. (The broader abstraction-layer approach in #206 was closed in
favour of this PR.) Each commit is self-contained and reverts cleanly. The
smaller portability fixes (os.path.isabs, Path.home() guard,
FileHandler(delay=True)) apply regardless of which multiplexer ships with
CAO.

Test plan

  • CI passes on Linux (no behaviour change expected)
  • Reviewer with psmux installed on Windows: cao launch --provider claude_code --agent-profile <profile> succeeds, agent gets CAO_TERMINAL_ID and any CLAUDE_CODE_USE_* env vars
  • No regression on macOS / Linux tmux flow

🤖 Generated with Claude Code

@marcfargas

Copy link
Copy Markdown
Author

For context: this PR is independent of #206 (the multiplexer-abstraction / WezTerm-backend work).

The two paths are complementary, not competing:

A few of the smaller fixes in here (e.g. the os.path.isabs() working-directory check, the Path.home() RuntimeError guard, FileHandler(delay=True)) are pure Windows-portability fixes that apply regardless of which backend you ship with — they could be cherry-picked independently if useful.

Happy to split / rework as preferred.

@marcfargas

marcfargas commented Apr 25, 2026

Copy link
Copy Markdown
Author

@psmux — flagging for confirmation. Thanks again for offering help in #206.

While porting CAO to run with psmux as a drop-in tmux replacement on Windows, we hit a handful of behavioural deltas vs upstream tmux. We've worked around all of them on the CAO side (every site tagged with `# psmux-workaround: …` in clients/tmux.py for grep-ability), but it would be ideal to land upstream fixes so the workarounds can be removed.

Tested against psmux 3.3 on Windows 11 with libtmux 0.51.0 in the loop.

Findings

  1. `list-sessions` ignores `-F` — the default `NAME: N windows (created DATE) [(attached)]` text is returned regardless of the format string. libtmux requests `-F #{session_id}…` and got the default text → empty parse → `TmuxObjectDoesNotExist`. We worked around by parsing the default output.

  2. `-F` is only honoured when separated from the format token in argv. `-F#{session_id}` (libtmux's default form, concatenated) is ignored; `-F #{session_id}` (space-separated argv entries) works. This caused `new-session -P -F#{session_id}` to echo back `-sNAME` instead of the session id, breaking libtmux's session-creation lookup.

  3. `has-session -t =NAME` exact-prefix not supported — libtmux uses the `=` prefix to disambiguate name vs prefix matches. We call without `=`.

  4. `new-session -e KEY=VAL` records the env on the session but doesn't propagate it into the spawned shell. CAO uses this to inject `CAO_TERMINAL_ID` and provider auth flags (e.g. `CLAUDE_CODE_USE_BEDROCK`); children couldn't see them on Windows. Workaround: stamp `$env:K = 'V'; ...` into the launch command before exec'ing the agent.

  5. Named paste buffers don't exist — `paste_buffers` appears to be a `Vec` capped at 10, with `-b NAME` parsing the name as a `usize` index that silently falls back to slot 0 when parse fails. CAO's UUID-named buffers all collapsed to the same slot under concurrent windows. We worked around by using one fixed buffer name and serialising calls per window. (Pointer based on user investigation: `src/types.rs:392` and `src/server/connection.rs:1050`.)

  6. `paste-buffer -p` ignores `-p` — dispatches `SendText` instead of `SendPaste`, so the bracketed-paste sequences (`\x1b[200~` / `\x1b[201~`) are never emitted. Not currently a blocker for CAO's flow on Windows but worth noting. (Pointer: `src/server/connection.rs:~1058`.)

Ask

Could you confirm any of these are accurate / scoped correctly? Findings #1#4 in particular feel like one-or-two-line server-side fixes that would let libtmux drive psmux directly without our subprocess workaround layer.

@haofeif haofeif added the enhancement New feature or request label Apr 26, 2026
@psmux

psmux commented Apr 26, 2026

Copy link
Copy Markdown

Will look into it and help you get it up and running in a short bit.

Also, you don't need to workaround except for OS specific issues. I will help you undo the workarounds if it's because of Psmux (except specific OS related stuff)

@psmux

psmux commented Apr 26, 2026

Copy link
Copy Markdown

Hey @marcfargas, great work on this PR. I went through every single workaround you documented and tested them all thoroughly. Here is a complete status report with proof.

All 6 Workarounds Tested and Verified

I wrote three separate test suites to verify each finding:

  1. 26/26 PowerShell E2E tests (tests/test_pr207_workaround_elimination.ps1) covering all 6 workarounds plus TCP path verification
  2. 25/25 Python subprocess tests (tests/test_pr207_libtmux.py) replicating the EXACT subprocess patterns from your tmux.py code, including the full CAO send_keys() workflow
  3. 22 Rust unit tests for the named buffer implementation (tests-rs/test_named_buffers.rs)

Full cargo test suite: 5619 tests, 0 failures.

Workaround Status

WA1: list-sessions ignores -F >> FIXED in psmux 3.3.3

list-sessions -F '#{session_name}' returns just the session name, not the default NAME: N windows text. Complex multi field formats like #{session_id} #{session_name} #{session_windows} all work correctly. Your workaround of parsing default output can be removed.

WA2: -F#{fmt} (concatenated) ignored >> FIXED in psmux 3.3.3

Both -F#{session_name} (concatenated) and -F '#{session_name}' (space separated) produce identical output. Tested across list-sessions, list-windows, and new-session -P. Your workaround of forcing space separated argv can be removed.

WA3: has-session -t =NAME not supported >> FIXED in psmux 3.3.3

Exact match semantics work correctly: =NAME matches only the exact session name, not prefixes. =pr207w_exact does NOT match a session named pr207w_exactmatch_full. Your workaround of calling without = can be removed.

WA4: -e KEY=VAL not propagated >> WORKS on psmux 3.3.3

Environment variables set via -e KEY=VAL on new-session are propagated into the child shell. Tested by setting CAO_TEST_VAR=hello_from_cao and reading it with $env:CAO_TEST_VAR inside the session. Your workaround of stamping $env:K = 'V' into the launch command is not needed.

WA5: Named paste buffers >> FIXED in psmux 3.3.3

This was the biggest gap. I implemented a full HashMap<String, String> named buffer system alongside the existing positional Vec<String> stack. UUID style buffer names now work exactly like tmux:

  • set-buffer -b <uuid> "content" stores in the named map
  • show-buffer -b <uuid> retrieves from the named map
  • delete-buffer -b <uuid> removes only that buffer
  • paste-buffer -b <uuid> -t session pastes into the target pane
  • load-buffer -b <uuid> - loads from stdin (your exact CAO pattern)
  • Multiple concurrent UUID named buffers are fully independent (no collision)
  • Numeric -b 0 still indexes the positional stack for backward compat

Your workaround of using a fixed buffer name and serializing per window can be removed.

WA6: paste-buffer -p (bracketed paste) >> Known limitation

As you noted, this is not currently a blocker for CAO. The -p flag does paste content into the pane, but does not wrap with \x1b[200~ / \x1b[201~ sequences. This is on the roadmap.

About libtmux's Server.sessions

You correctly identified that libtmux's session lookup breaks. I dug into the root cause: libtmux 0.55 generates a 126 field format string using U+241E (visible Record Separator symbol) as the delimiter. On Windows, Python's subprocess.Popen defaults to cp1252 encoding for stdout, which cannot represent U+241E. The UTF8 bytes E2 90 9E get decoded as a + z (mojibake), and the parse fails with zip() argument 2 is shorter. This is a libtmux encoding bug on Windows, not a psmux issue. Your approach of using direct subprocess.run() calls is actually the correct solution regardless.

Full CAO Workflow Proof

I replicated the EXACT send_keys() sequence from your tmux.py:

# Step 1: load-buffer from stdin
subprocess.run([psmux, "load-buffer", "-b", cao_uuid, "-"], input=command)
# Step 2: paste-buffer
subprocess.run([psmux, "paste-buffer", "-p", "-b", cao_uuid, "-t", session])
# Step 3: sleep
time.sleep(0.3)
# Step 4: send Enter
subprocess.run([psmux, "send-keys", "-t", session, "Enter"])
# Step 5: delete-buffer (finally)
subprocess.run([psmux, "delete-buffer", "-b", cao_uuid])

This full workflow passes end to end. The command is pasted, executed, and the buffer is cleaned up.

Summary

# Issue Status Workaround needed?
1 list-sessions ignores -F Fixed No
2 -F#{fmt} concatenated ignored Fixed No
3 has-session -t =NAME Fixed No
4 -e KEY=VAL not propagated Works No
5 Named paste buffers Fixed No
6 paste-buffer -p bracketed Known limitation Not a blocker

Workarounds 1 through 5 are proven unnecessary in psmux 3.3.3. I am happy to help you strip them out if you want, or you can grep for # psmux-workaround: and remove them yourselves. Let me know what works best.

All test files are in the psmux repo under tests/ if you want to verify independently.

@psmux

psmux commented Apr 26, 2026

Copy link
Copy Markdown

Update: All 6 Workarounds Eliminated + libtmux Native API Works

Commit: 10ebb81 (tag v3.3.0.4)

Install

cargo install --git https://github.com/psmux/psmux --tag v3.3.0.4

What's Fixed

Every workaround in PR #207 is now provably unnecessary with psmux:

# Workaround psmux Fix
WA1 list-sessions -F ignored → parse text output -F format flag works natively
WA2 -Fformat concatenated syntax broken Both -F format and -Fformat work
WA3 has-session prefix-matches → false positives = prefix exact-match supported
WA4 set-environment not propagated Environment variables propagated to panes
WA5 Single paste buffer → race conditions Named buffers (-b name) for concurrent ops
WA6 No bracketed paste → special chars garbled Bracketed paste mode supported

libtmux Native API Compatibility

libtmux's Server.sessions, .windows, .panes, new_window(), send_keys(), kill() all work natively with psmux — no workarounds needed:

  • $N session ID targets (e.g. -t $0) now resolve correctly
  • @N window ID targets (e.g. list-panes -t @2) now target the correct window
  • list-panes -t @N returns panes of the specified window (not just the active window)

Note: libtmux 0.55.1 on Windows requires a one-line patch to common.py (add encoding="utf-8" to subprocess.Popen) due to cp1252 locale on Windows. This is a libtmux upstream issue, not a psmux issue — tmux on Linux works because locale.getpreferredencoding() returns UTF-8 there.

Test Evidence

33 Python tests (all pass):

  • 25 subprocess-based tests covering all 6 workarounds + CAO workflow + concurrency
  • 8 libtmux native API tests: Server.sessions, .windows, .panes, new_window(), send_keys(), kill()

26 PowerShell E2E workaround elimination tests (all pass)

22 PowerShell E2E PR207 compat tests (all pass)

33 Rust unit tests (22 named buffers + 11 PR207 compat) (all pass)

1861 cargo tests (all pass)

@psmux

psmux commented Apr 26, 2026

Copy link
Copy Markdown

@marcfargas @haofeif Please try it and let me know how it goes

@marcfargas

Copy link
Copy Markdown
Author

Thanks @psmux — that's a comprehensive set of fixes. One snag on our end: we don't have a Rust toolchain on the test box, so cargo install --git ... --tag v3.3.0.4 isn't available. The Docker route also fails (Windows containers don't run on Windows ARM here).

The v3.3.0.4 tag is published but no GitHub release was cut for it, so there's no binary to download. The CI workflow doesn't upload artifacts either, and your release workflow is workflow_dispatch-only since 7e721a3.

Could you dispatch the Release workflow to publish a binary? Two options that would work for us:

  1. Tag v3.3.4 off current master (42ab3899) — gets the WA fixes plus the 3 doc commits since 10ebb81, and scoop will auto-update users from 3.3.33.3.4.
  2. Dispatch against the existing v3.3.0.4 tag — faster, but the version string won't trigger a scoop upgrade from 3.3.3.

Once a binary is out, I'll upgrade locally, strip workarounds 1–5 from clients/tmux.py, re-run the Windows test suite, and push. Will document psmux ≥ <version> as a hard requirement in the PR.

@psmux

psmux commented Apr 27, 2026

Copy link
Copy Markdown

Okay I need to run comprehensive testing overall which will go for several hours. Please give me sometime for the release

@codecov-commenter

codecov-commenter commented Apr 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 6.21469% with 166 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@3df3fa0). Learn more about missing BASE report.

Files with missing lines Patch % Lines
src/cli_agent_orchestrator/clients/tmux.py 5.34% 124 Missing ⚠️
...c/cli_agent_orchestrator/providers/opencode_cli.py 0.00% 8 Missing ⚠️
src/cli_agent_orchestrator/api/main.py 0.00% 7 Missing ⚠️
...rc/cli_agent_orchestrator/utils/skill_injection.py 0.00% 6 Missing ⚠️
src/cli_agent_orchestrator/providers/codex.py 0.00% 5 Missing ⚠️
...rc/cli_agent_orchestrator/providers/copilot_cli.py 0.00% 4 Missing ⚠️
src/cli_agent_orchestrator/providers/gemini_cli.py 0.00% 4 Missing ⚠️
src/cli_agent_orchestrator/providers/kimi_cli.py 0.00% 4 Missing ⚠️
src/cli_agent_orchestrator/utils/agent_profiles.py 33.33% 2 Missing ⚠️
...rc/cli_agent_orchestrator/providers/claude_code.py 75.00% 1 Missing ⚠️
... and 1 more
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #207   +/-   ##
=======================================
  Coverage        ?   10.81%           
=======================================
  Files           ?       52           
  Lines           ?     4326           
  Branches        ?        0           
=======================================
  Hits            ?      468           
  Misses          ?     3858           
  Partials        ?        0           
Flag Coverage Δ
unittests 10.81% <6.21%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

psmux added a commit to psmux/psmux that referenced this pull request Apr 30, 2026
The custom CreateProcessW command-line builder wrapped each value in
"..." and only replaced embedded quotes. For a value with spaces ending
in a backslash (e.g. C:\Program Files\Foo Bar\plugins\), it produced
"...plugins\" — the trailing \" is read by CommandLineToArgvW in the
spawned server as an *escaped* quote, swallowing every subsequent
argv entry into the value.

Replace the naive quoter with escape_arg_msvcrt, which doubles
backslash runs that immediately precede a " (literal or closing
quote), per Microsoft's documented parsing rules.

Tests: 10 new Rust units (incl. round-trip through real
CommandLineToArgvW) and 17 E2E checks (Python subprocess, cmd /c,
PowerShell native, control case, TUI verification).
1895 existing Rust tests pass.

Reported by @marcfargas (downstream of awslabs/cli-agent-orchestrator#207)
@marcfargas

Copy link
Copy Markdown
Author

Quick upstream status update on the remaining psmux workarounds:

  • psmux/psmux#264 (paste-buffer): maintainer could not reproduce on current master and added a proof harness in 10d8da6; this looks likely fixed post-3.3.4, but not yet in a released build we've verified against, so CAO keeps the send-keys -l workaround for now.
  • psmux/psmux#265 (trailing \\ in env values): no change; CAO still strips trailing backslashes as a workaround.
  • psmux/psmux#266 (automatic-rename overriding explicit -n NAME): maintainer reproduced it, then fixed it upstream in 4f31cde / tag v3.3.4-fix-266; CAO keeps the workaround until a released psmux build includes that fix.

I also updated the PR body to reflect this, but adding this comment so subscribers get notified.

marcfargas and others added 4 commits May 10, 2026 22:17
psmux runs on Windows under PowerShell; several modules assumed a
POSIX environment and failed silently or crashed on import.

- api/server.py: defer Unix-only signal/resource imports behind a
  platform check so cao-server can start on Windows at all.
- clients/tmux: replace startswith("/") path test with os.path.isabs()
  so Windows absolute paths (e.g. C:\...) are recognised correctly.
- providers/copilot_cli: guard Path.home() against RuntimeError when
  USERPROFILE/HOME is cleared by the test harness.
- utils/logging: open FileHandler with delay=True to avoid Windows
  file-lock errors during process cleanup.
- utils (skill_injection, agent_profiles, skills): use os.path.join
  and pathlib throughout instead of forward-slash string concatenation.
- tests: skip POSIX-permission (chmod 000) and symlink tests on Windows
  where those semantics are not available.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…3.3.4

libtmux used the Python tmux bindings which assumed a native tmux
socket and POSIX session management.  psmux (the Windows tmux shim)
diverges in ways that libtmux cannot paper over: session creation,
env propagation, and buffer commands all behave differently.

Replace the entire libtmux session layer in clients/tmux.py with
direct subprocess calls to the tmux/psmux binary.  This gives full
control over the exact argv passed at each step and lets us handle
platform differences explicitly.  Session env is injected via repeated
-e KEY=VAL flags; window names are forced with -n; paste-buffer IDs are
UUID-named to avoid collisions.

Providers are updated in parallel: commands are built as argv lists on
Windows (instead of a shell string) so quoting is handled by the OS
argument-passing layer rather than a POSIX shell that psmux doesn't
provide.  _pwsh_join is deduplicated into a single shared helper.

The result: psmux 3.3.4 on Windows is fully supported with no
platform-specific workarounds needed in the session layer.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…fixes

Several edge cases discovered during real-world psmux testing on Windows
after the subprocess client landed:

- Env var filter: tmux and psmux reject env-key names that contain
  characters outside [A-Za-z0-9_]; filter them out before building the
  -e KEY=VAL flags to avoid a silent launch failure.
- send-keys -l: psmux's paste-buffer implementation is broken on
  Windows; switch to send-keys -l (literal) which works correctly.
- UTF-8 decoding: force encoding='utf-8' on all subprocess stdout/stderr
  reads so non-ASCII output doesn't crash on Windows code pages.
- pwsh_join for claude_code provider: the provider was still building a
  bash-style shell string on Windows; route it through pwsh_join so
  quoting is correct for PowerShell.
- Trailing backslash in env values: Windows paths often end with \;
  psmux treats a trailing backslash as a line-continuation and drops
  the value.  Strip it before passing via -e.
- pwsh_join & prefix: the joined command string must be prefixed with &
  so PowerShell treats it as an invocation expression, not a string.
- Window rename: psmux ignores the -n flag on new-window when
  automatic-rename is on; explicitly call rename-window after creation.
- PSReadLine history: CAO-injected pwsh commands were being saved to
  the user's command history; suppress with Set-PSReadLineOption before
  each injected command.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a callout under the tmux install step noting that Windows users
should use psmux >= 3.3.4, as earlier versions lack the compatibility
fixes this client depends on.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
marcfargas and others added 2 commits May 10, 2026 22:59
Record the agreed approach for verifying this CAO fork through a local uv tool install before deeper runtime checks.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants