Skip to content

fix: prevent PTY file descriptor leak in ShellExecutionService#20916

Closed
stevenelliottjr wants to merge 1 commit intogoogle-gemini:mainfrom
stevenelliottjr:fix/pty-fd-leak-15945
Closed

fix: prevent PTY file descriptor leak in ShellExecutionService#20916
stevenelliottjr wants to merge 1 commit intogoogle-gemini:mainfrom
stevenelliottjr:fix/pty-fd-leak-15945

Conversation

@stevenelliottjr
Copy link
Copy Markdown

Summary

Fixes #15945

PTY master file descriptors were not being properly closed after process exit or manual kill, leading to system-wide PTY exhaustion on long-running sessions (macOS kern.tty.ptmx_max = 511).

Root causes identified and fixed:

  1. destroy() never called on PTY process — The onExit handler deleted the map entry and attempted optional-chained destroy?.(), but this was fragile and insufficient. Replaced with a dedicated destroyPtyProcess() helper that safely calls destroy() with fallback to kill().

  2. headlessTerminal.dispose() never called — The @xterm/headless Terminal with up to 300,000-line scrollback buffer was never disposed, leaking significant memory per execution. Added dispose() call in finalize() after output extraction.

  3. kill() method only deleted map entry — When a user manually kills a process, the PTY's master FD was never closed. Now uses cleanupPtyEntry() to properly destroy the PTY and dispose the terminal.

  4. Error path leaked FDs — If ptyInfo.module.spawn() succeeded but a later step (e.g. Terminal construction) threw, the spawned PTY was unreachable. Hoisted ptyProcess declaration outside the try block and added cleanup in the catch block.

  5. Duplicate activePtys.delete() call — Removed the redundant first delete in onExit (the second one in finalize() is the correct location, after output is extracted).

Changes:

  • Add destroyPtyProcess() — safely destroys a PTY to release its master FD
  • Add cleanupPtyEntry() — destroys PTY + disposes terminal + removes map entry
  • Destroy PTY immediately in onExit before waiting for output processing
  • Dispose headless terminal in finalize() after extracting output
  • Clean up PTY in catch block for spawn-succeeded-but-init-threw path
  • Use cleanupPtyEntry() in kill() instead of bare map delete

Test plan

  • All 55 existing tests pass (no regressions)
  • New test: kill() calls destroy() on PTY process and clears activePtys
  • New test: normal exit cleans up all resources (PTY destroyed, all maps cleared)
  • Manual verification: run 500+ commands in a session and verify /dev/ptmx FD count stays stable
  • Verify lsof -c node | grep ptmx | wc -l doesn't grow unbounded

…e-gemini#15945)

PTY master file descriptors were not being properly closed after process
exit or manual kill, leading to system-wide PTY exhaustion on long-running
sessions (macOS kern.tty.ptmx_max = 511).

Changes:
- Add destroyPtyProcess() helper that safely calls destroy() with fallback
  to kill() for closing PTY master FDs
- Add cleanupPtyEntry() helper that destroys PTY, disposes the headless
  terminal scrollback buffers, and removes the activePtys map entry
- Call destroyPtyProcess() immediately in onExit handler to release FDs
  before waiting for output processing
- Dispose headless terminal in finalize() after extracting output, freeing
  up to 300k-line scrollback buffer memory
- Clean up PTY in catch block when spawn succeeds but later initialization
  (e.g. Terminal construction) throws
- Use cleanupPtyEntry() in kill() to properly destroy PTY instead of only
  deleting the map entry
- Hoist ptyProcess declaration outside try block so catch block can access
  it for cleanup
- Remove duplicate activePtys.delete() call in onExit handler

Tests:
- Add test verifying kill() calls destroy() on PTY process
- Add test verifying all resources are cleaned up after normal exit
@stevenelliottjr stevenelliottjr requested a review from a team as a code owner March 3, 2026 02:46
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request delivers crucial fixes for resource leaks within the shell execution service, specifically targeting PTY (Pseudo-Terminal) processes and their associated headless terminals. By introducing dedicated cleanup functions and refining the lifecycle management of PTYs, it prevents the accumulation of open file descriptors and memory, thereby resolving issues like system-wide PTY exhaustion and memory leaks in long-running sessions.

Highlights

  • PTY File Descriptor Leak Prevention: Introduced destroyPtyProcess to safely close PTY master file descriptors, addressing system-wide PTY exhaustion.
  • Comprehensive Resource Cleanup: Implemented cleanupPtyEntry to centralize and ensure proper destruction of PTY processes and disposal of headless terminals.
  • Memory Leak Fix for Headless Terminals: Ensured headless terminals are explicitly disposed in finalize() after output extraction to prevent significant memory leaks.
  • Error Path Leak Mitigation: Addressed PTY file descriptor leaks that occurred when PTY spawn succeeded but subsequent initialization steps failed.
  • Refined PTY Lifecycle Management: Corrected PTY cleanup logic in onExit and kill methods, removing redundant deletions and ensuring timely resource release.
Changelog
  • packages/core/src/services/shellExecutionService.test.ts
    • Added a new test to verify that the kill() method correctly calls destroy() on the PTY process and clears activePtys.
    • Added a new test to confirm that all associated resources are properly cleaned up after a normal PTY exit.
  • packages/core/src/services/shellExecutionService.ts
    • Introduced a new private static method destroyPtyProcess to safely destroy a PTY process and release its file descriptors, with a fallback to kill() if destroy() is unavailable.
    • Implemented a new private static method cleanupPtyEntry to encapsulate the complete cleanup process for a PTY entry, including destroying the PTY, disposing the headless terminal, and removing it from activePtys.
    • Modified the executeWithPty method to declare ptyProcess outside the try block, allowing for its cleanup in the catch block if an error occurs after spawning but before full initialization.
    • Updated the onExit handler to immediately call destroyPtyProcess to release the PTY's master file descriptor as soon as the process exits.
    • Ensured that headlessTerminal.dispose() is called within the finalize() function after extracting the output, preventing memory leaks from the scrollback buffer.
    • Added a call to destroyPtyProcess in the catch block of executeWithPty to clean up PTY file descriptors if an error occurs during the setup phase after a successful spawn.
    • Replaced the direct this.activePtys.delete(pid) call in the kill method with this.cleanupPtyEntry(pid) to ensure comprehensive resource cleanup.
    • Removed a redundant this.activePtys.delete(ptyProcess.pid) call from the onExit handler, as the cleanup is now handled in finalize() or by cleanupPtyEntry().
Activity
  • No human activity has been recorded on this pull request yet.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively addresses several file descriptor and memory leaks in the ShellExecutionService by introducing robust cleanup mechanisms for PTY processes. The changes are well-structured, with new helper functions for destroying PTYs and cleaning up associated resources. The fixes cover normal exit, manual kill, and error paths during initialization. The added tests correctly verify the new cleanup logic. I've found one critical issue where a resource leak could still occur, and I've provided a suggestion to fix it.

@gemini-cli gemini-cli Bot added priority/p1 Important and should be addressed in the near term. area/core Issues related to User Interface, OS Support, Core Functionality help wanted We will accept PRs from all issues marked as "help wanted". Thanks for your support! labels Mar 3, 2026
@cocosheng-g
Copy link
Copy Markdown
Contributor

Hi @stevenelliottjr, this PR is being closed due to merge conflicts and 29 days of inactivity. To keep the review queue manageable, we are closing stale blocked PRs. Please feel free to re-open this PR or open a new one once the issues are resolved! Thank you for your contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/core Issues related to User Interface, OS Support, Core Functionality help wanted We will accept PRs from all issues marked as "help wanted". Thanks for your support! priority/p1 Important and should be addressed in the near term.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

System-wide PTY exhaustion due to file descriptor leak

2 participants