Skip to content

Fix Windows is_terminal() for write-only console handles (fixes #130974)#153219

Open
shri-prakhar wants to merge 1 commit intorust-lang:mainfrom
shri-prakhar:fix-windows-is-terminal-write-only-conout-130974
Open

Fix Windows is_terminal() for write-only console handles (fixes #130974)#153219
shri-prakhar wants to merge 1 commit intorust-lang:mainfrom
shri-prakhar:fix-windows-is-terminal-write-only-conout-130974

Conversation

@shri-prakhar
Copy link
Contributor

@shri-prakhar shri-prakhar commented Feb 28, 2026

On Windows, is_terminal() was incorrect for console handles opened without read access (e.g. write-only CONOUT$). GetConsoleMode needs GENERIC_READ; without it the API fails with ERROR_ACCESS_DENIED instead of reporting a console. The code treated that as “not a terminal,” so write-only console handles were misclassified.

This change fixes that by treating ERROR_ACCESS_DENIED from GetConsoleMode as “might be a console” and then using GetFileType: console buffers are FILE_TYPE_CHAR. The NUL device is also FILE_TYPE_CHAR but GetConsoleMode returns ERROR_INVALID_HANDLE for it, so NUL never hits this branch and is still correctly reported as non-terminal.

Fixes #130974.

Changes

  • library/std/src/sys/io/is_terminal/windows.rs: After GetConsoleMode fails, check for ERROR_ACCESS_DENIED and, in that case, confirm a console with GetFileType(handle) == FILE_TYPE_CHAR. Added a short comment explaining the logic and the NUL exception.
  • library/std/tests/is_terminal.rs: New Windows-only tests for IsTerminal:

Testing

  • python x.py test library/std (including the new is_terminal tests).
  • Manual check on Windows with a write-only CONOUT$ handle to confirm it is now reported as a terminal.

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Feb 28, 2026
@rustbot
Copy link
Collaborator

rustbot commented Feb 28, 2026

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: @ChrisDenton, libs
  • @ChrisDenton, libs expanded to 8 candidates
  • Random selection from Mark-Simulacrum, joboet

@rustbot

This comment has been minimized.

@Mark-Simulacrum
Copy link
Member

r? @ChrisDenton

(Or if you don't have bandwidth we can ping Windows group)

@rustbot
Copy link
Collaborator

rustbot commented Feb 28, 2026

ChrisDenton is currently at their maximum review capacity.
They may take a while to respond.

Copy link
Contributor

@Teapot4195 Teapot4195 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although this code of smells of AI generation, nothing looks obviously incorrect. A lot of the comments are redundant or highly verbose (in view of the assert messages and just test names).

Also, the asserts and the test specific comments should document expected behaviour, panic messages are redundant and can be removed.

View changes since this review

);
}
Err(_) => {
// No console attached (headless CI, service, etc.) — skip.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be blindly eating the case where \\.\CONOUT$ doesn't exist? This could mask away potential regressions if there is no conhost attached during testing.

The test should (ideally) create a conhost if it's not already attached to one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Teapot4195 Fair points all around. The verbose comments were AI-generated; I leaned on it for
documentation while I was working through the Windows console API surface, and I wasn't aware that
went against the review policy. That's on me. I used AI to help scaffold the tests documentation and comments
and that resulted in the overly verbose comments and redundant assert messages.
I should have cleaned the output more carefully before submitting lesson learned.
I'll strip the comments, remove the panic strings, and let the test names and
assertions speak for themselves.

On the \\.\CONOUT$ skip — agreed, silently eating Err(_) could hide
regressions in headless CI. I'll rework it so the test ensures a console
exists (e.g. via AllocConsole if not already attached) rather than bailing.

Will push a cleaned-up revision. Thanks for the thorough review.

@joboet
Copy link
Member

joboet commented Feb 28, 2026

Are consoles really the only character-device handles where GetConsoleMode returns ERROR_ACCESS_DENIED?

@shri-prakhar
Copy link
Contributor Author

shri-prakhar commented Mar 1, 2026

@joboet Based on the Windows docs, yes ERROR_ACCESS_DENIED from
GetConsoleMode is console-specific in practice.

The key distinction is how the kernel dispatches console API calls. GetConsoleMode
operates on console handles specifically (docs: "A handle to the console input buffer
or the console screen buffer. The handle must have the GENERIC_READ access right."
).
Non-console character devices (NUL, COM ports, etc.) aren't recognized as console objects
at all, so GetConsoleMode fails with ERROR_INVALID_HANDLE for those they never
reach the GENERIC_READ access check that produces ERROR_ACCESS_DENIED .

This is consistent with the Console Buffer Security docs, where the "wrong-way verbs"
section states: "Rejected calls will receive an access denied error code" describing
access checks that only apply to handles already identified as console buffers.

I verified this with NUL (\\.\NULGetConsoleMode returns error 6, not 5), and
it matches the behavior widely observed with redirected/piped handles (also error 6).

That said, the docs don't publish a formal error code table for GetConsoleMode, so
this relies on documented architecture + observed behavior rather than an explicit
guarantee. If a secondary confirmation is preferred (e.g. GetConsoleScreenBufferInfo
as a second check), I'm open to that.

@shri-prakhar shri-prakhar force-pushed the fix-windows-is-terminal-write-only-conout-130974 branch from bc69769 to 163c74d Compare March 1, 2026 08:36
@shri-prakhar
Copy link
Contributor Author

Pushed a revision.
Let me know your thoughts on ERROR_ACCESS_DENIED reasoning or if you'd prefer
a secondary confirmation check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

On Windows is_terminal always returns false if the handle is not opened with read access

6 participants