Try mark `no_hash` queries as green after execution #150156

zetanumbers · 2025-12-19T12:56:36Z

Fixes #150018
Fixes #142778
Fixes #141540

~~This PR allows parallel front-end's green-red node coloring processes in the query dep graph to race.~~ Recall that whenever one thread tries to mark query node green and then fails to reconstruct a subquery's key to force it, parent query is tried to be forced or executed too. The subquery could be marked green by the thread during execution the parent query because key is now present. Other thread during this process is able to mark green parent query itself. And so after some time the first thread finishes the execution and finds in its place a green node. This is bad for no_hash queries since those are conservatively presumed to be red but actually queries must be deterministic anyway and so no_hash nodes should be colored green if its deps are green too after its execution finishes. ~~As such we can consider green color to be "stronger" that the red color and thus former would replace the latter.~~

TODO: update rustc-dev-guide query docs and rustc comments on no_hash modifier

~~I have left some refactor-ish changes I did during my dirty debugging. Let me know if you would want those to be redone or removed.~~

Steps to reproduce

Quote from #150018, more details there:

Do a clean build x build library --stage 1 at 52fda2e, then git checkout to ba45f0b and without cleaning try rebuilding it the same way again. And don't forget to set RUSTFLAGS_BOOTSTRAP to -Zthreads=n where n > 1.

rustbot · 2025-12-19T12:56:41Z

r? @petrochenkov

rustbot has assigned @petrochenkov.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

petrochenkov · 2025-12-19T13:06:04Z

cc @Zoxc @cjgillot @nnethercote

zetanumbers · 2025-12-19T13:21:09Z

My first idea was to add a special "locked" color, but I wasn't sure what was even happening in there because coloring races currently are considered to be normal anyway.

matthiaskrgr · 2025-12-19T13:51:13Z

@bors try @rust-timer queue

Allow green-red query coloring race

zetanumbers · 2025-12-19T14:58:58Z

I ran rustc incremental builds as described in reproduction steps section for several hours with -Zthreads, debug=true and -Zincremental_verify_ich and it's looking good.

rust-bors · 2025-12-19T16:07:32Z

☀️ Try build successful (CI)
Build commit: bb5a19e (bb5a19e1ef3ed803b9a2acd98a3b223334df5063, parent: 0f0d850e9146f53b16fb1c6a56ed2f821962ff94)

rust-timer · 2025-12-19T16:48:58Z

Finished benchmarking commit (bb5a19e): comparison URL.

Overall result: ❌ regressions - please read the text below

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	2.3%	[0.3%, 6.6%]	117
Regressions ❌ (secondary)	1.7%	[0.2%, 4.9%]	75
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.3%	[0.3%, 6.6%]	117

Max RSS (memory usage)

Results (primary -3.5%, secondary -1.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-3.5%	[-3.5%, -3.5%]	1
Improvements ✅ (secondary)	-1.5%	[-2.2%, -0.8%]	2
All ❌✅ (primary)	-3.5%	[-3.5%, -3.5%]	1

Cycles

Results (primary 3.0%, secondary 3.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	3.0%	[2.9%, 3.1%]	2
Regressions ❌ (secondary)	3.4%	[2.3%, 4.9%]	3
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.0%	[2.9%, 3.1%]	2

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 479.055s -> 481.828s (0.58%)
Artifact size: 390.50 MiB -> 390.50 MiB (0.00%)

compiler/rustc_query_system/src/dep_graph/graph.rs

compiler/rustc_query_system/src/dep_graph/serialized.rs

compiler/rustc_query_system/src/dep_graph/graph.rs

petrochenkov · 2025-12-19T17:12:12Z

As usual, it would be nice to split the work into a few parts (especially given the perf regressions):

Preparatory changes and refactorings (passing the frame and feed around, the named iterator change, the COMPRESSED_RED_BIT change, and few others)
The assert hardening changes
The substantial changes modifying the logic in try_mark and alloc_and_color_node

But it may be better to wait for a high level design review from Zoxc or cjgillot before doing that.

@rustbot author

rustbot · 2025-12-19T17:12:16Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

cjgillot · 2025-12-25T15:21:11Z

Recall that whenever one thread tries to mark query node green and then fails to reconstruct a subquery's key to force it, parent query is tried to be forced or executed too. The subquery could be marked green by the thread during execution the parent query because key is now present.

Could you elaborate how this can happen? Key reconstructibility is not supposed to change.

I agree with @petrochenkov's suggestion to split pure reactors from the actual change. Parallelism and atomic are hard enough to make review as easy as possible.

zetanumbers · 2026-01-12T15:42:55Z

Recall that whenever one thread tries to mark query node green and then fails to reconstruct a subquery's key to force it, parent query is tried to be forced or executed too. The subquery could be marked green by the thread during execution the parent query because key is now present.

Could you elaborate how this can happen? Key reconstructibility is not supposed to change.

When some thread fails to mark some node green, it won't immediately mark that node red. It will try force the deepest possible node it can (aka one with the reconstructible key). While you are forcing a parent of that node you will execute that problematic child query you were unable to mark green (due to red dependencies) or force directly (due to unreconstructible key). That problematic query node could be marked as green if its computed result didn't change. But meanwhile another thread may try mark that parent node green as well too and so could now succeed, even tho the first thread is executing that query and will try mark it red if that query has no_hash modifier:

rust/compiler/rustc_query_system/src/dep_graph/graph.rs

Lines 739 to 742 in b115ea2

    
           // This is a red node, effectively: it existed in the previous compilation 
        
           // session, its query was re-executed, but it doesn't compute a result hash 
        
           // (i.e. it represents a `no_hash` query), so we have no way of determining 
        
           // whether or not the result was the same as before.

But I suggest that's not really true. If no_hash query's dependencies are all green, it has to be green too.

And yes, no_hash queries cannot be force directly either, so it is actually done indirectly through another query.

rustbot · 2026-01-13T12:38:18Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

petrochenkov · 2026-01-13T12:47:36Z

@bors try @rust-timer queue

Try mark `no_hash` queries as green after execution

zetanumbers · 2026-01-13T12:49:40Z

Seems like no node coloring race is required after all.

rust-bors · 2026-01-13T15:03:25Z

☀️ Try build successful (CI)
Build commit: 4b1294d (4b1294da0c6bc573b4608a908d2089ed17eeb095, parent: b2a322beb29110e22a1782e2ce5ed2a0719b81ed)

rust-timer · 2026-01-13T15:44:26Z

Finished benchmarking commit (4b1294d): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 473.958s -> 475.113s (0.24%)
Artifact size: 383.19 MiB -> 383.20 MiB (0.00%)

zetanumbers · 2026-01-13T17:44:36Z

@rustbot ready

petrochenkov · 2026-01-14T13:16:32Z

compiler/rustc_query_system/src/dep_graph/graph.rs

                    // This is a red node: it existed in the previous compilation, its query
                    // was re-executed, but it has a different result from before.
+                    #[cfg(debug_assertions)]
+                    if !feed && !cx.is_eval_always(key.kind) {


Suggested change

if !feed && !cx.is_eval_always(key.kind) {

if cfg!(debug_assertions) && !feed && !cx.is_eval_always(key.kind) {

Nit: so the code is at least type checked in all build modes.

petrochenkov · 2026-01-14T13:18:47Z

(Leaving this to @cjgillot to review.)

rustbot assigned petrochenkov Dec 19, 2025

This was referenced Dec 19, 2025

assertion failed: trying to encode a dep node twice #150018

Open

ICE: rustc_query_system: dep_graph: assertion failed (dep node index out of range) #142778

Open

A dep graph node (mir_drops_elaborated_and_const_checked) does not have an unique index #141540

Open

This comment has been minimized.

Sign in to view

rust-bors bot added a commit that referenced this pull request Dec 19, 2025

Auto merge of #150156 - zetanumbers:allow_red_green_race, r=<try>

bb5a19e

Allow green-red query coloring race

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Dec 19, 2025

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Dec 19, 2025

petrochenkov reviewed Dec 19, 2025

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 19, 2025

This comment was marked as duplicate.

Sign in to view

Try mark no_hash queries as green after execution

2fe32a3

zetanumbers force-pushed the allow_red_green_race branch from 0b6923b to 2fe32a3 Compare January 13, 2026 12:38

zetanumbers changed the title ~~Allow green-red query coloring race~~ Try mark no_hash queries as green after execution Jan 13, 2026

This comment has been minimized.

Sign in to view

rust-bors bot pushed a commit that referenced this pull request Jan 13, 2026

Auto merge of #150156 - zetanumbers:allow_red_green_race, r=<try>

4b1294d

Try mark `no_hash` queries as green after execution

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jan 13, 2026

zetanumbers closed this Jan 13, 2026

rustbot removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jan 13, 2026

zetanumbers reopened this Jan 13, 2026

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 13, 2026

This comment has been minimized.

Sign in to view

rustbot removed S-waiting-on-perf Status: Waiting on a perf run to be completed. perf-regression Performance regression. labels Jan 13, 2026

Update comment about no_hash query coloring

018c501

cjgillot self-assigned this Jan 13, 2026

petrochenkov approved these changes Jan 14, 2026

View reviewed changes

petrochenkov removed their assignment Jan 14, 2026

	if !feed && !cx.is_eval_always(key.kind) {
	if cfg!(debug_assertions) && !feed && !cx.is_eval_always(key.kind) {

Uh oh!

Try mark no_hash queries as green after execution #150156

Are you sure you want to change the base?

Try mark no_hash queries as green after execution #150156

Conversation

zetanumbers commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps to reproduce

Uh oh!

rustbot commented Dec 19, 2025

Uh oh!

petrochenkov commented Dec 19, 2025

Uh oh!

zetanumbers commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matthiaskrgr commented Dec 19, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

zetanumbers commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-bors bot commented Dec 19, 2025

Uh oh!

This comment has been minimized.

rust-timer commented Dec 19, 2025

Overall result: ❌ regressions - please read the text below

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

petrochenkov commented Dec 19, 2025

Uh oh!

rustbot commented Dec 19, 2025

Uh oh!

cjgillot commented Dec 25, 2025

Uh oh!

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

zetanumbers commented Jan 12, 2026

Uh oh!

rustbot commented Jan 13, 2026

Uh oh!

petrochenkov commented Jan 13, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

zetanumbers commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-bors bot commented Jan 13, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Jan 13, 2026

Overall result: no relevant changes - no action needed

Uh oh!

zetanumbers commented Jan 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

petrochenkov commented Jan 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Try mark `no_hash` queries as green after execution #150156

Try mark `no_hash` queries as green after execution #150156

zetanumbers commented Dec 19, 2025 •

edited

Loading

zetanumbers commented Dec 19, 2025 •

edited

Loading

zetanumbers commented Dec 19, 2025 •

edited

Loading

zetanumbers commented Jan 13, 2026 •

edited

Loading