Skip to content

Race condition in update_engine_version scripts causes 404 for engine_stamp.json on parallel flutter command invocations #184027

Description

@b055man

Steps to reproduce

below is a summary of a prolonged analysis generated with LLM's help

Description

When executing multiple flutter or dart commands in parallel (a common scenario when using monorepo tools like melos exec, or running parallel CI jobs), commands intermittently fail with a fatal error:

Failed to download https://storage.googleapis.com/flutter_infra_release/flutter//engine_stamp.json. Ensure you have network connectivity and then try again.
Exception: 404

Notice the malformed URL with // instead of the expected engine version hash.

Root Cause Analysis

  1. Unconditional Invocation: Every flutter or dart command unconditionally calls bin/internal/update_engine_version.sh (or .ps1) via the upgrade_flutter function in shared.sh, before the Dart-side _lock is acquired.

  2. Non-Atomic Write: The update_engine_version scripts write the engine hash to bin/cache/engine.stamp using non-atomic redirection:

    • echo $ENGINE_VERSION >"$FLUTTER_ROOT/bin/cache/engine.stamp" (Linux/macOS)
    • Set-Content -Path ... -Value $engineVersion (Windows)

    These shell commands first truncate the file to 0 bytes, then write the new content.

  3. The Race Condition: If Process A is truncating and writing to engine.stamp exactly when Process B's Dart isolate reaches await globals.cache.updateAll({DevelopmentArtifact.informative});, Process B will read an empty engine.stamp file.

  4. The Crash: Because the read returns an empty string, the FlutterEngineStamp artifact constructs an invalid URL (.../flutter//engine_stamp.json). The download naturally returns a 404, and because DevelopmentArtifact.informative failures are treated as fatal during startup, the entire command crashes.

Reproduction

This race condition is highly timing-dependent but can be consistently reproduced using a bash stress test that runs the script in parallel while continuously reading the stamp file:

#!/bin/bash
# Run from the Flutter SDK root
while true; do cat bin/cache/engine.stamp; done > reader.log &
READER_PID=$!

for i in {1..30}; do
  (for j in {1..50}; do bash bin/internal/update_engine_version.sh; done) &
done
wait

kill $READER_PID
# Count how many times an empty file was read
grep -c "^$" reader.log

On macOS/Linux, this stress test typically catches hundreds of empty reads within a few seconds.

Proposed Fix

Modify the update_engine_version scripts to use atomic writes. By writing to a temporary file first and then moving/renaming it, the OS guarantees that concurrent readers will always see either the complete old content or the complete new content—never an empty, truncated file.

1. bin/internal/update_engine_version.sh

Current:

# Write the engine version out so downstream tools know what to look for.
echo $ENGINE_VERSION >"$FLUTTER_ROOT/bin/cache/engine.stamp"

Proposed:

# Write the engine version out so downstream tools know what to look for.
# Use a temporary file and atomic mv to prevent race conditions during parallel flutter executions.
_es_tmp="$FLUTTER_ROOT/bin/cache/engine.stamp.tmp.$$"
echo "$ENGINE_VERSION" >"$_es_tmp" && mv "$_es_tmp" "$FLUTTER_ROOT/bin/cache/engine.stamp"

2. bin/internal/update_engine_version.ps1

Current:

# Write the engine version out so downstream tools know what to look for.
Set-Content -Path $flutterRoot/bin/cache/engine.stamp -Value $engineVersion -Encoding Ascii

Proposed:

# Write the engine version out so downstream tools know what to look for.
# Use a temporary file and atomic move to prevent race conditions during parallel flutter executions.
$esTmp = "$flutterRoot/bin/cache/engine.stamp.tmp.$PID"
Set-Content -Path $esTmp -Value $engineVersion -Encoding Ascii
Move-Item -Path $esTmp -Destination "$flutterRoot/bin/cache/engine.stamp" -Force

Impact

This is a low-risk, backward-compatible fix that completely eliminates the engine_stamp.json 404 flakiness in parallel build environments, CI pipelines, and monorepos.

Expected results

see the description

Actual results

see the description

Code sample

Code sample
[Paste your code here]

Screenshots or Video

Screenshots / Video demonstration

[Upload media here]

Logs

Logs
[Paste your logs here]

Flutter Doctor output

Doctor output
[Paste your output here]

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Important issues not at the top of the work listc: crashStack traces logged to the consolehas reproducible stepsThe issue has been confirmed reproducible and is ready to work onteam-toolOwned by Flutter Tool teamtoolAffects the "flutter" command-line tool. See also t: labels.triaged-toolTriaged by Flutter Tool team

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions