Skip to content

Regression: flutter test rebuilds build/unit_test_assets on every run, breaking concurrent invocations #187725

Description

@passsy

On the master channel, flutter test rebuilds the shared build/unit_test_assets directory on every invocation, even when nothing changed.
The rebuild deletes the directory and rewrites it non-atomically.
When two flutter test invocations run in the same project directory, one invocation's delete lands while the other's flutter_tester processes are reading an asset, and the engine throws:

Exception: Asset 'shaders/ink_sparkle.frag' not found
#0      new FragmentProgram._fromAsset (dart:ui/painting.dart:5415:7)
#1      FragmentProgram.fromAsset.<anonymous closure> (dart:ui/painting.dart:5443:39)

stable and beta do not rebuild on every run and are unaffected. flutter test -j 1 is also unaffected.

shaders/ink_sparkle.frag surfaces it because widget tests run as TargetPlatform.android, Material taps load it via InkSparkle, and _InkSparkleFactory.initializeShader loads it from a fire-and-forget Future with no error handler — so the rejection becomes an unhandled exception failing an unrelated test. Any asset read during a rebuild window is affected.

Reproduce

reproduce.shgenerates a minimal package with many tapping tests and runs four staggered flutter test invocations in the same directory:

flutter channel master && flutter upgrade
./reproduce.sh

On master (d0c4c5a892) it reports ~30 Asset 'shaders/ink_sparkle.frag' not found failures; on stable (3.44.0) it reports 0.

Real-world: passsy/spot fails this way on master CI. It is a widget-testing library whose suite runs flutter test in subprocesses with the package root as the working directory, so the nested invocations share build/unit_test_assets and run concurrently with the outer suite.

reproduce.sh

#!/usr/bin/env bash
#
# Repro for: `flutter test` intermittently throws
#
#     Exception: Asset 'shaders/ink_sparkle.frag' not found
#
# on the Flutter *master* channel when more than one `flutter test` runs
# concurrently in the same project directory. `stable`/`beta` are unaffected,
# and `flutter test -j 1` always passes.
#
# This script is fully self-contained: it generates a minimal Flutter package
# with many test files and then runs several `flutter test` invocations,
# staggered so that a later invocation rebuilds the shared asset bundle while
# an earlier invocation is still reading the shader from it.
#
# Usage:
#     flutter channel master && flutter upgrade   # once
#     ./reproduce.sh
#
# Tunables (env vars):
#     NUM_TEST_FILES   number of generated test files (default 32)
#     FLUTTER          flutter executable to use      (default "flutter")
#
set -euo pipefail

NUM_TEST_FILES="${NUM_TEST_FILES:-32}"
FLUTTER="${FLUTTER:-flutter}"
PROJECT_DIR="$(mktemp -d)/ink_sparkle_repro"
LOG_DIR="$(mktemp -d)"

echo "Flutter:"
"$FLUTTER" --version | sed 's/^/    /'
echo

# ---------------------------------------------------------------------------
# 1. Generate a minimal Flutter package with many test files.
#    Each test taps a Material button. The default test platform is Android, so
#    Material uses the InkSparkle splash, which lazily loads
#    shaders/ink_sparkle.frag on the first tap.
# ---------------------------------------------------------------------------
mkdir -p "$PROJECT_DIR/test"

cat > "$PROJECT_DIR/pubspec.yaml" <<'YAML'
name: ink_sparkle_repro
description: Minimal repro for flaky ink_sparkle.frag asset serving.
publish_to: none
environment:
  sdk: ">=3.5.0 <4.0.0"
dependencies:
  flutter:
    sdk: flutter
dev_dependencies:
  flutter_test:
    sdk: flutter
flutter:
  uses-material-design: true
YAML

for i in $(seq -w 1 "$NUM_TEST_FILES"); do
  cat > "$PROJECT_DIR/test/tap_${i}_test.dart" <<DART
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';

void main() {
  testWidgets('tap a Material button #$i', (tester) async {
    await tester.pumpWidget(const MaterialApp(home: _ButtonPage()));
    // Several taps to widen the window in which the shader is read from disk.
    for (var n = 0; n < 5; n++) {
      await tester.tap(find.byType(ElevatedButton));
      await tester.pump(const Duration(milliseconds: 50));
    }
    await tester.pumpAndSettle();
  });
}

class _ButtonPage extends StatelessWidget {
  const _ButtonPage();

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Center(
        child: ElevatedButton(onPressed: () {}, child: const Text('tap')),
      ),
    );
  }
}
DART
done

cd "$PROJECT_DIR"
echo "Generated $NUM_TEST_FILES test files in $PROJECT_DIR"
"$FLUTTER" pub get > /dev/null
echo

# ---------------------------------------------------------------------------
# 2. Run several `flutter test` invocations in the same directory, staggered.
#    On master each invocation unconditionally rebuilds build/unit_test_assets
#    (delete + rewrite). A later invocation's delete therefore lands while an
#    earlier invocation's flutter_tester processes are reading the shader.
# ---------------------------------------------------------------------------
echo "Running 4 staggered, concurrent 'flutter test' invocations..."
"$FLUTTER" test > "$LOG_DIR/run_A.log" 2>&1 &
sleep 4; "$FLUTTER" test > "$LOG_DIR/run_B.log" 2>&1 &
sleep 2; "$FLUTTER" test > "$LOG_DIR/run_C.log" 2>&1 &
sleep 2; "$FLUTTER" test > "$LOG_DIR/run_D.log" 2>&1 &
wait

# ---------------------------------------------------------------------------
# 3. Report.
# ---------------------------------------------------------------------------
NEEDLE="shaders/ink_sparkle.frag' not found"
TOTAL=0
for f in "$LOG_DIR"/run_*.log; do
  c=$(grep -c "$NEEDLE" "$f" || true)
  TOTAL=$((TOTAL + c))
  printf '    %s: %s "Asset not found" failures\n' "$(basename "$f")" "$c"
done

echo
if [ "$TOTAL" -gt 0 ]; then
  echo "REPRODUCED: $TOTAL \"Asset 'shaders/ink_sparkle.frag' not found\" failures."
  echo
  echo "Example stack trace:"
  { grep -A6 "$NEEDLE" "$LOG_DIR"/run_*.log || true; } | head -8 | sed 's/^/    /' || true
  echo
  echo "For comparison, this is always green:"
  echo "    cd $PROJECT_DIR && $FLUTTER test -j 1"
  exit 0
else
  echo "Not reproduced this run (it is timing-dependent)."
  echo "Re-run, or increase NUM_TEST_FILES (e.g. NUM_TEST_FILES=64 ./reproduce.sh)."
  echo "Slower / many-core machines and CI reproduce it most reliably."
  exit 1
fi

Expected

flutter test rebuilds build/unit_test_assets only when an asset actually changed (as on stable).

Cause

DevFSFileContent.isModifiedAfter() in packages/flutter_tools/lib/src/devfs.dart returns true whenever _fileStat == null, ignoring the time argument:

bool isModifiedAfter(DateTime time) {
  final (FileStat? currentStat, _) = _statFile();
  if (_fileStat == null && currentStat == null) {
    return false;
  }
  return _fileStat == null || currentStat == null || currentStat.modified.isAfter(time);
  //     ^^^^^^^^^^^^^^^^^^ always true for a freshly built bundle entry
}

_fileStat is the sync baseline, written only by markClean(). The freshly built AssetBundle that _needsRebuild() (in packages/flutter_tools/lib/src/commands/test.dart) inspects is never marked clean, so _fileStat is always null for it. isModifiedAfter() therefore returns true even when the asset's on-disk mtime is older than the manifest, _needsRebuild() always returns true, and writeBundle() deletes + rewrites the directory on every run.

This is a regression from #187488 ("fix(tool): initialize asset isModified state on startup to prevent 2x hot restart slowdown"), which made the stat accessors pure (_stat()_statFile()). Before it, the mutating _stat() populated _fileStat as a side effect of reads during the asset build, so _needsRebuild() happened to compare against a real baseline and worked. (Not #186902 — at that commit the stat accessors still used the mutating _stat().)

Bisection — repro run against each revision's tool code (same engine):

revision tool code result
becb39948f2 (parent) mutating _stat() green
21c65665fc5 (#186902) mutating _stat() green
1e492c7f421 (#187488) pure _statFile() flaky
d0c4c5a892 (current master) pure _statFile() flaky

Suggested fixes

  • Make isModifiedAfter(time) a pure comparison of the file's current modification time against time, without consulting the _fileStat sync baseline (matching DevFSByteContent/DevFSStringCompressingBytesContent, which already do _creationTime.isAfter(time)). The baseline is meaningful for isModified, not for an explicit reference time.
  • Independently, make writeBundle() swap the asset directory atomically (write to a temp dir, then rename) so even a legitimate rebuild never exposes missing files to concurrent readers.

@kevmoo

Version

Flutter 3.45.0-1.0.pre-413 • channel master
Framework • revision d0c4c5a892 • 2026-06-08
Engine    • hash 24096f9c3cb6c72c638a19b64bf3b0cb7aecfcfd
Tools     • Dart 3.13.0

Metadata

Metadata

Assignees

Labels

a: tests"flutter test", flutter_test, or one of our teststeam-toolOwned by Flutter Tool teamtoolAffects the "flutter" command-line tool. See also t: labels.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions