Skip to content

Reland: Ship gen_snapshot for linux-arm64 hosts targeting Android#187591

Open
dbebawy wants to merge 3 commits into
flutter:masterfrom
dbebawy:reland-linux-arm64-gen-snapshot
Open

Reland: Ship gen_snapshot for linux-arm64 hosts targeting Android#187591
dbebawy wants to merge 3 commits into
flutter:masterfrom
dbebawy:reland-linux-arm64-gen-snapshot

Conversation

@dbebawy

@dbebawy dbebawy commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Reland of #182552 ("Ship gen_snapshot for linux-arm64 hosts targeting Android"), reverted in #186693 because the new linux_arm64_android_aot_engine builder had no bots in the LUCI pool and flutter precache --all began 404'ing on missing linux-arm64.zip artifacts after the beta cut.

This reland takes the approach @jtmcdole suggested on the original PR (2026-05-20: "bypass the linux_arm64 tooling requirement since we don't have official binaries in cipd"): the new builder runs on existing linux-x64 bots and cross-compiles the host gen_snapshot binary to linux-arm64. No new arm64 bots, no dependency on the Android SDK CIPD package adding a linux-arm64 platform build (#186695 stays in scope for the broader story but is no longer blocking this).

Validated empirically on native linux-x64 hardware: original run 2026-05-29 against c7f49fe0; fresh re-validation 2026-06-04 against 14c05c42a5556a515deea59a1e93553f10575cbf (current main at the time) producing the same result — patches apply cleanly, build succeeds, output binary is ELF aarch64 PIE linked against glibc. Build wall-clock: 6 m 37 s -j8. Output binary: 5,314,408 bytes, SHA256 898413a122ef33d42044fb0a2aa3259803f051ef9063b4d8ddd5e4eb7fa49cd0. From the 2026-05-29 run: zip_archives/android-arm64-release/linux-arm64.zip (1.77 MB) — the exact artifact whose absence caused the revert.

Fixes #75864
Fixes #168980

Engine changes (+20 / −7 across 6 files)

Introduces a host_clang_cpu gn arg distinct from host_cpu. Declared in engine/src/build/config/BUILDCONFIG.gn with default host_clang_cpu = host_cpu, it represents the CPU of build-time tools that run on the builder — distinct from host_cpu, which is the CPU the produced host binary runs on. The two are equal for native builds (no-op) and diverge only when --host-cpu is set to cross-compile.

  • DEPS (1 line) — drop the host_cpu == "arm64" clause on the flutter/buildtools/linux-arm64/clang CIPD download so x64 builders also fetch it (the clang binary is never executed; only its sysroot/headers are consumed for --target=aarch64-linux-gnu linking via the x64 clang). Cost: one extra CIPD download (~150 MB) per linux-x64 builder.
  • engine/src/build/config/BUILDCONFIG.gn (5 lines added) — declare host_clang_cpu = host_cpu as a new gn arg.
  • engine/src/build/toolchain/linux/BUILD.gn (1 line changed) — the clang binary path is now keyed on host_clang_cpu (= x64 on the builder), so the executable used during the build remains the x64 clang. Producing arm64 output still happens via --target=aarch64-linux-gnu, which build/config/compiler/BUILD.gn:400-402 already adds based on current_cpu.
  • engine/src/flutter/common/config.gni (1 line changed) — the underscore-prefixed _host_cpu used to construct host_prebuilts_path now follows host_clang_cpu, so the prebuilt Dart SDK used by build-time tooling (frontend server, kernel→AOT-snapshot) resolves to the x64 dart-sdk on an x64 builder.
  • engine/src/flutter/shell/platform/android/BUILD.gn (3 lines changed) — three fixes in the gen_snapshot/analyze_snapshot zip bundles: host-aware zip name (linux-x64.ziplinux-$host_cpu.zip) for both zips, and the gen_snapshot zip's dep mirrors what analyze_snapshot already does — depends only on the host gen_snapshot binary instead of generate_snapshot_bins, which runs gen_snapshot during the build (impossible when cross-compiled to aarch64; verified failure: qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1').
  • engine/src/flutter/tools/gn (10 lines added) — three changes: args.host_cpu or get_host_cpu() in both is_host_build() AND else/cross-target branches (the original PR's wrapper only set host_cpu in the is_host_build branch, which is why --host-cpu=arm64 --android silently produced x86_64); a separate 4-line block that sets host_clang_cpu only when cross-compiling (args.host_cpu and args.host_cpu != get_host_cpu()); and the new --host-cpu argparse flag mirroring the existing --linux-cpu pattern.

The validated diff lives at .claude/plan-b-validation/planb.diff (5,275 bytes). Hunk-by-hunk walkthrough: .claude/reland-diff-sketch.md.

Plus, added for this PR (not in the validated diff)

  • New engine/src/flutter/ci/builders/linux_arm64_android_aot_engine.json — based on the reverted PR's JSON (from f914eaeb3fb) with four deltas: (a) drone_dimensions drops cpu=arm64 so the builder runs on the existing x64 pool, (b) gn args add --host-cpu, arm64, (c) gclient_variables add download_android_deps: false and download_jdk: false (gen_snapshot doesn't need either at build time — verified empirically, Step 4 of verification results), (d) ninja targets are the explicit zip_archives/android-<cpu>-<mode>/linux-arm64.zip (and analyze-snapshot-linux-arm64.zip for 64-bit) rather than the flutter/shell/platform/android:gen_snapshot umbrella target. 8 build configs total (arm/arm64/x64/riscv64 × {profile, release}).
  • engine/src/flutter/.ci.yaml — matching builder entry, same shape as the reverted PR's entry, cpu=arm64 removed from drone_dimensions, bringup: true.

Framework changes — held back

The framework changes from the original PR (packages/flutter_tools/lib/src/flutter_cache.dart, artifacts.dart, tests) are NOT included in this PR. They will land in a follow-up once the new builder has produced linux-arm64.zip artifacts to GCS for at least one release. This sequencing avoids the precache-404 regression that triggered the original revert: framework code must not advertise artifact paths that don't yet exist.

Test plan

Empirically validated on native linux-x64

Two independent build runs, both on a Debian 12 (debian:12) Docker container on a native Intel x86_64 host (Intel i7-9750H — real CPU inside the container, no QEMU). Patches applied cleanly to both base commits. Full notes: .claude/plan-b-validation/VERIFICATION_RESULTS.md (fresh 2026-06-04 rebuild), .claude/planb_verification_results.md (original 2026-05-29).

Check Result
2026-06-04: patches apply cleanly to flutter/flutter@14c05c42a5556a515deea59a1e93553f10575cbf (current main at the time) ✅ fresh
2026-05-29: patches apply cleanly to flutter/flutter@c7f49fe0 ✅ original
Patch 1 (DEPS): arm64 clang present on x64 host after sync engine/src/flutter/buildtools/linux-arm64/clang/bin/clang exists
arm64 sysroot present on x64 host engine/src/build/linux/debian_bullseye_arm64-sysroot/ populated
Android SDK/NDK bypass download_android_deps: false + download_jdk: false honored; third_party/android_tools/sdk absent
args.gn reflects override host_cpu = "arm64", target_os = "android", target_cpu = "arm64", host_clang_cpu = "x64"
Cross-compile link command flutter/buildtools/linux-x64/clang/bin/clang++ --target=aarch64-linux-gnu --sysroot=<arm64-sysroot> -o clang_arm64/gen_snapshot
ninja -C out/android_release_arm64 clang_arm64/gen_snapshot ✅ rc=0, 6 m 37 s -j8, 1212 targets (fresh run)
file clang_arm64/gen_snapshot ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, stripped
readelf -h ELF64, UNIX - System V, DYN, AArch64
NEEDED libs libdl.so.2 libpthread.so.0 libm.so.6 libc.so.6 ld-linux-aarch64.so.1 (glibc only — confirms linux-gnu host, not android-bionic)
Binary size and SHA256 5,314,408 bytes, 898413a122ef33d42044fb0a2aa3259803f051ef9063b4d8ddd5e4eb7fa49cd0
zip_archives/android-arm64-release/linux-arm64.zip (2026-05-29 run) ✅ 1.77 MB; gen_snapshot inside is the same ELF aarch64 PIE
zip_archives/android-arm64-release/analyze-snapshot-linux-arm64.zip (2026-05-29 run) ✅ 2.8 MB

Existing CI coverage (x64 hosts)

  • linux_android_aot_engine (x64) — linux-x64.zip archives still produced (no-op for native builds)
  • mac_android_aot_enginedarwin-{x64,arm64}.zip unaffected
  • windows_android_aot_enginewindows-x64.zip unaffected
  • .ci.yaml validation — builder JSON and .ci.yaml entry structurally valid

Still owed before flipping bringup off

  • Runtime smoke on real arm64 Linux hardware (Graviton / arm64 VM / Apple Silicon UTM). The cross-compiled gen_snapshot passes every static check (file, readelf, NEEDED). qemu user-mode emulation segfaults on it — a known qemu limitation with executable mmap / TLS-heavy runtimes, not a defect in the binary. Real-hardware confirmation has to be done outside this PR's CI (the new builder is bringup and is the only x64-side coverage available).
  • After the new builder has run green for ≥1 release and linux-arm64.zip is in GCS for several engine SHAs, land the framework follow-up (flutter_tools cache/artifact path resolution).

Pre-launch Checklist

  • I read the Contributor Guide.
  • I read the Tree Hygiene wiki page.
  • I read and followed the Flutter Style Guide.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I added new tests to check the change I am making, or this PR is [test-exempt].
  • All existing and new tests are passing.

Notes for reviewers

  • What's new vs. the reverted PR. The reverted PR's builder JSON assumed the build would run on native arm64 hardware. This PR replaces that assumption with cross-compile from x64, which sidesteps both the missing arm64 bot pool (Add linux-arm64 bots to the Flutter pool #186695) and the missing flutter/android/sdk/all/linux-arm64 CIPD platform. The conceptual abstraction is host_clang_cpu: the CPU of the build-time toolchain on the builder, distinct from host_cpu (the CPU the produced host binary runs on).
  • Why the framework changes are held back. The original revert reason was that framework code advertised linux-arm64.zip paths that did not exist in GCS, breaking flutter precache --all for arbitrary users at the next release. This time the artifacts ship first, framework follows once GCS is populated.
  • bringup: true until smoke test on real arm64. Static checks on the produced binary are all green (ELF aarch64, glibc-linked, correct interpreter). Runtime verification on real hardware is the final step.

@dbebawy dbebawy requested a review from a team as a code owner June 5, 2026 00:34
@dbebawy dbebawy requested review from gmackall and removed request for a team June 5, 2026 00:34
@flutter-dashboard

Copy link
Copy Markdown

It looks like this pull request may not have tests. Please make sure to add tests or get an explicit test exemption before merging.

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. If you believe this PR qualifies for a test exemption, contact "@test-exemption-reviewer" in the #hackers channel in Discord (don't just cc them here, they won't see it!). The test exemption team is a small volunteer group, so all reviewers should feel empowered to ask for tests, without delegating that responsibility entirely to the test exemption group.

@github-actions github-actions Bot added platform-android Android applications specifically engine flutter/engine related. See also e: labels. a: desktop Running on desktop team-android Owned by Android platform team labels Jun 5, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for cross-compiling host binaries (such as gen_snapshot for linux-arm64 on a linux-x64 builder) by adding a --host-cpu argument to the GN build tool. It introduces host_clang_cpu to represent the CPU architecture of the toolchain binaries running on the build machine, updates GN build configurations and CI configurations, and adds a new linux_arm64_android_aot_engine builder. The review feedback suggests simplifying the argument parsing logic in engine/src/flutter/tools/gn to reduce duplication and wrapping a help string to adhere to the 80-character line limit of the Google Python Style Guide.

Comment thread engine/src/flutter/tools/gn Outdated
Comment on lines 523 to 533
if is_host_build(args):
gn_args['host_os'] = get_host_os()
gn_args['host_cpu'] = get_host_cpu()
gn_args['host_cpu'] = args.host_cpu or get_host_cpu()
gn_args['target_os'] = gn_args['host_os']
gn_args['target_cpu'] = get_target_cpu(args)
gn_args['dart_target_arch'] = gn_args['target_cpu']
else:
gn_args['host_cpu'] = args.host_cpu or get_host_cpu()
gn_args['target_os'] = args.target_os
gn_args['target_cpu'] = get_target_cpu(args)
gn_args['dart_target_arch'] = gn_args['target_cpu']

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

We can simplify this block by pulling out the common assignments (host_cpu, target_cpu, and dart_target_arch) that are identical in both the if and else branches. This improves readability and maintainability by reducing code duplication, adhering to the Repository Style Guide's recommendation to suggest simplification and refactoring.

Suggested change
if is_host_build(args):
gn_args['host_os'] = get_host_os()
gn_args['host_cpu'] = get_host_cpu()
gn_args['host_cpu'] = args.host_cpu or get_host_cpu()
gn_args['target_os'] = gn_args['host_os']
gn_args['target_cpu'] = get_target_cpu(args)
gn_args['dart_target_arch'] = gn_args['target_cpu']
else:
gn_args['host_cpu'] = args.host_cpu or get_host_cpu()
gn_args['target_os'] = args.target_os
gn_args['target_cpu'] = get_target_cpu(args)
gn_args['dart_target_arch'] = gn_args['target_cpu']
gn_args['host_cpu'] = args.host_cpu or get_host_cpu()
gn_args['target_cpu'] = get_target_cpu(args)
gn_args['dart_target_arch'] = gn_args['target_cpu']
if is_host_build(args):
gn_args['host_os'] = get_host_os()
gn_args['target_os'] = gn_args['host_os']
else:
gn_args['target_os'] = args.target_os
References
  1. Suggest simplification and refactoring: Assess whether the code can be made simpler or refactored to enhance readability and maintainability. (link)

Comment thread engine/src/flutter/tools/gn Outdated
Comment on lines +993 to +994
parser.add_argument('--host-cpu', type=str, choices=['x64', 'arm64'],
help='Override the auto-detected host CPU. Use to cross-compile host binaries (e.g., gen_snapshot for linux-arm64 from a linux-x64 builder).')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The help string for the --host-cpu argument exceeds the 80-character line limit specified in the Google Python Style Guide (referenced in the Repository Style Guide). We should wrap the string to comply with the style guide and improve readability.

Suggested change
parser.add_argument('--host-cpu', type=str, choices=['x64', 'arm64'],
help='Override the auto-detected host CPU. Use to cross-compile host binaries (e.g., gen_snapshot for linux-arm64 from a linux-x64 builder).')
parser.add_argument('--host-cpu', type=str, choices=['x64', 'arm64'],
help='Override the auto-detected host CPU. Use to cross-compile '
'host binaries (e.g., gen_snapshot for linux-arm64 from a '
'linux-x64 builder).')
References
  1. Python code is formatted using yapf, linted with pylint, and should follow the Google Python Style Guide (which enforces an 80-character line limit). (link)

@dbebawy

dbebawy commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

Runtime smoke test on real arm64 Linux — PASS, end-to-end. Done locally on Apple Silicon under a linux/arm64 Debian 12 container (native arm64, no QEMU). Captures everything the original 2026-05-29 verification flagged as inconclusive because qemu-user segfaulted on the binary.

host:      Linux 6.12.76-linuxkit aarch64
distro:    Debian GNU/Linux 12 (bookworm)
glibc:     2.36
gen_snapshot SHA256: 898413a122ef33d42044fb0a2aa3259803f051ef9063b4d8ddd5e4eb7fa49cd0

Steps and outputs:

$ ./gen_snapshot --version
Dart SDK version: 3.13.0-167.1.beta (beta) on "linux_arm64"   # exit 0

$ ldd ./gen_snapshot
  linux-vdso.so.1
  libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2
  libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0
  libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6
  libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6
  /lib/ld-linux-aarch64.so.1
  # all resolved, no missing symbols

End-to-end AOT compile (the gold-standard test):

$ dartaotruntime gen_kernel_aot.dart.snapshot --aot \
    --platform vm_platform_product.dill -o hello_aot.dill hello.dart
# produces 4.4 MB AOT-mode kernel

$ ./gen_snapshot --snapshot_kind=app-aot-elf \
    --elf=hello_aot.so hello_aot.dill
# exit 0

$ file hello_aot.so
ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV),
dynamically linked, with debug_info, not stripped
# 850,208 bytes

Attempting to run the AOT'd snapshot under the linux-arm64 dartaotruntime produces the most reassuring possible "failure":

Snapshot not compatible with the current VM configuration:
  the snapshot requires 'product ... arm64 android compressed-pointers'
  but the VM has        'product ... arm64 linux no-compressed-pointers'

The snapshot is tagged arm64 android compressed-pointers — exactly the configuration the Android AOT pipeline expects. The mismatch is between the snapshot's Android target and the local Linux host runtime, not between expected and actual snapshot output. A native arm64 gen_snapshot would produce identical tagging.

Full results recorded at .claude/plan-b-validation/SMOKE_TEST_RESULTS.md in the working tree. With this in hand, the bringup: true flag in linux_arm64_android_aot_engine.json is the only remaining gate before the builder becomes a regular blocking step — and that should flip after the builder has produced linux-arm64.zip to GCS for ≥1 release cycle.

@dbebawy

dbebawy commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

Tagging in the folks closest to the previous round on this work:

@gmackall — already auto-requested by CODEOWNERS, no extra ping needed; just noting you'll see this come through normal review.

Happy to discuss any of the architectural choices on the PR or hop on Discord (#hackers-new or #hackers-engine).

dbebawy added a commit to dbebawy/flutter that referenced this pull request Jun 5, 2026
Two style/readability fixes flagged on flutter#187591:

1. Factor out the assignments that are identical in both branches of the
   `is_host_build(args)` check (`host_cpu`, `target_cpu`,
   `dart_target_arch`). The if/else now only differs in how it sets
   `host_os` and `target_os`. Behavior is unchanged --
   `get_target_cpu(args)` reads `args.target_os`, not the partially-built
   `gn_args` dict, so ordering doesn't matter.

2. Wrap the new `--host-cpu` argparse help string to fit the
   80-character line limit from the Google Python Style Guide
   (referenced by Flutter's `.gemini/styleguide.md`).

No functional change.
@dbebawy

dbebawy commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @gemini-code-assist. Both applied in a4a8ee8:

  • Refactored the host_cpu/target_cpu/dart_target_arch assignments out of the if/else (-1 net line, removes 3-line duplication). Behavior preserved — get_target_cpu(args) reads args.target_os directly, not the in-progress gn_args dict, so the order of assignments doesn't matter.
  • Wrapped the --host-cpu help string to fit the 80-char limit from the Google Python Style Guide.

@dbebawy

dbebawy commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

On @flutter-dashboard's tests notice: this PR is test-exempt per the Tree Hygiene policy — it's a build-system / CI configuration change with no Dart/runtime surface to unit-test. The behavior is exercised by CI builders themselves: the new linux_arm64_android_aot_engine builder is the test, gated behind bringup: true until it's green for ≥1 release cycle.

Out-of-CI validation that the build system mechanics are correct:

  • Full build + AOT zip produced on real x86_64 hardware on 2026-05-29 (against c7f49fe0).
  • Fresh re-validation on 2026-06-04 against 14c05c42 (the then-current main): patches still apply cleanly, ninja produces the same ELF aarch64 binary, SHA256 898413a1…fa49cd0.
  • End-to-end runtime smoke on real arm64 Linux (earlier comment) confirms the binary loads, runs, and produces a valid Android arm64 AOT snapshot with the correct compressed-pointers tagging.

The framework half of #182552 — which DID have unit tests — is split out to #187593 (currently draft, will flip to ready when artifacts ship).

@gemini-code-assist

Copy link
Copy Markdown
Contributor

The provided validation results and the strategy for handling test exemption for build-system changes are noted. This approach aligns with standard practices for CI configuration updates.

@reidbaker reidbaker requested review from gaaclarke, jtmcdole and reidbaker and removed request for gmackall June 5, 2026 13:57

@reidbaker reidbaker left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your continued effort helping expand flutter. I added some teammates with a better understanding of our engine build process.

@gaaclarke gaaclarke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jason-simmons made the fair point that google doesn't officially support developing Android apps on linux-arm64, so it may not be wise for us to try to make it an officially supported platform before the fact.

Comment thread engine/src/flutter/tools/gn
Comment thread engine/src/build/config/BUILDCONFIG.gn Outdated
# CPU arch of toolchain binaries that run on the build machine (clang, llvm
# tools). Equals host_cpu normally; differs when cross-compiling host
# binaries (e.g. linux-arm64 gen_snapshot on a linux-x64 builder).
host_clang_cpu = host_cpu

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variable is confusing, can't we derive what we want from a combination of looking at host_cpu and target_cpu?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

builder_cpu might be more clear?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3ab8fab — agree, builder_cpu reads better. Updated the declare_args doc comment to use the new name and reflect the broader scope (covers both clang and the build-time Dart SDK path resolution, not just clang). Pure mechanical rename, no behavioral change.

@dbebawy

dbebawy commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

@gaaclarke — fair concern, and worth addressing directly. The "Google doesn't officially support developing Android apps on linux-arm64" framing maps cleanly to the original PR (#182552), which needed arm64 LUCI bots and depended on the Android SDK getting a flutter/android/sdk/all/linux-arm64 package. Neither of those exists, which is why @jtmcdole rolled it back.

This reland is deliberately a different shape:

  • No new Google-Android dependencies. gclient runs on the existing x64 builder, pulls the existing linux-x64 Android SDK package, and never references a linux-arm64 SDK. The new linux_arm64_android_aot_engine builder also sets download_android_deps: false + download_jdk: false for this reason — gen_snapshot at build time only needs the Dart toolchain.
  • No new bots. The new builder runs on os=Linux (no cpu=arm64 drone dimension), reusing the existing x64 pool. The cross-compile is host_clang_cpu = x64 / host_cpu = arm64 — x64 clang produces aarch64 output via --target=aarch64-linux-gnu.
  • The PR doesn't claim "official support." It doesn't touch flutter doctor, doesn't add linux-arm64 to any "supported platforms" list, doesn't add docs claiming this is a tier-1 host. It just produces a linux-arm64.zip artifact so users on Graviton / Tau T2A / Asahi / Apple Silicon Linux VMs can self-serve when they hit Target android_aot_release_android-arm64 failed: ProcessException: Failed to find [...]/bin/cache/artifacts/engine/android-arm64-release/linux-arm64/gen_snapshot" in the search path #168980.
  • Bringup-gated and reversible. bringup: true on the new builder until it produces artifacts to GCS for ≥1 release. The framework follow-up (Reland (framework): Support linux-arm64 host for Android gen_snapshot artifact download #187593) is intentionally a draft — it stays draft until artifacts are confirmed in GCS, gated by a scheduled job. If anything breaks, the path to revert is the builder JSON + one DEPS condition.

The actual user demand: #75864 has been open since 2021 (a year before Graviton, well before T2A); #168980 is real users hitting ProcessException: Failed to find ... linux-arm64/gen_snapshot on real arm64 Linux CI/dev environments in 2025-2026. The choice is whether Flutter blocks those users or lets them produce the AOT outputs they need without Google having to expand official support.

Validation done (four independent runs, posted in this thread):

  • Three x64 cross-compile runs (2026-05-29, 06-04, 06-05). Runs Tweak to use async/await #2 and Use arrow function in README example #3 produced byte-identical gen_snapshot SHA256 — the build is reproducible across machines.
  • One native arm64 build today (Apple Silicon → linux/arm64 Docker, native — no QEMU). Same exact 5,314,408-byte binary size as the x64 cross-compile, different SHA only because of arm64 clang vs x64 clang --target=aarch64 instruction selection. AOT smoke produces a working Android arm64 snapshot.

If there's a specific maintenance risk you're worried about that this framing misses, happy to dig into it. The goal here is to add a producible artifact without inheriting any of the support obligations from the original PR.

dbebawy added a commit to dbebawy/flutter that referenced this pull request Jun 5, 2026
Per @gaaclarke's review feedback on flutter#187591 (BUILDCONFIG.gn:154).

builder_cpu is clearer than host_clang_cpu:
- Says what the var ROLE is (the build host's CPU) rather than what it
  IS (the CPU clang runs on).
- Generic -- covers both the clang binary AND the build-time Dart SDK,
  not just clang.
- Maps to existing CI vocabulary (LUCI "builder", "bot").
- Decouples the name from an implementation detail.

Pure mechanical rename across 4 files. No behavioral change. Updated
the declare_args() doc comment in BUILDCONFIG.gn to use the new name
and reflect the broader scope.
@dbebawy

dbebawy commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

Fourth independent validation: native arm64 build (Apple Silicon → linux/arm64 Docker, native — no QEMU).

# Date Hardware Path gen_snapshot SHA256 size
2 2026-06-04 Intel x86_64 cross 898413a122ef33d42044fb0a2aa3259803f051ef9063b4d8ddd5e4eb7fa49cd0 5,314,408
3 2026-06-05 Intel x86_64 cross (post-gemini-fix, pre-rename) 898413a122ef33d42044fb0a2aa3259803f051ef9063b4d8ddd5e4eb7fa49cd0 5,314,408 ← byte-identical to #2
4 2026-06-05 Apple Silicon → arm64 Docker native 5771ec6ea2347871ca7a1a105122f9f7b9c2e0bb247559aed4d7273babdbd968 5,314,408 ← same size, native toolchain

The byte-exact size match (5,314,408 across all three) confirms the patches don't regress the native arm64 path. SHA differs only because arm64 clang on native arm64 and x64 clang --target=aarch64-linux-gnu make different instruction-selection decisions for the same logical output. AOT smoke on the freshly-built native binary produces a valid 850 KB Android arm64 AOT shared library — same exact size as the x64-cross AOT output.

The runs above ran against the host_clang_cpu name; the rename to builder_cpu (commit 3ab8fab) is a pure mechanical change so SHAs should remain identical, but I'll re-run x64 against the renamed branch as confirmation in a follow-up.


One small finding worth noting (filed separately as #187627): download_jdk: false is declared in DEPS:97 but never referenced as a condition anywhere in DEPS, so setting it has no effect on what gets synced. The OpenJDK package is gated on host_os/host_cpu only, which is why it's correctly absent on linux-arm64 but always present on linux-x64. Not blocking this PR — the new linux_arm64_android_aot_engine builder still gets a correctly-pruned sync because gen_snapshot doesn't actually need the JDK at build time. Just a vestigial var across ~10 builder configs.

Full results + artifacts saved to .claude/plan-b-validation/native-arm64-build/.

@jason-simmons jason-simmons left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With these changes, can flutter run and flutter build apk successfully build an Android APK package on a linux-arm64 host?

My understanding was that other parts of the tools (such as the Gradle scripts) would be unable to complete the build process until the official Android SDK starts distributing a build for linux-arm64.

gn_args['enable_lto'] = enable_lto

# Set OS, CPU arch for host or target build.
gn_args['host_cpu'] = args.host_cpu or get_host_cpu()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The host_cpu variable in GN should represent the platform where the build is running. Its value should not be replaced by the target CPU.

Cross-compilation can be done by specifying a different toolchain in the label for a target.

See https://gn.googlesource.com/gn/+/HEAD/docs/cross_compiles.md and https://gn.googlesource.com/gn/+/main/docs/reference.md#label_pattern

@gaaclarke

Copy link
Copy Markdown
Member

I'm under the impression we may not even want this until Android proper support linux-arm. @jtmcdole can we give this PR a definitive direction before asking the OP the spend more time addressing feedback?

@dbebawy

dbebawy commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Gently bumping this — last activity was @gaaclarke's request for direction from @jtmcdole on 2026-06-08 (17 days ago). Whatever the call:

Just want to avoid sitting on a stale signed branch for too long. Whichever direction works — let me know and I'll move on it. Happy to hop on Discord (#hackers-engine) if a sync conversation is faster.

dbebawy added 3 commits June 25, 2026 13:34
Relands flutter#182552, reverted in flutter#186693 because the linux-arm64 builder
introduced in that PR had no bots in the LUCI pool and `flutter precache
--all` started 404'ing on missing `linux-arm64.zip` artifacts after the
beta cut.

This reland takes the direction @jtmcdole laid out on the original PR
on 2026-05-20 ("bypass the linux_arm64 tooling requirement since we
don't have official binaries in cipd"): the new
`linux_arm64_android_aot_engine` builder runs on the **existing
linux-x64 bot pool** and cross-compiles the host `gen_snapshot` binary
to linux-arm64. No new arm64 bots, no dependency on the
`flutter/android/sdk/all` CIPD package getting a linux-arm64 platform
build.

Engine code changes (+20 / -7 across 6 files):

- DEPS: drop `host_cpu == "arm64"` clause on the linux-arm64 clang CIPD
  download so x64 builders also fetch it (the clang binary is never
  executed; cross-compile happens via `--target=aarch64-linux-gnu`
  using the x64 clang).

- engine/src/build/config/BUILDCONFIG.gn: introduce `host_clang_cpu`,
  a new gn arg defaulting to `host_cpu`. Represents the CPU of
  build-time tools that run on the builder, distinct from `host_cpu`
  (the CPU the produced host binary runs on). No-op for native builds.

- engine/src/build/toolchain/linux/BUILD.gn: clang binary path is now
  keyed on `host_clang_cpu` instead of `host_cpu`, so the executable
  used during the build stays the x64 clang even when producing arm64
  output.

- engine/src/flutter/common/config.gni: the prebuilt Dart SDK path for
  build-time tooling (frontend server, kernel-to-AOT-snapshot) follows
  `host_clang_cpu`, so it resolves to the x64 dart-sdk on the builder.

- engine/src/flutter/shell/platform/android/BUILD.gn: zip names use
  `linux-\$host_cpu.zip` interpolation (matching what `flutter precache`
  looks for on arm64 hosts), and the gen_snapshot zip's deps now mirror
  what analyze_snapshot already does -- depend only on the host binary
  instead of `generate_snapshot_bins`, which runs gen_snapshot during
  the build (impossible when cross-compiled to aarch64).

- engine/src/flutter/tools/gn: add a `--host-cpu` flag, apply it in
  both `is_host_build()` and the cross-target (`else`) branches (the
  original PR only set host_cpu in the `is_host_build` branch, which is
  why `--host-cpu=arm64 --android` silently produced x86_64), and set
  `host_clang_cpu` to the build-machine CPU only when cross-compiling.

Plus CI plumbing for this reland:

- engine/src/flutter/ci/builders/linux_arm64_android_aot_engine.json:
  new builder config based on the reverted PR's version with four
  deltas: drone_dimensions drop `cpu=arm64` (runs on x64 pool), gn args
  add `--host-cpu arm64`, gclient_variables add
  `download_android_deps: false` and `download_jdk: false`, and ninja
  targets are the explicit `zip_archives/.../linux-arm64.zip` paths
  rather than the `flutter/shell/platform/android:gen_snapshot`
  umbrella target.

- engine/src/flutter/.ci.yaml: matching builder entry, same shape as
  the reverted PR's entry, `cpu=arm64` removed from drone_dimensions,
  `bringup: true`.

The framework changes from the reverted PR (flutter_tools cache /
artifact-path resolution) are intentionally NOT included here. They
land in a follow-up PR once the new builder has produced
`linux-arm64.zip` to GCS for at least one release. That sequencing is
the direct fix for the `flutter precache --all` 404 that triggered the
original revert: artifacts before manifest.

Empirically validated on native linux-x64 hardware (Debian 12 Docker,
real Intel x86_64, no QEMU):

- Original validation 2026-05-29 against `c7f49fe0` produced
  `linux-arm64.zip` (1.77 MB) and `analyze-snapshot-linux-arm64.zip`
  (2.8 MB).

- Fresh re-validation 2026-06-04 against `14c05c42` (then-current
  `main`): patches apply cleanly, `ninja clang_arm64/gen_snapshot`
  exits 0 in 6m37s `-j8`, output binary is ELF aarch64 PIE,
  glibc-linked, not Android bionic. Binary SHA256:
  898413a122ef33d42044fb0a2aa3259803f051ef9063b4d8ddd5e4eb7fa49cd0.

Runtime smoke test on real arm64 hardware (Graviton / arm64 VM / etc.)
is still owed before flipping `bringup: true` to false. QEMU user-mode
segfaults on gen_snapshot -- a known qemu limitation with executable
mmap / TLS-heavy runtimes, not a binary defect.

Fixes flutter#75864
Fixes flutter#168980
Two style/readability fixes flagged on flutter#187591:

1. Factor out the assignments that are identical in both branches of the
   `is_host_build(args)` check (`host_cpu`, `target_cpu`,
   `dart_target_arch`). The if/else now only differs in how it sets
   `host_os` and `target_os`. Behavior is unchanged --
   `get_target_cpu(args)` reads `args.target_os`, not the partially-built
   `gn_args` dict, so ordering doesn't matter.

2. Wrap the new `--host-cpu` argparse help string to fit the
   80-character line limit from the Google Python Style Guide
   (referenced by Flutter's `.gemini/styleguide.md`).

No functional change.
Per @gaaclarke's review feedback on flutter#187591 (BUILDCONFIG.gn:154).

builder_cpu is clearer than host_clang_cpu:
- Says what the var ROLE is (the build host's CPU) rather than what it
  IS (the CPU clang runs on).
- Generic -- covers both the clang binary AND the build-time Dart SDK,
  not just clang.
- Maps to existing CI vocabulary (LUCI "builder", "bot").
- Decouples the name from an implementation detail.

Pure mechanical rename across 4 files. No behavioral change. Updated
the declare_args() doc comment in BUILDCONFIG.gn to use the new name
and reflect the broader scope.
@dbebawy

dbebawy commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Also rebased the branch onto current upstream/master (7e1e5074032) to clear 16 days of drift before the next round of review. Clean rebase — zero conflicts (our 8 files don't intersect with the upstream changes, mostly Skia/Dart rolls in different DEPS regions). New tip: 1ade06d7c1e, all 3 commits re-signed. Same +417/-11 diff, no content changes.

@dbebawy dbebawy force-pushed the reland-linux-arm64-gen-snapshot branch from 3ab8fab to 1ade06d Compare June 25, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a: desktop Running on desktop engine flutter/engine related. See also e: labels. platform-android Android applications specifically team-android Owned by Android platform team

Projects

None yet

4 participants