Skip to content

Ship gen_snapshot for linux-arm64 hosts targeting Android#182552

Merged
auto-submit[bot] merged 6 commits into
flutter:masterfrom
dbebawy:linux-arm64-gen-snapshot
Apr 29, 2026
Merged

Ship gen_snapshot for linux-arm64 hosts targeting Android#182552
auto-submit[bot] merged 6 commits into
flutter:masterfrom
dbebawy:linux-arm64-gen-snapshot

Conversation

@dbebawy

@dbebawy dbebawy commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Enable Android profile and release builds on ARM64 Linux hosts (AWS Graviton, GCP Tau T2A, self-hosted ARM64 runners, etc.), which currently fail because gen_snapshot is only shipped for linux-x64 hosts.

Supersedes #182275 and #182276 — combined into a single atomic commit per reviewer feedback.

Engine changes

  • BUILD.gn: Make gen_snapshot and analyze_snapshot zip output names host-architecture-aware (linux-arm64.zip vs linux-x64.zip) using explicit if (host_os == "linux" && host_cpu == "arm64") branching, matching the existing pattern for mac/windows.
  • linux_arm64_android_aot_engine.json (new): CI builder config following the secondary builder pattern (matching mac_android_aot_engine.json). Only produces host-specific archives — shared artifacts (artifacts.zip, symbols.zip, embedding/abi jars) continue to be produced by the existing x64 Linux builder.
  • .ci.yaml: New builder entry with bringup: true, os=Linux, cpu=arm64.

Framework changes

  • flutter_cache.dart: Convert _linuxBinaryDirs from a hardcoded linux-x64 const list to a function parameterized by host architecture, using the existing cache.getHostPlatformArchName() pattern from LinuxEngineArtifacts.
  • artifacts.dart: Clean up stale TODO comment — replace with accurate explanation of why darwin-arm64 is remapped to darwin-x64 (universal binary). No Linux remapping needed since Linux ships native binaries per host architecture.
  • Tests: Add tests for both x64 and arm64 host artifact resolution in cache_test.dart and artifacts_test.dart, following the existing LinuxEngineArtifacts test pattern.

Infrastructure notes

  • The builder is set to bringup: true to allow stabilization before becoming a blocking builder.
  • Requires ARM64 Linux machines in the LUCI swarming pool (os=Linux + cpu=arm64).
  • 8 build configurations: {arm, arm64, x64, riscv64} × {profile, release}. This matches the x64 Linux builder's target coverage. The macOS equivalent has 6 builds (no riscv64).
  • Uses RBE (--rbe) for remote compilation.

Fixes #75864
Fixes #168980

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

Test plan

Verified by PR CI (x64 hosts)

  • linux_android_aot_engine (x64) passed — BUILD.gn change still produces linux-x64.zip correctly
  • mac_android_aot_engine passed — darwin-x64.zip unaffected
  • windows_android_aot_engine passed — windows-x64.zip unaffected
  • ci.yaml validation passed — builder JSON and .ci.yaml entry are structurally valid
  • Archive paths are unique across all builder configs (verified locally)
  • Unit tests pass for AndroidGenSnapshotArtifacts.getBinaryDirs() returning linux-x64 entries on x64 hosts and linux-arm64 entries on arm64 hosts
  • Unit tests pass for getArtifactPath resolving gen_snapshot to correct host-architecture path on both x64 and arm64 Linux

Requires ARM64 Linux hardware (post-bringup)

The new linux_arm64_android_aot_engine builder is bringup: true and does not run on PR checks. The BUILD.gn code path (host_cpu == "arm64") is only exercised on ARM64 hosts. The following need to be validated once the builder runs on ARM64 infrastructure:

  • gen_snapshot compiles and links on ARM64 Linux
  • CI builder produces linux-arm64.zip archives for all Android target CPUs (arm, arm64, x64, riscv64) in both profile and release modes
  • analyze_snapshot produces analyze-snapshot-linux-arm64.zip for 64-bit targets
  • Artifacts download correctly on an ARM64 Linux host via flutter precache

@dbebawy dbebawy requested a review from a team as a code owner February 18, 2026 17:53
@github-actions github-actions Bot added platform-android Android applications specifically tool Affects the "flutter" command-line tool. See also t: labels. engine flutter/engine related. See also e: labels. team-android Owned by Android platform team labels Feb 18, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enables building Android profile and release artifacts on ARM64 Linux hosts by making gen_snapshot and analyze_snapshot host-architecture-aware. A new CI builder for linux-arm64 is added, along with corresponding build configurations. The Flutter tool's artifact handling for Android gen_snapshot is updated to correctly resolve paths based on the host architecture on Linux. The changes are well-tested with new unit tests for artifact resolution and cache behavior. The code is clean and the changes are logical.

@dbebawy dbebawy force-pushed the linux-arm64-gen-snapshot branch 2 times, most recently from 2980c04 to c4bd392 Compare February 18, 2026 18:05
@dbebawy

dbebawy commented Feb 18, 2026

Copy link
Copy Markdown
Contributor Author

Tagging reviewers from the original PRs (#182275 and #182276) which were combined here per feedback:

@jtmcdole — Added bringup: true to the .ci.yaml entry as you flagged. Thanks for the catch.
@reidbaker — This is the combined PR with all changes. My detailed response to your maintenance burden question is on #182275. Happy to discuss further here.
@jmagman — Combined engine and framework into a single atomic commit as you suggested. Thanks for the heads up about the monorepo merge.

@dbebawy dbebawy force-pushed the linux-arm64-gen-snapshot branch 2 times, most recently from f216e41 to c907882 Compare March 2, 2026 21:49
@reidbaker

Copy link
Copy Markdown
Contributor

This pr need approval from someone on infra and from someone on the engine team before landing. @jtmcdole is both but may delegate to others for one or both of those roles.

@jtmcdole jtmcdole added the CICD Run CI/CD label Mar 19, 2026
@dbebawy dbebawy force-pushed the linux-arm64-gen-snapshot branch from c907882 to 2bfbb64 Compare March 20, 2026 02:41
@github-actions github-actions Bot removed the CICD Run CI/CD label Mar 20, 2026
dbebawy added 2 commits March 23, 2026 21:22
Build gen_snapshot and analyze_snapshot for ARM64 Linux hosts targeting
Android, enabling profile and release builds on ARM64 Linux machines
(AWS Graviton, GCP Tau T2A, etc.).

Engine changes:
- Make gen_snapshot and analyze_snapshot zip output names in BUILD.gn
  host-architecture-aware (linux-arm64.zip vs linux-x64.zip)
- Add linux_arm64_android_aot_engine.json CI builder config following
  the secondary builder pattern (matching mac_android_aot_engine.json)
- Add .ci.yaml entry for the new ARM64 Linux builder (bringup: true)

The ARM64 builder only produces host-specific archives (linux-arm64.zip
and analyze-snapshot-linux-arm64.zip). Shared artifacts (artifacts.zip,
symbols.zip, embedding/abi jars) continue to be produced by the
existing x64 Linux builder.

Framework changes:
- Convert _linuxBinaryDirs from a const list to a function
  parameterized by host architecture, using the existing
  cache.getHostPlatformArchName() pattern from LinuxEngineArtifacts
- Clean up stale TODO comment in artifacts.dart
- Add tests for both x64 and arm64 host artifact resolution

Fixes flutter#75864
Fixes flutter#168980
- Replace getCurrentHostPlatform() with _operatingSystemUtils.hostPlatform
  in _getAndroidArtifactPath to use constructor-injected dependency instead
  of zone-injected global, matching the pattern used elsewhere in
  CachedArtifacts (lines 500, 538)
- Apply dart format to test files with long test names
@dbebawy dbebawy force-pushed the linux-arm64-gen-snapshot branch from 2bfbb64 to 84c7cd8 Compare March 24, 2026 01:22
@dbebawy

dbebawy commented Mar 24, 2026

Copy link
Copy Markdown
Contributor Author

Hi @jtmcdole @reidbaker — it looks like the LUCI CI suite isn't triggering on this PR. Could someone approve the pipeline or add the CICD label to kick it off? Is there anything else I can do on my end to help move this forward? Thanks!

@reidbaker reidbaker added the CICD Run CI/CD label Mar 24, 2026
@github-actions github-actions Bot removed the CICD Run CI/CD label Mar 24, 2026
@reidbaker reidbaker added the CICD Run CI/CD label Mar 24, 2026
@dbebawy

dbebawy commented May 13, 2026

Copy link
Copy Markdown
Contributor Author

Validated the framework changes on linux/arm64 (Debian 12 via Docker Desktop on Apple Silicon) against master at 3b4ef3e9ec / engine 054314414e. Confirmed f914eaeb is in HEAD.

Framework path — works as intended

  • flutter --version bootstrapped cleanly: Linux arm64 Dart SDK downloaded, tool built, no errors.
  • flutter precache --android now requests linux-arm64.zip paths instead of linux-x64.zip, confirming flutter_cache.dart / artifacts.dart resolve correctly for an arm64 host.

Engine artifacts — not yet on GCS

android-arm-profile/linux-arm64.zip     → 404
android-arm-release/linux-arm64.zip     → 404
android-arm64-profile/linux-arm64.zip   → 404
android-arm64-release/linux-arm64.zip   → 404
android-x64-profile/linux-arm64.zip     → 404
android-x64-release/linux-arm64.zip     → 404
android-arm-profile/linux-x64.zip       → 200  (reference)
android-arm64-release/linux-x64.zip     → 200  (reference)

flutter precache --android fails with Exception: 404 on the first artifact, as expected — linux_arm64_android_aot_engine is still bringup: true and not publishing to flutter_infra_release. This matches the unchecked checklist item in the PR description ("CI builder produces linux-arm64.zip archives… post-bringup").

Context: downstream Flutter app, currently pinning CI runners to docker-amd64 solely because of this gap. Happy to re-run validation once the builder exits bringup.

@eyebrowsoffire eyebrowsoffire added the revert_wf Revert using Github Workflow label May 18, 2026
@flutteractionsbot

Copy link
Copy Markdown
Contributor

A reason for requesting a revert of flutter/flutter/182552 could not be found or the reason was not properly formatted. Begin a comment with 'Reason for revert:' to tell the bot why this issue is being reverted.

@flutteractionsbot flutteractionsbot removed the revert_wf Revert using Github Workflow label May 18, 2026
@eyebrowsoffire

Copy link
Copy Markdown
Contributor

Reason for revert: We do not have any bots capable of servicing this builder, and it is breaking our releases as well as flutter precache --all

@eyebrowsoffire eyebrowsoffire added the revert_wf Revert using Github Workflow label May 18, 2026
@flutteractionsbot

Copy link
Copy Markdown
Contributor

Successfully created revert PR: #186693

@flutteractionsbot flutteractionsbot removed the revert_wf Revert using Github Workflow label May 18, 2026
MarlonJD pushed a commit to MarlonJD/flutter that referenced this pull request May 20, 2026
…lutter#186693)

Reverts: [Ship gen_snapshot for linux-arm64 hosts targeting
Android](flutter#182552)

Initiated by: @eyebrowsoffire

Reason for reverting: We do not have any bots capable of servicing this
builder, and it is breaking our releases as well as `flutter precache
--all`

Original PR Author: @dbebawy

Reviewed By: @reidbaker

The original PR description is provided below:

## Summary

Enable Android **profile** and **release** builds on ARM64 Linux hosts
(AWS Graviton, GCP Tau T2A, self-hosted ARM64 runners, etc.), which
currently fail because `gen_snapshot` is only shipped for `linux-x64`
hosts.

Supersedes flutter#182275 and flutter#182276 — combined into a single atomic commit
per reviewer feedback.

### Engine changes

- **`BUILD.gn`**: Make `gen_snapshot` and `analyze_snapshot` zip output
names host-architecture-aware (`linux-arm64.zip` vs `linux-x64.zip`)
using explicit `if (host_os == "linux" && host_cpu == "arm64")`
branching, matching the existing pattern for mac/windows.
- **`linux_arm64_android_aot_engine.json`** (new): CI builder config
following the secondary builder pattern (matching
`mac_android_aot_engine.json`). Only produces host-specific archives —
shared artifacts (`artifacts.zip`, `symbols.zip`, embedding/abi jars)
continue to be produced by the existing x64 Linux builder.
- **`.ci.yaml`**: New builder entry with `bringup: true`, `os=Linux`,
`cpu=arm64`.

### Framework changes

- **`flutter_cache.dart`**: Convert `_linuxBinaryDirs` from a hardcoded
`linux-x64` const list to a function parameterized by host architecture,
using the existing `cache.getHostPlatformArchName()` pattern from
`LinuxEngineArtifacts`.
- **`artifacts.dart`**: Clean up stale TODO comment — replace with
accurate explanation of why `darwin-arm64` is remapped to `darwin-x64`
(universal binary). No Linux remapping needed since Linux ships native
binaries per host architecture.
- **Tests**: Add tests for both x64 and arm64 host artifact resolution
in `cache_test.dart` and `artifacts_test.dart`, following the existing
`LinuxEngineArtifacts` test pattern.

### Infrastructure notes

- The builder is set to `bringup: true` to allow stabilization before
becoming a blocking builder.
- Requires ARM64 Linux machines in the LUCI swarming pool (`os=Linux` +
`cpu=arm64`).
- 8 build configurations: {arm, arm64, x64, riscv64} × {profile,
release}. This matches the x64 Linux builder's target coverage. The
macOS equivalent has 6 builds (no riscv64).
- Uses RBE (`--rbe`) for remote compilation.

Fixes flutter#75864
Fixes flutter#168980

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

## Test plan

### Verified by PR CI (x64 hosts)

- [x] `linux_android_aot_engine` (x64) passed — `BUILD.gn` change still
produces `linux-x64.zip` correctly
- [x] `mac_android_aot_engine` passed — `darwin-x64.zip` unaffected
- [x] `windows_android_aot_engine` passed — `windows-x64.zip` unaffected
- [x] `ci.yaml validation` passed — builder JSON and `.ci.yaml` entry
are structurally valid
- [x] Archive paths are unique across all builder configs (verified
locally)
- [x] Unit tests pass for `AndroidGenSnapshotArtifacts.getBinaryDirs()`
returning `linux-x64` entries on x64 hosts and `linux-arm64` entries on
arm64 hosts
- [x] Unit tests pass for `getArtifactPath` resolving `gen_snapshot` to
correct host-architecture path on both x64 and arm64 Linux

### Requires ARM64 Linux hardware (post-bringup)

The new `linux_arm64_android_aot_engine` builder is `bringup: true` and
does not run on PR checks. The `BUILD.gn` code path (`host_cpu ==
"arm64"`) is only exercised on ARM64 hosts. The following need to be
validated once the builder runs on ARM64 infrastructure:

- [ ] `gen_snapshot` compiles and links on ARM64 Linux
- [ ] CI builder produces `linux-arm64.zip` archives for all Android
target CPUs (arm, arm64, x64, riscv64) in both profile and release modes
- [ ] `analyze_snapshot` produces `analyze-snapshot-linux-arm64.zip` for
64-bit targets
- [ ] Artifacts download correctly on an ARM64 Linux host via `flutter
precache`

---------

Co-authored-by: Jackson Gardner <jacksongardner@google.com>
@jtmcdole

Copy link
Copy Markdown
Member

We need to see if we can bypass the linux_arm64 tooling requirement since we don't have official binaries in cipd.

eyebrowsoffire added a commit to eyebrowsoffire/flutter that referenced this pull request May 20, 2026
…lutter#186693)

Reverts: [Ship gen_snapshot for linux-arm64 hosts targeting
Android](flutter#182552)

Initiated by: @eyebrowsoffire

Reason for reverting: We do not have any bots capable of servicing this
builder, and it is breaking our releases as well as `flutter precache
--all`

Original PR Author: @dbebawy

Reviewed By: @reidbaker

The original PR description is provided below:

## Summary

Enable Android **profile** and **release** builds on ARM64 Linux hosts
(AWS Graviton, GCP Tau T2A, self-hosted ARM64 runners, etc.), which
currently fail because `gen_snapshot` is only shipped for `linux-x64`
hosts.

Supersedes flutter#182275 and flutter#182276 — combined into a single atomic commit
per reviewer feedback.

### Engine changes

- **`BUILD.gn`**: Make `gen_snapshot` and `analyze_snapshot` zip output
names host-architecture-aware (`linux-arm64.zip` vs `linux-x64.zip`)
using explicit `if (host_os == "linux" && host_cpu == "arm64")`
branching, matching the existing pattern for mac/windows.
- **`linux_arm64_android_aot_engine.json`** (new): CI builder config
following the secondary builder pattern (matching
`mac_android_aot_engine.json`). Only produces host-specific archives —
shared artifacts (`artifacts.zip`, `symbols.zip`, embedding/abi jars)
continue to be produced by the existing x64 Linux builder.
- **`.ci.yaml`**: New builder entry with `bringup: true`, `os=Linux`,
`cpu=arm64`.

### Framework changes

- **`flutter_cache.dart`**: Convert `_linuxBinaryDirs` from a hardcoded
`linux-x64` const list to a function parameterized by host architecture,
using the existing `cache.getHostPlatformArchName()` pattern from
`LinuxEngineArtifacts`.
- **`artifacts.dart`**: Clean up stale TODO comment — replace with
accurate explanation of why `darwin-arm64` is remapped to `darwin-x64`
(universal binary). No Linux remapping needed since Linux ships native
binaries per host architecture.
- **Tests**: Add tests for both x64 and arm64 host artifact resolution
in `cache_test.dart` and `artifacts_test.dart`, following the existing
`LinuxEngineArtifacts` test pattern.

### Infrastructure notes

- The builder is set to `bringup: true` to allow stabilization before
becoming a blocking builder.
- Requires ARM64 Linux machines in the LUCI swarming pool (`os=Linux` +
`cpu=arm64`).
- 8 build configurations: {arm, arm64, x64, riscv64} × {profile,
release}. This matches the x64 Linux builder's target coverage. The
macOS equivalent has 6 builds (no riscv64).
- Uses RBE (`--rbe`) for remote compilation.

Fixes flutter#75864
Fixes flutter#168980

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

## Test plan

### Verified by PR CI (x64 hosts)

- [x] `linux_android_aot_engine` (x64) passed — `BUILD.gn` change still
produces `linux-x64.zip` correctly
- [x] `mac_android_aot_engine` passed — `darwin-x64.zip` unaffected
- [x] `windows_android_aot_engine` passed — `windows-x64.zip` unaffected
- [x] `ci.yaml validation` passed — builder JSON and `.ci.yaml` entry
are structurally valid
- [x] Archive paths are unique across all builder configs (verified
locally)
- [x] Unit tests pass for `AndroidGenSnapshotArtifacts.getBinaryDirs()`
returning `linux-x64` entries on x64 hosts and `linux-arm64` entries on
arm64 hosts
- [x] Unit tests pass for `getArtifactPath` resolving `gen_snapshot` to
correct host-architecture path on both x64 and arm64 Linux

### Requires ARM64 Linux hardware (post-bringup)

The new `linux_arm64_android_aot_engine` builder is `bringup: true` and
does not run on PR checks. The `BUILD.gn` code path (`host_cpu ==
"arm64"`) is only exercised on ARM64 hosts. The following need to be
validated once the builder runs on ARM64 infrastructure:

- [ ] `gen_snapshot` compiles and links on ARM64 Linux
- [ ] CI builder produces `linux-arm64.zip` archives for all Android
target CPUs (arm, arm64, x64, riscv64) in both profile and release modes
- [ ] `analyze_snapshot` produces `analyze-snapshot-linux-arm64.zip` for
64-bit targets
- [ ] Artifacts download correctly on an ARM64 Linux host via `flutter
precache`

---------

Co-authored-by: Jackson Gardner <jacksongardner@google.com>
auto-submit Bot pushed a commit that referenced this pull request May 21, 2026
…roid" (#186818)

This is a cherry-pick of this revert, which is needed because the change that was reverted caused a breakage in our release build.

Reverts: [Ship gen_snapshot for linux-arm64 hosts targeting Android](#182552)

Initiated by: @eyebrowsoffire

Reason for reverting: We do not have any bots capable of servicing this builder, and it is breaking our releases as well as `flutter precache --all`

Original PR Author: @dbebawy

Reviewed By: @reidbaker

The original PR description is provided below:

## Summary

Enable Android **profile** and **release** builds on ARM64 Linux hosts (AWS Graviton, GCP Tau T2A, self-hosted ARM64 runners, etc.), which currently fail because `gen_snapshot` is only shipped for `linux-x64` hosts.

Supersedes #182275 and #182276 — combined into a single atomic commit per reviewer feedback.

### Engine changes

- **`BUILD.gn`**: Make `gen_snapshot` and `analyze_snapshot` zip output names host-architecture-aware (`linux-arm64.zip` vs `linux-x64.zip`) using explicit `if (host_os == "linux" && host_cpu == "arm64")` branching, matching the existing pattern for mac/windows.
- **`linux_arm64_android_aot_engine.json`** (new): CI builder config following the secondary builder pattern (matching
`mac_android_aot_engine.json`). Only produces host-specific archives — shared artifacts (`artifacts.zip`, `symbols.zip`, embedding/abi jars) continue to be produced by the existing x64 Linux builder.
- **`.ci.yaml`**: New builder entry with `bringup: true`, `os=Linux`, `cpu=arm64`.

### Framework changes

- **`flutter_cache.dart`**: Convert `_linuxBinaryDirs` from a hardcoded `linux-x64` const list to a function parameterized by host architecture, using the existing `cache.getHostPlatformArchName()` pattern from `LinuxEngineArtifacts`.
- **`artifacts.dart`**: Clean up stale TODO comment — replace with accurate explanation of why `darwin-arm64` is remapped to `darwin-x64` (universal binary). No Linux remapping needed since Linux ships native binaries per host architecture.
- **Tests**: Add tests for both x64 and arm64 host artifact resolution in `cache_test.dart` and `artifacts_test.dart`, following the existing `LinuxEngineArtifacts` test pattern.

### Infrastructure notes

- The builder is set to `bringup: true` to allow stabilization before becoming a blocking builder.
- Requires ARM64 Linux machines in the LUCI swarming pool (`os=Linux` + `cpu=arm64`).
- 8 build configurations: {arm, arm64, x64, riscv64} × {profile, release}. This matches the x64 Linux builder's target coverage. The macOS equivalent has 6 builds (no riscv64).
- Uses RBE (`--rbe`) for remote compilation.

Fixes #75864 Fixes #168980

https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

## Test plan

### Verified by PR CI (x64 hosts)

### Requires ARM64 Linux hardware (post-bringup)

The new `linux_arm64_android_aot_engine` builder is `bringup: true` and does not run on PR checks. The `BUILD.gn` code path (`host_cpu == "arm64"`) is only exercised on ARM64 hosts. The following need to be validated once the builder runs on ARM64 infrastructure:

---------

<!--
Thanks for filing a pull request!
Reviewers are typically assigned within a week of filing a request.
To learn more about code review, see our documentation on Tree Hygiene: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
-->

*Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.*

*List which issues are fixed by this PR. You must list at least one issue. An issue is not required if the PR fixes something trivial like a typo.*

*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*

If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.

**Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.
matthewhendrix pushed a commit to matthewhendrix/flutter that referenced this pull request May 23, 2026
…lutter#186693)

Reverts: [Ship gen_snapshot for linux-arm64 hosts targeting
Android](flutter#182552)

Initiated by: @eyebrowsoffire

Reason for reverting: We do not have any bots capable of servicing this
builder, and it is breaking our releases as well as `flutter precache
--all`

Original PR Author: @dbebawy

Reviewed By: @reidbaker

The original PR description is provided below:

## Summary

Enable Android **profile** and **release** builds on ARM64 Linux hosts
(AWS Graviton, GCP Tau T2A, self-hosted ARM64 runners, etc.), which
currently fail because `gen_snapshot` is only shipped for `linux-x64`
hosts.

Supersedes flutter#182275 and flutter#182276 — combined into a single atomic commit
per reviewer feedback.

### Engine changes

- **`BUILD.gn`**: Make `gen_snapshot` and `analyze_snapshot` zip output
names host-architecture-aware (`linux-arm64.zip` vs `linux-x64.zip`)
using explicit `if (host_os == "linux" && host_cpu == "arm64")`
branching, matching the existing pattern for mac/windows.
- **`linux_arm64_android_aot_engine.json`** (new): CI builder config
following the secondary builder pattern (matching
`mac_android_aot_engine.json`). Only produces host-specific archives —
shared artifacts (`artifacts.zip`, `symbols.zip`, embedding/abi jars)
continue to be produced by the existing x64 Linux builder.
- **`.ci.yaml`**: New builder entry with `bringup: true`, `os=Linux`,
`cpu=arm64`.

### Framework changes

- **`flutter_cache.dart`**: Convert `_linuxBinaryDirs` from a hardcoded
`linux-x64` const list to a function parameterized by host architecture,
using the existing `cache.getHostPlatformArchName()` pattern from
`LinuxEngineArtifacts`.
- **`artifacts.dart`**: Clean up stale TODO comment — replace with
accurate explanation of why `darwin-arm64` is remapped to `darwin-x64`
(universal binary). No Linux remapping needed since Linux ships native
binaries per host architecture.
- **Tests**: Add tests for both x64 and arm64 host artifact resolution
in `cache_test.dart` and `artifacts_test.dart`, following the existing
`LinuxEngineArtifacts` test pattern.

### Infrastructure notes

- The builder is set to `bringup: true` to allow stabilization before
becoming a blocking builder.
- Requires ARM64 Linux machines in the LUCI swarming pool (`os=Linux` +
`cpu=arm64`).
- 8 build configurations: {arm, arm64, x64, riscv64} × {profile,
release}. This matches the x64 Linux builder's target coverage. The
macOS equivalent has 6 builds (no riscv64).
- Uses RBE (`--rbe`) for remote compilation.

Fixes flutter#75864
Fixes flutter#168980

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

## Test plan

### Verified by PR CI (x64 hosts)

- [x] `linux_android_aot_engine` (x64) passed — `BUILD.gn` change still
produces `linux-x64.zip` correctly
- [x] `mac_android_aot_engine` passed — `darwin-x64.zip` unaffected
- [x] `windows_android_aot_engine` passed — `windows-x64.zip` unaffected
- [x] `ci.yaml validation` passed — builder JSON and `.ci.yaml` entry
are structurally valid
- [x] Archive paths are unique across all builder configs (verified
locally)
- [x] Unit tests pass for `AndroidGenSnapshotArtifacts.getBinaryDirs()`
returning `linux-x64` entries on x64 hosts and `linux-arm64` entries on
arm64 hosts
- [x] Unit tests pass for `getArtifactPath` resolving `gen_snapshot` to
correct host-architecture path on both x64 and arm64 Linux

### Requires ARM64 Linux hardware (post-bringup)

The new `linux_arm64_android_aot_engine` builder is `bringup: true` and
does not run on PR checks. The `BUILD.gn` code path (`host_cpu ==
"arm64"`) is only exercised on ARM64 hosts. The following need to be
validated once the builder runs on ARM64 infrastructure:

- [ ] `gen_snapshot` compiles and links on ARM64 Linux
- [ ] CI builder produces `linux-arm64.zip` archives for all Android
target CPUs (arm, arm64, x64, riscv64) in both profile and release modes
- [ ] `analyze_snapshot` produces `analyze-snapshot-linux-arm64.zip` for
64-bit targets
- [ ] Artifacts download correctly on an ARM64 Linux host via `flutter
precache`

---------

Co-authored-by: Jackson Gardner <jacksongardner@google.com>
dhananjay6561 pushed a commit to dhananjay6561/flutter that referenced this pull request May 26, 2026
…s on linux-arm64 (flutter#186917)

flutter#182552 added support in the
Flutter tools for using a version of gen_snapshot that runs on
linux-arm64 hosts and targets Android devices.

However, the engine build infrastructure is not currently set up to
build this kind of gen_snapshot binary.

This PR contains changes needed to build engine artifacts on linux-arm64
machines. With this PR a linux-arm64 build environment can:
* run "gclient sync"
* build targets like zip_archives/android-arm64-profile/linux-arm64.zip
that were added in flutter#182552
@dbebawy

dbebawy commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

Reland PR is open: #187591. Runtime smoke on real arm64 Linux included. Framework follow-up will land after artifacts ship to GCS.


Reland update for anyone following this thread.

@jtmcdole — taking the direction you laid out on 2026-05-20 (bypass the linux_arm64 tooling requirement since we don't have official binaries in cipd). The reland uses the existing linux-x64 bot pool and cross-compiles the host gen_snapshot to linux-arm64. No new bots, no dependency on the Android SDK CIPD package getting a linux-arm64 platform build.

Validated empirically on native linux-x64 hardware, twice:

  • 2026-05-29 against c7f49fe0: full build + zip artifact produced (linux-arm64.zip 1.77 MB, the exact artifact whose absence triggered the revert).
  • 2026-06-04 fresh re-validation against 14c05c42 (current main at the time): patches apply cleanly, ninja clang_arm64/gen_snapshot rc=0 in 6 m 37 s -j8 (1212 targets).

Output binary on both runs: ELF 64-bit LSB pie executable, ARM aarch64, ... interpreter /lib/ld-linux-aarch64.so.1, stripped (glibc, not Android bionic). download_android_deps: false + download_jdk: false honored — no Android SDK/NDK fetched.

Engine change is a ~6-file diff (+20/−7) introducing a host_clang_cpu var (= the CPU of build-time tools on the builder) distinct from host_cpu (= the CPU the produced host binary runs on). They're equal for native builds, diverge only with --host-cpu.

The new linux_arm64_android_aot_engine.json builder runs on existing os=Linux bots and passes --host-cpu=arm64. bringup: true until a real-arm64 runtime smoke (qemu-user emulation segfaults on gen_snapshot — known qemu limitation with TLS/executable mmap, not a binary defect).

Framework changes from the original PR are intentionally held back until the builder has produced linux-arm64.zip to GCS for ≥1 release. That sequencing is the direct fix for what broke flutter precache --all last time.

dbebawy added a commit to dbebawy/flutter that referenced this pull request Jun 25, 2026
Relands flutter#182552, reverted in flutter#186693 because the linux-arm64 builder
introduced in that PR had no bots in the LUCI pool and `flutter precache
--all` started 404'ing on missing `linux-arm64.zip` artifacts after the
beta cut.

This reland takes the direction @jtmcdole laid out on the original PR
on 2026-05-20 ("bypass the linux_arm64 tooling requirement since we
don't have official binaries in cipd"): the new
`linux_arm64_android_aot_engine` builder runs on the **existing
linux-x64 bot pool** and cross-compiles the host `gen_snapshot` binary
to linux-arm64. No new arm64 bots, no dependency on the
`flutter/android/sdk/all` CIPD package getting a linux-arm64 platform
build.

Engine code changes (+20 / -7 across 6 files):

- DEPS: drop `host_cpu == "arm64"` clause on the linux-arm64 clang CIPD
  download so x64 builders also fetch it (the clang binary is never
  executed; cross-compile happens via `--target=aarch64-linux-gnu`
  using the x64 clang).

- engine/src/build/config/BUILDCONFIG.gn: introduce `host_clang_cpu`,
  a new gn arg defaulting to `host_cpu`. Represents the CPU of
  build-time tools that run on the builder, distinct from `host_cpu`
  (the CPU the produced host binary runs on). No-op for native builds.

- engine/src/build/toolchain/linux/BUILD.gn: clang binary path is now
  keyed on `host_clang_cpu` instead of `host_cpu`, so the executable
  used during the build stays the x64 clang even when producing arm64
  output.

- engine/src/flutter/common/config.gni: the prebuilt Dart SDK path for
  build-time tooling (frontend server, kernel-to-AOT-snapshot) follows
  `host_clang_cpu`, so it resolves to the x64 dart-sdk on the builder.

- engine/src/flutter/shell/platform/android/BUILD.gn: zip names use
  `linux-\$host_cpu.zip` interpolation (matching what `flutter precache`
  looks for on arm64 hosts), and the gen_snapshot zip's deps now mirror
  what analyze_snapshot already does -- depend only on the host binary
  instead of `generate_snapshot_bins`, which runs gen_snapshot during
  the build (impossible when cross-compiled to aarch64).

- engine/src/flutter/tools/gn: add a `--host-cpu` flag, apply it in
  both `is_host_build()` and the cross-target (`else`) branches (the
  original PR only set host_cpu in the `is_host_build` branch, which is
  why `--host-cpu=arm64 --android` silently produced x86_64), and set
  `host_clang_cpu` to the build-machine CPU only when cross-compiling.

Plus CI plumbing for this reland:

- engine/src/flutter/ci/builders/linux_arm64_android_aot_engine.json:
  new builder config based on the reverted PR's version with four
  deltas: drone_dimensions drop `cpu=arm64` (runs on x64 pool), gn args
  add `--host-cpu arm64`, gclient_variables add
  `download_android_deps: false` and `download_jdk: false`, and ninja
  targets are the explicit `zip_archives/.../linux-arm64.zip` paths
  rather than the `flutter/shell/platform/android:gen_snapshot`
  umbrella target.

- engine/src/flutter/.ci.yaml: matching builder entry, same shape as
  the reverted PR's entry, `cpu=arm64` removed from drone_dimensions,
  `bringup: true`.

The framework changes from the reverted PR (flutter_tools cache /
artifact-path resolution) are intentionally NOT included here. They
land in a follow-up PR once the new builder has produced
`linux-arm64.zip` to GCS for at least one release. That sequencing is
the direct fix for the `flutter precache --all` 404 that triggered the
original revert: artifacts before manifest.

Empirically validated on native linux-x64 hardware (Debian 12 Docker,
real Intel x86_64, no QEMU):

- Original validation 2026-05-29 against `c7f49fe0` produced
  `linux-arm64.zip` (1.77 MB) and `analyze-snapshot-linux-arm64.zip`
  (2.8 MB).

- Fresh re-validation 2026-06-04 against `14c05c42` (then-current
  `main`): patches apply cleanly, `ninja clang_arm64/gen_snapshot`
  exits 0 in 6m37s `-j8`, output binary is ELF aarch64 PIE,
  glibc-linked, not Android bionic. Binary SHA256:
  898413a122ef33d42044fb0a2aa3259803f051ef9063b4d8ddd5e4eb7fa49cd0.

Runtime smoke test on real arm64 hardware (Graviton / arm64 VM / etc.)
is still owed before flipping `bringup: true` to false. QEMU user-mode
segfaults on gen_snapshot -- a known qemu limitation with executable
mmap / TLS-heavy runtimes, not a binary defect.

Fixes flutter#75864
Fixes flutter#168980
via-guy pushed a commit to via-guy/flutter that referenced this pull request Jun 26, 2026
…lutter#186693)

Reverts: [Ship gen_snapshot for linux-arm64 hosts targeting
Android](flutter#182552)

Initiated by: @eyebrowsoffire

Reason for reverting: We do not have any bots capable of servicing this
builder, and it is breaking our releases as well as `flutter precache
--all`

Original PR Author: @dbebawy

Reviewed By: @reidbaker

The original PR description is provided below:

## Summary

Enable Android **profile** and **release** builds on ARM64 Linux hosts
(AWS Graviton, GCP Tau T2A, self-hosted ARM64 runners, etc.), which
currently fail because `gen_snapshot` is only shipped for `linux-x64`
hosts.

Supersedes flutter#182275 and flutter#182276 — combined into a single atomic commit
per reviewer feedback.

### Engine changes

- **`BUILD.gn`**: Make `gen_snapshot` and `analyze_snapshot` zip output
names host-architecture-aware (`linux-arm64.zip` vs `linux-x64.zip`)
using explicit `if (host_os == "linux" && host_cpu == "arm64")`
branching, matching the existing pattern for mac/windows.
- **`linux_arm64_android_aot_engine.json`** (new): CI builder config
following the secondary builder pattern (matching
`mac_android_aot_engine.json`). Only produces host-specific archives —
shared artifacts (`artifacts.zip`, `symbols.zip`, embedding/abi jars)
continue to be produced by the existing x64 Linux builder.
- **`.ci.yaml`**: New builder entry with `bringup: true`, `os=Linux`,
`cpu=arm64`.

### Framework changes

- **`flutter_cache.dart`**: Convert `_linuxBinaryDirs` from a hardcoded
`linux-x64` const list to a function parameterized by host architecture,
using the existing `cache.getHostPlatformArchName()` pattern from
`LinuxEngineArtifacts`.
- **`artifacts.dart`**: Clean up stale TODO comment — replace with
accurate explanation of why `darwin-arm64` is remapped to `darwin-x64`
(universal binary). No Linux remapping needed since Linux ships native
binaries per host architecture.
- **Tests**: Add tests for both x64 and arm64 host artifact resolution
in `cache_test.dart` and `artifacts_test.dart`, following the existing
`LinuxEngineArtifacts` test pattern.

### Infrastructure notes

- The builder is set to `bringup: true` to allow stabilization before
becoming a blocking builder.
- Requires ARM64 Linux machines in the LUCI swarming pool (`os=Linux` +
`cpu=arm64`).
- 8 build configurations: {arm, arm64, x64, riscv64} × {profile,
release}. This matches the x64 Linux builder's target coverage. The
macOS equivalent has 6 builds (no riscv64).
- Uses RBE (`--rbe`) for remote compilation.

Fixes flutter#75864
Fixes flutter#168980

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

## Test plan

### Verified by PR CI (x64 hosts)

- [x] `linux_android_aot_engine` (x64) passed — `BUILD.gn` change still
produces `linux-x64.zip` correctly
- [x] `mac_android_aot_engine` passed — `darwin-x64.zip` unaffected
- [x] `windows_android_aot_engine` passed — `windows-x64.zip` unaffected
- [x] `ci.yaml validation` passed — builder JSON and `.ci.yaml` entry
are structurally valid
- [x] Archive paths are unique across all builder configs (verified
locally)
- [x] Unit tests pass for `AndroidGenSnapshotArtifacts.getBinaryDirs()`
returning `linux-x64` entries on x64 hosts and `linux-arm64` entries on
arm64 hosts
- [x] Unit tests pass for `getArtifactPath` resolving `gen_snapshot` to
correct host-architecture path on both x64 and arm64 Linux

### Requires ARM64 Linux hardware (post-bringup)

The new `linux_arm64_android_aot_engine` builder is `bringup: true` and
does not run on PR checks. The `BUILD.gn` code path (`host_cpu ==
"arm64"`) is only exercised on ARM64 hosts. The following need to be
validated once the builder runs on ARM64 infrastructure:

- [ ] `gen_snapshot` compiles and links on ARM64 Linux
- [ ] CI builder produces `linux-arm64.zip` archives for all Android
target CPUs (arm, arm64, x64, riscv64) in both profile and release modes
- [ ] `analyze_snapshot` produces `analyze-snapshot-linux-arm64.zip` for
64-bit targets
- [ ] Artifacts download correctly on an ARM64 Linux host via `flutter
precache`

---------

Co-authored-by: Jackson Gardner <jacksongardner@google.com>
via-guy pushed a commit to via-guy/flutter that referenced this pull request Jun 26, 2026
…s on linux-arm64 (flutter#186917)

flutter#182552 added support in the
Flutter tools for using a version of gen_snapshot that runs on
linux-arm64 hosts and targets Android devices.

However, the engine build infrastructure is not currently set up to
build this kind of gen_snapshot binary.

This PR contains changes needed to build engine artifacts on linux-arm64
machines. With this PR a linux-arm64 build environment can:
* run "gclient sync"
* build targets like zip_archives/android-arm64-profile/linux-arm64.zip
that were added in flutter#182552
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD engine flutter/engine related. See also e: labels. platform-android Android applications specifically team-android Owned by Android platform team tool Affects the "flutter" command-line tool. See also t: labels. will affect goldens Changes to golden files

Projects

None yet

8 participants