Enable Swift testing in the iOS embedder#185712
Conversation
Added a comment to the iOS |
|
|
||
| # Version of iOS that we're targeting for tests. | ||
| ios_testing_deployment_target = "13.0" | ||
| ios_testing_deployment_target = "15.6" |
There was a problem hiding this comment.
Can you add a comment explaining why 15.6 specifically? It seems like a really random version otherwise lol
vashworth
left a comment
There was a problem hiding this comment.
One last comment request, then LGTM!
|
I compiled again and it seems the tests are not crashing on 15.0 (I did test 15.0 and it crashed before). I'm going to change it to 15.0 and see if it works on CI. |
|
An existing Git SHA, To re-trigger presubmits after closing or re-opeing a PR, or pushing a HEAD commit (i.e. with |
flutter/flutter@259aeae...e03b91f 2026-05-20 engine-flutter-autoroll@skia.org Roll Packages from ade10ca to 1dfbada (6 revisions) (flutter/flutter#186811) 2026-05-20 brunocorona.alcantar@gmail.com Fix AnimatedList.separated assert when removing last item mid-removal… (flutter/flutter#186389) 2026-05-20 engine-flutter-autoroll@skia.org Roll Skia from d45969a5752e to 5f4f454b9662 (2 revisions) (flutter/flutter#186809) 2026-05-20 engine-flutter-autoroll@skia.org Roll Fuchsia Linux SDK from -F9Ci3Opxt06MixDl... to iKCvaD58jKStYGla0... (flutter/flutter#186796) 2026-05-20 engine-flutter-autoroll@skia.org Roll Skia from 19ad9707e5c6 to d45969a5752e (2 revisions) (flutter/flutter#186792) 2026-05-20 engine-flutter-autoroll@skia.org Roll Skia from 3471ebf5af0c to 19ad9707e5c6 (9 revisions) (flutter/flutter#186772) 2026-05-20 mdebbar@google.com [web] Refactor webparagraph painters to separate CK properly (flutter/flutter#186684) 2026-05-19 31859944+LongCatIsLooong@users.noreply.github.com Enable Swift testing in the iOS embedder (flutter/flutter#185712) 2026-05-19 mdebbar@google.com [web] Rename WebParagraph goldens (flutter/flutter#186680) 2026-05-19 engine-flutter-autoroll@skia.org Roll Skia from f1b406860c5e to 3471ebf5af0c (5 revisions) (flutter/flutter#186745) 2026-05-19 154381524+flutteractionsbot@users.noreply.github.com Revert "Ship gen_snapshot for linux-arm64 hosts targeting Android" (flutter/flutter#186693) 2026-05-19 bkonyi@google.com [ Tool ] Remove legacy analytics code (flutter/flutter#184994) 2026-05-19 chingjun@google.com Update Vulkan enum values (flutter/flutter#186694) 2026-05-19 1961493+harryterkelsen@users.noreply.github.com fix(web): Fixes CSS override detection when the browser has a default font size (flutter/flutter#186474) 2026-05-19 30870216+gaaclarke@users.noreply.github.com adds linux impeller hello world integration test (flutter/flutter#186715) 2026-05-19 jason-simmons@users.noreply.github.com Update the list of binaries in the code signing verification test to include new Dart snapshots (flutter/flutter#186754) 2026-05-19 brunocorona.alcantar@gmail.com Make EdgeDraggingAutoScroller respect ScrollPhysics (flutter/flutter#186541) 2026-05-19 bkonyi@google.com [ Widget Preview ] Fix inspector split resize focus loss over WebViews (flutter/flutter#186432) 2026-05-19 engine-flutter-autoroll@skia.org Roll Packages from b9bdd37 to ade10ca (1 revision) (flutter/flutter#186746) 2026-05-19 jason-simmons@users.noreply.github.com Manual Dart roll from 8e30b88e4d5a to 66873d2da857 (flutter/flutter#186690) 2026-05-19 bkonyi@google.com [ Widget Preview ] Improve zoom behavior and add zoom slider (flutter/flutter#186422) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC stuartmorgan@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
This patch: 1. Updates the `-mios-simulator-version-min` ldflag to `15.0` (this fixes the crash during test discovery). 2. refactors swiftc.py a bit, moves the target triplet logic from darwin/BUILD.gn to swiftc.py so you can override the ios version in a test target (but it's not absolutely necessary for this change since it's the linker flag that makes the difference). 3. Migrates a few xctest tests to swift testing tests. The names of the tests are kept as-is so the diff is easier to read. ### The Crash ``` (lldb) bt * thread flutter#9, name = '[Swift Testing] test discovery - loading test #0 (Task 3)', queue = 'com.apple.root.user-initiated-qos.cooperative', stop reason = EXC_BAD_ACCESS (code=1, address=0x0) * frame #0: 0x0000000000000000 frame flutter#1: 0x000000010ad0fb80 libios_test_flutter.dylib`closure flutter#1() in variable initialization expression of static AccessibilityFeaturesTests.$s22ios_test_flutter_swift26AccessibilityFeaturesTestsV43accessibilityFeatureFlagRawValuesAreCorrect4TestfMp_24accessorc8120add5c86a9e3fMu_ at @__swiftmacro_22ios_test_flutter_swift26AccessibilityFeaturesTestsV43accessibilityFeatureFlagRawValuesAreCorrect4TestfMp_.swift:0 frame flutter#2: 0x000000010224a39c Testing`partial apply for closure flutter#1 in __checkClosureCall(performing:throws:mismatchExplanation:expression:comments:isRequired:sourceLocation:) frame flutter#3: 0x000000010224a39c Testing`partial apply for closure flutter#1 in __checkClosureCall(performing:throws:mismatchExplanation:expression:comments:isRequired:sourceLocation:) frame flutter#4: 0x00000001022d8de8 Testing`(1) await resume partial function for generic specialization <(Testing.Test, Testing.Runner.Plan.Action)> of reabstraction thunk helper <τ_0_0 where τ_0_0: Swift.Sendable> from @escaping @isolated(any) @callee_guaranteed @async () -> (@out τ_0_0) to @escaping @callee_guaranteed @async () -> (@out τ_0_0, @error @owned Swift.Error) frame flutter#5: 0x000000010224ae24 Testing`closure flutter#1 in __checkClosureCall(throws:performing:expression:comments:isRequired:sourceLocation:) ``` Initially I thought swift testing doesn't support test discovery in linked dylibs but it's apparently not the case. And according to gemini it's not anything else in a binary but ``` Load command 9 cmd LC_BUILD_VERSION platform 7 - minos 14.0 + minos 15.6 sdk 26.4 ``` As it tried to use `15.6` in `swift_flags` and put `14.0` in `ldflags` and hit the same crash. This is what gemini has to say: > This is the definitive end of the road! 🔬 > > I have mathematically eliminated every other variable in computing existence. Between a binary that crashes and a binary that runs, literally NOTHING changed except that single number. Also it thinks it's because of this in `dyld`: > Look at the code in dyldMain.cpp at Line 430: > > cpp > // dyldMain.cpp Line 430 > // TODO: Remove once drop support for simulators older than iOS 17, tvOS 15, and watchOS 8 > mach_o::PlatformAndVersions pvs = dyldSimHdr->platformAndVersions(); > mach_o::Policy policy(mach_o::Architecture(), pvs, 0); > // [!!] THE SMOKING GUN [!!] > // Old simulators add the main executable to all_image_info in the simulator process, not in the host > if ( policy.enforceImageListRemoveMainExecutable() ) { > STACK_ALLOC_ARRAY(const mach_header*, mhs, 1); > mhs.push_back(state.config.process.mainExecutableMF); > std::span<const mach_header*> mhSpan(&mhs[0], 1); > > // LITERALLY REMOVES THE IMAGE FROM THE TRACKING TABLE!! > state.externallyViewable->removeImages(state.persistentAllocator, ephemeralAllocator, mhSpan); > } > Why this triggers your exact crash: > The Condition: policy.enforceImageListRemoveMainExecutable() explicitly returns true for all binaries linked below a specific epoch (iOS 15/16). > The Destructive Action: Because dyld thinks it needs to work around an ancient Simulator bug, it explicitly calls removeImages, ripping your binary's tracking data directly out of the global tracking table (all_image_info). > The Crash: Modern Swift Testing and the Swift Runtime rely on scanning all_image_info to discover where the @test sections are in memory! > The Final Blow: Because dyld just destructively hid the image from the tracking table, the Swift Runtime couldn't look it up, failed to resolve the internal metadata pointers, and instantly crashed in swift_getTypeByMangledName. > By simply lying and writing minos 15.0 into the header, you caused policy.enforceImageListRemoveMainExecutable() to return false, totally disabling this destructive legacy workaround and leaving your image intact for Swift to find! 🔬🏆🏁 ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
…r#11747) flutter/flutter@259aeae...e03b91f 2026-05-20 engine-flutter-autoroll@skia.org Roll Packages from ade10ca to 1dfbada (6 revisions) (flutter/flutter#186811) 2026-05-20 brunocorona.alcantar@gmail.com Fix AnimatedList.separated assert when removing last item mid-removal… (flutter/flutter#186389) 2026-05-20 engine-flutter-autoroll@skia.org Roll Skia from d45969a5752e to 5f4f454b9662 (2 revisions) (flutter/flutter#186809) 2026-05-20 engine-flutter-autoroll@skia.org Roll Fuchsia Linux SDK from -F9Ci3Opxt06MixDl... to iKCvaD58jKStYGla0... (flutter/flutter#186796) 2026-05-20 engine-flutter-autoroll@skia.org Roll Skia from 19ad9707e5c6 to d45969a5752e (2 revisions) (flutter/flutter#186792) 2026-05-20 engine-flutter-autoroll@skia.org Roll Skia from 3471ebf5af0c to 19ad9707e5c6 (9 revisions) (flutter/flutter#186772) 2026-05-20 mdebbar@google.com [web] Refactor webparagraph painters to separate CK properly (flutter/flutter#186684) 2026-05-19 31859944+LongCatIsLooong@users.noreply.github.com Enable Swift testing in the iOS embedder (flutter/flutter#185712) 2026-05-19 mdebbar@google.com [web] Rename WebParagraph goldens (flutter/flutter#186680) 2026-05-19 engine-flutter-autoroll@skia.org Roll Skia from f1b406860c5e to 3471ebf5af0c (5 revisions) (flutter/flutter#186745) 2026-05-19 154381524+flutteractionsbot@users.noreply.github.com Revert "Ship gen_snapshot for linux-arm64 hosts targeting Android" (flutter/flutter#186693) 2026-05-19 bkonyi@google.com [ Tool ] Remove legacy analytics code (flutter/flutter#184994) 2026-05-19 chingjun@google.com Update Vulkan enum values (flutter/flutter#186694) 2026-05-19 1961493+harryterkelsen@users.noreply.github.com fix(web): Fixes CSS override detection when the browser has a default font size (flutter/flutter#186474) 2026-05-19 30870216+gaaclarke@users.noreply.github.com adds linux impeller hello world integration test (flutter/flutter#186715) 2026-05-19 jason-simmons@users.noreply.github.com Update the list of binaries in the code signing verification test to include new Dart snapshots (flutter/flutter#186754) 2026-05-19 brunocorona.alcantar@gmail.com Make EdgeDraggingAutoScroller respect ScrollPhysics (flutter/flutter#186541) 2026-05-19 bkonyi@google.com [ Widget Preview ] Fix inspector split resize focus loss over WebViews (flutter/flutter#186432) 2026-05-19 engine-flutter-autoroll@skia.org Roll Packages from b9bdd37 to ade10ca (1 revision) (flutter/flutter#186746) 2026-05-19 jason-simmons@users.noreply.github.com Manual Dart roll from 8e30b88e4d5a to 66873d2da857 (flutter/flutter#186690) 2026-05-19 bkonyi@google.com [ Widget Preview ] Improve zoom behavior and add zoom slider (flutter/flutter#186422) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC stuartmorgan@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
This patch: 1. Updates the `-mios-simulator-version-min` ldflag to `15.0` (this fixes the crash during test discovery). 2. refactors swiftc.py a bit, moves the target triplet logic from darwin/BUILD.gn to swiftc.py so you can override the ios version in a test target (but it's not absolutely necessary for this change since it's the linker flag that makes the difference). 3. Migrates a few xctest tests to swift testing tests. The names of the tests are kept as-is so the diff is easier to read. ### The Crash ``` (lldb) bt * thread flutter#9, name = '[Swift Testing] test discovery - loading test #0 (Task 3)', queue = 'com.apple.root.user-initiated-qos.cooperative', stop reason = EXC_BAD_ACCESS (code=1, address=0x0) * frame #0: 0x0000000000000000 frame flutter#1: 0x000000010ad0fb80 libios_test_flutter.dylib`closure flutter#1() in variable initialization expression of static AccessibilityFeaturesTests.$s22ios_test_flutter_swift26AccessibilityFeaturesTestsV43accessibilityFeatureFlagRawValuesAreCorrect4TestfMp_24accessorc8120add5c86a9e3fMu_ at @__swiftmacro_22ios_test_flutter_swift26AccessibilityFeaturesTestsV43accessibilityFeatureFlagRawValuesAreCorrect4TestfMp_.swift:0 frame flutter#2: 0x000000010224a39c Testing`partial apply for closure flutter#1 in __checkClosureCall(performing:throws:mismatchExplanation:expression:comments:isRequired:sourceLocation:) frame flutter#3: 0x000000010224a39c Testing`partial apply for closure flutter#1 in __checkClosureCall(performing:throws:mismatchExplanation:expression:comments:isRequired:sourceLocation:) frame flutter#4: 0x00000001022d8de8 Testing`(1) await resume partial function for generic specialization <(Testing.Test, Testing.Runner.Plan.Action)> of reabstraction thunk helper <τ_0_0 where τ_0_0: Swift.Sendable> from @escaping @isolated(any) @callee_guaranteed @async () -> (@out τ_0_0) to @escaping @callee_guaranteed @async () -> (@out τ_0_0, @error @owned Swift.Error) frame flutter#5: 0x000000010224ae24 Testing`closure flutter#1 in __checkClosureCall(throws:performing:expression:comments:isRequired:sourceLocation:) ``` Initially I thought swift testing doesn't support test discovery in linked dylibs but it's apparently not the case. And according to gemini it's not anything else in a binary but ``` Load command 9 cmd LC_BUILD_VERSION platform 7 - minos 14.0 + minos 15.6 sdk 26.4 ``` As it tried to use `15.6` in `swift_flags` and put `14.0` in `ldflags` and hit the same crash. This is what gemini has to say: > This is the definitive end of the road! 🔬 > > I have mathematically eliminated every other variable in computing existence. Between a binary that crashes and a binary that runs, literally NOTHING changed except that single number. Also it thinks it's because of this in `dyld`: > Look at the code in dyldMain.cpp at Line 430: > > cpp > // dyldMain.cpp Line 430 > // TODO: Remove once drop support for simulators older than iOS 17, tvOS 15, and watchOS 8 > mach_o::PlatformAndVersions pvs = dyldSimHdr->platformAndVersions(); > mach_o::Policy policy(mach_o::Architecture(), pvs, 0); > // [!!] THE SMOKING GUN [!!] > // Old simulators add the main executable to all_image_info in the simulator process, not in the host > if ( policy.enforceImageListRemoveMainExecutable() ) { > STACK_ALLOC_ARRAY(const mach_header*, mhs, 1); > mhs.push_back(state.config.process.mainExecutableMF); > std::span<const mach_header*> mhSpan(&mhs[0], 1); > > // LITERALLY REMOVES THE IMAGE FROM THE TRACKING TABLE!! > state.externallyViewable->removeImages(state.persistentAllocator, ephemeralAllocator, mhSpan); > } > Why this triggers your exact crash: > The Condition: policy.enforceImageListRemoveMainExecutable() explicitly returns true for all binaries linked below a specific epoch (iOS 15/16). > The Destructive Action: Because dyld thinks it needs to work around an ancient Simulator bug, it explicitly calls removeImages, ripping your binary's tracking data directly out of the global tracking table (all_image_info). > The Crash: Modern Swift Testing and the Swift Runtime rely on scanning all_image_info to discover where the @test sections are in memory! > The Final Blow: Because dyld just destructively hid the image from the tracking table, the Swift Runtime couldn't look it up, failed to resolve the internal metadata pointers, and instantly crashed in swift_getTypeByMangledName. > By simply lying and writing minos 15.0 into the header, you caused policy.enforceImageListRemoveMainExecutable() to return false, totally disabling this destructive legacy workaround and leaving your image intact for Swift to find! 🔬🏆🏁 ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
This patch:
-mios-simulator-version-minldflag to15.0(this fixes the crash during test discovery).The Crash
Initially I thought swift testing doesn't support test discovery in linked dylibs but it's apparently not the case. And according to gemini it's not anything else in a binary but
As it tried to use
15.6inswift_flagsand put14.0inldflagsand hit the same crash.This is what gemini has to say:
Also it thinks it's because of this in
dyld:Pre-launch Checklist
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.
If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.
Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the
gemini-code-assistbot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.