Support running x86_64 logic tests on Apple silicon by bogo · Pull Request #40 · google/xctestrunner

bogo · 2021-11-07T09:58:43Z

This PR adds support for running x86_64 logic tests on Apple silicon by following the patterns introduced in 7f8fc8. This is most helpful in the current Bazel/rules_apple context, where the only Simulator binaries produced are x86. This approach to running logic tests on Apple silicon is discussed in MobileNativeFoundation/discussions#141.

Since xctest is stripped down to x86 only if the x86 slice is present, this should not affect existing Intel Macs, and should be automatically skipped once rules_apple and Bazel correctly support Apple silicon Simulator.

keith · 2021-11-11T20:08:33Z

test_runner/logic_test_util.py

+  test_archs = bundle_util.GetFileArchTypes(test_executable)
+
+  # if a logic bundle is built w/ x86 on Apple silicon, it won't be able to launch on the ARM64 sim; rework the xctest to fix this
+  if ios_constants.ARCH.X86_64 in test_archs:


Does this need to be an exclusive check? Because you could be testing a fat binary in which case you still wouldn't want to do this I think?

Fair point. Do you think parametrizing the test runner with something like --force-run-x86 would be sufficient here? Then it could be passed as a test parameter from Bazel invocation.

seems like host arch == arm64 && test_archs == x86_64 && no test host is a find conditional for now to make things "just work" for folks, but up to the google folks

Keith, why testing with a fat binary is not expected? I don't have a local M1 machine to test it out. My understanding is the current implementation can work for both x86_64 only test bundle and fat test bundle.

if you have a fat binary, this conditional will be true, but you should probably not do this and test the native arm64 slice instead if you have it.

Make sense.

Bogo, could you help change the condition to be ios_constants.ARCH.X86_64 in test_archs and len(test_archs) == 1? Doing the host arch check seems not necessary. No matter if the host is M1 or intel, we can use x86_64 only xctest tool to execute x86_64 test bundle.

I will also need to change

xctestrunner/test_runner/xctestrun.py

Line 558 in db0fce1

bundle_util.RemoveArchType(uitest_runner_exec, ios_constants.ARCH.ARM64)

my conditional mentioned above also makes sure this wouldn't happen in cases where it's unnecessary, but it's not required

@keith I thought about testing the host architecture, but figured it might be better to have the code paths as close as possible on Apple silicon and x86. Can add the necessary helpers if you feel it'd be worth it though.

ivan-golub · 2022-01-31T23:51:57Z

Hey @bogo! This PR seems to be stale for quite some time. Is it abandoned? Do you need any help to finalize it? My team would like to be able to run x86 sim tests via bazel which mean we need to teach xctestrunner first

thii · 2022-02-03T04:49:56Z

@albertdai Can you take a look at this too?

albertdai · 2022-02-03T19:34:47Z

test_runner/logic_test_util.py

+  xctest_tool = shutil.copy(
+    xcode_info_util.GetXctestToolPath(ios_constants.SDK.IPHONESIMULATOR),
+    os.path.join(tempfile.mkdtemp(), "xctest")
+  )


You can change LeaveOnlyArchType to LeaveOnlyArchType(input_file_path, output_file_path, arch_type). So we don't need to do this copy

bogo · 2022-02-08T18:44:06Z

Thanks for the help, @keith and @albertdai! Applied the changes you suggested.

thii · 2022-03-15T21:26:46Z

Seems like this patch needs tweaks for Xcode 13.3. Getting the following error when using it with Xcode 13.3:

2022-03-16 06:03:33.959 xctest[76139:18007203] The bundle “ATest.xctest” couldn’t be loaded. Try reinstalling the bundle.
2022-03-16 06:03:33.962 xctest[76139:18007203] (dlopen(/tmp/test_runner_work_dir.fetEaf/ATest/ATest.xctest/ATest, 0x0109): Library not loaded: @rpath/libXCTestSwiftSupport.dylib
  Referenced from: /private/tmp/test_runner_work_dir.fetEaf/ATest/ATest.xctest/ATest

  Reason: tried: '/Applications/Xcode-13.3.0.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/swift/libXCTestSwiftSupport.dylib' (no such file), '/usr/lib/swift/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/tmp3710xrhn/Frameworks/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/test_runner_work_dir.fetEaf/ATest/ATest.xctest/Frameworks/libXCTestSwiftSupport.dylib' (no such file), '/Applications/Xcode-13.3.0.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/swift/libXCTestSwiftSupport.dylib' (no such file), '/usr/lib/swift/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/tmp3710xrhn/Frameworks/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/test_runner_work_dir.fetEaf/ATest/ATest.xctest/Frameworks/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/tmp3710xrhn/../../Frameworks/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/tmp3710xrhn/../../PrivateFrameworks/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/tmp3710xrhn/../../../usr/lib/libXCTestSwiftSupport.dylib' (no such file))

ivan-golub · 2022-03-16T16:51:45Z

Hey @bogo @keith @albertdai ! We have been using custom xctestrunner from this commit with great success but would love to get this in main branch to get other fixes like #41 in too. Do you need any help with this PR to move it forward? Also looks like @thii caught some issues on 13.3 which means some more adjustments are needed

keith · 2022-03-16T16:55:34Z

Seems like we'd need to fix the new issues before doing anything, but note that on this repo @albertdai is the decider on this stuff as the only person with commit access

albertdai · 2022-03-16T18:40:36Z

Seems like this patch needs tweaks for Xcode 13.3. Getting the following error when using it with Xcode 13.3:

2022-03-16 06:03:33.959 xctest[76139:18007203] The bundle “ATest.xctest” couldn’t be loaded. Try reinstalling the bundle.
2022-03-16 06:03:33.962 xctest[76139:18007203] (dlopen(/tmp/test_runner_work_dir.fetEaf/ATest/ATest.xctest/ATest, 0x0109): Library not loaded: @rpath/libXCTestSwiftSupport.dylib
  Referenced from: /private/tmp/test_runner_work_dir.fetEaf/ATest/ATest.xctest/ATest

  Reason: tried: '/Applications/Xcode-13.3.0.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/swift/libXCTestSwiftSupport.dylib' (no such file), '/usr/lib/swift/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/tmp3710xrhn/Frameworks/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/test_runner_work_dir.fetEaf/ATest/ATest.xctest/Frameworks/libXCTestSwiftSupport.dylib' (no such file), '/Applications/Xcode-13.3.0.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/swift/libXCTestSwiftSupport.dylib' (no such file), '/usr/lib/swift/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/tmp3710xrhn/Frameworks/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/test_runner_work_dir.fetEaf/ATest/ATest.xctest/Frameworks/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/tmp3710xrhn/../../Frameworks/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/tmp3710xrhn/../../PrivateFrameworks/libXCTestSwiftSupport.dylib' (no such file), '/private/tmp/tmp3710xrhn/../../../usr/lib/libXCTestSwiftSupport.dylib' (no such file))

I don't have a local Apple silicon machine to test out this issue on Xcode 13.3. @bogo Could you help double check with that from your side?

albertdai · 2022-03-16T19:41:42Z

I tested out locally with my intel macbook pro and confirmed the test runner with the commits can not work with Swift logic test on Xcode 13.3.

So we need to fix the issues before merging the commits into master.

ivan-golub · 2022-04-07T18:15:22Z

@bogo do you need help with this PR? This is really valuable change

bogo · 2022-04-18T18:59:57Z

@ivan-golub @albertdai will take a look at whether I can make it work again either later this week or early next week!

albertdai · 2022-04-18T20:25:02Z

Thanks Bogo.

I feel the xctest tool may have some dependencies with relative path under the new Xcode version. You can consider creating the temp xctest tool under the same directory of xctest. I have not tested it out yet.

ivan-golub · 2022-04-22T19:19:30Z

FYI: failure for swift logic tests seem to be present only for iOS 15.4 simulator, ie this will fail

ios_test_runner(
    name = "iphone_test_runner",
    device_type = "iPhone 8",
    os_version = "15.4",
    visibility = ["//visibility:public"],
)

but this will not

ios_test_runner(
    name = "iphone_test_runner",
    device_type = "iPhone 8",
    os_version = "15.2", # or 15.0
    visibility = ["//visibility:public"],
)

ivan-golub · 2022-04-22T19:33:54Z

test_runner/logic_test_util.py

+
+  # if a logic bundle is built w/ x86 on Apple silicon, it won't be able to launch on the ARM64 sim; rework the xctest to fix this
+  if ios_constants.ARCH.X86_64 in test_archs and len(test_archs) == 1:
+    replacement_xctest_tool = os.path.join(tempfile.mkdtemp(), "xctest")


Suggested change

replacement_xctest_tool = os.path.join(tempfile.mkdtemp(), "xctest")

replacement_xctest_tool = os.path.join(os.path.dirname(xctest_tool), "xctest_x86_64")

this fixes swift logic test issue with xcode 13.3 and ios 15.4 as @albertdai suspected here #40 (comment)

this could fail depending on how the user installed xcode, as it won't have permissions in some cases

keith · 2022-05-17T01:08:09Z

FYI this broke with Xcode 13.4.0, the issue appears to be a change in iOS, I'm debugging a bit

keith · 2022-05-17T01:54:51Z

test_runner/logic_test_util.py

+    )
+    xctest_tool = replacement_xctest_tool
+    platform_developer_dir = os.path.join(xcode_info_util.GetSdkPlatformPath(ios_constants.SDK.IPHONESIMULATOR), "Developer")
+    simctl_env_vars[_SIMCTL_ENV_VAR_PREFIX + "DYLD_FALLBACK_LIBRARY_PATH"] = "{0}/usr/lib".format(platform_developer_dir)


Suggested change

simctl_env_vars[_SIMCTL_ENV_VAR_PREFIX + "DYLD_FALLBACK_LIBRARY_PATH"] = "{0}/usr/lib".format(platform_developer_dir)

simctl_env_vars[_SIMCTL_ENV_VAR_PREFIX + "DYLD_LIBRARY_PATH"] = "{0}/usr/lib".format(platform_developer_dir)

this fixes Xcode 13.4 support, and still works for earlier versions. I'm not sure why yet but something about the semantics of this changed.

Support running logic tests on M1

0454d81

keith reviewed Nov 11, 2021

View reviewed changes

albertdai reviewed Feb 3, 2022

View reviewed changes

Apply suggested fixes

1b6c804

Switch tool names around

a3f4ce8

jpsim mentioned this pull request Apr 22, 2022

[bazelw] Use arm64 binary on darwin-arm64 envoyproxy/envoy-mobile#2185

Merged

ivan-golub reviewed Apr 22, 2022

View reviewed changes

keith reviewed May 17, 2022

View reviewed changes

bogo closed this Oct 16, 2022

	replacement_xctest_tool = os.path.join(tempfile.mkdtemp(), "xctest")
	replacement_xctest_tool = os.path.join(os.path.dirname(xctest_tool), "xctest_x86_64")

	simctl_env_vars[_SIMCTL_ENV_VAR_PREFIX + "DYLD_FALLBACK_LIBRARY_PATH"] = "{0}/usr/lib".format(platform_developer_dir)
	simctl_env_vars[_SIMCTL_ENV_VAR_PREFIX + "DYLD_LIBRARY_PATH"] = "{0}/usr/lib".format(platform_developer_dir)

Conversation

bogo commented Nov 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivan-golub commented Jan 31, 2022

Uh oh!

thii commented Feb 3, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bogo commented Feb 8, 2022

Uh oh!

thii commented Mar 15, 2022

Uh oh!

ivan-golub commented Mar 16, 2022

Uh oh!

keith commented Mar 16, 2022

Uh oh!

albertdai commented Mar 16, 2022

Uh oh!

albertdai commented Mar 16, 2022

Uh oh!

ivan-golub commented Apr 7, 2022

Uh oh!

bogo commented Apr 18, 2022

Uh oh!

albertdai commented Apr 18, 2022

Uh oh!

ivan-golub commented Apr 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

keith commented May 17, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bogo commented Nov 7, 2021 •

edited

Loading

ivan-golub commented Apr 22, 2022 •

edited

Loading