Context
We have two timeouts in XHarness. First one, --launch-timeouts, starts when XHarness starts and is discarded when the application boots and test execution starts (we have TCP connection). Second, --timeout, runs from the beginning and breaks the whole run when is hit.
Problem
We see TIMED_OUT results after --launch-timeout starts but this timeout should give us PACKAGE_INSTALLATION_TIMEOUT probably because we never start executing.
Or we have started executing the app and this timeout should not matter anymore. We can then have TIMED_OUT if the second timeout for the whole run is hit.
We shouldn't have TIMED_OUT after --launch-timeout is over though.
Example logs (full logs)
XHarness command issued: apple test --app /tmp/helix/working/A62F0846/w/A14608E7/e/Microsoft.Extensions.Configuration.UserSecrets.Tests.app --output-directory /tmp/helix/working/A62F0846/w/A14608E7/uploads --target tvos-simulator --timeout 00:30:00 --xcode /Applications/Xcode115.app -v --launch-timeout 00:03:00
[12:38:15] info: Preparing run for tvos-simulator
[12:38:15] info: Getting app bundle information from '/tmp/helix/working/A62F0846/w/A14608E7/e/Microsoft.Extensions.Configuration.UserSecrets.Tests.app'
[12:38:15] dbug:
[12:38:15] dbug: Running /usr/libexec/PlistBuddy -c "Print CFBundleName" /tmp/helix/working/A62F0846/w/A14608E7/e/Microsoft.Extensions.Configuration.UserSecrets.Tests.app/Info.plist
[12:38:15] dbug: Process PlistBuddy exited with 0
[12:38:15] dbug:
[12:38:15] dbug: Running /usr/libexec/PlistBuddy -c "Print CFBundleIdentifier" /tmp/helix/working/A62F0846/w/A14608E7/e/Microsoft.Extensions.Configuration.UserSecrets.Tests.app/Info.plist
[12:38:15] dbug: Process PlistBuddy exited with 0
[12:38:15] dbug:
[12:38:15] dbug: Running /usr/libexec/PlistBuddy -c "Print UIRequiredDeviceCapabilities" /tmp/helix/working/A62F0846/w/A14608E7/e/Microsoft.Extensions.Configuration.UserSecrets.Tests.app/Info.plist
[12:38:15] dbug: Process PlistBuddy exited with 1
[12:38:15] dbug: Property UIRequiredDeviceCapabilities not present in Info.plist, assuming 32-bit is not supported
[12:38:15] info: Looking for available tvos-simulator simulators..
[12:38:15] dbug: Looking for available tvos-simulator simulators. Storing logs into list-tvos-simulator-20211130_123815.log
[12:38:32] info: Found simulator device 'Apple TV (tvOS 13.4) - created by XHarness'
[12:38:32] info: Uninstalling any previous instance of 'net.dot.Microsoft.Extensions.Configuration.UserSecrets.Tests' from 'Apple TV (tvOS 13.4) - created by XHarness'
[12:38:32] dbug:
[12:38:32] dbug: Running /Applications/Xcode115.app/Contents/Developer/usr/bin/simctl uninstall F564F86E-0133-4B19-99A9-044549D844D8 net.dot.Microsoft.Extensions.Configuration.UserSecrets.Tests
[12:38:32] dbug: An error was encountered processing the command (domain=com.apple.CoreSimulator.SimError, code=165):
[12:38:32] dbug: Unable to lookup in current state: Shutdown
[12:38:32] dbug: Process simctl exited with 165
[12:38:32] dbug: Preemptive uninstallation of application failed
[12:38:32] info: Installing application 'Microsoft.Extensions.Configuration.UserSecrets.Tests' on 'Apple TV (tvOS 13.4) - created by XHarness'
[12:38:32] dbug: Installing '/tmp/helix/working/A62F0846/w/A14608E7/e/Microsoft.Extensions.Configuration.UserSecrets.Tests.app' to 'Apple TV (tvOS 13.4) - created by XHarness' (46.84 MB)
[12:38:32] dbug:
[12:38:32] dbug: Running /private/tmp/helix/working/A62F0846/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21576.2/tools/net6.0/any/../../../runtimes/any/native/mlaunch/bin/mlaunch --sdkroot /Applications/Xcode115.app --device=:v2:udid=F564F86E-0133-4B19-99A9-044549D844D8 --installsim /tmp/helix/working/A62F0846/w/A14608E7/e/Microsoft.Extensions.Configuration.UserSecrets.Tests.app -v -v
[12:38:33] dbug: Using Xcode 11.5 found in /Applications/Xcode115.app
[12:38:33] dbug: Xamarin.Hosting: Xamarin.Hosting
[12:38:33] dbug: Xamarin.Hosting: Version: efad4f0bae (main)
[12:38:33] dbug: Xamarin.Hosting: Xcode: /Applications/Xcode115.app
[12:38:33] dbug: Xamarin.Hosting: Xcode Version: 11.5
[12:38:33] dbug: Xamarin.Hosting: Verbosity: 2
[12:38:34] dbug: Xamarin.Hosting: Booting Apple TV (tvOS 13.4) - created by XHarness...
[12:38:35] dbug: Xamarin.Hosting: Booted Apple TV (tvOS 13.4) - created by XHarness successfully.
[12:38:35] dbug: Xamarin.Hosting: Installing /tmp/helix/working/A62F0846/w/A14608E7/e/Microsoft.Extensions.Configuration.UserSecrets.Tests.app with Bundle Identifier net.dot.Microsoft.Extensions.Configuration.UserSecrets.Tests on 'tvOS 13.4 (17L255) - Apple TV'...
[12:39:50] dbug: Xamarin.Hosting: Installed 'net.dot.Microsoft.Extensions.Configuration.UserSecrets.Tests' from /tmp/helix/working/A62F0846/w/A14608E7/e/Microsoft.Extensions.Configuration.UserSecrets.Tests.app
[12:39:51] dbug: Xamarin.Hosting: Can't process potentially bad load commands, because the app is signed.
[12:39:51] dbug: Xamarin.Hosting: The bundle id net.dot.Microsoft.Extensions.Configuration.UserSecrets.Tests was successfully installed.
[12:39:57] dbug: Xamarin.Hosting: Shutdown Apple TV (tvOS 13.4) - created by XHarness successfully.
[12:39:57] dbug: Process mlaunch exited with 0
[12:39:57] info: Application 'Microsoft.Extensions.Configuration.UserSecrets.Tests' was installed successfully on 'Apple TV (tvOS 13.4) - created by XHarness'
[12:39:57] info: Starting test run for net.dot.Microsoft.Extensions.Configuration.UserSecrets.Tests..
[12:39:57] dbug: Test log server listening on: 0.0.0.0:55477
[12:39:57] dbug: *** Executing 'Microsoft.Extensions.Configuration.UserSecrets.Tests' on tvos-simulator 'Apple TV (tvOS 13.4) - created by XHarness' ***
[12:39:57] dbug: System log for the 'Apple TV (tvOS 13.4) - created by XHarness' simulator is: /private/var/root/Library/Logs/CoreSimulator/F564F86E-0133-4B19-99A9-044549D844D8/system.log
[12:39:57] dbug: Starting test run
[12:39:57] dbug:
[12:39:57] dbug: Running /private/tmp/helix/working/A62F0846/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21576.2/tools/net6.0/any/../../../runtimes/any/native/mlaunch/bin/mlaunch --sdkroot /Applications/Xcode115.app -setenv=NUNIT_AUTOEXIT=true -setenv=NUNIT_HOSTPORT=55477 -setenv=NUNIT_ENABLE_XML_OUTPUT=true -setenv=NUNIT_XML_VERSION=xUnit -setenv=NUNIT_HOSTNAME=127.0.0.1 --device=:v2:udid=F564F86E-0133-4B19-99A9-044549D844D8 --stdout=/tmp/helix/working/A62F0846/w/A14608E7/uploads/net.dot.Microsoft.Extensions.Configuration.UserSecrets.Tests.log --stderr=/tmp/helix/working/A62F0846/w/A14608E7/uploads/net.dot.Microsoft.Extensions.Configuration.UserSecrets.Tests.log --launchsimbundleid=net.dot.Microsoft.Extensions.Configuration.UserSecrets.Tests -v -v -v -v -v
[12:39:58] dbug: Using Xcode 11.5 found in /Applications/Xcode115.app
[12:39:58] dbug: Xamarin.Hosting: Xamarin.Hosting
[12:39:58] dbug: Xamarin.Hosting: Version: efad4f0bae (main)
[12:39:58] dbug: Xamarin.Hosting: Xcode: /Applications/Xcode115.app
[12:39:58] dbug: Xamarin.Hosting: Xcode Version: 11.5
[12:39:58] dbug: Xamarin.Hosting: Verbosity: 5
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/DVTFoundation
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DVTiPhoneSimulatorRemoteClient.framework/DVTiPhoneSimulatorRemoteClient
[12:39:58] dbug: Xamarin.Hosting: Loaded /Library/Developer/PrivateFrameworks/CoreSimulator.framework/CoreSimulator
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DTDeviceKitBase.framework/DTDeviceKitBase
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DVTKit.framework/DVTKit
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DTDeviceKit.framework/DTDeviceKit
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DTXConnectionServices.framework/DTXConnectionServices
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DVTSourceControl.framework/DVTSourceControl
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DVTServices.framework/DVTServices
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DVTPortal.framework/DVTPortal
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DVTDocumentation.framework/DVTDocumentation
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DVTAnalyticsClient.framework/DVTAnalyticsClient
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/ContentDeliveryServices.framework/ContentDeliveryServices
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/DVTAnalytics.framework/DVTAnalytics
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/SourceKit.framework/SourceKit
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/Frameworks/IDEFoundation.framework/IDEFoundation
[12:39:58] dbug: Xamarin.Hosting: Loaded /Applications/Xcode115.app/Contents/SharedFrameworks/IDEProducts.framework/IDEProducts
[12:39:58] dbug: Xamarin.Hosting: Simulator watchdogs are already disabled for 'tvOS 13.4 (17L255) - Apple TV'.
[12:39:59] dbug: Xamarin.Hosting: Launching simulator application 'com.apple.iphonesimulator'
[12:41:15] dbug: Killing process 2045 as it was cancelled
[12:41:15] dbug: Test execution timed out after 1.3038806166666668 minute(s).
[12:41:15] dbug: Process mlaunch exited with 137
[12:41:15] dbug: Test run failed
[12:41:15] dbug: Could not find pid in mtouch output.
[12:41:16] dbug: Test run never launched
[12:41:16] dbug: No crash reports, waiting 5 seconds for the crash report service...
[12:41:22] warn: Application test run timed out
Context
We have two timeouts in XHarness. First one,
--launch-timeouts, starts when XHarness starts and is discarded when the application boots and test execution starts (we have TCP connection). Second,--timeout, runs from the beginning and breaks the whole run when is hit.Problem
We see
TIMED_OUTresults after--launch-timeoutstarts but this timeout should give usPACKAGE_INSTALLATION_TIMEOUTprobably because we never start executing.Or we have started executing the app and this timeout should not matter anymore. We can then have
TIMED_OUTif the second timeout for the whole run is hit.We shouldn't have
TIMED_OUTafter--launch-timeoutis over though.Example logs (full logs)