Integration Tests: relaunch tests when retrying failures#1925
Merged
Conversation
I've observed that a lot of the time flaky tests still don't pass after a retry. This change can probably decrease the likelyhood that a retry will still fail, by ensuring that we create a brand new `SKTestSession` in a new simulator session.
Contributor
Author
|
Welp flaky tests are still flaky but I think this is still useful to minimize issues with CI's simulators. |
tonidero
approved these changes
Sep 22, 2022
tonidero
left a comment
Contributor
There was a problem hiding this comment.
Since test keeps failing, not sure if we want to merge it but looks good and shouldn't hurt. Also, would this slow down tests too much? I'm guessing spinning new processes would be slower.
Contributor
Author
|
This failed in 5 minutes vs another sample failure in |
NachoSoto
added a commit
that referenced
this pull request
Sep 27, 2022
**This is an automatic release.** ### New Features * 🚨 `StoreKit 2` is now enabled by default 🚨 (#1922) via NachoSoto (@NachoSoto) * Extracted `PurchasesType` and `PurchasesSwiftType` (#1912) via NachoSoto (@NachoSoto) ### Bugfixes * `StoreKit 1`: changed result of cancelled purchases to be consistent with `StoreKit 2` (#1910) via NachoSoto (@NachoSoto) * `PaymentQueueWrapper`: handle promotional purchase requests from App Store when SK1 is disabled (#1901) via NachoSoto (@NachoSoto) ### Other Changes * Fixed iOS 12 tests (#1936) via NachoSoto (@NachoSoto) * `CacheableNetworkOperation`: fixed race condition in new test (#1932) via NachoSoto (@NachoSoto) * `BasePurchasesTests`: changed default back to SK1 (#1935) via NachoSoto (@NachoSoto) * `Logger`: refactored default `LogLevel` definition (#1934) via NachoSoto (@NachoSoto) * `AppleReceipt`: refactored declarations into nested types (#1933) via NachoSoto (@NachoSoto) * `Integration Tests`: relaunch tests when retrying failures (#1925) via NachoSoto (@NachoSoto) * `CircleCI`: downgraded release jobs to Xcode 13.x (#1927) via NachoSoto (@NachoSoto) * `ErrorUtils`: added test to verify that `PublicError`s can be `catch`'d as `ErrorCode` (#1924) via NachoSoto (@NachoSoto) * `StoreKitIntegrationTests`: print `AppleReceipt` data whenever `verifyEntitlementWentThrough` fails (#1929) via NachoSoto (@NachoSoto) * `OperationQueue`: log debug message when requests are found in cache and skipped (#1926) via NachoSoto (@NachoSoto) * `GetCustomerInfoAPI`: avoid making a request if there's any `PostReceiptDataOperation` in progress (#1911) via NachoSoto (@NachoSoto) * `PurchaseTester`: allow HTTP requests and enable setting `ProxyURL` (#1917) via NachoSoto (@NachoSoto)
NachoSoto
added a commit
that referenced
this pull request
Oct 24, 2023
See https://github.com/RevenueCat/purchases-ios/blob/4.28.1/Tests/TestPlans/CI-BackendIntegration.xctestplan#L28 This was changed to 5 repetitions in #3218, but the `Fastfile` was overriding that. Also #1925 didn't have any effect because of this, and that actually makes tests take significantly longer, so this reverts that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I've observed that a lot of the time flaky tests still don't pass after a retry.
This change can probably decrease the likelyhood that a retry will still fail, by ensuring that we create a brand new
SKTestSessionin a new simulator session.