Allow CircleCI to generate new test snapshots#1561
Closed
joshdholtz wants to merge 1 commit into
Closed
Conversation
e146fbf to
e0bf949
Compare
242dd2e to
696d98e
Compare
NachoSoto
reviewed
May 13, 2022
NachoSoto
left a comment
Contributor
There was a problem hiding this comment.
Awesome this looks ready! Amazing job.
The only think I don't understand is what's happening with the iOS 12 snapshots generated. They're way too many. Could you invoke a test run with raw output? I think something is happening with those requests that's making them retry many times for some reason.
| steps: | ||
| - run: | ||
| name: Run create_snapshot_pr | ||
| command: bundle exec fastlane create_snapshot_pr version:"tvos" |
Contributor
There was a problem hiding this comment.
We don't actually need this one. tvOS tests use the equivalent version (so tvOS 15 uses iOS 15 snapshots). Could it be removed?
| branch_name = "generated_snapshots/#{base_branch}-#{version}" | ||
| sh("git", "checkout", "-b", branch_name) | ||
|
|
||
| sh("git", "add", "../Tests") |
Member
Author
|
@NachoSoto Pushed a commit where no Xcode formatter (raw output) will be used when generating snapshots! |
6b6b552 to
5c5b7c5
Compare
60df869 to
6823c7e
Compare
5c5b7c5 to
21c5086
Compare
Contributor
|
Looks like this is working 🎉 But let's run it against #1560 to see if it generates the new ones correctly too. |
NachoSoto
reviewed
May 20, 2022
| prelaunch_simulator: true, | ||
| output_types: 'junit', | ||
| number_of_retries: 5, | ||
| number_of_retries: generate_snapshots ? 0 : 5, |
Member
|
what's the status of this one? |
1 task
Contributor
|
Should we close this for now? |
Member
Author
|
Ahhh... this one was super frustrating! CircleCI wasn't generating the things properly even though it should have. Probably should close it 😬 |
Contributor
|
We can pair to finish it, I'm sure it's almost ready? |
Member
Author
|
I'd be down to pair sometime 😊 |
Fixes https://app.circleci.com/pipelines/github/RevenueCat/purchases-ios/6737/workflows/b945f5f6-6f31-4716-b1ac-6e16e3848c7e/jobs/26321 For some reason, on iOS 12.x only, this: ```swift JSONDecoder.default.decode(jsonData: "null".data(using: .utf8)!) ``` Fails: ``` dataCorrupted(Swift.DecodingError.Context(codingPath: [], debugDescription: "The given data was not valid JSON.", underlyingError: Optional(Error Domain=NSCocoaErrorDomain Code=3840 "JSON text did not start with array or object and option to allow fragments not set." UserInfo={NSDebugDescription=JSON text did not start with array or object and option to allow fragments not set.}))) ``` We don't rely on decoding fragments, so I just updated the test to check `.null` within a dictionary instead. `Snapshotting.formattedData`: use `JSONEncoder.OutputFormatting.withoutEscapingSlashes` if available This was removed after a [PR review comment](#1366 (comment)) because it would have lead to different snapshots in iOS 12. However, since #1504 we have separate snapshots for each iOS version, so this cleans up the output when possible. ```diff - "url" : "https:\/\/api.revenuecat.com\/v1\/subscribers\/user\/alias" + "url" : "https://api.revenuecat.com/v1/subscribers/user/alias" ``` Update CircleCI config to use pipeline parameters to set generate snapshots env Run all in one serially Split OS version into their own jobs Create PR from all os version jobs This is the one Added requestby user in body Fix for pr actually creating Reordered some things Remove tvos from generated jobs Raw output when generating snapshots Don't retry tests when generating snapshots Maybe this will make the PRs Maybe this This is it
835a255 to
5fb1433
Compare
Contributor
|
Closing in favor of #2472. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #1589
Motivation
Our snapshot testing sometimes needs regeneration and it requires regeneration of iOS 12. iOS 12 simulators only run on Intel machines and not all of us have them.
This PR will allow a CircleCI job to be manually triggered to regenerate snapshot tests
Description
Step 1: Trigger CircleCI workflow
Run
bundle exec fastlane generate_snapshotsStep 2: Run CircleCI workflow with generate snapshots
This will start a series of jobs on CircleCI that will have the
CIRCLECI_TESTS_GENERATE_SNAPSHOTSset totruethat will regenerate new/updated snapshot test assets.Step 3: Create pull requests
If there are any files changed, a new PR will be created against the base branch (set earlier in the
generate_snapshotslane)Example PR 👉 #1578