Skip to content

[CI] Run unit and functional tests separately in CI#368

Merged
kevinlind merged 7 commits intoadobe:devfrom
timkimadobe:build-process-fix
Jul 17, 2023
Merged

[CI] Run unit and functional tests separately in CI#368
kevinlind merged 7 commits intoadobe:devfrom
timkimadobe:build-process-fix

Conversation

@timkimadobe
Copy link
Copy Markdown
Contributor

@timkimadobe timkimadobe commented Jul 14, 2023 •
edited
Loading

Description

This PR:

  1. Splits unit and functional test steps in the makefile to prevent the occasional interleaving of unit test build process with functional test case execution

Related Issue

Motivation and Context

For future reference, the old method required extracting the true scheme, with output like (note the duplicate AEPEdge scheme differentiated with parentheses):

List of available shared Schemes in xcworkspace
xcodebuild -workspace AEPEdge.xcworkspace -list
Command line invocation:
    /Applications/Xcode-14.1.app/Contents/Developer/usr/bin/xcodebuild -workspace AEPEdge.xcworkspace -list

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

Information about workspace "AEPEdge":
    Schemes:
        AEPAssurance
        AEPCore
        AEPEdge (AEPEdge project)
        AEPEdge (Pods project)
        AEPEdgeConsent
        AEPEdgeIdentity
        AEPEdgeXCF
        AEPRulesEngine
        AEPServices
        FunctionalTests
        Pods-AEPEdge
        Pods-FunctionalTests
        Pods-TestAppiOS
        Pods-TestApptvOS
        Pods-UnitTests
        Pods-UpstreamIntegrationTests
        SwiftLint
        TestAppiOS
        TestApptvOS
        UnitTests
        UpstreamIntegrationTests

final_scheme=""; \
	if xcodebuild -workspace AEPEdge.xcworkspace -list | grep -q "(AEPEdge project)"; \
	then \
	   final_scheme="AEPEdge (AEPEdge project)" ; \
	   echo $final_scheme ; \
	else \
	   final_scheme="AEPEdge" ; \
	   echo $final_scheme ; \
	fi; \
	xcodebuild test -workspace AEPEdge.xcworkspace -scheme "$final_scheme" -destination 'platform=iOS Simulator,name=iPhone 14' -derivedDataPath build/out -resultBundlePath iosresults.xcresult -enableCodeCoverage YES ADB_SKIP_LINT=YES
AEPEdge (AEPEdge project)
Command line invocation:
    /Applications/Xcode-14.1.app/Contents/Developer/usr/bin/xcodebuild test -workspace AEPEdge.xcworkspace -scheme "AEPEdge (AEPEdge project)" -destination "platform=iOS Simulator,name=iPhone 14" -derivedDataPath build/out -resultBundlePath iosresults.xcresult -enableCodeCoverage YES ADB_SKIP_LINT=YES

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • I have signed the Adobe Open Source CLA.
  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@timkimadobe
Split unit and functional test steps
5d63bc5 
Update CircleCI script accordingly
@timkimadobe timkimadobe changed the title Build process fix CI test process fix Jul 14, 2023
@timkimadobe timkimadobe changed the title CI test process fix CI testing workflow fix Jul 14, 2023
@codecov
Copy link
Copy Markdown

codecov bot commented Jul 15, 2023 •
edited
Loading

Codecov Report

Merging #368 (61b1076) into dev (1cf1b14) will increase coverage by 0.05%.
The diff coverage is 100.00%.

❗ Current head 61b1076 differs from pull request most recent head bff3336. Consider uploading reports for the commit bff3336 to get more accurate results

@@            Coverage Diff             @@
##              dev     #368      +/-   ##
==========================================
+ Coverage   96.68%   96.73%   +0.05%     
==========================================
  Files          27       27              
  Lines        1626     1653      +27     
==========================================
+ Hits         1572     1599      +27     
  Misses         54       54

@timkimadobe timkimadobe mentioned this pull request Jul 15, 2023
Merged
10 tasks
emdobrin
emdobrin reviewed Jul 15, 2023
View reviewed changes
timkimadobe
timkimadobe commented Jul 15, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor Author

@timkimadobe timkimadobe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @emdobrin! Reordered unit and functional tests

@timkimadobe timkimadobe requested a review from emdobrin July 15, 2023 00:15
emdobrin
emdobrin reviewed Jul 15, 2023
View reviewed changes
kevinlind
kevinlind reviewed Jul 15, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@kevinlind kevinlind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change looks good but it appears there are still test failures in CI. Are those failures expected?

@kevinlind
Copy link
Copy Markdown
Contributor

kevinlind commented Jul 15, 2023

In the CircleCI configuration, I'd recommend running the unit and functional tests in separate steps, instead of just one step which runs both sequentially. The reason is when a failure occurs it is hard to know if the failure is from the unit or functional tests. For example, in this failure the last few lines displayed show a functional test failure but searching through the entire log shows a unit test failed as well.
https://app.circleci.com/pipelines/github/adobe/aepsdk-edge-ios/1094/workflows/8f8abbf7-815b-4877-88ec-2e70a4af49b8/jobs/1745

emdobrin
emdobrin reviewed Jul 17, 2023
View reviewed changes
emdobrin
emdobrin reviewed Jul 17, 2023
View reviewed changes
@kevinlind kevinlind changed the title CI testing workflow fix [CI] Run unit and functional tests separately in CI Jul 17, 2023
@kevinlind kevinlind merged commit f114e14 into adobe:dev Jul 17, 2023
@timkimadobe timkimadobe deleted the build-process-fix branch July 18, 2023 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@emdobrin emdobrin emdobrin approved these changes

@kevinlind kevinlind kevinlind approved these changes

@addb addb Awaiting requested review from addb

@cacheung cacheung Awaiting requested review from cacheung

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@timkimadobe @kevinlind @emdobrin