Edge Network (Konductor) integration test with GitHub Action workflow by timkimadobe · Pull Request #321 · adobe/aepsdk-edge-ios

timkimadobe · 2023-03-28T23:25:59Z

Description

This PR:

Creates a new workflow for integration testing between Edge network extension and Edge Network (Konductor).
Updates documentation to use correct links

The new integration test target can be run:

Locally
Manual trigger through GitHub action
Automated GitHub action workflow run

The new action has configuration settings for:

Code branch to run the action off of (defaults to main)
ID for the job run (optional) - useful for uniquely identifying a run, especially when integrating with other automated CI/CD systems
Edge Network environment level to test (prod, pre-prod, int)
Edge location hint to set before each test case (optional)

Dropdown menus are used for options 3 and 4 to help with ease of selection and preventing typos, since these values are known and have static values

The action is set up in my fork of the repo here: https://github.com/timkimadobe/aepsdk-edge-ios/actions/workflows/edge-network-integration-test.yml

Examples of manual workflow trigger:

Overview of workflow

The entry points are:

Manual trigger
Automated trigger
Local run

flowchart TD
    D(Manual trigger) & E(Automated trigger) --> A
    A("<b>GitHub Action</b><br>Select job run settings (env vars)<br>Checkout + Caching + Make") --> |"Env vars (Makefile context)"| B
    
    B(<b>Makefile</b><br>Pod install + xcodebuild<br>Set env vars in shell context) --> |"Sanitized env vars<br>based on Xcode scheme"| C    
    C(<b>Xcode test</b><br>Integration test Edge + Edge Network)
    
    F(<b>Local run</b><br>Makefile command + specify env vars) --> |"Env vars (Makefile context)"| B

Env vars = Environment variables

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

I have signed the Adobe Open Source CLA.
My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

add conditional check warning output for KONDUCTOR_ENV

test colon in test run echo update env var extraction documentation in test file

Tests/FunctionalTests/util/FunctionalTestBase.swift

.github/workflows/edge-network-integration-test.yml

emdobrin · 2023-03-31T20:59:39Z

Tests/KonductorIntegrationTests/EdgeNetworkIntegrationTests.swift

+    private var edgeLocationHint: EdgeLocationHint? = nil
+
+    /// Edge Network (Konductor) environment levels that correspond to their deployment environment levels
+    enum EdgeEnvironment: String {


I think they can go in individual files, easier to read/find than UpstreamIntegrationTestEnums.

Tests/UpstreamIntegrationTests/util/UpstreamIntegrationTestEnums.swift

emdobrin · 2023-03-31T21:12:43Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+
+        let waitForRegistration = CountDownLatch(1)
+        MobileCore.setLogLevel(.trace)
+        MobileCore.configureWith(appId: "94f571f308d5/6b1be84da76a/launch-023a1b64f561-development")


will we capture in a different task the set up of different app ids per the selected environment?

I don't mind setting up the required properties in this task, but I'm not sure how to create one in the stage version of Experience Platform? I don't seem to have access to any orgs in that env

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

Update usage site in test setup

codecov · 2023-04-04T00:02:48Z

Codecov Report

Merging #321 (18867c9) into feature/upstream-integration-tests (079595c) will not change coverage.
The diff coverage is n/a.

❗ Current head 18867c9 differs from pull request most recent head b536150. Consider uploading reports for the commit b536150 to get more accurate results

@@                         Coverage Diff                         @@
##           feature/upstream-integration-tests     #321   +/-   ##
===================================================================
  Coverage                               96.54%   96.54%           
===================================================================
  Files                                      28       28           
  Lines                                    1649     1649           
===================================================================
  Hits                                     1592     1592           
  Misses                                     57       57

Move NetworkTestingDelegate to network file

timkimadobe

Thanks for the followup review @emdobrin! Addressed feedback with latest updates, with some outstanding questions

Tests/FunctionalTests/util/FunctionalTestBase.swift

timkimadobe · 2023-04-03T23:16:31Z

Tests/KonductorIntegrationTests/EdgeNetworkIntegrationTests.swift

+    private var edgeLocationHint: EdgeLocationHint? = nil
+
+    /// Edge Network (Konductor) environment levels that correspond to their deployment environment levels
+    enum EdgeEnvironment: String {


Ok I've split them into separate files and kept the common enum extraction logic in a shared file called "EnumUtils", please let me know what you think!

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

Tests/UpstreamIntegrationTests/util/UpstreamIntegrationTestEnums.swift

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

timkimadobe · 2023-04-04T00:02:18Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+
+        let waitForRegistration = CountDownLatch(1)
+        MobileCore.setLogLevel(.trace)
+        MobileCore.configureWith(appId: "94f571f308d5/6b1be84da76a/launch-023a1b64f561-development")


I don't mind setting up the required properties in this task, but I'm not sure how to create one in the stage version of Experience Platform? I don't seem to have access to any orgs in that env

timkimadobe · 2023-04-04T00:07:54Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+
+    override func setUp() {
+        let networkService = IntegrationTestNetworkService()
+        networkService.testingDelegate = testingDelegate


Love this idea, thank you! I think it definitely makes the network response testing logic more robust, and easier to write.

I can create a new task to flesh out this suggested implementation. In the meantime, I've moved the NetworkTestingDelegate class into the IntegrationTestNetworkService file as suggested

timkimadobe · 2023-04-04T00:13:56Z

Makefile

+		echo '---------------------------------------------------------------'; \
+		echo ''; \
+	fi; \
+	xcodebuild test \


It looks like even with test failure, -quiet is a bit too quiet; it doesn't provide any of the output from XCTAssertEqual and similar on test failure (I guess their output is not considered a warning or error level log?)
see example job: https://github.com/timkimadobe/aepsdk-edge-ios/actions/runs/4602180121/jobs/8130873840

However, I found a helper action that can take the .xcresult file and create github action summary: https://github.com/timkimadobe/aepsdk-edge-ios/actions/runs/4602401813/jobs/8131396224

The only downside is I don't know how useful that is for the integration case (where the run is triggered by a different system like Jenkins) and console log text would be preferable..

addb · 2023-04-04T23:48:22Z

Makefile

+	-scheme UpstreamIntegrationTests \
+	-destination 'platform=iOS Simulator,name=iPhone 14' \
+	-derivedDataPath build/out \
+	-resultBundlePath iosresults.xcresult \


We should add folder to the coverage to avoid conflicts with coverage from our unit and functional tests
-resultBundlePath coverage/upstreamIntegrationTest/iosresults.xcresult

nit: command name test-integration-upstream-ios

@emdobrin does it make sense to have it with tvOS as well?

In my opinion we should thoroughly test the various platforms at functional test level and keep this simple for now, we can revisit this if we have platform specific test cases in the future.

Updated -resultBundlePath to suggested value, and I will hold off on updating command name until we have platform specific integration tests

emdobrin

Thanks Tim, looks great

emdobrin · 2023-04-04T20:30:29Z

.github/workflows/upstream-integration-test.yml

+        show-code-coverage: false
+      if: success() || failure()
+      # ^ This is important because the action will be run
+      # even if the test fails in the previous step.


usually you specify that per test file - https://developer.apple.com/documentation/xctest/xctestcase/1496260-continueafterfailure
or is this check in place for failures across test suites?

Also the xcresult looks awesome, we can give it a try to see how it works as we start adding new tests and the output increases.
Nit: can we rename the results to test-integration-upstream-results so they match

For the test results we can explore other output formats if needed, for now I think we should be ok as long as the status on the job is reported correctly success/fail

I believe this if statement for unconditional run on success/failure is at the GitHub Action job level; that is, if the Xcode test step emits a failure code (because of a test case fail, or otherwise) then this step will still run to create the test report

Updated test result name to suggested value, and as we discussed earlier I will move this step along with refinements to xcode test output to a separate followup PR for more comprehensive review by the team!

emdobrin · 2023-04-04T20:40:16Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+        if let locationHint = EdgeLocationHint() {
+            self.edgeLocationHint = locationHint
+        }


nit: this can be self.edgeLocationHint = locationHint

Updated! And added hint comment to note the default value logic for the environment variables

addb · 2023-04-05T00:09:47Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+        """#
+
+        // MARK: Response Event assertions
+        registerEdgeLocationHintListener() { event in


Can this be extracted to function ex: ValidateLocationHintResponseHandleEvent().

So your test case can be

test { test api validateNetworkReq() validateResponseEvent() }

Or we could have a common function that can be used for registering listeners for a specific handle type, locationHint being one of them.

We may need to also have support for testing multiple types and multiple handles for the same type in one test, or assert no other unexpected handles are received.
e.g.:

expectHandles(withType: "locationHint:result", times: 1) expectHandles(withType: "other:example", times: 2, expectedHandles: [{...}, {...}]) ignoreUnexpectedHandles(false)

Again, I think this is outside the scope of this PR, but something we should consider.

I can work on creating this kind of testing utility in a follow up PR that implements the specific test cases (covered by task #320)

addb · 2023-04-05T00:10:44Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+            {
+              "ttlSeconds" : 1800,
+              "scope" : "EdgeNetwork",
+              "hint" : "or2"


@emdobrin
If github action is triggered from some other region, the location hint might be different right?

@timkimadobe Should we base expectation based on the input value for the githubAction? Or this is just a setup and will be made dynamic in the upcoming PR?

@addb you are correct, we should capture that when we start adding test cases in future PRs. This PR should account for the GitHub action setup and making the configured params accessible to the test, this is more of an example.

@addb yes you're right, my understanding is that this should be dynamic based on either:

the input value for the action triggering the workflow

if no input value is set (defaults to no explicit location hint), then we can validate the returned hint is within the expected set

As @emdobrin mentioned, this will be implemented in the PR that creates all the specific test cases; this is a placeholder for now

timkimadobe

Thanks for the review @addb and @emdobrin! Addressed feedback with the latest update

timkimadobe · 2023-04-05T18:58:26Z

.github/workflows/upstream-integration-test.yml

+        show-code-coverage: false
+      if: success() || failure()
+      # ^ This is important because the action will be run
+      # even if the test fails in the previous step.


I believe this if statement for unconditional run on success/failure is at the GitHub Action job level; that is, if the Xcode test step emits a failure code (because of a test case fail, or otherwise) then this step will still run to create the test report

Updated test result name to suggested value, and as we discussed earlier I will move this step along with refinements to xcode test output to a separate followup PR for more comprehensive review by the team!

timkimadobe · 2023-04-05T18:59:27Z

Makefile

+	-scheme UpstreamIntegrationTests \
+	-destination 'platform=iOS Simulator,name=iPhone 14' \
+	-derivedDataPath build/out \
+	-resultBundlePath iosresults.xcresult \


Updated -resultBundlePath to suggested value, and I will hold off on updating command name until we have platform specific integration tests

timkimadobe · 2023-04-05T19:07:28Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+        if let locationHint = EdgeLocationHint() {
+            self.edgeLocationHint = locationHint
+        }


Updated! And added hint comment to note the default value logic for the environment variables

timkimadobe · 2023-04-05T19:10:08Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+            {
+              "ttlSeconds" : 1800,
+              "scope" : "EdgeNetwork",
+              "hint" : "or2"


@addb yes you're right, my understanding is that this should be dynamic based on either:

the input value for the action triggering the workflow

if no input value is set (defaults to no explicit location hint), then we can validate the returned hint is within the expected set

As @emdobrin mentioned, this will be implemented in the PR that creates all the specific test cases; this is a placeholder for now

timkimadobe · 2023-04-06T18:39:14Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+        """#
+
+        // MARK: Response Event assertions
+        registerEdgeLocationHintListener() { event in


I can work on creating this kind of testing utility in a follow up PR that implements the specific test cases (covered by task #320)

…adobe#321) * initial e2e test * update pods for e2e test target * update workflow config settings * implement pods cache to prevent high macos usage time * Fix method typo in functional test helper * test for locationHint * Update integration test case * Remove specific hint value * update action to include build cache step * update buildcache to upload logs for debug * fix flag typo * trying different cache system * try with command line settings * update to include key * fix var bug in makefile command * remove playground * Remove build cache testing * passing env vars to test target working * Update e2e flow action, yaml, and test scheme for konductor env * Fix action command to updated name * Updating action text and defaults * Update make command for e2e test to improve documentation add conditional check warning output for KONDUCTOR_ENV * update test names * update job name * Update env var extraction logic * update test to use edge location hint * update action to have location hint options * fix github action var name * Update makefile env vars and include edge location hint * update makefile integration test docs test colon in test run echo update env var extraction documentation in test file * update test output to use raw string values * test using shell if else for job run id * fix spacing syntax * Update to user proper Edge Network name * Update to EDGE_ENVIRONMENT * Small doc updates * Update documentation to correct links * Update dropdown description * Add quiet flag to xcodebuild command for integration test * Rename integration test make command Reorder position to under tvOS test * Update name to UpstreamIntegrationTests Test empty string option for dropdown * Update action script to use updated make command * Split enums into separate file, add env var extraction inits * run pod install after new target name * Remove FunctionalTestBase inheritance and unused compile sources * Add validation for network request portion * Rename and simplify IntegrationTestNetworkService to only required methods and properties * Update to use raw multiline string instead of resource files * Revert podfile lock version upgrade * Update action job name to align with makefile name * Update target name in makefile * Remove extra newlines Remove marketing version from integration test debug target build settings * Remove implementation specific documentation * Test spacing in makefile * Test remove silent option * test moving back to the bottom * test updating command name * test newline * test name change again * Revert all test name changes - the workflow error was not specifying the correct branch to run off of * Add job failure help text * Add doc comment to on failure step * Revert changes in functional test utils * Split test enums into separate files with shared util method * Clean up comments and update test method * Update EdgeEnvironment enum to have .prod default value Update usage site in test setup * Update to include xcresult visualizer tool * Update with example test failure case * Update env file ID to use helper method Move NetworkTestingDelegate to network file * Omit coverage from test results to save characters * Fix clang flag warning from cocoapods * Remove Xcode report tool step from this PR * Update result bundle path to avoid name conflicts with other tests * Add testing documentation comment hint

* Edge Network (Konductor) integration test with GitHub Action workflow (#321) * initial e2e test * update pods for e2e test target * update workflow config settings * implement pods cache to prevent high macos usage time * Fix method typo in functional test helper * test for locationHint * Update integration test case * Remove specific hint value * update action to include build cache step * update buildcache to upload logs for debug * fix flag typo * trying different cache system * try with command line settings * update to include key * fix var bug in makefile command * remove playground * Remove build cache testing * passing env vars to test target working * Update e2e flow action, yaml, and test scheme for konductor env * Fix action command to updated name * Updating action text and defaults * Update make command for e2e test to improve documentation add conditional check warning output for KONDUCTOR_ENV * update test names * update job name * Update env var extraction logic * update test to use edge location hint * update action to have location hint options * fix github action var name * Update makefile env vars and include edge location hint * update makefile integration test docs test colon in test run echo update env var extraction documentation in test file * update test output to use raw string values * test using shell if else for job run id * fix spacing syntax * Update to user proper Edge Network name * Update to EDGE_ENVIRONMENT * Small doc updates * Update documentation to correct links * Update dropdown description * Add quiet flag to xcodebuild command for integration test * Rename integration test make command Reorder position to under tvOS test * Update name to UpstreamIntegrationTests Test empty string option for dropdown * Update action script to use updated make command * Split enums into separate file, add env var extraction inits * run pod install after new target name * Remove FunctionalTestBase inheritance and unused compile sources * Add validation for network request portion * Rename and simplify IntegrationTestNetworkService to only required methods and properties * Update to use raw multiline string instead of resource files * Revert podfile lock version upgrade * Update action job name to align with makefile name * Update target name in makefile * Remove extra newlines Remove marketing version from integration test debug target build settings * Remove implementation specific documentation * Test spacing in makefile * Test remove silent option * test moving back to the bottom * test updating command name * test newline * test name change again * Revert all test name changes - the workflow error was not specifying the correct branch to run off of * Add job failure help text * Add doc comment to on failure step * Revert changes in functional test utils * Split test enums into separate files with shared util method * Clean up comments and update test method * Update EdgeEnvironment enum to have .prod default value Update usage site in test setup * Update to include xcresult visualizer tool * Update with example test failure case * Update env file ID to use helper method Move NetworkTestingDelegate to network file * Omit coverage from test results to save characters * Fix clang flag warning from cocoapods * Remove Xcode report tool step from this PR * Update result bundle path to avoid name conflicts with other tests * Add testing documentation comment hint * Flexible JSON comparison system (#332) * Implement JSON comparison system * Update to use AEPServices AnyCodable * Move assertion helpers to AnyCodableUtils Update assertion methods to accept file and line args for inline errors on test failure Update key path logic to pretty print * Create flexible validation system, with exact match pathing * Exact match wildcard example * Update general wildcard logic to apply to all elements Rename arguments and usage sites * Complete revamp of flexible json comparison system * Convert AnyCodable test assertion helpers to protocol with default implementations Update test classes to adhere to new protocol and update usages Update flexible assertion to use single method with parameter for default mode Clean up code and update documentation for helper methods Fix bug with escaped keys and add unit test case * Apply swift lint * Extract AnyCodable array extension into separate file * Remove unused test setup methods * Remove redundant test cases covered by AnyCodable unit tests * Switch from protocol to XCTestCase extension Update usages Update documentation text * Update keyPathAsString signature and usages * Simplify implementation of AnyCodable array comparison * Remove unused EventSpec * Update filenames Remove Bool return from assertEqual methods Refactor to allow single public API for assertEqual * Update flexible comparison APIs * Add additional test cases for AnyCodable unit tests * Apply swift lint formatting * Update to use positive case * Update method order Update method signature styling * Update actual condition * Update flexible text setup to propagate file and line Upgrade getCapturedRegexGroups to a test failure Upgrade alternate path index errors to TEST ERROR, add file and line * Fix incorrect find and replace * Update regex capture to handle special cases Add test cases that validate special cases * Extract regex logic into shared function * Update shared testing utilities for integration tests (#334) * Move shared test files * Refactor FunctionalTestBase and FunctionalTestNetworkService to allow for dual mode This allows for sharing common test utilities between functional and integration tests * Renaming all usages of FunctionalTestBase to updated name * Rename mock mode bool * Update method docs and cleanup code comments * Update initializer param Create separate data structs for mocked and real network responses Update connectAsync logic structure * Move XCTestCase+AnyCodableAsserts to shared test utils Update method docs for TestNetworkService Update mock bool in TestBase Move AnyCodable helper methods to shared test utils * Update TestConstants name and usages * Consolidate networkResponses into single data struct With mocked behavior controlled by mockNetworkService flag * Update FunctionalTestConst usages * Temporarily moving TestNetworkService for review * Update set get logic for network responses Consolidate awaitRequest logic Simplify NetworkRequest construction Use force unwrap for test constructions * Move NetworkRequest extension to FTNS * Integration tests - TestNetworkService mock and server API split (#337) * Move FTNS to shared test utils * Rename FTNS to TNS * Initial split of mock and server test network service implementations Before removal of network logic from TestBase * Migrate TestBase network APIs to TestNetworkService Move mock or server specific APIs to respective child classes * Update log source name Update reset test expectations logic * Remove resetting network service logic from test base * Rename delayed network response param and var Remove outdated method docs for return value * Update setup logic Update reset test expectations usage * Update network service to static immutable var Update usages to Self * Remove todo as task is wont do * Update mock response API param name * Update integration test to use static network service Also add helper comments to the purpose of the different setup methods * Rename Server to Real TNS Update base class inheritance for both mock and real TNS Update base TNS to be a shared network request helper Update usages within mock and real TNS to use new shared helper as instance var Add passthrough APIs for both mock and real TNS to access helper methods as needed Refactor static networkService in test class to instance to remove need for Self prefix Update associated setUp logic Add mock prefix to networkService in functional test for clarity * Rename mock and real network service classes * Clean up implementation Update doc comment Add implementation note for isEqual * Move shared network service logic to helper Update usage sites Remove unused import * Remove unused imports * Move NetworkRequest flatten body method into NetworkRequest extension Update usages * Update access level * Update doc class names * Remove unneeded commented code * Move testbase debug flag to per case setup Update doc comment * Refactor CompletionHandlerFunctionalTests to use MockNetworkService * Refactor EdgeConsentTests to use MockNetworkService * Add networkService reset * Add NetworkService reset * Refactor EdgePublicAPITests to use MockNetworkService * Refactor AEPEdgePathOverwriteTests Add test flow doc comments * Refactor IdentityStateFunctionalTests Move debug flag set to after setup * Refactor NoConfigFunctionalTests Update doc comment for what method is used to determine equality * Refactor SampleFunctionalTests * Apply lint autocorrect to PR files * Remove unneeded commented code * Integration test cases (#346) * Implementation notes * WIP invalid datastream test * Add test cases Complex XDM, complex data Preset location hint expected error dataset ID expected error invalid location hint * Update setExpectation API to accept only NetworkRequest Update docs Update usages of updated API Update networkService API usages in integration test class * Clean up code comments * Fix for unpassed params * Update class docs Add test note on how JSON comparison system works * Apply swift lint autocorrect * Rename vars and add additional assert on response count Add changes lost in rebase * Updated flexible JSON comparison notes Refactored assertEdgeResonseHandle API and usages Refactored location hint test case to use single variable for location hint value * Update network service class docs * Update first test case to have two event validation * Remove unused API and refactor used API to call helper directly * Update assertEdgeResponseHandle signature and usages Create new helper methods to construct interact URLs with location hint * Add strict count assertion for all responses Update matchedResponses name to be uniform across test suite * Update assertEdgeResponseError method to use getDispatchedEventsWith directly Revert refactor of assertEdgeResponseHandle and usages Add expectedCount argument to both APIs and update assertion logic to check all events Add org ID related exact match assertions and exact match paths (non-wildcard) Add exact match requirement for error type URL * Remove unused API (actually) * Update test cases to use direct assertions * Refactor assert*Match APIs to non-nil expected AnyCodable * Update assertExpectedEvents API to allow for custom timeout period * Update to use conditional check on location hint if initial value is set Simplify 2x event handle test case Add longer timeout for event expectations * Update test case setup Remove outdated API comment * Remove unused helper API * Refactor functional tests from dev branch to use new testing utilities * Use longer extension registration timeout * Add per test case teardown network service reset * Test extending teardown sleep duration * Add higher timeout value for startup event validation * Add sleep to allow more buffer for listener registration * Extend setup timeout to 10s * Restore original teardown sleep time * Remove sleep from test case * Change order of operations for test teardown * Test unregistering instrumented extension in teardown process * Add sleep to startup process to allow event hub setup time * Add mock network service teardown to all functional tests * Trigger CI workflow * Revert changes in TestBase teardown * Integration testing workflow update (#349) * Add new integration test job * Update device for integration to 8 * Update integration test job name * Remove extra space * Remove code coverage upload from integration job * Test job conditional in circleci workflow * Add non conditional step * Update triggering branches * Update initiating branch name condition to staging * test fetching branch name from fork * remove spaces from command * Test extract both base and head branch names * Add semicolons * Update extraction commands * Only fetch base branch name Make integration setup steps non conditional * Update job trigger conditions * Update simulator for integration test to 14 * Update integration test workflow to use conditional at job level * Update integration test job Xcode version * Fix conditional in integration test job * Move checkout step to non-conditional level * Test current branch name also triggers integration test workflow * Test branch name 2 * Remove test branch name * Add main branch to check * Update deployment version for integration target to 11 * Make integration test make command result output consistent with other tests Add removal of existing old results part of the job like other tests * Refactor to remove branch name conditionals from job level and use workflow job branch filter * Test workflow job filter when set on current branch * Update filter criteria * Remove test branch filter * Reorder jobs so conditional ones come after mandatory * Update integration test target in podfile --------- Co-authored-by: Emilia Dobrin <33132425+emdobrin@users.noreply.github.com>

timkimadobe added 30 commits February 28, 2023 15:43

initial e2e test

ca433ce

update pods for e2e test target

98475ee

update workflow config settings

541e94a

implement pods cache to prevent high macos usage time

8e39b4a

Fix method typo in functional test helper

a235c46

test for locationHint

92e5a87

Update integration test case

ac26527

Remove specific hint value

8f11dd5

update action to include build cache step

1ad14a6

update buildcache to upload logs for debug

747c989

fix flag typo

ef4e780

trying different cache system

727ae7d

try with command line settings

ffa3613

update to include key

31a448f

fix var bug in makefile command

0ec64c7

remove playground

777a388

Remove build cache testing

704326a

passing env vars to test target working

b8bb181

Update e2e flow action, yaml, and test scheme for konductor env

d227e24

Fix action command to updated name

b5dc0d1

Updating action text and defaults

2d5d491

Update make command for e2e test to improve documentation

30c4dcf

add conditional check warning output for KONDUCTOR_ENV

update test names

0860806

update job name

1de75b9

Update env var extraction logic

cbc1f54

update test to use edge location hint

825dd1c

update action to have location hint options

29d73bd

fix github action var name

92edef3

Update makefile env vars and include edge location hint

b1a1269

update makefile integration test docs

af9648b

test colon in test run echo update env var extraction documentation in test file

emdobrin reviewed Mar 31, 2023

View reviewed changes

timkimadobe added 6 commits April 3, 2023 16:09

Revert changes in functional test utils

bc0a4f8

Split test enums into separate files with shared util method

8176e69

Clean up comments and update test method

9344412

Update EdgeEnvironment enum to have .prod default value

2ca27be

Update usage site in test setup

Update to include xcresult visualizer tool

9e7482f

Update with example test failure case

8e5b9aa

Update env file ID to use helper method

836f7fe

Move NetworkTestingDelegate to network file

timkimadobe commented Apr 4, 2023

View reviewed changes

timkimadobe added 2 commits April 3, 2023 17:21

Omit coverage from test results to save characters

b21da6f

Fix clang flag warning from cocoapods

18867c9

timkimadobe changed the base branch from dev to feature/upstream-integration-tests April 4, 2023 00:27

timkimadobe requested a review from emdobrin April 4, 2023 23:12

addb reviewed Apr 4, 2023

View reviewed changes

emdobrin reviewed Apr 4, 2023

View reviewed changes

addb reviewed Apr 5, 2023

View reviewed changes

timkimadobe added 3 commits April 5, 2023 12:05

Remove Xcode report tool step from this PR

e6e5703

Update result bundle path to avoid name conflicts with other tests

265f058

Add testing documentation comment hint

b536150

timkimadobe commented Apr 6, 2023

View reviewed changes

timkimadobe requested review from addb and emdobrin April 6, 2023 19:13

emdobrin approved these changes Apr 6, 2023

View reviewed changes

timkimadobe merged commit cd5b992 into adobe:feature/upstream-integration-tests Apr 6, 2023

timkimadobe linked an issue Apr 25, 2023 that may be closed by this pull request

Update integration testing script to allow non-file-modifying configuration settings #317

Closed

Conversation

timkimadobe commented Mar 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Overview of workflow

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Apr 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

timkimadobe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emdobrin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emdobrin Apr 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timkimadobe commented Mar 28, 2023 •

edited

Loading

codecov bot commented Apr 4, 2023 •

edited

Loading

emdobrin Apr 5, 2023 •

edited

Loading