Flexible JSON comparison system by timkimadobe · Pull Request #332 · adobe/aepsdk-edge-ios

timkimadobe · 2023-04-24T18:02:52Z

Description

This PR extends the capability from #325 to allow for flexible validation which means only the keys from the expected JSON side are required. There are two default modes for flexible validation:

Default type match mode - when comparing values, only the type needs to match (and be non-nil).
Default exact match mode - when comparing values, the value needs to match exactly.

Each mode has the option to specify key paths that switch the default behavior to the other mode from that key onward.

This allows for keeping expected JSON minimal; you only have to define and maintain what keys and values are relevant for your test case.

Use case

This supports the use case of validating only sub-portions from upstream responses (ex: Edge Network) that the Edge Network extension is interested in and allows testing to better align with real usage of upstream system responses in business logic today.

Example: Extraction of location hint from Edge Network response
Edge Network extension is only interested in

Even though the full response from Edge Network for location hint contains:

{
   "payload": [
    {
     "ttlSeconds" : 1800,
     "scope" : "Target",
     "hint" : "35"
  },
  {
     "ttlSeconds" : 1800,
     "scope" : "AAM",
     "hint" : "9"
  },
  {
     "ttlSeconds" : 1800,
     "scope" : "EdgeNetwork",
     "hint" : "or2"
  }
]
}

Edge Network extension is only interested in (and extracts) the EdgeNetwork hint using a combination of Event.Type and scope: "EdgeNetwork" (see implementation for reference)

{
  "ttlSeconds" : 1800,
  "scope" : "EdgeNetwork",
  "hint" : "or2"
}

By using flexible validation with wildcard matching, it is possible to easily specify only the validation structure, and follow the same logical path as used in the extension itself (an in-order array traversal where first); see the example usage below:

func testFlexibleValidation() {
  let expectedJSON = #"""
  {
    "payload": [
      {
        "ttlSeconds" : 1800,
        "scope" : "EdgeNetwork",
        "hint" : 1
      }
    ]
  }
  """#
  
  let actualJSON = #"""
 {
   "payload": [
     {
       "ttlSeconds" : 1800,
       "scope" : "Target",
       "hint" : "35"
     },
     {
       "ttlSeconds" : 1800,
       "scope" : "AAM",
       "hint" : "9"
     },
     {
       "ttlSeconds" : 1800,
       "scope" : "EdgeNetwork",
       "hint" : "or2"
     }
   ]
 }
 """#
  
  let expected = getAnyCodable(expectedJSON)!
  let actual = getAnyCodable(actualJSON)!
  AnyCodableUtils.assertContainsDefaultTypeMatch(
      expected: expected,
      actual: actual,
      exactMatchPaths: ["payload[*].scope"]
  )
}

Simplification of Consent for Edge Network test case assertions:
From: https://github.com/emdobrin/aepsdk-edge-ios/blob/templates/Tests/FunctionalTests/Edge%2BConsentTests.swift#L193
To:

aepsdk-edge-ios/Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

Line 339 in 52c8010

func testConsentUpdate_onlyExpectedKeys_usingGeneralWildcard() {

This greatly improves test logic alignment with business logic, reduces the burden of maintenance on test writers, and helps improves test clarity.

Logic

The type/exact match paths array is converted into a logical tree
Each validation step checks the exact match tree to see if flexible or exact matching validation should be used
If validation needs to go deeper, the current step provides the validation tree of the next step to the next call

Because of the recursive nature of the programmatic traversal, any combination of flexible and/or exact matching can be used, and this also works with wildcard matching.

Array validation logic works as follows:

Standard behavior is to use the validation array as the primary traversal with the value on either side with equality following the same flexible validation
- If validation side has fewer elements, validation passes and the rest of the items in the input side are ignored

flowchart LR
    A("0") -->|"flexible match ✅"| B("0")
    C("1") -->|"flexible match ✅"| D("1")
               F("2")
               G("Test passes ✅")
    subgraph Validation
    A
    C
    end

    subgraph Input
    B
    D
    F
    end

    subgraph Result
    G
    end

Test writer can specify special matching cases in the following ways:
1. Specific index, using [<INT>] notation (ex: [3])
  - This means the specific index should use the alternate validation mode
2. Wildcard with index, using [*<INT>] notation (ex: [*0], [*12], etc.)
  - This means the specific index should use wildcard matching logic and the alternate validation mode
3. General wildcard, using [*] notation, which applies wildcard matching to all indexes on the expected side not already covered by 1 or 2

Precedence:

Explicit index exact matching (ex: key1[0])
Wildcard with index matching (ex: key1[*2])
General wildcard (ex: key1[*])
Unspecified indexes (that is, ones not covered by 1-3)

Known limitations

Regex

Array index regex does not do strong validation from the end of the string
- Ex: key1[0]key2[1] still gets extracted as array access [0] [1] but string validation will extract key1 (up to the first open square bracket)

Related Issue

Motivation and Context

As seen in the use cases examples and example test cases implemented in UpstreamIntegrationTests.swift, JSON validation is a requirement of our SDK architecture. The test cases cover a broad range of intended behavior, and currently it is up to test writers to manually implement digging into JSON structures, write their own validation logic, and maintain these across many repos.

This comparison tool allows for bringing all the complexity of JSON comparison and validation into a single spot which has two primary benefits:

Rigorously test the system itself both through dedicated unit tests and tests reliant on the system, which creates a robust, battle tested, reliable backing validation system for all our tests across all repos
Assertion logic is greatly simplified and easy to follow and maintain
- Copy-paste-ability of expected JSON structures (for example, from Assurance, Charles, test logs, etc.)
- No need to create Swift dictionaries, use if let/guard as? [String: Any] casting, manually digging into dictionaries, etc.
- Flexible validation with modifiable strictness

All this is to say that the complexity of JSON validation is required somewhere; if we use a shared, powerful backing system, then instead of having to reinvent the JSON validation wheel in every test suite we can focus on creating even more test cases to validate the SDK logic.

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

I have signed the Adobe Open Source CLA.
My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

Update assertion methods to accept file and line args for inline errors on test failure Update key path logic to pretty print

Rename arguments and usage sites

…plementations Update test classes to adhere to new protocol and update usages Update flexible assertion to use single method with parameter for default mode Clean up code and update documentation for helper methods Fix bug with escaped keys and add unit test case

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

emdobrin · 2023-04-26T23:48:17Z

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

+    }
+
+    /// Performs flexible comparison testing assertions between two `[AnyCodable]` instances.
+    private func assertFlexibleEqual(expected: [AnyCodable]?, actual: [AnyCodable]?, keyPath: [Any], pathTree: [String: Any]?, defaultExactEqualityMode: Bool, file: StaticString = #file, line: UInt = #line, shouldAssert: Bool = true) -> Bool {


this method is unexpectedly long L361->L632 😅
can you split some of the logic in dedicated functions (max 50-80lines) to increase readability?

Thank you! That totally makes sense, and based on your feedback I found a lot of areas where there is repeated logic, just with different toggles; I've updated this method with the following changes:

Condensed shared base logic into inner functions

Added documentation above each inner function on what state values it relies on and/or mutates

I used inner functions because there is shared mutated state between calls; this could be handled by an inout/returned mutated state, but I thought since these particular functions are only used in the array comparison context it made more sense to keep them inline. I think it still helps with separating the logic and making it easier to reason about without all the extra overhead of explicitly passing/handling all the shared state at the call site. What do you think?

Updated a lot of the nested logic to use guards instead to reduce pyramid of dooming - thank you! I think it looks a lot cleaner

Upgraded the print warnings to explicit test failures, to help test writers maintain cleaner assertions and not (relatively) silently allow conflicting assertions

Added more documentation on the logical flow of the method

The overall function itself does still exceed 80 lines (195 lines), but I feel that splitting it further could make understanding the logical progression harder, since this method performs all the required array logic in place, in order. And a lot of the lines come from:

Multiline strings

Method calls split over multiple lines for clarity (both methods with long param lists, and chained method calls)

Tests/UpstreamIntegrationTests/AnyCodableUtilsTests.swift

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

emdobrin · 2023-04-27T19:37:52Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+        }
+        """#
+
+        let actualJSON = #"""


I assume we will take the actualJSON from the network request body for these tests?

Yes! These were just placeholder examples to demonstrate the capabilities; will update these test cases with the test case implementation PR to follow

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

emdobrin · 2023-04-27T19:43:17Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

    }
-}

+    // MARK: - Instrumented Extension helpers


L767->L892 Since these are copied from the functional test helpers, I would like to extract those so they can be reused across both integration&functional tests.
Feel free to include that into another PR, but let's at least remove them from this PR if possible or add a todo to remove the duplicated code

That sounds great! Removed these methods and put a placeholder TODO, to be updated in the upcoming test implementation PR

Update usages Update documentation text

timkimadobe

Thank you so much for the review and feedback @emdobrin! Addressed all feedback with latest update to PR, with some outstanding questions

Tests/UpstreamIntegrationTests/AnyCodableUtilsTests.swift

timkimadobe · 2023-04-27T20:54:50Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

+        }
+        """#
+
+        let actualJSON = #"""


Yes! These were just placeholder examples to demonstrate the capabilities; will update these test cases with the test case implementation PR to follow

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

timkimadobe · 2023-04-27T20:57:35Z

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

    }
-}

+    // MARK: - Instrumented Extension helpers


That sounds great! Removed these methods and put a placeholder TODO, to be updated in the upcoming test implementation PR

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

timkimadobe · 2023-04-27T23:35:25Z

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

+    }
+
+    /// Performs flexible comparison testing assertions between two `[AnyCodable]` instances.
+    private func assertFlexibleEqual(expected: [AnyCodable]?, actual: [AnyCodable]?, keyPath: [Any], pathTree: [String: Any]?, defaultExactEqualityMode: Bool, file: StaticString = #file, line: UInt = #line, shouldAssert: Bool = true) -> Bool {


Thank you! That totally makes sense, and based on your feedback I found a lot of areas where there is repeated logic, just with different toggles; I've updated this method with the following changes:

Condensed shared base logic into inner functions

Added documentation above each inner function on what state values it relies on and/or mutates

I used inner functions because there is shared mutated state between calls; this could be handled by an inout/returned mutated state, but I thought since these particular functions are only used in the array comparison context it made more sense to keep them inline. I think it still helps with separating the logic and making it easier to reason about without all the extra overhead of explicitly passing/handling all the shared state at the call site. What do you think?

Updated a lot of the nested logic to use guards instead to reduce pyramid of dooming - thank you! I think it looks a lot cleaner

Upgraded the print warnings to explicit test failures, to help test writers maintain cleaner assertions and not (relatively) silently allow conflicting assertions

Added more documentation on the logical flow of the method

The overall function itself does still exceed 80 lines (195 lines), but I feel that splitting it further could make understanding the logical progression harder, since this method performs all the required array logic in place, in order. And a lot of the lines come from:

Multiline strings

Method calls split over multiple lines for clarity (both methods with long param lists, and chained method calls)

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

emdobrin · 2023-04-28T01:13:57Z

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

+    /// - The default behavior is that elements from the expected JSON side are compared in order, up to the last element of the expected array
+    ///
+    /// - Parameters:
+    ///    - defaultExactEqualityMode: The default mode to use for the validation process. `true` uses exact match, and values require


hmm.. it still seems confusing. What do you think about providing convenience wrappers for these asserts for more clarity, so instead of parametrized assertContains we have for e.g.:

assertTypeMatch(expected, actual, exactMatchPaths) -> type match, except exactMatchPaths assertTypeMatch(expected, actual) -> all type match assertExactMatch(expected, actual, typeMatchPaths) -> exact match, except typeMatchPaths assertExactMatch(expected, actual) -> all exact match assertEqual(expected, actual)

Remove Bool return from assertEqual methods Refactor to allow single public API for assertEqual

timkimadobe

Thank you for the follow up review @emdobrin! Addressed feedback with latest update, with some outstanding questions

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

timkimadobe · 2023-04-28T02:36:48Z

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

+    /// - The default behavior is that elements from the expected JSON side are compared in order, up to the last element of the expected array
+    ///
+    /// - Parameters:
+    ///    - defaultExactEqualityMode: The default mode to use for the validation process. `true` uses exact match, and values require


That sounds good! I've updated to the suggested signatures and updated usage sites

Tests/UpstreamIntegrationTests/UpstreamIntegrationTests.swift

Tests/UpstreamIntegrationTests/util/XCTestCase+AnyCodableAsserts.swift

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

Update method signature styling

timkimadobe

Thank you for the review @emdobrin! Addressed all feedback with latest update

Tests/UpstreamIntegrationTests/util/XCTestCase+AnyCodableAsserts.swift

Tests/UpstreamIntegrationTests/util/AnyCodableUtils.swift

emdobrin

looks good, and let's resolve the todos and commented steps in the followup pr.

Upgrade getCapturedRegexGroups to a test failure Upgrade alternate path index errors to TEST ERROR, add file and line

Add test cases that validate special cases

Tests/UpstreamIntegrationTests/util/XCTestCase+AnyCodableAsserts.swift

timkimadobe

Thanks for the review @emdobrin! Updated according to feedback with latest update

Tests/UpstreamIntegrationTests/util/XCTestCase+AnyCodableAsserts.swift

* Implement JSON comparison system * Update to use AEPServices AnyCodable * Move assertion helpers to AnyCodableUtils Update assertion methods to accept file and line args for inline errors on test failure Update key path logic to pretty print * Create flexible validation system, with exact match pathing * Exact match wildcard example * Update general wildcard logic to apply to all elements Rename arguments and usage sites * Complete revamp of flexible json comparison system * Convert AnyCodable test assertion helpers to protocol with default implementations Update test classes to adhere to new protocol and update usages Update flexible assertion to use single method with parameter for default mode Clean up code and update documentation for helper methods Fix bug with escaped keys and add unit test case * Apply swift lint * Extract AnyCodable array extension into separate file * Remove unused test setup methods * Remove redundant test cases covered by AnyCodable unit tests * Switch from protocol to XCTestCase extension Update usages Update documentation text * Update keyPathAsString signature and usages * Simplify implementation of AnyCodable array comparison * Remove unused EventSpec * Update filenames Remove Bool return from assertEqual methods Refactor to allow single public API for assertEqual * Update flexible comparison APIs * Add additional test cases for AnyCodable unit tests * Apply swift lint formatting * Update to use positive case * Update method order Update method signature styling * Update actual condition * Update flexible text setup to propagate file and line Upgrade getCapturedRegexGroups to a test failure Upgrade alternate path index errors to TEST ERROR, add file and line * Fix incorrect find and replace * Update regex capture to handle special cases Add test cases that validate special cases * Extract regex logic into shared function

* Edge Network (Konductor) integration test with GitHub Action workflow (#321) * initial e2e test * update pods for e2e test target * update workflow config settings * implement pods cache to prevent high macos usage time * Fix method typo in functional test helper * test for locationHint * Update integration test case * Remove specific hint value * update action to include build cache step * update buildcache to upload logs for debug * fix flag typo * trying different cache system * try with command line settings * update to include key * fix var bug in makefile command * remove playground * Remove build cache testing * passing env vars to test target working * Update e2e flow action, yaml, and test scheme for konductor env * Fix action command to updated name * Updating action text and defaults * Update make command for e2e test to improve documentation add conditional check warning output for KONDUCTOR_ENV * update test names * update job name * Update env var extraction logic * update test to use edge location hint * update action to have location hint options * fix github action var name * Update makefile env vars and include edge location hint * update makefile integration test docs test colon in test run echo update env var extraction documentation in test file * update test output to use raw string values * test using shell if else for job run id * fix spacing syntax * Update to user proper Edge Network name * Update to EDGE_ENVIRONMENT * Small doc updates * Update documentation to correct links * Update dropdown description * Add quiet flag to xcodebuild command for integration test * Rename integration test make command Reorder position to under tvOS test * Update name to UpstreamIntegrationTests Test empty string option for dropdown * Update action script to use updated make command * Split enums into separate file, add env var extraction inits * run pod install after new target name * Remove FunctionalTestBase inheritance and unused compile sources * Add validation for network request portion * Rename and simplify IntegrationTestNetworkService to only required methods and properties * Update to use raw multiline string instead of resource files * Revert podfile lock version upgrade * Update action job name to align with makefile name * Update target name in makefile * Remove extra newlines Remove marketing version from integration test debug target build settings * Remove implementation specific documentation * Test spacing in makefile * Test remove silent option * test moving back to the bottom * test updating command name * test newline * test name change again * Revert all test name changes - the workflow error was not specifying the correct branch to run off of * Add job failure help text * Add doc comment to on failure step * Revert changes in functional test utils * Split test enums into separate files with shared util method * Clean up comments and update test method * Update EdgeEnvironment enum to have .prod default value Update usage site in test setup * Update to include xcresult visualizer tool * Update with example test failure case * Update env file ID to use helper method Move NetworkTestingDelegate to network file * Omit coverage from test results to save characters * Fix clang flag warning from cocoapods * Remove Xcode report tool step from this PR * Update result bundle path to avoid name conflicts with other tests * Add testing documentation comment hint * Flexible JSON comparison system (#332) * Implement JSON comparison system * Update to use AEPServices AnyCodable * Move assertion helpers to AnyCodableUtils Update assertion methods to accept file and line args for inline errors on test failure Update key path logic to pretty print * Create flexible validation system, with exact match pathing * Exact match wildcard example * Update general wildcard logic to apply to all elements Rename arguments and usage sites * Complete revamp of flexible json comparison system * Convert AnyCodable test assertion helpers to protocol with default implementations Update test classes to adhere to new protocol and update usages Update flexible assertion to use single method with parameter for default mode Clean up code and update documentation for helper methods Fix bug with escaped keys and add unit test case * Apply swift lint * Extract AnyCodable array extension into separate file * Remove unused test setup methods * Remove redundant test cases covered by AnyCodable unit tests * Switch from protocol to XCTestCase extension Update usages Update documentation text * Update keyPathAsString signature and usages * Simplify implementation of AnyCodable array comparison * Remove unused EventSpec * Update filenames Remove Bool return from assertEqual methods Refactor to allow single public API for assertEqual * Update flexible comparison APIs * Add additional test cases for AnyCodable unit tests * Apply swift lint formatting * Update to use positive case * Update method order Update method signature styling * Update actual condition * Update flexible text setup to propagate file and line Upgrade getCapturedRegexGroups to a test failure Upgrade alternate path index errors to TEST ERROR, add file and line * Fix incorrect find and replace * Update regex capture to handle special cases Add test cases that validate special cases * Extract regex logic into shared function * Update shared testing utilities for integration tests (#334) * Move shared test files * Refactor FunctionalTestBase and FunctionalTestNetworkService to allow for dual mode This allows for sharing common test utilities between functional and integration tests * Renaming all usages of FunctionalTestBase to updated name * Rename mock mode bool * Update method docs and cleanup code comments * Update initializer param Create separate data structs for mocked and real network responses Update connectAsync logic structure * Move XCTestCase+AnyCodableAsserts to shared test utils Update method docs for TestNetworkService Update mock bool in TestBase Move AnyCodable helper methods to shared test utils * Update TestConstants name and usages * Consolidate networkResponses into single data struct With mocked behavior controlled by mockNetworkService flag * Update FunctionalTestConst usages * Temporarily moving TestNetworkService for review * Update set get logic for network responses Consolidate awaitRequest logic Simplify NetworkRequest construction Use force unwrap for test constructions * Move NetworkRequest extension to FTNS * Integration tests - TestNetworkService mock and server API split (#337) * Move FTNS to shared test utils * Rename FTNS to TNS * Initial split of mock and server test network service implementations Before removal of network logic from TestBase * Migrate TestBase network APIs to TestNetworkService Move mock or server specific APIs to respective child classes * Update log source name Update reset test expectations logic * Remove resetting network service logic from test base * Rename delayed network response param and var Remove outdated method docs for return value * Update setup logic Update reset test expectations usage * Update network service to static immutable var Update usages to Self * Remove todo as task is wont do * Update mock response API param name * Update integration test to use static network service Also add helper comments to the purpose of the different setup methods * Rename Server to Real TNS Update base class inheritance for both mock and real TNS Update base TNS to be a shared network request helper Update usages within mock and real TNS to use new shared helper as instance var Add passthrough APIs for both mock and real TNS to access helper methods as needed Refactor static networkService in test class to instance to remove need for Self prefix Update associated setUp logic Add mock prefix to networkService in functional test for clarity * Rename mock and real network service classes * Clean up implementation Update doc comment Add implementation note for isEqual * Move shared network service logic to helper Update usage sites Remove unused import * Remove unused imports * Move NetworkRequest flatten body method into NetworkRequest extension Update usages * Update access level * Update doc class names * Remove unneeded commented code * Move testbase debug flag to per case setup Update doc comment * Refactor CompletionHandlerFunctionalTests to use MockNetworkService * Refactor EdgeConsentTests to use MockNetworkService * Add networkService reset * Add NetworkService reset * Refactor EdgePublicAPITests to use MockNetworkService * Refactor AEPEdgePathOverwriteTests Add test flow doc comments * Refactor IdentityStateFunctionalTests Move debug flag set to after setup * Refactor NoConfigFunctionalTests Update doc comment for what method is used to determine equality * Refactor SampleFunctionalTests * Apply lint autocorrect to PR files * Remove unneeded commented code * Integration test cases (#346) * Implementation notes * WIP invalid datastream test * Add test cases Complex XDM, complex data Preset location hint expected error dataset ID expected error invalid location hint * Update setExpectation API to accept only NetworkRequest Update docs Update usages of updated API Update networkService API usages in integration test class * Clean up code comments * Fix for unpassed params * Update class docs Add test note on how JSON comparison system works * Apply swift lint autocorrect * Rename vars and add additional assert on response count Add changes lost in rebase * Updated flexible JSON comparison notes Refactored assertEdgeResonseHandle API and usages Refactored location hint test case to use single variable for location hint value * Update network service class docs * Update first test case to have two event validation * Remove unused API and refactor used API to call helper directly * Update assertEdgeResponseHandle signature and usages Create new helper methods to construct interact URLs with location hint * Add strict count assertion for all responses Update matchedResponses name to be uniform across test suite * Update assertEdgeResponseError method to use getDispatchedEventsWith directly Revert refactor of assertEdgeResponseHandle and usages Add expectedCount argument to both APIs and update assertion logic to check all events Add org ID related exact match assertions and exact match paths (non-wildcard) Add exact match requirement for error type URL * Remove unused API (actually) * Update test cases to use direct assertions * Refactor assert*Match APIs to non-nil expected AnyCodable * Update assertExpectedEvents API to allow for custom timeout period * Update to use conditional check on location hint if initial value is set Simplify 2x event handle test case Add longer timeout for event expectations * Update test case setup Remove outdated API comment * Remove unused helper API * Refactor functional tests from dev branch to use new testing utilities * Use longer extension registration timeout * Add per test case teardown network service reset * Test extending teardown sleep duration * Add higher timeout value for startup event validation * Add sleep to allow more buffer for listener registration * Extend setup timeout to 10s * Restore original teardown sleep time * Remove sleep from test case * Change order of operations for test teardown * Test unregistering instrumented extension in teardown process * Add sleep to startup process to allow event hub setup time * Add mock network service teardown to all functional tests * Trigger CI workflow * Revert changes in TestBase teardown * Integration testing workflow update (#349) * Add new integration test job * Update device for integration to 8 * Update integration test job name * Remove extra space * Remove code coverage upload from integration job * Test job conditional in circleci workflow * Add non conditional step * Update triggering branches * Update initiating branch name condition to staging * test fetching branch name from fork * remove spaces from command * Test extract both base and head branch names * Add semicolons * Update extraction commands * Only fetch base branch name Make integration setup steps non conditional * Update job trigger conditions * Update simulator for integration test to 14 * Update integration test workflow to use conditional at job level * Update integration test job Xcode version * Fix conditional in integration test job * Move checkout step to non-conditional level * Test current branch name also triggers integration test workflow * Test branch name 2 * Remove test branch name * Add main branch to check * Update deployment version for integration target to 11 * Make integration test make command result output consistent with other tests Add removal of existing old results part of the job like other tests * Refactor to remove branch name conditionals from job level and use workflow job branch filter * Test workflow job filter when set on current branch * Update filter criteria * Remove test branch filter * Reorder jobs so conditional ones come after mandatory * Update integration test target in podfile --------- Co-authored-by: Emilia Dobrin <33132425+emdobrin@users.noreply.github.com>

timkimadobe added 8 commits April 6, 2023 14:47

Implement JSON comparison system

4c8cba9

Update to use AEPServices AnyCodable

6877108

Move assertion helpers to AnyCodableUtils

de0593b

Update assertion methods to accept file and line args for inline errors on test failure Update key path logic to pretty print

Create flexible validation system, with exact match pathing

690464a

Exact match wildcard example

5156a6b

Update general wildcard logic to apply to all elements

804fa86

Rename arguments and usage sites

Complete revamp of flexible json comparison system

52c8010

timkimadobe requested review from addb, cacheung, emdobrin and kevinlind April 26, 2023 22:01

Apply swift lint

bfea171

emdobrin requested changes Apr 27, 2023

View reviewed changes

timkimadobe added 6 commits April 27, 2023 14:04

Extract AnyCodable array extension into separate file

efa5431

Remove unused test setup methods

757dd15

Remove redundant test cases covered by AnyCodable unit tests

817cf16

Switch from protocol to XCTestCase extension

b83dc71

Update usages Update documentation text

Update keyPathAsString signature and usages

c9f5981

Simplify implementation of AnyCodable array comparison

bf81937

timkimadobe commented Apr 27, 2023

View reviewed changes

emdobrin reviewed Apr 28, 2023

View reviewed changes

timkimadobe added 5 commits April 27, 2023 18:24

Remove unused EventSpec

885054e

Update filenames

9011554

Remove Bool return from assertEqual methods Refactor to allow single public API for assertEqual

Update flexible comparison APIs

2c71223

Add additional test cases for AnyCodable unit tests

de70ec5

Apply swift lint formatting

ecb46f4

timkimadobe commented Apr 28, 2023

View reviewed changes

emdobrin reviewed Apr 28, 2023

View reviewed changes

emdobrin mentioned this pull request Apr 28, 2023

JSON comparison system - integration tests #325

Closed

10 tasks

timkimadobe added 2 commits April 28, 2023 14:36

Update to use positive case

6177c6d

Update method order

dac8bf5

Update method signature styling

timkimadobe commented Apr 28, 2023

View reviewed changes

Update actual condition

8da72f2

emdobrin approved these changes Apr 28, 2023

View reviewed changes

timkimadobe added 3 commits April 28, 2023 15:15

Update flexible text setup to propagate file and line

008c6f3

Upgrade getCapturedRegexGroups to a test failure Upgrade alternate path index errors to TEST ERROR, add file and line

Fix incorrect find and replace

f384623

Update regex capture to handle special cases

32cc3ad

Add test cases that validate special cases

emdobrin reviewed May 2, 2023

View reviewed changes

Extract regex logic into shared function

8219afd

timkimadobe commented May 3, 2023

View reviewed changes

Tests/UpstreamIntegrationTests/util/XCTestCase+AnyCodableAsserts.swift Outdated Show resolved Hide resolved

Tests/UpstreamIntegrationTests/util/XCTestCase+AnyCodableAsserts.swift Outdated Show resolved Hide resolved

emdobrin approved these changes May 3, 2023

View reviewed changes

emdobrin merged commit 420bb57 into adobe:feature/upstream-integration-tests May 3, 2023

timkimadobe deleted the flexible-json-comarison-system branch May 3, 2023 18:35

emdobrin linked an issue May 4, 2023 that may be closed by this pull request

Integration testing - Investigate options for JSON dictionary comparison #319

Closed

timkimadobe mentioned this pull request Aug 8, 2023

Integration test JSON comparison system adobe/aepsdk-edge-android#79

Merged

10 tasks

Conversation

timkimadobe commented Apr 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Use case

Logic

Known limitations

Regex

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timkimadobe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timkimadobe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

timkimadobe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emdobrin left a comment

Choose a reason for hiding this comment

timkimadobe commented Apr 24, 2023 •

edited

Loading