Skip to content

Integration test cases#346

Merged
emdobrin merged 23 commits intoadobe:feature/upstream-integration-testsfrom
timkimadobe:integration-test-cases-REBASE-post-merge
Jun 29, 2023
Merged

Integration test cases#346
emdobrin merged 23 commits intoadobe:feature/upstream-integration-testsfrom
timkimadobe:integration-test-cases-REBASE-post-merge

Conversation

@timkimadobe
Copy link
Copy Markdown
Contributor

@timkimadobe timkimadobe commented May 18, 2023

Description

This PR:

Implements the integration test cases:

Standard sendEvent

  1. multiple times returns the expected event handles
  2. complex XDM (multiple keys and data types)
  3. complex data (multiple keys and data types)

Standard config changes

  1. valid location hint

Expected errors

  1. invalid datastream ID
  2. invalid location hint value

It takes advantage of the AnyCodable assertion helpers and test methods to easily construct events and assert against event payloads with varying levels of strictness as required per test case.

Updates network service test APIs

to use NetworkRequest instances directly, and for new test cases in the integration test class, instantiates a NetworkRequest to reuse throughout the test.

  • That is, network service APIs use NetworkRequest instances by default, and the previous url string + httpmethod are converted to convenience methods

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • I have signed the Adobe Open Source CLA.
  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Complex XDM, complex data
Preset location hint
expected error dataset ID
expected error invalid location hint
Update docs
Update usages of updated API
Update networkService API usages in integration test class
@timkimadobe timkimadobe changed the title Integration test cases rebase post merge Integration test cases May 18, 2023
@timkimadobe timkimadobe linked an issue May 18, 2023 that may be closed by this pull request
@codecov
Copy link
Copy Markdown

codecov bot commented May 18, 2023

Codecov Report

Merging #346 (36bf720) into feature/upstream-integration-tests (e85aeb9) will not change coverage.
The diff coverage is n/a.

@@                         Coverage Diff                         @@
##           feature/upstream-integration-tests     #346   +/-   ##
===================================================================
  Coverage                               96.61%   96.61%           
===================================================================
  Files                                      28       28           
  Lines                                    1650     1650           
===================================================================
  Hits                                     1594     1594           
  Misses                                     56       56           

// Setup

// Test constructs should always be valid
let standardNetworkRequest = NetworkRequest(urlString: "https://obumobile5.data.adobedc.net/ee/v1/interact", httpMethod: .post)!
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extract the URL to a constant if used repeatedly.

let INTERACT_ENDPOINT = "https://obumobile5.data.adobedc.net/ee/v1/interact"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is something @emdobrin and I discussed previously in terms of test readability; we were thinking especially for outside teams that read the test cases (upon test run failure or otherwise), it would be easier for them to see the URLs inline to the test code itself

What do you think?

Add test note on how JSON comparison system works
Copy link
Copy Markdown
Contributor Author

@timkimadobe timkimadobe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review @addb! Updated based on feedback with some outstanding questions

// Setup

// Test constructs should always be valid
let standardNetworkRequest = NetworkRequest(urlString: "https://obumobile5.data.adobedc.net/ee/v1/interact", httpMethod: .post)!
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is something @emdobrin and I discussed previously in terms of test readability; we were thinking especially for outside teams that read the test cases (upon test run failure or otherwise), it would be easier for them to see the URLs inline to the test code itself

What do you think?

@timkimadobe timkimadobe requested a review from addb May 19, 2023 01:01
Copy link
Copy Markdown
Contributor Author

@timkimadobe timkimadobe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @emdobrin! Updated based on feedback with some outstanding questions

Refactored assertEdgeResonseHandle API and usages
Refactored location hint test case to use single variable for location hint value
Update matchedResponses name to be uniform across test suite
…directly

Revert refactor of assertEdgeResponseHandle and usages
Add expectedCount argument to both APIs and update assertion logic to check all events
Add org ID related exact match assertions and exact match paths (non-wildcard)
Add exact match requirement for error type URL
Copy link
Copy Markdown
Contributor Author

@timkimadobe timkimadobe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for the review @emdobrin! Updated according to feedback with a few outstanding questions

Copy link
Copy Markdown
Contributor Author

@timkimadobe timkimadobe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @emdobrin! Updated the test cases to use direct assertions, and updated some of the structure of the tests to get more granular coverage

Copy link
Copy Markdown
Contributor

@emdobrin emdobrin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some cleanup questions, but it looks great to me!

Copy link
Copy Markdown
Contributor Author

@timkimadobe timkimadobe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for the review @emdobrin! Updated based on feedback

@timkimadobe timkimadobe requested a review from emdobrin June 28, 2023 21:06
Remove outdated API comment
Copy link
Copy Markdown
Contributor Author

@timkimadobe timkimadobe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @emdobrin! Updated based on feedback

@emdobrin emdobrin merged commit 33e04dd into adobe:feature/upstream-integration-tests Jun 29, 2023
timkimadobe added a commit to timkimadobe/aepsdk-edge-ios that referenced this pull request Jul 11, 2023
* Implementation notes

* WIP invalid datastream test

* Add test cases

Complex XDM, complex data
Preset location hint
expected error dataset ID
expected error invalid location hint

* Update setExpectation API to accept only NetworkRequest

Update docs
Update usages of updated API
Update networkService API usages in integration test class

* Clean up code comments

* Fix for unpassed params

* Update class docs

Add test note on how JSON comparison system works

* Apply swift lint autocorrect

* Rename vars and add additional assert on response count

Add changes lost in rebase

* Updated flexible JSON comparison notes

Refactored assertEdgeResonseHandle API and usages
Refactored location hint test case to use single variable for location hint value

* Update network service class docs

* Update first test case to have two event validation

* Remove unused API and refactor used API to call helper directly

* Update assertEdgeResponseHandle signature and usages

Create new helper methods to construct interact URLs with location hint

* Add strict count assertion for all responses

Update matchedResponses name to be uniform across test suite

* Update assertEdgeResponseError method to use getDispatchedEventsWith directly

Revert refactor of assertEdgeResponseHandle and usages
Add expectedCount argument to both APIs and update assertion logic to check all events
Add org ID related exact match assertions and exact match paths (non-wildcard)
Add exact match requirement for error type URL

* Remove unused API (actually)

* Update test cases to use direct assertions

* Refactor assert*Match APIs to non-nil expected AnyCodable

* Update assertExpectedEvents API to allow for custom timeout period

* Update to use conditional check on location hint if initial value is set

Simplify 2x event handle test case
Add longer timeout for event expectations

* Update test case setup

Remove outdated API comment

* Remove unused helper API
@timkimadobe timkimadobe deleted the integration-test-cases-REBASE-post-merge branch July 19, 2023 23:37
timkimadobe added a commit that referenced this pull request Jul 21, 2023
* Edge Network (Konductor) integration test with GitHub Action workflow (#321)

* initial e2e test

* update pods for e2e test target

* update workflow config settings

* implement pods cache to prevent high macos usage time

* Fix method typo in functional test helper

* test for locationHint

* Update integration test case

* Remove specific hint value

* update action to include build cache step

* update buildcache to upload logs for debug

* fix flag typo

* trying different cache system

* try with command line settings

* update to include key

* fix var bug in makefile command

* remove playground

* Remove build cache testing

* passing env vars to test target working

* Update e2e flow action, yaml, and test scheme for konductor env

* Fix action command to updated name

* Updating action text and defaults

* Update make command for e2e test to improve documentation

add conditional check warning output for KONDUCTOR_ENV

* update test names

* update job name

* Update env var extraction logic

* update test to use edge location hint

* update action to have location hint options

* fix github action var name

* Update makefile env vars and include edge location hint

* update makefile integration test docs

test colon in test run echo
update env var extraction documentation in test file

* update test output to use raw string values

* test using shell if else for job run id

* fix spacing syntax

* Update to user proper Edge Network name

* Update to EDGE_ENVIRONMENT

* Small doc updates

* Update documentation to correct links

* Update dropdown description

* Add quiet flag to xcodebuild command for integration test

* Rename integration test make command

Reorder position to under tvOS test

* Update name to UpstreamIntegrationTests

Test empty string option for dropdown

* Update action script to use updated make command

* Split enums into separate file, add env var extraction inits

* run pod install after new target name

* Remove FunctionalTestBase inheritance and unused compile sources

* Add validation for network request portion

* Rename and simplify IntegrationTestNetworkService to only required methods and properties

* Update to use raw multiline string instead of resource files

* Revert podfile lock version upgrade

* Update action job name to align with makefile name

* Update target name in makefile

* Remove extra newlines

Remove marketing version from integration test debug target build settings

* Remove implementation specific documentation

* Test spacing in makefile

* Test remove silent option

* test moving back to the bottom

* test updating command name

* test newline

* test name change again

* Revert all test name changes - the workflow error was not specifying the correct branch to run off of

* Add job failure help text

* Add doc comment to on failure step

* Revert changes in functional test utils

* Split test enums into separate files with shared util method

* Clean up comments and update test method

* Update EdgeEnvironment enum to have .prod default value

Update usage site in test setup

* Update to include xcresult visualizer tool

* Update with example test failure case

* Update env file ID to use helper method

Move NetworkTestingDelegate to network file

* Omit coverage from test results to save characters

* Fix clang flag warning from cocoapods

* Remove Xcode report tool step from this PR

* Update result bundle path to avoid name conflicts with other tests

* Add testing documentation comment hint

* Flexible JSON comparison system (#332)

* Implement JSON comparison system

* Update to use AEPServices AnyCodable

* Move assertion helpers to AnyCodableUtils

Update assertion methods to accept file and line args for inline errors on test failure
Update key path logic to pretty print

* Create flexible validation system, with exact match pathing

* Exact match wildcard example

* Update general wildcard logic to apply to all elements

Rename arguments and usage sites

* Complete revamp of flexible json comparison system

* Convert AnyCodable test assertion helpers to protocol with default implementations

Update test classes to adhere to new protocol and update usages
Update flexible assertion to use single method with parameter for default mode
Clean up code and update documentation for helper methods
Fix bug with escaped keys and add unit test case

* Apply swift lint

* Extract AnyCodable array extension into separate file

* Remove unused test setup methods

* Remove redundant test cases covered by AnyCodable unit tests

* Switch from protocol to XCTestCase extension

Update usages
Update documentation text

* Update keyPathAsString signature and usages

* Simplify implementation of AnyCodable array comparison

* Remove unused EventSpec

* Update filenames

Remove Bool return from assertEqual methods
Refactor to allow single public API for assertEqual

* Update flexible comparison APIs

* Add additional test cases for AnyCodable unit tests

* Apply swift lint formatting

* Update to use positive case

* Update method order

Update method signature styling

* Update actual condition

* Update flexible text setup to propagate file and line

Upgrade getCapturedRegexGroups to a test failure
Upgrade alternate path index errors to TEST ERROR, add file and line

* Fix incorrect find and replace

* Update regex capture to handle special cases

Add test cases that validate special cases

* Extract regex logic into shared function

* Update shared testing utilities for integration tests (#334)

* Move shared test files

* Refactor FunctionalTestBase and FunctionalTestNetworkService to allow for dual mode

This allows for sharing common test utilities between functional and integration tests

* Renaming all usages of FunctionalTestBase to updated name

* Rename mock mode bool

* Update method docs and cleanup code comments

* Update initializer param

Create separate data structs for mocked and real network responses
Update connectAsync logic structure

* Move XCTestCase+AnyCodableAsserts to shared test utils

Update method docs for TestNetworkService
Update mock bool in TestBase
Move AnyCodable helper methods to shared test utils

* Update TestConstants name and usages

* Consolidate networkResponses into single data struct

With mocked behavior controlled by mockNetworkService flag

* Update FunctionalTestConst usages

* Temporarily moving TestNetworkService for review

* Update set get logic for network responses

Consolidate awaitRequest logic
Simplify NetworkRequest construction
Use force unwrap for test constructions

* Move NetworkRequest extension to FTNS

* Integration tests - TestNetworkService mock and server API split (#337)

* Move FTNS to shared test utils

* Rename FTNS to TNS

* Initial split of mock and server test network service implementations

Before removal of network logic from TestBase

* Migrate TestBase network APIs to TestNetworkService

Move mock or server specific APIs to respective child classes

* Update log source name

Update reset test expectations logic

* Remove resetting network service logic from test base

* Rename delayed network response param and var

Remove outdated method docs for return value

* Update setup logic

Update reset test expectations usage

* Update network service to static immutable var

Update usages to Self

* Remove todo as task is wont do

* Update mock response API param name

* Update integration test to use static network service

Also add helper comments to the purpose of the different setup methods

* Rename Server to Real TNS

Update base class inheritance for both mock and real TNS
Update base TNS to be a shared network request helper
Update usages within mock and real TNS to use new shared helper as instance var
Add passthrough APIs for both mock and real TNS to access helper methods as needed
Refactor static networkService in test class to instance to remove need for Self prefix
Update associated setUp logic
Add mock prefix to networkService in functional test for clarity

* Rename mock and real network service classes

* Clean up implementation

Update doc comment
Add implementation note for isEqual

* Move shared network service logic to helper

Update usage sites
Remove unused import

* Remove unused imports

* Move NetworkRequest flatten body method into NetworkRequest extension

Update usages

* Update access level

* Update doc class names

* Remove unneeded commented code

* Move testbase debug flag to per case setup

Update doc comment

* Refactor CompletionHandlerFunctionalTests to use MockNetworkService

* Refactor EdgeConsentTests to use MockNetworkService

* Add networkService reset

* Add NetworkService reset

* Refactor EdgePublicAPITests to use MockNetworkService

* Refactor AEPEdgePathOverwriteTests

Add test flow doc comments

* Refactor IdentityStateFunctionalTests

Move debug flag set to after setup

* Refactor NoConfigFunctionalTests

Update doc comment for what method is used to determine equality

* Refactor SampleFunctionalTests

* Apply lint autocorrect to PR files

* Remove unneeded commented code

* Integration test cases (#346)

* Implementation notes

* WIP invalid datastream test

* Add test cases

Complex XDM, complex data
Preset location hint
expected error dataset ID
expected error invalid location hint

* Update setExpectation API to accept only NetworkRequest

Update docs
Update usages of updated API
Update networkService API usages in integration test class

* Clean up code comments

* Fix for unpassed params

* Update class docs

Add test note on how JSON comparison system works

* Apply swift lint autocorrect

* Rename vars and add additional assert on response count

Add changes lost in rebase

* Updated flexible JSON comparison notes

Refactored assertEdgeResonseHandle API and usages
Refactored location hint test case to use single variable for location hint value

* Update network service class docs

* Update first test case to have two event validation

* Remove unused API and refactor used API to call helper directly

* Update assertEdgeResponseHandle signature and usages

Create new helper methods to construct interact URLs with location hint

* Add strict count assertion for all responses

Update matchedResponses name to be uniform across test suite

* Update assertEdgeResponseError method to use getDispatchedEventsWith directly

Revert refactor of assertEdgeResponseHandle and usages
Add expectedCount argument to both APIs and update assertion logic to check all events
Add org ID related exact match assertions and exact match paths (non-wildcard)
Add exact match requirement for error type URL

* Remove unused API (actually)

* Update test cases to use direct assertions

* Refactor assert*Match APIs to non-nil expected AnyCodable

* Update assertExpectedEvents API to allow for custom timeout period

* Update to use conditional check on location hint if initial value is set

Simplify 2x event handle test case
Add longer timeout for event expectations

* Update test case setup

Remove outdated API comment

* Remove unused helper API

* Refactor functional tests from dev branch to use new testing utilities

* Use longer extension registration timeout

* Add per test case teardown network service reset

* Test extending teardown sleep duration

* Add higher timeout value for startup event validation

* Add sleep to allow more buffer for listener registration

* Extend setup timeout to 10s

* Restore original teardown sleep time

* Remove sleep from test case

* Change order of operations for test teardown

* Test unregistering instrumented extension in teardown process

* Add sleep to startup process to allow event hub setup time

* Add mock network service teardown to all functional tests

* Trigger CI workflow

* Revert changes in TestBase teardown

* Integration testing workflow update (#349)

* Add new integration test job

* Update device for integration to 8

* Update integration test job name

* Remove extra space

* Remove code coverage upload from integration job

* Test job conditional in circleci workflow

* Add non conditional step

* Update triggering branches

* Update initiating branch name condition to staging

* test fetching branch name from fork

* remove spaces from command

* Test extract both base and head branch names

* Add semicolons

* Update extraction commands

* Only fetch base branch name

Make integration setup steps non conditional

* Update job trigger conditions

* Update simulator for integration test to 14

* Update integration test workflow to use conditional at job level

* Update integration test job Xcode version

* Fix conditional in integration test job

* Move checkout step to non-conditional level

* Test current branch name also triggers integration test workflow

* Test branch name 2

* Remove test branch name

* Add main branch to check

* Update deployment version for integration target to 11

* Make integration test make command result output consistent with other tests

Add removal of existing old results part of the job like other tests

* Refactor to remove branch name conditionals from job level and use workflow job branch filter

* Test workflow job filter when set on current branch

* Update filter criteria

* Remove test branch filter

* Reorder jobs so conditional ones come after mandatory

* Update integration test target in podfile

---------

Co-authored-by: Emilia Dobrin <33132425+emdobrin@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integration testing - Implement test cases based on test plan

3 participants