Skip to content

Fix flaky tests #365

@timkimadobe

Description

@timkimadobe

Task description

There are tests across both unit and functional tests that intermittently fail, particularly when processing speeds are slow and timeout thresholds are hit (although this is not the only flaky test failure mode).

Examples of flaky failures

1. Test cleanup failure - AEPEdgeFunctionalTests.swift

testSendEvent_edgeNetworkResponseContainsLocationHint_sendEventDoesNotIncludeExpiredLocationHint
runs before
testSendEvent_fatalError

The latter case fails with an unexpected locationHint:result event in the setup process - a locationHint:result type event is the mocked response from the former case


testSendEvent_twoConsecutiveCalls_resetBefore_appendsReceivedClientSideStore runs before
testSendEvent_twoConsecutiveCalls_resetInBetween_doesNotAppendReceivedClientSideStore

The latter case fails with an unexpected state:store event in the setup process - a state:store type event is the mocked response from the former case

See job failure

2. Test setup timeout failure - AEPEdgeFunctionalTests.swift

func testSendEvent_twoConsecutiveCalls_appendsReceivedClientSideStore() {

Fails with: XCTAssertFalse failed - Timed out waiting for event type com.adobe.eventType.hub and source com.adobe.eventSource.sharedState, expected 4, but received 1

Hypothesis is the setup process doesn't have enough time to process all the shared state events before the timeout triggers, especially on slower machines

See job failure


testSendEvent_fatalError
XCTAssertFalse failed - Timed out waiting for event type com.adobe.eventType.hub and source com.adobe.eventSource.sharedState, expected 4, but received 1

See job failure


testSendEvent_fatalError400
XCTAssertFalse failed - Timed out waiting for event type com.adobe.eventType.configuration and source com.adobe.eventSource.responseContent, expected 1, but received 0

See job failure

3. Event listener timeout failure - UnitTests EdgePublicAPITests

testGetLocationHint_dispatchesEdgeRequestIdentity_receivesResponseIdentity_withEmptyHint

func testGetLocationHint_dispatchesEdgeRequestIdentity_receivesResponseIdentity_withEmptyHint() {

Fails on


XCTAssertNil failed: "AEPError"

See job failure


testGetLocationHint_dispatchesEdgeRequestIdentity_receivesResponseIdentity_withValidHint

func testGetLocationHint_dispatchesEdgeRequestIdentity_receivesResponseIdentity_withValidHint() {

Fails on

XCTAssertEqual("or2", hint)

4. MockDataQueue empty failure - UnitTests EdgeExtensionsTests

func testHandleExperienceEventRequest_validData_consentYes_queues() {

Fails on

XCTAssertEqual(1, mockDataQueue.count())

XCTAssertEqual failed: ("1") is not equal to ("0")

See job failure

Additional implementation details or code snippet(s)

No response

Metadata

Metadata

Labels

taskItem that can be completed as a standalone code change or is part of an epic.Used by issue templates

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions