Skip to content

Fix under-determined test: testWithParam#2737

Merged
bjagg merged 3 commits into
uPortal-Project:masterfrom
KiruthikaJanakiraman:fix-flaky-testWithParam
Dec 22, 2023
Merged

Fix under-determined test: testWithParam#2737
bjagg merged 3 commits into
uPortal-Project:masterfrom
KiruthikaJanakiraman:fix-flaky-testWithParam

Conversation

@KiruthikaJanakiraman

Copy link
Copy Markdown
Contributor
Checklist
Description of change

This PR aims to fix an under-determined test: org.apereo.portal.url.CasLoginRefUrlEncoderTest.testWithParam

I found and confirmed the failure of the test using an open-source research tool NonDex, which shuffles implementations of nondeterminism operations.

Problem
The test case org.apereo.portal.url.CasLoginRefUrlEncoderTest.testWithParam fails due to the below assertion:

assertEquals(
"https://cas:80/cas/login?service=myhost:8080/uPortal/Login%3FrefUrl%3Dhttp%3A%2F%2Fmyhost%3A8080%2FuPortal%2Fp%2Ffname%253FpP_announcementId%253D1834",
encoder.encodeLoginAndRefUrl(request));

This is because, findMatchingServerName returns the first element from allServerNames collection.

Since allServerNames is a HashSet, the order of the items "myhost:8080", "theirhost:8443" in the HashSet is not preserved. Hence the server name returned by findMatchingServerName may vary which may cause the test testWithParam to fail in a hypothetical future.

Solution
This PR fixes the test by using a LinkedHashSet to store allServerNames since LinkedHashSet preserves the order of the elements.

Steps to reproduce
The following command can be used to reproduce the assertion errors and verify the fix.

Integrate NonDex:

Add the following snippet to the top of the build.gradle in $PROJ_DIR:

plugins {
    id 'edu.illinois.nondex' version '2.1.1-1'
}

Append to build.gradle in $PROJ_DIR:

apply plugin: 'edu.illinois.nondex'

Execute Test with Gradle:

./gradlew --info uPortal-security:test --tests org.apereo.portal.url.CasLoginRefUrlEncoderTest.testWithParam

Run NonDex:

./gradlew --info uPortal-security:nondexTest --tests=org.apereo.portal.url.CasLoginRefUrlEncoderTest.testWithParam --nondexRuns=50 -x autoLintGradle

Test Environment:

Java version "1.8.0_381"
macOS Venture Version 13.4.1 (22F82)

Please let me know if you have any concerns or questions.

@ChristianMurphy

ChristianMurphy commented Nov 18, 2023

Copy link
Copy Markdown
Member

Hey @KiruthikaJanakiraman 👋

This is because, findMatchingServerName returns the first element from allServerNames collection.

I'm not sure I follow, findMatchingServerName returns the first match not the first element.
Both in the test and in most real world scenarios there will be no overlapping servernames so only one match will be returned.

@KiruthikaJanakiraman

Copy link
Copy Markdown
Contributor Author

Hi,

If I'm not wrong, the case where comparisonHost is empty, the function returns the first element of allServerNames:

@ChristianMurphy

Copy link
Copy Markdown
Member

@jgribonvald do you have insights on the intent of this method?

@bjagg bjagg merged commit 44e377c into uPortal-Project:master Dec 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants