[BUG] [CI] Investigate Flaky test failure for windows CI tasks #3021
Description
What is the bug?
Windows CI has been flaky with recent runs for tasks, specifically citest, dlicRestApiTest.
See these runs for example:
https://github.com/opensearch-project/security/actions/runs/5577190458/attempts/1 <-- 3 tasks failed
https://github.com/opensearch-project/security/actions/runs/5577190458/attempts/2 <-- 2 tasks failed (1 prev failed task passed)
https://github.com/opensearch-project/security/actions/runs/5577190458/attempts/3 <-- 1 task failed
https://github.com/opensearch-project/security/actions/runs/5577190458/attempts/5 <-- all tasks passed
Failure Details
| (citest, windows-latest, 11) | (citest, windows-latest, 17) | |
|---|---|---|
| 1st run | - org.opensearch.security.SecurityAdminTests.testSecurityAdmin - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesEnabled - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testComplianceEnable - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesDisabled - org.opensearch.security.multitenancy.test.MultitenancyTests.testTenantParametersSubstitution - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery - org.opensearch.security.httpclient.HttpClientTest.testPlainConnection - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testComplianceEnable - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesEnabled |
- com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticatorTest.initialConnectionFailureTest - com.amazon.dlic.auth.ldap2.LdapBackendIntegTest2.testIntegLdapAuthenticationSSL - org.opensearch.security.SecurityAdminTests.testIsLegacySecurityIndexOnV7Index - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testComplianceEnable - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesDisabled - org.opensearch.security.multitenancy.test.TenancyPrivateTenantEnabledTests.testPrivateTenantDisabled_Update_EndToEnd - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testSensitiveMethodRedaction - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testScroll - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testScroll - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction |
| 2nd run | - org.opensearch.security.SecurityAdminTests.testSecurityAdminRegularUpdate - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery - org.opensearch.security.multitenancy.test.TenancyMultitenancyEnabledTests.testMultitenancyDisabled_endToEndTest - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery |
- com.amazon.dlic.auth.ldap2.LdapBackendIntegTest2.testAttributesWithImpersonation - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testWriteHistory - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction - org.opensearch.security.multitenancy.test.MultitenancyTests.testTenantParametersSubstitution - org.opensearch.security.multitenancy.test.MultitenancyTests.testMt - org.opensearch.security.multitenancy.test.MultitenancyTests.testMtMulti - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction |
| 3rd run | - org.opensearch.security.SecurityAdminTests.testSecurityAdmin - org.opensearch.security.SecurityAdminTests.testSecurityAdminInvalidCert - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesEnabled - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiNewUser - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testUpdate - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesEnabled - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery - org.opensearch.security.multitenancy.test.MultitenancyTests.testMt - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery - org.opensearch.security.multitenancy.test.TenancyMultitenancyEnabledTests.testMultitenancyDisabled_endToEndTest - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiNewUser - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testUpdate |
Test Methods with failures across all mentioned runs
- org.opensearch.security.SecurityAdminTests.testSecurityAdmin
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesEnabled
- org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testComplianceEnable
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesDisabled
- org.opensearch.security.multitenancy.test.MultitenancyTests.testTenantParametersSubstitution
- org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery
- org.opensearch.security.httpclient.HttpClientTest.testPlainConnection
- com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticatorTest.initialConnectionFailureTest
- com.amazon.dlic.auth.ldap2.LdapBackendIntegTest2.testIntegLdapAuthenticationSSL
- org.opensearch.security.SecurityAdminTests.testIsLegacySecurityIndexOnV7Index
- org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction
- org.opensearch.security.multitenancy.test.TenancyPrivateTenantEnabledTests.testPrivateTenantDisabled_Update_EndToEnd
- org.opensearch.security.auditlog.integration.BasicAuditlogTest.testSensitiveMethodRedaction
- org.opensearch.security.auditlog.integration.BasicAuditlogTest.testScroll
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction
- org.opensearch.security.multitenancy.test.MultitenancyTests.testMt
- org.opensearch.security.multitenancy.test.MultitenancyTests.testMtMulti
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testWriteHistory
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiNewUser
- org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testUpdate
- org.opensearch.security.SecurityAdminTests.testSecurityAdminInvalidCert
- org.opensearch.security.multitenancy.test.TenancyMultitenancyEnabledTests.testMultitenancyDisabled_endToEndTest
What is the expected behavior?
No flakiness.