[1.2] Update FIPS API libraries of Bouncy Castle (#1853) by tlfeng · Pull Request #1888 · opensearch-project/OpenSearch

tlfeng · 2022-01-11T23:57:17Z

Description

Backport PR #1853 / commit db23f72 into 1.2 branch.
Update the versions of all the remaining API libraries of org.bouncycastle, which are mainly FIPS APIs.

Update the version of bc-fips from 1.0.2 to 1.0.2.1 to reduce the vulnerability CVE-2020-15522
Update bcpg-fips from 1.0.4 to 1.0.5.1
Update bctls-fips from 1.0.9 to 1.0.12.2
Apply the unified defined version of bouncycastle to bcpkix-jdk15on, in HDFS testing fixture.

Issues Resolved

None.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

* Update bc-fips to 1.0.2.1 Signed-off-by: Tianli Feng <ftl94@live.com> * Update bcpg-fips to 1.0.5.1 Signed-off-by: Tianli Feng <ftl94@live.com> * Update bctls-fips to 1.0.12.2 Signed-off-by: Tianli Feng <ftl94@live.com> * Use the unified bouncycastle version for bcpkix-jdk15on in HDFS testing fixture Signed-off-by: Tianli Feng <ftl94@live.com>

opensearch-ci-bot · 2022-01-11T23:58:03Z

Can one of the admins verify this patch?

opensearch-ci-bot · 2022-01-12T00:23:31Z

❌ Gradle Check failure 4be4ee7
Log 1865

Reports 1865

tlfeng · 2022-01-12T00:30:22Z

In Log 1865:

> Task :plugins:transport-nio:test

REPRODUCE WITH: ./gradlew ':plugins:transport-nio:test' --tests "org.opensearch.http.nio.NioHttpServerTransportTests.testLargeCompressedResponse" -Dtests.seed=37F8E8D16987C4E5 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=fr-CA -Dtests.timezone=Asia/Samarkand -Druntime.java=15

org.opensearch.http.nio.NioHttpServerTransportTests > testLargeCompressedResponse FAILED
    java.lang.AssertionError:

REPRODUCE WITH: ./gradlew ':distribution:tools:plugin-cli:test' --tests "org.opensearch.plugins.InstallPluginCommandTests" -Dtests.method="testOfficialPluginSnapshot {p0=sun.nio.fs.LinuxFileSystem@1b1b9d68 p1=org.opensearch.plugins.InstallPluginCommandTests$$Lambda$228/0x0000000800d87d10@53e11ea2}" -Dtests.seed=37F8E8D16987C4E5 -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=uk-UA -Dtests.timezone=MST7MDT -Druntime.java=15

org.opensearch.plugins.InstallPluginCommandTests > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([37F8E8D16987C4E5]:0)

opensearch-ci-bot · 2022-01-12T04:50:23Z

❌ Gradle Check failure 991f488
Log 1871

Reports 1871

tlfeng · 2022-01-12T05:03:00Z

In Log 1871:

REPRODUCE WITH: ./gradlew ':modules:transport-netty4:internalClusterTest' --tests "org.opensearch.transport.netty4.OpenSearchLoggingHandlerIT.testLoggingHandler" -Dtests.seed=9B5DBA413F09806A -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-PY -Dtests.timezone=America/Nome -Druntime.java=15

org.opensearch.transport.netty4.OpenSearchLoggingHandlerIT > testLoggingHandler FAILED
    java.lang.AssertionError: 
    Expected: an empty collection
         but: <[org.apache.logging.log4j.spi.AbstractLogger caught

The above is reproducible locally.

> Task :distribution:tools:plugin-cli:test

REPRODUCE WITH: ./gradlew ':distribution:tools:plugin-cli:test' --tests "org.opensearch.plugins.InstallPluginCommandTests" -Dtests.method="testOfficialPluginSnapshot {p0=com.google.common.jimfs.JimfsFileSystem@11ccd49d p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d866b8@e71a611}" -Dtests.seed=9B5DBA413F09806A -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sk -Dtests.timezone=SystemV/PST8 -Druntime.java=15

org.opensearch.plugins.InstallPluginCommandTests > testOfficialPluginSnapshot {p0=com.google.common.jimfs.JimfsFileSystem@11ccd49d p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d866b8@e71a611} FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([9B5DBA413F09806A]:0)

REPRODUCE WITH: ./gradlew ':distribution:tools:plugin-cli:test' --tests "org.opensearch.plugins.InstallPluginCommandTests" -Dtests.method="testOfficialPluginSnapshot {p0=com.google.common.jimfs.JimfsFileSystem@11ccd49d p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d866b8@e71a611}" -Dtests.seed=9B5DBA413F09806A -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sk -Dtests.timezone=SystemV/PST8 -Druntime.java=15

org.opensearch.plugins.InstallPluginCommandTests > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([9B5DBA413F09806A]:0)

This one might be related to the bouncy castle library.

saratvemulapalli · 2022-01-12T17:42:39Z

In Log 1871:

REPRODUCE WITH: ./gradlew ':modules:transport-netty4:internalClusterTest' --tests "org.opensearch.transport.netty4.OpenSearchLoggingHandlerIT.testLoggingHandler" -Dtests.seed=9B5DBA413F09806A -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-PY -Dtests.timezone=America/Nome -Druntime.java=15

org.opensearch.transport.netty4.OpenSearchLoggingHandlerIT > testLoggingHandler FAILED
    java.lang.AssertionError: 
    Expected: an empty collection
         but: <[org.apache.logging.log4j.spi.AbstractLogger caught

The above is reproducible locally.

> Task :distribution:tools:plugin-cli:test

REPRODUCE WITH: ./gradlew ':distribution:tools:plugin-cli:test' --tests "org.opensearch.plugins.InstallPluginCommandTests" -Dtests.method="testOfficialPluginSnapshot {p0=com.google.common.jimfs.JimfsFileSystem@11ccd49d p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d866b8@e71a611}" -Dtests.seed=9B5DBA413F09806A -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sk -Dtests.timezone=SystemV/PST8 -Druntime.java=15

org.opensearch.plugins.InstallPluginCommandTests > testOfficialPluginSnapshot {p0=com.google.common.jimfs.JimfsFileSystem@11ccd49d p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d866b8@e71a611} FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([9B5DBA413F09806A]:0)

REPRODUCE WITH: ./gradlew ':distribution:tools:plugin-cli:test' --tests "org.opensearch.plugins.InstallPluginCommandTests" -Dtests.method="testOfficialPluginSnapshot {p0=com.google.common.jimfs.JimfsFileSystem@11ccd49d p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d866b8@e71a611}" -Dtests.seed=9B5DBA413F09806A -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sk -Dtests.timezone=SystemV/PST8 -Druntime.java=15

org.opensearch.plugins.InstallPluginCommandTests > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([9B5DBA413F09806A]:0)

This one might be related to the bouncy castle library.

I see, its a timeout exceeded problem on CI. There were other PRs hitting this similar problem.

saratvemulapalli · 2022-01-12T17:42:45Z

start gradle check

opensearch-ci-bot · 2022-01-12T17:45:10Z

❌ Gradle Check failure 991f488
Log 1877

Reports 1877

tlfeng · 2022-01-12T18:12:12Z

Let me merge the upstream commits to this branch.

opensearch-ci-bot · 2022-01-12T18:18:44Z

❌ Gradle Check failure 5ebd120
Log 1879

Reports 1879

saratvemulapalli · 2022-01-12T18:51:20Z

❌ Gradle Check failure 991f488 Log 1877

Reports 1877

JCenter is down, we are seeing similar issues with build: opensearch-project/opensearch-build#1456 and Job Scheduler

opensearch-ci-bot · 2022-01-13T01:23:04Z

❌ Gradle Check failure f1dd9f5
Log 1894

Reports 1894

tlfeng · 2022-01-13T03:26:15Z

In log 1894:

> Task :distribution:tools:plugin-cli:test

REPRODUCE WITH: ./gradlew ':distribution:tools:plugin-cli:test' --tests "org.opensearch.plugins.InstallPluginCommandTests" -Dtests.method="testOfficialPluginStaging {p0=com.google.common.jimfs.JimfsFileSystem@1b1b9d68 p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d86378@53e11ea2}" -Dtests.seed=D702C5A2973EEACB -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=vi -Dtests.timezone=Africa/Lagos -Druntime.java=15

org.opensearch.plugins.InstallPluginCommandTests > testOfficialPluginStaging {p0=com.google.common.jimfs.JimfsFileSystem@1b1b9d68 p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d86378@53e11ea2} FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([D702C5A2973EEACB]:0)

REPRODUCE WITH: ./gradlew ':distribution:tools:plugin-cli:test' --tests "org.opensearch.plugins.InstallPluginCommandTests" -Dtests.method="testOfficialPluginStaging {p0=com.google.common.jimfs.JimfsFileSystem@1b1b9d68 p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d86378@53e11ea2}" -Dtests.seed=D702C5A2973EEACB -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=vi -Dtests.timezone=Africa/Lagos -Druntime.java=15

org.opensearch.plugins.InstallPluginCommandTests > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([D702C5A2973EEACB]:0)

tlfeng · 2022-01-13T03:31:24Z

start gradle check

opensearch-ci-bot · 2022-01-13T03:40:18Z

❌ Gradle Check failure f1dd9f5
Log 1898

Reports 1898

tlfeng · 2022-01-13T03:43:30Z

In log 1898:

> Task :distribution:tools:upgrade-cli:test

REPRODUCE WITH: ./gradlew ':distribution:tools:upgrade-cli:test' --tests "org.opensearch.upgrade.DetectEsInstallationTaskTests.testTaskExecution" -Dtests.seed=CC775139B6D1C442 -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=cs -Dtests.timezone=America/Recife -Druntime.java=15

org.opensearch.upgrade.DetectEsInstallationTaskTests > testTaskExecution FAILED
    java.lang.AssertionError: 
    Expected: a collection with size <0>
         but: collection size was <5>
        at __randomizedtesting.SeedInfo.seed([CC775139B6D1C442:866D75B262868BC7]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.opensearch.upgrade.DetectEsInstallationTaskTests.testTaskExecution(DetectEsInstallationTaskTests.java:53)

It is reported in #1846, but can't be reproduced locally.

tlfeng · 2022-01-13T03:47:05Z

start gradle check

opensearch-ci-bot · 2022-01-13T04:26:30Z

❌ Gradle Check failure f1dd9f5
Log 1899

Reports 1899

tlfeng · 2022-01-13T04:51:40Z

In log 1899:

The failure also occurred above and in the PR #1546 (comment), and being tracked in the issue #1564. Maybe there are some issues in 1.2 branch.

> Task :distribution:tools:plugin-cli:test

REPRODUCE WITH: ./gradlew ':distribution:tools:plugin-cli:test' --tests "org.opensearch.plugins.InstallPluginCommandTests" -Dtests.method="testOfficialPlatformPluginSnapshot {p0=com.google.common.jimfs.JimfsFileSystem@1b1b9d68 p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d86378@53e11ea2}" -Dtests.seed=77245BA4D7BC0948 -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=Australia/Melbourne -Druntime.java=15

org.opensearch.plugins.InstallPluginCommandTests > testOfficialPlatformPluginSnapshot {p0=com.google.common.jimfs.JimfsFileSystem@1b1b9d68 p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d86378@53e11ea2} FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([77245BA4D7BC0948]:0)

REPRODUCE WITH: ./gradlew ':distribution:tools:plugin-cli:test' --tests "org.opensearch.plugins.InstallPluginCommandTests" -Dtests.method="testOfficialPlatformPluginSnapshot {p0=com.google.common.jimfs.JimfsFileSystem@1b1b9d68 p1=org.opensearch.plugins.InstallPluginCommandTests$1Parameter$$Lambda$227/0x0000000800d86378@53e11ea2}" -Dtests.seed=77245BA4D7BC0948 -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=Australia/Melbourne -Druntime.java=15

org.opensearch.plugins.InstallPluginCommandTests > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([77245BA4D7BC0948]:0)

dblock · 2022-01-13T15:00:57Z

start gradle check

opensearch-ci-bot · 2022-01-13T15:36:18Z

✅ Gradle Check success f1dd9f5
Log 1908

Reports 1908

tlfeng added >upgrade Label used when upgrading library dependencies (e.g., Lucene) backport PRs or issues specific to backporting features or enhancments CVE Fixes a CVE v1.2.4 labels Jan 11, 2022

VachaShah approved these changes Jan 12, 2022

View reviewed changes

Merge remote-tracking branch 'upstream/1.2' into 1.2-bc-fips

991f488

saratvemulapalli approved these changes Jan 12, 2022

View reviewed changes

Merge branch '1.2' into 1.2-bc-fips

5ebd120

tlfeng added 2 commits January 13, 2022 00:36

Merge remote-tracking branch 'upstream/1.2' into 1.2-bc-fips

b91dd07

Merge remote-tracking branch 'origin/1.2-bc-fips' into 1.2-bc-fips

f1dd9f5

tlfeng mentioned this pull request Jan 13, 2022

[BUG] InstallPluginCommandTests.testOfficialPluginSnapshot (Random Test Failure) #1564

Closed

dblock merged commit e505b10 into opensearch-project:1.2 Jan 13, 2022

tlfeng deleted the 1.2-bc-fips branch January 20, 2022 00:24

jmazanec15 mentioned this pull request Feb 10, 2022

[Backport 1.2] [BUG FIX] Add space type default and ef search parameter in warmup opensearch-project/k-NN#285

Merged

Conversation

tlfeng commented Jan 11, 2022

Description

Issues Resolved

Check List

Uh oh!

opensearch-ci-bot commented Jan 11, 2022

Uh oh!

opensearch-ci-bot commented Jan 12, 2022

Uh oh!

tlfeng commented Jan 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

opensearch-ci-bot commented Jan 12, 2022

Uh oh!

tlfeng commented Jan 12, 2022

Uh oh!

saratvemulapalli commented Jan 12, 2022

Uh oh!

saratvemulapalli commented Jan 12, 2022

Uh oh!

opensearch-ci-bot commented Jan 12, 2022

Uh oh!

tlfeng commented Jan 12, 2022

Uh oh!

opensearch-ci-bot commented Jan 12, 2022

Uh oh!

saratvemulapalli commented Jan 12, 2022

Uh oh!

opensearch-ci-bot commented Jan 13, 2022

Uh oh!

tlfeng commented Jan 13, 2022

Uh oh!

tlfeng commented Jan 13, 2022

Uh oh!

opensearch-ci-bot commented Jan 13, 2022

Uh oh!

tlfeng commented Jan 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tlfeng commented Jan 13, 2022

Uh oh!

opensearch-ci-bot commented Jan 13, 2022

Uh oh!

tlfeng commented Jan 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dblock commented Jan 13, 2022

Uh oh!

opensearch-ci-bot commented Jan 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tlfeng commented Jan 12, 2022 •

edited

Loading

tlfeng commented Jan 13, 2022 •

edited

Loading

tlfeng commented Jan 13, 2022 •

edited

Loading