GH-34157: [C++] Configure bundled AWS SDK to use aws-lc instead of OpenSSL by js8544 · Pull Request #34159 · apache/arrow

js8544 · 2023-02-13T09:48:02Z

Using OpenSSL causes various issues like #33808 (comment) and #34111. We should try to use aws-lc for libcrypto and libssl. We need to hide them inside s2n-tls to avoid name conflicts with OpenSSL used by other libraries.

Some notes:

Only linux needs s2n-tls and aws-lc.
Because aws-c-http requires curl which requires libcrypto from OpenSSL, we need to hide Aws::libcrypto by S2N_INTERN_LIBCRYPTO=ON.
aws-c-cal also uses libcrypto, but it can't hide it. So we need to use OpenSSL instead of aws-lc for it.
Mac and windows don't need s2n-tls, but do need some security libs from system, like -framework Security and ncrypt.lib.
arrow-s3fs-test is re-enabled.
Force Java Gandiva to use static protobuf lib.

Closes: [C++] Configure bundled AWS SDK to use aws-lc instead of OpenSSL #34157

github-actions · 2023-02-13T09:48:26Z

Closes: [C++] Configure bundled AWS SDK to use aws-lc instead of OpenSSL #34157

js8544 · 2023-02-13T09:49:37Z

@github-actions crossbow submit -g cpp -g r

js8544 · 2023-02-13T09:49:48Z

@github-actions crossbow submit java-jars

js8544 · 2023-02-13T11:17:42Z

@github-actions crossbow submit -g cpp -g r

js8544 · 2023-02-13T11:17:54Z

@github-actions crossbow submit java-jars

js8544 · 2023-02-13T12:16:04Z

@github-actions crossbow submit java-jars

js8544 · 2023-02-13T12:16:16Z

@github-actions crossbow submit -g cpp -g r

js8544 · 2023-02-13T14:28:27Z

@github-actions crossbow submit java-jars

js8544 · 2023-02-13T14:28:41Z

@github-actions crossbow submit -g cpp -g r

js8544 · 2023-02-13T14:33:09Z

@github-actions crossbow submit java-jars

js8544 · 2023-02-13T14:33:24Z

@github-actions crossbow submit -g cpp -g r

js8544 · 2023-02-13T14:37:49Z

@github-actions crossbow submit java-jars

java/gandiva/CMakeLists.txt

js8544 · 2023-02-15T04:25:04Z

Ah, this is caused by Homebrew/homebrew-core#122277 . Homebrew stopped providing libprotobuf.a.

We need to prepare libprotobuf.a for us. But it is not related to this pull request. Could you open a new issue for it?

Nice find! I thought of this reason but didn't find this change. I created a new issue #34189 for this.

js8544 · 2023-02-15T06:27:57Z

CI failures are unrelated. I think this is ready to to merged. cc @kou

kou · 2023-02-15T06:31:10Z

@github-actions crossbow submit -g cpp -g r java-jars

js8544 · 2023-02-15T07:07:36Z

@github-actions crossbow submit -g cpp -g r java-jars
I just rebased to fix the zstd version issue.

github-actions · 2023-02-15T07:14:10Z

Revision: f44c1bf

Submitted crossbow builds: ursacomputing/crossbow @ actions-7b13f07b9d

Task	Status
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-r41
homebrew-r-autobrew
homebrew-r-brew
java-jars
r-binary-packages
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-10-cpp-amd64
test-debian-10-cpp-i386
test-debian-11-cpp-amd64
test-debian-11-cpp-i386
test-fedora-35-cpp
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-library-r-base-latest
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-18.04-cpp
test-ubuntu-18.04-cpp-release
test-ubuntu-18.04-cpp-static
test-ubuntu-18.04-r-sanitizer
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-20
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp

kou · 2023-02-15T07:49:21Z

Sorry. Can we try this?

diff --git a/ci/scripts/java_jni_macos_build.sh b/ci/scripts/java_jni_macos_build.sh
index 391d05642..37bbe1283 100755
--- a/ci/scripts/java_jni_macos_build.sh
+++ b/ci/scripts/java_jni_macos_build.sh
@@ -109,6 +109,7 @@ fi
 popd
 
 
+export JAVA_JNI_CMAKE_ARGS="-DProtobuf_ROOT=${build_dir}/cpp/protobuf_ep-install"
 ${arrow_dir}/ci/scripts/java_jni_build.sh \
   ${arrow_dir} \
   ${install_dir} \

js8544 · 2023-02-15T07:53:01Z

@github-actions crossbow submit java-jars

github-actions · 2023-02-15T07:55:31Z

Revision: 89c547c

Submitted crossbow builds: ursacomputing/crossbow @ actions-4bf574a4e3

Task	Status
java-jars

js8544 · 2023-02-15T10:12:12Z

Sorry. Can we try this?

diff --git a/ci/scripts/java_jni_macos_build.sh b/ci/scripts/java_jni_macos_build.sh
index 391d05642..37bbe1283 100755
--- a/ci/scripts/java_jni_macos_build.sh
+++ b/ci/scripts/java_jni_macos_build.sh
@@ -109,6 +109,7 @@ fi
 popd
 
 
+export JAVA_JNI_CMAKE_ARGS="-DProtobuf_ROOT=${build_dir}/cpp/protobuf_ep-install"
 ${arrow_dir}/ci/scripts/java_jni_build.sh \
   ${arrow_dir} \
   ${install_dir} \

It works. Thanks for suggesting this!

kou

+1

Thanks!

kou · 2023-02-15T20:45:06Z

@github-actions crossbow submit -g wheel

github-actions · 2023-02-15T20:48:48Z

Revision: 89c547c

Submitted crossbow builds: ursacomputing/crossbow @ actions-5407ab76c8

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-mojave-cp310-amd64
wheel-macos-mojave-cp311-amd64
wheel-macos-mojave-cp37-amd64
wheel-macos-mojave-cp38-amd64
wheel-macos-mojave-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp311-amd64
wheel-manylinux2014-cp311-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

kou · 2023-02-15T23:20:10Z

@js8544 It seems that we need more work for manylinux wheels. Could you open a new issue for it?

ursabot · 2023-02-15T23:36:56Z

Benchmark runs are scheduled for baseline = b955da0 and contender = 798b417. 798b417 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.18% ⬆️0.03%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.22% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 798b417e ec2-t3-xlarge-us-east-2
[Finished] 798b417e test-mac-arm
[Finished] 798b417e ursa-i9-9960x
[Finished] 798b417e ursa-thinkcentre-m75q
[Finished] b955da05 ec2-t3-xlarge-us-east-2
[Failed] b955da05 test-mac-arm
[Finished] b955da05 ursa-i9-9960x
[Finished] b955da05 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2023-02-15T23:50:29Z

['Python', 'R'] benchmarks have high level of regressions.
test-mac-arm

…ed AWS (#34215) To fix build failures such as #34159 (comment), we need to help it find OpenSSL. * Closes: #34214 Authored-by: Jin Shang <shangjin1997@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

… of OpenSSL (apache#34159) Using OpenSSL causes various issues like apache#33808 (comment) and apache#34111. We should try to use aws-lc for libcrypto and libssl. We need to hide them inside s2n-tls to avoid name conflicts with OpenSSL used by other libraries. Some notes: 1. Only linux needs s2n-tls and aws-lc. 2. Because aws-c-http requires curl which requires libcrypto from OpenSSL, we need to hide Aws::libcrypto by S2N_INTERN_LIBCRYPTO=ON. 3. aws-c-cal also uses libcrypto, but it can't hide it. So we need to use OpenSSL instead of aws-lc for it. 4. Mac and windows don't need s2n-tls, but do need some security libs from system, like `-framework Security` and `ncrypt.lib`. 5. arrow-s3fs-test is re-enabled. 6. Force Java Gandiva to use static protobuf lib. * Closes: apache#34157 Lead-authored-by: Jin Shang <shangjin1997@gmail.com> Co-authored-by: jinshang <jinshang@tencent.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

js8544 mentioned this pull request Feb 13, 2023

GH-34157: [C++] Configure bundled AWS SDK to use aws-lc instead of OpenSSL #34158

Closed

github-actions bot added the Component: C++ label Feb 13, 2023

This comment was marked as outdated.

Sign in to view

js8544 mentioned this pull request Feb 13, 2023

[CI][Packaging][Java] java-jars was broken by updating bundled aws-sdk-cpp #34111

Closed

js8544 force-pushed the jinshang/use_aws_lc branch from 8b547a8 to ad4a5bc Compare February 13, 2023 10:06

This comment was marked as outdated.

Sign in to view

js8544 force-pushed the jinshang/use_aws_lc branch from cb07ab5 to 2f77681 Compare February 13, 2023 11:23

This comment was marked as outdated.

Sign in to view

js8544 force-pushed the jinshang/use_aws_lc branch from 561a147 to e4dcae8 Compare February 13, 2023 14:32

This comment was marked as outdated.

Sign in to view

js8544 force-pushed the jinshang/use_aws_lc branch from e4dcae8 to 09bb8b5 Compare February 13, 2023 14:37

This comment was marked as outdated.

Sign in to view

kou reviewed Feb 14, 2023

View reviewed changes

java/gandiva/CMakeLists.txt Outdated Show resolved Hide resolved

java/gandiva/CMakeLists.txt Outdated Show resolved Hide resolved

js8544 mentioned this pull request Feb 15, 2023

[Java] Provide static libprotobuf.a for java-jars build #34189

Closed

This comment was marked as outdated.

Sign in to view

js8544 force-pushed the jinshang/use_aws_lc branch from d33a98f to f44c1bf Compare February 15, 2023 07:07

This comment was marked as outdated.

Sign in to view

kou approved these changes Feb 15, 2023

View reviewed changes

kou merged commit 798b417 into apache:master Feb 15, 2023

This was referenced Feb 16, 2023

[C++] Bundled AWS::c_cal can't find libcrypto on manylinux #34214

Closed

GH-34214: [C++] Pass OPENSSL_ROOT_HINT to CMAKE_PREFIX_PATH for bundled AWS #34215

Merged

kou mentioned this pull request Feb 26, 2024

[C++][CMake] Improve description why we need to initialize AWS C++ SDK in arrow-s3fs-test #40228

Closed

Conversation

js8544 commented Feb 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 13, 2023

Uh oh!

js8544 commented Feb 13, 2023

Uh oh!

js8544 commented Feb 13, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

js8544 commented Feb 13, 2023

Uh oh!

js8544 commented Feb 13, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

js8544 commented Feb 13, 2023

Uh oh!

js8544 commented Feb 13, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

js8544 commented Feb 13, 2023

Uh oh!

js8544 commented Feb 13, 2023

Uh oh!

This comment was marked as outdated.

js8544 commented Feb 13, 2023

Uh oh!

js8544 commented Feb 13, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

js8544 commented Feb 13, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

Uh oh!

js8544 commented Feb 15, 2023

Uh oh!

js8544 commented Feb 15, 2023

Uh oh!

kou commented Feb 15, 2023

Uh oh!

This comment was marked as outdated.

js8544 commented Feb 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

github-actions bot commented Feb 15, 2023

Uh oh!

kou commented Feb 15, 2023

Uh oh!

js8544 commented Feb 15, 2023

Uh oh!

github-actions bot commented Feb 15, 2023

Uh oh!

js8544 commented Feb 15, 2023

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

kou commented Feb 15, 2023

Uh oh!

github-actions bot commented Feb 15, 2023

Uh oh!

kou commented Feb 15, 2023

Uh oh!

ursabot commented Feb 15, 2023

Uh oh!

ursabot commented Feb 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

js8544 commented Feb 13, 2023 •

edited

Loading

js8544 commented Feb 15, 2023 •

edited

Loading