ARROW-6437: [R] Add AWS SDK to system dependencies for macOS and Windows #7928
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
@github-actions crossbow submit homebrew-r-autobrew |
|
Revision: 65001f14bf78efafad024907d7c91dd404d40c09 Submitted crossbow builds: ursa-labs/crossbow @ actions-471
|
Member
Author
|
@github-actions crossbow submit homebrew-r-autobrew |
|
Revision: d8cc705f2e0f42a56109a49024f2a122be668ced Submitted crossbow builds: ursa-labs/crossbow @ actions-473
|
Member
Author
|
@github-actions crossbow submit homebrew-r-autobrew |
|
Revision: 9a8364dd3e89052f2925f34fc591f5e17563ab90 Submitted crossbow builds: ursa-labs/crossbow @ actions-474
|
Member
Author
|
Rtools40 is passing. Rtools35 needs a different library build, which Jeroen is working on: https://github.com/r-windows/rtools-backports/compare/aws-sdk autobrew build fails like this: |
Member
|
No idea. Can we have the entire output from |
Member
Author
|
@github-actions crossbow submit homebrew-cpp-autobrew |
|
Revision: 9a8364dd3e89052f2925f34fc591f5e17563ab90 Submitted crossbow builds: ursa-labs/crossbow @ actions-475
|
Contributor
|
Maybe there is a problem with the unity builds of aws-sdk-cpp or arrow |
Member
Author
|
@github-actions crossbow submit homebrew-cpp-autobrew |
|
Revision: 3e72b7ad35355b66ae9b34d2d26dbcb879d19f82 Submitted crossbow builds: ursa-labs/crossbow @ actions-476
|
Contributor
|
If I disable the arrow unity build, then it succeeds in linking aws-sdk, but then it fails building the vendored jemalloc. If I disable both jemalloc and unity-build, then everything works for me. |
Member
Author
|
In https://travis-ci.org/github/ursa-labs/crossbow/builds/716930465 it seems that cmake can't find aws-c-common, though I see it is downloaded, so I wonder if we're hitting that cmake issue I referenced yesterday @jeroen. |
Member
Author
Where do you see that? |
3e72b7a to
34534b2
Compare
Member
Author
|
Ok I've conditionally enabled |
Member
Author
|
I ran the autobrew locally and think I've got the full error here. It looks like it's linking some parquet lib and maybe needs to include the aws-sdk-cpp libs but isn't? I don't understand what it's trying to do or why though, particularly since we're building with Does this provide any insight @pitrou @jeroen? |
Member
Author
|
@github-actions crossbow submit homebrew-r-autobrew |
|
Revision: f990f73a94a392306fdbdb5ce7735c3b4387193d Submitted crossbow builds: ursa-labs/crossbow @ actions-483
|
Contributor
|
@nealrichardson this works for me: autobrew/homebrew-core#17 |
Member
Author
|
If the choice is between jemalloc and S3 support, I think we have to go with jemalloc. But hopefully we don't have to choose. I got something working locally, just trying now to see if it works on Travis. |
Member
Author
|
Hey this seemed to work: turning off the unity build (but jemalloc is still on), plus a few tweaks to the |
Member
Author
|
@github-actions crossbow submit homebrew-r-autobrew |
|
Revision: f4afc43981c2cdae9f1758d521ec0ac0e28921fc Submitted crossbow builds: ursa-labs/crossbow @ actions-484
|
Contributor
|
For me that didn't work on at least one version of macos. Can you try to submit a draft pr to autobrew-core to confirm ? |
Member
Author
|
So on retry that failed like: Which I also saw once locally but not subsequently. I don't understand what would cause |
Member
|
Just retry the build? This could be a sporadic issue. |
Member
|
No idea. Perhaps you can find someone with CMake and macOS expertise... :-S |
Member
Author
|
Googling the error pointed me back to our own JIRA. Looks like #456 fixed this issue previously, but #2779 undid the fix. Unclear why we're seeing this race condition only here, but I'll re-fix it (in a separate PR, to help with the next time we do |
nealrichardson
commented
Aug 18, 2020
ci/scripts/r_windows_build.sh
Outdated
Member
Author
There was a problem hiding this comment.
@jeroen have you updated the CRAN repo with aws-sdk-cpp yet?
Contributor
There was a problem hiding this comment.
No not yet. Will do soon if we confirm that it works (and you are sure you don't want to add more modules again :))
Member
Author
There was a problem hiding this comment.
I believe the macOS jemalloc install issue has been fixed now but I don't think I can easily confirm that on autobrew/homebrew-core#19 because you'd need to brew install --HEAD to pull the fix, and the brew test-bot command doesn't support head.
From the arrow perspective, builds are passing, so if you're satisfied that this is working, we can merge. And once we merge, no modules can get added to the cmake config without breaking the build, so there shouldn't be any more surprises.
Contributor
There was a problem hiding this comment.
You can try to cherry-pick the patch in the homebrew formula like so: https://github.com/autobrew/homebrew-core/blob/adebb0d668a1625c2c28d01df145ab7376ef5aef/Formula/mariadb-connector-c.rb#L9-L12
Member
Author
There was a problem hiding this comment.
Cool, that worked: https://travis-ci.org/github/autobrew/homebrew-core/builds/719332651
Member
Author
There was a problem hiding this comment.
Ok @jeroen, LMK when the CRAN rtools-packages is updated and I'll switch off of the bintray. No problem if that can't be done easily/quickly, I can merge this and swap the repos later.
Contributor
There was a problem hiding this comment.
I have uploaded it to my server, it should sync to the mirrors in a few hours.
kou
pushed a commit
that referenced
this pull request
Aug 19, 2020
On ARROW-6437 (#7928) we saw occasional "File exists" errors on `jemalloc_ep` on macOS. Googling the error message led back to ARROW-739 (#456), which fixed this before by forcing install with `-j1`. ARROW-3492 later made it so jemalloc would build (but not install) in parallel. Then ARROW-3539 (#2779) was addressing a problem with that change and, along with fixing the build parallelization issue, unfortunately reverted the original `make -j1 install` fix. This patch restores the fix from ARROW-739. Closes #7995 from nealrichardson/jemalloc-install Authored-by: Neal Richardson <neal.p.richardson@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
45705a9 to
363ee27
Compare
Member
Author
|
@github-actions crossbow submit homebrew-r-autobrew |
|
Revision: 97517a9 Submitted crossbow builds: ursa-labs/crossbow @ actions-494
|
This was referenced Aug 20, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds S3 support to the binary macOS and Windows (Rtools40 only, i.e. R >= 4.0) packages.