Doc and deployment script fixes for EMR bootstrap failures#634
Merged
rwgdrummer merged 1 commit intomasterfrom Feb 22, 2016
Merged
Doc and deployment script fixes for EMR bootstrap failures#634rwgdrummer merged 1 commit intomasterfrom
rwgdrummer merged 1 commit intomasterfrom
Conversation
rwgdrummer
added a commit
that referenced
this pull request
Feb 22, 2016
Doc and deployment script fixes for EMR bootstrap failures
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This request contains fixes for #633, "GeoWave on EMR Fails During Bootstrap". Copies of these files have already been placed in the GeoWave S3 bucket used by the Jenkins EMR launch script.
The first problem was http://apache.mirrors.tds.net/accumulo/ removed all of the Accumulo binary archive files except for the recently released 1.6.5 and 1.7.0 and we defaulted to 1.6.4 To fix I made ACCUMULO_DOWNLOAD_BASE_URL a setting that defaults to https://archive.apache.org/dist/accumulo which has both recent and past releases. We could also vendor our own copies somewhere else if artifacts disappearing or moving continues to be a problem.
The second problem was the URL to the RPM GPG signing key for the Apache BigTop repo was recently moved and the repo file I was curling into /etc/yum.repos.d still listed the old version so zookeeper was not getting installed causing Accumulo to error out during install and the whole cluster would get terminated.
While looking into this problem I noticed Amazon recently put out a new version of EMR v4.3.0 which includes a bundled version of the BigTop repo. So the fix was to just use the repo included in the newer version of EMR.
Version upgrades in EMR 4.3.0
From: http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-whatsnew.html
I tested that the cluster would start as expected with the newly released Accumulo 1.6.5 and Accumulo 1.7.0 but didn't do any extensive testing with GeoWave to validate.