Skip to content

Build against Apache Hadoop#242

Merged
chrisbennight merged 2 commits intolocationtech:masterfrom
dlyle65535:build-against-apache
Feb 27, 2015
Merged

Build against Apache Hadoop#242
chrisbennight merged 2 commits intolocationtech:masterfrom
dlyle65535:build-against-apache

Conversation

@dlyle65535
Copy link
Copy Markdown
Contributor

Create separate build profiles for Apache (default), Hortonworks (-P horonworks), and Cloudera (-P cloudera) distributions.

Addresses issue #241.

…hortonworks), and Cloudera (-P cloudera) distributions.
@coveralls
Copy link
Copy Markdown

Coverage Status

Coverage remained the same at 34.86% when pulling 0672ae6 on dlyle65535:build-against-apache into 7bdc096 on ngageoint:master.

@chrisbennight
Copy link
Copy Markdown
Contributor

Looks like we are having a zookeeper issue in the hadoop 2.6 version. (Guess based on the logs). My guess is a transitive dependency thing coming in from cloudera but not apache.

@dlyle65535
Copy link
Copy Markdown
Contributor Author

Yeah. It worked twice (once locally and once on Travis). What's your
recommendation for how to proceed?

On Wed, Feb 18, 2015 at 5:49 PM, Chris Bennight notifications@github.com
wrote:

Looks like we are having a zookeeper issue in the hadoop 2.6 version.
(Guess based on the logs). My guess is a transitive dependency thing coming
in from cloudera but not apache.


Reply to this email directly or view it on GitHub
#242 (comment).

@rfecher
Copy link
Copy Markdown
Contributor

rfecher commented Feb 18, 2015

Yeah, I do see it worked here: https://travis-ci.org/dlyle65535/geowave/jobs/51268058

So, I just restarted the travis job that failed and we'll see how that goes...its a bit concerning that it can randomly fail like that. Is there perhaps an xml file on the classpath of the hortonworks and cloudera distro's that reliably sets some config we may be missing? I'm just at a loss and throwing darts with that guess.

@dlyle65535
Copy link
Copy Markdown
Contributor Author

I'm also puzzled and concerned. What I tried to accomplish was keep the
repos separate. That is, don't use Hortonworks for Cloudera and vice versa.
Use neither for Apache.

On Wed, Feb 18, 2015 at 6:18 PM, rfecher notifications@github.com wrote:

Yeah, I do see it worked here:
https://travis-ci.org/dlyle65535/geowave/jobs/51268058

So, I just restarted the travis job that failed and we'll see how that
goes...its a bit concerning that it can randomly fail like that. Is there's
perhaps an xml file on the classpath of the hortonworks and cloudera
distro's that reliably sets some config we may be missing? I'm just at a
loss and throwing darts with that guess.


Reply to this email directly or view it on GitHub
#242 (comment).

@chrisbennight
Copy link
Copy Markdown
Contributor

I pulled it down over here and am checking to see if I can replicate the failure locally (make sure it's not a travis/load thing, which we have seen before with random failures - typically as the build matrix gets bigger).

@chrisbennight
Copy link
Copy Markdown
Contributor

So I ran all the tests on my ubuntu box (I need to build a 2.6 binary to get it running on windows - there's an ABI change (unsatisfied link error) with 2.5.2 native libs against 2.6)

They all passed except for the 2.6.0 one - re-running again now with debug mode + capturing output.

I want to run down exactly what is going on before we merge this in - if you can hold off a day or two we can hopefully ID and fix the issue, and then accept the pull request

@dlyle65535
Copy link
Copy Markdown
Contributor Author

Absolutely. Let's get it right. Thanks!

On Wed, Feb 18, 2015 at 6:57 PM, Chris Bennight notifications@github.com
wrote:

So I ran all the tests on my ubuntu box (I need to build a 2.6 binary to
get it running on windows - there's an ABI change (unsatisfied link error)
with 2.5.2 native libs against 2.6)

They all passed except for the 2.6.0 one - re-running again now with debug
mode + capturing output.

I want to run down exactly what is going on before we merge this in - if
you can hold off a day or two we can hopefully ID and fix the issue, and
then accept the pull request


Reply to this email directly or view it on GitHub
#242 (comment).

@chrisbennight
Copy link
Copy Markdown
Contributor

So I left a couple of builds running overnight on the ubuntu box
I ran 40 builds in different configurations, focusing on zookeeper

*Config 1: Mixed 3.3.6 and 3.4.6 (transitive from Accumulo and Hadoop 2.6.0, respectively). Integration tests failed 12 times (all centered around what looked like a closed local zookeeper, propagating tons of other errors - my suspicion is in the all in on / integration test harness configuration)
*Config 2: Forced 3.3.6. No failures
*Config 3: forced 3.4.6. No failures
*Config 4: forced 3.4.5-cdh5.2.0 No failures
*Config 5: Mixed 3.3.6 and 3.4.5-cdh5.2.0 1 failure. Looked similar to the zookeeper issue in (1)

Since (1) was repeatedly failing I stood up a small VM cluster and ran though an ingest (portion of the OSMGpx data), and scripted some random WMS queries. I was not able to cause any failures there - hence my guess above it's something in the all in one configuration. I also re-ran this with (3) above and received the same results (I serialized the results from (1) and compared them to (3))

ABI wise zookeeper looks pretty stable, so I'm going to vote for explicitly setting the zookeeper version to 3.4.6. Here's the diff between the pull request and the changes: 056de08
it hasn't had any issues on travis.

Thoughts?

@dlyle65535
Copy link
Copy Markdown
Contributor Author

Hi Chris-
Somehow this got lost in my e-mail box. Sorry about that. I have modified my branch. Thanks for digging into this!

@coveralls
Copy link
Copy Markdown

Coverage Status

Coverage remained the same at 34.86% when pulling 984a043 on dlyle65535:build-against-apache into 7bdc096 on ngageoint:master.

2 similar comments
@coveralls
Copy link
Copy Markdown

Coverage Status

Coverage remained the same at 34.86% when pulling 984a043 on dlyle65535:build-against-apache into 7bdc096 on ngageoint:master.

@coveralls
Copy link
Copy Markdown

Coverage Status

Coverage remained the same at 34.86% when pulling 984a043 on dlyle65535:build-against-apache into 7bdc096 on ngageoint:master.

chrisbennight added a commit that referenced this pull request Feb 27, 2015
@chrisbennight chrisbennight merged commit fb59922 into locationtech:master Feb 27, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants