Skip to content

Speed up CircleCI builds#1283

Merged
ebuchman merged 1 commit intodevelopfrom
feature/xla-run-integration-release
Mar 6, 2018
Merged

Speed up CircleCI builds#1283
ebuchman merged 1 commit intodevelopfrom
feature/xla-run-integration-release

Conversation

@xla
Copy link
Contributor

@xla xla commented Mar 6, 2018

To achieve faster feedback cycles for our feature PRs this change reduces the average buildtime from 35 to ~6min by utilising their new 2.0 offering based on docker and nomad. We make use of parallel build steps wherever possible so that the duration is determined by the slowest test suite (p2p).

This is an intermediate step until we move our CI/CD completely on-premise for more control and added security.


The workflow in action: https://circleci.com/workflow-run/1500b172-1104-4920-854c-eb68fe7a816f

@xla xla added the T:test Type: Tests that need love label Mar 6, 2018
@xla xla self-assigned this Mar 6, 2018
@xla xla requested review from greg-szabo and melekes March 6, 2018 14:45
@xla xla requested a review from ebuchman as a code owner March 6, 2018 14:45
@codecov-io
Copy link

codecov-io commented Mar 6, 2018

Codecov Report

Merging #1283 into develop will increase coverage by 0.42%.
The diff coverage is 0%.

@@             Coverage Diff             @@
##           develop    #1283      +/-   ##
===========================================
+ Coverage    59.96%   60.39%   +0.42%     
===========================================
  Files          127      112      -15     
  Lines        11623    10860     -763     
===========================================
- Hits          6970     6559     -411     
+ Misses        3988     3656     -332     
+ Partials       665      645      -20
Impacted Files Coverage Δ
types/priv_validator/socket.go 64.21% <0%> (ø) ⬆️
consensus/reactor.go 71.97% <0%> (-8.11%) ⬇️
p2p/node_info.go 54.71% <0%> (-3.78%) ⬇️
evidence/reactor.go 67.74% <0%> (-3.23%) ⬇️
p2p/peer.go 63.38% <0%> (-2.82%) ⬇️
mempool/reactor.go 68.49% <0%> (-2.74%) ⬇️
consensus/state.go 76.72% <0%> (-2.03%) ⬇️
p2p/listener.go 50% <0%> (-1.93%) ⬇️
p2p/pex/pex_reactor.go 66.26% <0%> (-1.61%) ⬇️
p2p/conn/connection.go 82.33% <0%> (-1.14%) ⬇️
... and 20 more

Makefile Outdated
@echo "--> Installing tools"
go get -u -v $(GOTOOLS)
# @gometalinter.v2 --install
@$(GOBIN)/gometalinter.v2 --install
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we had a long discussion about that. I would prefer for user to add GOPATH/bin to PATH as https://golang.org/doc/install#install suggests.

Example: If I installed dep from other source to /usr/local/bin, I still should be able to run make get_vendor_deps.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd support that sentiment, if we would check for existence of the binary in any location. But our entire Makefile infrastructure leverages the go toolchain to manage binaries.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally we shouldn't take any version of the tool, but one that we can control to avoid incompatibilities or other surprises.

Copy link
Contributor

@melekes melekes Mar 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if I don't have GOBIN? Do I have to set it manually?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/vagrant/go"
GORACE=""
GOROOT="/usr/lib/go-1.9"
GOTOOLDIR="/usr/lib/go-1.9/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build418845291=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed, no more $GOBIN in the Makefile.

To achieve faster feedback cycles for our feature PRs this change
reduces the average buildtime from 35 to ~6min by utilising their new
2.0 offering based on docker and nomad. We make use of parallel build
steps wherever possible so that the duration is determined by the
slowest test suite (p2p).

This is an intermediate step until we move our CI/CD completely
on-premise for more control and added security.
@xla xla force-pushed the feature/xla-run-integration-release branch from 1c0b50d to b7ce89e Compare March 6, 2018 16:40
Copy link
Contributor

@ebuchman ebuchman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ !

@ebuchman ebuchman merged commit 45d07a3 into develop Mar 6, 2018
@ebuchman ebuchman deleted the feature/xla-run-integration-release branch March 6, 2018 18:38

# TODO: install everything

export PATH="$GOBIN:$PATH"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we still need this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

T:test Type: Tests that need love

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants