Run test plans on Docker Swarm (AWS) by raulk · Pull Request #126 · testground/testground

raulk · 2019-11-06T00:16:33Z

This PR builds on #112. It introduces:

a cluster:swarm runner that takes a docker image of a test plan, and runs a test case within a Docker swarm cluster on AWS.
docker:go builder can now push to Docker registries, particularly AWS ECR registries. It does the aws ecr get-login dance and all!
an environment configuration file for things like AWS secrets, docker endpoints, etc. This is what mine looks like (eliding sensitive info):

["aws"]
access_key_id = "access key id"
secret_access_key = "secret access key"
region = "us-west-2"

[run_strategies."cluster:swarm"]
docker_endpoint = "tcp://localhost:4545"
docker_tls = false
docker_tls_ca_cert_path = "/"
docker_tls_cert_path = "/"
docker_tls_key_path = "/"

I'm hitting the remote Docker Swarm manager endpoint by creating an SSH tunnel for port forwarding:

$ ssh -nNT -L 4545:/var/run/docker.sock ubuntu@...compute.amazonaws.com

Needs to connect to the right network and fetch the right image.

Just need to make those a bit more dynamic and we will have a working runner!

…try.

It now schedules containers on AWS. By default it runs on the foreground and tails service logs, until all replicas are done, at which point it deletes the service. These behaviours are configurable via runner cfg.

Even if DOCKER_HOST is set.

daviddias · 2019-11-06T20:54:30Z

Could you:

Create a secrets-example.toml with the placeholder you pasted at Run test plans on Docker Swarm (AWS) #126 (comment) and add to .gitignore a secrets.toml
Add to the README the note on how to create that file (i.e. cp secrets-example.toml secrets.toml and add the the user own secrets)

raulk · 2019-11-06T20:57:31Z

@daviddias good ideas!

daviddias

Awesome work! Made some comments and suggestions. That said, @Stebalien should review as there is a lot of go things here that are beyond my current knowledge.

cmd/run.go

pkg/build/golang/docker.go

pkg/engine/engine.go

pkg/runner/cluster_swarm.go

README.md

daviddias

Thanks for applying CR :)

We now check the executable directory and the working directory.

cmd/run.go

pkg/aws/ecr.go

pkg/build/golang/docker.go

pkg/engine/engine.go

pkg/runner/cluster_swarm.go

daviddias · 2019-11-07T08:47:05Z

Only missing a @Stebalien 👍🏽 for the merge? :)

daviddias

CI lint is unhappy.

daviddias · 2019-11-07T09:41:44Z

pkg/runner/local_docker.go

 	}

+	if !cfg.KeepContainers {
+		defer deleteContainers(cli, log, containers)


Go is unhappy here - #126 (review)

Suggested change

defer deleteContainers(cli, log, containers)

defer func () {

err := deleteContainers(cli, log, containers)

if err != nil {

log.Errorw("error while deleting containers", "error", err)

}

}()

Or is there other best practice to silence this lint error?

In this case, we probably want to log.

daviddias

❓

jimpick and others added 3 commits November 6, 2019 00:09

Call Docker Swarm API from go

f2fd787

Needs to connect to the right network and fetch the right image.

Working example docker service with hardcoded redis, network and image

6e112bb

Just need to make those a bit more dynamic and we will have a working runner!

wip cluster:swarm runner on AWS, docker:go now pushes images to regis…

6aa9be6

…try.

raulk force-pushed the runner/cluster-swarm-aws branch from 222f0c4 to 6aa9be6 Compare November 6, 2019 00:18

raulk mentioned this pull request Nov 6, 2019

WIP: cluster:swarm runner #112

Closed

raulk added 6 commits November 6, 2019 10:21

propagate ECR auth token; check if redis is running.

3e3e8ed

clarify the role/usage of package pkg.

2648fcc

cluster:swarm runner: it works! 🙌

a315bfa

It now schedules containers on AWS. By default it runs on the foreground and tails service logs, until all replicas are done, at which point it deletes the service. These behaviours are configurable via runner cfg.

run command: option to bypass building.

1868549

docker:go builder: always run builds locally.

c09084e

Even if DOCKER_HOST is set.

improve logs.

0162b62

raulk requested review from Stebalien, daviddias and jimpick November 6, 2019 20:44

raulk marked this pull request as ready for review November 6, 2019 20:44

fix linter issues.

c7b7621

daviddias suggested changes Nov 6, 2019

View reviewed changes

cmd/run.go Outdated Show resolved Hide resolved

pkg/build/golang/docker.go Show resolved Hide resolved

pkg/engine/engine.go Show resolved Hide resolved

pkg/engine/engine.go Outdated Show resolved Hide resolved

pkg/runner/cluster_swarm.go Show resolved Hide resolved

raulk added 3 commits November 6, 2019 21:07

provide an example .env.toml file.

3dff93f

readme: usage instructions for .env.toml.

a56b82c

address review comments.

6a8afe1

raulk requested a review from daviddias November 6, 2019 21:22

docker:go builder: abort push when no registry type is specified

a69343a

daviddias suggested changes Nov 6, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

raulk requested a review from daviddias November 6, 2019 21:41

daviddias approved these changes Nov 6, 2019

View reviewed changes

feat: harden the logic to guess the source directory. (#130)

6dae0b4

We now check the executable directory and the working directory.

Stebalien suggested changes Nov 6, 2019

View reviewed changes

daviddias mentioned this pull request Nov 6, 2019

WIP: simplified smlbench - two containers (for comparison) #74

Closed

raulk added 2 commits November 6, 2019 23:28

bugfix: local:docker would remove containers too early (autoremove).

50a0f14

address review comments.

e84277f

raulk requested a review from Stebalien November 6, 2019 23:29

Stebalien approved these changes Nov 7, 2019

View reviewed changes

Merge branch 'master' into runner/cluster-swarm-aws

a5d3f65

daviddias suggested changes Nov 7, 2019

View reviewed changes

daviddias reviewed Nov 7, 2019

View reviewed changes

daviddias added the P0 - Critical label Nov 7, 2019

daviddias assigned raulk Nov 7, 2019

raulk force-pushed the runner/cluster-swarm-aws branch from a5d3f65 to 5687b96 Compare November 7, 2019 10:01

pacify linter.

c93a876

raulk force-pushed the runner/cluster-swarm-aws branch from 5687b96 to c93a876 Compare November 7, 2019 10:05

daviddias approved these changes Nov 7, 2019

View reviewed changes

daviddias merged commit 860b1fc into master Nov 7, 2019

daviddias deleted the runner/cluster-swarm-aws branch November 7, 2019 10:13

This was referenced Nov 7, 2019

Impl Plan - Bitswap Tuning #100

Closed

Test Plan - Message Delivery (PubSub) #98

Closed

jimpick mentioned this pull request Nov 9, 2019

TestGround Test Plans run on Cloud Infrastructure and its results are captured in a Dashboard #103

Closed

daviddias mentioned this pull request Nov 14, 2019

Ansible + Terraform of Docker Swarm Cluster in AWS - (Temporary ansible scripts - will be replaced with 'roles' layout in a separate PR) #111

Merged

-		defer deleteContainers(cli, log, containers)
+		defer func () {
+			err := deleteContainers(cli, log, containers)
+			if err != nil {
+				log.Errorw("error while deleting containers", "error", err)
+			}
+		}()

Conversation

raulk commented Nov 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daviddias commented Nov 6, 2019 • edited by raulk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raulk commented Nov 6, 2019

Uh oh!

daviddias left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

daviddias left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

daviddias commented Nov 7, 2019

Uh oh!

daviddias left a comment

Choose a reason for hiding this comment

Uh oh!

daviddias Nov 7, 2019 • edited by Stebalien Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Stebalien Nov 7, 2019

Choose a reason for hiding this comment

Uh oh!

Stebalien Nov 7, 2019

Choose a reason for hiding this comment

Uh oh!

daviddias left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

raulk commented Nov 6, 2019 •

edited

Loading

daviddias commented Nov 6, 2019 •

edited by raulk

Loading

daviddias Nov 7, 2019 •

edited by Stebalien

Loading