Conversation
| var AllRunners = []runner.Runner{ | ||
| &runner.LocalDockerRunner{}, | ||
| &runner.LocalExecutableRunner{}, | ||
| &runner.ClusterSwarmRunner{}, |
There was a problem hiding this comment.
| &runner.ClusterSwarmRunner{}, | |
| &runner.ClusterDockerSwarmRunner{}, |
There was a problem hiding this comment.
I'm not entirely sure if naming yet another Runner just for Docker Swarm is the best path. It probably worth to just have Docker Swarm as the default Docker Runner
There was a problem hiding this comment.
Or perhaps this is me just getting confused by seeing Docker Swarm being called a Runner (given that in the end it is just the same thing running in a Docker container, but managed by Docker Swarm)
There was a problem hiding this comment.
I think calling this ClusterSwarmRunner is fine. I was using the convention scope:technology, e.g. local:exec, local:docker, etc. to name runners. We can also rename "runners" to "schedulers" if we feel that's more accurate.
pkg/runner/cluster_swarm.go
Outdated
| // docker service create --replicas 1 --name helloworld alpine ping docker.com | ||
|
|
||
| replicas := uint64(input.Instances) | ||
| spec := swarm.ServiceSpec{ |
There was a problem hiding this comment.
Can I get a clarification of this way of specing the service vs. using docker-compose.yml? (https://docs.docker.com/compose/)
There was a problem hiding this comment.
Now that we've decided to create a temporary overlay network per run, we could use Docker Compose to create both the network and the service in one go. However, IIRC Docker Compose is really a CLI tool written in Python that transforms a YAML into docker calls. We cannot use it from Golang :-(
| } | ||
|
|
||
| // Temp Redis fix | ||
| env = append(env, "REDIS_HOST=172.31.14.166") |
There was a problem hiding this comment.
Just flagging things to discuss.
pkg/runner/cluster_swarm.go
Outdated
| }, | ||
| TaskTemplate: swarm.TaskSpec{ | ||
| ContainerSpec: &swarm.ContainerSpec{ | ||
| Image: "909427826938.dkr.ecr.us-west-2.amazonaws.com/testground:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855@sha256:e175f10c2fc0545ede1de08458dffbea5b3efb3c023963028eac9129f4fd5920", |
There was a problem hiding this comment.
Based on discussion today, the docker:go builder should push the image to ECR if we receive a docker_registry param.
pkg/runner/cluster_swarm.go
Outdated
| }, | ||
| Networks: []swarm.NetworkAttachmentConfig{ | ||
| { | ||
| Target: "hw6dcms11qhf3iv3rr2j8a2vb", |
There was a problem hiding this comment.
We need to create a new overlay network per deployment.
There was a problem hiding this comment.
i.e. the data plane would be specific to the test case.
pkg/runner/cluster_swarm.go
Outdated
| Env: env, | ||
| }, | ||
| RestartPolicy: &swarm.RestartPolicy{ | ||
| Condition: swarm.RestartPolicyConditionNone, |
| var AllRunners = []runner.Runner{ | ||
| &runner.LocalDockerRunner{}, | ||
| &runner.LocalExecutableRunner{}, | ||
| &runner.ClusterSwarmRunner{}, |
There was a problem hiding this comment.
I think calling this ClusterSwarmRunner is fine. I was using the convention scope:technology, e.g. local:exec, local:docker, etc. to name runners. We can also rename "runners" to "schedulers" if we feel that's more accurate.
pkg/runner/cluster_swarm.go
Outdated
| // docker service create --replicas 1 --name helloworld alpine ping docker.com | ||
|
|
||
| replicas := uint64(input.Instances) | ||
| spec := swarm.ServiceSpec{ |
There was a problem hiding this comment.
Now that we've decided to create a temporary overlay network per run, we could use Docker Compose to create both the network and the service in one go. However, IIRC Docker Compose is really a CLI tool written in Python that transforms a YAML into docker calls. We cannot use it from Golang :-(
|
Taking this to land it on top of @jimpick's work on the infra side of things. |
Needs to connect to the right network and fetch the right image.
Just need to make those a bit more dynamic and we will have a working runner!
f9200fb to
222f0c4
Compare
I have an automated Ansible setup for Docker swarm in the aws-ansible branch (needs just a bit more cleanup). I first tested it with a shell script, but then I realized that it wouldn't be too much work to modify the local:docker runner to talk to the swarm API.
Here's a demo running the smlbench2 2-container bitswap test across 2 different EC2 vms in the Docker swarm cluster:
https://github.com/ipfs/testground/blob/aws-ansible/pkg/runner/cluster_swarm.go
It's not quite done yet ... I've hard-coded the image (pulling from ECR), the REDIS_HOST environment variable, and the network. We'll want to upload the image to ECR after building it before creating the service.
Not complete yet ... some hard-coded values. See demo of it running in #110 (along with notes about what is still missing)
I just hacked this together to test the ansible setup I've been prototyping in the aws-ansible branch:
https://github.com/ipfs/testground/tree/aws-ansible
(tested with the smlbench2 test, which isn't on master yet ... that's going to be the basis of #94)