We should add an integration test suite to support testing against different versions of the daemon (platforms, EE/CE, API versions).

Proposal

A new test suite should be written for the docker/cli repo. The suite should accept two variables:

• DOCKER_BINARY_PATH - the path to the compiled docker binary (default to build/docker)
• DOCKER_HOST - the url to the docker daemon API

CI should test PRs against an appropriate matrix of daemons.

Each test case should be written in an end-to-end test style:

• set requirements for minimum supported API version, platform, experimental, etc
• setup appropriate test state required for the test
• run the binary using docker/docker/pkg/testutil/cmd
• assert expected ExitCode, Stdout/Stderr, and local filesystem state (The state of the daemon should not be tested! That behaviour is outside of the CLI, so is already covered by daemon test cases).

Guidelines for integration testing

All code changes should have unit test coverage. Integration tests do not replace unit tests.

Each feature (subcommand) should generally only have one integration testing for the success case ("happy path"). The test should include all (or most) flags/options supported by that feature.

In some rare cases a couple additional integration tests may be written for a sufficiently complex and critical feature (ex: container run, service create, docker build may have ~3 cases each).

In some rare cases a sufficiently critical error path may have a single integration test case.

In all other cases the behaviour should be covered by unit tests.

If a code change adds a new flag, that flag should be added to the existing "success case" integration test.

If a code change fixes a bug, that bug fix should be covered either by adding assertions to the existing integration test, or with unit tests.

Integration suite details

• use go test with testify.assert/suite and docker/docker/pkg/testutil
• add a Jenkinsfile to the repo to run the test matrix on each PR: moby/moby master (sometimes failures can be ignored), latest CE release, latest EE release, current RC release)
• tests should go into ./test-e2e/ (or ./features, or something like that) and mirror the directory structure of commands:

  .
  |- test-e2e
  |     |- image
  |     |    |- pull_test.go
  |     |    |- build_test.go
  |     |    \- ...
  |     |- container
  |     |    | ...
  

Related

• unit tests on commands #37 - tracking issue for missing unit tests
• [Testing] Tracking issue for missing tests #201 - tracking issue for missing integration tests