Automatic service rollback on failure #31108

aaronlehmann · 2017-02-17T02:41:00Z

This PR adds support for a "rollback" failure action. When this is selected, the orchestrator will automatically roll back to the previous version of a service if an update hits the failure threshold.

It adds the following options to control the behavior of rollbacks. They are kept separate from the existing --update-* options, since there are situations where the behavior should be different for rollbacks. For example, an administrator may wish to roll out a new version of a service very slowly, but if failure thresholds are exceeded, switch back to the old version quickly. Arguably, we may be able to live with hardcoded defaults for some of these options, but it seemed better to stay consistent with the --update-* options than to introduce arbitrary differences.

--rollback-delay
--rollback-failure-action
--rollback-max-failure-ratio
--rollback-monitor
--rollback-parallelism

I'm open to reducing the set of options if desired, but at a minimum we should keep --rollback-delay and --rollback-parallelism.

This PR also switches the manually invoked service update --rollback to work on the server side, instead of being a purely client side operation. This allows manually-initiated rollbacks to respect the new rollback parameters, and it also has the benefits of avoiding some nasty hacks we previously used. The client is version-aware, so it will still use the old method against an older daemon.

Note that --rollback has been changed so that it can't be used in combination with other service update flags. This is a compromise to avoid adding further hacks. While it's unfortunate to be restricting this functionality, it does make the implementation much cleaner.

cc @aluzzardi @dongluochen

yu8833 · 2017-02-17T07:59:51Z

docs/reference/commandline/service_create.md

add "|rollback"?
📝

Fixed, thanks

aaronlehmann · 2017-02-21T18:02:15Z

Rebased

dongluochen · 2017-02-22T18:52:55Z

api/swagger.yaml

=> Amount of time between rollback iterations,

dongluochen · 2017-02-22T19:08:42Z

daemon/cluster/convert/service.go

If the value from GRPC is unrecognized, there is nothing we can really do here. We could return an error, but I think that would be too extreme. This is not changing the old behavior, just reorganizing the code and adding a new case.

dongluochen · 2017-02-22T19:11:15Z

daemon/cluster/executor/container/controller.go

Is there a reason this PR is mixed with rollback change?

This was causing problems testing the change. I can break it out to a separate PR if you want.

I'm fine if this will be merged soon. This PR might be cherry-picked.

I agree we should not block the shutdown because a failure in the deactivate service binding, unless user is given a command to fix it. I was just looking at it, while I came to know from @dongluochen this was taken care in this PR as a side change.

Given this PR may take some time to go through, I am also thinking it would be good to have a separate PR which can be quickly merged.

I pulled this part out into #31279

dongluochen · 2017-02-22T19:14:14Z

docs/reference/commandline/service_create.md

(pause|continue|rollback) => (pause|continue)

My mistake, this was a rebase issue.

dongluochen · 2017-02-22T19:15:25Z

docs/reference/commandline/service_update.md

no rollback on rollback failure.

My mistake, this was a rebase issue.

aaronlehmann · 2017-02-22T20:59:12Z

@dongluochen: Addressed your comments, thanks.

dongluochen · 2017-02-22T21:54:08Z

LGTM

aluzzardi · 2017-02-25T01:08:09Z

LGTM

aaronlehmann · 2017-03-02T01:05:45Z

Rebased

vdemeester

LGTM 🐯

yongtang · 2017-03-03T17:17:00Z

Docs change LGTM.

yongtang · 2017-03-03T17:18:48Z

bash and zsh completion could be done in the follow up PRs.

yongtang · 2017-03-03T19:16:45Z

I think it might be good to add a simple example section in the docs, and docs/api/version-history.md may need to be updated to reflect the changes in API.

Also, the PR needs a rebase.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

Server-side rollback can take advantage of the rollback-specific update parameters, instead of being treated as a normal update that happens to go back to a previous version of the spec. Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

aaronlehmann · 2017-03-04T00:36:57Z

Thanks @yongtang. I've updated the API version history and added some content to service_update.md.

yongtang · 2017-03-04T02:47:12Z

LGTM. Thanks!

yongtang · 2017-03-04T02:49:27Z

/cc @albers @sdurrheimer for bash/zsh completion.

GordonTheTurtle added the status/0-triage label Feb 17, 2017

aaronlehmann force-pushed the automatic-rollback branch from 367c595 to 96b1ef6 Compare February 17, 2017 02:42

cpuguy83 added status/1-design-review and removed status/0-triage labels Feb 17, 2017

yu8833 reviewed Feb 17, 2017

View reviewed changes

aaronlehmann force-pushed the automatic-rollback branch 2 times, most recently from 4d498a6 to 9b1bbf9 Compare February 17, 2017 18:08

aaronlehmann mentioned this pull request Feb 18, 2017

Synchronous service create and service update #31144

Merged

7 tasks

aaronlehmann force-pushed the automatic-rollback branch from 9b1bbf9 to 9b99e39 Compare February 21, 2017 18:01

dongluochen reviewed Feb 22, 2017

View reviewed changes

aaronlehmann force-pushed the automatic-rollback branch from 9b99e39 to d874a3d Compare February 22, 2017 20:59

aaronlehmann mentioned this pull request Feb 23, 2017

Shutdown leaks an error when the container was never started #31279

Merged

vdemeester added status/2-code-review and removed status/1-design-review labels Feb 26, 2017

aaronlehmann mentioned this pull request Feb 27, 2017

swarm: Add update/rollback order #30261

Merged

aaronlehmann force-pushed the automatic-rollback branch from d874a3d to 014be19 Compare March 2, 2017 01:05

vdemeester approved these changes Mar 2, 2017

View reviewed changes

aaronlehmann added status/3-docs-review and removed status/2-code-review labels Mar 2, 2017

GordonTheTurtle assigned yongtang Mar 3, 2017

aaronlehmann mentioned this pull request Mar 3, 2017

Fix error caused by overlapping merge of #30733 #31533

Merged

Add support for the "rollback" failure action

cc9d046

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

Aaron Lehmann added 2 commits March 3, 2017 16:33

Add support for rollback flags

3a88a24

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

aaronlehmann force-pushed the automatic-rollback branch from 014be19 to f9bd8ec Compare March 4, 2017 00:36

yongtang merged commit 3385658 into moby:master Mar 4, 2017

GordonTheTurtle added this to the 17.04.0 milestone Mar 4, 2017

aaronlehmann deleted the automatic-rollback branch March 4, 2017 02:50

thaJeztah added impact/api impact/changelog impact/cli impact/documentation labels Mar 4, 2017

albers mentioned this pull request Mar 5, 2017

Add bash completion for service create|update --rollback-* #31558

Merged

thaJeztah mentioned this pull request Mar 6, 2017

Service update failure thresholds and rollback #26421

Merged

nishanttotla mentioned this pull request Mar 10, 2017

Regression: no warnings printed when digest pinning fails on service update #31716

Closed

thaJeztah mentioned this pull request Mar 20, 2017

A better docker service update behaviour toward services with replicas = 1 #31955

Open

sdurrheimer mentioned this pull request Mar 22, 2017

Add zsh completion for 'docker service {create,update} --rollback-*' #32020

Merged

cpuguy83 mentioned this pull request Mar 28, 2017

Swarm replica goes in restart loop when it cannot retrieve image after service update #32158

Closed

mdlinville mentioned this pull request Apr 12, 2017

Document service rollback docker/docs#2754

Closed

aaronlehmann mentioned this pull request Apr 12, 2017

Support rollback options in stack deploy #32585

Closed

thaJeztah mentioned this pull request May 30, 2017

Can't rollback service automatically after update #33427

Closed

amegianeg mentioned this pull request May 31, 2017

Document 'rollback' for 'update_config.failure_action' in docker compose docker/cli#141

Closed

thaJeztah mentioned this pull request Jun 1, 2017

Allow --detach and --quiet flags when using --rollback docker/cli#144

Merged

Automatic service rollback on failure #31108

Automatic service rollback on failure #31108

Uh oh!

Conversation

aaronlehmann commented Feb 17, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aaronlehmann commented Feb 21, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aaronlehmann commented Feb 22, 2017

Uh oh!

dongluochen commented Feb 22, 2017

Uh oh!

aluzzardi commented Feb 25, 2017

Uh oh!

aaronlehmann commented Mar 2, 2017

Uh oh!

vdemeester left a comment

Choose a reason for hiding this comment

Uh oh!

yongtang commented Mar 3, 2017

Uh oh!

yongtang commented Mar 3, 2017

Uh oh!

yongtang commented Mar 3, 2017

Uh oh!

aaronlehmann commented Mar 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yongtang commented Mar 4, 2017

Uh oh!

yongtang commented Mar 4, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

aaronlehmann commented Mar 4, 2017 •

edited

Loading