Add ON-CALL playbook by andraxylia · Pull Request #3466 · istio/istio

andraxylia · 2018-02-14T00:01:22Z

replaces https://goo.gl/9xUCRB and https://goo.gl/Hrg94p

ldemailly · 2018-02-14T00:26:05Z

try making branches only after pulling
the circle lint failure was because your branch was out of date
#3471

ldemailly · 2018-02-14T00:26:17Z

/lgtm

hklai · 2018-02-14T00:32:01Z

ON-CALL.md

+Every week, one Istio developer is on-call and is responsible for maintaining Istio build and help out users and other developers. The on-call person should prioritize on-call duties on top of their daily work.
+
+## Schedule
+The on-call schedule can be found [here][https://docs.google.com/spreadsheets/d/1FaHwPpad3F3hva2suJweNeTnocjKtLnbgLkyMRPzgUY/edit#gid=1475801904].


Incorrect md syntax.
Use () after [] for hyperlinks.

You can preview the text using the "view" button above too.

The [] works for me in my golang IDE viewer, but I will change it.

hklai · 2018-02-14T00:36:16Z

ON-CALL.md

+* Help with creating the release when needed.
+* Check the schedule sheet to make sure the next on call is defined.
+* On Friday, notify the next on-call.
+* On Tuesday morning, handoff to next oncall and send a summary to istio-dev containing these stats, that can be obtained by querying [github with dates ranges][https://help.github.com/articles/searching-issues-and-pull-requests/]


Please send hand off summary to istio-oncall@googlegroups.com

hklai · 2018-02-14T00:37:01Z

ON-CALL.md

+* Join google groups [istio-oncall][https://groups.google.com/forum/#!forum/istio-oncall]
+* Join the `oncall` Slack channel
+
+## Responsibilities


Please add post submit monitoring and fix

ldemailly · 2018-02-14T00:57:04Z

~~I would add in the summary that it replaces the other doc and in the other doc a pointer to this~~
EDIT: I did that

also make sure nothing is lost

ldemailly · 2018-02-14T00:57:43Z

/hold

ldemailly · 2018-02-14T00:58:32Z

(my lgtm was mostly to re-kick the build btw - also I (wrongly) assumed the preview had been looked and we'd iterate)

istio-merge-robot · 2018-02-14T03:07:33Z

/lgtm cancel //PR changed after LGTM, removing LGTM. @andraxylia @ldemailly

linsun · 2018-02-14T03:16:54Z

ON-CALL.md

+
+## Responsibilities
+* Build cop: monitor the builds, the presubmit automated tests, the postsubmit automated tests:
+  * Familiarize yourself with the current [open issues affecting automated tests](https://github.com/istio/istio/issues?q=is%3Aopen+is%3Aissue+label%3Akind%2Ftest-failure).


That seems a too long list to be familiar with. Is there a shortened list just for build/critical test break issue?

This is the list unfortunately, 20 open issues. If it is too long, people should prioritize fixing those bugs.

This is the list for critical issues, not all the issues.

linsun · 2018-02-14T03:19:09Z

ON-CALL.md

+#PRs with review approved / in flight:
+53 baseline / 22 current
+```
+


Would be good to have queries for these for reuse purpose.

They have to be adapted anyway with the new dates, so no point in adding them here. Please feel free to add them later if you find some that can be re-used.

linsun · 2018-02-14T03:21:14Z

ON-CALL.md

+  * If there are new failures, open issues and label them with kind/test-failure, with the appropriate area label, with "prow" or "circleci" label,
+  and assign them either directly to an engineer or to the area lead.
+  The issue must contain a link to the failed test log.
+  Add a comment in github cc-ing the assignees and explaining this must be fixed or reverted with highest priority. Nag people when needed.


What is the process of disable the required test when the test is determined to be not stable?

I have instructions to disable prow in an email, I can add them, but this can only be done by an administrator.

I added instructions, PTAL.

linsun · 2018-02-14T03:22:38Z

ON-CALL.md

+
+#Issues total
+488 baseline / 526 current
+


How do I interpret baseline? the # of issues at the beginning of the on call?

wonder if we should put the data in an online spreadsheet so that we can easily see trends among multiple weeks. that'd be more interesting data.

i am keeping a log in the oncall folder, maybe we can turn it into a sheet with weekly data

Sure, that can be done later. K8s does fancy graphs with the data, this is more for information purposes. It is not the on call duty to make judgments calls related to the number of open issues.

ldemailly · 2018-02-14T05:18:59Z

I find keeping an eye on https://github.com/orgs/istio/dashboard useful
and during handoff look at https://github.com/istio/istio/pulse which conveniently defaults to looking at 1 week in the past

andraxylia · 2018-02-14T16:20:22Z

Only a handful of people have github permissions to disable tests. Instructions are here for reference, but it will not be actually the on-call doing it. There is no need to add the note about the slippery slope, I hope those who have been given github admin permissions understand this.

andraxylia · 2018-02-14T16:21:15Z

Can I get an approval to merge this? People can iterate subsequently.

geeknoid · 2018-02-14T16:58:44Z

ON-CALL.md

+# Istio On-call Playbook
+
+## Who
+Every week, one Istio developer is on-call and is responsible for maintaining Istio build and for helping out users and other developers. The on-call person should prioritize on-call duties on top of the daily work.


maintaining THE Istio build PROCESS
on top of THEIR daily work

And should we mention it's not 24*7 on-call?

I will fix the grammar mistakes. I already mentioned this is during business hours for the time zone.

geeknoid · 2018-02-14T17:01:06Z

ON-CALL.md

+  * If there are new failures, open issues and label them with kind/test-failure, with the appropriate area label, with "prow" or "circleci" label,
+  and assign them either directly to an engineer or to the area lead.
+  The issue must contain a link to the failed test log.
+  Add a comment in github cc-ing the assignees and explaining this must be fixed or reverted with highest priority. Nag people when needed.


github -> GitHub

here and elsewhere in the doc.

geeknoid · 2018-02-14T17:06:29Z

ON-CALL.md

+* Help with creating the release when needed.
+* Check the schedule sheet to make sure the next on call is defined.
+* On Friday, notify the next on-call.
+* On Tuesday morning, handoff to next oncall and send a summary to istio-oncall and istio-dev. Include the stats below, that can be obtained by querying [github with dates ranges:](https://help.github.com/articles/searching-issues-and-pull-requests/)


oncall -> on-call

douglas-reid

a few comments.

douglas-reid · 2018-02-14T21:22:51Z

ON-CALL.md

+## Schedule
+The on-call schedule can be found [here](https://docs.google.com/spreadsheets/d/1FaHwPpad3F3hva2suJweNeTnocjKtLnbgLkyMRPzgUY/edit#gid=1475801904).
+
+On-call duty starts on Tuesday at noon PST, ends the following week on Tuesday at noon PST, and is performed during regular working hours for your time zone.


so, for non-PST timezoned folks, this implies that their oncall cycle might start at the beginning of their regular working hours some time thereafter, correct? and their cycles would end before the handoff?

For the oncall members in Israel, noon PST is like 10PM and 10AM is midnight PST. I'm not sure how much this matters, but is worth mentioning, especially as it regards oncall expectations.

Right, it is kind of implied by the business hours I mention previously.

douglas-reid · 2018-02-14T21:24:32Z

ON-CALL.md

+* Help with creating the release when needed.
+* Check the schedule sheet to make sure the next on call is defined.
+* On Friday, notify the next on-call.
+* On Tuesday morning, handoff to next oncall and send a summary to istio-oncall and istio-dev. Include the stats below, that can be obtained by querying [github with dates ranges:](https://help.github.com/articles/searching-issues-and-pull-requests/)


i'd be careful with "morning" given timezone differences. I think I'd phrase this as "at the end of your oncall shift".

douglas-reid · 2018-02-14T21:26:24Z

ON-CALL.md

+* Join the `oncall` Slack channel
+
+## Responsibilities
+* Build cop: monitor the builds, the presubmit automated tests, the postsubmit automated tests:


can we add a link to the test grid (that shows post-submit test status across PRs, etc.) somewhere here? http://k8s-testgrid.appspot.com/istio#Summary

wattli · 2018-02-14T23:49:26Z

ON-CALL.md

+* Uncheck the affected tests.
+
+
+


Remove empty lines.

wattli · 2018-02-14T23:50:12Z

ON-CALL.md

+# Istio On-call Playbook
+
+## Who
+Every week, one Istio developer is on-call and is responsible for maintaining Istio build and for helping out users and other developers. The on-call person should prioritize on-call duties on top of the daily work.


And should we mention it's not 24*7 on-call?

wattli · 2018-02-14T23:55:10Z

ON-CALL.md

+## Who
+Every week, one Istio developer is on-call and is responsible for maintaining Istio build and for helping out users and other developers. The on-call person should prioritize on-call duties on top of the daily work.
+
+## Schedule


And how do we do this oncall rotation? Based on volunteer or we have some rotation mechanism like round robin

I do not know this.

andraxylia · 2018-02-15T01:23:30Z

Updated the doc, PTAL. Remember this is not a legally binding document, so we can iterate if we find it does not work.

ldemailly · 2018-02-15T02:47:23Z

Thanks Andra ! I'll add more too after my shift
/lgtm

istio-merge-robot · 2018-02-15T02:48:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ldemailly

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these OWNERS Files:

~~OWNERS~~ [ldemailly]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

istio-merge-robot · 2018-02-15T02:48:23Z

/test all [submit-queue is verifying that this PR is safe to merge]

istio-merge-robot · 2018-02-15T03:31:23Z

Automatic merge from submit-queue.

Add ON-CALL playbook

26aade3

andraxylia requested a review from a team February 14, 2018 00:01

googlebot added the cla: yes label Feb 14, 2018

andraxylia requested review from ldemailly, louiscryan and mangchiandjjoe February 14, 2018 00:01

Merge branch 'master' into oncall

d7b5329

istio-testing assigned ldemailly Feb 14, 2018

istio-testing added the lgtm label Feb 14, 2018

istio-merge-robot added the approved label Feb 14, 2018

hklai reviewed Feb 14, 2018

View reviewed changes

istio-testing added the do-not-merge/hold Block automatic merging of a PR. label Feb 14, 2018

andraxylia added 2 commits February 13, 2018 19:06

Fix ()

96ff51c

Merge branch 'oncall' of https://github.com/istio/istio into oncall

42575f7

istio-merge-robot removed the lgtm label Feb 14, 2018

Minor fixes

5eb4095

linsun reviewed Feb 14, 2018

View reviewed changes

ldemailly removed the do-not-merge/hold Block automatic merging of a PR. label Feb 14, 2018

Add instructions for disabling Prow/CircleCI

33b52b7

geeknoid reviewed Feb 14, 2018

View reviewed changes

douglas-reid reviewed Feb 14, 2018

View reviewed changes

wattli reviewed Feb 14, 2018

View reviewed changes

Address review comments

3b40b56

istio-testing added the lgtm label Feb 15, 2018

istio-merge-robot merged commit ed69c3a into master Feb 15, 2018

ldemailly deleted the oncall branch February 15, 2018 05:38

Conversation

andraxylia commented Feb 14, 2018 • edited by ldemailly Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldemailly commented Feb 14, 2018

Uh oh!

ldemailly commented Feb 14, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldemailly commented Feb 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldemailly commented Feb 14, 2018

Uh oh!

ldemailly commented Feb 14, 2018

Uh oh!

istio-merge-robot commented Feb 14, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldemailly commented Feb 14, 2018

Uh oh!

andraxylia commented Feb 14, 2018

Uh oh!

andraxylia commented Feb 14, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

douglas-reid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andraxylia commented Feb 14, 2018 •

edited by ldemailly

Loading

ldemailly commented Feb 14, 2018 •

edited

Loading