Create Post Take-Master Processes Hook by daniel-2647 · Pull Request #859 · openark/orchestrator

daniel-2647 · 2019-04-05T13:25:34Z

Description

This PR creates a hook which would only be called upon a successful Take-Master event. This would enable custom scripts to be called at those times when a post failover hook is not called, i.e. When we do a graceful failover of an intermediary master (See #799 for full details)

This PR introduces 1 new config option:

PostTakeMasterProcesses : "some PostTakeMasterHook here"
There is an issue created for this a few months back with details on the reasoning behind this PR: #799

contributed code is using same conventions as original code
code is formatted via gofmt (please avoid goimports)
code is built via ./build.sh
code is tested via go test ./go/...

Added PostTakeMasterProcessesEnabled and PostTakeMasterProcesses to config

Imported go/os and created new functions to enable Post Take-Master Hooks

Added sample config options to enable Post TakeMaster Processes

daniel-2647 · 2019-04-05T19:17:31Z

What does this error mean?

+gofmt -s -w go/
+git diff --exit-code --quiet
The command "script/cibuild" exited with 1.'''

…Master event

daniel-2647 · 2019-04-09T14:32:06Z

Sample outcome when using this functionality: (this PR addresses this issue #799)

Pre take-master:

[root@lab-proxysql-1 ~]# orchestrator-client -c topology -i lab-applogdb-1:3306
lab-shadowmaster:3306   [0s,ok,10.3.13-MariaDB-log,rw,MIXED,>>]
+ lab-applogdb-1:3306   [0s,ok,10.3.13-MariaDB-log,rw,MIXED,>>,GTID]
  + lab-applogdb-2:3306 [0s,ok,10.3.13-MariaDB-log,ro,MIXED,>>,GTID]
  + lab-applogdb-3:3306 [0s,ok,10.3.13-MariaDB-log,ro,MIXED,>>,GTID]

[root@lab-proxysql-1 ~]# echo "select * from mysql_servers" | proxysql
hostgroup_id	hostname	port	status	weight	compression	max_connections	max_replication_lag	use_ssl	max_latency_ms	comment
0	lab-applogdb-1	3306	ONLINE	1000	0	1000	0	0	0
1	lab-applogdb-2	3306	ONLINE	200	0	1000	0	0	0
1	lab-applogdb-3	3306	ONLINE	10	0	1000	0	0	0
1	lab-applogdb-1	3306	ONLINE	1000	0	1000	0	0	0

[root@lab-proxysql-1 ~]# consul kv get -recurse
mysql/intmaster/STAGE/lab-applogdb-1:192.168.96.101
mysql/master/STAGE/lab-shadowmaster:3306:lab-shadowmaster:3306
mysql/master/STAGE/lab-shadowmaster:3306/hostname:lab-shadowmaster
mysql/master/STAGE/lab-shadowmaster:3306/ipv4:192.168.96.107
mysql/master/STAGE/lab-shadowmaster:3306/ipv6:
mysql/master/STAGE/lab-shadowmaster:3306/port:3306
mysql/slave/STAGE/lab-applogdb-2:192.168.96.102
mysql/slave/STAGE/lab-applogdb-3:192.168.96.103

Post successful take-master (the hook is only called at the end and upon success of take-master process) is called and custom changes are applied:

[root@lab-proxysql-1 ~]# orchestrator-client -c topology -i lab-applogdb-1:3306
lab-shadowmaster:3306   [0s,ok,10.3.13-MariaDB-log,rw,MIXED,>>]
+ lab-applogdb-2:3306   [0s,ok,10.3.13-MariaDB-log,rw,MIXED,>>,GTID]
  + lab-applogdb-1:3306 [0s,ok,10.3.13-MariaDB-log,ro,MIXED,>>,GTID]
  + lab-applogdb-3:3306 [0s,ok,10.3.13-MariaDB-log,ro,MIXED,>>,GTID]

[root@lab-proxysql-1 ~]# consul kv get -recurse
mysql/intmaster/STAGE/lab-applogdb-2:192.168.96.102
mysql/master/STAGE/lab-shadowmaster:3306:lab-shadowmaster:3306
mysql/master/STAGE/lab-shadowmaster:3306/hostname:lab-shadowmaster
mysql/master/STAGE/lab-shadowmaster:3306/ipv4:192.168.96.107
mysql/master/STAGE/lab-shadowmaster:3306/ipv6:
mysql/master/STAGE/lab-shadowmaster:3306/port:3306
mysql/slave/STAGE/lab-applogdb-1:192.168.96.101
mysql/slave/STAGE/lab-applogdb-3:192.168.96.103

[root@lab-proxysql-1 ~]# echo "select * from mysql_servers" | proxysql
hostgroup_id	hostname	port	status	weight	compression	max_connections	max_replication_lag	use_ssl	max_latency_ms	comment
1	lab-applogdb-1	3306	ONLINE	200	0	1000	0	0	0
0	lab-applogdb-2	3306	ONLINE	1000	0	1000	0	0	0
1	lab-applogdb-2	3306	ONLINE	1000	0	1000	0	0	0
1	lab-applogdb-3	3306	ONLINE	10	0	1000	0	0	0

shlomi-noach

Thank you for this PR! I've made a few requests for changes.

go/inst/instance_topology.go

…g.go, instance_topology.go and orchestrator-sample.conf.json

shlomi-noach · 2019-04-18T13:21:03Z

Thank you! I apologize for the delayed response. Completely swamped, and will be out next week. Will hope to respond by end of this month.

daniel-2647 · 2019-04-18T13:54:24Z

Thank you! I apologize for the delayed response. Completely swamped, and will be out next week. Will hope to respond by end of this month.

No worries at all Shlomi, thanks for all the work 🥇

shlomi-noach · 2019-04-30T12:13:08Z

What does this error mean?

The tests ensure code is formatted via go fmt.

shlomi-noach

Looking good, thank you!

shlomi-noach · 2019-04-30T12:22:09Z

For next time, to make the process smoother, please use a branch other than master 🙏

daniel-2647 · 2019-04-30T13:19:17Z

Thanks Shlomi, appreciate you looking at the request.

daniel-2647 · 2019-04-30T13:26:51Z

For next time, to make the process smoother, please use a branch other than master 🙏

Will do sir, my apologies for the confusion and the extra work I've caused. Thanks for looking through this PR and approving the changes.

yakirgb · 2020-06-25T17:30:28Z

Hi @daniel-2647 , general question on your comment:

[root@lab-proxysql-1 ~]# consul kv get -recurse
mysql/intmaster/STAGE/lab-applogdb-1:192.168.96.101
mysql/master/STAGE/lab-shadowmaster:3306:lab-shadowmaster:3306
mysql/master/STAGE/lab-shadowmaster:3306/hostname:lab-shadowmaster
mysql/master/STAGE/lab-shadowmaster:3306/ipv4:192.168.96.107
mysql/master/STAGE/lab-shadowmaster:3306/ipv6:
mysql/master/STAGE/lab-shadowmaster:3306/port:3306
mysql/slave/STAGE/lab-applogdb-2:192.168.96.102
mysql/slave/STAGE/lab-applogdb-3:192.168.96.103

How did you add/update consul mysql/slave or mysql/intmaster? in your hook of take-over?

daniel-2647 · 2020-07-06T20:03:49Z

Hi @daniel-2647 , general question on your comment:

[root@lab-proxysql-1 ~]# consul kv get -recurse
mysql/intmaster/STAGE/lab-applogdb-1:192.168.96.101
mysql/master/STAGE/lab-shadowmaster:3306:lab-shadowmaster:3306
mysql/master/STAGE/lab-shadowmaster:3306/hostname:lab-shadowmaster
mysql/master/STAGE/lab-shadowmaster:3306/ipv4:192.168.96.107
mysql/master/STAGE/lab-shadowmaster:3306/ipv6:
mysql/master/STAGE/lab-shadowmaster:3306/port:3306
mysql/slave/STAGE/lab-applogdb-2:192.168.96.102
mysql/slave/STAGE/lab-applogdb-3:192.168.96.103

How did you add/update consul mysql/slave or mysql/intmaster? in your hook of take-over?

I created a postfailoverhook.sh script which gets called by the post failover hook. So, in the config, I have something like this:

"PostGracefulTakeoverProcesses": [
    "/usr/local/orchestrator/postfailoverhook.sh >> /tmp/recovery.log 2>&1",
    "echo 'Planned takeover complete' >> /tmp/recovery.log"
  ],

and the shell script makes calls to update the consul store, sample below: (not complete code)

echo "Removing new master from slave pool: ${slave_pool}/${NewMaster}"
${CONSUL_EXEC} kv delete ${slave_pool}/${NewMaster}

echo "Adding old master to slave pool: ${ORC_FAILED_HOST}"
${CONSUL_EXEC} kv put ${slave_pool}/${ORC_FAILED_HOST} ${OldMasterIP}

yakirgb · 2020-08-11T13:10:15Z

Thank you @daniel-2647 .

yakirgb · 2020-08-11T13:11:17Z

Also PreTakeMasterProcessescan be useful

yakirgb · 2020-08-11T13:11:17Z

Also PreTakeMasterProcessescan be useful

daniel-2647 added 3 commits April 5, 2019 09:16

Added new options to config

7815d36

Added PostTakeMasterProcessesEnabled and PostTakeMasterProcesses to config

Created new functions and added missing import

ffe0ed5

Imported go/os and created new functions to enable Post Take-Master Hooks

Added sample config

4346f07

Added sample config options to enable Post TakeMaster Processes

daniel-2647 mentioned this pull request Apr 5, 2019

Create Post Take-Master Processes Hook #858

Closed

4 tasks

Added ref to goos and update env var

5c6b159

daniel-2647 and others added 2 commits April 9, 2019 02:05

Created function to allow a hook to be called after a successful Take…

6e22daa

…Master event

Merge branch 'master' of github.com:daniel-2647/orchestrator

a2f688b

shlomi-noach suggested changes Apr 10, 2019

View reviewed changes

Made changes based on suggestions by Shlomi. Files changed were confi…

445fe27

…g.go, instance_topology.go and orchestrator-sample.conf.json

Merge branch 'master' into master

5ad9587

shlomi-noach approved these changes Apr 30, 2019

View reviewed changes

shlomi-noach changed the base branch from master to take-master-hook April 30, 2019 12:19

shlomi-noach changed the base branch from take-master-hook to master April 30, 2019 12:20

shlomi-noach mentioned this pull request Apr 30, 2019

Create Post Take-Master Processes Hook #869

Merged

shlomi-noach temporarily deployed to production/mysql_cluster=concertmaster April 30, 2019 12:23 Inactive

shlomi-noach temporarily deployed to production/mysql_cluster=conductor May 1, 2019 08:28 Inactive

shlomi-noach merged commit a641bc5 into openark:master May 1, 2019

Conversation

daniel-2647 commented Apr 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

daniel-2647 commented Apr 5, 2019

Uh oh!

daniel-2647 commented Apr 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shlomi-noach left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shlomi-noach commented Apr 18, 2019

Uh oh!

daniel-2647 commented Apr 18, 2019

Uh oh!

shlomi-noach commented Apr 30, 2019

Uh oh!

shlomi-noach left a comment

Choose a reason for hiding this comment

Uh oh!

shlomi-noach commented Apr 30, 2019

Uh oh!

daniel-2647 commented Apr 30, 2019

Uh oh!

daniel-2647 commented Apr 30, 2019

Uh oh!

yakirgb commented Jun 25, 2020

Uh oh!

daniel-2647 commented Jul 6, 2020

Uh oh!

yakirgb commented Aug 11, 2020

Uh oh!

yakirgb commented Aug 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yakirgb commented Aug 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

daniel-2647 commented Apr 5, 2019 •

edited

Loading

daniel-2647 commented Apr 9, 2019 •

edited

Loading

yakirgb commented Aug 11, 2020 •

edited

Loading

yakirgb commented Aug 11, 2020 •

edited

Loading