Skip to content

box: make bootstrap_leader cfg option dynamic#10769

Merged
sergepetrenko merged 1 commit intotarantool:masterfrom
sergepetrenko:gh-10604-bootstrap-leader-dynamic
Nov 12, 2024
Merged

box: make bootstrap_leader cfg option dynamic#10769
sergepetrenko merged 1 commit intotarantool:masterfrom
sergepetrenko:gh-10604-bootstrap-leader-dynamic

Conversation

@sergepetrenko
Copy link
Collaborator

bootstrap_leader configuration option only takes effect during instance bootstrap. For this reason, the option was made static. One can't change the bootstrap_leader after initial box.cfg() call.

So when a user has bootstrap_strategy = 'config' and joins new nodes during replicaset lifespan, he ends up with instances having different bootstrap_leader configuration, depending on which node was writable at the moment of each instance bootstrap.

It would be better to have matching configuration on each replica set member with no exceptions and to be able to set bootstrap_leader to the same value on all the instances of the replica set, even on the ones which are already bootstrapped.

Let's allow changing bootstrap_leader option even though it affects nothing.

Closes #10604

NO_DOC=the doc already states the option is dynamic

@sergepetrenko sergepetrenko requested a review from a team as a code owner October 31, 2024 12:54
@sergepetrenko sergepetrenko self-assigned this Oct 31, 2024
@coveralls
Copy link

coveralls commented Oct 31, 2024

Coverage Status

coverage: 87.286% (-0.03%) from 87.312%
when pulling b5779ea on sergepetrenko:gh-10604-bootstrap-leader-dynamic
into a6aa249
on tarantool:master
.

Copy link
Contributor

@Totktonada Totktonada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see no flaws. Thank you!

@Totktonada Totktonada removed their assignment Oct 31, 2024
@sergepetrenko sergepetrenko force-pushed the gh-10604-bootstrap-leader-dynamic branch from 3c36ff4 to b429ae7 Compare November 1, 2024 12:38
Copy link
Contributor

@Serpentian Serpentian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! No objections from my side

@sergepetrenko sergepetrenko added the full-ci Enables all tests for a pull request label Nov 1, 2024
@sergepetrenko
Copy link
Collaborator Author

Found a memory leak:

[041] server1 | =================================================================
[041] server1 | ==77643==ERROR: LeakSanitizer: detected memory leaks
[041] server1 | 
[041] server1 | Direct leak of 60 byte(s) in 1 object(s) allocated from:
[041] server1 |     #0 0x556b8f89958f in malloc (/__w/tarantool/tarantool/src/tarantool+0x33858f) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] server1 |     #1 0x556b8f8121ac in strndup (/__w/tarantool/tarantool/src/tarantool+0x2b11ac) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] server1 |     #2 0x556b904f7eca in uri_create /__w/tarantool/tarantool/src/lib/uri/uri.c:266:17
[041] server1 |     #3 0x556b8fb5ce57 in box_check_uri(uri*, char const*, bool) /__w/tarantool/tarantool/src/box/box.cc:1242:6
[041] server1 |     #4 0x556b8fb5ce57 in box_check_bootstrap_leader(uri*, tt_uuid*, char*) /__w/tarantool/tarantool/src/box/box.cc:1601:6
[041] server1 |     #5 0x556b8fb7a04d in box_set_bootstrap_leader() /__w/tarantool/tarantool/src/box/box.cc:2166:9
[041] server1 |     #6 0x556b8fb7a04d in box_cfg_xc() /__w/tarantool/tarantool/src/box/box.cc:5689:6
[041] server1 |     #7 0x556b8fb794eb in box_cfg /__w/tarantool/tarantool/src/box/box.cc:5885:3
[041] server1 |     #8 0x556b8f8dc7df in load_cfg /__w/tarantool/tarantool/src/main.cc:507:2
[041] server1 |     #9 0x556b8fd569fe in lbox_cfg_load(lua_State*) /__w/tarantool/tarantool/src/box/lua/cfg.cc:61:3
[041] server1 |     #10 0x556b8fef5c42 in lj_BC_FUNCC /__w/tarantool/tarantool/third_party/luajit/src/lj_vm.S:811
[041] server1 | 
[041] server1 | Direct leak of 6 byte(s) in 1 object(s) allocated from:
[041] server1 |     #0 0x556b8f89958f in malloc (/__w/tarantool/tarantool/src/tarantool+0x33858f) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] server1 |     #1 0x556b8f8121ac in strndup (/__w/tarantool/tarantool/src/tarantool+0x2b11ac) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] server1 |     #2 0x556b904f7e7e in uri_create /__w/tarantool/tarantool/src/lib/uri/uri.c:265:14
[041] server1 |     #3 0x556b8fb5ce57 in box_check_uri(uri*, char const*, bool) /__w/tarantool/tarantool/src/box/box.cc:1242:6
[041] server1 |     #4 0x556b8fb5ce57 in box_check_bootstrap_leader(uri*, tt_uuid*, char*) /__w/tarantool/tarantool/src/box/box.cc:1601:6
[041] server1 |     #5 0x556b8fd56aad in lbox_cfg_set_bootstrap_leader(lua_State*) /__w/tarantool/tarantool/src/box/lua/cfg.cc:87:6
[041] server1 |     #6 0x556b8fef5c42 in lj_BC_FUNCC /__w/tarantool/tarantool/third_party/luajit/src/lj_vm.S:811
[041] server1 | 
[041] server1 | Direct leak of 6 byte(s) in 1 object(s) allocated from:
[041] server1 |     #0 0x556b8f89958f in malloc (/__w/tarantool/tarantool/src/tarantool+0x33858f) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] server1 |     #1 0x556b8f8121ac in strndup (/__w/tarantool/tarantool/src/tarantool+0x2b11ac) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] server1 |     #2 0x556b904f7e7e in uri_create /__w/tarantool/tarantool/src/lib/uri/uri.c:265:14
[041] server1 |     #3 0x556b8fb5ce57 in box_check_uri(uri*, char const*, bool) /__w/tarantool/tarantool/src/box/box.cc:1242:6
[041] server1 |     #4 0x556b8fb5ce57 in box_check_bootstrap_leader(uri*, tt_uuid*, char*) /__w/tarantool/tarantool/src/box/box.cc:1601:6
[041] server1 |     #5 0x556b8fb7a04d in box_set_bootstrap_leader() /__w/tarantool/tarantool/src/box/box.cc:2166:9
[041] server1 |     #6 0x556b8fb7a04d in box_cfg_xc() /__w/tarantool/tarantool/src/box/box.cc:5689:6
[041] server1 |     #7 0x556b8fb794eb in box_cfg /__w/tarantool/tarantool/src/box/box.cc:5885:3
[041] server1 |     #8 0x556b8f8dc7df in load_cfg /__w/tarantool/tarantool/src/main.cc:507:2
[041] server1 |     #9 0x556b8fd569fe in lbox_cfg_load(lua_State*) /__w/tarantool/tarantool/src/box/lua/cfg.cc:61:3
[041] server1 |     #10 0x556b8fef5c42 in lj_BC_FUNCC /__w/tarantool/tarantool/third_party/luajit/src/lj_vm.S:811
[041] server1 | 
[041] server1 | SUMMARY: AddressSanitizer: 72 byte(s) leaked in 3 allocation(s).
[041] not ok 10	gh-7999-bootstrap-strategy-config.test_dynamic_leader_cfg
[041] #   ...antool/tarantool/test-run/lib/luatest/luatest/server.lua:582: Memory leak during process execution (alias: server1, workdir: server1-xkZCxJxyJl_i, pid: 77643)
[041] #   started logging into a pipe, SIGHUP log rotation disabled
[041] #   
[041] #   =================================================================
[041] #   ==77643==ERROR: LeakSanitizer: detected memory leaks
[041] #   
[041] #   Direct leak of 60 byte(s) in 1 object(s) allocated from:
[041] #       #0 0x556b8f89958f in malloc (/__w/tarantool/tarantool/src/tarantool+0x33858f) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] #       #1 0x556b8f8121ac in strndup (/__w/tarantool/tarantool/src/tarantool+0x2b11ac) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] #       #2 0x556b904f7eca in uri_create /__w/tarantool/tarantool/src/lib/uri/uri.c:266:17
[041] #       #3 0x556b8fb5ce57 in box_check_uri(uri*, char const*, bool) /__w/tarantool/tarantool/src/box/box.cc:1242:6
[041] #       #4 0x556b8fb5ce57 in box_check_bootstrap_leader(uri*, tt_uuid*, char*) /__w/tarantool/tarantool/src/box/box.cc:1601:6
[041] #       #5 0x556b8fb7a04d in box_set_bootstrap_leader() /__w/tarantool/tarantool/src/box/box.cc:2166:9
[041] #       #6 0x556b8fb7a04d in box_cfg_xc() /__w/tarantool/tarantool/src/box/box.cc:5689:6
[041] #       #7 0x556b8fb794eb in box_cfg /__w/tarantool/tarantool/src/box/box.cc:5885:3
[041] #       #8 0x556b8f8dc7df in load_cfg /__w/tarantool/tarantool/src/main.cc:507:2
[041] #       #9 0x556b8fd569fe in lbox_cfg_load(lua_State*) /__w/tarantool/tarantool/src/box/lua/cfg.cc:61:3
[041] #       #10 0x556b8fef5c42 in lj_BC_FUNCC /__w/tarantool/tarantool/third_party/luajit/src/lj_vm.S:811
[041] #   
[041] #   Direct leak of 6 byte(s) in 1 object(s) allocated from:
[041] #       #0 0x556b8f89958f in malloc (/__w/tarantool/tarantool/src/tarantool+0x33858f) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] #       #1 0x556b8f8121ac in strndup (/__w/tarantool/tarantool/src/tarantool+0x2b11ac) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] #       #2 0x556b904f7e7e in uri_create /__w/tarantool/tarantool/src/lib/uri/uri.c:265:14
[041] #       #3 0x556b8fb5ce57 in box_check_uri(uri*, char const*, bool) /__w/tarantool/tarantool/src/box/box.cc:1242:6
[041] #       #4 0x556b8fb5ce57 in box_check_bootstrap_leader(uri*, tt_uuid*, char*) /__w/tarantool/tarantool/src/box/box.cc:1601:6
[041] #       #5 0x556b8fd56aad in lbox_cfg_set_bootstrap_leader(lua_State*) /__w/tarantool/tarantool/src/box/lua/cfg.cc:87:6
[041] #       #6 0x556b8fef5c42 in lj_BC_FUNCC /__w/tarantool/tarantool/third_party/luajit/src/lj_vm.S:811
[041] #   
[041] #   Direct leak of 6 byte(s) in 1 object(s) allocated from:
[041] #       #0 0x556b8f89958f in malloc (/__w/tarantool/tarantool/src/tarantool+0x33858f) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] #       #1 0x556b8f8121ac in strndup (/__w/tarantool/tarantool/src/tarantool+0x2b11ac) (BuildId: 2a52badb58f8a8196558df2bd051941fd194df54)
[041] #       #2 0x556b904f7e7e in uri_create /__w/tarantool/tarantool/src/lib/uri/uri.c:265:14
[041] #       #3 0x556b8fb5ce57 in box_check_uri(uri*, char const*, bool) /__w/tarantool/tarantool/src/box/box.cc:1242:6
[041] #       #4 0x556b8fb5ce57 in box_check_bootstrap_leader(uri*, tt_uuid*, char*) /__w/tarantool/tarantool/src/box/box.cc:1601:6
[041] #       #5 0x556b8fb7a04d in box_set_bootstrap_leader() /__w/tarantool/tarantool/src/box/box.cc:2166:9
[041] #       #6 0x556b8fb7a04d in box_cfg_xc() /__w/tarantool/tarantool/src/box/box.cc:5689:6
[041] #       #7 0x556b8fb794eb in box_cfg /__w/tarantool/tarantool/src/box/box.cc:5885:3
[041] #       #8 0x556b8f8dc7df in load_cfg /__w/tarantool/tarantool/src/main.cc:507:2
[041] #       #9 0x556b8fd569fe in lbox_cfg_load(lua_State*) /__w/tarantool/tarantool/src/box/lua/cfg.cc:61:3
[041] #       #10 0x556b8fef5c42 in lj_BC_FUNCC /__w/tarantool/tarantool/third_party/luajit/src/lj_vm.S:811
[041] #   
[041] #   SUMMARY: AddressSanitizer: 72 byte(s) leaked in 3 allocation(s).
[041] #   
[041] #   stack traceback:
[041] #   	...ool/test/replication-luatest/bootstrap_strategy_test.lua:190: in function <...ool/test/replication-luatest/bootstrap_strategy_test.lua:189>
[041] #   	...
[041] #   	[C]: in function 'xpcall'
[041] #   artifacts:
[041] #   	server1 -> /tmp/t/041_replication-luatest/artifacts/rs-t-MwKaKScjtj/server1-xkZCxJxyJl_i
[041] # Starting group: gh-7999-bootstrap-strategy-config-fail

@sergepetrenko sergepetrenko removed the full-ci Enables all tests for a pull request label Nov 1, 2024
@sergepetrenko
Copy link
Collaborator Author

Changes since the previous version:

diff
diff --git a/src/box/box.cc b/src/box/box.cc
index 2b25f14502..d53dc7ac3c 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -1579,6 +1579,8 @@ static int
 box_check_bootstrap_leader(struct uri *uri, struct tt_uuid *uuid, char *name)
 {
        *uuid = uuid_nil;
+       if (!uri_is_nil(uri))
+               uri_destroy(uri);
        uri_create(uri, NULL);
        *name = '\0';
        const char *source = cfg_gets("bootstrap_leader");
@@ -1977,6 +1979,7 @@ box_check_config(void)
        box_check_replication_sync_timeout();
        if (box_check_bootstrap_strategy() == BOOTSTRAP_STRATEGY_INVALID)
                diag_raise();
+       uri_create(&uri, NULL);
        if (box_check_bootstrap_leader(&uri, &uuid, name) != 0)
                diag_raise();
        uri_destroy(&uri);
diff --git a/src/box/replication.cc b/src/box/replication.cc
index 2b228a8098..ac4d8cbe3e 100644
--- a/src/box/replication.cc
+++ b/src/box/replication.cc
@@ -265,6 +265,9 @@ replication_free(void)
        latch_destroy(&replicaset.applier.order_latch);
        applier_free();
 
+       if (!uri_is_nil(&cfg_bootstrap_leader_uri))
+               uri_destroy(&cfg_bootstrap_leader_uri);
+
        TRASH(&replicaset);
 }
 

bootstrap_leader configuration option only takes effect during instance
bootstrap. For this reason, the option was made static. One can't change
the bootstrap_leader after initial box.cfg() call.

So when a user has bootstrap_strategy = 'config' and joins new nodes
during replicaset lifespan, he ends up with instances having different
`bootstrap_leader` configuration, depending on which node was writable
at the moment of each instance bootstrap.

It would be better to have matching configuration on each replica
set member with no exceptions and to be able to set bootstrap_leader to
the same value on all the instances of the replica set, even on the ones
which are already bootstrapped.

Let's allow changing `bootstrap_leader` option even though it affects
nothing.

While we're at it, fix a memory leak of bootstrap leader uri on
tarantool shutdown.

Closes tarantool#10604

@TarantoolBot document
Title: clarify `bootstrap_leader` configuration option meaning

The option `bootstrap_leader` is dynamic, but it's important to note
that changing it after instance bootstrap affects nothing. It's left
dynamic simply for the sake of centralized configuration, when it's
preferable to have matching configurations on all instances of a replica
set.
@sergepetrenko sergepetrenko force-pushed the gh-10604-bootstrap-leader-dynamic branch from b429ae7 to b5779ea Compare November 6, 2024 12:53
@sergepetrenko sergepetrenko added the asan-ci Enables asan build tests for a pull request label Nov 6, 2024
@sergepetrenko sergepetrenko removed their assignment Nov 6, 2024
@sergepetrenko
Copy link
Collaborator Author

@Serpentian, PTAL. I've fixed the memory leak.

@sergepetrenko sergepetrenko added full-ci Enables all tests for a pull request and removed asan-ci Enables asan build tests for a pull request labels Nov 11, 2024
@sergepetrenko sergepetrenko merged commit 97ec171 into tarantool:master Nov 12, 2024
@sergepetrenko sergepetrenko deleted the gh-10604-bootstrap-leader-dynamic branch March 17, 2025 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Enables all tests for a pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make box.cfg.bootstrap_leader option dynamic

5 participants