tracing: enabling trace redaction on 21.2.3 does not work on multi-tenant deployments #73938
Description
Describe the problem
Enabling the trace.redactable.enabled to true does not take effect as expected in a multi-tenant deployment.
To Reproduce
Set up secure tenant and kv certs and servers
mkdir certs
./cockroach cert create-ca --certs-dir=./certs/ --ca-key=./certs/ca.key
./cockroach cert create-node 127.0.0.1 ::1 localhost "*.local" --certs-dir=./certs/ --ca-key=./certs/ca.key
./cockroach cert create-client root --certs-dir=./certs --ca-key=./certs/ca.key
./cockroach mt cert create-tenant-client 123 --certs-dir=./certs --ca-key=./certs/ca.key
Start kv server:
./cockroach start-single-node --certs-dir=./certs --store=redact --listen-addr=127.0.0.1:26257
Connect to kv server (to set cluster setting):
./cockroach sql --certs-dir=./certs --host=localhost:26257
Create tenant on KV:
SELECT crdb_internal.create_tenant(123);
Start tenant:
./cockroach mt start-sql --sql-addr=localhost:26259 --http-addr=:8082 --tenant-id=123 --certs-dir=./cert
Connect to tenant (to run queries and view traces):
./cockroach sql --certs-dir=./certs --host=localhost:26259
Demonstrate trace redactability
Commands run on tenant after initial CREATE TABLE test (id INT);:
root@localhost:26259/defaultdb> set tracing = on;
root@localhost:26259/defaultdb> insert into test values (1);
root@localhost:26259/defaultdb> show trace for session;
With set cluster setting trace.redactable.enabled = false on KV:
Relevant output subset on tenant:
dist sender send | 22
2021-12-16 17:36:28.751254+00 | 00:00:02.72649 | sending request to 127.0.0.1:26257 | [sql,client=127.0.0.1:51396,hostssl,user=root,txn=2c033d9d] | rpc/nodedialer/nodedialer.go:136 | dist sender send | 22
2021-12-16 17:36:28.751262+00 | 00:00:02.726498 | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 23
2021-12-16 17:36:28.751551+00 | 00:00:02.726787 | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751605+00 | 00:00:02.726841 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751678+00 | 00:00:02.726914 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751711+00 | 00:00:02.726947 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751736+00 | 00:00:02.726972 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751746+00 | 00:00:02.726982 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751766+00 | 00:00:02.727002 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751789+00 | 00:00:02.727025 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751801+00 | 00:00:02.727037 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751814+00 | 00:00:02.72705 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751973+00 | 00:00:02.727209 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.751997+00 | 00:00:02.727233 | × | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:36:28.773951+00 | 00:00:02.749187 | fast path completed | [sql,client=127.0.0.1:51396,hostssl,user=root] | sql/plan_node_to_row_source.go:159 |
This output does not change when the cluster setting above is set to true.
Expected behavior
With set cluster setting trace.redactable.enabled = true on KV:
Relevant output subset on tenant:
dist sender send | 22
2021-12-16 17:39:05.240753+00 | 00:00:03.785012 | sending request to 127.0.0.1:26257 | [sql,client=127.0.0.1:51455,hostssl,user=root,txn=d3392f40] | rpc/nodedialer/nodedialer.go:136 | dist sender send | 22
2021-12-16 17:39:05.240759+00 | 00:00:03.785018 | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 23
2021-12-16 17:39:05.241011+00 | 00:00:03.78527 | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.241041+00 | 00:00:03.7853 | node received request: × | [n=1] | server/node.go:907 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.241082+00 | 00:00:03.785341 | executing ConditionalPut [×,×), EndTxn(parallel commit) [×], [txn: d3392f40], [can-forward-ts] | [n=1,s=1] | kv/kvserver/store_send.go:157 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.241103+00 | 00:00:03.785362 | read-write path | [n=1,s=1,r=44/1:×] | kv/kvserver/replica_send.go:105 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.241117+00 | 00:00:03.785376 | sequencing request | [n=1,s=1,r=44/1:×] | kv/kvserver/concurrency/concurrency_manager.go:187 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.241125+00 | 00:00:03.785384 | acquiring latches | [n=1,s=1,r=44/1:×] | kv/kvserver/concurrency/concurrency_manager.go:259 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.241136+00 | 00:00:03.785395 | scanning lock table for conflicting locks | [n=1,s=1,r=44/1:×] | kv/kvserver/concurrency/concurrency_manager.go:300 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.241152+00 | 00:00:03.785411 | applied timestamp cache | [n=1,s=1,r=44/1:×] | kv/kvserver/replica_write.go:144 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.241172+00 | 00:00:03.785431 | executing read-write batch | [n=1,s=1,r=44/1:×] | kv/kvserver/replica_write.go:436 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.241193+00 | 00:00:03.785452 | attempting 1PC execution | [n=1,s=1,r=44/1:×] | kv/kvserver/replica_write.go:505 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.24132+00 | 00:00:03.785579 | evaluated ConditionalPut command ×, txn=<nil> : resp=×, err=<nil> | [n=1,s=1,r=44/1:×] | kv/kvserver/replica_evaluate.go:521 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.24134+00 | 00:00:03.785599 | evaluated request | [n=1,s=1,r=44/1:×] | kv/kvserver/replica_raft.go:90 | /cockroach.roachpb.Internal/Batch | 24
2021-12-16 17:39:05.282375+00 | 00:00:03.826634 | fast path completed | [sql,client=127.0.0.1:51455,hostssl,user=root] | sql/plan_node_to_row_source.go:159 |
Environment:
- CockroachDB version: 21.2.3
- Client app: Cockroach SQL