multitenant: demo startup performance is poor with multiple tenants and simulated latencies #76305
Description
cockroach demo --global --nodes 9 --empty takes several minutes to startup. When multi-tenancy is disabled, it takes only a few seconds. It's not clear what's causing this poor performance, but a quick peek into the logs shows that each tenant takes between 20 and 60 seconds to start.
I220209 13:16:17.066637 1 cli/democluster/demo_cluster.go:389 ⋮ [start-demo-cluster,phase=8] 2880 starting tenant nodes
I220209 13:16:17.066657 1 cli/democluster/demo_cluster.go:394 ⋮ [start-demo-cluster,phase=8] 2881 starting tenant node 0
I220209 13:16:38.293483 1 cli/democluster/demo_cluster.go:394 ⋮ [start-demo-cluster,phase=8,nsql1] 6010 starting tenant node 1
I220209 13:16:57.371877 1 cli/democluster/demo_cluster.go:394 ⋮ [start-demo-cluster,phase=8,nsql1] 8435 starting tenant node 2
I220209 13:17:15.858256 1 cli/democluster/demo_cluster.go:394 ⋮ [start-demo-cluster,phase=8,nsql1] 10585 starting tenant node 3
I220209 13:18:01.189788 1 cli/democluster/demo_cluster.go:394 ⋮ [start-demo-cluster,phase=8,nsql1] 13284 starting tenant node 4
I220209 13:18:52.356343 1 cli/democluster/demo_cluster.go:394 ⋮ [start-demo-cluster,phase=8,nsql1] 15015 starting tenant node 5
I220209 13:19:31.760119 1 cli/democluster/demo_cluster.go:394 ⋮ [start-demo-cluster,phase=8,nsql1] 17341 starting tenant node 6
I220209 13:20:20.155604 1 cli/democluster/demo_cluster.go:394 ⋮ [start-demo-cluster,phase=8,nsql1] 20181 starting tenant node 7
I220209 13:21:19.373402 1 cli/democluster/demo_cluster.go:394 ⋮ [start-demo-cluster,phase=8,nsql1] 23542 starting tenant node 8
More investigation is required.
Epic: CRDB-18596
Jira issue: CRDB-13070