Hi everyone. I’ve noticed that online schema changes got slower in 20.1 version compared to 19.2;
I know that it popped up before and there are #48621 (included in 20.1.1 release) and #50923 (included in 20.1.4 release) that supposed to fix this, but the problem still exists even for 20.1.4 version.

Setup for the benchmark is the following:
I am running 400 concurrent Go tests against my local single node cluster, each test does:

1. Connects to the cockroachdb server
2. Creates a fresh new database
3. Executes SQL with a single transaction that contains all schema changes.

Here is the SQL with the schema:

BEGIN;
SAVEPOINT cockroach_restart;
CREATE TABLE users (
    id UUID PRIMARY KEY,
    name STRING NOT NULL
);
CREATE TABLE user_posts (
    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    post_id UUID NOT NULL,
    PRIMARY KEY (user_id, post_id)
);
RELEASE SAVEPOINT cockroach_restart;
COMMIT;

This takes 15.3 seconds on cockroach 20.1.4 and 5.9s on cockroach 19.2.9
If I increase the number of tests it scales not linearly.
I investigated and found the SQL statement that causes the problem, it was REFERENCES. If I remove it from users_posts table definition, the times are the same for 20.1.4 and 19.2.9;
Apart from REFERENCES statement, I discovered that a separate CREATE INDEX statement causes the same problem:

BEGIN;
SAVEPOINT cockroach_restart;
CREATE TABLE users (
    id UUID PRIMARY KEY,
    name STRING NOT NULL
);
CREATE TABLE user_posts (
    user_id UUID NOT NULL,
    post_id UUID NOT NULL,
    PRIMARY KEY (user_id, post_id)
);
CREATE INDEX ON user_posts (user_id);
RELEASE SAVEPOINT cockroach_restart;
COMMIT;

This takes 14.5 seconds on cockroach 20.1.4 and 5.2 seconds on 19.2.9;
But if replace a separate CREATE INDEX with INDEX inside the table definition, the problem is gone, times are the same for both versions.
Also, If I create an interleaved index, instead of the regular one, like that:

BEGIN;
SAVEPOINT cockroach_restart;
CREATE TABLE users (
    id UUID PRIMARY KEY,
    name STRING NOT NULL
);
CREATE TABLE user_posts (
    user_id UUID NOT NULL,
    post_id UUID NOT NULL,
    PRIMARY KEY (user_id, post_id)
);
CREATE INDEX ON user_posts (user_id) INTERLEAVE IN PARENT users (user_id);
RELEASE SAVEPOINT cockroach_restart;
COMMIT;

The difference is even bigger, it takes 24.5 seconds on 20.1.4 and 6.2 seconds on 19.2.9

Jira issue: CRDB-3938