roachtest: Add tpcc1k test to identify long running schema change txns.#39096
Conversation
|
This change is |
|
@dt I am still running the test with tpcc1k and will post the time taken for each schema change once it completes. Just wanted you to have a look if this fulfills what core requires, thanks. |
3763bc5 to
5f137f5
Compare
dt
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @adityamaru27, @ajwerner, and @dt)
pkg/cmd/roachtest/schemachange.go, line 435 at r1 (raw file):
}) }, MinVersion: "v2.2.0",
I'd go ahead and make this 19.1 -- I think we know some of these take a very long time on 2.2, and I'm not actually sure they all work at all.
ajwerner
left a comment
There was a problem hiding this comment.
Nice!
Are there things we want to assert? Maybe that all of the statements finished at least once? Perhaps about how long things take? Can follow up after we collect some data.
Reviewable status:
pkg/cmd/roachtest/schemachange.go, line 435 at r1 (raw file):
Previously, dt (David Taylor) wrote…
I'd go ahead and make this 19.1 -- I think we know some of these take a very long time on 2.2, and I'm not actually sure they all work at all.
👍
5f137f5 to
8b14371
Compare
|
@ajwerner |
|
Following are timings of the schema change statements: • Importing fixtures + queries took a total of 3h12m5.065694801s. |
|
All these look OK to me (they're longer than 45s) except for |
|
Yeah, |
|
Thanks! Yeah, let's make a second constraint with |
8b14371 to
a5aa7e4
Compare
|
Will update the time for the new Time taken for tpcc.district VALIDATE : 7m27s |
bed498f to
c9de72d
Compare
This change adds a roachtest which spins up a 5 node, 16 cpu cluster with a tpcc-1k workload, and runs the schema changes outlined in cockroachdb#37083. The need for this test arises to have a single source of truth for long running schema change txns, and the impact of lowering the closed ts interval. Release note: None
c9de72d to
80cd61a
Compare
|
@dt can i merge this? We can collect some data and make tweaks if needed over the next few days. |
|
oh, yeah, sorry, I thought I LGTM'ed it awhile ago. |
|
TFTR |
39096: roachtest: Add tpcc1k test to identify long running schema change txns. r=adityamaru27 a=adityamaru27 This change adds a roachtest which spins up a 5 node, 16 cpu cluster with a tpcc-1k workload, and runs the schema changes outlined in #37083. The need for this test arises to have a single source of truth for long running schema change txns, and the impact of lowering the closed ts interval. Co-authored-by: Aditya Maru <adityamaru@cockroachlabs.com>
Build succeeded |
This PR dramatically shortens the closed timestamp target interval. With this setting the experimental_follower_read_timestamp() will now be 3.7 seconds in the past as opposed to the current 48 seconds in the past. This relatively aggressive setting of 5s is intended to shake out issues. This value has been verified to work on the `schemachange/mixed/tpcc` roachtest introduced in cockroachdb#39096 and for vanilla TPC-C runs around where we have established a baseline in the past using tpccbench. Specifically several runs at 2300 warehouses on 3x c5d.4xlarge nodes which is right at the passing boundary without the change were run only a very small difference in efficiency or tail latency was observed. It seems reasonable to attempt to live with this value on master for a while and see what happens. Fixes cockroachdb#37083. Release note: None
39643: closedts: shorten target_duration from 30s to 5s r=ajwerner a=ajwerner This PR dramatically shortens the closed timestamp target interval. With this setting the experimental_follower_read_timestamp() will now be 8.5 seconds in the past as opposed to the current 48 seconds in the past. This relatively aggressive setting of 5s is intended to shake out issues as we head into the stability period. This value has been verified to work on the `schemachange/mixed/tpcc` roachtest introduced in #39096 and for vanilla TPC-C runs around where we have established a baseline in the past using tpccbench. Specifically several runs at 2300 warehouses on 3x c5d.4xlarge nodes which is right at the passing boundary without the change were run only a very small difference in efficiency or tail latency was observed. It seems reasonable to attempt to live with this value on master for a while and see what happens. Fixes #37083. Release note: None Co-authored-by: Andrew Werner <ajwerner@cockroachlabs.com>
This PR dramatically shortens the closed timestamp target interval. With this setting the experimental_follower_read_timestamp() will now be 4.8 seconds in the past as opposed to the current 48 seconds in the past. This relatively aggressive setting of 3s is intended to shake out issues. This value has been verified to work on the `schemachange/mixed/tpcc` roachtest introduced in cockroachdb#39096 and for vanilla TPC-C runs around where we have established a baseline in the past using tpccbench. Specifically several runs at 2300 warehouses on 3x c5d.4xlarge nodes which is right at the passing boundary without the change were run only a very small difference in efficiency or tail latency was observed. It seems reasonable to attempt to live with this value on master for a while and see what happens. Fixes cockroachdb#37083. Release note (enterprise change): Shorten the default closed timestamp target interval from 30s to 3s leading to permit follower reads at 4.8 seconds in the past rather than the previous 48 seconds.
This PR dramatically shortens the closed timestamp target interval. With this setting the experimental_follower_read_timestamp() will now be 4.8 seconds in the past as opposed to the current 48 seconds in the past. This relatively aggressive setting of 3s is intended to shake out issues. This value has been verified to work on the `schemachange/mixed/tpcc` roachtest introduced in cockroachdb#39096 and for vanilla TPC-C runs around where we have established a baseline in the past using tpccbench. Specifically several runs at 2300 warehouses on 3x c5d.4xlarge nodes which is right at the passing boundary without the change were run only a very small difference in efficiency or tail latency was observed. It seems reasonable to attempt to live with this value on master for a while and see what happens. Fixes cockroachdb#37083. Release note (enterprise change): Shorten the default closed timestamp target interval from 30s to 3s leading to permit follower reads at 4.8 seconds in the past rather than the previous 48 seconds.
This PR dramatically shortens the closed timestamp target interval. With this setting the experimental_follower_read_timestamp() will now be 4.8 seconds in the past as opposed to the current 48 seconds in the past. This relatively aggressive setting of 3s is intended to shake out issues. This value has been verified to work on the `schemachange/mixed/tpcc` roachtest introduced in cockroachdb#39096 and for vanilla TPC-C runs around where we have established a baseline in the past using tpccbench. Specifically several runs at 2300 warehouses on 3x c5d.4xlarge nodes which is right at the passing boundary without the change were run only a very small difference in efficiency or tail latency was observed. It seems reasonable to attempt to live with this value on master for a while and see what happens. Fixes cockroachdb#37083. Release note (enterprise change): Shorten the default closed timestamp target interval from 30s to 3s leading to permit follower reads at 4.8 seconds in the past rather than the previous 48 seconds.
43147: closedts: shorten target_duration from 30s to 3s r=ajwerner a=ajwerner This PR dramatically shortens the closed timestamp target interval. With this setting the experimental_follower_read_timestamp() will now be 4.8 seconds in the past as opposed to the current 48 seconds in the past. This relatively aggressive setting of 3s is intended to shake out issues. This value has been verified to work on the `schemachange/mixed/tpcc` roachtest introduced in #39096 and for vanilla TPC-C runs around where we have established a baseline in the past using tpccbench. Specifically several runs at 2300 warehouses on 3x c5d.4xlarge nodes which is right at the passing boundary without the change were run only a very small difference in efficiency or tail latency was observed. It seems reasonable to attempt to live with this value on master for a while and see what happens. Fixes #37083. Release note (enterprise change): Shorten the default closed timestamp target interval from 30s to 3s leading to permit follower reads at 4.8 seconds in the past rather than the previous 48 seconds. 43226: sql: fix namespace name resolution fallback code r=arulajmani a=arulajmani Previously, the fallback for name resolution was broken. If an entry is not present in the new `system.namespace`, we should look it up in the old `system.namespace`. This was not happening, as we would only fallback to the old `system.namespace` if the entry was a table. Thus, databases created before the new `system.namespace` table was created were not being resolved at all. This PR fixes that. Fixes #43141 Release note: None Co-authored-by: Andrew Werner <ajwerner@cockroachlabs.com> Co-authored-by: arulajmani <arulajmani@gmail.com>
This change adds a roachtest which spins up a 5 node, 16 cpu cluster
with a tpcc-1k workload, and runs the schema changes outlined in #37083.
The need for this test arises to have a single source of truth for long
running schema change txns, and the impact of lowering the closed ts
interval.