Overview of the Issue
We encountered a panic in vtgate while running concurrent schema changes in different keyspaces. This is caused by a race condition in the vtgate schema tracker which leads to fatal error: concurrent map iteration and map write followed by a panic and process exit. See log below for details.
Reproduction Steps
- Enable schema tracking on vtgate:
--schema_change_signal
- Enable schema change signaling and schema tracking on vttablet
--queryserver-config-schema-change-signal, --watch_replication_stream, --track_schema_versions (not sure if the last two are necessary, but it's how we run in production and perhaps it results in the schema changes being sent to vtgate upon completion vs periodically)
- Run schema changes concurrently on multiple keyspaces
Binary Version
This occurred in production using an internally patched version of v15.0.3 but I've been able to replicate it locally on main (f2c1e01)
Operating System and Environment details
Operating system (output of cat /etc/os-release)
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Kernel version (output of uname -sr)
$ uname -sr
Linux 5.15.0-78-generic
Architecture (output of uname -m)
Log Fragments
I0725 13:58:35.703272 1 vschema_manager.go:177] Sent vschema to subscriber
I0725 13:58:35.710735 1 vschema_manager.go:177] Sent vschema to subscriber
I0725 13:58:40.692117 1 vschema_manager.go:164] Received schema update
I0725 13:58:40.694105 1 vschema_manager.go:164] Received schema update
I0725 13:58:40.699426 1 vschema_manager.go:177] Sent vschema to subscriber
I0725 13:58:40.700444 1 vschema_manager.go:164] Received schema update
I0725 13:58:40.701199 1 vschema_manager.go:177] Sent vschema to subscriber
I0725 13:58:40.703746 1 vschema_manager.go:164] Received schema update
I0725 13:58:40.708684 1 vschema_manager.go:177] Sent vschema to subscriber
I0725 13:58:40.709758 1 vschema_manager.go:164] Received schema update
I0725 13:58:40.711174 1 vschema_manager.go:177] Sent vschema to subscriber
I0725 13:58:40.715259 1 vschema_manager.go:164] Received schema update
I0725 13:58:40.716919 1 vschema_manager.go:177] Sent vschema to subscriber
fatal error: concurrent map iteration and map write
I0725 13:58:40.719495 1 vschema_manager.go:164] Received schema update
goroutine 33027864 [running]:
runtime.throw({0x1f07b40?, 0xc00d558c00?})
runtime/panic.go:992 +0x71 fp=0xc00342fd38 sp=0xc00342fd08 pc=0x436291
runtime.mapiternext(0x1bfdac0?)
runtime/map.go:871 +0x4eb fp=0xc00342fda8 sp=0xc00342fd38 pc=0x40fe6b
vitess.io/vitess/go/vt/vtgate.(*VSchemaManager).updateFromSchema(0xc0055458b0, 0x0?)
vitess.io/vitess/go/vt/vtgate/vschema_manager.go:194 +0x109 fp=0xc00342ff00 sp=0xc00342fda8 pc=0xfd2409
vitess.io/vitess/go/vt/vtgate.(*VSchemaManager).buildAndEnhanceVSchema(0xc0055458b0, 0xc010e64f90?)
vitess.io/vitess/go/vt/vtgate/vschema_manager.go:185 +0x45 fp=0xc00342ff28 sp=0xc00342ff00 pc=0xfd22c5
vitess.io/vitess/go/vt/vtgate.(*VSchemaManager).Rebuild(0xc0055458b0)
vitess.io/vitess/go/vt/vtgate/vschema_manager.go:170 +0xa6 fp=0xc00342ff70 sp=0xc00342ff28 pc=0xfd2166
vitess.io/vitess/go/vt/vtgate.(*VSchemaManager).Rebuild-fm()
<autogenerated>:1 +0x26 fp=0xc00342ff88 sp=0xc00342ff70 pc=0xfe3666
vitess.io/vitess/go/vt/vtgate/schema.(*updateController).consume(0xc00e738b80)
vitess.io/vitess/go/vt/vtgate/schema/update_controller.go:79 +0x144 fp=0xc00342ffc8 sp=0xc00342ff88 pc=0xf9f424
vitess.io/vitess/go/vt/vtgate/schema.(*updateController).add.func2()
vitess.io/vitess/go/vt/vtgate/schema/update_controller.go:154 +0x26 fp=0xc00342ffe0 sp=0xc00342ffc8 pc=0xf9faa6
runtime.goexit()
runtime/asm_amd64.s:1571 +0x1 fp=0xc00342ffe8 sp=0xc00342ffe0 pc=0x469301
created by vitess.io/vitess/go/vt/vtgate/schema.(*updateController).add
vitess.io/vitess/go/vt/vtgate/schema/update_controller.go:154 +0x16f
Overview of the Issue
We encountered a panic in vtgate while running concurrent schema changes in different keyspaces. This is caused by a race condition in the vtgate schema tracker which leads to
fatal error: concurrent map iteration and map writefollowed by a panic and process exit. See log below for details.Reproduction Steps
--schema_change_signal--queryserver-config-schema-change-signal,--watch_replication_stream,--track_schema_versions(not sure if the last two are necessary, but it's how we run in production and perhaps it results in the schema changes being sent to vtgate upon completion vs periodically)Binary Version
This occurred in production using an internally patched version of v15.0.3 but I've been able to replicate it locally on main (f2c1e01)
Operating System and Environment details
Operating system (output of cat /etc/os-release)
Kernel version (output of uname -sr)
Architecture (output of uname -m)
Log Fragments
I0725 13:58:35.703272 1 vschema_manager.go:177] Sent vschema to subscriber I0725 13:58:35.710735 1 vschema_manager.go:177] Sent vschema to subscriber I0725 13:58:40.692117 1 vschema_manager.go:164] Received schema update I0725 13:58:40.694105 1 vschema_manager.go:164] Received schema update I0725 13:58:40.699426 1 vschema_manager.go:177] Sent vschema to subscriber I0725 13:58:40.700444 1 vschema_manager.go:164] Received schema update I0725 13:58:40.701199 1 vschema_manager.go:177] Sent vschema to subscriber I0725 13:58:40.703746 1 vschema_manager.go:164] Received schema update I0725 13:58:40.708684 1 vschema_manager.go:177] Sent vschema to subscriber I0725 13:58:40.709758 1 vschema_manager.go:164] Received schema update I0725 13:58:40.711174 1 vschema_manager.go:177] Sent vschema to subscriber I0725 13:58:40.715259 1 vschema_manager.go:164] Received schema update I0725 13:58:40.716919 1 vschema_manager.go:177] Sent vschema to subscriber fatal error: concurrent map iteration and map write I0725 13:58:40.719495 1 vschema_manager.go:164] Received schema update goroutine 33027864 [running]: runtime.throw({0x1f07b40?, 0xc00d558c00?}) runtime/panic.go:992 +0x71 fp=0xc00342fd38 sp=0xc00342fd08 pc=0x436291 runtime.mapiternext(0x1bfdac0?) runtime/map.go:871 +0x4eb fp=0xc00342fda8 sp=0xc00342fd38 pc=0x40fe6b vitess.io/vitess/go/vt/vtgate.(*VSchemaManager).updateFromSchema(0xc0055458b0, 0x0?) vitess.io/vitess/go/vt/vtgate/vschema_manager.go:194 +0x109 fp=0xc00342ff00 sp=0xc00342fda8 pc=0xfd2409 vitess.io/vitess/go/vt/vtgate.(*VSchemaManager).buildAndEnhanceVSchema(0xc0055458b0, 0xc010e64f90?) vitess.io/vitess/go/vt/vtgate/vschema_manager.go:185 +0x45 fp=0xc00342ff28 sp=0xc00342ff00 pc=0xfd22c5 vitess.io/vitess/go/vt/vtgate.(*VSchemaManager).Rebuild(0xc0055458b0) vitess.io/vitess/go/vt/vtgate/vschema_manager.go:170 +0xa6 fp=0xc00342ff70 sp=0xc00342ff28 pc=0xfd2166 vitess.io/vitess/go/vt/vtgate.(*VSchemaManager).Rebuild-fm() <autogenerated>:1 +0x26 fp=0xc00342ff88 sp=0xc00342ff70 pc=0xfe3666 vitess.io/vitess/go/vt/vtgate/schema.(*updateController).consume(0xc00e738b80) vitess.io/vitess/go/vt/vtgate/schema/update_controller.go:79 +0x144 fp=0xc00342ffc8 sp=0xc00342ff88 pc=0xf9f424 vitess.io/vitess/go/vt/vtgate/schema.(*updateController).add.func2() vitess.io/vitess/go/vt/vtgate/schema/update_controller.go:154 +0x26 fp=0xc00342ffe0 sp=0xc00342ffc8 pc=0xf9faa6 runtime.goexit() runtime/asm_amd64.s:1571 +0x1 fp=0xc00342ffe8 sp=0xc00342ffe0 pc=0x469301 created by vitess.io/vitess/go/vt/vtgate/schema.(*updateController).add vitess.io/vitess/go/vt/vtgate/schema/update_controller.go:154 +0x16f