cherry-pick #24359 to release-4.0
You can switch your code base to this Pull Request by using git-extras:

# In tidb repo:
git pr https://github.com/pingcap/tidb/pull/26030

After apply modifications, you can push your change to this PR via:

git push git@github.com:ti-srebot/tidb.git pr/26030:release-4.0-0f10bef470f4

What problem does this PR solve?

Issue Number: close #24326

Problem Summary:

The existing global vars cache only caches for 2 seconds, and does not perform notification to other servers when cache is invalid. This PR changes the design to be basically the same as the privileges system cache.

It fixes two immediate bugs:

• Read after write consistency from setting global vars (on the initiating instance only)
• Running SHOW VARIABLES for the first time in a session previously took 1 second in some cases(!). It should now read the values from memory.

What is changed and how it works?

What's Changed:

• The global vars cache is replaced with a new cache.
• All operations check the cache first rather than read from mysql.global_variables.
• Updating a global variable sends a notification to other servers via etcd.
• Updating a global variable is read-after-write consistent on a single server.

This opens the door, but does not fix some remaining issues - the session cache should be populated with a copy of all session vars when the session starts. Currently there is a lazy loading mechanism which is not MySQL compatible. It also caches some global variables in the session systems[] which is incorrect. In a followup PR I hope to remove the array builtinGlobalVariable and simplify loadCommonGlobalVariablesIfNeeded to just copy session vars.

Related changes

• Docs PR Add docs for sysvar cache docs#5563

Check List

Tests

• Unit test
• Manual test (add detailed scripts or steps below)

I manually verified that behavior is correct with multiple servers receiving notice from etcd.

Side effects

• There is an small upgrade issue with this PR: because the cache now lives for a long time, it relies on a notification from etcd that the cache is stale and needs to be refreshed. If the cluster is running with mixed-versions, and the SET GLOBAL statement is run on the older version, it will not send an etcd notification to newer servers that their cache is stale. I think this behavior is acceptable, since the cache will be refreshed within a few minutes, but it should be made clear in the release notes.

Release note

• The system variables cache has been replaced to follow a similar design to privileges cache, where all TiDB servers are immediately notified of changes to variables. This lifts a previous limitation where changes to global variables may take 2 seconds to take effect. To take advantage of this feature, all TiDB servers need to be upgraded so that they are able to both publish and subscribe to system variable changes. Operating with mixed versions might result in an issue where system variable changes take longer (30 seconds) to propagate to the cluster.