Set socket_timeout for memcached connections to 1s#1429
Merged
vuntz merged 2 commits intocrowbar:masterfrom Nov 16, 2017
Merged
Set socket_timeout for memcached connections to 1s#1429vuntz merged 2 commits intocrowbar:masterfrom
vuntz merged 2 commits intocrowbar:masterfrom
Conversation
The default is 3 seconds, which is very high. When one memcached server goes down, then every connection pool for memcached (in each process, on each node) need to identify that this server is down, and that has some dramatic impact on performance. Lowering this to 1 second, while still being a reasonable value, helps quite a bit. I would even consider lowering this to 0.25, but oslo.cache doesn't support float values there (see https://bugs.launchpad.net/oslo.cache/+bug/1731921). Helps with https://bugzilla.suse.com/show_bug.cgi?id=1038223
The keystone_authtoken section has some configuration for memcache, and the default for memcache_pool_socket_timeout is 3 seconds. When one memcached server goes down, then every connection pool for memcached (in each process, on each node) need to identify that this server is down, and that has some dramatic impact on performance. Lowering this to 1 second, while still being a reasonable value, helps quite a bit. I would even consider lowering this to 0.25, but keystonemiddleware doesn't support float values there (see https://bugs.launchpad.net/oslo.cache/+bug/1731921). Helps with https://bugzilla.suse.com/show_bug.cgi?id=1038223
Member
Author
|
Btw, I noticed that ec2-api doesn't seem to have the memcached settings in |
stefannica
requested changes
Nov 14, 2017
Contributor
stefannica
left a comment
There was a problem hiding this comment.
I'm not 100% sure, but I don't think this is needed anymore, because only one memcached server is used - the local one - even when HA is enabled. Please take a look at #1341 and let me know if I'm wrong.
Member
Author
Check again, we didn't merge that PR because it's actually incorrect to only use the local memcached server :-) |
bmwiedemann
approved these changes
Nov 14, 2017
Member
bmwiedemann
left a comment
There was a problem hiding this comment.
With network latency being in the millisecond range, 1s = 1000ms is actually on the very safe side.
rsalevsky
approved these changes
Nov 16, 2017
stefannica
approved these changes
Nov 16, 2017
gozer-gerrit
pushed a commit
to ArdanaCLM/glance-ansible
that referenced
this pull request
Dec 21, 2017
The deprecated keystonemiddleware.auth_token in-process token cache is replaced with a memcached server. Configure the auth_token keystone middleware for glance to authenticate and encrypt the token data stored in memcached. See Also: https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#memcache-protection Set socket_timeout for memcached connections to 1s. The default for memcache_pool_socket_timeout is 3 seconds. When one memcached server goes down, then every connection pool for memcached (in each process, on each node) need to identify that this server is down, and that has some dramatic impact on performance. See crowbar/crowbar-openstack#1429 Merged: Idb8a278d87ea46b1bce335357bce64f74a68dca0 Merged: I65218e71dd0260bf135e733ff4bf5dfde09d7819 Change-Id: I469d2babae33ff4b8602fbbc46584c7960557eb7
gozer-gerrit
pushed a commit
to ArdanaCLM/cinder-ansible
that referenced
this pull request
Dec 21, 2017
The deprecated keystonemiddleware.auth_token in-process token cache is replaced with a memcached server. Configure the auth_token keystone middleware for glance to authenticate and encrypt the token data stored in memcached. See Also: https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#memcache-protection Set socket_timeout for memcached connections to 1s. The default for memcache_pool_socket_timeout is 3 seconds. When one memcached server goes down, then every connection pool for memcached (in each process, on each node) need to identify that this server is down, and that has some dramatic impact on performance. See crowbar/crowbar-openstack#1429 Change-Id: I6c178ebbd8bc8847bc160501078325568a3e5728 Merged: Idb8a278d87ea46b1bce335357bce64f74a68dca0 Merged: I65218e71dd0260bf135e733ff4bf5dfde09d7819
openstack-gerrit
pushed a commit
to openstack/oslo.cache
that referenced
this pull request
Jul 22, 2019
Lowering this setting still allows for reasonable confidence of a timeout without tanking performance. Certain downstream projects have already made this optimization: crowbar/crowbar-openstack#1429 Change-Id: Iaca2f8cc09a6acae425c020a9d14da3ffde05dec
openstack-gerrit
pushed a commit
to openstack/openstack
that referenced
this pull request
Jul 22, 2019
* Update oslo.cache from branch 'master'
- Lower the memcache_socket_timeout to one second
Lowering this setting still allows for reasonable confidence of a
timeout without tanking performance.
Certain downstream projects have already made this optimization:
crowbar/crowbar-openstack#1429
Change-Id: Iaca2f8cc09a6acae425c020a9d14da3ffde05dec
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The default is 3 seconds, which is very high. When one memcached server
goes down, then every connection pool for memcached (in each process, on
each node) need to identify that this server is down, and that has some
dramatic impact on performance.
Lowering this to 1 second, while still being a reasonable value, helps
quite a bit. I would even consider lowering this to 0.25, but oslo.cache
doesn't support float values there (see
https://bugs.launchpad.net/oslo.cache/+bug/1731921).
Helps with https://bugzilla.suse.com/show_bug.cgi?id=1038223