qa/keystone: use uwsgi for daemon by tobias-urdin · Pull Request #66517 · ceph/ceph

tobias-urdin · 2025-12-04T20:05:30Z

Instead of using the builtin development server that can only serve a single request and is discouraged start Keystone with uwsgi instead.

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

cbodley · 2025-12-04T20:12:17Z

thanks, do you need any help testing this?

do you think this could resolve https://tracker.ceph.com/issues/72968?

cbodley · 2025-12-04T21:35:55Z

single job pending in https://pulpito.ceph.com/cbodley-2025-12-04_21:25:40-rgw:tempest-main-distro-default-smithi/

tobias-urdin · 2025-12-05T09:20:48Z

@cbodley my hope is that it will help, I've updated the patch again as I was installing uwsgi in the wrong venv, can you test it again please thanks!

cbodley · 2025-12-05T14:21:25Z

https://pulpito.ceph.com/cbodley-2025-12-05_14:17:02-rgw:tempest-main-distro-default-smithi/

tobias-urdin · 2025-12-05T15:05:17Z

2025-12-05T14:45:32.028 INFO:tasks.keystone.client.0.smithi031.stderr:/home/ubuntu/cephtest/keystone/.tox/venv/bin/uwsgi: unrecognized option '--plugins python'

@cbodley updated again

cbodley · 2025-12-05T15:07:32Z

scheduled again with:

$ teuthology-suite -m smithi -s rgw/tempest --ceph-repo https://github.com/ceph/ceph.git -c main -S 8611241dc028e3b148e236f6c5d350f9f0381cb1 --suite-repo https://github.com/tobias-urdin/ceph.git --suite-branch keystone-uwsgi --filter tasks/tempest -p 75

(using recent packages for 'main' to avoid waiting for builds)

pending in https://pulpito.ceph.com/cbodley-2025-12-05_15:06:57-rgw:tempest-main-distro-default-smithi/

github-actions · 2026-02-03T15:13:26Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

ivancich · 2026-02-19T15:10:12Z

@tobias-urdin Just wanted to ping you and see where things stand here.

tobias-urdin · 2026-02-20T07:50:25Z

I don't recall what the last failure was here so I think we need to run it again to get new logs to analyze.

cbodley · 2026-02-26T15:15:31Z

pending in https://pulpito.ceph.com/cbodley-2025-12-05_15:06:57-rgw:tempest-main-distro-default-smithi/

logs are gone from before the ~~apocalypse~~ lab move, so i'll schedule another one

cbodley · 2026-02-26T17:44:38Z

i'll schedule another one

sorry, i forgot that keystone is currently broken due to https://tracker.ceph.com/issues/75041. #67549 includes further investigation

cbodley · 2026-02-26T18:19:36Z

testing this with #67549 gave this output https://qa-proxy.ceph.com/teuthology/cbodley-2026-02-26_18:00:26-rgw:tempest-main-distro-default-trial/72598/teuthology.log

2026-02-26T18:13:14.879 DEBUG:teuthology.orchestra.run.trial073:> /home/ubuntu/cephtest/keystone/.tox/venv/bin/uwsgi --env OS_KEYSTONE_CONFIG_FILES=/home/ubuntu/cephtest/keystone/etc/keystone.conf --uwsgi-socket 127.0.0.1:5001 --buffer-size 65535 --master --enable-threads --processes 4 --thunder-lock --plugins python --lazy-apps --file /home/ubuntu/cephtest/keystone/.tox/venv/bin/keystone-wsgi-public
2026-02-26T18:13:14.893 INFO:tasks.keystone.client.0:Started
2026-02-26T18:13:14.904 DEBUG:teuthology.orchestra.run.trial073:> cd /home/ubuntu/cephtest/keystone && source .tox/venv/bin/activate && sleep 15
2026-02-26T18:13:14.917 INFO:tasks.keystone.client.0.trial073.stderr:open("./python_plugin.so"): No such file or directory [core/utils.c line 3709]
2026-02-26T18:13:14.929 INFO:tasks.keystone.client.0.trial073.stderr:!!! UNABLE to load uWSGI plugin: ./python_plugin.so: cannot open shared object file: No such file or directory !!!
2026-02-26T18:13:14.941 INFO:tasks.keystone.client.0.trial073.stderr:*** Starting uWSGI 2.0.31 (64bit) on [Thu Feb 26 18:13:14 2026] ***
2026-02-26T18:13:14.951 INFO:tasks.keystone.client.0.trial073.stderr:compiled with version: 11.4.0 on 26 February 2026 18:12:53
2026-02-26T18:13:14.963 INFO:tasks.keystone.client.0.trial073.stderr:os: Linux-5.15.0-171-generic #181-Ubuntu SMP Fri Feb 6 22:44:50 UTC 2026
2026-02-26T18:13:14.975 INFO:tasks.keystone.client.0.trial073.stderr:nodename: trial073
2026-02-26T18:13:14.986 INFO:tasks.keystone.client.0.trial073.stderr:machine: x86_64
2026-02-26T18:13:14.998 INFO:tasks.keystone.client.0.trial073.stderr:clock source: unix
2026-02-26T18:13:15.009 INFO:tasks.keystone.client.0.trial073.stderr:detected number of CPU cores: 24
2026-02-26T18:13:15.021 INFO:tasks.keystone.client.0.trial073.stderr:current working directory: /home/ubuntu
2026-02-26T18:13:15.032 INFO:tasks.keystone.client.0.trial073.stderr:detected binary path: /home/ubuntu/cephtest/keystone/.tox/venv/bin/uwsgi
2026-02-26T18:13:15.044 INFO:tasks.keystone.client.0.trial073.stderr:!!! no internal routing support, rebuild with pcre support !!!
2026-02-26T18:13:15.055 INFO:tasks.keystone.client.0.trial073.stderr:your processes number limit is 511108
2026-02-26T18:13:15.067 INFO:tasks.keystone.client.0.trial073.stderr:your memory page size is 4096 bytes
2026-02-26T18:13:15.085 INFO:tasks.keystone.client.0.trial073.stderr:detected max file descriptor number: 1024
2026-02-26T18:13:15.096 INFO:tasks.keystone.client.0.trial073.stderr:lock engine: pthread robust mutexes
2026-02-26T18:13:15.108 INFO:tasks.keystone.client.0.trial073.stderr:thunder lock: enabled
2026-02-26T18:13:15.120 INFO:tasks.keystone.client.0.trial073.stderr:uwsgi socket 0 bound to TCP address 127.0.0.1:5001 fd 3
2026-02-26T18:13:15.131 INFO:tasks.keystone.client.0.trial073.stderr:Python version: 3.10.12 (main, Jan 26 2026, 14:55:28) [GCC 11.4.0]
2026-02-26T18:13:15.142 INFO:tasks.keystone.client.0.trial073.stderr:Python main interpreter initialized at 0x5636dbd62490
2026-02-26T18:13:15.153 INFO:tasks.keystone.client.0.trial073.stderr:python threads support enabled
2026-02-26T18:13:15.165 INFO:tasks.keystone.client.0.trial073.stderr:your server socket listen backlog is limited to 100 connections
2026-02-26T18:13:15.176 INFO:tasks.keystone.client.0.trial073.stderr:your mercy for graceful operations on workers is 60 seconds
2026-02-26T18:13:15.188 INFO:tasks.keystone.client.0.trial073.stderr:mapped 671795 bytes (656 KB) for 4 cores
2026-02-26T18:13:15.199 INFO:tasks.keystone.client.0.trial073.stderr:*** Operational MODE: preforking ***
2026-02-26T18:13:15.210 INFO:tasks.keystone.client.0.trial073.stderr:*** uWSGI is running in multiple interpreter mode ***
2026-02-26T18:13:15.221 INFO:tasks.keystone.client.0.trial073.stderr:spawned uWSGI master process (pid: 21247)
2026-02-26T18:13:15.233 INFO:tasks.keystone.client.0.trial073.stderr:spawned uWSGI worker 1 (pid: 21248, cores: 1)
2026-02-26T18:13:15.244 INFO:tasks.keystone.client.0.trial073.stderr:spawned uWSGI worker 2 (pid: 21249, cores: 1)
2026-02-26T18:13:15.256 INFO:tasks.keystone.client.0.trial073.stderr:spawned uWSGI worker 3 (pid: 21250, cores: 1)
2026-02-26T18:13:15.268 INFO:tasks.keystone.client.0.trial073.stderr:spawned uWSGI worker 4 (pid: 21251, cores: 1)
2026-02-26T18:13:15.280 INFO:tasks.keystone.client.0.trial073.stderr:failed to open python file /home/ubuntu/cephtest/keystone/.tox/venv/bin/keystone-wsgi-public
2026-02-26T18:13:15.295 INFO:tasks.keystone.client.0.trial073.stderr:failed to open python file /home/ubuntu/cephtest/keystone/.tox/venv/bin/keystone-wsgi-public
2026-02-26T18:13:15.306 INFO:tasks.keystone.client.0.trial073.stderr:unable to load app 0 (mountpoint='') (callable not found or import error)
2026-02-26T18:13:15.317 INFO:tasks.keystone.client.0.trial073.stderr:unable to load app 0 (mountpoint='') (callable not found or import error)
2026-02-26T18:13:15.329 INFO:tasks.keystone.client.0.trial073.stderr:*** no app loaded. going in full dynamic mode ***
2026-02-26T18:13:15.341 INFO:tasks.keystone.client.0.trial073.stderr:*** no app loaded. going in full dynamic mode ***
2026-02-26T18:13:15.352 INFO:tasks.keystone.client.0.trial073.stderr:failed to open python file /home/ubuntu/cephtest/keystone/.tox/venv/bin/keystone-wsgi-public
2026-02-26T18:13:15.364 INFO:tasks.keystone.client.0.trial073.stderr:failed to open python file /home/ubuntu/cephtest/keystone/.tox/venv/bin/keystone-wsgi-public
2026-02-26T18:13:15.376 INFO:tasks.keystone.client.0.trial073.stderr:unable to load app 0 (mountpoint='') (callable not found or import error)
2026-02-26T18:13:15.387 INFO:tasks.keystone.client.0.trial073.stderr:unable to load app 0 (mountpoint='') (callable not found or import error)
2026-02-26T18:13:15.399 INFO:tasks.keystone.client.0.trial073.stderr:*** no app loaded. going in full dynamic mode ***
2026-02-26T18:13:15.411 INFO:tasks.keystone.client.0.trial073.stderr:*** no app loaded. going in full dynamic mode ***

failed to open python file /home/ubuntu/cephtest/keystone/.tox/venv/bin/keystone-wsgi-public

this part is because 2025.2 removed keystone-wsgi-public. do you know to accomplish this part? it was recommended by 2025.2 release notes

Deployment tooling should instead reference the Python module paths for the service, keystone.wsgi.api, if their chosen WSGI server supports this (gunicorn, uWSGI) or implement a .wsgi script themselves if not (mod_wsgi).

cbodley · 2026-02-26T18:27:27Z

in case it helps, a suggestion from red hat:

Please ask them to try the following systemd unit from upstream:

[Unit]
Description = Devstack devstack@keystone.service

[Service]
RestartForceExitStatus = 100
NotifyAccess = all
Restart = always
KillMode = process
Type = notify
ExecReload = /usr/bin/kill -HUP $MAINPID
ExecStart = /bin/uwsgi --procname-prefix keystone --ini /etc/keystone/keystone-uwsgi-public.ini --venv /opt/stack/data/venv
User = stack
SyslogIdentifier = devstack@keystone.service
Environment = "PATH=/bin:/opt/stack/data/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/sbin"

[Install]
WantedBy = multi-user.target

Instead of using the builtin development server that can only serve a single request and is discouraged start Keystone with uwsgi instead. Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

tobias-urdin · 2026-02-27T09:50:41Z

@cbodley Thanks! I updated the PR so that it instead loads Keystone from module in uwsgi and not the file that is now removed in 2024.2

diff --git a/qa/tasks/keystone.py b/qa/tasks/keystone.py
index c19bf05dcef..1bc9249d55f 100644
--- a/qa/tasks/keystone.py
+++ b/qa/tasks/keystone.py
@@ -52,7 +52,7 @@ def get_keystone_uwsgi_cmd(ctx, conf_file):
     cmd = [uwsgibin, '--env', conf_env, '--uwsgi-socket', '127.0.0.1:5001',
            '--buffer-size', '65535', '--master', '--enable-threads',
            '--processes', '4', '--thunder-lock', '--plugins', 'python',
-           '--lazy-apps', '--file', keystonebin]
+           '--lazy-apps', '--module', 'keystone.wsgi.api:application']
     return cmd

 @contextlib.contextmanager

cbodley · 2026-03-02T15:37:22Z

thanks @tobias-urdin. my last attempt from https://qa-proxy.ceph.com/teuthology/cbodley-2026-02-26_19:17:01-rgw:tempest-main-distro-default-trial/72610/teuthology.log used --ini to point at keystone-uwsgi-public.ini which references module = keystone.wsgi.api:application for that

it does find the .ini file but fails to load the python plugin for it:

2026-02-26T19:29:56.624 INFO:tasks.keystone.client.0.trial073.stderr:open("./python_plugin.so"): No such file or directory [core/utils.c line 3709]
2026-02-26T19:29:56.636 INFO:tasks.keystone.client.0.trial073.stderr:!!! UNABLE to load uWSGI plugin: ./python_plugin.so: cannot open shared object file: No such file or directory !!!

i had added apt install uwsgi-plugin-python3 for that but it didn't work. that package installs a /usr/lib/uwsgi/plugins/python310_plugin.so according to https://packages.ubuntu.com/jammy/amd64/uwsgi-plugin-python3/filelist

cbodley · 2026-03-02T15:52:16Z

i had added apt install uwsgi-plugin-python3 for that but it didn't work. that package installs a /usr/lib/uwsgi/plugins/python310_plugin.so according to https://packages.ubuntu.com/jammy/amd64/uwsgi-plugin-python3/filelist

i guess the problem is that we're installing the plugin as a system package but uwsgi from pip in the virtualenv? i didn't see uwsgi plugins available from pip, so i'll try installing the uwsgi package too

tobias-urdin · 2026-03-05T15:35:38Z

closing in favor of #67660

tobias-urdin requested a review from a team as a code owner December 4, 2025 20:05

github-actions bot added the tests label Dec 4, 2025

cbodley approved these changes Dec 4, 2025

View reviewed changes

tobias-urdin force-pushed the keystone-uwsgi branch from 4887c52 to 1b3d9d2 Compare December 5, 2025 09:19

tobias-urdin force-pushed the keystone-uwsgi branch from 1b3d9d2 to 901c109 Compare December 5, 2025 15:03

github-actions bot added the stale label Feb 3, 2026

github-actions bot removed the stale label Feb 19, 2026

cbodley mentioned this pull request Feb 26, 2026

qa/rgw/keystone: bump repo again to stable/2025.2 #67549

Closed

qa/keystone: use uwsgi for daemon

345f3fa

Instead of using the builtin development server that can only serve a single request and is discouraged start Keystone with uwsgi instead. Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

tobias-urdin force-pushed the keystone-uwsgi branch from 901c109 to 345f3fa Compare February 27, 2026 09:49

kshtsk mentioned this pull request Mar 5, 2026

qa/tasks/keystone: upgrade keystone to 2025.2 #67660

Merged

14 tasks

tobias-urdin closed this Mar 5, 2026

Conversation

tobias-urdin commented Dec 4, 2025

Contribution Guidelines

Checklist

Uh oh!

cbodley commented Dec 4, 2025

Uh oh!

cbodley commented Dec 4, 2025

Uh oh!

tobias-urdin commented Dec 5, 2025

Uh oh!

cbodley commented Dec 5, 2025

Uh oh!

tobias-urdin commented Dec 5, 2025

Uh oh!

cbodley commented Dec 5, 2025

Uh oh!

github-actions bot commented Feb 3, 2026

Uh oh!

ivancich commented Feb 19, 2026

Uh oh!

tobias-urdin commented Feb 20, 2026

Uh oh!

cbodley commented Feb 26, 2026

Uh oh!

cbodley commented Feb 26, 2026

Uh oh!

cbodley commented Feb 26, 2026

Uh oh!

cbodley commented Feb 26, 2026

Uh oh!

tobias-urdin commented Feb 27, 2026

Uh oh!

cbodley commented Mar 2, 2026

Uh oh!

cbodley commented Mar 2, 2026

Uh oh!

tobias-urdin commented Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants