Add templating, via jinja2, to cephadm#52492
Conversation
|
Open question: If we are going to add external deps to cephadm and will do so via pip, should we also include hashes in our requreiments file? Example: |
eeab467 to
02a2315
Compare
02a2315 to
79eb613
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
79eb613 to
f43a389
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
f43a389 to
8cd1d05
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
8cd1d05 to
cdcdadd
Compare
src/cephadm/cephadm.py
Outdated
| _TMPL_DAEMON_UNIT = """# generated by cephadm | ||
| [Unit] | ||
| Description=Ceph %i for {fsid} | ||
| Description=Ceph %i for {{fsid}} |
There was a problem hiding this comment.
maybe now is a good time to formally move these into a .j2 (or .jinja) template file?
There was a problem hiding this comment.
I think that's doable. I was looking at the jinj2 docs (and source) recently and the package loader appears to support zipimport (which IIUC is the basis for zipapp).
What I'll do is try this out in a an experimental branch and if all goes smoothly I'll fold that branch back into this one. If things don't work out I'll report my findings here.
cdcdadd to
bf4f12e
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
fa519b1 to
9d157d6
Compare
797de3e to
fd86aa1
Compare
fd86aa1 to
384a5ff
Compare
|
jenkins test dashboard cephadm |
adk3798
left a comment
There was a problem hiding this comment.
Builds are now all working. But seeing some test failures.
d6569f9 to
d5bf8a3
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
We can not rely on any particular python version (py 3.6+ is supported) and can not assume any particular architecture. So using wheels based on the build system is pointless. Installing binary .so files compiled from C/C++ similarly so. Attempt to block the behaviors when adding dependencies to the zipapp. Signed-off-by: John Mulligan <jmulligan@redhat.com>
Unfortunately, a single simple call to pip does not work on all the distributions that ceph is built on. In particular, Ubuntu 20.04 and Ubuntu 22.04 come with pip versions that can not correctly handle disabling wheels and installing Jinja2 (it tries to use the markupsafe dependency before it is installed). This can be worked around by using a virtual env and updating pip before proceeding. However, this is not enough because CentOS/RHEL 8 uses python 3.6 and there is no version of pip that supports 3.6 that we can update to that is new enough to fix the issue with disabling wheels. The workaround in this case is to install each dependency one at a time through multiple calls to pip. Because of this extra complexity is it simpler to eschew the use of a requirements.txt file in build.py entirely. Thus the zipapp is built using build.py only. Requirements files for cephadm are for setting up the tox environments *only*. For completeness a new option is added that gives the caller control over when build.py uses a virtualenv or not. Thus the build.py script requires at least one of: a working pip that handles disabling wheels; or, a virtualenv (venv) and the ability to update to a working version of pip. If the list of distros ceph supports (and the python versions they use) ever becomes simpler/newer some of this complexity could be removed from the build.py script. Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add tests that cover the four main distros that ceph is built on (in the ceph infra). These tests should not be run by automation as they are slow and have special requirements like a working podman. Instead, these are provided to be run by a dev when build.py is updated. Signed-off-by: John Mulligan <jmulligan@redhat.com>
Signed-off-by: John Mulligan <jmulligan@redhat.com>
d5bf8a3 to
7a0f17f
Compare
Asserts more about the unit files. Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add tests that check the generation of the standard systemd unit for cephadm services. This test ensures that non trivial changes to the content of these files are noticed. Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add Jinja2 and MarkupSafe dependencies to a requirements.txt style file. This file tracks the dependencies needed to run the cephadm libs in the unit test framework. The actual dependencies that get added to the ziapp are managed by build.py but mirrored here. Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add `-rzipapp-reqs.txt` to the unit tests and mypy environments in tox.ini, enabling the use of dependencies outside the stdlib. Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add a simple wrapper function for templating from a string to a string. Signed-off-by: John Mulligan <jmulligan@redhat.com>
The somewhat complex string assembly of the main systemd unit file for cephadm services can benefit from using a standard templating approach. Signed-off-by: John Mulligan <jmulligan@redhat.com>
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Expand the templating module so that templates can be source from the python package. Add (more) convenient to use methods. Signed-off-by: John Mulligan <jmulligan@redhat.com>
Copy the current template strings into files under the `templates` dir. Add a enum for holding the names of known template files. Signed-off-by: John Mulligan <jmulligan@redhat.com>
Switch off of the embedded template strings to using the recently added template files. Signed-off-by: John Mulligan <jmulligan@redhat.com>
7a0f17f to
ae8a218
Compare
|
reruns of failed jobs: https://pulpito.ceph.com/adking-2023-11-05_15:58:41-orch:cephadm-wip-phlogistonjohn-testing-2023-11-01-2015-distro-default-smithi/ After reruns, 1 failure:
|
There was a problem hiding this comment.
I would imagine at some point a user might want to override this file similar to the service templates in the mgr
Contribution Guidelines
To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
Checklist
opened tracker ticketShow available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume toxjenkins test windows