Note: the Containerfile is not in the root of the repo and is instead placed into ./container for some practical reasons:

1. for developers using non-Linux systems to build the container image (i.e., MacOS users), the build context needs to copy the entire ceph repo into a VM for the build, which is unnecessary and takes a truly massive amount of time (for no benefit)
2. it provides a space for adding scripts/makefile to help with the container build into the ./container dir, keeping any container tooling in the same place without cluttering the repo root

the squid release will be dropping support for centos 8, and there's a need a build containers based on centos 9. there's been some discussions around this in #53901 and elsewhere on slack. if we're starting from scratch here, it would be wonderful if we could start with centos 9 instead

the squid release will be dropping support for centos 8, and there's a need a build containers based on centos 9. there's been some discussions around this in #53901 and elsewhere on slack. if we're starting from scratch here, it would be wonderful if we could start with centos 9 instead

I agree that it would be nice, but I (or someone) would need to spend time making sure all the el9 locations are ready. Ideally quincy, reef, squid, and main all have el9 builds, and I'm not sure that's true today or not.

Can we at least get this merged and then convert to el9 later? This PR is clean and good to go from my end, but I'm waiting to hear back from @guits about checking the secret so that the quay login works.

#54575 (comment)

[update]

Just realized that this is probably not a configuration issue, but a security thing. GitHub doesn't allow secrets to be used from pull requests so that users can't "exfiltrate" secrets by making arbitrary PRs.
https://stackoverflow.com/questions/73866908/github-actions-main-repository-secret-not-picked-up-from-pull-request-build

This means that the GH action is good for builds from committed PRs, but not viable for building/publishing images related to pull requests. I guess the shaman system would be good for that, and it seems like something is already set up to build the quay.io/ceph-ci images like that. @dmick does that sound right?

this has an approval. does it need any further review/testing to merge? afaik nothing depends on this yet so there's little risk to merging

what are the next steps?

"This means that the GH action is good for builds from committed PRs, but not viable for building/publishing images related to pull requests. I guess the shaman system would be good for that, and it seems like something is already set up to build the quay.io/ceph-ci images like that. @dmick does that sound right?"

I can't speak to the GH actions; I don't understand anything about the context they run in, or how one might possibly handle secrets safely. I know that things like those secrets are stored in Jenkins, which is what does the CI builds (shaman and chacra are just storage/db for the build results). I do want to point out however that we generally don't push CI images to quay.io, but rather our own quay.ceph.io (quay.io has usage limits and we do a lot of builds).

I've been working on github.com/dmick/ceph-buildah which uses the buildah utility to build the containers. The huge advantage is that it allows a series of shell commands to make incremental changes to a working container, and then commits the final result, so all the conditional logic can be coded in the toplevel shell command, which is a huge help at bug diagnosis. I think I very much want to go this direction rather than a Dockerfile for the revamp. I've been building containers of centos8 and centos9 x86 and arm and having some luck; I want to do that formally and run some formal container-based tests (probably the rbd iscsi tests to catch "is the iscsi inclusion good as well), and then I think I'd like to propose integrating that into ceph.git rather than this.

Red Hat's build system (OSBS) input is a single Dockerfile per container, and I think a Dockerfile is generally the input format for other build systems (like Tekton). I think a Dockerfile is going to be easier to maintain.

For debugging, I like to use the podman build --force-rm=false option to avoid deleting intermediate stages. (it leaves a lot behind, so podman image prune cleans up.)

oh well that might be a good argument for dockerfiles I guess. crap. Maybe modern dockerfiles are less painful to do this ridiculous handstanding in.

@dmick I was pointed at this conversation during the most recent infra meeting. I am also interested in the topic and am happy to discuss any challenges you are hitting wrt building images: esp. wrt "this ridiculous handstanding" might entail :-)

I also am a fan of buildah, but as Ken notes it's not so widespread and more niche than Dockerfiles.

Mostly it's "if somehow we end up with another 200-line continued bash statement I will descend upon the author with a mighty and terrible force" but I think multiple RUN statements and --squash are the way out of that horror. ARG and short conditional bash are probably maintainable/usable.

Mostly it's "if somehow we end up with another 200-line continued bash statement I will descend upon the author with a mighty and terrible force"

lol :-)

but I think multiple RUN statements and --squash are the way out of that horror. ARG and short conditional bash are probably maintainable/usable.

That seems fine. I also dislike having a bunch of logic on a single RUN line. If the logic is necessary I'll often tend to farm the work out to a real script outside the {Dock,Contain}erfile.

In our project we've run into several problems around CentOS Stream, causing us to look for Debian based Ceph container images. As we could not find suitable ones, we are now looking at building them ourselves and would like to use the code from this PR as base instead of the one based on Make (I arrived here through issue 2171).

So I would like to ask what state this PR is in? I see some open questions / work-in-progress-notes while I do not find anything which, at least to me, looks like testing, e.g. compared with the previous target test.staging (though I only just started reading through that code, so no idea what it actually does/tests, yet).

More work is needed; I don't know how far this is from final form yet. But container building isn't particularly magic, and if you end up at a slightly different place than we do (besides the obvious 'based on Debian' thing, which will involve a fair bit of package name munging as I'm sure you know), as long as you have the dependencies you need it'll probably be fine. I have a separate repo I've been working in, dmick/container.git, that might or might not be a better starting place.

Whats the status of this? c8 is dead now.

More work is needed; I don't know how far this is from final form yet. But container building isn't particularly magic, and if you end up at a slightly different place than we do (besides the obvious 'based on Debian' thing, which will involve a fair bit of package name munging as I'm sure you know), as long as you have the dependencies you need it'll probably be fine. I have a separate repo I've been working in, dmick/container.git, that might or might not be a better starting place.

I keep hitting this problem though I still have hopes we can work-around it (for our specific usecase) because we were able to install a kind-of-working cluster on bare-metal, so without any containers, using the DEB packages. Since we want to leverage cephadm, bare-metal is not our preference at all.

I'm mentioning this here because, according to the comments in the linked bug report, Centos Stream 9 is also affected and with Centos Stream 8 being EOL I'm hoping someone here has found a solution/workaround to share.

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

This initiative seems to have fallen asleep somewhat. For what it's worth, I made an image build based on this PR work using latest Debian Bookworm as base. It took some tinkering with Ceph sources, completely disabling the built-in mgr dashboard to work around the PyO3 multi-init error. I'm still working on improving it and, so far, testing yielded no issues.

Hopefully this gives someone a push to move forward with this PR :-)

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

For those watching this PR or who stumble across it in the future, PR #59868 has been merged and this adds the necessary {Container,Docker}file and scripts to build containers from ceph/ceph repo in the existing ceph build infra.
@BlaineEXE may want to close this if that appears sufficient, otherwise I expect the bot will close this after some time of inactivity.

I have the amazing ability to always paste the wrong thing. PR #59868 is it. I have also updated my previous comment (but edits don't appear in email notifications, thus this comment). :-)