Skip to content

Providing SDE docker images from CI#105

Merged
fruffy merged 16 commits into
p4lang:mainfrom
n1tr0-5urf3r:main
Jun 28, 2025
Merged

Providing SDE docker images from CI#105
fruffy merged 16 commits into
p4lang:mainfrom
n1tr0-5urf3r:main

Conversation

@n1tr0-5urf3r

Copy link
Copy Markdown
Contributor

This PR adds an automated CI workflow for building and pushing Docker images to the GitHub Container Registry (#103). The container image can be used to build and verify data plane builds in CI and to compile a P4 project locally for the Tofino model, eliminating the need to compile p4studio. I have included a Dockerfile and a lightweight profile for P4Studio, which may still require some adjustment. For example, are the BFRT and GRPC drivers required for the image?

Also, ccache doesnt work with the docker build process in CI, so building the image is 2-3h hours currently.

Further, some steps are redundant with the test matrix from the ubuntu-test CI, i.e., building p4studio (without the Dockerfile and with a different profile though). We could either

  1. Keep both CI steps in separate workflows
  2. Use the testing profile for the Docker image as well, and use the Dockerfile to build p4studio, build the image, and run the tests in one CI file.

Option 1 requires more CI building resources since p4studio is built twice, but I prefer this option.

After pushing the image, it may have to be made public in the organization's settings under "Packages"

@fruffy

fruffy commented Jun 22, 2025

Copy link
Copy Markdown
Contributor

Great!

ccache might just require a little bit of extra set up. We have something similar for p4c:
https://github.com/p4lang/p4c/blob/main/.github/workflows/disabled/ci-ubuntu-18-nightly.yml#L41

You may also have to set the directory: https://github.com/p4lang/p4c/blob/main/tools/ci-build.sh#L133

Also, please sign the DCO:
https://github.com/p4lang/p4c/blob/main/CONTRIBUTING.md#contributing-license

@n1tr0-5urf3r n1tr0-5urf3r force-pushed the main branch 2 times, most recently from 6ef8dbe to dd84d09 Compare June 25, 2025 10:24
@n1tr0-5urf3r

n1tr0-5urf3r commented Jun 25, 2025

Copy link
Copy Markdown
Contributor Author

The current workflow for p4c (https://github.com/p4lang/p4c/blob/main/.github/workflows/ci-container-image.yml) uses the Dockerx build step which differs from the one you mentioned in https://github.com/p4lang/p4c/blob/main/.github/workflows/disabled/ci-ubuntu-18-nightly.yml#L41.
To get it to work with Dockerx building, I now mount a cache volume for ccache to the Docker image which should work without the need for the export_ccache.sh script.

However, the cache hit rate seems marginal:

 #14 7068.5 Summary:
#14 7068.5   Hits:               84 /  6185 (1.36 %)
#14 7068.5     Direct:           59 /  6254 (0.94 %)
#14 7068.5     Preprocessed:     25 /  6173 (0.40 %)
#14 7068.5   Misses:           6101
#14 7068.5     Direct:         6195
#14 7068.5     Preprocessed:   6148
#14 7068.5   Uncacheable:      1099
#14 7068.5 Primary storage:
#14 7068.5   Hits:              143 / 12485 (1.15 %)
#14 7068.5   Misses:          12342
#14 7068.5   Cache size (GB):  0.40 /  5.00 (7.97 %)

The DCO verifier should be happy now.

EDIT: Seems that #98 has broken something for the Docker CI:

#14 5437.4 [ 35%] Generating bf_kdrv.ko
#14 5437.4 make[3]: *** modules: No such file or directory.  Stop.
#14 5437.4 make[2]: *** [pkgsrc/bf-drivers/kdrv/bf_kdrv/CMakeFiles/bf_kdrv.dir/build.make:76: pkgsrc/bf-drivers/kdrv/bf_kdrv/bf_kdrv.ko] Error 2
#14 5437.4 make[1]: *** [CMakeFiles/Makefile2:7197: pkgsrc/bf-drivers/kdrv/bf_kdrv/CMakeFiles/bf_kdrv.dir/all] Error 2
#14 5437.4 make[1]: *** Waiting for unfinished jobs....
#14 5437.4 [ 35%] Building C object pkgsrc/bf-drivers/src/dru_sim/CMakeFiles/dru_model.dir/dru_intf_tcp.c.o
#14 5437.4 [ 35%] Building C object pkgsrc/bf-drivers/src/dru_sim/CMakeFiles/dru_model.dir/__/lld/lld_dr_regs_tof.c.o
#14 5437.4 [ 35%] Building C object pkgsrc/bf-drivers/src/dru_sim/CMakeFiles/dru_model.dir/__/lld/lld_dr_regs_tof2.c.o
#14 5437.4 [ 35%] Built target dru_sim_o
#14 5437.4 [ 35%] Building C object pkgsrc/bf-drivers/src/dru_sim/CMakeFiles/dru_model.dir/__/lld/lld_dr_regs_tof3.c.o
#14 5437.4 [ 35%] Building C object pkgsrc/bf-drivers/src/dru_sim/CMakeFiles/dru_model.dir/__/lld/lld_dr_regs.c.o
#14 5437.4 [ 35%] Linking C static library ../../../../../install/lib/libdru_model.a
#14 5437.4 [ 35%] Built target dru_model
#14 5437.4 make: *** [Makefile:156: all] Error 2
#14 5437.4   - libcrafter: Cmd 'make --jobs=4 install' took: 0:00:17.641680, status: 2
#14 5437.4 
#14 5437.4 For more details, see: /open-p4studio/p4studio/logs/p4studio_2025-06-25_10-27-33.log
#14 5437.4 Error: Installation completed unsuccessfully
#14 ERROR: process "/bin/sh -c ./p4studio/p4studio profile apply --jobs $(nproc) ./p4studio/profiles/docker.yaml &&     ccache -s" did not complete successfully: exit code: 1

Not sure why though... It works for the testing CI

@fruffy

fruffy commented Jun 25, 2025

Copy link
Copy Markdown
Contributor

The current workflow for p4c (https://github.com/p4lang/p4c/blob/main/.github/workflows/ci-container-image.yml) uses the Dockerx build step which differs from the one you mentioned in https://github.com/p4lang/p4c/blob/main/.github/workflows/disabled/ci-ubuntu-18-nightly.yml#L41.

Yes, for the container workflow in P4C we have the same problem that ccache does not work actually. For the Ubuntu 18.04 workflow things were working using the steps I provided. We can leave it for now and figure it out later, but it would be nice to make use of ccache. Otherwise the build will take a loooong time.

EDIT: Seems that #98 has broken something for the Docker CI:

This step builds a kernel module. Possible that these folders are not mapped in Docker?

@n1tr0-5urf3r

Copy link
Copy Markdown
Contributor Author

Alright, I will create an issue later so that the ccache feature for the docker CI does not fall into oblivion :) Should I also remove all ccache related parts for now?

Yes, it seems that this is the problem, and building kernel modules in Docker does not look that straightforward. Is there a configuration flag or environment variable that we could set to disable kernel module building?

@fruffy

fruffy commented Jun 25, 2025

Copy link
Copy Markdown
Contributor

Is there a configuration flag or environment variable that we could set to disable kernel module building?

Yes, there should be an option in the CMakelists to build kernel modules, You can toggle that.

@fruffy

fruffy commented Jun 25, 2025

Copy link
Copy Markdown
Contributor

Feel free to create the issue and remove the ccache changes. Maybe add them to the issue for the future.

@n1tr0-5urf3r

Copy link
Copy Markdown
Contributor Author

Is there a configuration flag or environment variable that we could set to disable kernel module building?

Yes, there should be an option in the CMakelists to build kernel modules, You can toggle that.

I tried to set it with CMAKE_ARGS="-DKERNEL-MODULES=OFF" ./p4studio/p4studio profile apply --jobs $(nproc) ./p4studio/profiles/docker.yaml and CMAKE_FLAGS without success. I now simply added a sed replace to the Dockerfile.

@fruffy

fruffy commented Jun 26, 2025

Copy link
Copy Markdown
Contributor

Is there a configuration flag or environment variable that we could set to disable kernel module building?

Yes, there should be an option in the CMakelists to build kernel modules, You can toggle that.

I tried to set it with CMAKE_ARGS="-DKERNEL-MODULES=OFF" ./p4studio/p4studio profile apply --jobs $(nproc) ./p4studio/profiles/docker.yaml and CMAKE_FLAGS without success. I now simply added a sed replace to the Dockerfile.

There is a way to pass CMake args to the studio build, or use the configuration file to pass the CMake arguments. But I do not quite remember how. I would need to check.

If passing this option does not work its a bug. Let's not use the sed hack.

@vgurevich

Copy link
Copy Markdown
Contributor

@fruffy , @n1tr0-5urf3r !

Sorry for chiming in late, but if you are using p4studio tool to build open-p4studio (and I think that this is the right approach, since that's what others will use anyway), then p4studio's profiles have a standard parameter that turns building the kernel modules on or off.

Here is a trivial profile for your reference:

global-options: {}
features:
  drivers:
    kernel-modules: true
  p4-examples:
  - p4-16-programs
architectures:
- tofino

You can set this kernel-modules: parameter to true or false and be done in no time.

@n1tr0-5urf3r

Copy link
Copy Markdown
Contributor Author

You can set this kernel-modules: parameter to true or false and be done in no time.

Thank you @vgurevich, this was exactly what I was looking for! I further added the kernel-modules flag to p4studio/profiles/README.md for documentation.

jafingerhut and others added 15 commits June 27, 2025 08:31
…port (p4lang#84)

Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: fruffy <fruffy@nyu.edu>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Vladimir Gurevich <vag@p4ica.com>
Signed-off-by: fruffy <fruffy@nyu.edu>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
@fruffy fruffy merged commit 21a91e7 into p4lang:main Jun 28, 2025
2 checks passed
rcgoodfellow pushed a commit to oxidecomputer/tofino-sde that referenced this pull request Feb 21, 2026
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: fruffy <fruffy@nyu.edu>
Signed-off-by: Vladimir Gurevich <vag@p4ica.com>
Co-authored-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Co-authored-by: Fabian Ruffy <5960321+fruffy@users.noreply.github.com>
Co-authored-by: Vladimir Gurevich <vag@p4ica.com>
jafingerhut added a commit to jafingerhut/open-p4studio that referenced this pull request Feb 26, 2026
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Fabian Ihle <fabian.ihle@uni-tuebingen.de>
Signed-off-by: fruffy <fruffy@nyu.edu>
Signed-off-by: Vladimir Gurevich <vag@p4ica.com>
Co-authored-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Co-authored-by: Fabian Ruffy <5960321+fruffy@users.noreply.github.com>
Co-authored-by: Vladimir Gurevich <vag@p4ica.com>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants