Out of curiosity, when you say modernize, do you perhaps mean accepting the perhaps-unpleasant change in pip3 that prevents (or very strongly discourages) one from installing Python packages in system-wide directories? Perhaps considering using a Python virtual environment?

Out of curiosity, when you say modernize, do you perhaps mean accepting the perhaps-unpleasant change in pip3 that prevents (or very strongly discourages) one from installing Python packages in system-wide directories? Perhaps considering using a Python virtual environment?

Yes. We might have to go with the times here because the workarounds are quite aggressive/invasive.

I am exploring using poetry + virtualenv. If I work the kinks out you should be able to set up the entire Python environment in few commands:

sudo apt-get install -y python3-poetry python3-venv
cd p4c
poetry install
poetry shell

sudo applications can be run using sudo -E env PATH="$PATH" python3

FYI, my install-p4dev-v8.sh script here: https://github.com/jafingerhut/p4-guide/blob/master/bin/install-p4dev-v8.sh

can run as a normal user, with liberal sudo commands sprinkled throughout when needed, that start from unmodified Ubuntu 20.04, 22.04, or 24.04 system and install gRPC, Thrift, PI, BMv2, p4c, ptf, and mininet, all working together at least enough for the p4lang/tutorials repo exercises to work, and it uses a single Python venv for all Python packages installed.

I doubt that exactly what it does is what you need for this work, but just wanted to point it out as a working example of using a venv for all these tools together.

I believe I also needed LD_LIBRARY_PATH=$LD_LIBRARY_PATH in addition to PATH=$PATH from some sudo commands, perhaps ones like make check run in p4c/build.

Oh, and sometimes I found that instead of a command like sudo PATH=$PATH ... mn, where mn was installed in the Python venv, the best working alternative I found so far is sudo PATH=$PATH ... which mn``.

FYI, my install-p4dev-v8.sh script here: https://github.com/jafingerhut/p4-guide/blob/master/bin/install-p4dev-v8.sh

can run as a normal user, with liberal sudo commands sprinkled throughout when needed, that start from unmodified Ubuntu 20.04, 22.04, or 24.04 system and install gRPC, Thrift, PI, BMv2, p4c, ptf, and mininet, all working together at least enough for the p4lang/tutorials repo exercises to work, and it uses a single Python venv for all Python packages installed.

I doubt that exactly what it does is what you need for this work, but just wanted to point it out as a working example of using a venv for all these tools together.

I believe I also needed LD_LIBRARY_PATH=$LD_LIBRARY_PATH in addition to PATH=$PATH from some sudo commands, perhaps ones like make check run in p4c/build.

Yes, I am aware of these and have recommended it to students I work with! But for the sake of the compiler installation we can stay be much simpler. Nonetheless, the changes here should definitely integrate with your scripts. I'll explore further.

@jafingerhut Poetry works really well for managing Python dependencies. It was not difficult at all to set this up for all our distributions. All you need to do is call poetry shell in the p4c folder and you are working the correct Python environment.

Poetry also exposes a bunch of small issues we had hidden in the environment. For example, we are relying on the external p4runtime Python packages instead of the local p4runtime version from Github for testing.

One thing: It looks Poetry is not supported well Ubuntu18.04. This might be a good opportunity to finally deprecate this version.

@vbnogueira Trying to refactor the Python setup for TC STF I am seeing timeouts. Do you know possible reasons for that? Could be a simple fix.

@vbnogueira Trying to refactor the Python setup for TC STF I am seeing timeouts. Do you know possible reasons for that? Could be a simple fix.

I tested it out and I believe the VM is not getting started because the runner is not finding the virtme-run executable. It was expecting to find it under backends/tc/runtime/virtme-ng/bin/, but there is a change that seems to skip that part of the installation process [1].
I wasn't able to figure out a fix yet because today unfortunately was a complex day, but will do tomorrow.
Probably just need to figure a sane way to change where it looks for virtme-run

[1] https://github.com/p4lang/p4c/pull/5047/files#diff-76f06548e16be4f20ba423b3768e29bfca82a8ecd274407c02b37b3b5816f0c8L233

@vbnogueira Trying to refactor the Python setup for TC STF I am seeing timeouts. Do you know possible reasons for that? Could be a simple fix.

I tested it out and I believe the VM is not getting started because the runner is not finding the virtme-run executable. It was expecting to find it under backends/tc/runtime/virtme-ng/bin/, but there is a change that seems to skip that part of the installation process [1]. I wasn't able to figure out a fix yet because today unfortunately was a complex day, but will do tomorrow. Probably just need to figure a sane way to change where it looks for virtme-run

[1] https://github.com/p4lang/p4c/pull/5047/files#diff-76f06548e16be4f20ba423b3768e29bfca82a8ecd274407c02b37b3b5816f0c8L233

Thanks, that already helps me. I can likely fix this.

@vbnogueira Trying to refactor the Python setup for TC STF I am seeing timeouts. Do you know possible reasons for that? Could be a simple fix.

I tested it out and I believe the VM is not getting started because the runner is not finding the virtme-run executable. It was expecting to find it under backends/tc/runtime/virtme-ng/bin/, but there is a change that seems to skip that part of the installation process [1]. I wasn't able to figure out a fix yet because today unfortunately was a complex day, but will do tomorrow. Probably just need to figure a sane way to change where it looks for virtme-run

[1] https://github.com/p4lang/p4c/pull/5047/files#diff-76f06548e16be4f20ba423b3768e29bfca82a8ecd274407c02b37b3b5816f0c8L233

I finally found a way to fix the P4-TC tests, by using virtme-ng provided by the Debian repositories. Let me know whether that works for you.

@jafingerhut The transition to Ubuntu 24.04 is almost completed with this. #5183 is a prerequisite, I will clean up that PR later this afternoon and then it is ready for review.

The Fedora failures are unrelated and happen because of the distro changes.

@vbnogueira Trying to refactor the Python setup for TC STF I am seeing timeouts. Do you know possible reasons for that? Could be a simple fix.

I tested it out and I believe the VM is not getting started because the runner is not finding the virtme-run executable. It was expecting to find it under backends/tc/runtime/virtme-ng/bin/, but there is a change that seems to skip that part of the installation process [1]. I wasn't able to figure out a fix yet because today unfortunately was a complex day, but will do tomorrow. Probably just need to figure a sane way to change where it looks for virtme-run
[1] https://github.com/p4lang/p4c/pull/5047/files#diff-76f06548e16be4f20ba423b3768e29bfca82a8ecd274407c02b37b3b5816f0c8L233

I finally found a way to fix the P4-TC tests, by using virtme-ng provided by the Debian repositories. Let me know whether that works for you.

I think I tried that, but it didn't work at the time (probably because the virtme-ng for the previous ubuntu version was too outdated). Glad to see it works now, makes things easier to setup.

@jafingerhut This is ready for review now.