datapath/loader: Add new complexity tests for verifier#40367
Merged
dylandreimerink merged 4 commits intomainfrom Jul 16, 2025
Merged
datapath/loader: Add new complexity tests for verifier#40367dylandreimerink merged 4 commits intomainfrom
dylandreimerink merged 4 commits intomainfrom
Conversation
Member
Author
|
/ci-verifier |
7fe7bc7 to
b5287e4
Compare
Member
Author
|
/ci-verifier |
b5287e4 to
baab150
Compare
Member
Author
|
/ci-verifier |
baab150 to
040e8bb
Compare
Member
Author
|
/ci-verifier |
040e8bb to
7ab236b
Compare
Member
Author
|
/ci-verifier |
7ab236b to
16ade69
Compare
Member
Author
|
/ci-verifier |
16ade69 to
ed636f4
Compare
Member
Author
|
/ci-verifier |
ed636f4 to
3d910cf
Compare
Member
Author
|
/ci-verifier |
1 similar comment
Member
Author
|
/ci-verifier |
d50b767 to
d23eb05
Compare
Member
Author
|
/ci-verifier |
d23eb05 to
a1ab085
Compare
Member
Author
|
/ci-verifier |
a1ab085 to
326171e
Compare
Member
Author
|
/ci-verifier |
326171e to
c1d4d37
Compare
Member
Author
|
/ci-verifier |
There are some differences in the flags used by the makefile and the loader package when invoking the compiler. This commit ensures that the flags used by the makefile and the loader package match. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
This commit introduces a new test runner for the BPF verifier complexity tests. This new runner still uses the existing files to get all of the build permutations to be ran, in addition it also allows for multiple permutations of load-time config to be tried for each build permutation. The new runner also has a number of other improvements. First, we new reuse code from the loader to invoke the compiler in the exact same way we do in the cilium agent at runtime. Second, we now run the tests in parallel, all build permutations and load-time configurations are run in parallel, which should speed up the test suite significantly. Third, we tell the verifier to not spit out the entire verifier log, just the stats. This increases the speed of verification and lowers memory usage. When a load fails, we enable detailed verifier logs and re-attempt to gather the full log in such cases. On load failure, this full verifier log is emitted as test artifact along with the object file that failed to load. The `--full-log` flag can be used to force the full log to be emitted for all loads, even when successful. In case we want to do analysis of the successful logs. Last, we enable additional stats gathering in the verifier, giving us the stack depth (amount of bytes of stack used by a program) and verifier verification time in microseconds (more accurate than time the full syscall takes). All stats collected are always emitted in a test artifact as a JSON file, which should be easy to query with jq or similar tools. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
In v1.18 we changed the minimum supported kernel version to v5.10. For a while we still were testing RHEL 8.6 against v5.4, but that is no longer the case since #40390. So we can remove the compile permutations for v5.4. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
Now that we have a new complexity test runner that lives in the loader package we can delete the old one. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
c07ff12 to
48ce612
Compare
Member
Author
|
/test |
8 tasks
dylandreimerink
added a commit
that referenced
this pull request
Sep 26, 2025
Since #40367 the verifier complexity test has gained the ability to output JSON files with all the results. So far these were emitted as artifacts per kernel. This commit expends the JSON files by adding the kernel version which produced the results. This allows us to merge all results together into one big JSON file. The workflow is modified to download all separate artifacts and to merge them into one big JSON file which is then uploaded as an artifact. This will allow us to collect tests results from scheduled runs to track complexity over time. This commit also changes the workflow to run the complexity tests twice when triggered on a PR. The first run is on the base/merge branch and then on the HEAD of the PR branch. The merged results of both are then compared by a new tool `complexity-diff` which outputs the diff in a separate JSON file as well as outputting a markdown summary. The summary states the most significant changes in complexity and stack depth as result of the current PR. It also states the 15 largest programs in terms of complexity and stack depth. For every "program" the summary lists the permutation with the smallest number "min" and the largest number "max". Though results for all permutations are available in the JSON files. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
dylandreimerink
added a commit
that referenced
this pull request
Sep 26, 2025
Since #40367 the verifier complexity test has gained the ability to output JSON files with all the results. So far these were emitted as artifacts per kernel. This commit expends the JSON files by adding the kernel version which produced the results. This allows us to merge all results together into one big JSON file. The workflow is modified to download all separate artifacts and to merge them into one big JSON file which is then uploaded as an artifact. This will allow us to collect tests results from scheduled runs to track complexity over time. This commit also changes the workflow to run the complexity tests twice when triggered on a PR. The first run is on the base/merge branch and then on the HEAD of the PR branch. The merged results of both are then compared by a new tool `complexity-diff` which outputs the diff in a separate JSON file as well as outputting a markdown summary. The summary states the most significant changes in complexity and stack depth as result of the current PR. It also states the 15 largest programs in terms of complexity and stack depth. For every "program" the summary lists the permutation with the smallest number "min" and the largest number "max". Though results for all permutations are available in the JSON files. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
dylandreimerink
added a commit
that referenced
this pull request
Sep 26, 2025
Since #40367 the verifier complexity test has gained the ability to output JSON files with all the results. So far these were emitted as artifacts per kernel. This commit expends the JSON files by adding the kernel version which produced the results. This allows us to merge all results together into one big JSON file. The workflow is modified to download all separate artifacts and to merge them into one big JSON file which is then uploaded as an artifact. This will allow us to collect tests results from scheduled runs to track complexity over time. This commit also changes the workflow to run the complexity tests twice when triggered on a PR. The first run is on the base/merge branch and then on the HEAD of the PR branch. The merged results of both are then compared by a new tool `complexity-diff` which outputs the diff in a separate JSON file as well as outputting a markdown summary. The summary states the most significant changes in complexity and stack depth as result of the current PR. It also states the 15 largest programs in terms of complexity and stack depth. For every "program" the summary lists the permutation with the smallest number "min" and the largest number "max". Though results for all permutations are available in the JSON files. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
dylandreimerink
added a commit
that referenced
this pull request
Sep 29, 2025
Since #40367 the verifier complexity test has gained the ability to output JSON files with all the results. So far these were emitted as artifacts per kernel. This commit expends the JSON files by adding the kernel version which produced the results. This allows us to merge all results together into one big JSON file. The workflow is modified to download all separate artifacts and to merge them into one big JSON file which is then uploaded as an artifact. This will allow us to collect tests results from scheduled runs to track complexity over time. This commit also changes the workflow to run the complexity tests twice when triggered on a PR. The first run is on the base/merge branch and then on the HEAD of the PR branch. The merged results of both are then compared by a new tool `complexity-diff` which outputs the diff in a separate JSON file as well as outputting a markdown summary. The summary states the most significant changes in complexity and stack depth as result of the current PR. It also states the 15 largest programs in terms of complexity and stack depth. For every "program" the summary lists the permutation with the smallest number "min" and the largest number "max". Though results for all permutations are available in the JSON files. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
dylandreimerink
added a commit
that referenced
this pull request
Oct 1, 2025
Since #40367 the verifier complexity test has gained the ability to output JSON files with all the results. So far these were emitted as artifacts per kernel. This commit expends the JSON files by adding the kernel version which produced the results. This allows us to merge all results together into one big JSON file. The workflow is modified to download all separate artifacts and to merge them into one big JSON file which is then uploaded as an artifact. This will allow us to collect tests results from scheduled runs to track complexity over time. This commit also changes the workflow to run the complexity tests twice when triggered on a PR. The first run is on the base/merge branch and then on the HEAD of the PR branch. The merged results of both are then compared by a new tool `complexity-diff` which outputs the diff in a separate JSON file as well as outputting a markdown summary. The summary states the most significant changes in complexity and stack depth as result of the current PR. It also states the 15 largest programs in terms of complexity and stack depth. For every "program" the summary lists the permutation with the smallest number "min" and the largest number "max". Though results for all permutations are available in the JSON files. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
dylandreimerink
added a commit
that referenced
this pull request
Nov 19, 2025
Since #40367 the verifier complexity test has gained the ability to output JSON files with all the results. So far these were emitted as artifacts per kernel. This commit expends the JSON files by adding the kernel version which produced the results. This allows us to merge all results together into one big JSON file. The workflow is modified to download all separate artifacts and to merge them into one big JSON file which is then uploaded as an artifact. This will allow us to collect tests results from scheduled runs to track complexity over time. This commit also changes the workflow to run the complexity tests twice when triggered on a PR. The first run is on the base/merge branch and then on the HEAD of the PR branch. The merged results of both are then compared by a new tool `complexity-diff` which outputs the diff in a separate JSON file as well as outputting a markdown summary. The summary states the most significant changes in complexity and stack depth as result of the current PR. It also states the 15 largest programs in terms of complexity and stack depth. For every "program" the summary lists the permutation with the smallest number "min" and the largest number "max". Though results for all permutations are available in the JSON files. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
dylandreimerink
added a commit
that referenced
this pull request
Nov 19, 2025
Since #40367 the verifier complexity test has gained the ability to output JSON files with all the results. So far these were emitted as artifacts per kernel. This commit expends the JSON files by adding the kernel version which produced the results. This allows us to merge all results together into one big JSON file. The workflow is modified to download all separate artifacts and to merge them into one big JSON file which is then uploaded as an artifact. This will allow us to collect tests results from scheduled runs to track complexity over time. This commit also changes the workflow to run the complexity tests twice when triggered on a PR. The first run is on the base/merge branch and then on the HEAD of the PR branch. The merged results of both are then compared by a new tool `complexity-diff` which outputs the diff in a separate JSON file as well as outputting a markdown summary. The summary states the most significant changes in complexity and stack depth as result of the current PR. It also states the 15 largest programs in terms of complexity and stack depth. For every "program" the summary lists the permutation with the smallest number "min" and the largest number "max". Though results for all permutations are available in the JSON files. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
dylandreimerink
added a commit
that referenced
this pull request
Nov 19, 2025
Since #40367 the verifier complexity test has gained the ability to output JSON files with all the results. So far these were emitted as artifacts per kernel. This commit expends the JSON files by adding the kernel version which produced the results. This allows us to merge all results together into one big JSON file. The workflow is modified to download all separate artifacts and to merge them into one big JSON file which is then uploaded as an artifact. This will allow us to collect tests results from scheduled runs to track complexity over time. This commit also changes the workflow to run the complexity tests twice when triggered on a PR. The first run is on the base/merge branch and then on the HEAD of the PR branch. The merged results of both are then compared by a new tool `complexity-diff` which outputs the diff in a separate JSON file as well as outputting a markdown summary. The summary states the most significant changes in complexity and stack depth as result of the current PR. It also states the 15 largest programs in terms of complexity and stack depth. For every "program" the summary lists the permutation with the smallest number "min" and the largest number "max". Though results for all permutations are available in the JSON files. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
dylandreimerink
added a commit
that referenced
this pull request
Nov 21, 2025
Since #40367 the verifier complexity test has gained the ability to output JSON files with all the results. So far these were emitted as artifacts per kernel. This commit expends the JSON files by adding the kernel version which produced the results. This allows us to merge all results together into one big JSON file. The workflow is modified to download all separate artifacts and to merge them into one big JSON file which is then uploaded as an artifact. This will allow us to collect tests results from scheduled runs to track complexity over time. This commit also changes the workflow to run the complexity tests twice when triggered on a PR. The first run is on the base/merge branch and then on the HEAD of the PR branch. The merged results of both are then compared by a new tool `complexity-diff` which outputs the diff in a separate JSON file as well as outputting a markdown summary. The summary states the most significant changes in complexity and stack depth as result of the current PR. It also states the 15 largest programs in terms of complexity and stack depth. For every "program" the summary lists the permutation with the smallest number "min" and the largest number "max". Though results for all permutations are available in the JSON files. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Nov 24, 2025
Since #40367 the verifier complexity test has gained the ability to output JSON files with all the results. So far these were emitted as artifacts per kernel. This commit expends the JSON files by adding the kernel version which produced the results. This allows us to merge all results together into one big JSON file. The workflow is modified to download all separate artifacts and to merge them into one big JSON file which is then uploaded as an artifact. This will allow us to collect tests results from scheduled runs to track complexity over time. This commit also changes the workflow to run the complexity tests twice when triggered on a PR. The first run is on the base/merge branch and then on the HEAD of the PR branch. The merged results of both are then compared by a new tool `complexity-diff` which outputs the diff in a separate JSON file as well as outputting a markdown summary. The summary states the most significant changes in complexity and stack depth as result of the current PR. It also states the 15 largest programs in terms of complexity and stack depth. For every "program" the summary lists the permutation with the smallest number "min" and the largest number "max". Though results for all permutations are available in the JSON files. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces a new test runner for the BPF verifier complexity tests. This new runner still uses the existing files to get all of the build permutations to be ran, in addition it also allows for multiple permutations of load-time config to be tried for each build permutation.
The new runner also has a number of other improvements. First, we new reuse code from the loader to invoke the compiler in the exact same way we do in the cilium agent at runtime.
Second, we now run the tests in parallel, all build permutations and load-time configurations are run in parallel, which should speed up the test suite significantly.
Third, we tell the verifier to not spit out the entire verifier log, just the stats. This increases the speed of verification and lowers memory usage. When a load fails, we enable detailed verifier logs and re-attempt to gather the full log in such cases. On load failure, this full verifier log is emitted as test artifact along with the object file that failed to load. The
--full-logflag can be used to force the full log to be emitted for all loads, even when successful. In case we want to do analysis of the successful logs.Last, we enable additional stats gathering in the verifier, giving us the stack depth (amount of bytes of stack used by a program) and verifier verification time in microseconds (more accurate than time the full syscall takes). All stats collected are always emitted in a test artifact as a JSON file, which should be easy to query with jq or similar tools.
Fixes: #39143