Skip to content

ci: Enable junit xml reports#5190

Closed
dougch wants to merge 4 commits intoaws:mainfrom
dougch:nix_junit
Closed

ci: Enable junit xml reports#5190
dougch wants to merge 4 commits intoaws:mainfrom
dougch:nix_junit

Conversation

@dougch
Copy link
Copy Markdown
Contributor

@dougch dougch commented Mar 13, 2025 •
edited
Loading

Release Summary:

Resolved issues:

Partial #5097

Description of changes:

This enables ctest to emit junit reports. This requires ctest >=3.24, which is currently only available for CI in our nix environments and Ubuntu24.

In nix, if you just use unit or integ without specifying an individual test, it produces an aggregated report, e.g.: unit.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="Linux-gcc"
        tests="273"
        failures="0"
        disabled="0"
        skipped="0"
        hostname="devdesktop22"
        time="19"
        timestamp="2025-03-21T00:18:23"
        >
        <testcase name="s2n_3des_test" classname="s2n_3des_test" time="3.06185" status="run">
                <system-out>Running /home/dougch/gitrepos/s2n-tls/tests/unit/s2n_3des_test.c ... PASSED     153695 tests
            </system-out>
        </testcase>
        <testcase name="s2n_aead_aes_test" classname="s2n_aead_aes_test" time="12.7899" status="run">
                <system-out>Running /home/dougch/gitrepos/s2n-tls/tests/unit/s2n_aead_aes_test.c ... PASSED   32098262 tests
               </system-out>
        </testcase>
...

Sample integration test sni match:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="Linux-gcc"
        tests="1"
        failures="0"
        disabled="0"
        skipped="0"
        hostname="devdesktop22"
        time="4"
        timestamp="2025-03-21T00:19:22"
        >
        <testcase name="integrationv2_sni_match" classname="integrationv2_sni_match" time="4.39539" status="run">
                <system-out>============================= test session starts ==============================
platform linux -- Python 3.10.11, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/dougch/gitrepos/s2n-tls/tests/integrationv2
plugins: forked-1.4.0, html-3.1.1, metadata-2.0.2, xdist-2.5.0, rerunfailures-10.2
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I
[gw0] Python 3.10.11 (main, Apr  4 2023, 22:10:32) [GCC 11.3.0]
[gw1] Python 3.10.11 (main, Apr  4 2023, 22:10:32) [GCC 11.3.0]
[gw2] Python 3.10.11 (main, Apr  4 2023, 22:10:32) [GCC 11.3.0]
[gw3] Python 3.10.11 (main, Apr  4 2023, 22:10:32) [GCC 11.3.0]
[gw4] Python 3.10.11 (main, Apr  4 2023, 22:10:32) [GCC 11.3.0]
[gw5] Python 3.10.11 (main, Apr  4 2023, 22:10:32) [GCC 11.3.0]
[gw6] Python 3.10.11 (main, Apr  4 2023, 22:10:32) [GCC 11.3.0]
[gw7] Python 3.10.11 (main, Apr  4 2023, 22:10:32) [GCC 11.3.0]
gw0 [88] / gw1 [88] / gw2 [88] / gw3 [88] / gw4 [88] / gw5 [88] / gw6 [88] / gw7 [88]

scheduling tests via LoadScheduling

test_sni_match.py::test_sni_match[...
[This part of the test output was removed since it exceeds the threshold of 1024 bytes.]
</system-out>
        </testcase>
</testsuite>

A sample of junit reports for all the integration jobs are available on CF:

Call-outs:

Pytest can also generate junit reports, but the detail is... verbose. We can discuss offline, but I don't think we need every param passed to every test.

Testing:

How is this change tested (unit tests, fuzz tests, etc.)? locally

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@github-actions github-actions bot added the s2n-core team label Mar 13, 2025
@dougch dougch changed the title ci: Enable junit xml and html reports. ci: Enable junit xml reports. Mar 21, 2025
@dougch dougch requested a review from lrstewart March 27, 2025 20:06
@dougch dougch marked this pull request as ready for review March 27, 2025 20:07
@dougch dougch requested a review from jmayclin March 27, 2025 20:08
@lrstewart lrstewart changed the title ci: Enable junit xml reports. ci: Enable junit xml reports Mar 31, 2025
@dougch dougch marked this pull request as draft March 31, 2025 22:03
@dougch dougch removed request for jmayclin and lrstewart March 31, 2025 22:03
@dougch dougch closed this Apr 1, 2025
@dougch dougch mentioned this pull request Apr 1, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

s2n-core team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dougch