Add e4s_testsuite tutorial section by wspear · Pull Request #1368 · buildtesters/buildtest

wspear · 2023-02-03T01:22:30Z

No description provided.

First draft for a larger commit. Do not merge yet.

shahzebsiddiqui · 2023-02-03T14:24:40Z

Please fix document ci check it's not building

wspear · 2023-02-03T16:01:18Z

Please fix document ci check it's not building
@shahzebsiddiqui I think we might have a chicken egg problem. I couldn't find any formatting errors reported in the readthedocs build but it looks like the data files I added aren't in the image readthedocs uses for its build yet.

/checkouts/1368/docs/buildspecs/e4s_testsuite.rst:36: WARNING: Include file '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/1368/examples/spack/e4s_testsuite_mpich.yml' not found or reading it failed
WARNING: Unexpected return code 1 from command Command(command=('cat', 'buildtest_tutorial_examples/spack/build/e4s_testuite_mpich.txt'), shell=False, hide_standard_error=False, working_directory='/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/1368/docs') (output='cat: buildtest_tutorial_examples/spack/build/e4s_testuite_mpich.txt: No such file or directory')
WARNING: Unexpected return code 1 from command Command(command=('cat', 'buildtest_tutorial_examples/spack/inspect/e4s_testuite_mpich.yml'), shell=False, hide_standard_error=False, working_directory='/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/1368/docs') (output='cat: buildtest_tutorial_examples/spack/inspect/e4s_testuite_mpich.yml: No such file or directory')
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/1368/docs/buildspecs/spack.rst:6: WARNING: duplicate label buildtest_spack_integration, other instance in /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/1368/docs/buildspecs/e4s_testsuite.rst
looking for now-outdated files... none found

docs/buildspecs/e4s_testsuite.rst

shahzebsiddiqui · 2023-02-03T16:24:14Z

@jscook2345 can you please help review Wyatt PR, adding additional eyes can help

shahzebsiddiqui · 2023-02-03T16:25:50Z

Please fix document ci check it's not building
@shahzebsiddiqui I think we might have a chicken egg problem. I couldn't find any formatting errors reported in the readthedocs build but it looks like the data files I added aren't in the image readthedocs uses for its build yet.

/checkouts/1368/docs/buildspecs/e4s_testsuite.rst:36: WARNING: Include file '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/1368/examples/spack/e4s_testsuite_mpich.yml' not found or reading it failed
WARNING: Unexpected return code 1 from command Command(command=('cat', 'buildtest_tutorial_examples/spack/build/e4s_testuite_mpich.txt'), shell=False, hide_standard_error=False, working_directory='/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/1368/docs') (output='cat: buildtest_tutorial_examples/spack/build/e4s_testuite_mpich.txt: No such file or directory')
WARNING: Unexpected return code 1 from command Command(command=('cat', 'buildtest_tutorial_examples/spack/inspect/e4s_testuite_mpich.yml'), shell=False, hide_standard_error=False, working_directory='/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/1368/docs') (output='cat: buildtest_tutorial_examples/spack/inspect/e4s_testuite_mpich.yml: No such file or directory')
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/1368/docs/buildspecs/spack.rst:6: WARNING: duplicate label buildtest_spack_integration, other instance in /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/1368/docs/buildspecs/e4s_testsuite.rst
looking for now-outdated files... none found

Is the test getting a nonzero return code?

examples/spack/e4s_testsuite_mpich.yaml

shahzebsiddiqui · 2023-02-03T16:37:23Z

Can you add test to

buildtest/buildtest/tools/docs.py

Line 69 in 87cbe8c

commands_to_run = {

that way it's automatic generated as pose to copying the output.

Next in container run buildtest tutorial-examples see https://buildtest.readthedocs.io/en/devel/contributing/build_documentation.html#generating-documentation-examples-for-buildtest-tutorial

Note you need to volume mount buildtest into container it's provided in docs.

You will see lots of files change which is expected

codecov · 2023-02-03T19:33:46Z

Codecov Report

Base: 72.76% // Head: 69.46% // Decreases project coverage by -3.30% ⚠️

Coverage data is based on head (f544d4a) compared to base (8f0a9c8).
Patch coverage: 3.42% of modified lines in pull request are covered.

❗ Current head f544d4a differs from pull request most recent head 58a0fa6. Consider uploading reports for the commit 58a0fa6 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##            devel    #1368      +/-   ##
==========================================
- Coverage   72.76%   69.46%   -3.30%     
==========================================
  Files          57       57              
  Lines        6160     6277     +117     
  Branches     1066     1112      +46     
==========================================
- Hits         4482     4360     -122     
- Misses       1678     1915     +237     
- Partials        0        2       +2

Impacted Files	Coverage Δ
buildtest/builders/base.py	`55.75% <0.00%> (-1.00%)`	⬇️
buildtest/buildsystem/checks.py	`6.03% <3.68%> (-0.96%)`	⬇️
buildtest/cli/compilers.py	`27.59% <0.00%> (-46.80%)`	⬇️
buildtest/system.py	`54.50% <0.00%> (-15.50%)`	⬇️
buildtest/cli/cdash.py	`88.61% <0.00%> (-0.99%)`	⬇️
buildtest/config.py	`49.75% <0.00%> (-0.25%)`	⬇️
buildtest/executors/cobalt.py	`22.11% <0.00%> (-0.24%)`	⬇️
buildtest/executors/setup.py	`72.95% <0.00%> (+0.13%)`	⬆️
buildtest/cli/report.py	`90.59% <0.00%> (+0.27%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

wspear · 2023-02-04T00:56:34Z

The container example build worked for my new stuff. Here's the PR render for quick review: https://buildtest--1368.org.readthedocs.build/en/1368/buildspecs/e4s_testsuite.html

The compilers/steam_example_metrics.yml buildspec failed so I didn't recommit the other rebuilt example output. I didn't see why it was invalid at a glance and as far as I can tell the error didn't occur in the github build.

Valid Buildspecs: 0
Invalid Buildspecs: 1
/home/spack/buildtest/examples/compilers/stream_example_metrics.yml: INVALID

buildtest is unable to create any tests because there are no valid buildspecs.

I don't think the container build error came from this commit either:

#14 exporting to image
#14 pushing layers 0.6s done
#14 ERROR: failed to push ghcr.io/buildtesters/buildtest_spack:latest: unexpected status: 403 Forbidden
------
 > exporting to image:
------
ERROR: failed to solve: failed to push ghcr.io/buildtesters/buildtest_spack:latest: unexpected status: 403 Forbidden
Error: buildx failed with: ERROR: failed to solve: failed to push ghcr.io/buildtesters/buildtest_spack:latest: unexpected status: 403 Forbidden

shahzebsiddiqui · 2023-02-06T15:48:46Z

ok @wspear thanks for headsup, ill try to fix the error for compilers/stream_example_metrics.yml in a separate PR. Yes you dont need to commit all the example diff since it is going to be a really big commit diff. What you did was correct, only commit the changes relevant to e4s testsuite

docs/buildspecs/e4s_testsuite.rst

shahzebsiddiqui · 2023-02-06T15:54:53Z

docs/buildspecs/e4s_testsuite.rst

+E4S is a spack based software stack but because not all spack packages provide internal testing functionality it 
+is useful to have an alternative framework for testing deployed software.


i think this sentence can be removed, we don't need to explain to user why we need an alternative framework for testing spack packages.

Actually, this question comes up a lot in discussions about Spack Tests/E4S Testsuite.

docs/buildspecs/e4s_testsuite.rst

shahzebsiddiqui · 2023-02-06T19:44:03Z

Congratulations @wspear on your first contribution 👏

wspear added 7 commits February 2, 2023 15:56

Create e4s_testsuite.rst

7756ca6

First draft for a larger commit. Do not merge yet.

Create e4s_testsuite_mpich.yaml

d37c6a4

Update e4s_testsuite_mpich.yaml

5d8cbed

Create e4s_testsuite_mpich.txt

abe926e

Create e4s_testsuite_mpich.txt

457779c

Update e4s_testsuite.rst

c24b77d

Update buildspec_tutorial.rst

b5bd4be

pull-request-size bot added the size/L label Feb 3, 2023

shahzebsiddiqui reviewed Feb 3, 2023

View reviewed changes

docs/buildspecs/e4s_testsuite.rst Outdated Show resolved Hide resolved

shahzebsiddiqui reviewed Feb 3, 2023

View reviewed changes

docs/buildspecs/e4s_testsuite.rst Outdated Show resolved Hide resolved

shahzebsiddiqui reviewed Feb 3, 2023

View reviewed changes

docs/buildspecs/e4s_testsuite.rst Outdated Show resolved Hide resolved

shahzebsiddiqui requested a review from jscook2345 February 3, 2023 16:22

shahzebsiddiqui assigned wspear Feb 3, 2023

shahzebsiddiqui reviewed Feb 3, 2023

View reviewed changes

examples/spack/e4s_testsuite_mpich.yaml Outdated Show resolved Hide resolved

shahzebsiddiqui reviewed Feb 3, 2023

View reviewed changes

examples/spack/e4s_testsuite_mpich.yaml Outdated Show resolved Hide resolved

shahzebsiddiqui reviewed Feb 3, 2023

View reviewed changes

examples/spack/e4s_testsuite_mpich.yaml Outdated Show resolved Hide resolved

wspear added 4 commits February 3, 2023 11:14

Address changes requested in the PR

9635f8c

Remove comments from buildspec

75936c6

Fix path suffix

8d4e770

Run e4s testsuite tests as part of doc gen

01e9e02

Fixed typo. Rebuilt example output.

5b0a9c7

shahzebsiddiqui reviewed Feb 6, 2023

View reviewed changes