add support for assert greater equal. Update metrics property by shahzebsiddiqui · Pull Request #1313 · buildtesters/buildtest

shahzebsiddiqui · 2022-12-16T17:39:21Z

Here is the stream benchmark take note of the output lines

stream_test/7305fc15: testing metric: copy if 7981.3 >= 5000.0
stream_test/7305fc15: testing metric: scale if 9484.8 >= 5000.0

(buildtest)  ~/Documents/github/buildtest/ [perf_status] buildtest bd -b tutorials/stream.yml
╭─────────────────────────────────────────── buildtest summary ───────────────────────────────────────────╮                                                                                                                                
│                                                                                                         │                                                                                                                                
│ User:               siddiq90                                                                            │                                                                                                                                
│ Hostname:           DOE-7086392.vpn-dhcp.lbl.gov                                                        │                                                                                                                                
│ Platform:           Darwin                                                                              │                                                                                                                                
│ Current Time:       2022/12/16 12:41:03                                                                 │                                                                                                                                
│ buildtest path:     /Users/siddiq90/Documents/github/buildtest/bin/buildtest                            │                                                                                                                                
│ buildtest version:  1.0                                                                                 │                                                                                                                                
│ python path:        /Users/siddiq90/.local/share/virtualenvs/buildtest-Ir4AdrfC/bin/python3             │                                                                                                                                
│ python version:     3.7.3                                                                               │                                                                                                                                
│ Configuration File: /Users/siddiq90/Documents/github/buildtest/buildtest/settings/config.yml            │                                                                                                                                
│ Test Directory:     /Users/siddiq90/Documents/github/buildtest/var/tests                                │                                                                                                                                
│ Report File:        /Users/siddiq90/Documents/github/buildtest/var/report.json                          │                                                                                                                                
│ Command:            /Users/siddiq90/Documents/github/buildtest/bin/buildtest bd -b tutorials/stream.yml │                                                                                                                                
│                                                                                                         │                                                                                                                                
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                                                                                
─────────────────────────────────────────────────────────────────────────────────────────────────────────  Discovering Buildspecs ─────────────────────────────────────────────────────────────────────────────────────────────────────────
                       Discovered buildspecs                       
╔═════════════════════════════════════════════════════════════════╗
║ buildspec                                                       ║
╟─────────────────────────────────────────────────────────────────╢
║ /Users/siddiq90/Documents/github/buildtest/tutorials/stream.yml ║
╚═════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
─────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/Users/siddiq90/Documents/github/buildtest/tutorials/stream.yml: VALID
Total builder objects created: 1
                                                                                               Builders by type=script                                                                                               
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder              ┃ type   ┃ executor           ┃ compiler ┃ nodes ┃ procs ┃ description                                                     ┃ buildspecs                                                      ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ stream_test/7305fc15 │ script │ generic.local.bash │ None     │ None  │ None  │ Run stream test with metrics example using assert greater equal │ /Users/siddiq90/Documents/github/buildtest/tutorials/stream.yml │
└──────────────────────┴────────┴────────────────────┴──────────┴───────┴───────┴─────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
stream_test/7305fc15: Creating test directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/stream/stream_test/7305fc15
stream_test/7305fc15: Creating the stage directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/stream/stream_test/7305fc15/stage
stream_test/7305fc15: Writing build script: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/stream/stream_test/7305fc15/stream_test_build.sh
────────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 1 processes for processing builders
─────────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────
stream_test/7305fc15 does not have any dependencies adding test to queue
In this iteration we are going to run the following tests: [stream_test/7305fc15]
stream_test/7305fc15: Running Test via command: bash --norc --noprofile -eo pipefail stream_test_build.sh
stream_test/7305fc15: Test completed in 2.67788 seconds
stream_test/7305fc15: Test completed with returncode: 0
stream_test/7305fc15: Writing output file -  /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/stream/stream_test/7305fc15/stream_test.out
stream_test/7305fc15: Writing error file - /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/stream/stream_test/7305fc15/stream_test.err
stream_test/7305fc15: testing metric: copy if 7981.3 >= 5000.0
stream_test/7305fc15: testing metric: scale if 9484.8 >= 5000.0
                                                   Test Summary                                                    
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder              ┃ executor           ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ stream_test/7305fc15 │ generic.local.bash │ PASS   │ False False False                   │ 0          │ 2.67788 │
└──────────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /Users/siddiq90/Documents/github/buildtest/var/report.json
Writing Logfile to: /Users/siddiq90/Documents/github/buildtest/var/logs/buildtest_jfnd4nfo.log

The metrics are captured based on regular expession and they are converted to int, float, or str. In this example we are capturing the 1st item from the list

(buildtest)  ~/Documents/github/buildtest/ [perf_status] buildtest inspect query -o stream_test 
──────────────────────────────────────────────────────────────────────────────────────────── stream_test/7305fc15-dceb-4c4e-b094-fadf4ed6a7b9 ─────────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Run stream test with metrics example using assert greater equal
State: PASS
Returncode: 0
Runtime: 2.67788 sec
Starttime: 2022/12/16 12:41:03
Endtime: 2022/12/16 12:41:06
Command: bash --norc --noprofile -eo pipefail stream_test_build.sh
Test Script: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/stream/stream_test/7305fc15/stream_test.sh
Build Script: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/stream/stream_test/7305fc15/stream_test_build.sh
Output File: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/stream/stream_test/7305fc15/stream_test.out
Error File: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/stream/stream_test/7305fc15/stream_test.err
Log File: /Users/siddiq90/Documents/github/buildtest/var/logs/buildtest_jfnd4nfo.log
     Metrics      
┏━━━━━━━┳━━━━━━━━┓
┃ Name  ┃ Value  ┃
┡━━━━━━━╇━━━━━━━━┩
│ copy  │ 7981.3 │
│ scale │ 9484.8 │
└───────┴────────┘
──────────────────────────────────────────────────── Output File: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/stream/stream_test/7305fc15/stream_test.out ─────────────────────────────────────────────────────
-------------------------------------------------------------                                                                                                                                                                              
STREAM version $Revision: 5.10 $                                                                                                                                                                                                           
-------------------------------------------------------------                                                                                                                                                                              
This system uses 8 bytes per array element.                                                                                                                                                                                                
-------------------------------------------------------------                                                                                                                                                                              
Array size = 10000000 (elements), Offset = 0 (elements)                                                                                                                                                                                    
Memory per array = 76.3 MiB (= 0.1 GiB).                                                                                                                                                                                                   
Total memory required = 228.9 MiB (= 0.2 GiB).                                                                                                                                                                                             
Each kernel will be executed 10 times.                                                                                                                                                                                                     
 The *best* time for each kernel (excluding the first iteration)                                                                                                                                                                           
 will be used to compute the reported bandwidth.                                                                                                                                                                                           
-------------------------------------------------------------                                                                                                                                                                              
Your clock granularity/precision appears to be 1 microseconds.                                                                                                                                                                             
Each test below will take on the order of 25614 microseconds.                                                                                                                                                                              
   (= 25614 clock ticks)                                                                                                                                                                                                                   
Increase the size of the arrays if this shows that                                                                                                                                                                                         
you are not getting at least 20 clock ticks per test.                                                                                                                                                                                      
-------------------------------------------------------------                                                                                                                                                                              
WARNING -- The above is only a rough guideline.                                                                                                                                                                                            
For best results, please be sure you know the                                                                                                                                                                                              
precision of your system timer.                                                                                                                                                                                                            
-------------------------------------------------------------                                                                                                                                                                              
Function    Best Rate MB/s  Avg time     Min time     Max time                                                                                                                                                                             
Copy:            7981.3     0.020670     0.020047     0.022545                                                                                                                                                                             
Scale:           9484.8     0.018009     0.016869     0.021216                                                                                                                                                                             
Add:            10859.3     0.022774     0.022101     0.024963                                                                                                                                                                             
Triad:          11370.7     0.022250     0.021107     0.024320                                                                                                                                                                             
-------------------------------------------------------------                                                                                                                                                                              
Solution Validates: avg error less than 1.000000e-13 on all three arrays                                                                                                                                                                   
-------------------------------------------------------------

…perform assert greater equal using 'metrics' to capture key values from the test output. A new key 'assert_ge' was added that will be added to 'status' check that can be used to determine how test passes.

codecov · 2022-12-16T17:44:14Z

Codecov Report

Base: 73.22% // Head: 72.64% // Decreases project coverage by -0.58% ⚠️

Coverage data is based on head (eb7f39a) compared to base (20c9354).
Patch coverage: 1.96% of modified lines in pull request are covered.

❗ Current head eb7f39a differs from pull request most recent head 904b8f5. Consider uploading reports for the commit 904b8f5 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##            devel    #1313      +/-   ##
==========================================
- Coverage   73.22%   72.64%   -0.58%     
==========================================
  Files          56       56              
  Lines        5923     5972      +49     
  Branches     1051     1058       +7     
==========================================
+ Hits         4337     4338       +1     
- Misses       1584     1632      +48     
  Partials        2        2

Impacted Files	Coverage Δ
buildtest/builders/base.py	`51.15% <1.96%> (-4.66%)`	⬇️
buildtest/tools/stylecheck.py	`89.83% <0.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

update stream test to incorporate two more metrics 'add' and 'triad' fix exception issue raised during testing where we got undefined variable reference

update the item property with minimum value of 0

add buildspec for stream benchmark test that will demonstrate how to …

4c91345

…perform assert greater equal using 'metrics' to capture key values from the test output. A new key 'assert_ge' was added that will be added to 'status' check that can be used to determine how test passes.

pull-request-size bot added the size/L label Dec 16, 2022

shahzebsiddiqui added 4 commits December 16, 2022 17:05

add documentation on assert greater equal

009417f

update stream test to incorporate two more metrics 'add' and 'triad' fix exception issue raised during testing where we got undefined variable reference

add regression test for testing assert_ge

be51d05

update documentation.

eb7f39a

update the item property with minimum value of 0

slight changes to documentation

904b8f5

shahzebsiddiqui merged commit 07a7132 into devel Dec 19, 2022

shahzebsiddiqui deleted the perf_status branch December 19, 2022 17:14

shahzebsiddiqui mentioned this pull request Dec 19, 2022

Status Check for Assert Less Than Equal - assert_le #1314

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add support for assert greater equal. Update metrics property#1313

add support for assert greater equal. Update metrics property#1313
shahzebsiddiqui merged 5 commits intodevelfrom
perf_status

shahzebsiddiqui commented Dec 16, 2022 •

edited

Loading

Uh oh!

codecov bot commented Dec 16, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

shahzebsiddiqui commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

shahzebsiddiqui commented Dec 16, 2022 •

edited

Loading

codecov bot commented Dec 16, 2022 •

edited

Loading