Test #690 unit tests #691

MichaelStritt · 2021-07-01T15:09:35Z

Linked issue

Check out #690

Comments

Added some small unit tests.

MichaelStritt · 2021-07-05T14:42:18Z

QC Meeting (Henk & Michael)

standardize unit test names 👍 implemented in 37f8a83
get test name from script name 👍 implemented in 37f8a83
get test file type form script name as well 👍 implemented in 37f8a83
the image processing ones would be the most interesting ones, like xASL_spm_reslice e.g., maybe we can define test scenarios and requirements together

MichaelStritt · 2021-07-05T16:44:13Z

Quick test run

====================================== TEST RESULTS ======================================

                        name                           unit          tests        passed
    ____________________________________________    __________    ____________    ______

    'xASL_adm_CatchNumbersFromString'               'function'    [1×3 struct]    true  
    'xASL_adm_CheckFileCount'                       'function'    [1×2 struct]    true  
    'xASL_adm_CompareLists'                         'function'    [1×2 struct]    true  
    'xASL_adm_CorrectName'                          'function'    [1×2 struct]    true  
    'xASL_adm_OrderFields'                          'function'    [1×1 struct]    true  
    'xASL_bids_BIDS2Legacy'                         'function'    [1×2 struct]    true  
    'xASL_bids_Config'                              'function'    [1×1 struct]    true  
    'xASL_bids_CreateDatasetDescriptionTemplate'    'function'    [1×2 struct]    true  
    'xASL_bids_JsonCheck'                           'function'    [1×3 struct]    true  
    'xASL_bids_VendorFieldCheck'                    'function'    [1×1 struct]    true  
    'xASL_io_ExportVTK'                             'function'    [1×5 struct]    true  
    'xASL_io_Nifti2Im'                              'function'    [1×1 struct]    true  
    'xASL_stat_PSNR'                                'function'    [1×2 struct]    true  
    'xASL_stat_QuantileNan'                         'function'    [1×1 struct]    true  
    'xASL_stat_StdNan'                              'function'    [1×1 struct]    true  
    'xASL_str2num'                                  'function'    [1×2 struct]    true  
    'xASL_test_GetLogContent'                       'function'    [1×2 struct]    true  
    'xASL_tsvRead'                                  'function'    [1×2 struct]    true  
    'xASL_tsvWrite'                                 'function'    [1×2 struct]    true  

==========================================================================================

MichaelStritt · 2021-07-06T09:13:07Z

Test AMC (for backwards compatibility #574 )

HenkMutsaerts

Very nice work, few minor suggestions

HenkMutsaerts · 2021-07-11T15:06:23Z

Testing/UnitTests/xASL_ut_function_xASL_adm_CheckFileCount.m


+try
+    fclose('all');
+catch


also print message if you catch error

I just removed it, this shouldn't fail anyway. Fix in bfbadb7

Testing/UnitTests/xASL_ut_function_xASL_bids_BIDS2Legacy.m

HenkMutsaerts · 2021-07-11T15:09:45Z

Testing/UnitTests/xASL_ut_function_xASL_adm_CheckFileCount.m

-fclose(fid);
-fid = fopen(fullfile(TestRepository,'UnitTesting','working_directory','test_3.txt'), 'wt');
-fclose(fid);
+fid1 = fopen(fullfile(TestRepository,'UnitTesting','working_directory','test_1.txt'), 'wt');


Please add the function name in the commit title ;)
Weird why this doesn't work. In Linux there is a much easier way to create an empty file, which is called "touching" a file": system('touch test_1.txt');

Yeah, I always try to avoid system statements wherever possible, because then we have to take these Windows/Unix differences into account.

HenkMutsaerts · 2021-07-11T15:10:36Z

Testing/xASL_test_UnitTesting.m

    clc
-    fprintf('================================= TEST RESULTS =================================\n\n')
+    fprintf('====================================== TEST RESULTS ======================================\n\n')
    disp(UnitTestsTable);


Replace disp or display by fprintf('%s\n', MessageToPrint);

But the UnitTestsTable isn't a string, it's a table. You can't fprintf a table.

HenkMutsaerts · 2021-07-11T15:17:23Z

Testing/UnitTests/xASL_ut_UnitTest_function_StdNan.m

+
+% Run your test here
+y1 = xASL_stat_StdNan([0, 0, 0, 0, 0]);
+y2 = xASL_stat_StdNan([0, 1, 2, 3, 4]);


@MichaelStritt
I know that we are not ready to sweep across a number of parameters, but this would be the desired unit test, here we only test a limited amount of values. What do you think of creating a function xASL_ut_CreateInputValues, where you provide the upper and lower limit as well as the "resolution/number bins" of the values to test. So e.g., upper limit of 1,000,000 and lower limit of -1,000,000 with resolution 10 would create as input parameters for the unit test:

>> [-1000000:2000000/10:1000000] ans = Columns 1 through 3 -1000000 -800000 -600000 Columns 4 through 6 -400000 -200000 0 Columns 7 through 9 200000 400000 600000 Columns 10 through 11 800000 1000000

Also, we now do this unit test by assuming that another unit (==Matlab's std) is perfect. Whereas, this unit xASL_stat_StdNaN probably uses Matlab's std internally :) And whereas Matlab's std may differ between versions (theoretically)

Also, we now do this unit test by assuming that another unit (==Matlab's std) is perfect

I don't understand what you would do differently. Do you want to open an old math book and use reference values?

And whereas Matlab's std may differ between versions (theoretically)

Same here. Matlab is tested and approved software. If you don't trust Matlab to this level, you shouldn't use it at all.

I know that we are not ready to sweep across a number of parameters, but this would be the desired unit test, here we only test a limited amount of values.

Check out my comments for xASL_ut_UnitTest_function_str2num. We can implement this, but I don't think it's needed right now. Let's start as simple as possible. We should focus on testing more scripts and not on testing the few ones more thoroughly.

HenkMutsaerts · 2021-07-11T15:18:43Z

Testing/UnitTests/xASL_ut_UnitTest_function_str2num.m

+% Run your test here
+numOutA = xASL_str2num('123');
+numOutB = xASL_str2num('-123');
+numOutC = xASL_str2num('123.456');


Same suggestion here as for the other unit test below, why do you test 123456, why not 1234567890? But that's just my theoretical thinking probably :)

From the scientific point of view this is definitely a valid argument, but on the other hand we can't test for every integer/double value. In my opinion testing with one positive integer, one negative interger and a double should be enough. We can write something like...

index = 1; for iTest = -1000:+1000 testResult(index) = xASL_str2num(num2str(iTest)); index = index +1; end

... though.

I don't expect relevant testing results here though. What could be more interesting is to test the cases where we have problems regarding rounding a bit better. So tests like:

numOutX = xASL_str2num('0.000000000001'); numOutY = xASL_str2num('-0.000000000001');

jan-petr

Really great. We should start making stats on how many errors the UT discovered :)

I made some minor comments on what unusual cases to test - it's usually these that crash.

jan-petr · 2021-07-14T09:24:25Z

Testing/UnitTests/xASL_ut_function_xASL_adm_CorrectName.m

+testTime = tic;
+
+% Run your test here
+strOut = xASL_adm_CorrectName('abc$%&def');


you want to have some backslashes and umlauts - these were a horror

jan-petr · 2021-07-14T09:27:42Z

Testing/UnitTests/xASL_ut_function_xASL_stat_StdNan.m

+testTime = tic;
+
+% Run your test here
+y1 = xASL_stat_StdNan([0, 0, 0, 0, 0]);


you want to test also 3D or 4D files...

jan-petr · 2021-07-14T09:29:19Z

Testing/UnitTests/xASL_ut_function_xASL_str2num.m

+testTime = tic;
+
+% Run your test here
+numOutA = xASL_str2num('123');


also stuff like 10e6 , 5.3e-6, .12, and some nonsense stuff - to see if it doesn't crash if a non-number is provided.

MichaelStritt · 2021-07-14T14:42:22Z

@HenkMutsaerts: I added everything you and @jan-petr asked for. If I'm not completely wrong, then the accuracy of xASL_stat_StdNan is really quite low. Especially if we look at the third unit test (Multidimensional test with NaN values (2D)). Can you re-approve so that we can merge the current state? Would be easier to add unit tests step by step.

HenkMutsaerts

Looks good.

The std inaccuracy could be explained by the way of taking the mean difference. They should both do a parametric mean difference (i.e., "standard" deviation) as opposed to a non-parametric one (i.e., mean absolute difference). But one thing that we should check, is if they do sum(diff)/n or sum(diff)/(n-1). Matlab's std probably does the latter by default, which accounts for the loss of one degree of freedom (from calculating the mean diff), but I usually find sum(diff)/n easier to interpret.

I like these unit tests a lot. Would be good to still think of ways to generalize them.

Minor fix for xASL_stat_QuantileNan, fix header of StdNan unit test

Added module tests for Structural, ASL, and Population. Renamed ExploreASL_Master unit test. Added TSV export functionality to unit testing master script.

…ment

MichaelStritt added the testing Either unit testing or full pipeline testing for bugs label Jul 1, 2021

MichaelStritt added this to the Release 1.8.0 milestone Jul 1, 2021

MichaelStritt requested a review from HenkMutsaerts July 1, 2021 15:09

MichaelStritt self-assigned this Jul 1, 2021

MichaelStritt linked an issue Jul 1, 2021 that may be closed by this pull request

Add new unit tests #690

Closed

MichaelStritt requested a review from jan-petr July 2, 2021 08:17

HenkMutsaerts requested changes Jul 11, 2021

View reviewed changes

MichaelStritt requested a review from HenkMutsaerts July 12, 2021 12:14

jan-petr approved these changes Jul 14, 2021

View reviewed changes

HenkMutsaerts approved these changes Jul 15, 2021

View reviewed changes

MichaelStritt added 16 commits July 15, 2021 12:46

#690 xASL_adm_CatchNumbersFromString: Add unit test

d6c1fe8

#690 xASL_adm_CheckFileCount: Add unit test

908e4c7

#690 xASL_adm_CompareLists: Add unit test

9ae9742

#690 xASL_adm_CorrectName: Add unit test

29a1acd

#690 xASL_adm_OrderFields: Add unit test

3e54935

#690 xASL_bids_Config: Add unit test

677ed0e

#690 xASL_bids_CreateDatasetDescriptionTemplate: Add unit test

be2af31

#690 xASL_bids_JsonCheck: Add unit test

c043b9d

#690 xASL_bids_VendorFieldCheck: Add unit test

26911a9

#690 Update headers

8b54de8

#690 xASL_str2num: Add unit test

1d934c3

#690 xASL_str2num: Super minor fix

4c3ff2b

#690 xASL_stat_StdNan: Add unit test

e09588c

#690 xASL_stat_QuantileNan: Add unit test

535cab3

Minor fix for xASL_stat_QuantileNan, fix header of StdNan unit test

#690 xASL_stat_PSNR: Add unit tests

8c89739

#690 Update module tests, add new export functionality

4d8c85b

Added module tests for Structural, ASL, and Population. Renamed ExploreASL_Master unit test. Added TSV export functionality to unit testing master script.

MichaelStritt added 12 commits July 15, 2021 12:46

#690 Automatic unit test name and type functionality

bcec16c

#690 Reorder Unit Tests structure and improve output

084add2

#690 Minor fix: closing file handles did not work on linux

30e5a86

#690 Minor fix: backwards compatibility

c9a5839

#690 Minor fix: backwards compatibility

2c5e00a

#690 Minor fix: new xasl format

52d256c

#690 Minor fix: Permission error on windows

8077661

#690 xASL_ut_function_xASL_adm_CheckFileCount: Remove try-catch state…

890f218

…ment

#690 xASL_adm_CorrectName: Add umlaut and slash test

ae9b42c

#690 xASL_adm_stat_StdNan: Add multi dim tests

3631d0c

#690 xASL_str2num: Improve unit test

691ddf2

#690 Fix three unit tests

e431b9a

MichaelStritt force-pushed the test-#690_UnitTests branch from d30610e to e431b9a Compare July 15, 2021 10:47

MichaelStritt merged commit e431b9a into develop Jul 15, 2021

MichaelStritt deleted the test-#690_UnitTests branch July 15, 2021 10:47

Test #690 unit tests #691

Test #690 unit tests #691

Uh oh!

Conversation

MichaelStritt commented Jul 1, 2021

Linked issue

Comments

Uh oh!

MichaelStritt commented Jul 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QC Meeting (Henk & Michael)

Uh oh!

MichaelStritt commented Jul 5, 2021

Quick test run

Uh oh!

MichaelStritt commented Jul 6, 2021

Test AMC (for backwards compatibility #574 )

Uh oh!

HenkMutsaerts left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HenkMutsaerts Jul 11, 2021 • edited by MichaelStritt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jan-petr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichaelStritt commented Jul 14, 2021

Uh oh!

HenkMutsaerts left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MichaelStritt commented Jul 5, 2021 •

edited

Loading

HenkMutsaerts Jul 11, 2021 •

edited by MichaelStritt

Loading