Release 1.8.0 #816

MichaelStritt · 2021-09-04T15:27:30Z

Linked issue

Release PR, related issue #805.

How to test

Check out testing descriptions in #805.

Comments

Starting to test now, discussion (& release) on monday (if possible).

The major problem was that xASL_adm_LoadParms just dumps all fields of the JSON into the x struct. This is not really helpful and maybe we should do this in a more controlled way in a future release. For now it should mostly be fixed. We use a standardized script called xASL_io_CheckDeprecatedFieldsX for this now. We use the same script for xASL_io_ReadDataPar and we could also integrate it in the LoadX function, we just have to use OldX.x for x there I think.

…behavior

…able

… table generation A single missing backslash t or missing comma within a TSV or CSV file should not break the pipeline before writing the results to a mat file

…reASL

…te instead

I encountered some bugs where the TSV reading did not work if we use xASL_tsvWrite with empty strings within the cell array. We should probably fix it there as well. For know, I fixed this in xASL_stat_PrintStat and xASL_stat_GetDICOMStatistics. I think the multiple subjects/sessions should work now, too. I will run the ten test datasets and then we can see if everything is correct now.

I am a bit surprised how this did work before. For me this breaks the code. It is not possible to assign a character array to a field of a numeric array. This would work for cell array though.

MichaelStritt · 2021-09-09T07:27:39Z

Test results from before `084349b`, `86591ae` & `14913c3`

Everything fine here...

====================================== TEST RESULTS ======================================

                        name                           unit           tests        passed
    ____________________________________________    __________    _____________    ______

    'xASL_adm_CatchNumbersFromString'               'function'    [1×3  struct]    true  
    'xASL_adm_CheckFileCount'                       'function'    [1×2  struct]    true  
    'xASL_adm_CompareLists'                         'function'    [1×2  struct]    true  
    'xASL_adm_CorrectName'                          'function'    [1×3  struct]    false 
    'xASL_adm_DeleteFileList'                       'function'    [1×2  struct]    true  
    'xASL_adm_OrderFields'                          'function'    [1×1  struct]    true  
    'xASL_bids_BIDS2Legacy'                         'function'    [1×2  struct]    true  
    'xASL_bids_CheckDatasetDescription'             'function'    [1×2  struct]    true  
    'xASL_bids_Config'                              'function'    [1×1  struct]    true  
    'xASL_bids_CreateDatasetDescriptionTemplate'    'function'    [1×2  struct]    true  
    'xASL_bids_JsonCheck'                           'function'    [1×3  struct]    true  
    'xASL_bids_VendorFieldCheck'                    'function'    [1×1  struct]    true  
    'xASL_io_ExportVTK'                             'function'    [1×5  struct]    true  
    'xASL_io_Nifti2Im'                              'function'    [1×1  struct]    true  
    'xASL_stat_PSNR'                                'function'    [1×2  struct]    true  
    'xASL_stat_QuantileNan'                         'function'    [1×1  struct]    true  
    'xASL_stat_StdNan'                              'function'    [1×3  struct]    true  
    'xASL_str2num'                                  'function'    [1×3  struct]    true  
    'xASL_test_GetLogContent'                       'function'    [1×2  struct]    true  
    'xASL_tsvRead'                                  'function'    [1×2  struct]    true  
    'xASL_tsvWrite'                                 'function'    [1×2  struct]    true  
    'xASL_ExploreASL_Master'                        'master'      [1×11 struct]    true  
    'xASL_module_ASL'                               'module'      [1×1  struct]    true  
    'xASL_module_Population'                        'module'      [1×1  struct]    true  
    'xASL_module_Structural'                        'module'      [1×1  struct]    true  

==========================================================================================

MichaelStritt · 2021-09-09T07:43:18Z

@HenkMutsaerts: let's discuss this here, because whatsapp isn't really that good for this kind of stuff

084349b and 14913c3 are okay, but 86591ae definitely isn't. I'm just saying that we should change...

    if bUseBasilQuantification
        % Here we perform FSL BASIL
        [PWI, resultFSL] = xASL_quant_Basil(PWI, x);
    end

...back to...

    if bUseBasilQuantification
        % Here we perform FSL BASIL
        [PWI, resultFSL] = xASL_quant_Basil(PWI, x);
    else
        resultFSL = NaN;
    end

I can't use the windows machine to test everything again today and I'm not in the office tomorrow, but somebody can test everything on linux and then it should basically be fine, because we had multiple testing rounds already. Agree @jan-petr & @HenkMutsaerts ?

Edit: The optimal solution IMHO would be to use the check FSL script before quant BASIL and then skip this part. This could be a nice follow-up issue for 1.9.0. I don't care that much about all of this, it was just super non-transparent before and hard for me to find out what was going wrong without checking each and every log file (if you don't have FSL installed).

MichaelStritt · 2021-09-09T07:51:04Z

@MichaelStritt can you please undo this, I already commented before that xASL_adm_CreateDir automatically checks if it has to create a folder yes or no right?

But I don't want to have an arbitrary warning (folder exists already) at this point. I want to warn the user that this specific BIDS subject exists already. There's a difference there.

AND

I know you want to do this in more detail and discuss it and so on, but it is helpful to have it in 1.8.0 and it was tested a lot. That's just not moving in the right direction and I don't have time for this kind of stuff.

Edit: Just realized that this doesn't really work as intended for multi-subject, multi-session datasets. I added a fix below. This way we know where to start for the next release and this helps with debugging.

Henk asked for this a while ago, since we have to think this through. The previous version did obviously not work for multi-subject multi-session datasets. I removed the return statement and added another print statement. Please verify that my changes do not break anything.

jan-petr · 2021-09-09T12:58:08Z

One more try during the day ;)

Philips_2DEPI_FLAIR_DisabledQuantification_M0_LoQ

Yes, the problem was probably that the quantification wasn't skipped. If we skip it, we obtain similar numbers. Managed to reproduce the old 1.7.0 with 1.8.0. We are thus fine with the 2-fold increase in CBF.

The other two are probably also fine - one was caused by the M0Processing and the other by the other parameters. But I guess we can fix the values now. But we should then save the 1.8.0 values as reference for both Windows and Linux.

MichaelStritt · 2021-09-09T13:11:46Z

One more try during the day ;)

Philips_2DEPI_FLAIR_DisabledQuantification_M0_LoQ

Yes, the problem was probably that the quantification wasn't skipped. If we skip it, we obtain similar numbers. Managed to reproduce the old 1.7.0 with 1.8.0. We are thus fine with the 2-fold increase in CBF.

The other two are probably also fine - one was caused by the M0Processing and the other by the other parameters. But I guess we can fix the values now. But we should then save the 1.8.0 values as reference for both Windows and Linux.

Sounds good. How do we handle the BASIL dataset?

Edit: discussed, should be fine

MichaelStritt · 2021-09-10T05:32:09Z

Windows test datasets

It seems that my changes for the BASIL quantification now crash the pipeline at a later point if you don't have FSL/BASIL installed. I'd be more interested if all of this works nicely on a system with FSL/BASIL though. @jan-petr & @HenkMutsaerts: does this need fixing?

Philips_2DEPI_Lesion_NoFLAIR_NoM0_HiQ

Quantification CBF using FSL Basil:
Note that any file not found warnings can be ignored, this pertains to the use of symbolic links by BASIL
Basil: CASL/PcASL model
Basil: Single-compartment (white paper mode)
Basil: Fixed bolus duration component
Basil: Single-delay data - cannot infer ATT or arterial component
Basil: Fixed arterial arrival time
Detected windows PC without WSL, needs to be installed first if you want to use FSL
This warning can be ignored if you dont need to run FSL-specific processing, e.g. TopUp

Warning: No FSL installation found, skipping FSL function
     xASL_fsl_RunFSL, line 64
     xASL_quant_Basil, line 74

...

Using single-compartment CASL
Using DICOM (re)scale slopes 2.7392 * 0.037159

ERROR: ASL module terminated for subject 1: Sub-0001
     Error using xASL_quant_SinglePLD line 241
     PWI dimensions too small compared to M0 and/or ScaleImage dimensions
     Error in xASL_wrp_Quantify, line 389

CONT: but continue with next iteration!

Philips_3DGRASE_FLAIR_M0_2ndRun_HiQ

Quantification CBF using FSL Basil:
Note that any file not found warnings can be ignored, this pertains to the use of symbolic links by BASIL
Basil: CASL/PcASL model
Basil: Single-compartment (white paper mode)
Basil: Fixed bolus duration component
Basil: Single-delay data - cannot infer ATT or arterial component
Basil: Fixed arterial arrival time
Detected windows PC without WSL, needs to be installed first if you want to use FSL
This warning can be ignored if you dont need to run FSL-specific processing, e.g. TopUp

Warning: No FSL installation found, skipping FSL function
     xASL_fsl_RunFSL, line 64

...

Using single-compartment CASL
=========================================== Scaling ==========================================
Using DICOM (re)scale slopes 1.3722 * 6.5894e-06

ERROR: ASL module terminated for subject 1: MCI-0013
    Error using xASL_quant_SinglePLD, line 241
    PWI dimensions too small compared to M0 and/or ScaleImage dimensions

CONT: but continue with next iteration!

Windows flavor tests

Conversion

The BIDS conversion never worked that good before 😉

>> xASL_test_Flavors(pathExploreASL, pathTest, [0 0 1 0 0 0 0], x);
Dataset: GE_PCASL_2DEPI_23.0LX_1
Dataset: GE_PCASL_3Dspiral_14.0LX_1
Dataset: GE_PCASL_3Dspiral_15.0LX_WIP_1
Dataset: GE_PCASL_3Dspiral_22.0LX_1
Dataset: GE_PCASL_3Dspiral_23.1LX_1
Dataset: GE_PCASL_3Dspiral_24.0LX_WIP_1
Dataset: GE_PCASL_3Dspiral_25.0LX_ADNI3_027_S_5079_1
Dataset: Philips_PASL_2DEPI_5.4.1.0_ADNI3_006_S_6681_1
Dataset: Philips_PCASL_2DEPI_3.2.1.1_1
Dataset: Philips_PCASL_2DEPI_3.2.1.1_2
Dataset: Philips_PCASL_2DEPI_3.2.1.1_3
Dataset: Philips_PCASL_2DEPI_3.2.1.1_dummyLL_1
Dataset: Philips_PCASL_2DEPI_3.2.1.1_dummyMultiPLD_1
Dataset: Philips_PCASL_2DEPI_3.2.1.1_dummyQUASAR_1
Dataset: Philips_PCASL_2DEPI_3.2.2.0_glioma_1
Dataset: Philips_PCASL_2DEPI_3.2.2.1_1
Dataset: Philips_PCASL_2DEPI_3.2.3.0_BSupN_1
Dataset: Philips_PCASL_2DEPI_3.2.3.1_1
Dataset: Philips_PCASL_2DEPI_3.2.3.2_1
Dataset: Philips_PCASL_2DEPI_4.1.2.1_1
Dataset: Philips_PCASL_2DEPI_5.1.7.2_1
Dataset: Philips_PCASL_2DEPI_5.1.7.2_2
Dataset: Philips_PCASL_2DEPI_5.1.8.2_1
Dataset: Philips_PCASL_2DEPI_5.1.9.1_GMB_1
Dataset: Philips_PCASL_2DEPI_5.3.1.1_1
Dataset: Philips_PCASL_2DEPI_PARREC_1
Dataset: Philips_PCASL_3DGRASE_5.1.7.2_1
Dataset: Philips_PCASL_3DGRASE_5.4.0.2_1
Dataset: Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1
Dataset: Siemens_PASL_2DEPI_VB15A_ADNI2_941_S_4764_1
Dataset: Siemens_PASL_2DEPI_VB17A_ADNI2_941_S_5193_1
Dataset: Siemens_PASL_2DEPI_VB17A_BSupN_1
Dataset: Siemens_PASL_2DEPI_VB17A_BSupN_2
Dataset: Siemens_PASL_2DEPI_VB19A_BSupN_1
Dataset: Siemens_PASL_3DGRASE_E11_1
Dataset: Siemens_PASL_3DGRASE_VB17A_1
Dataset: Siemens_PASL_3DGRASE_VD13D_1
Dataset: Siemens_PASL_3DGRASE_VE11C_1
Dataset: Siemens_PASL_3DGRASE_VE11C_2
Dataset: Siemens_PCASL_2DEPI_NA_1
Dataset: Siemens_PCASL_2DEPI_VB17A_BSupN_1
Dataset: Siemens_PCASL_2DEPI_VB17A_BSupN_2
Dataset: Siemens_PCASL_3DGRASE_VB17A_TopUp_1
Dataset: Siemens_PCASL_3DGRASE_VB17A_TopUp_2
Dataset: Siemens_PCASL_3DGRASE_VB17A_multiPLD_1
Dataset: Siemens_PCASL_3DGRASE_VD13A_Hadamard4-FME_1
Dataset: Siemens_PCASL_3DGRASE_VD13A_Hadamard8-FME_1
Dataset: Siemens_PCASL_3DGRASE_VD13D_1
Dataset: Siemens_PCASL_3DGRASE_VD13D_2
Dataset: Siemens_PCASL_3DGRASE_VE11C_1
Dataset: Siemens_PCASL_3DGRASE_VE11C_2

Processing

Philips_PASL_2DEPI_5.4.1.0_ADNI3_006_S_6681_1
- ASL module terminated for subject 1: sub-001 -> Reference to non-existent field 'LabelingDuration'.
Philips_PCASL_3DGRASE_5.1.7.2_1
- ASL module terminated for subject 1: sub-Patient1_1 -> Error xASL_module_ASL, line 220
Siemens_PASL_3DGRASE_E11_1
- ASL module terminated for subject 1: sub-Sub1_1 -> Error spm_realign>error_message, line 525
Siemens_PASL_3DGRASE_VE11C_2
- ASL module terminated for subject 1: sub-Sub1_1 -> Error xASL_quant_SinglePLD, line 51, xASL_wrp_Quantify, line 389
Siemens_PCASL_3DGRASE_VB17A_TopUp_1
- ASL module terminated for subject 1: sub-Sub1_1 -> Error xASL_spm_admin, line 29, xASL_spm_coreg, line 108
Siemens_PCASL_3DGRASE_VB17A_TopUp_2
- ASL module terminated for subject 1: sub-SUB002_1 -> Reference to non-existent field 'LabelingDuration'.
Siemens_PCASL_3DGRASE_VB17A_multiPLD_1
- ASL module terminated for subject 1: sub-Sub1_1 -> Error using zeros

HenkMutsaerts · 2021-09-10T07:39:54Z

You know that you should be able to copy paste the output you're showing in that Table to this Github issue? For me this always gives a very nice layout on Github automatically
Why do we have both Release 1.8.0 #805 and Release 1.8.0 #816? :)
I will test now, having FSL. I guess we should issue a proper warning if bUseBasilQuantification && FSL is not found, and then set bUseBasilQuantification to false. Then the pipeline doesn't crash if you request Basil but don't have it. We would have to test the same for TopUp (first step of the ASL module) where you also use FSL.
We discussed already with the FSL developers, best would be to include a light-weight version of FSL in ExploreASL

jan-petr · 2021-09-10T08:07:28Z

@MichaelStritt Yes, it definitively needs to be fixed. For once, it always does the normal quantification as well, even if BASIL is not installed, so it should never crash and always deliver results.

HenkMutsaerts · 2021-09-10T11:26:41Z

I get this error, any ideas?

ExploreASL will run the import workflow and will load the dataset...
==============================================================================================
 ________                      __                                 ______    ______   __        
/        |                    /  |                               /      \  /      \ /  |      
########/  __    __   ______  ## |  ______    ______    ______  /######  |/######  |## |      
## |__    /  \  /  | /      \ ## | /      \  /      \  /      \ ## |__## |## \__##/ ## |      
##    |   ##  \/##/ /######  |## |/######  |/######  |/######  |##    ## |##      \ ## |      
#####/     ##  ##<  ## |  ## |## |## |  ## |## |  ##/ ##    ## |######## | ######  |## |      
## |_____  /####  \ ## |__## |## |## \__## |## |      ########/ ## |  ## |/  \__## |## |_____ 
##       |/##/ ##  |##    ##/ ## |##    ##/ ## |      ##       |## |  ## |##    ##/ ##       |
########/ ##/   ##/ #######/  ##/  ######/  ##/        #######/ ##/   ##/  ######/  ########/ 
                    ## |                                                                      
                    ## |                                                                      
                    ##/  

==================================== ExploreASL Settings =====================================
Dataset Root        ...ers/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1
Import Modules      DCM2NII 
Process Modules     
bPause              False
iWorker             1
nWorkers            1
==============================================================================================
ExploreASL v1.8.0 initialized ... 
================================== DICOM to NIFTI CONVERSION =================================

Matching files (#=2):
Sub1/ASL
Sub1/M0

Running DCM2NIIX...
==============================================================================================
Importing subject=Sub1_1:   
>>> Subject=Sub1, visit=1, session=1, scan=ASL4D
executing: [/Users/henk/ExploreASL/ExploreASL/External/MRIcron/20190902/dcm2nii-osx -o /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/dcm2nii_temp_ASL4D /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/sourcedata/Sub1/ASL/00001.dcm]
Chris Rorden's dcm2niiX version v1.0.20190902  (JP2:OpenJPEG) (JP-LS:CharLS) Clang8.1.0 (64-bit MacOS)
Found 10 DICOM file(s)
Warning: Empty protocol name(s) (0018,1030)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Using RWVSlope:RWVIntercept = 0.1:0
 Philips Scaling Values RS:RI:SS = 0.1:0:10 (see PMC3998685)
Convert 10 DICOM as /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/dcm2nii_temp_ASL4D/ASL__20181204173011_801 (80x80x10x1)
Conversion required 0.144819 seconds (0.023110 for core code).
==============================================================================================
Chris Rorden's dcm2niiX version v1.0.20190902  (JP2:OpenJPEG) (JP-LS:CharLS) Clang8.1.0 (64-bit MacOS)
Found 10 DICOM file(s)
Warning: Empty protocol name(s) (0018,1030)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Using RWVSlope:RWVIntercept = 0.1:0
 Philips Scaling Values RS:RI:SS = 0.1:0:10 (see PMC3998685)
Convert 10 DICOM as /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/dcm2nii_temp_ASL4D/ASL__20181204173011_801 (80x80x10x1)
Conversion required 0.144819 seconds (0.023110 for core code).
==============================================================================================
status: 0
Recreating parameter file: /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/ASL4D.js100%Field ImageType contains empty fields, skipping
>>> Subject=Sub1, visit=1, session=1, scan=M0
executing: [/Users/henk/ExploreASL/ExploreASL/External/MRIcron/20190902/dcm2nii-osx -o /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/dcm2nii_temp_M0 /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/sourcedata/Sub1/M0/AP0]
Chris Rorden's dcm2niiX version v1.0.20190902  (JP2:OpenJPEG) (JP-LS:CharLS) Clang8.1.0 (64-bit MacOS)
Found 38 DICOM file(s)
Warning: Empty protocol name(s) (0018,1030)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Using RWVSlope:RWVIntercept = 1.31746:0
 Philips Scaling Values RS:RI:SS = 1.31746:0:0.00291533 (see PMC3998685)
Convert 19 DICOM as /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/dcm2nii_temp_M0/M0__20181204173011_601 (80x80x19x1)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Using RWVSlope:RWVIntercept = 1.3011:0
 Philips Scaling Values RS:RI:SS = 1.3011:0:0.00259727 (see PMC3998685)
Convert 19 DICOM as /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/dcm2nii_temp_M0/M0__20181204173011_701 (80x80x19x1)
Conversion required 0.394456 seconds (0.065465 for core code).
==============================================================================================
Chris Rorden's dcm2niiX version v1.0.20190902  (JP2:OpenJPEG) (JP-LS:CharLS) Clang8.1.0 (64-bit MacOS)
Found 38 DICOM file(s)
Warning: Empty protocol name(s) (0018,1030)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Using RWVSlope:RWVIntercept = 1.31746:0
 Philips Scaling Values RS:RI:SS = 1.31746:0:0.00291533 (see PMC3998685)
Convert 19 DICOM as /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/dcm2nii_temp_M0/M0__20181204173011_601 (80x80x19x1)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Using RWVSlope:RWVIntercept = 1.3011:0
 Philips Scaling Values RS:RI:SS = 1.3011:0:0.00259727 (see PMC3998685)
Convert 19 DICOM as /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/dcm2nii_temp_M0/M0__20181204173011_701 (80x80x19x1)
Conversion required 0.394456 seconds (0.065465 for core code).
==============================================================================================
status: 0
Recreating parameter file: /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/M0_601_00601.json
Recreating parameter file: /Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/M0_701_00701.js100%Field ImageType contains empty fields, skipping
Field ImageType contains empty fields, skipping



Error using xASL_Move (line 48)
Source doesnt exist:
/Users/henk/ExploreASL/TmpConversion/Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1/temp/Sub1/ASL_1/M0_601_00001.nii

Error in xASL_test_Flavors_DCM2BIDS (line 67)
			xASL_Move(fullfile(DirASL, 'M0_601_00001.nii'), fullfile(DirASL,
                        'M0.nii'), 1);

Error in xASL_test_Flavors (line 127)
	xASL_test_Flavors_DCM2BIDS(conversionPath, x);

HenkMutsaerts · 2021-09-10T16:43:16Z

Testing on macOS, with Jan, on Friday-eve trying to find out what is going on...

Someone decided to use Matlab's built in dir without explicitly using the -s command for recursive, or even using our own xASL_adm_GetFileList; and if Matlab is not higher than 2016 (my version! ;) than it gives the following cryptical fprintf to the screen:

This method is not able to find empty directories right now...

MichaelStritt · 2021-09-12T20:37:14Z

Why do we have both Release 1.8.0 #805 and Release 1.8.0 #816? :)

The first one is an issue, the second one is this PR right here that you're commenting in.

We discussed already with the FSL developers, best would be to include a light-weight version of FSL in ExploreASL

I thought the conclusion was that this wouldn't work for a commercial version.

…no FSL

…tatement

MichaelStritt · 2021-09-12T21:20:02Z

@jan-petr & @HenkMutsaerts:

For the two BASIL datasets in the 10 test datasets I had to change the BASIL fallback/default. Returning an empty image has to lead to a pipeline crash. Instead we could return the original/backup PWI. I implemented this here.
Philips_PASL_2DEPI_5.4.1.0_ADNI3_006_S_6681_1 crashed just because the corresponding fields were missing in Q. Here I just check for their existence. Like this it runs the full pipeline without crashing already. Should they exist though? Another big issue here is that we get this warning Field SliceReadoutTime in xASL structure has a value of 41975, which is outside of the recommended range even though we explicitly define the SliceTiming of 41.975. Easy to see that we scale ms to seconds here. So we just need to use 0.041975 in the studyPar JSON instead of "SliceTiming": 41.975?
Philips_PCASL_3DGRASE_5.1.7.2_1 here we have an ASL image which has the exact same value everywhere. Because of this the pipeline stops here isempty(tempASL) || max(tempASL(:))==0 || numel(unique(tempASL(:)))==1.
Siemens_PASL_3DGRASE_VE11C_2 here the M0 is only NaN values, which is why xASL_quant_SinglePLD stops.

MichaelStritt · 2021-09-12T21:45:15Z

Testing on macOS, with Jan, on Friday-eve trying to find out what is going on...

Someone decided to use Matlab's built in dir without explicitly using the -s command for recursive, or even using our own xASL_adm_GetFileList; and if Matlab is not higher than 2016 (my version! ;) than it gives the following cryptical fprintf to the screen:

This method is not able to find empty directories right now...

This has nothing to do with dir not working correctly. I originally wrote the code on Matlab 2019a and Jan also always used 2019b or newer. Once Mathijs wanted to test with an older version and mentioned that it didn't work for him. Therefore we added this code section below for backwards compatibility...

    % Get files and folders of datasets A and B
    [~, dateVersion] = version;
    if xASL_str2num(dateVersion(end-4:end))>2016 % dir does not work recursively in older versions
        filesA = dir(fullfile(pathDatasetA, '**','*.*'));
        filesB = dir(fullfile(pathDatasetB, '**','*.*'));
    else
        [filesA, filesB] = xASL_bids_CompareStructures_GetFileListsUnix(pathDatasetA,pathDatasetB);
    end

... the fallback function for older versions is not able to find empty directories right now though, which is why we added this print statement. For us this wasn't super relevant at the time, since only me and Jan did the flavor testing so far. Sometimes you have to find a good trade of there. If you know a nice way to make this work, feel free to improve the code 👍

I don't think the bids comparison function is the problem here though, I think our code tries to move an M0 that's not there. I guess something with the TopUp dataset doesn't work correctly in the import? Those are work in progress as far as I know, but please correct me if I'm wrong here @jan-petr & @HenkMutsaerts 👍

MichaelStritt · 2021-09-13T06:53:23Z

Windows 10 test datasets

MichaelStritt · 2021-09-13T06:59:14Z

Flavor testing

Philips_PCASL_3DGRASE_5.1.7.2_1

Warning: ...\TmpConversion\Philips_PCASL_3DGRASE_5.1.7.2_1\derivatives\ExploreASL\sub-Patient1_1
didnt contain a T1w structural image, skipping... 
> In xASL_module_Structural (line 65)

...

ERROR: ASL module terminated for subject 1: sub-Patient1_1
ans =
    'Error using xASL_module_ASL (line 220)
     Invalid ASL image
...

Philips_PCASL_3DGRASE_5.4.0.2_1

Warning: Corrupt ...\TmpConversion\Philips_PCASL_3DGRASE_5.4.0.2_1\derivatives\ExploreASL\sub-Sub001_1\T1.nii 
> In xASL_io_Nifti2Im (line 81)

...

------------------------------------------------------------------------
CAT Preprocessing error for T1:
------------------------------------------------------------------------
insufficient image overlap
------------------------------------------------------------------------
  474 - error_message
  177 - spm_affreg
...

jan-petr · 2021-09-13T07:06:40Z

` Linux Matlab2019b

{'Data'             }    {'mean_qCBF_TotalGM'}    {'median_qCBF_Tot…'}    {'median_qCBF_Dee…'}    {'CoV_qCBF_TotalGM'}    {'GMvol' }    {'WMvol' }      {'CSFvol'}    {'PipelineCompleted'}    {'TC_ASL_Registra…'}    {'TC_M0_Registrat…'}
{'GE_3Dspiral_T1p…'}    {'65.2658'          }    {'52.4785'          }    {'31.8834'          }    {'0.29875'         }    {[0.6399]}    {[0.5052]}      {[0.4329]}    {[                1]}    {[           0.9005]}    {[           0.9319]}
{'Philips_2DEPI_F…'}    {'39.0707'          }    {'27.7337'          }    {'6.7209'           }    {'0.71364'         }    {[0.5349]}    {[0.4587]}     {[0.4051]}    {[                1]}    {[           0.7297]}    {[           0.8139]}
{'Philips_2DEPI_F…'}    {'74.0677'          }    {'56.3657'          }    {'22.9638'          }    {'0.33674'         }    {[0.5442]}    {[0.3921]}     {[0.3564]}    {[                1]}    {[           0.9191]}    {[           0.9479]}
{'Philips_2DEPI_F…'}    {'48.8506'          }    {'43.3796'          }    {'20.2502'          }    {'0.41961'         }    {[0.6982]}    {[0.5828]}     {[0.2224]}    {[                1]}    {[           0.8134]}    {[           0.8345]}
{'Philips_2DEPI_L…'}    {'empty'            }    {'14.436'           }    {'3.4311'           }    {'0.89236'         }    {[0.7099]}    {[0.5136]}     {[0.1939]}    {[                1]}    {[           0.7443]}    {[           0.8896]}
{'Philips_2DEPI_S…'}    {'45.9493'          }    {'41.1539'          }    {'9.4059'           }    {'0.36499'         }    {[0.8167]}    {[0.4596]}     {[0.1516]}    {[                1]}    {[           0.8543]}    {[           0.8027]}
{'Philips_3DGRASE…'}    {'35.2382'          }    {'26.316'           }    {'7.6074'           }    {'0.66134'         }    {[0.5709]}    {[0.5279]}     {[0.4218]}    {[                1]}    {[           0.8571]}    {[           0.7547]}
{'Philips_3DGRASE…'}    {'32.1459'          }    {'24.3005'          }    {'8.5921'           }    {'0.43058'         }    {[0.7239]}    {[0.5520]}     {[0.3564]}    {[                1]}    {[           0.8413]}    {[           0.8952]}
{'Siemens3DGRASE_…'}    {'37.5377'          }    {'37.2731'          }    {'21.767'           }    {'0.78865'         }    {[0.5191]}    {[0.4658]}     {[0.4054]}    {[                1]}    {[           0.6914]}    {[           0.4987]}
{'Siemens_2DEPI_n…'}    {'67.5267'          }    {'50.6621'          }    {'13.4265'          }    {'0.31694'         }    {[0.6887]}    {[0.6031]}    {[0.3101]}    {[                1]}    {[           0.8758]}    {[           0.9589]} `

MichaelStritt · 2021-09-13T07:16:22Z

BIDS conversion

Here everything seems to work fine...

>> xASL_test_Flavors(pathExploreASL, pathTest, [0 0 1 0 0 0 0], x);
Dataset: GE_PCASL_2DEPI_23.0LX_1
Dataset: GE_PCASL_3Dspiral_14.0LX_1
Dataset: GE_PCASL_3Dspiral_15.0LX_WIP_1
Dataset: GE_PCASL_3Dspiral_22.0LX_1
Dataset: GE_PCASL_3Dspiral_23.1LX_1
Dataset: GE_PCASL_3Dspiral_24.0LX_WIP_1
Dataset: GE_PCASL_3Dspiral_25.0LX_ADNI3_027_S_5079_1
Dataset: Philips_PASL_2DEPI_5.4.1.0_ADNI3_006_S_6681_1
Dataset: Philips_PCASL_2DEPI_3.2.1.1_1
Dataset: Philips_PCASL_2DEPI_3.2.1.1_2
Dataset: Philips_PCASL_2DEPI_3.2.1.1_3
Dataset: Philips_PCASL_2DEPI_3.2.1.1_dummyLL_1
Dataset: Philips_PCASL_2DEPI_3.2.1.1_dummyMultiPLD_1
Dataset: Philips_PCASL_2DEPI_3.2.1.1_dummyQUASAR_1
Dataset: Philips_PCASL_2DEPI_3.2.2.0_glioma_1
Dataset: Philips_PCASL_2DEPI_3.2.2.1_1
Dataset: Philips_PCASL_2DEPI_3.2.3.0_BSupN_1
Dataset: Philips_PCASL_2DEPI_3.2.3.1_1
Dataset: Philips_PCASL_2DEPI_3.2.3.2_1
Dataset: Philips_PCASL_2DEPI_4.1.2.1_1
Dataset: Philips_PCASL_2DEPI_5.1.7.2_1
Dataset: Philips_PCASL_2DEPI_5.1.7.2_2
Dataset: Philips_PCASL_2DEPI_5.1.8.2_1
Dataset: Philips_PCASL_2DEPI_5.1.9.1_GMB_1
Dataset: Philips_PCASL_2DEPI_5.3.1.1_1
Dataset: Philips_PCASL_2DEPI_PARREC_1
Dataset: Philips_PCASL_3DGRASE_5.1.7.2_1
Dataset: Philips_PCASL_3DGRASE_5.4.0.2_1
Warning: Corrupt
M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\FlavorTests\TmpConversion\Philips_PCASL_3DGRASE_5.4.0.2_1\rawdata\sub-Sub001\anat\sub-Sub001_T1w_tmp_copy.nii.gz 
> In xASL_io_Nifti2Im (line 81)
  In xASL_im_CompareNiftis (line 90)
  In xASL_bids_CompareStructures>compareNIFTI (line 590)
  In xASL_bids_CompareStructures>checkFileContents (line 332)
  In xASL_bids_CompareStructures (line 201)
  In xASL_test_Flavors (line 137) 
Message: File is smaller than the dimensions say it should be.
Dataset: Philips_PCASL_3DGRASE_5.4.1.0_TopUp_1
Dataset: Philips_PCASL_3DGRASE_5.7.1.1_classic
Dataset: Siemens_PASL_2DEPI_VB15A_ADNI2_941_S_4764_1
Dataset: Siemens_PASL_2DEPI_VB17A_ADNI2_941_S_5193_1
Dataset: Siemens_PASL_2DEPI_VB17A_BSupN_1
Dataset: Siemens_PASL_2DEPI_VB17A_BSupN_2
Dataset: Siemens_PASL_2DEPI_VB19A_BSupN_1
Dataset: Siemens_PASL_3DGRASE_E11_1
Dataset: Siemens_PASL_3DGRASE_VB17A_1
Dataset: Siemens_PASL_3DGRASE_VD13D_1
Dataset: Siemens_PASL_3DGRASE_VE11C_1
Dataset: Siemens_PASL_3DGRASE_VE11C_2
Dataset: Siemens_PCASL_2DEPI_NA_1
Dataset: Siemens_PCASL_2DEPI_VB17A_BSupN_1
Dataset: Siemens_PCASL_2DEPI_VB17A_BSupN_2
Dataset: Siemens_PCASL_3DGRASE_VB17A_TopUp_1
Dataset: Siemens_PCASL_3DGRASE_VB17A_TopUp_2
Dataset: Siemens_PCASL_3DGRASE_VB17A_multiPLD_1
Dataset: Siemens_PCASL_3DGRASE_VD13A_Hadamard4-FME_1
Dataset: Siemens_PCASL_3DGRASE_VD13A_Hadamard8-FME_1
Dataset: Siemens_PCASL_3DGRASE_VD13D_1
Dataset: Siemens_PCASL_3DGRASE_VD13D_2
Dataset: Siemens_PCASL_3DGRASE_VE11C_1
Dataset: Siemens_PCASL_3DGRASE_VE11C_2

MichaelStritt and others added 30 commits July 27, 2021 21:31

#717 xASL_init_PrintCheckSettings: x.Quality -> x.settings.Quality

7a80d43

#717 xASL_spm_coreg: x.Quality -> x.settings.Quality

b2cf695

#717 xASL_bids_BIDS2Legacy: x.settings bug

fac3465

#717 xASL_init_VisualizationSettings: Simplify code

5af8f13

#717 xASL_io_ReadDataPar: Add JSON template

898c41f

#717 xASL_quant_SliceTiming: Update comment

1502074

#717 xASL_module_ASL: Add missing Q field

00749e7

#717 xASL_module_ASL: Add missing modules.asl

c3e5f59

#717 xASL_init_FileSystem: Add missing dir

c222456

#717 Use the same deprecated fields table for dataParReader and LoadX

9f0d6b4

#717 Use the same deprecated fields table for dataParReader and LoadX

3c43a05

#717 Cosmetics (indentation)

89bc4a4

#717 xASL_adm_LoadX: Simplify the code

e80b0c2

#717 xASL_adm_LoadX: Fix bug where existing SUBJECTDIR is overwritten

c4621f6

#717 xASL_adm_LoadParms: Fix missing settings field

b76ae13

#717 xASL_adm_DeleteManyTempFiles: Add try-catch because of instable …

e81387c

…behavior

#717 xASL_adm_DeleteManyTempFiles: Input check and fix undefined vari…

004791d

…able

#717 xASL_qc_TestExploreASL: Make sure the code does not break during…

13686af

… table generation A single missing backslash t or missing comma within a TSV or CSV file should not break the pipeline before writing the results to a mat file

#717 x.settings.M0 -> x.Q.M0

aff15bd

#717 M0_ConventionalProcessing -> modules.asl

fd8eb76

#717 xASL_init_SubStructs: Improve modularity and robustness of Explo…

9cd42da

…reASL

#717 xASL_init_SubStructs: Minor fix

6ed7b0b

#717 x.HadamardType -> x.modules.asl.HadamardType

d2939d3

#717 xASL_stat_PrintStats: Fix TSV writing behavior, use xASL_tsvWri…

303d474

…te instead

#717 xASL_adm_GetDeprecatedFields: Minor fix

dbf307a

#717 xASL_qc_TestExploreASL: Add n/a for failed cases

87a3648

#717 xASL_stat_GetDICOMStatistics: Assigning n/a to a numeric array

3eb740d

I am a bit surprised how this did work before. For me this breaks the code. It is not possible to assign a character array to a field of a numeric array. This would work for cell array though.

#717 xASL_spm_deformations: Minor fix parameter admin

5d501d1

#805 xASL_fsl_RunFSL: Finish comment 39a2e8e

14913c3

MichaelStritt mentioned this pull request Sep 9, 2021

Check behavior of metadata prioritization #744

Closed

MichaelStritt added 2 commits September 12, 2021 23:00

#805 xASL_quant_Basil: return original PWI instead of empty image if …

0df76a1

…no FSL

#805 xASL_quant_SinglePLD: stop code from crashing because of print s…

bcf7aec

…tatement

MichaelStritt added 3 commits September 13, 2021 10:04

#805 xASL_quant_...: Basil crashes for missing FSL with error message

f00590d

#805 Update CHANGES.md

e101ab4

#805 Cosmetics CHANGES.md

8b6ac51

jan-petr assigned MichaelStritt Sep 13, 2021

#805 Update ReferenceValues.tsv

1f984fc

MichaelStritt merged commit 1f984fc into main Sep 13, 2021

jan-petr removed the request for review from HenkMutsaerts September 15, 2021 16:12

jan-petr deleted the release-1.8.0 branch September 15, 2021 16:12

Release 1.8.0 #816

Release 1.8.0 #816

Uh oh!

Conversation

MichaelStritt commented Sep 4, 2021

Linked issue

How to test

Comments

Uh oh!

MichaelStritt commented Sep 9, 2021

Test results from before 084349b, 86591ae & 14913c3

Uh oh!

MichaelStritt commented Sep 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaelStritt commented Sep 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jan-petr commented Sep 9, 2021

Philips_2DEPI_FLAIR_DisabledQuantification_M0_LoQ

Uh oh!

MichaelStritt commented Sep 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Philips_2DEPI_FLAIR_DisabledQuantification_M0_LoQ

Uh oh!

MichaelStritt commented Sep 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Windows test datasets

Philips_2DEPI_Lesion_NoFLAIR_NoM0_HiQ

Philips_3DGRASE_FLAIR_M0_2ndRun_HiQ

Windows flavor tests

Conversion

Processing

Uh oh!

HenkMutsaerts commented Sep 10, 2021

Uh oh!

jan-petr commented Sep 10, 2021

Uh oh!

HenkMutsaerts commented Sep 10, 2021

Uh oh!

HenkMutsaerts commented Sep 10, 2021

Uh oh!

MichaelStritt commented Sep 12, 2021

Uh oh!

MichaelStritt commented Sep 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaelStritt commented Sep 12, 2021

Uh oh!

MichaelStritt commented Sep 13, 2021

Windows 10 test datasets

Uh oh!

MichaelStritt commented Sep 13, 2021

Flavor testing

Philips_PCASL_3DGRASE_5.1.7.2_1

Philips_PCASL_3DGRASE_5.4.0.2_1

Uh oh!

jan-petr commented Sep 13, 2021

Uh oh!

MichaelStritt commented Sep 13, 2021

BIDS conversion

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Test results from before `084349b`, `86591ae` & `14913c3`

MichaelStritt commented Sep 9, 2021 •

edited

Loading

MichaelStritt commented Sep 9, 2021 •

edited

Loading

MichaelStritt commented Sep 9, 2021 •

edited

Loading

MichaelStritt commented Sep 10, 2021 •

edited

Loading

MichaelStritt commented Sep 12, 2021 •

edited

Loading