-
Notifications
You must be signed in to change notification settings - Fork 13
#683 Revamp data loading #695
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
QC Meeting (Henk & Michael)
|
Test: data import>> pathTest = 'M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer';
>> x = ExploreASL(pathTest,1,0,0);
ExploreASL will run the import workflow and will run the initialization...
==============================================================================================
________ __ ______ ______ __
/ | / | / \ / \ / |
########/ __ __ ______ ## | ______ ______ ______ /###### |/###### |## |
## |__ / \ / | / \ ## | / \ / \ / \ ## |__## |## \__##/ ## |
## | ## \/##/ /###### |## |/###### |/###### |/###### |## ## |## \ ## |
#####/ ## ##< ## | ## |## |## | ## |## | ##/ ## ## |######## | ###### |## |
## |_____ /#### \ ## |__## |## |## \__## |## | ########/ ## | ## |/ \__## |## |_____
## |/##/ ## |## ##/ ## |## ##/ ## | ## |## | ## |## ##/ ## |
########/ ##/ ##/ #######/ ##/ ######/ ##/ #######/ ##/ ##/ ######/ ########/
## |
## |
##/
==================================== ExploreASL Settings =====================================
Dataset Root ...nt\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer
Import Modules DCM2NII NII2BIDS DEFACE BIDS2LEGACY
Process Modules
bPause False
iWorker 1
nWorkers 1
==============================================================================================
ExploreASL v1.8.0_BETA initialized ...
Matching files:
'Sub103\ASL'
'Sub103\T1w'
#=2
Running import (i.e. dcm2niiX)
==============================================================================================
Importing subject=Sub103_1:
>>> Subject=Sub103, visit=1, session=1, scan=ASL4D
executing: ["M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\ExploreASL\External\MRIcron\20190902\dcm2nii.exe" -f "%f_%p_%t_%s_%r" -o "M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\ASL_1\dcm2nii_temp_ASL4D" "M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\sourcedata\Sub103\ASL\i1445064.MRDC..dcm"]
Chris Rorden's dcm2niiX version v1.0.20190902 (JP2:OpenJPEG) (JP-LS:CharLS) MSC1900 (64-bit Windows)
Found 80 DICOM file(s)
Warning: Empty protocol name(s) (0018,1030)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Convert 40 DICOM as M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\ASL_1\dcm2nii_temp_ASL4D\ASL__20170920100637_4_1 (128x128x40x1)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Convert 40 DICOM as M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\ASL_1\dcm2nii_temp_ASL4D\ASL__20170920100637_4_41 (128x128x40x1)
Conversion required 0.459000 seconds.
==============================================================================================
Chris Rorden's dcm2niiX version v1.0.20190902 (JP2:OpenJPEG) (JP-LS:CharLS) MSC1900 (64-bit Windows)
Found 80 DICOM file(s)
Warning: Empty protocol name(s) (0018,1030)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Convert 40 DICOM as M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\ASL_1\dcm2nii_temp_ASL4D\ASL__20170920100637_4_1 (128x128x40x1)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Convert 40 DICOM as M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\ASL_1\dcm2nii_temp_ASL4D\ASL__20170920100637_4_41 (128x128x40x1)
Conversion required 0.459000 seconds.
==============================================================================================
status: 0
Recreating parameter file: M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\ASL_1\ASL4D_4_00001.json
Recreating parameter file: M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\ASL_1\ASL4D_4_00041.js01%Parameter RescaleSlope not found, default used
Parameter RescaleSlopeOriginal not found, default used
Parameter MRScaleSlope not found, default used
Parameter RescaleIntercept not found, default used
Parameter AcquisitionTime not found, default used
Parameter NumberOfTemporalPositions not found, default used
Parameter RWVIntercept not found, default used
Parameter RWVSlope not found, default used
Parameter AssetRFactor not found, default used
Parameter EffectiveEchoSpacing not found, default us51%Parameter RescaleSlope not found, default used
Parameter RescaleSlopeOriginal not found, default used
Parameter MRScaleSlope not found, default used
Parameter RescaleIntercept not found, default used
Parameter AcquisitionTime not found, default used
Parameter NumberOfTemporalPositions not found, default used
Parameter RWVIntercept not found, default used
Parameter RWVSlope not found, default used
Parameter AssetRFactor not found, default used
Parameter EffectiveEchoSpacing not found, default used
Parameter InversionTime not found, default us100%Field ImageType contains empty fields, skipping
Field ImageType contains empty fields, skipping
ASL context: deltam,m0scan
Sorted NIfTIs based on InstanceNumbers...
Warning: concatenating multiple NIfTIs & jsons as output from dcm2niiX
Corrected dcm2niiX output for GE files and found the correct scan order:
M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\ASL_1\ASL4D.nii
deltam,m0scan
>>> Subject=Sub103, visit=1, session=1, scan=T1w
executing: ["M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\ExploreASL\External\MRIcron\20190902\dcm2nii.exe" -f "%f_%p_%t_%s_%r" -o "M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\T1w_1\dcm2nii_temp_T1w" "M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\sourcedata\Sub103\T1w\i1444892.MRDC..dcm"]
Chris Rorden's dcm2niiX version v1.0.20190902 (JP2:OpenJPEG) (JP-LS:CharLS) MSC1900 (64-bit Windows)
Found 172 DICOM file(s)
Warning: Empty protocol name(s) (0018,1030)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Convert 172 DICOM as M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\T1w_1\dcm2nii_temp_T1w\T1w__20170920100637_3_1 (256x256x172x1)
Conversion required 1.739000 seconds.
==============================================================================================
Chris Rorden's dcm2niiX version v1.0.20190902 (JP2:OpenJPEG) (JP-LS:CharLS) MSC1900 (64-bit Windows)
Found 172 DICOM file(s)
Warning: Empty protocol name(s) (0018,1030)
Warning: Unable to append protocol name (0018,1030) to filename (it is empty).
Convert 172 DICOM as M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\T1w_1\dcm2nii_temp_T1w\T1w__20170920100637_3_1 (256x256x172x1)
Conversion required 1.739000 seconds.
==============================================================================================
status: 0
Recreating parameter file: M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\temp\Sub103\T1w_1\T1w.js00%Parameter RescaleSlope not found, default used
Parameter RescaleSlopeOriginal not found, default used
Parameter MRScaleSlope not found, default used
Parameter RescaleIntercept not found, default used
Parameter AcquisitionTime not found, default used
Parameter NumberOfTemporalPositions not found, default used
Parameter RWVIntercept not found, default used
Parameter RWVSlope not found, default used
Parameter AssetRFactor not found, default used
Parameter EffectiveEchoSpacing not found, default used
Parameter GELabelingDuration not found, default us100%Field ImageType contains empty fields, skipping
dataset_description.json is missing the following RECOMMENDED fields:
HEDVersion, DatasetType, License
Missing Recommended ASL fields: SliceEncodingDirection, LabelingOrientation
The field ArterialSpinLabelingType is PCASL, please check the dependencies below:
The recommended fields are missing: PCASLType, LabelingPulseAverageGradient, LabelingPulseMaximumGradient, LabelingPulseAverageB1, LabelingPulseDuration, LabelingPulseFlipAngle, LabelingPulseInterval
Defacing image for privacy reasons...
-----------------------------------------------------------------------
Running SPM job 'De-face Images'
Done 'De-face Images'
Done
xASL_Move: overwriting M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\rawdata\sub-Sub103\anat\sub-Sub103_T1w.nii
There is no dataPar.json file in the study root directory. Default settings will be used...
Parsing BIDS scans: 100%
Note that any warnings may have only printed once if they were repeated for multiple scans
Always run your rawdata folder through the BIDS validator first and aim to avoid warnings
Converting from BIDS to Legacy: M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunte100%
M0 parsed for M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer\derivatives\ExploreASL\sub-Sub103\ASL_1\ASL4D.nii.gz
Creating default dataPar.json since file was not found...
Overwriting x.dir.dataPar...Test: data loading>> x = ExploreASL(pathTest,0,0,0);
ExploreASL will load the dataset...
==============================================================================================
________ __ ______ ______ __
/ | / | / \ / \ / |
########/ __ __ ______ ## | ______ ______ ______ /###### |/###### |## |
## |__ / \ / | / \ ## | / \ / \ / \ ## |__## |## \__##/ ## |
## | ## \/##/ /###### |## |/###### |/###### |/###### |## ## |## \ ## |
#####/ ## ##< ## | ## |## |## | ## |## | ##/ ## ## |######## | ###### |## |
## |_____ /#### \ ## |__## |## |## \__## |## | ########/ ## | ## |/ \__## |## |_____
## |/##/ ## |## ##/ ## |## ##/ ## | ## |## | ## |## ##/ ## |
########/ ##/ ##/ #######/ ##/ ######/ ##/ #######/ ##/ ##/ ######/ ########/
## |
## |
##/
==================================== ExploreASL Settings =====================================
Dataset Root ...nt\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer
Import Modules
Process Modules
bPause False
iWorker 1
nWorkers 1
==============================================================================================
ExploreASL v1.8.0_BETA initialized ...
Automatically defining sessions...
==================================== Additional Settings =====================================
1 scans - 0 exclusions, resulting in 1 scans of:
Longitudinal timePoint 1 = 1 scans - 0 exclusions = 1 scans
ASL sessions: 1
Ancillary data, sets: 4 sets are defined for 1 "SubjectsSessions"
Set 1 = "session" options "ASL_1", codes for paired data
Set 2 = "LongitudinalTimePoint" options "TimePoint_1", codes for paired data
Set 3 = "SubjectNList" options "SubjectNList", codes for paired data
Set 4 = "Site" options "SingleSite", codes for two-sample data
x.D.ROOT M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCA ...
... SL_3Dspiral_volunteer\derivatives\ExploreASL
x.DELETETEMP 1 (delete temporary files)
x.Quality 1 (0 = fast try-out; 1 = normal high quality)
==============================================================================================Test: data processing with pause>> x = ExploreASL(pathTest,0,1,1);
ExploreASL will run the processing pipeline...
==============================================================================================
________ __ ______ ______ __
/ | / | / \ / \ / |
########/ __ __ ______ ## | ______ ______ ______ /###### |/###### |## |
## |__ / \ / | / \ ## | / \ / \ / \ ## |__## |## \__##/ ## |
## | ## \/##/ /###### |## |/###### |/###### |/###### |## ## |## \ ## |
#####/ ## ##< ## | ## |## |## | ## |## | ##/ ## ## |######## | ###### |## |
## |_____ /#### \ ## |__## |## |## \__## |## | ########/ ## | ## |/ \__## |## |_____
## |/##/ ## |## ##/ ## |## ##/ ## | ## |## | ## |## ##/ ## |
########/ ##/ ##/ #######/ ##/ ######/ ##/ #######/ ##/ ##/ ######/ ########/
## |
## |
##/
==================================== ExploreASL Settings =====================================
Dataset Root ...nt\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer
Import Modules
Process Modules STRUCTURAL ASL POPULATION
bPause True
iWorker 1
nWorkers 1
==============================================================================================
ExploreASL v1.8.0_BETA initialized ...
Automatically defining sessions...
==================================== Additional Settings =====================================
1 scans - 0 exclusions, resulting in 1 scans of:
Longitudinal timePoint 1 = 1 scans - 0 exclusions = 1 scans
ASL sessions: 1
Ancillary data, sets: 4 sets are defined for 1 "SubjectsSessions"
Set 1 = "session" options "ASL_1", codes for paired data
Set 2 = "LongitudinalTimePoint" options "TimePoint_1", codes for paired data
Set 3 = "SubjectNList" options "SubjectNList", codes for paired data
Set 4 = "Site" options "SingleSite", codes for two-sample data
x.D.ROOT M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCA ...
... SL_3Dspiral_volunteer\derivatives\ExploreASL
x.DELETETEMP 1 (delete temporary files)
x.Quality 1 (0 = fast try-out; 1 = normal high quality)
==============================================================================================
Press any key to start processing & analyzing
Please ensure you have a read-only copy of your original data as they may be overwritten
Or press CTRL/command-C to cancel...
...Test: data processing without pause>> x = ExploreASL(pathTest,0,1,0);
ExploreASL will run the processing pipeline...
==============================================================================================
________ __ ______ ______ __
/ | / | / \ / \ / |
########/ __ __ ______ ## | ______ ______ ______ /###### |/###### |## |
## |__ / \ / | / \ ## | / \ / \ / \ ## |__## |## \__##/ ## |
## | ## \/##/ /###### |## |/###### |/###### |/###### |## ## |## \ ## |
#####/ ## ##< ## | ## |## |## | ## |## | ##/ ## ## |######## | ###### |## |
## |_____ /#### \ ## |__## |## |## \__## |## | ########/ ## | ## |/ \__## |## |_____
## |/##/ ## |## ##/ ## |## ##/ ## | ## |## | ## |## ##/ ## |
########/ ##/ ##/ #######/ ##/ ######/ ##/ #######/ ##/ ##/ ######/ ########/
## |
## |
##/
==================================== ExploreASL Settings =====================================
Dataset Root ...nt\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer
Import Modules
Process Modules STRUCTURAL ASL POPULATION
bPause False
iWorker 1
nWorkers 1
==============================================================================================
ExploreASL v1.8.0_BETA initialized ...
Automatically defining sessions...
==================================== Additional Settings =====================================
1 scans - 0 exclusions, resulting in 1 scans of:
Longitudinal timePoint 1 = 1 scans - 0 exclusions = 1 scans
ASL sessions: 1
Ancillary data, sets: 4 sets are defined for 1 "SubjectsSessions"
Set 1 = "session" options "ASL_1", codes for paired data
Set 2 = "LongitudinalTimePoint" options "TimePoint_1", codes for paired data
Set 3 = "SubjectNList" options "SubjectNList", codes for paired data
Set 4 = "Site" options "SingleSite", codes for two-sample data
x.D.ROOT M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCA ...
... SL_3Dspiral_volunteer\derivatives\ExploreASL
x.DELETETEMP 1 (delete temporary files)
x.Quality 1 (0 = fast try-out; 1 = normal high quality)
==============================================================================================
Running xASL_module_Structural ...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
=== Subject: sub-Sub103, Module: xASL_module_Structural, 05-Jul-2021 20:26:26 ===
...Output argument warning>> ExploreASL_Master(pathTest,0,0,0);
ExploreASL will load the dataset...
==============================================================================================
________ __ ______ ______ __
/ | / | / \ / \ / |
########/ __ __ ______ ## | ______ ______ ______ /###### |/###### |## |
## |__ / \ / | / \ ## | / \ / \ / \ ## |__## |## \__##/ ## |
## | ## \/##/ /###### |## |/###### |/###### |/###### |## ## |## \ ## |
#####/ ## ##< ## | ## |## |## | ## |## | ##/ ## ## |######## | ###### |## |
## |_____ /#### \ ## |__## |## |## \__## |## | ########/ ## | ## |/ \__## |## |_____
## |/##/ ## |## ##/ ## |## ##/ ## | ## |## | ## |## ##/ ## |
########/ ##/ ##/ #######/ ##/ ######/ ##/ #######/ ##/ ##/ ######/ ########/
## |
## |
##/
==================================== ExploreASL Settings =====================================
Dataset Root ...nt\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCASL_3Dspiral_volunteer
Import Modules
Process Modules
bPause False
iWorker 1
nWorkers 1
==============================================================================================
ExploreASL v1.8.0_BETA initialized ...
Automatically defining sessions...
==================================== Additional Settings =====================================
1 scans - 0 exclusions, resulting in 1 scans of:
Longitudinal timePoint 1 = 1 scans - 0 exclusions = 1 scans
ASL sessions: 1
Ancillary data, sets: 4 sets are defined for 1 "SubjectsSessions"
Set 1 = "session" options "ASL_1", codes for paired data
Set 2 = "LongitudinalTimePoint" options "TimePoint_1", codes for paired data
Set 3 = "SubjectNList" options "SubjectNList", codes for paired data
Set 4 = "Site" options "SingleSite", codes for two-sample data
x.D.ROOT M:\SoftwareDevelopment\MATLAB\m.stritt\Server_xASL\New_TestData\GE_PCA ...
... SL_3Dspiral_volunteer\derivatives\ExploreASL
x.DELETETEMP 1 (delete temporary files)
x.Quality 1 (0 = fast try-out; 1 = normal high quality)
==============================================================================================
Data loading requested but no output structure defined...
Try adding "x = " to the command to load data into the x structure |
jan-petr
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very good work! Two small things - bOnlyLoad was sometimes set to 1 and sometimes to true, I have changed that to boolean on all occurrences - check my latest commit. There's a small comment in the code about commentaries. But the rest can be merged.
| % ExploreASL_Initialize - This wrapper initializes ExploreASL: managing paths, deployment, etc. | ||
| % ExploreASL_ImportMaster - Multi-step import workflow for DCM2NII, NII2BIDS & BIDS2LEGACY. | ||
| % ExploreASL_ProcessMaster - Multi-step processing workflow for the STRUCTURAL, ASL and POPULATION module. | ||
| % I. Initialization: ExploreASL_Initialize |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not a huge issue of course. But is there any reason to have Roman instead Arabic numbers?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jan-petr: I talked with Henk about that. We thought it could be helpful if we do not use arabic numbers for the import and process modules which are themselves part of the ExploreASL_Master workflow. Alternatively we could use A, B, C, D etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now we have:
I.InitializationII.Import Master1.DICOM TO NII2.NII TO BIDS3.DEFACE4.BIDS TO Legacy
III.Data loadingIV.Print user feedback (before pipeline)V.Processing Master1.Structural Module2.ASL Module3.Population Module
VI.Print user feedback (after pipeline)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice!
Moved data loading to separate script, simplified master script, moved user feedback to separate script
I simplified the behavior by introducing bLoadData and removing bReinitialize. Within the data loading script I simplified the condition as well.
I moved the user feedback to the print user feedback script
Move function to separate script. Fix nargout bug.
6605a82 to
4f104ea
Compare
Linked issue
Check out #683