fix EDF loading: Resample channels after loading whole file#11549
fix EDF loading: Resample channels after loading whole file#11549larsoner merged 27 commits intomne-tools:mainfrom
Conversation
|
can you convert the images/scripts above into a unit tests? |
…ling # Conflicts: # mne/io/base.py # mne/io/edf/edf.py
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
…sue10635-edf-resampling
for more information, see https://pre-commit.ci
We squash and merge so the chaos evaporates :) Let me know when I should look |
|
Should be ready now @larsoner :) |
| verbose="error", | ||
| test_preloading=False, | ||
| preload=True, # no preload=False for mixed sfreqs | ||
| ) |
There was a problem hiding this comment.
Can we add a second run of this with something like:
with pytest.warns(...):
_test_raw_reader(...)
? In theory it should still pass, right?
There was a problem hiding this comment.
No, the problem is that the _test_raw_reader also tests for array equivalence when loading slices without preloading vs loading slices with preloading. These will have different edge artifacts and therefore will fail the numpy tests. Therefore _test_raw_reader will not only throw a warning but raise an AssertionError :-/
|
Marking for merge when green, thanks in advance @skjerns ! |
Reference issue
Example: Fixes #10635
TODO before merge:
What does this implement/fix?
Previously, when loading EDF files with different sample frequencies, data was loaded in chunks and each chunk was resampled independently. This created edge artifacts for each chunk, see also:
mne-python/mne/io/edf/edf.py
Lines 368 to 372 in bf25021
I changed the loading such that minimal changes in the code appear. All channel data is loaded completely, and then resampled. Downside is that it needs 2x as much memory, but only briefly.
However, some tests now fail, as in
test_raw.py, it is tested whether data from certain time selections loaded "on the fly" is equivalent to when data is preloaded. I suspect this fails now, as the "on the fly" loaded resampled data contains different edge artifacts, however, I don't fully understand yet why in some cases the edge artifacts look so different.Details
One tests fails when
preload=Falseand specific timeframes are accessed. I presume this is due to edge artifacts, as it only happens for channels which were resampled:this test:
mne-python/mne/io/edf/tests/test_edf.py
Lines 177 to 182 in bf25021
which calls this test in
io/tests/test_raw.py, when usingpreload=False, the other tests works (preload=buffer)mne-python/mne/io/tests/test_raw.py
Lines 120 to 125 in bf25021
Plotting differences between the loaded signals reveals A9 having clear edge artefacts, as it is indeed the only resampled channel
However, some other time indices look weird and not like clear edge artifacts (plottet are diffs between data1 and data2):
Any idea what to do with this? Simply remove the test?