Skip to content

ENH: Allow removal of (up to two) bad marker coils in read_raw_kit()#12394

Merged
larsoner merged 17 commits intomne-tools:mainfrom
JD-Zhu:handle_bad_marker_coils
Apr 15, 2024
Merged

ENH: Allow removal of (up to two) bad marker coils in read_raw_kit()#12394
larsoner merged 17 commits intomne-tools:mainfrom
JD-Zhu:handle_bad_marker_coils

Conversation

@JD-Zhu
Copy link
Copy Markdown
Contributor

@JD-Zhu JD-Zhu commented Jan 30, 2024

@larsoner

Related to #8981 #8989

The previous fix deals with missing marker coils when reading in KIT data. However, when all five marker coils are present in the data, sometimes there are bad coils that need to be removed post-hoc. This step needs to happen before the device-head transformation is computed during the call to read_raw_kit().

Removal of bad coils cannot be achieved by simply editing the relevant digitisation file (.elp), as this causes another error downstream in fit_matched_points() due to a mismatch in the number of marker coils between the .elp and .mrk files. The .mrk file cannot be edited as it's not in plain text format.

The proposed solution here allows a list of bad coils to be passed in when calling read_raw_kit(), so that they can be removed once the marker coil info has been read in from the .elp and .mrk files. Up to two bad coils are allowed, so we can ensure there are at least three marker coils remaining to be used for the tranformation.

@larsoner
Copy link
Copy Markdown
Member

@JD-Zhu if you look at the style run by clicking "Details" you'll see this:

mne/io/kit/coreg.py:175:89: E501 Line too long (90 > 88)
mne/io/kit/kit.py:130:19: B006 Do not use mutable data structures for argument defaults
mne/io/kit/kit.py:915:15: B006 Do not use mutable data structures for argument defaults

From CircleCI details you'll see:

RuntimeError: Error documenting RawKIT:
'kit_badcoils'

I'll comment more inline but in general when a CI fails if you scroll toward the bottom of the failed/red step it should have helpful info about what went wrong!

Copy link
Copy Markdown
Member

@larsoner larsoner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a good start! I think a simple little test would be to look through our existing MNE-Python KIT test datasets to see if one of them has enough coils where you could remove one or two of the markers using bad_coils and ensure the dev_head_t that gets computed is similar (but not identical!). Let me know if anything is unclear or you want help with any of this!

@drammock
Copy link
Copy Markdown
Member

when all five marker coils are present in the data, sometimes there are bad coils that need to be removed post-hoc. This step needs to happen before the device-head transformation is computed during the call to read_raw_kit(). [...] The proposed solution here allows a list of bad coils to be passed in when calling read_raw_kit(), so that they can be removed once the marker coil info has been read in from the .elp and .mrk files.

Perhaps this is a naive question, but does this mean that users need to call read_raw_kit() twice? (once to load all 5 coils and notice that some are bad, and a second time to exclude the bad ones)? Would it be possible to load once, notice the bad coils, then call a new method (RawKIT.remove_marker_coils(bads=[...]) or so) that would re-compute the dev_head_t in-place? If so, that seems like a preferable / more intuitive user experience to me, and also avoids doing the I/O twice (which is usually slow).

@JD-Zhu
Copy link
Copy Markdown
Contributor Author

JD-Zhu commented Jan 31, 2024

@larsoner Thanks so much for the detailed comments and suggested changes! Is there a way to accept all the suggestions ("Add suggestion to batch" is grayed out) or should I just click "Commit suggestion" for each one? Thanks.

@JD-Zhu
Copy link
Copy Markdown
Contributor Author

JD-Zhu commented Jan 31, 2024

when all five marker coils are present in the data, sometimes there are bad coils that need to be removed post-hoc. This step needs to happen before the device-head transformation is computed during the call to read_raw_kit(). [...] The proposed solution here allows a list of bad coils to be passed in when calling read_raw_kit(), so that they can be removed once the marker coil info has been read in from the .elp and .mrk files.

Perhaps this is a naive question, but does this mean that users need to call read_raw_kit() twice? (once to load all 5 coils and notice that some are bad, and a second time to exclude the bad ones)? Would it be possible to load once, notice the bad coils, then call a new method (RawKIT.remove_marker_coils(bads=[...]) or so) that would re-compute the dev_head_t in-place? If so, that seems like a preferable / more intuitive user experience to me, and also avoids doing the I/O twice (which is usually slow).

Thanks for the comment. In this solution, users only need to call read_raw_kit() once. The bad coils are usually known beforehand (most likely written down during data acquisition), so they can be passed into read_raw_kit() straight away. That said, I think having an option to remove bad coils and re-compute the dev_head_t (as well as updating 'dig' and 'hpi_results'?) for raw data that have already been read in (as per your suggestion) is also not a bad idea. This can also be quite useful on epochs/evoked data, e.g. if sensor-space analysis has been carried out previously (without worrying about bad coils as it probably didn't matter for sensor space), it would be nice to be able to load the previous results (e.g. -epo.fif) and then apply bad marker coil removal to obtain the correct transformations for source analysis.

Edit: when I said "post-hoc" in the original post, I meant after data acquisition (rather than after reading the data in). Sorry for the confusion!

@larsoner
Copy link
Copy Markdown
Member

Is there a way to accept all the suggestions ("Add suggestion to batch" is grayed out) or should I just click "Commit suggestion" for each one? Thanks.

On the "Files changed" tab of this page (which links here) yes you can "add to batch" then commit multiple changes at once. But at least a couple of my suggestions were more ideas that you'd have to push changes for anyway (or one suggestion that applies in multiple places in principle) so probably easier just to push a commit!

@JD-Zhu JD-Zhu changed the title FIX: Allow removal of (up to two) bad marker coils in read_raw_kit() ENH: Allow removal of (up to two) bad marker coils in read_raw_kit() Feb 2, 2024
@JD-Zhu
Copy link
Copy Markdown
Contributor Author

JD-Zhu commented Feb 2, 2024

Looks like a good start! I think a simple little test would be to look through our existing MNE-Python KIT test datasets to see if one of them has enough coils where you could remove one or two of the markers using bad_coils and ensure the dev_head_t that gets computed is similar (but not identical!). Let me know if anything is unclear or you want help with any of this!

Sounds like a good idea! Will try to test this once I can figure out the virtual environments (which always seem to get me in trouble).
Alternatively, is there a way to call functions in the dev version I'm working on rather than the stable version I have installed (e.g. maybe specify the path when importing?)

@larsoner
Copy link
Copy Markdown
Member

larsoner commented Feb 2, 2024

Alternatively, is there a way to call functions in the dev version I'm working on rather than the stable version I have installed (e.g. maybe specify the path when importing?)

In theory if you are in the mne-python/ directory, having mne-python/mne/ directly beneath it should make it so that import mne imports from that directory. Looks like that's the case:

~$ pip uninstall -q mne
Proceed (Y/n)? y
larsoner:~$ pip install mne
Collecting mne
  Downloading mne-1.6.1-py3-none-any.whl.metadata (13 kB)
Successfully installed mne-1.6.1
~$ python -c "import mne; print(mne)"
<module 'mne' from '/home/larsoner/python/virtualenvs/base/lib/python3.11/site-packages/mne/__init__.py'>
~$ cd python/mne-python
~/python/mne-python$ python -c "import mne; print(mne)"
<module 'mne' from '/home/larsoner/python/mne-python/mne/__init__.py'>
~/python/mne-python$ pytest mne/io/kit/tests/test_kit.py 
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.11.6, pytest-8.0.0, pluggy-1.3.0
PyQt6 6.6.0 -- Qt runtime 6.6.0 -- Qt compiled 6.6.0
MNE 1.6.1 -- /home/larsoner/python/mne-python/mne
...

You can see where mne gets imported from. Even though the version is reported incorrectly, the path is correct, and I verified that adding a raise RuntimeError to mne/__init__.py caused the RuntimeError to happen in the pytest call.

@larsoner larsoner enabled auto-merge (squash) April 15, 2024 16:27
@larsoner larsoner added this to the 1.7 milestone Apr 15, 2024
@larsoner larsoner merged commit 0aa4bec into mne-tools:main Apr 15, 2024
@larsoner
Copy link
Copy Markdown
Member

Pushed a few commits to make CIs happy then merged, thanks @JD-Zhu !

larsoner added a commit to larsoner/mne-python that referenced this pull request Apr 16, 2024
* upstream/main:
  fix prefilter management for EDF/BDF (mne-tools#12441)
  [pre-commit.ci] pre-commit autoupdate (mne-tools#12541)
  ENH: Allow removal of (up to two) bad marker coils in read_raw_kit() (mne-tools#12394)
  Implement `picks` argument to `Raw.plot()` (mne-tools#12467)
  Add Meggie under Related Software documentation (mne-tools#12540)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants