Conversation
|
Those limitations all seem reasonable, we can expand later once this is in. Is this ready for a round of review? |
agramfort
left a comment
There was a problem hiding this comment.
there is much more but it's a start
mne/io/egi/egimff.py
Outdated
| Path for the file | ||
| Returns | ||
| ------------------------ | ||
| info : dictionary |
mne/io/egi/egimff.py
Outdated
| input_fname : str | ||
| Path for the file | ||
| Returns | ||
| ------------------------ |
There was a problem hiding this comment.
Returns
------------------------
->
Returns
--------
mne/io/egi/egimff.py
Outdated
| # version = version.byteswap().astype(np.uint32) | ||
| # else: | ||
| # ValueError('Watchout. This does not seem to be a simple ' | ||
| # 'binary EGI file.') |
mne/io/egi/egimff.py
Outdated
| def _read_events(input_fname, hdr, info): | ||
| """Read events for the record. | ||
|
|
||
| in: |
mne/io/egi/egimff.py
Outdated
|
|
||
|
|
||
| @verbose | ||
| def read_raw_egi_mff(input_fname, montage=None, eog=None, misc=None, |
There was a problem hiding this comment.
why not factorize this with the read_raw_egi to based on extension decide if you use mff or not?
mne/io/egi/source/general.py
Outdated
| def _get_signalfname(filepath, infontype): | ||
| import os | ||
| from xml.dom.minidom import parse | ||
| import re |
There was a problem hiding this comment.
to import at the top of the file
mne/io/egi/source/general.py
Outdated
| ## | ||
|
|
||
| def _u2sample(microsecs, samprate): | ||
| import numpy as np |
mne/io/egi/source/general.py
Outdated
| # sampDuration = 1000000/sampRate; | ||
| # sampleNum = microsecs/sampDuration; | ||
| # remainder = uint64(rem(microsecs, sampDuration)); | ||
| # sampleNum = fix(sampleNum); |
mne/io/egi/source/general.py
Outdated
| """Read data signal.""" | ||
| import numpy as np | ||
|
|
||
| binfile = filepath + '/' + nbinfile |
mne/io/egi/source/header.py
Outdated
| blockbeginsamps[x + 1] = blockbeginsamps[x] + blocknumsamps[x] | ||
| # ----------------------------------------------------- | ||
| summaryinfo = dict(blocks=signal_blocks['blocks'], | ||
| eegFilename=signal_blocks['eegFile'], |
There was a problem hiding this comment.
if don't use CamelCase for variable names but snake_case
|
@ramonapariciog thanks for taking a stab at this. There is a bit of work |
|
@ramonapariciog excellent Job, @agramfort Ramon did great Job, all camelCases was my bad :-) |
|
@ramonapariciog @Eric89GXL @agramfort may be we all work on this during hackathon ? |
|
Sheraz, I think Roman and Jaakko will meet soon to finalize this one in a
pair programming session :) So it will be merged rather soon I predict.
…On Thu, Mar 2, 2017 at 8:07 PM Sheraz Khan ***@***.***> wrote:
@ramonapariciog <https://github.com/ramonapariciog> @Eric89GXL
<https://github.com/Eric89GXL> @agramfort <https://github.com/agramfort>
may be we all work on this during hackathon ?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#4017 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB0finMZAiWPW_cN7Zczkcpk44_oLgzhks5rhxNggaJpZM4MLm6H>
.
|
|
@dengemann you guys are awesome :) |
|
Now the |
larsoner
left a comment
There was a problem hiding this comment.
I'll stop reviewing for now. I pointed out a few small pattern things, but I suspect you were already planning to clean up most of them.
mne/io/egi/egimff.py
Outdated
| """ | ||
| Created on Fri Jan 13 11:01:39 2017. | ||
|
|
||
| @author: ramonapariciog |
mne/io/egi/egimff.py
Outdated
| time_n = dateutil.parser.parse(mff_hdr['date']) | ||
| info = dict( # leer del info.xml desde read_mff_header | ||
| version=version, # duda | ||
| year=int(time_n.strftime('%Y')), # duda |
There was a problem hiding this comment.
That's the current president of Poland, maybe related? 😉
mne/io/egi/egimff.py
Outdated
|
|
||
|
|
||
| @verbose | ||
| def read_raw_egi_mff(input_fname, montage=None, eog=None, misc=None, |
mne/io/egi/egimff.py
Outdated
| on timestamps received by the Netstation. As a consequence, | ||
| triggers have only short durations. | ||
|
|
||
| This step will fail if events are not mutually exclusive. |
There was a problem hiding this comment.
This .. note:: is very long. Let's move it to the Notes section. If you want, you can add a on or two sentence .. note:: here like "this function will attempt to generate a synthetic trigger channel. For details, see Notes below.".
| montage : str | None | instance of montage | ||
| Path or instance of montage containing electrode positions. | ||
| If None, sensor locations are (0,0,0). See the documentation of | ||
| :func:`mne.channels.read_montage` for more information. |
There was a problem hiding this comment.
I think we should stop putting these in the constructor and have people use the methods instead
There was a problem hiding this comment.
Now this is fused with the egi reader. Deprecation should be done in another PR
mne/io/egi/egimff.py
Outdated
| if len(data) != len(info['ch_names']): | ||
| raise ValueError('len(data) does not match ' | ||
| 'len(info["ch_names"])') | ||
| logger.info('Creating RawArray with %s data, n_channels=%s, ' |
mne/io/egi/source/data.py
Outdated
|
|
||
|
|
||
| def read_mff_data(filepath, indtype, startind, lastind, hdr): | ||
| """Function for load the data in a list. |
There was a problem hiding this comment.
Don't say "function for" or "helper" anymore, use imperative
mne/io/egi/source/data.py
Outdated
| """ | ||
| import numpy as np | ||
| import os | ||
| from .general import _bls2blns, _read_signaln, _get_gains |
There was a problem hiding this comment.
don't nest these
hopefully general does not create a circular import. If it does, some helper function probably needs to be moved
mne/io/egi/source/data.py
Outdated
| info_fp = os.path.join(filepath, suminfo['infoFile'][0]) | ||
| gains = _get_gains(info_fp) | ||
| else: | ||
| print('Multisubject not suported') |
|
I can take a look tomorrow. So the test_egi.mff is supposedly continuous data, not epoched? |
|
Hello Jakko, |
|
Thank you Jakko, if you need something i will try to respond you quickly. |
|
Now it should work. I'm a bit worried about the event timing. It seems that the event times in the test data are all smaller than the |
mne/io/egi/egimff.py
Outdated
| warn('Event outside data range (%ss).' % (i / | ||
| info['samp_rate'])) | ||
| continue | ||
| events[n][i] = 2**n |
There was a problem hiding this comment.
I changed it to use powers of 2 so it'll be easy to mask overlaps.
There was a problem hiding this comment.
Does this create inconsistency with how the other readers work, or do we not assign consecutive values elsewhere?
There was a problem hiding this comment.
Yeah, I noticed that before it was doing basically the same thing as the egi reader (consecutive values). The problem is that it doesn't allow overlapping events.
There was a problem hiding this comment.
Okay then we should probably have a parameter for this. event_numbering='linear' | 'log' or 'consecutive' | 'power' or something. If we want to make 2**n the default here, it should probably be part of this PR, or as part of an immediate follow-up.
mne/io/egi/egimff.py
Outdated
| # STI 014 is simply the sum of all event channels (powers of 2). | ||
| if len(egi_events) > 0: | ||
| egi_events = np.vstack([egi_events, np.sum(egi_events, axis=0)]) | ||
| data[n_channels:] = egi_events |
There was a problem hiding this comment.
I also made the stim channel construction lazy like this. Now it's created on data fetch stage.
There was a problem hiding this comment.
Ok Jakko, thanks i will test it with other files.
|
ping @ramonapariciog |
mne/io/egi/source/general.py
Outdated
| nsamples = block['nsamples'] | ||
| else: | ||
| raise NotImplementedError('Only continuos files are supported') | ||
| numblocks = int(l // blocksize) |
There was a problem hiding this comment.
@ramonapariciog the problem was here. It was previously not doing integer division and then adding an extra block if the data did not align. If I understand correctly, the proper thing to do is to just dump the extra data, since it is only available for the first channels in the block.
|
I'll just say I'd appreciate no longer having to use EEGLAB to import MFF files :) |
|
There's still a lot of cleaning to do. Seems like there's a lot of redundant meta data stored. I think the best is to store only the necessary data and then add more later if needed. |
|
+1 be pragmatic. Store what is needed for conversion.
…On Mon, 24 Apr 2017 at 09:03, jaeilepp ***@***.***> wrote:
There's still a lot of cleaning to do. Seems like there's a lot of
redundant meta data stored. I think the best is to store only the necessary
data and then add more later if needed.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4017 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB0fihEaEgIh16vmcE81ygNwrch7pGU1ks5rzJ2igaJpZM4MLm6H>
.
|
|
Sure, i will check which data is needed from the first version to discart it. |
|
Hi Jakko, i have problems to use the function again :'( i update numpy scipy and also, to use the new version of autoreject and this happens when i try load a file. |
|
Strange that I can't reproduce this even after creating an environment with the same settings as travis. |
Codecov Report
@@ Coverage Diff @@
## master #4017 +/- ##
==========================================
+ Coverage 86.18% 86.21% +0.03%
==========================================
Files 358 349 -9
Lines 65020 65084 +64
Branches 9914 9983 +69
==========================================
+ Hits 56040 56115 +75
+ Misses 6247 6222 -25
- Partials 2733 2747 +14 |
|
Currently this data is not used. I assume at least GCAL should be used for calibration, but I have no idea what ICAL is. EEGLAB reader seems to recognize |
|
Gain and intercept (multiplicative and additive) would make sense for GCAL
and ICAL
|
|
Hello Jakko, i check again and now i think that is loading the data, but, you already tried using the data with Autoreject? Do you think that could be the timeline? |
|
I think this is close to being ready. I couldn't figure out how to use the ICAL values, but the GCALs are now in use. I compared to the converted egi files I have and the data scales are very close to each other. |
|
Can you update |
|
Regarding ICAL/GCAL can you cross-check with FieldTrip (e.g., http://www.fieldtriptoolbox.org/faq/how_can_i_read_egi_mff_data_without_the_jvm) |
mne/io/egi/egimff.py
Outdated
| signal_blocks = _get_blocks(fname) | ||
| blocknumsamps = np.sum(signal_blocks['blockNumSamps']) | ||
|
|
||
| pibhasref = False |
There was a problem hiding this comment.
either make the name more explicit or add a comment, I don't know what pib means
mne/io/egi/egimff.py
Outdated
|
|
||
| summaryinfo = dict(eeg_fname=eeg_file, | ||
| info_fname=info_files[0], | ||
| pibNChans=pibnchans, |
mne/io/egi/egimff.py
Outdated
| ntrials = 1 | ||
|
|
||
| # Add the sensor info. | ||
| sensor_layout_file = filepath + '/sensorLayout.xml' |
mne/io/egi/egimff.py
Outdated
| for sensor in sensors: | ||
| sensortype = int(sensor.getElementsByTagName('type')[0] | ||
| .firstChild.data) | ||
| if sensortype == 0 or sensortype == 1: |
mne/io/egi/egimff.py
Outdated
| if sensor.getElementsByTagName('name')[0].firstChild is None: | ||
| sn = sensor.getElementsByTagName('number')[0].firstChild.data | ||
| sn = sn.encode() | ||
| tmp_label = 'E' + sn.decode() |
There was a problem hiding this comment.
why encode then decode? why not just u'E' + text_type(sn)?
mne/io/egi/egimff.py
Outdated
| gain=0, | ||
| bits=0, | ||
| value_range=0) | ||
| unsegmented = 1 if mff_hdr['nTrials'] == 1 else 0 |
mne/io/egi/egimff.py
Outdated
| unsegmented = 1 if mff_hdr['nTrials'] == 1 else 0 | ||
| precision = 4 | ||
| if precision == 0: | ||
| RuntimeError('Floating point precision is undefined.') |
There was a problem hiding this comment.
... also this can never happen because precision = 4 immediately above...?
mne/io/egi/egimff.py
Outdated
|
|
||
|
|
||
| class RawMff(BaseRaw): | ||
| """RAWMff class.""" |
mne/io/egi/egimff.py
Outdated
| def _read_segment_file(self, data, idx, fi, start, stop, cals, mult): | ||
| """Read a chunk of data.""" | ||
| from ..utils import _mult_cal_one | ||
| dtype = '<f4' |
There was a problem hiding this comment.
it seems like this should be read from the header somewhere but it's not currently (and the check above is broken)
There was a problem hiding this comment.
I can't find it anywhere in the meta data and it seems fieldtrip has the size hard coded too.
There was a problem hiding this comment.
Sounds reasonable, would be good to add a comment along these lines
| from glob import glob | ||
| from os.path import basename, join, splitext | ||
|
|
||
| from xml.etree.ElementTree import parse |
There was a problem hiding this comment.
this should be grouped with the standard imports
|
I couldn't find anything about the calibrations in fieldtrip code base. |
larsoner
left a comment
There was a problem hiding this comment.
One last structural thing. @agramfort a while ago mentioned rolling this into read_raw_egi and triaging based on file extension. Is that possible?
| montage : str | None | instance of montage | ||
| Path or instance of montage containing electrode positions. | ||
| If None, sensor locations are (0,0,0). See the documentation of | ||
| :func:`mne.channels.read_montage` for more information. |
mne/io/egi/egimff.py
Outdated
| def _read_segment_file(self, data, idx, fi, start, stop, cals, mult): | ||
| """Read a chunk of data.""" | ||
| from ..utils import _mult_cal_one | ||
| dtype = '<f4' |
There was a problem hiding this comment.
Sounds reasonable, would be good to add a comment along these lines
mne/io/egi/events.py
Outdated
|
|
||
|
|
||
| def _ns(s): | ||
| """Remove namespace, but only it there is a namespace to begin with.""" |
mne/io/egi/tests/test_egi.py
Outdated
| assert_true('RawMff' in repr(raw)) | ||
| include = ['DIN1', 'DIN2', 'DIN3', 'DIN4', | ||
| 'DIN5', 'DIN7'] | ||
| with warnings.catch_warnings(record=True): |
There was a problem hiding this comment.
add comment about what warnings are being caught
|
Hello guys, |
|
@ramonapariciog @SherazKhan can you guys test and report any difficulty? thanks heaps @jaeilepp for the hard work ! |
|
Yeah sure, thanks @jaeilepp for all, i will test. |
|
ping @ramonapariciog @SherazKhan can you guys test and report any difficulty? |
|
ok let's say it's working :) @jaeilepp please update what's new and merge |
|
Ok I'll update what's new in master. I feel like closing PRs... |
|
see 82306db |
|
ohhh, i have problems with my computer for now, but yeah, the last version of the function was working without problems. |
|
no need to delete.
please report problem with what I merged in master yesterday
|
* Create read_egi_mff I need to understand from where part of the _BaseRaw class __init__() the _read_segment_file are called to execution, because the data from Sheraz reads the data and only adapting it is enough for a first test. * The egimff.py was added with the read_raw_egi_mff function. To import use from mne.io import read_raw_egi_mff * Correction of some PEP8 errors and delete the folder mff bad added * Raw reader. * preload=False * Cleaning. * Fixes to event reading. * Fixes. * Fixes. * cleaning useless functions * Cleaning. * Reading of meta data at the end. * Update tests. Cleaning. * Cleaning. Correct number of samples. * More cleaning. * Cleaning. * Update testing dataset. * Update test. * Fix. * Test fix. * Fixes. * Docs. Switched back to running number events. * Gains. Fix to events. * Cleaning. Apply gains. * Fixes. * Fixes. * Read coordinates. Cleaning. * Using _combine_triggers to create STI 014 and event_id attribute * Refactoring. * Fix.
mne-mff-reader BETA
MNE Class and functions for loading the EGI '.mff' files from EGI Netstation EEG.
How to import:
Only with the following line:
from mne.io.mff import read_raw_egi_mff
and for create the raw instance:
raw = read_raw_egi_mff(filepath, exclude, include, verbose)
All the pararameters are the same than for the read_raw_egi.
Limitations:
1.- At this moment doesn't support multisubject records, but can be adapted for that.
2.- At the moment has been tested by comparing the results with the obtained by a single subject and notsegmented record raw file, and the tests in the test_egi.py
3.- At the moment all data is loaded to memory, using the preload=True mode.
Thanks to PhD. Guillaume Dumas, PhD. Dennis A. Engeman & PhD. Sheraz Khan