Skip to content

Reading large EDF files with preload = False raises memory error #10634

@arnaumanasanch

Description

@arnaumanasanch

Issue/Bug

When reading in an IPython notebook a large .edf file which is 10Gb size (4 hours/ 150 channels/ 2048Hz sampling frequency) with:

raw_edf = mne.io.read_raw_edf('file.edf', preload=False)

the kernel crashes as the RAM memory, which is 12 Gb, gets full.

I thought the preload argument was loading only metadata (which should not be more than a few hundred Mbs).

Would there be a way in which I could read the raw_edf (only the metadata) and then with the get_data() method, be able to just load into memory a small piece of data (n channels and x time range) without the system failing because of the RAM being full?

If I try the same with a higher RAM (32Gb), there is no problem and once it is loaded, the RAM goes back to normal (e.g: before loading it is 1Gb, when loading it goes up to 15Gb, once loaded back to 1 Gb approx.). We need to make it functional, if possible, with the 12 Gb RAM.

For privacy reasons, I can’t share the file that I am using.

Thanks in advance.

Additional information

MNE version: e.g. 1.0.3
operating system: / Windows 10

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions