coreneuron reports#3507
Conversation
|
✔️ d1223ac -> artifacts URL |
|
✔️ 0945084 -> artifacts URL |
|
✔️ d1223ac -> Azure artifacts URL |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3507 +/- ##
==========================================
+ Coverage 68.42% 68.44% +0.02%
==========================================
Files 684 685 +1
Lines 116720 116809 +89
==========================================
+ Hits 79863 79950 +87
- Misses 36857 36859 +2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
✔️ 5f189fc -> artifacts URL |
|
✔️ 5f189fc -> Azure artifacts URL |
|
✔️ 6abb56d -> artifacts URL |
|
✔️ 3fab19d -> artifacts URL |
|
✔️ 81aa62e -> artifacts URL |
|
✔️ 81aa62e -> Azure artifacts URL |
|
✔️ e859f99 -> artifacts URL |
|
✔️ 7f9e6e8 -> artifacts URL |
|
✔️ 7f9e6e8 -> Azure artifacts URL |
|
✔️ 968379a -> artifacts URL |
|
✔️ 787af02 -> artifacts URL |
|
✔️ ac2ca54 -> artifacts URL |
|
✔️ 53f7292 -> artifacts URL |
|
✔️ 53f7292 -> Azure artifacts URL |
|
✔️ 3eee388 -> Azure artifacts URL |
|
✔️ 3eee388 -> artifacts URL |
|
✔️ bbd84e9 -> Azure artifacts URL |
|
✔️ bbd84e9 -> artifacts URL |
|
✔️ 70eed7e -> artifacts URL |
|
✔️ 70eed7e -> Azure artifacts URL |
|
✔️ cbef6eb -> artifacts URL |
|
✔️ cbef6eb -> Azure artifacts URL |
|
|
✔️ 009f1f8 -> artifacts URL |
|
✔️ 009f1f8 -> Azure artifacts URL |
## Why This is the continuation of this #3507 to enable `CompartmentSet` reports in coreneuron. They are nested on top of `Compartment` reports. In fact, conceptually, they are a filtered version since we report the values typically found in a `compartment` report but only for some specific points; specified in an additional file. The specs are described [here](https://sonata-extension.readthedocs.io/en/latest/sonata_report.html#fine-grained-compartment-report). Neurodamus/neuron already handle most of this. However, coreneuron generates reports by itself (using libsonatareport) so we need to add support in coreneuron itself. ## How Qualitatively this is done in a few steps: - add a way to pass the points to coreneuron because it does not read `compartment_sets.json` by itself. This is done by adding a few lines of binary values to `report.conf`. A point is defined as a triplet: (`gid`, `section_id`, `compartment_id`) where: - `gid` is the usual gid of a cell - `section_id` is the progressive id of the sections. Local of each cell and it depends on the stection types that are on the cell. Qualitatively, it uniquely defines a section on the cell and you can always pass for example `dend[16] <-> section_id`. - `compartment_id` I am not 100% sure of how these ids are really taken. However I see that I can: - pick them from a `seg` (compartment) with `soma[0](0.5).node_index()` - they are non-repeating per thread and can identify a `node`. A node is an element in one of the basic neuron matrices - differently from the nodes, they are not contiguous. The mapping can be retrieveed from `Memb_list.nodeindices` where: `Memb_list.nodeindices[i_node] = compartment_id`. - reversing `Memb_list.nodeindices` allows me to retrieve the node id to, finally, pick the correct `double*` necessary for libsoantareport to work. This lazy mapping is done in `fill_segment_id_2_node_id(`. `get_var(` is in charge of calling this when necessary. The only thing left to manage this cache is to decide when to clear the cache (every new gid gets a new cache). Fortunately gids are progressive so this is already optimal. ## Changelog ### `nrnreport.hpp` - [x] add `CompartmentSet` report type - [x] remove `TargetType` and split it in `Compartments` and `SectionType`: no need to encode/decode 2 enums into 1 in a non-critical part of the code - [x] add a few fields to `ReportConfiguration` to manage `CompartmentSet` report and the removal of `TargetType` ### `report_configuration_parser.cpp` - [x] simplify reading of `report.conf` - [x] standardize the reading of binary `vector<int>` from `report.conf` with `fill_vec_int` - [x] add a few `nrn_assert`: `compartment` reports cannot pick multiple variables for the same point ### `report_handler.cpp` - [x] standardized errors and asserts - [x] reorder function position to make most of the functions local in this translation unit: they were never used outside - [x] lazy caching with `fill_segment_id_2_node_id`. Moved out of `get_var` to carify its purpose. In some particular cases it may be useful to have it as a stand-alone function - [x] remove `check_section_type`: if the `ReportType` is unknown we throw an error directly in `create_report` - [x] reworked `intersection_gids` to create a list of ids to pick in `report.target`. `CompartmentSet` can easily have repeated `gids` (not a thing before) which were discarded before. This made the previous list badly suited for a 3-vector id system as in `CompartmentSet`. It could have been done without this change but this was by far the simplest, cleanest, and most elegant solution. The change for the existing functions is minimal. From: ``` for (const auto& gid: gids_to_report) { ``` to: ``` for (const auto& intersection_id: intersection_ids) { const auto gid = report.target[intersection_id]; ``` - [x] remove `nodes_to_gids` and `map_gids` which is, now, obsolete and inefficient. It requires a loop over all the nodes instead of picking only the good ones. - [x] `all_compartments` is not part of the configuration (in a nice enum: `Compartments`) ### `report_handler.hpp` - [x] remove most of the member methods as they were used only in `report_handler.cpp`. Faster compilation. ### `test/external/CMakeLists.txt` - [x] updated external tests and its tag
## Context This should finally take care of the compartment sets for coreneuron Fix: #345 #322 testing for: neuronsimulator/nrn#3507 neuronsimulator/nrn#3542 ## Scope - [x] rework report handling (mostly in node and target_manager this time. i.e. removal of `_report_setup`) - [x] new `CoreReportConfig` to finally handle in a proper way `report.conf` - [x] remove `write_sim_config`, `update_report_conf`, `write_report_conf` and similar functions in `CoreConfig`. Use `CoreSimulationConfig` and `CoreReportConfig` instead - [x] move `cell-permute` in another issue/pr. EDIT done: #367 - [x] scaling moved to `report_parameters.py` - [x] `CumulativeError` moved in `pyutils` - [x] `ReportParams` now has its own file alleviating the burden of the `node.py` file. `_report_build_params` (previously in `node.py`) is also inside the new file. - [x] `target_type` is no more. We do not need encoding/decoding. We just write `sections` and `compartments` in the `report.conf` - [x] streamlined `enable_reports`. It is still a complex function but it should be easier to read and with less edge cases - [x] use the decorator `@cache_errors` to just fill a `CumulativeError` instead of raising an error and raise it at the end - [x] `merge_dicts` (in `tests/utils.py`) allows for overrides and suppression of sections from the child dict with the special keywords: `override_field` and `delete_field` ### `report.conf` changes - [x] `type_name` has been split into `sections` and `compartments`. No need to encode everything in an enum to decode later. This simplifies code and makes the `report.conf`. The code is slightly slower `0(1)` in a part of the code that was not performance-critical - [x] `scaling` added at the end of the report line (after buffer size) - [x] add 2 additional binary lines in case of a compartment set report to pass the exact positions of the compartment sets. They are necessary if and only if the report type is `compartment_set`. Having them when it is not or not having them when it is is an error. ## Testing - [x] add integration-e2e tests in `test_reports.py` ## Testing - [x] test various bug fixes in coreneuron - [x] add lfp test. Check that it fails with neuron - [x] fix integration-e2e - [x] fix scientific - [x] fix unit
## Context This should finally take care of the compartment sets for coreneuron Fix: #345 #322 testing for: neuronsimulator/nrn#3507 neuronsimulator/nrn#3542 ## Scope - [x] rework report handling (mostly in node and target_manager this time. i.e. removal of `_report_setup`) - [x] new `CoreReportConfig` to finally handle in a proper way `report.conf` - [x] remove `write_sim_config`, `update_report_conf`, `write_report_conf` and similar functions in `CoreConfig`. Use `CoreSimulationConfig` and `CoreReportConfig` instead - [x] move `cell-permute` in another issue/pr. EDIT done: #367 - [x] scaling moved to `report_parameters.py` - [x] `CumulativeError` moved in `pyutils` - [x] `ReportParams` now has its own file alleviating the burden of the `node.py` file. `_report_build_params` (previously in `node.py`) is also inside the new file. - [x] `target_type` is no more. We do not need encoding/decoding. We just write `sections` and `compartments` in the `report.conf` - [x] streamlined `enable_reports`. It is still a complex function but it should be easier to read and with less edge cases - [x] use the decorator `@cache_errors` to just fill a `CumulativeError` instead of raising an error and raise it at the end - [x] `merge_dicts` (in `tests/utils.py`) allows for overrides and suppression of sections from the child dict with the special keywords: `override_field` and `delete_field` ### `report.conf` changes - [x] `type_name` has been split into `sections` and `compartments`. No need to encode everything in an enum to decode later. This simplifies code and makes the `report.conf`. The code is slightly slower `0(1)` in a part of the code that was not performance-critical - [x] `scaling` added at the end of the report line (after buffer size) - [x] add 2 additional binary lines in case of a compartment set report to pass the exact positions of the compartment sets. They are necessary if and only if the report type is `compartment_set`. Having them when it is not or not having them when it is is an error. ## Testing - [x] add integration-e2e tests in `test_reports.py` ## Testing - [x] test various bug fixes in coreneuron - [x] add lfp test. Check that it fails with neuron - [x] fix integration-e2e - [x] fix scientific - [x] fix unit



Context
coreneuron reports should behave like for neuron:
https://sonata-extension.readthedocs.io/en/latest/sonata_report.html
However, I have found various bugs that I fixed in this way:
mech_namewasi_membrane. In general we would like to have the same behavior for neuron and coreneuron to report on various variables. For now there are all the mechanisms + v + i_membrane. An error is raised for an unknown variable (before the program was reporting voltage)Heavy rework
get_summation_vars_to_reporthas been heavily reworked.Before
i_membranewas added, differently from all the other mechanisms, a posteriori, with a loop over the section/compartments. Meanwhile the mechansims were added with a loop over the ndoes.Now, everything passes by a function that picks the variable (and another one to pick the scaling factor). In this way it can be reused in the compartment report. However, this implies that we need to unify the way we pick the variables. I erased completely the loop over the nodes, focussing on the sections/compartments
Testing