Skip to content

coreneuron reports#3507

Merged
cattabiani merged 60 commits into
masterfrom
katta/corenrn_reports
Aug 22, 2025
Merged

coreneuron reports#3507
cattabiani merged 60 commits into
masterfrom
katta/corenrn_reports

Conversation

@cattabiani

@cattabiani cattabiani commented Jun 30, 2025

Copy link
Copy Markdown
Member

Context

coreneuron reports should behave like for neuron:

https://sonata-extension.readthedocs.io/en/latest/sonata_report.html

However, I have found various bugs that I fixed in this way:

  • no scaling: the value was not even passed to coreneuron and the "scaling" that was there was just for an inversion for the currents. In addition the scaling was an int, effectively cutting the decimals of the area
  • compartment reports on v and i_membrane: compartment report was reporting on voltage except if the mech_name was i_membrane. In general we would like to have the same behavior for neuron and coreneuron to report on various variables. For now there are all the mechanisms + v + i_membrane. An error is raised for an unknown variable (before the program was reporting voltage)
  • limit the use of char*: use string_view where is possible
  • use class enum: and add functions to convert from and to strings. They are easier to handle and harder to missuse
  • use SectionType enum instead of string: around the code the sectionType was used as a string and compared to other strings even if it is an enum. I am properly enforcing conversion to class enum and use of the class enum.
  • use shared_ptrs in nrnsection_mapping (coreneuron file)
  • expand compartment report to report on all the variables considered by the summation report

Heavy rework

get_summation_vars_to_report has been heavily reworked.

Before i_membrane was added, differently from all the other mechanisms, a posteriori, with a loop over the section/compartments. Meanwhile the mechansims were added with a loop over the ndoes.

Now, everything passes by a function that picks the variable (and another one to pick the scaling factor). In this way it can be reused in the compartment report. However, this implies that we need to unify the way we pick the variables. I erased completely the loop over the nodes, focussing on the sections/compartments

Testing

  • test_mech_mapping: add unit test for mech_mapping

@cattabiani cattabiani self-assigned this Jun 30, 2025
@github-actions

Copy link
Copy Markdown
Contributor

✔️ d1223ac -> artifacts URL

@github-actions

Copy link
Copy Markdown
Contributor

✔️ 0945084 -> artifacts URL

@azure-pipelines

Copy link
Copy Markdown

✔️ d1223ac -> Azure artifacts URL

@codecov

codecov Bot commented Jun 30, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 58.59873% with 65 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.44%. Comparing base (801ce19) to head (009f1f8).
⚠️ Report is 46 commits behind head on master.

Files with missing lines Patch % Lines
src/coreneuron/io/reports/report_handler.cpp 0.00% 36 Missing ⚠️
...eneuron/io/reports/report_configuration_parser.cpp 45.45% 12 Missing ⚠️
src/coreneuron/io/reports/nrnreport.hpp 65.62% 11 Missing ⚠️
src/coreneuron/mechanism/mech_mapping.cpp 80.95% 4 Missing ⚠️
src/coreneuron/io/nrnsection_mapping.hpp 75.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3507      +/-   ##
==========================================
+ Coverage   68.42%   68.44%   +0.02%     
==========================================
  Files         684      685       +1     
  Lines      116720   116809      +89     
==========================================
+ Hits        79863    79950      +87     
- Misses      36857    36859       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 4, 2025

Copy link
Copy Markdown
Contributor

✔️ 5f189fc -> artifacts URL

@azure-pipelines

Copy link
Copy Markdown

✔️ 5f189fc -> Azure artifacts URL

@github-actions

github-actions Bot commented Jul 4, 2025

Copy link
Copy Markdown
Contributor

✔️ 6abb56d -> artifacts URL

@github-actions

github-actions Bot commented Jul 4, 2025

Copy link
Copy Markdown
Contributor

✔️ 3fab19d -> artifacts URL

@github-actions

github-actions Bot commented Jul 4, 2025

Copy link
Copy Markdown
Contributor

✔️ 81aa62e -> artifacts URL

@azure-pipelines

Copy link
Copy Markdown

✔️ 81aa62e -> Azure artifacts URL

@github-actions

github-actions Bot commented Jul 7, 2025

Copy link
Copy Markdown
Contributor

✔️ e859f99 -> artifacts URL

@github-actions

github-actions Bot commented Jul 7, 2025

Copy link
Copy Markdown
Contributor

✔️ 7f9e6e8 -> artifacts URL

@azure-pipelines

Copy link
Copy Markdown

✔️ 7f9e6e8 -> Azure artifacts URL

@github-actions

github-actions Bot commented Jul 7, 2025

Copy link
Copy Markdown
Contributor

✔️ 968379a -> artifacts URL

@github-actions

github-actions Bot commented Jul 7, 2025

Copy link
Copy Markdown
Contributor

✔️ 787af02 -> artifacts URL

@github-actions

github-actions Bot commented Jul 7, 2025

Copy link
Copy Markdown
Contributor

✔️ ac2ca54 -> artifacts URL

@github-actions

Copy link
Copy Markdown
Contributor

✔️ 53f7292 -> artifacts URL

@azure-pipelines

Copy link
Copy Markdown

✔️ 53f7292 -> Azure artifacts URL

@cattabiani cattabiani requested a review from nrnhines August 12, 2025 11:15
@azure-pipelines

Copy link
Copy Markdown

✔️ 3eee388 -> Azure artifacts URL

@github-actions

Copy link
Copy Markdown
Contributor

✔️ 3eee388 -> artifacts URL

@azure-pipelines

Copy link
Copy Markdown

✔️ bbd84e9 -> Azure artifacts URL

@github-actions

Copy link
Copy Markdown
Contributor

✔️ bbd84e9 -> artifacts URL

@github-actions

Copy link
Copy Markdown
Contributor

✔️ 70eed7e -> artifacts URL

@azure-pipelines

Copy link
Copy Markdown

✔️ 70eed7e -> Azure artifacts URL

@github-actions

Copy link
Copy Markdown
Contributor

✔️ cbef6eb -> artifacts URL

@azure-pipelines

Copy link
Copy Markdown

✔️ cbef6eb -> Azure artifacts URL

@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

✔️ 009f1f8 -> artifacts URL

@azure-pipelines

Copy link
Copy Markdown

✔️ 009f1f8 -> Azure artifacts URL

@cattabiani cattabiani merged commit 8b46733 into master Aug 22, 2025
43 checks passed
@cattabiani cattabiani deleted the katta/corenrn_reports branch August 22, 2025 09:05
cattabiani added a commit that referenced this pull request Aug 22, 2025
## Why

This is the continuation of this #3507 to enable `CompartmentSet` reports in coreneuron. They are nested on top of `Compartment` reports. In fact, conceptually, they are a filtered version since we report the values typically found in a `compartment` report but only for some specific points; specified in an additional file. The specs are described [here](https://sonata-extension.readthedocs.io/en/latest/sonata_report.html#fine-grained-compartment-report).

Neurodamus/neuron already handle most of this. However, coreneuron generates reports by itself (using libsonatareport) so we need to add support in coreneuron itself. 

## How

Qualitatively this is done in a few steps:

- add a way to pass the points to coreneuron because it does not read `compartment_sets.json` by itself. This is done by adding a few lines of binary values to `report.conf`. A point is defined as a triplet: (`gid`, `section_id`, `compartment_id`) where:
- `gid` is the usual gid of a cell
- `section_id` is the progressive id of the sections. Local of each cell and it depends on the stection types that are on the cell. Qualitatively, it uniquely defines a section on the cell and you can always pass for example `dend[16] <-> section_id`.
- `compartment_id` I am not 100% sure of how these ids are really taken. However I see that I can: 
    - pick them from a `seg` (compartment) with `soma[0](0.5).node_index()`
    - they are non-repeating per thread and can identify a `node`. A node is an element in one of the basic neuron matrices 
    - differently from the nodes, they are not contiguous. The mapping can be retrieveed from `Memb_list.nodeindices` where: `Memb_list.nodeindices[i_node] = compartment_id`. 
    - reversing `Memb_list.nodeindices` allows me to retrieve the node id to, finally, pick the correct `double*` necessary for libsoantareport to work. This lazy mapping is done in `fill_segment_id_2_node_id(`. `get_var(` is in charge of calling this when necessary. The only thing left to manage this cache is to decide when to clear the cache (every new gid gets a new cache). Fortunately gids are progressive so this is already optimal.

## Changelog

### `nrnreport.hpp`
- [x] add `CompartmentSet` report type
- [x] remove `TargetType` and split it in `Compartments` and `SectionType`: no need to encode/decode 2 enums into 1 in a non-critical part of the code
- [x] add a few fields to `ReportConfiguration` to manage `CompartmentSet` report and the removal of `TargetType`

### `report_configuration_parser.cpp`
- [x] simplify reading of `report.conf`
- [x] standardize the reading of binary `vector<int>` from `report.conf` with `fill_vec_int`
- [x] add a few `nrn_assert`: `compartment` reports cannot pick multiple variables for the same point

### `report_handler.cpp`
- [x] standardized errors and asserts
- [x] reorder function position to make most of the functions local in this translation unit: they were never used outside
- [x] lazy caching with `fill_segment_id_2_node_id`. Moved out of `get_var` to carify its purpose. In some particular cases it may be useful to have it as a stand-alone function
- [x] remove `check_section_type`: if the `ReportType` is unknown we throw an error directly in `create_report`
- [x] reworked `intersection_gids` to create a list of ids to pick in `report.target`. `CompartmentSet` can easily have repeated `gids` (not a thing before) which were discarded before. This made the previous list badly suited for a 3-vector id system as in `CompartmentSet`. It could have been done without this change but this was by far the simplest, cleanest, and most elegant solution. The change for the existing functions is minimal. From:
```
for (const auto& gid: gids_to_report) {
```
to:

```
 for (const auto& intersection_id: intersection_ids) {
        const auto gid = report.target[intersection_id];
 ```
- [x] remove `nodes_to_gids` and `map_gids` which is, now, obsolete and inefficient. It requires a loop over all the nodes instead of picking only the good ones.
- [x] `all_compartments` is not part of the configuration (in a nice enum: `Compartments`)

### `report_handler.hpp`
- [x] remove most of the member methods as they were used only in `report_handler.cpp`. Faster compilation. 

### `test/external/CMakeLists.txt`
- [x] updated external tests and its tag
cattabiani added a commit to openbraininstitute/neurodamus that referenced this pull request Sep 4, 2025
## Context

This should finally take care of the compartment sets for coreneuron

Fix: #345 #322

testing for:

neuronsimulator/nrn#3507
neuronsimulator/nrn#3542

## Scope

- [x] rework report handling (mostly in node and target_manager this time. i.e. removal of `_report_setup`)
- [x] new `CoreReportConfig` to finally handle in a proper way `report.conf`
- [x] remove `write_sim_config`, `update_report_conf`, `write_report_conf` and similar functions in `CoreConfig`. Use `CoreSimulationConfig` and `CoreReportConfig` instead
- [x] move `cell-permute` in another issue/pr. EDIT done: #367
- [x] scaling moved to `report_parameters.py`
- [x] `CumulativeError` moved in `pyutils`
- [x] `ReportParams` now has its own file alleviating the burden of the `node.py` file. `_report_build_params` (previously in `node.py`) is also inside the new file.
- [x] `target_type` is no more. We do not need encoding/decoding. We just write `sections` and `compartments` in the `report.conf` 
- [x] streamlined `enable_reports`. It is still a complex function but it should be easier to read and with less edge cases 
- [x] use the decorator `@cache_errors` to just fill a `CumulativeError` instead of raising an error and raise it at the end
- [x] `merge_dicts` (in `tests/utils.py`) allows for overrides and suppression of sections from the child dict with the special keywords: `override_field` and `delete_field`

### `report.conf` changes

- [x] `type_name` has been split into `sections` and `compartments`. No need to encode everything in an enum to decode later. This simplifies code and makes the `report.conf`. The code is slightly slower `0(1)` in a part of the code that was not performance-critical
- [x] `scaling` added at the end of the report line (after buffer size)
- [x] add 2 additional binary lines in case of a compartment set report to pass the exact positions of the compartment sets. They are necessary if and only if the report type is `compartment_set`. Having them when it is not or not having them when it is is an error.

## Testing

- [x] add integration-e2e tests in `test_reports.py`

## Testing
- [x] test various bug fixes in coreneuron
- [x] add lfp test. Check that it fails with neuron
- [x] fix integration-e2e
- [x] fix scientific
- [x] fix unit
@JCGoran JCGoran mentioned this pull request Oct 17, 2025
1 task
NghiV1412 pushed a commit to ABL-Lab/neurodamus that referenced this pull request Dec 18, 2025
## Context

This should finally take care of the compartment sets for coreneuron

Fix: #345 #322

testing for:

neuronsimulator/nrn#3507
neuronsimulator/nrn#3542

## Scope

- [x] rework report handling (mostly in node and target_manager this time. i.e. removal of `_report_setup`)
- [x] new `CoreReportConfig` to finally handle in a proper way `report.conf`
- [x] remove `write_sim_config`, `update_report_conf`, `write_report_conf` and similar functions in `CoreConfig`. Use `CoreSimulationConfig` and `CoreReportConfig` instead
- [x] move `cell-permute` in another issue/pr. EDIT done: #367
- [x] scaling moved to `report_parameters.py`
- [x] `CumulativeError` moved in `pyutils`
- [x] `ReportParams` now has its own file alleviating the burden of the `node.py` file. `_report_build_params` (previously in `node.py`) is also inside the new file.
- [x] `target_type` is no more. We do not need encoding/decoding. We just write `sections` and `compartments` in the `report.conf` 
- [x] streamlined `enable_reports`. It is still a complex function but it should be easier to read and with less edge cases 
- [x] use the decorator `@cache_errors` to just fill a `CumulativeError` instead of raising an error and raise it at the end
- [x] `merge_dicts` (in `tests/utils.py`) allows for overrides and suppression of sections from the child dict with the special keywords: `override_field` and `delete_field`

### `report.conf` changes

- [x] `type_name` has been split into `sections` and `compartments`. No need to encode everything in an enum to decode later. This simplifies code and makes the `report.conf`. The code is slightly slower `0(1)` in a part of the code that was not performance-critical
- [x] `scaling` added at the end of the report line (after buffer size)
- [x] add 2 additional binary lines in case of a compartment set report to pass the exact positions of the compartment sets. They are necessary if and only if the report type is `compartment_set`. Having them when it is not or not having them when it is is an error.

## Testing

- [x] add integration-e2e tests in `test_reports.py`

## Testing
- [x] test various bug fixes in coreneuron
- [x] add lfp test. Check that it fails with neuron
- [x] fix integration-e2e
- [x] fix scientific
- [x] fix unit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants