EDITOR

Dear @andruhamax,

Thank you for your submission. I will serve as editor for your article and will soon assign two reviewers.

@heplesser Could you review this one ? I know you're already reviewing another submission but this submission is based on NEST.

Thanks, that would be great if you can suggest someone.

@eroesch Could you review this submission ?

@rougier I saw the full author list and affiliations now. I am closely collaborating with van Albada and Diesmann, so I have a clear conflict of interest here and cannot review this submission.

@heplesser Ok, thank for telling me. Could you suggest another reviewer ?

@mstimberg Could you review this submission ?

Hi @rougier, I would have really liked to do this one but unfortunately we are in the middle of exam period, and won't have time for at least an other month.

Hi @rougier, I am happy to do the review, but I probably won't be able to work on it before July 18th.

EDITOR

@mstimberg Thanks ! Given the period of the year, 18th of July is great ok.

EDITOR

@eroesch Thanks for the reply. End of July might be a bit late.

EDITOR

@rossant Can you review this submission ?

@rougier ok, I'll do it by July 22th

EDITOR

@rossant Thanks that's great.
@andruhamax The review will start in 2/3 weeks.

REVIEWER 2

I'm trying to install and run the code on Ubuntu 16.04. First, a few remarks. I think it would help to:

• give direct link to download nest 2.8.0 source code (otherwise one might go to the downloads page and take the latest version, which is not 2.8.0)
• give instructions to download the code since it's not obvious for someone who isn't used to git (git clone, git checkout the particular branch, etc.)
• clarify the fact that some dependencies need to be installed (one has to lookup the nest installation instructions) before running bootstrap.sh

Then, running python Compte2003.py doesn't work because my default Python installation is Python 3 (miniconda3) whereas the code is written for Python 2. First option (the best one if it is feasible) would be to update the code to make it Python 3-compatible so that it would just work for someone like me who already has a default Py3 installation. Second option (less ideal) would be to explain in the instructions how to use a particular distribution of Python (2) before building.

REVIEWER 1

I had a first go at running the code (under Ubuntu 14.04) and it worked with a few minor problems (see below) -- I did not yet look into the results or the code in any detail. I also agree with @rossant's comments (for the download link to NEST 2.8 maybe consider the Zenodo DOI?).

Issues in code/README.md:

• "Unpack NEST source code to nest_2.8.0" should be nest-2.8.0
• Make clear that nest-2.8.0-build should be a directory in parallel to nest-2.8.0 (not inside)
• $HOME/<path-to-nest>: the $HOME seems somewhat redundant, for me <path-to-nest> already includes the user's home. Maybe consider defining a PREFIX variable that can be used later as well (as in the NEST documentation)
• the definition of PATH and PYTHONPATH is incorrect (refers to /install-2.8.0 instead of $PREFIX/install-2.8.0)
• The last line refers to Compte2003_R_scheme.py but the file is called Compte2003_R_method.py

Not sure whether this is worth mentioning (as it is not specific to this code or even NEST), but I could not run the code within my anaconda environment due to a readline issue (see e.g. here: ContinuumIO/anaconda-issues#152). I used the workaround from that bug report (removing anaconda's readline and using pip install readline instead) and then it worked.

EDITOR

@andruhamax Can you address the comments on installation ?

Dear reviewers, thanks a lot for your reply.
Because I was not available last weeks, I am starting to work on your comments now.

Installation readme.md file is updated to account for corrections from both reviewers.

Transition to Python 3 would require a major code revision, because differences between Python 2 and Python 3 are considerable. For a moment, we would like to stay with Python 2, which is still widely used.

REVIEWER 1

My general assessment of the article, the code and its documentation is that it fulfills the quality standards of ReScience: all results are replicable using a standard tool of the community (NEST) when following the authors' instructions, and all code and analysis scripts are well documented. However, the replication of the original study remains qualitative, there are discrepancies between the results and the original results that need to be cleared up (see major issues below).

I cannot judge the details of the C++ implementation of the neuron/synapse models that are added to NEST itself, but 1) they seem to be based on established models; 2) they are clearly linked to the model description in the paper (e.g. the variable names are intuitive); 3) the code has a reasonable number of comments, stating important details (such as the expected scale of parameters).

Their approach of simplifying the synaptic model seems justified and they present simulations that support their claim that they successfully replicated the basic neuron and synapse dynamics. The authors also present an extensive investigation of the membrane input resistance measurements with some apparently relevant insights (I have to admit that I am not very familiar with these kind of measurements so I cannot judge this part on a detailed level).

Article

Major issues

The authors state (p.10): "However, despite close reproduction of the model constituents, the simulated network activity is characterized by unreasonably high firing rates due to the dominance of NMDA conductances far exceeding the potency of the opposing GABAergic inhibition. To achieve a network state characterized by comparable excitation and inhibition, we modify the synaptic strengths W of all synapse types". There are several issues here:

1. Why do the values provided by the original authors not work? This would be an important error in the original paper and therefore needs thorough investigation.
2. How were new parameter values chosen? All the parameter values for the synaptic weights are different from the corresponding maximal synaptic conductances in the original paper, both in absolute values but also in relative terms (e.g. NMDA to excitatory cells is 6 times stronger than NMDA to inhibitory cells in the present manuscript but only 1.2 times stronger in the original paper). For the absolute values it is unclear how to compare them to the original values given the differences in the synaptic model.

I wonder whether differences in the connectivity pattern could explain the different results. In the Compte et al. paper they state: "The neurons in the network are sparsely connected to each other through a fixed number of connections that are set at the beginning of the simulation. Neurons make 20±5 (SD) contacts to their postsynaptic partners (multiple contacts onto the same target, but no autapses, are allowed)." The authors of the present manuscript interpret this in a way that each excitatory cell connects (on average) to 20 other excitatory cells and to 20 other inhibitory cells, the same for inhibitory cells. I think there are other possible interpretations: e.g. each neuron could have 20 outgoing connections in total (of which 4/5 would go to excitatory cells since they outnumber excitatory cells by 4:1). Also, the authors of the present manuscript assume that all excitatory connections form one AMPA and one NMDA synapse (which means that excitatory synapses could be modeled as one AMPA-NMDA hybrid synapse) – the original paper does not make any clear statement on the distribution of AMPA vs. NMDA synapses, but in principle the above quote about the connectivity does not exclude that excitatory cells form AMPA and NMDA synapses independently.

The authors have to make their decisions and their understanding of the original paper clear and motivate why they chose the specific values they chose (and the conclusion should not deemphasize these changes as "minor modifications").

The results that the authors achieve with the adjusted weight values reproduce several of the results of the original paper (spontaneous up-down oscillations, dynamics of membrane resistance and ionic conductances) but also look different in quantitative terms: firing rate in Compte et al. is up to ~10Hz (exc. neurons) and ~20Hz (inh. neurons) while it is up to ~20Hz (exc. neurons) and ~35Hz (inh. neurons) in the present manuscript. The dynamics of the firing rate during an up-state is also quite different, it is uni-modal in the Compte et al. study but has a strong peak followed by a lower level plateau in the present manuscript. Also, the wave-like propagation is far from clear and inhibitory neurons do not seem to fire before excitatory neurons as in the original study. Finally, the length of one Up period is ~2.5s, followed by a Down period of approximately the same length whereas in the original paper an Up periods lasts ~ 0.5s to 1s, followed by a Down period of ~2.5s.

The authors chose to focus on a subset of the results of the original paper (which is of course fine in general), it's a pity that the paper does not include some robustness investigation, though. In particular, some manipulations like stronger or weaker inhibition/excitation (Fig. 6B,C in the Compte et al. paper) could be performed with the present code just by changing one parameter value.

Minor issues

• p.2., last line above the figure: I find the term "pacemaking activity" not ideal because it has connotations of oscillatory (and potentially bursting) activity – I'd prefer the simpler term "spontaneous activity".
• p.3, caption Fig. 2: The explanation of the DC currents in the first sentence is a bit confusing (e.g. it somewhat implies that the same DC current leads to different synaptic activation rates for different channels and also it sounds odd to say that the DC current activates the synaptic channels in the first place – the DC current makes the neuron spike and this is used to trigger the synaptic channels)
• p.7, Table 3: some conventions seem to be different than in the original paper, e.g. the sign of the synaptic current and the sign of the first term in the Ca2+ and Na+ concentration. If those are not typos, a short mention of this difference in the caption would be helpful.
• p.8, Table 4: The RHS of m_inf for "Na+-dependent K+ current, soma" has a spurious vertical bar at the end.
• p.10, Table 7: the standard deviation of g^soma_L for inhibitory neurons should be 0.5 instead of 0.05, I think.
• A general remark: there are a few places where it is not immediately obvious whether the authors refer to their text/figure or the text/figure in the Compte et al. paper: e.g.
  • p.11, l.10: "(see Figure 6)", better: "(see their Figure 6)"
  • p.15, Fig.7C: "measured with the method suggested by the authors", better "measured with the method suggested in [1]" or "... by the authors of [1]"
• Figure 5 could benefit from a few adjustment to make it easier to compare to the corresponding figure of the original paper: Fig. 5B cut be cut higher to show less of the uninformative spikes and resemble more closely Fig. 5B of the original paper; Fig. 5C and 5D should use the same axis limits to facilitate the quantitative comparison of excitatory and inhibitory conductances (as in Fig. 5C of the original paper)

Installation instructions

With the changes made by the authors I am now happy with the installation instructions and I can install NEST and their code following them step by step. The only thing I am not sure about is whether it is necessary to mention the installation from the deadsnake PPA. This is not necessary for current versions of Ubuntu and while it may become necessary in the future, relying on an unofficial PPA is not great. I would maybe link to it with something like "if your Ubuntu distribution no longer provides a Python 2.7 package, you can use the following PPA". In any case, telling users how to install Python 2 on their machine (which they most likely already have) without telling them how to install numpy and matplotlib (which they might not have), is a bit odd.

Code

Minor issues

• Compte2003.py, l. 505 (before saving the data) states "# transform data to standard form" – a slightly more detailed comment could be helpful here
• The code uses g_0 for the coupling between soma and dendrite, instead of g_sd (as in the papers) – maybe this has been done for consistency with other NEST code?

General

• I find it a bit confusing to have empty data and notebook directories in the repository, but this seems to be the case for the other published articles as well so I guess this is part of the editorial policy (@rougier ?).

PS: Apologies in advance, but vacation time is getting into the way again: I will not be available for the coming two weeks, I will be back on August 16th.

REVIEWER 2

The code is clear and well-documented. A few minor remarks:

• I agree with @mstimberg that you should add the fix for conda users in the installation instructions: conda remove --force readline && pip install readline (I had the same problem)
• The end of life of Python 2.7 will happen in 3.5 years. Further, the next major versions of core scientific packages like IPython (which is a recommended dependency for Nest) will be incompatible with Python 2.7. The number of users unable to run your code will only increase with time, which might be problematic for this journal (@rougier there might be other articles with this problem?).
• In any case, I was able to update your Python code to Python 3 almost instantly using the command 2to3 *.py -w (it only involved minor changes to print and range). There were a few bugs due to the following issue.
• You should use range() on integers rather than floats (VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future). Please update the code accordingly.
• With these changes, the simulation ran successfully with Python 3. Please incorporate the Python 3 changes to the code (of course it remains compatible with Python 2). Here is the patch.
• FYI, after the simulation, the script crashed. I had to install texlive-latex-extra and dvipng. It might be worth specifying these dependencies in the instructions.
• You should add an estimated running time in the instructions so that users know how long they'll have to wait until the simulation is complete.

I didn't have time to look at the article yet. I'll do it when I come back from vacation in 10 days.

EDITOR

@rossant Thanks for the code review, we'll wait for your article review in 10 days.

@andruhamax Can you address these comments ?

AUTHOR

yes, I am working on that.

REVIEWER 2

This submission's quality is good, the code and article are clear, and the results are qualitatively similar to the original article. Like @mstimberg, the main concern I have is that the network activity is not reproduced with the same synapse parameters, and the authors had to use different values than in the original implementation. I think this deserves further investigations and explanations.

EDITOR

@andruhamax Any progress ?

Sorry for the delay, the update is coming within few days.

EDITOR

@andruhamax Any update ?

Answer to reviewer 1:
1)
“Why do the values provided by the original authors not work? This would be an important error in the original paper and therefore needs thorough investigation.”

Our analysis suggests that the activation of single synapses elicits different postsynaptic responses in the authors' work and in our non-simplified implementation of the original synaptic dynamics. However, due to the absence of the source code, we cannot find the origin of this discrepancy.

In the revised manuscript, we explain this issue as follows:

“This difference from the original results might be a consequence of the different implementation of synaptic dynamics. Closer examination of the original Figure 5C suggests that unitary excitatory and inhibitory postsynaptic responses are approximately 0.5 nS during periods of network silence. This contradicts our implementation of the non-modified synaptic dynamics (see Fig. 2), where single synaptic activations (with synaptic weights as in the original model) would result in 5.4 and 0.8 nS postsynaptic response for AMPA and GABA channels, respectively. This suggests an inconsistency between the reported dynamics and that which was implemented already in the original work. Furthermore, we assumed the units of the function defining the dependence of the synaptic conductances on the presynaptic
membrane potential to be ms−1, but this is not strictly specified in the original work, providing another potential source of discrepancies.”

and

“Besides the reasons mentioned above, differences between the simplified and original synaptic kinetics during network activity, not brought out by the simple inputs in Fig. 2, may contribute to the need for weight changes to obtain a network state similar to that in the original study. However, this is unlikely to account for the need to strongly scale synaptic weights relative to each other. In the absence of the original code and since our network implementation does not allow using the original synapse dynamics, we cannot resolve this issue here.”

“How were new parameter values chosen?”

We find that it is not possible to reach the desired network regime with a simple scaling of all synaptic weights, so that ratios stay unchanged. This is probably connected to discrepancies in the implementation of the synaptic dynamics, mentioned above. Therefore, we scaled synaptic weight values individually based on the ability of the combined set of weights to provide the desired network dynamics. This approach is in full agreement with the approach chosen in the original work, where the proper network dynamics is the key factor, rather than a close match of the synaptic weights to known biology.

In the revised manuscript, we clarify the basis for our choice of parameter values:

“To achieve an appropriate network regime, we modify the synaptic strengths W of all synapse types. Specifically, we match the firing rates, the duration of Up and Down states, the wave propagation speed, and the response to network stimulation.”

“The authors have to make their decisions and their understanding of the original paper clear and motivate why they chose the specific values they chose”

In our hands, multiple sets of synaptic weights could provide the desired network dynamics. We chose a particular realization from among these sets based on relative proximity to the originally reported weight values. We mention this in the revised manuscript:

“However, a range of different synaptic weight settings was able to produce qualitatively identical network activity. Therefore, we chose a set of weights from among these possibilities based on relative proximity to the originally reported weight values.”

“I wonder whether differences in the connectivity pattern could explain the different results.”

We completely agree with the reviewer in his interpretation of the network connectivity, and corrected our model accordingly. However, this change did not improve the network dynamics. Further adjustment of the synaptic weights was required.

“the authors of the present manuscript assume that all excitatory connections form one AMPA and one NMDA synapse”

From biology we know that cortical connections between excitatory neurons have both AMPA and NMDA receptors combined at postsynaptic sites. In the original paper, the authors speak about excitatory and inhibitory connections with no indication about separation into AMPA and NMDA types. Therefore, we think it is safe to assume that AMPA and NMDA are colocalized also in the original model.

“The results that the authors achieve...look different in quantitative terms”

For the revised version of our manuscript, we adjusted the parameters for a closer quantitative matching of the network dynamics. This includes firing rates, Up and Down state durations, wave propagation speed, and response to network stimulation (an additional figure (Fig.5) appeared). The duration of Up states in our model is approx. 0.6 sec, similar to their Figure 3. The duration of Down states in our model is approx. 2.6 sec, similar to their Figure 3.

“it's a pity that the paper does not include some robustness investigation”

To address this issue, in the revised version we analyze the network response to 10% modification of synaptic weights, as well as to blockade of AMPA, NDMA, and GABA channels (see top of p. 17 and new Figure 10).

The conclusion section is slightly rewritten to incorporate the minor changes due to the modification of the synaptic parameters.

“We provide a closer look at the method which the authors of the original study use to access neuronal conductance (or resistance). Their measure reflects an “instantaneous chord conductance”, which results in the model in values around 15 nS in isolated pyramidal neuron, and around 20 nS throughout network activity. In experimental works, the typical measure approximates a “steady-state slope conductance” (see [@jack1983] for classification), which results in a membrane conductance of less than 1 nS for a slightly depolarized isolated pyramidal neuron. Throughout network activity, this yields around 2 nS during Down states, strongly fluctuating membrane conductance during subthreshold Up state periods, and a strong negative conductance lasting on average 32 ms before each spike event. These low and even negative long-lasting conductances are typically not reported in the experimental literature. Therefore, the dynamics of membrane conductance in the model merits further investigation. The present model successfully reproduces the network response to blockade of AMPA, NMDA, or GABA channels. However, the sensitivity of the present model to changes in the synaptic strengths is weaker than in the original model.

We could not obtain the desired network regime with the synaptic weights reported in the original work after calibration to account for the change in the synaptic model. We suggest the synaptic dynamics to be the source of discrepancy, as suggested by the inconsistency between our synapse implementation and the results shown in the original Figure 5C.”

We corrected all the minor issues. Also, we noticed some superfluous variables in the neuron model code, and removed these. Further, the axial conductance now is called 'g_conn' for consistency with other NEST models.

Answer to reviewer 2:

1. The code is now adapted to Python 3.
2. We added the instruction for the conda fix.
3. We added texlive-latex-extra and dvipng installation instructions.

@rossant @mstimberg Can you check @andruhamax answers and tell me if you can make your decision ?

@rossant @mstimberg 🔔

@rougier The authors answered to all of my comments and I now accept the submission.

REVIEWER 1

The latest updates address all of my concerns. The discrepancies in synaptic weights to the original paper are still somewhat unsatisfying, but with the additional remarks in the text the observed differences and the authors' reasoning are now clear enough IMO. I therefore recommend the acceptance of the submission.

Just a few final minor remarks/suggestions (no need to do another round of reviews for those, I think):

• The README file now states that "The network model and further analysis are implemented with Python (v.3.5.2)", while the article abstract states "The model and analysis scripts are implemented using Python 2.7.6" -- this should be unified, stating that it works with both.
• The code for Compte2003_R_method.py still has instances of "floats instead of ints", leading to warnings like the following:

Compte2003_R_method.py:74: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  time_temp = time[t_start:t_stop]

• Caption Figure 2: "DC currents, simultaneous injected" --> "simultaneously"
• Table 3, Spike detection: "no spike emission is allowed during a fixed time" -- maybe state the actual time used (I think it is 1ms)?

@rossant @mstimberg Thank you.
@andruhamax Congratulations, your submission has been accepted. Can you address the last comments and tell me when it's done?

@andruhamax 🔔

@andruhamax Thank you. Last question, can you give me some keywords ?

Python, Neuroscience, NEST, Network Modeling, Up-Down Oscillations, Cortex, Membrane Resistance Measurement

@andruhamax fig9.png is missing in code directory, woudl it possible to add it (to rebuild the pdf) ?

@andruhamax And the code repository needs a license

Publication has been published and will appear soon on https://rescience.github.io/read