Thanks for your submission. An editor will soon be assigned.

Dear @Vahidrostami, I just realized that one of the new author (Sonja Grün) is also one of the author of the original article. Could you tell me a bit more about that. It's ok since the paper is alsmost 20 years old but I wonder what langage was the original implementation and if the new implementation is a brand new one or a simple adaptation.

Dear @rougier, The current implementation of the Unitary Events analysis method, which is the basis for the reproduced findings, is a completely new reimplementation of the analysis method in Python written by author VR (myself). This reimplementation was done from scratch without access to the original codes.

As we detail in the manuscript, tracing the provenance of the original figures was a major part of our reproduction. It turned out, that (i) two different analysis codes (one in IDL, the other in Matlab) were used for the original publication and (ii) that they are either no longer available (IDL version), or only available in a version that was heavily modified over the years to include additional features, however, without version control or unit testing (Matlab version). Thus, the reimplementation was a necessity. For maximum benefit for the community, the method was integrated as part of a recently established resource for tools for neuronal activity data analysis, the Electrophysiology Analysis Toolkit (Elephant). In the manuscript, we recount the detailed process of tracing what was done in the original study, which was a necessary step to arrive at the final reproduced figures using the reimplemented method.

Thank you for the detailed explanation. In the meantime, I've started to read the paper where it is also nicely explained.

@Vahidrostami I will edit your submission and will soon assign two reviewers.

@rougier, Thank you very much for the quick reply.

In the meantime, you need to update the README.md file in the code directory to explain requirements and dependencies. And you need to add a LICENSE.txt in the code directory.
For the root directory, you can also update the README.md to explain what this repository is about (you can check for example https://github.com/ReScience-Archives/Vitay-2016)

Ok. I will update the repository of our submission in the upcoming days.

@rossant @apdavison Can you review this submission ?

I have an existing collaboration with Sonja Grün; I also contribute to the Elephant toolbox mentioned in the article. Perhaps this is too much of a conflict of interest?

@apdavison Thank you for the quick answer and yes, the existing collaboration might be a problem. Would you have some name to suggest (someone familiar with Elephant and Neo) by any chance ?

@benoit-girard Could you review this submission ?

@rougier I just updated:

1. the code repository by adding a Licence for our code and explaining the requirements and dependencies.
2. the README.md in the root directory to explain what this repository is about.

Please take a look and let me know if something is missing. Thanks

@rougier unfortunately I don't have time to do a full review, although I could do a quick code review if that's useful...

@rossant Thanks for the answer.

Authors addressed all of my comments. The source code is now commented and running out-of-the-box (bugs have been fixed). Authors have compared their results with the original ones using a more rigorous method. I recommend acceptance of the current version. Below I provide some minor comments regarding mostly some typos.

Minor comments

1. page 7, Results section, first paragraph, first sentence: "the data still available -> since for the [...] available anymore" -> since for the rest of the figures data were either incomplete [...] or not available.
2. page 7, Results section, first paragraph, last sentence: "arbitrary numbers of neurons" -> arbitrary number of neurons
3. Page 10, Results section, third paragraph: "spike times correspond to to the" -> spike times correspond to the
4. Page 12, Conclusion, first paragraph, third sentence: "This difference in ... in the results"-> This difference in the alignment would not affect the results
5. Page 15, Conclusion, fifth paragraph, second sentence: "the reproduction is based the values"-> is based on the values
6. Could you please provide an English version of Figure 5?

@gdetor Thanks.
@benureau Are you satisfied with the corrections?
@Vahidrostami Can you address the minor comments from @gdetor ?

@rougier
@gdetor Thanks a lot for the careful reading. I have fixed the typos and provided an English version of Figure 5.

The authors have done a remarkable job of addressing my comments. I particularly appreciate the quantitative analysis provided by the new Figure 3. So much that I have a couple of remarks about it:

Figure 3 not reproduced

• The Plotting_figure.py code and the notebook talk about reproducing Figure 3, which is now Figure 6. And Figure 3, which is quite interesting, is not reproduced. Since extracting the data from the original plot was not trivial (involved extracting command in the PDF), it would be valuable for this data to be shared in the replication.
• And even better, the code for extracting the data from the PDF could be made available too, as it should be valuable for other ReScience efforts.
• Figure 3C is a bit busy. I would advise plotting differences with spikes and UEs in two different figures (this could allow to plots differences in UEs for Figure 1 as well).
• Figure 3 is missing all units; it should be on the figures or in the caption.

Ease of use

• Saving figure with Plotting_figure.py to disk should be by default. I use the Agg renderer with matplotlib, which means—being a non-interactive renderer—that show() has no effect for me.
• I get a superposition of Figure 1 and Figure 2 when saving to disk (a detail, I can live without it being solved.):
• When I talked about a requirements.txt file, I talked about a file compatible with pip (cf. https://pip.pypa.io/en/stable/user_guide/#requirements-files), so that all dependencies could be installed with a pip install -r requirements.txt. This is easily done by replacing the current one:

- Elephant>=0.5.0 (https://github.com/NeuralEnsemble/elephant)
- neo>=0.4.0 (https://github.com/NeuralEnsemble/python-neo)
- quantities>=0.9.0
- numpy>=1.6.2
- matplotlib>=1.5.1

by

Elephant>=0.4.0
neo>=0.4.0
quantities>=0.9.0
numpy>=1.6.2
matplotlib>=1.5.1

By the way, I assume you meant Elephant>=0.4.0, not 0.5.0 which is not released yet (latest release is 0.4.1, README.md is affected as well) ?

• Optional setup and run instructions for non-apt-get users:

cd code
pip install -r requirements.txt
python Plotting_Figures.py

• The notebook is missing three sets of parenthesis to be Python 3 compatible (and seems slightly behind the .py version), for instance, in the last cell:

  File "<ipython-input-5-90fba530d061>", line 16
    print 'calculating UE ...'
                             ^
SyntaxError: Missing parentheses in call to 'print'

Conclusions

Beyond those remarks, I fully recommend accepting the contribution.

@benureau Thanks for the review !
@Vahidrostami Congratulations, your submission is accepted. Could you address the remaining minor comments before we proceed with publication ?

Following a recent discussion (ReScience/ReScience#43) about the non-reproducibility of articles published in ReScience, I did a quick test on this one using the accepted but not yet published code. As explained in the earlier discussion, all I checked is if I can install the dependencies and run the software - I don't look at the results, since I am not competent in judging them anyway.

First remark: the instructions quote version numbers for everything except Python itself. In particular, it isn't clear if I should use Python 2 or Python 3.

Using Python 3.5 under macOS 10.11, I installed the dependencies via pip and ran Plotting_Figures.py. The output:

plotting Figure 1 with trigger:  PS_4 ...
plotting raster plot ...
plotting Spike Rates ...
plotting Raw Coincidences ...
plotting emp. and exp. coincidences rate ...
plotting Surprise ...
plotting UEs ...

plotting Figure 2 with trigger:  RS_4 ...
plotting raster plot ...
plotting Spike Rates ...
plotting Raw Coincidences ...
plotting emp. and exp. coincidences rate ...
plotting Surprise ...
plotting UEs ...

plotting Figure 3 ...
calculating UE ...
calculating UE ...
/Users/hinsen/Temp/ReScience-submission-VRostami-JIto-MDenker-SGruen-2016/code/utils.py:455: RuntimeWarning: invalid value encountered in greater_equal
  Js_dict_lst[N_cnt]['Js'] >= Js_sig)[0]

If that warning is expected and harmless, it would be nice to mention this in the instructions.

Thanks @khinsen, I totally forgot this discussion and your proposal of including a third reviewer not from the field to test for the correctness of the instruction.

@Vahidrostami Do you know if the warning is expected ?

@rougier This is expected. This happens because the array contains nan when it tests for greater_equal. This warning occurs with python 3 not python 2. As we wrote the whole code in python 2 this warning slipped our attention. However as the reviewer suggested we are making the code compatible with Python 3.

@khinsen Thanks for your test. In the final version we will take care of this warning.

@rougier
@benureau Thanks a lot for your comments. Please find below a detailed response to your comments:
"

Figure 3 not reproduced

• The Plotting_figure.py code and the notebook talk about reproducing Figure 3, which is now Figure 6. And Figure 3, which is quite interesting, is not reproduced. Since extracting the data from the original plot was not trivial (involved extracting command in the PDF), it would be valuable for this data to be shared in the replication.
• And even better, the code for extracting the data from the PDF could be made available too, as it should be valuable for other ReScience efforts.
  "

Regarding these two points, we are happy to share our code for extracting plot points from a PDF file, but we do not think that it should be included into the publication, for the following reason.
We implemented our code so that it does NOT directly read the original PDF file, but it reads an EPS file (which is a text file, much easier for a Python script to read) containing only the figure part of the original PDF.
We generated this EPS file, using Adobe Acrobat XI Pro on Windows OS, by deleting everything in the original PDF document except for the figure part and then exporting the result to an EPS file.
Thus, if we are to submit the code in a form testable by reviewers, we also need to submit this EPS file, which is a modified version of the original publication, but we guess that re-distributing a modified version of a publication is generally prohibited in most licenses.
We would suggest, instead, to publish the code only (but with a documentation on how to prepare the EPS file) as supplementary information or something like.
@rougier We wonder if such option is available in ReScience journal.

"

• Figure 3C is a bit busy. I would advise plotting differences with spikes and UEs in two different figures (this could allow to plots differences in UEs for Figure 1 as well).
  "
  As the reviewer suggested, we have separated the distribution of differences for spikes and UEs in two different subplots.
  Plotting the distribution of differences for UEs in Figure 1 is not possible as the number of UEs extracted from the original publication differs from the reproduced one in Figure 1.

"

• Figure 3 is missing all units; it should be on the figures or in the caption.
  "
  We thank the reviewer for pointing this out. We added the missing units to Figure 3.

"

Ease of use

"
We have corrected all the mentioned points.

@Vahidrostami: I think sharing the code for extracting the data from the EPS and descriptive instructions on how the EPS was created, along with the data generated and the code of Figure 3, would be perfectly fine for me.

Maybe you can illustrate the technique on a "fake" PDF file. Fro extracting the figure part, you can also use inkscape to open the PDF directly and then remove pretty much everything.

@Vahidrostami Byt the way, it it's too much trouble, I think we can publish without this specific part.

@rougier It clearly goes way beyond this particular review, and IANAL, and the risks are possibly not worth taking, but this may fall under Fair Use (http://fairuse.stanford.edu/overview/fair-use/what-is-fair-use/). Transformative, partial, with the intent to comment, and "the public reaps benefits from your review".

Also, we could ask.

I'll wait for @Vahidrostami to answer if this too much work or not. Since we've already accepted the publication, it might not be fair to delay the publication too much because of this.

@Vahidrostami What is your decision concerning @benureau request ? As I said, we can publish now without the requested supplementary code or we can wait a bit more if you intend to supply it. Juste tell us.

@rougier As @benureau suggested, we have added now the following codes:

• A script for extracting the data from the EPS file (extract_values_from_eps.py in the code folder). As we discussed before this code can not be run, as the reduced EPS file from Riehle et al (1997) does not exist. However, there is a detail description in the documentation of the code on how to generate the reduced EPS file.
• The code for plotting Figure 3. This plot needs the extracted values from the EPS file which are provided with extracted_data.npy in the data folder.

@Vahidrostami: Thanks for this additional material.
@rougier: Everything is good for me.

Sorry, I'm late for publishing this submission, will do by end of next week.

This submission has been published and will soon appear at http://rescience.github.io/read/