Thanks for your submission. I will edit it and will invite reviewers soon.
Don't hesitate to ask here for progress in the review.

@anne-urai @qihongl Could you review this submission ?

Hi @rougier, I'm happy to review and will aim to do so before the end of this week.

@anne-urai Great, thank you !

Dear @hstojic, @rougier,

I have ran the code provided (using Rstudio 1.0.153, on my Macbook Pro laptop running OS X El Capitan) and I was able to reproduce the author's original figures. I applaud the author for reviving this classic, and found the paper generally clear and concise. I have a few suggestions for further improvement.

Major suggestions:

• The author's point about the difference in the asymptote of P(Undecided) is an important one, which could be strengthened by visualising data that's already present. Specifically, I suggest including a figure with all simulations overlaid (possibly in separate panels for the 3 types of alleles), and a histogram showing the range of endpoints after 50 generations. For comparison, the endpoint of the original Hinton & Nowlan allele frequencies could be added. This way, it should become clear if the original figure could have resulted from a draw of the same process as specified here. Showing that the original draw does not fall within the simulation distribution would give more weight to the argument that the original fitness function used was a flawed one.
• In Algorithm 1, line 14, 1-19 should be 1+19 (according to both the R code and the original Hinton & Nowlan paper).

Suggestions for improving the useability of the code:

• rather than listing the necessary R packages in the README, it would be helpful if util.R could automatically check the user's installed packages, and install any missing ones.
• since the simulations take quite long, please output some form of a status message - preferably including time spent and estimated time to finish.
• when simulating only 1 trace, no plot is created, and when simulating 2 traces, there are no grey error bars visible. Please check that the code can deal with a single trace, and confirm that error bars are indeed plotted but barely visible due to the high number of simulations.
• the code is well commented, but the link between the lines in Algorithm 1 and the R script are not immediately clear. It would be very helpful to explicitly refer to sections of Algorithm 1 in the code.

Suggestions for the paper:

• consider providing a legend with Algorithm 1, explaining the notation used. For readers more familiar with code than mathematical notation (myself, and probably many cognitive scientists), it would help a lot to clarify e.g. \forall and why |G| is used rather than noAgents.
• a quick Google search for "Hinton Nowlan code" returns several hits for previously published code (e.g. here and here). While I do not think this is a problem for the current submission, please remove from the paper "I am also not aware of other implementations elsewhere."
  • More generally, what is @ReScience/editors take on algorithms which implementation is available elsewhere, albeit in a less clear and non-peer reviewed form?
• If you have their permission, it would be nice to mention which of the author you've corresponded with, and summarize the discussion.
• I find "This result has made a substantial impact in the cognitive science literature." weak without accompanying citations.
• For the reader unfamiliar with the topic, it may be nice to briefly recap the of the Baldwin effect and cite the original paper. Specifically, I would contrast this with Lamarckian inheritance where learned information is passed on to the offspring's genotype. Note that this is not of vital importance since the reference article explains this well, but it would still be nice to very briefly recap the debate to which the reference paper applies.
• In footnote 1, it would be helpful to explicitly state original figures. Also, a full stop at the end of the footnote is missing.

Sorry for the delayed response! The notification was buried by other messages...

I would love to review it! I will try to respond by the end of this week!

@anne-urai @qihongl Thank you both for accepting the review. I'm supposed to write now that review can start but @anne-urai already sent her review 😄.

@hstojic You can wait for second review to make suggested changes if your prefer to address comments all at once.

@rougier @anne-urai @qihongl Thank you very much for accepting the editor/reviewer roles and for being so prompt about it.

I will wait for Quihong's coments and then address both reviews.

Hi, Folks ( @hstojic , @rougier, @anne-urai ),

Sorry for the waiting! Nice replication! I was able to run your code (R version 3.4.0 (2017-04-21), MacOS 10.12.6). I really enjoyed reading Hinton and others' early work! @anne-urai nicely covered many points in my mind. Though I do have some additional thoughts that might be worth considering.

First of all, I think having a clean and well-documented code for the referenced work is valuable, since the original paper is of interest to many people. It is highly cited. Multiple attempts of replicating the result also reflect people's interest.

I really appreciate your effort for trying to understand the interesting difference between your result and the original result. I think it would be even better if you can recover the original plot, if the original finding is flawed as you said. I agree with @anne-urai that summarizing your discussion with the author would be very informative. However, in my opinion, successfully replicating the original plot is more convincing than the author's approval. I guess some potential readers of your work would be interested to see concrete evidence as well.

By the way, in the MATLAB replication listed by @anne-urai, the frequency of learnable genes converged at around 0.2. (instead of decreasing monotonically). This trend is more similar to the original result. Understanding how does this implementation differ from yours might be useful for recovering the original finding. (Thanks! @anne-urai)

Second, your code can be slightly more general. For example, the number of alleles in your code is set to 20. I think it is useful to let it be a parameter that user can specify at the beginning of the simulation code, so that the user can explore some potential effect of varying the number of alleles. One reason why this is interesting to explore is articulated here: https://egtheory.wordpress.com/2014/02/07/learning-guide-evolution/

This is just one example. My general point is that, though the purpose of the code (and this journal) is to replicate the finding, having slightly a more generic implementation (that goes above and beyond replicating the original result) might provide additional educational value.

Some detail:

1. errorbar
   "...barely visible grey ribbons are standard errors"

• How many standard errors? (alternatively, It might be helpful to translate them to a (1-alpha) confidence interval.)
• I replicated what @anne-urai mentioned, so I share her concerns.

2. In table 1, the description of p_x is "Probability that allele is of type x". It is probably more reader friendly to say what "type x" is. For example, I would describe p_? as "Probability that allele is genetically undecided (or learnable?)". Anyway, make sure the choices of language are consistent with the plots.

3. I found using "noSim", "noAgents" to represent the number of simulations and the number of agents to be slightly confusing. I would use "numSim" and "numAgent", or "nSim" and "nAgent". Nevermind if your choice is commonly accepted coding in your field though...

4. At the beginning of the simulation file, It might be useful to add "rm(list=ls())" to remove all environmental variables and functions.

5. Finally, I think some wording can be slightly more concise.

Example 5.1:

• "The main aim of this article is to replicate the simulations reported in the reference article [4]. I have corresponded with one of the authors and learned that the original implementation of the simulations is no longer available. I am also not aware of other implementations elsewhere. The simulations are relatively simple, and I thus propose a replication based on description from the reference article. Replication code relies on R programming language [6] and several R packages [8, 10, 9, 1, 2, 3, 7, 5]."

Something like the following might be better?

• "I replicated the reference article [4] in R [6] with some auxiliary packages [8, 10, 9, 1, 2, 3, 7, 5], because other existing replications are ... "

Example 5.2:
The following discussion might be unnecessary if you can recover the original plot:

"... One explanation is that the results reported in Figure 2 in the reference article stem from a non-typical draw. This is unlikely, as upon examining the evolutionary paths of Undecided alleles in all simulation runs, they all show a strong downward trend already in the first 50 generations, differing only in the exact generation when this decrease onsets."

Thank you very much for spending the time to read my thoughts. I hope some of them are useful. And feel free to implement them as you see fit!

Thank you all for quick responses and all the comments - they are very helpful and will allow me to improve the article. I'll think about all the issues raised in the next few days and give you my response.

Response to reviews

I provide a comment below each suggestion of the reviewer.

Anne

Dear @hstojic, @rougier,

I have ran the code provided (using Rstudio 1.0.153, on my Macbook Pro laptop running OS X El Capitan) and I was able to reproduce the author's original figures. I applaud the author for reviving this classic, and found the paper generally clear and concise. I have a few suggestions for further improvement.

Major suggestions:

• The author's point about the difference in the asymptote of P(Undecided) is an important one, which could be strengthened by visualising data that's already present. Specifically, I suggest including a figure with all simulations overlaid (possibly in separate panels for the 3 types of alleles), and a histogram showing the range of endpoints after 50 generations. For comparison, the endpoint of the original Hinton & Nowlan allele frequencies could be added. This way, it should become clear if the original figure could have resulted from a draw of the same process as specified here. Showing that the original draw does not fall within the simulation distribution would give more weight to the argument that the original fitness function used was a flawed one.
  • HS: Indeed, this is how I checked it, thanks for the suggestion for adding it to the paper. I opted for adding a figure with paths and a histogram of end points for Undecided alleles since these are the focus of that discussion.
• In Algorithm 1, line 14, 1-19 should be 1+19 (according to both the R code and the original Hinton & Nowlan paper).
  • HS: Thank you for spotting this, corrected!

Suggestions for improving the useability of the code:

• rather than listing the necessary R packages in the README, it would be helpful if util.R could automatically check the user's installed packages, and install any missing ones.

  • HS: Good point. I actually modified the code so that utils.R automatically installs the version of the package with which the code is developed. This is really needed so that the code does not break with more recent version of the packages. It will overwrite the packages if some of them are already installed though.

• since the simulations take quite long, please output some form of a status message - preferably including time spent and estimated time to finish.

  • HS: Good suggestion, I provided some rudimentary estimates.

• when simulating only 1 trace, no plot is created, and when simulating 2 traces, there are no grey error bars visible. Please check that the code can deal with a single trace, and confirm that error bars are indeed plotted but barely visible due to the high number of simulations.

  • HS: Thanks for the suggestion. I have made sure the figure is plotted regardless of the number of simulations. I also confirmed that error bars are plotted (computation before was hardcoded, thats why ribbons were not showing with 2 simulations).

• the code is well commented, but the link between the lines in Algorithm 1 and the R script are not immediately clear. It would be very helpful to explicitly refer to sections of Algorithm 1 in the code.

  • HS: Algorithm 1 is only a pseudocode, the structure of the code itself is a bit different for efficiency reasons. I have put more comments in the code trying to connect it to the Algorithm 1 more clearly. I also changed the names of some variables in the code to make it more similar.

Suggestions for the paper:

• consider providing a legend with Algorithm 1, explaining the notation used. For readers more familiar with code than mathematical notation (myself, and probably many cognitive scientists), it would help a lot to clarify e.g. \forall and why |G| is used rather than noAgents.

  • HS: I have eliminated the math notation.

• a quick Google search for "Hinton Nowlan code" returns several hits for previously published code (e.g. here and here). While I do not think this is a problem for the current submission, please remove from the paper "I am also not aware of other implementations elsewhere."

  • HS: It seems I have used wrong keywords, but true, I did not put a serious effort in searching for the code, so I dropped the sentence. And thanks for spotting them, It will be useful for comparison, to see if we get similar results.

• More generally, what is @ReScience/editors take on algorithms which implementation is available elsewhere, albeit in a less clear and non-peer reviewed form?

• If you have their permission, it would be nice to mention which of the author you've corresponded with, and summarize the discussion.

  • HS: Email exchange with Geoff Hinton was brief, what I have provided is a summary. In another correspondence he agreed to name him here.

• I find "This result has made a substantial impact in the cognitive science literature." weak without accompanying citations.

  • HS: I was referring primarily to the number of citations, so I rephrased the sentence in that manner. I also saw it on reading lists in machine learning / AI courses.

• For the reader unfamiliar with the topic, it may be nice to briefly recap the of the Baldwin effect and cite the original paper. Specifically, I would contrast this with Lamarckian inheritance where learned information is passed on to the offspring's genotype. Note that this is not of vital importance since the reference article explains this well, but it would still be nice to very briefly recap the debate to which the reference paper applies.

  • HS: I have thought this was relatively clear from the introduction. I have put the footnote about the Baldwin effect in the text now and tried to contrast them more clearly.

• In footnote 1, it would be helpful to explicitly state original figures. Also, a full stop at the end of the footnote is missing.

  • HS: I am not sure that I understand the suggestion. It does not make sense to put whole captions, they are rather big.

Quihong

Hi, Folks ( @hstojic , @rougier, @anne-urai ),

Sorry for the waiting! Nice replication! I was able to run your code (R version 3.4.0 (2017-04-21), MacOS 10.12.6). I really enjoyed reading Hinton and others' early work! @anne-urai nicely covered many points in my mind. Though I do have some additional thoughts that might be worth considering.

First of all, I think having a clean and well-documented code for the referenced work is valuable, since the original paper is of interest to many people. It is highly cited. Multiple attempts of replicating the result also reflect people's interest.

• I really appreciate your effort for trying to understand the interesting difference between your result and the original result. I think it would be even better if you can recover the original plot, if the original finding is flawed as you said. I agree with @anne-urai that summarizing your discussion with the author would be very informative. However, in my opinion, successfully replicating the original plot is more convincing than the author's approval. I guess some potential readers of your work would be interested to see concrete evidence as well.

  • HS: I agree that would be great, but without the original code it would be too difficult to replicate the original result, I simply do not see how to figure that out in an efficient manner. I would welcome any suggestions on how to do that.

• By the way, in the MATLAB replication listed by @anne-urai, the frequency of learnable genes converged at around 0.2. (instead of decreasing monotonically). This trend is more similar to the original result. Understanding how does this implementation differ from yours might be useful for recovering the original finding. (Thanks! @anne-urai)

  • HS: The Matlab code actually has a bug in lines 66-68, and it runs the simulation only once. When these issues are resolved I get the same results as in my simulations. This suggests that I correctly implemented the simulations.

• Second, your code can be slightly more general. For example, the number of alleles in your code is set to 20. I think it is useful to let it be a parameter that user can specify at the beginning of the simulation code, so that the user can explore some potential effect of varying the number of alleles. One reason why this is interesting to explore is articulated here: https://egtheory.wordpress.com/2014/02/07/learning-guide-evolution/

  • HS: Thanks. I made it a parameter defined at the top of the script. And thanks for the reference, that was an interesting discussion on limitations of the original simulations.

This is just one example. My general point is that, though the purpose of the code (and this journal) is to replicate the finding, having slightly a more generic implementation (that goes above and beyond replicating the original result) might provide additional educational value.

Some detail:

• errorbar "...barely visible grey ribbons are standard errors". How many standard errors? (alternatively, It might be helpful to translate them to a (1-alpha) confidence interval.)

  • HS: Thanks. I made it more precise in the caption. These are usual standard errors of the mean, sd/sqrt(N).

• I replicated what @anne-urai mentioned, so I share her concerns. In table 1, the description of p_x is "Probability that allele is of type x". It is probably more reader friendly to say what "type x" is. For example, I would describe p_? as "Probability that allele is genetically undecided (or learnable?)". Anyway, make sure the choices of language are consistent with the plots.

  • HS: Thanks. I made ? more reader friendly, while 0 and 1 has to stay as they are, as there is no correspondence with figures - Correct allele can be either 1 or 0.

• I found using "noSim", "noAgents" to represent the number of simulations and the number of agents to be slightly confusing. I would use "numSim" and "numAgent", or "nSim" and "nAgent". Nevermind if your choice is commonly accepted coding in your field though...

  • HS: This choice is arbitrary, no. is usual acronym for numbers I believe, but indeed, choices might differ in different fields. I would stick with the one I used.

• At the beginning of the simulation file, It might be useful to add "rm(list=ls())" to remove all environmental variables and functions.

  • HS: Thanks, I corrected it.

Finally, I think some wording can be slightly more concise.

Example 5.1:

• "The main aim of this article is to replicate the simulations reported in the reference article [4]. I have corresponded with one of the authors and learned that the original implementation of the simulations is no longer available. I am also not aware of other implementations elsewhere. The simulations are relatively simple, and I thus propose a replication based on description from the reference article. Replication code relies on R programming language [6] and several R packages [8, 10, 9, 1, 2, 3, 7, 5]."

Something like the following might be better?

• "I replicated the reference article [4] in R [6] with some auxiliary packages [8, 10, 9, 1, 2, 3, 7, 5], because other existing replications are ... "
  • HS: Thanks, I shortened it a bit.

Example 5.2:

• The following discussion might be unnecessary if you can recover the original plot: "... One explanation is that the results reported in Figure 2 in the reference article stem from a non-typical draw. This is unlikely, as upon examining the evolutionary paths of Undecided alleles in all simulation runs, they all show a strong downward trend already in the first 50 generations, differing only in the exact generation when this decrease onsets."
  • HS: See my response to your first comment.

Thank you very much for spending the time to read my thoughts. I hope some of them are useful. And feel free to implement them as you see fit!

Dear @rougier @anne-urai and @qihongl,

thanks again for all the comments, I have updated the article reflecting all your suggestions, let me know if you are satisfied with the changes and the final article and the code.

Best,
Hrvoje

Thanks. @anne-urai @qihongl are you satisfied with all the modifications by @hstojic ?
If this is the case, can you post your decision here?

Thanks! @hstojic @rougier

I am traveling for a conference this week and I will get back to you ASAP!

Hi @hstojic,

Unfortunately your changes to package management seem to have caused the code to malfunction in my installation of R. devtools should be loaded also when already installed, and the packages should only be installed if not already present (now, it seems like they will be re-installed at each run of the code? e.g. https://gist.github.com/stevenworthington/3178163).

If I manually set the packages and uncomment this part of the code, it runs smoothly. It would be great if you could still check this, so that future users can run the script easily.

Hi @anne-urai,

thanks, I have modified the code so that now it first checks whether a package exists before installing. Nevertheless, it still check whether a correct version exists and installs it even if it finds a different version. I think this is important as packages do change over time and the code might not work in the future otherwise.

Best,
Hrvoje

Hi @hstojic, @rougier,

thanks for the quick update. The code now runs smoothly, and all my comments have been addressed. I recommend publication of the paper. Thank you for making this code available to the community!

Thanks @anne-urai. @qihongl Are your back from your conference? Did you make your decision ?

Sorry for the delayed response. I'm overall happy with all of the modifications. I was just slightly concerned about the differences between the replication and the original result.

Since the referenced paper is highly cited, to claim that the original implementation is flawed, I think we're responsible for providing a crystal clear explanation. And I think this replication would be more impactful if we can understand exactly what was going on. I do understand that it might be hard to recover the original result. In this case, it might be useful provide a thorough mathematical analysis of why the number of undecided alleles should decrease continuously over time.

What do you folks think? @anne-urai @hstojic @rougier If people agree that a more precise explanation of the deviation from the original result is not necessary, I would recommend this paper for publication. The current result does show learning can act as "2nd order mutation" that guide exploration over certain optimization landscapes, which was the main point of the original paper.

By the way, overriding user's packages might not be a good idea. I guess a better practice is to use a container (e.g. Docker) or getting a virtual machine image.

Dear @qihongl,

thanks for a prompt response.

I agree, I would be happier as well if I could explain exactly what went wrong. Nevertheless, in my opinion tracking down the source of it is beyond the scope of a replication, and same holds for doing an additional mathematical analysis. Perhaps I would be tempted to put more effort in getting at the bottom of it if it weren't for correspondence with Geoff where he agreed with the replication results. I have also checked Java and Matlab code found by @anne-urai and with those same results are obtained (with Matlab code after correcting some bugs). Finally, the main result and conclusion of the paper still holds, as you also indicated.

Indeed, overriding packages is not nice, but I thought it's necessary. Perhaps a Docker is a better idea, do @rougier and @anne-urai have opinion on this? That might be a good idea for a general journal policy, to make the code in the articles more reproducible (independent of operating systems etc).

Best,
Hrvoje

@qihongl I see your point but in the same time it seems impossible to recover the original code/result. As for a thorough mathematical analysis, I think it may be beyond the scope of this replication (but I've no estimation of how much work this would represent). @hstojic Do you have any idea on that matter ?

Another option would be to develop such an analysis in a new paper (I mean not a replication) if it brings new interpretation.

Genetic algorithms are not really my domain of expertise, so I'm not sure how much work would it be. If I would put an effort into it, then I think the paper would go beyond a replication and I could submit it to a different journal as well.

@hstojic Make sense. @qihongl I think we can accept the submission as it is (modulo the docker question) but I would need you to confirm your decision.

The docker question has been raised a couple of times and we have no definitive answer. I'm not familar at all with R so I've no real opinion (I'm not sure what it means to override packages). Can't you use virtual environment of local packages ?

I see! Then I would recommend this paper for publication! Thank you very much again @hstojic !

BTW @hstojic sorry about asking too much... I used to get stuck on some research project due to seemingly irrelevant differences between the replicated result vs. the original result. And I still have a little bit of PTSD I guess... lol

I therefore accept the submission. Congratulations @hstojic and many thanks @qihongl @anne-urai for the fast and detailed reviews.

The paper will soon be published.

Here is the DOI:

The article will be soon listed on http://rescience.github.io/read/

Done ! 🎉