Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

The case for open computer programs

Nature volume 482, pages 485–488 (2012)Cite this article

Subjects

Abstract

Scientific communication relies on evidence that cannot be entirely included in publications, but the rise of computational science has added a new layer of inaccessibility. Although it is now accepted that data should be made available on request, the current regulations regarding the availability of software are inconsistent. We argue that, with some exceptions, anything less than the release of source programs is intolerable for results that depend on computation. The vagaries of hardware, software and natural language will always ensure that exact reproducibility remains uncertain, but withholding code increases the chances that efforts to reproduce results will fail.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to the full article PDF.

39,95 €

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

References

  1. Schwab, M., Karrenbach, M. & Claerbout, J. Making scientific computations reproducible. Comput. Sci. Eng. 2, 61–67 (2000)

    Article  Google Scholar 

  2. Barnes, N. Publish your computer code: it is good enough. Nature 467, 753 (2010)

    Article  CAS  Google Scholar 

  3. McCafferty, D. Should code be released? Commun. ACM 53, 16–17 (2010)

    Article  Google Scholar 

  4. Merali, Z. …Error. Nature 467, 775–777 (2010)

    Article  CAS  Google Scholar 

  5. Hanson, B., Sugden, A. & Alberts, B. Making data maximally available. Science 331, 649 (2011)

    Article  CAS  ADS  Google Scholar 

  6. Peng, R. D. Reproducible research and biostatistics. Biostatistics 10, 405–408 (2009)This work provides a succinct and convincing argument for reproducibility— Biostatistics is at the forefront of ensuring that code and data are provided for other researchers.

    Article  Google Scholar 

  7. Devil in the details. Nature 470, 305–306 (2011)

  8. Pedersen, T. Empiricism is not a matter of faith. Comput. Linguist. 34, 465–470 (2008)

    Article  Google Scholar 

  9. Donoho, D. L. An invitation to reproducible computational research. Biostatistics 11, 385–388 (2010)

    Article  Google Scholar 

  10. Raymond, E. S. The Cathedral and the Bazaar (O’Reilly, 2001)

    Google Scholar 

  11. He, Y. & Ding, C. H. Q. Using accurate arithmetics to improve numerical reproducibility and stability in parallel applications. J. Supercomput. 18, 259–277 (2001)

    Article  Google Scholar 

  12. Hatton, L. et al. The seismic kernel system—a large scale exercise in Fortran 77 portability. Softw. Pract. Exper. 18, 301–329 (1988)

    Article  ADS  Google Scholar 

  13. Hatton, L. The T experiments: errors in scientific software. IEEE Comput. Sci. Eng. 4, 27–38 (1997)

    Article  Google Scholar 

  14. Hey, T. Tansley, S. Tolle, K., eds. The Fourth Paradigm: Data-Intensive Scientific Discovery (Microsoft Press, 2009)

  15. Gervasi, V. & Zowghi, D. On the role of ambiguity in RE. In Requirements Engineering: Foundation for Software Quality (eds Wieringa, R. & Persson, A ) 248–254 (Springer, 2010)

    Book  Google Scholar 

  16. van Deemter, K. Not Exactly: In Praise of Vagueness (Oxford University Press, 2010)

    Google Scholar 

  17. Yang, H., Willis, A., De Roeck, A. & Nuseibeh, B. Automatic detection of nocuous coordination ambiguities in natural language requirements. In Proc. IEEE/ACM Intl Conf. on Automated Software Engineering 53–62 (ACM, 2010)

    Book  Google Scholar 

  18. de Bruijn, F. & Dekkers, H. L. Ambiguity in natural language software requirements: a case study. In Requirements Engineering: Foundation for Software Quality (eds Wieringa, R, & Persson, A., ) 233–247 (Springer, 2010)

    Book  Google Scholar 

  19. Pfleeger, S. L. & Hatton, L. Investigating the influence of formal methods. IEEE Computer 30, 33–43 (1997)

    Article  Google Scholar 

  20. van der Meulen, M. J. P. & Revilla, M. A. The effectiveness of software diversity in a large population of programs. IEEE Trans. Softw. Eng. 34, 753–764 (2008)

    Article  Google Scholar 

  21. Boehm, B., Rombach, H. D., Zelkowitz, M. V., eds. Foundations of Empirical Software Engineering: the Legacy of Victor R. Basili (Springer, 2005)

  22. Monniaux, D. The pitfalls of verifying floating-point computations. ACM Trans. Programming Languages Systems 30 (3). 1–41 (2008)

  23. Thomas, S. J., Hacker, J. P., Desagne, M. & Stull, R. B. An ensemble analysis of forecast errors related to floating point performance. Weather Forecast. 17, 898–906 (2002)

    Article  ADS  Google Scholar 

  24. Revol, N. Standardized interval arithmetic and interval arithmetic used in libraries. Proc. 3rd Intl Congr. Mathematical Software 337–341. (2010)

  25. Rump, S. M., Ogita, T. & Oishi, S. Accurate floating point summation. Part 1: Faithful rounding. SIAM J. Sci. Comput. 31, 189–224 (2009)

    Article  Google Scholar 

  26. Rump, S. M. Accurate and reliable computing in floating-point arithmetic. Proc. 3rd Intl Congr. Mathematical Software 105–108. (2010)

  27. Badin, M., Bic, L. & Dillencourt, M. &. Nicolau, A. Improving accuracy for matrix multiplications on GPUs. Sci. Prog. 19, 3–11 (2011)

    Google Scholar 

  28. Pan, V. Y., Murphy, B., Qian, G. & Rosholt, R. E. A new error-free floating-point summation algorithm. Comput. Math. Appl. 57, 560–564 (2009)

    Article  MathSciNet  Google Scholar 

  29. Boldo, S. & Muller, J.-M. Exact and approximated error of the FMA. IEEE Trans. Comput. 60, 157–164 (2011)

    Article  MathSciNet  Google Scholar 

  30. Kahan, W. Desperately needed remedies for the undebuggability of large floating-point computations in science and engineering. IFIP/SIAM/NIST Working Conf. Uncertainty Quantification in Scientific Computing (Springer, in the press).

  31. Mesirov, J. P. Accessible reproducible research. Science 327, 415–416 (2010)

    Article  CAS  Google Scholar 

  32. Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M. & Stodden, V. Reproducible research in computational harmonic analysis. Comput. Sci. Eng. 11, 8–18 (2009)Donoho and fellow researchers have been at the forefront of reproducibility for many years; this article reviews their work, including facilities for code presentation.

    Article  Google Scholar 

  33. Yool, A., Popova, E. E. & Anderson, T. R. Medusa-1.0: a new intermediate complexity plankton ecosystem model for the global domain. Geosci. Model Develop. 4, 381–417 (2011)An example of an article from a journal that asks for code release for model description papers and encourages code release for other categories of paper.

    Article  ADS  Google Scholar 

  34. Easterbrook, S. M. & Johns, T. C. Engineering the software for understanding climate change. Comput. Sci. Eng. 11, 65–74 (2009)

    Article  Google Scholar 

  35. Brohan, P., Kennedy, J. J., Harris, I., Tett, S. F. B. & Jones, P. D. Uncertainty estimates in regional and global observed temperature changes: a new dataset from 1850. J. Geophys. Res. 111, D12106 (2006)

    Article  ADS  Google Scholar 

  36. Adams, E. N. Optimizing preventive service of software products. IBM J. Res. Develop. 28, 2–14 (1984)

    Article  Google Scholar 

  37. Hatton, L. & Roberts, A. How accurate is scientific software? IEEE Trans. Softw. Eng. 20, 785–797 (1994)

    Article  Google Scholar 

Download references

Acknowledgements

We thank D. Hales of the Department of Design at the Open University and P. Piwak of the Department of Computing for pointing out some reproducibility work outside computing. J.G.-C. is grateful to I. Goz for pointing out the calculation errors in the CRUTEM data from the Meteorological Office.

Author information

Authors and Affiliations

  1. Department of Computing Open University, Walton Hall, Milton Keynes MK7 6AA, UK,

    Darrel C. Ince

  2. School of Computing and Information Systems, Kingston University, Kingston KT1 2EE, UK,

    Leslie Hatton

  3. 83 Victoria Street, London SW1H 0HW, UK,

    John Graham-Cumming

Authors
  1. Darrel C. Ince
    View author publications

    Search author on:PubMed Google Scholar

  2. Leslie Hatton
    View author publications

    Search author on:PubMed Google Scholar

  3. John Graham-Cumming
    View author publications

    Search author on:PubMed Google Scholar

Contributions

D.C.I., L.H. and J.G.-C. contributed to all aspects of this article.

Corresponding author

Correspondence to Darrel C. Ince.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ince, D., Hatton, L. & Graham-Cumming, J. The case for open computer programs. Nature 482, 485–488 (2012). https://doi.org/10.1038/nature10836

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/nature10836

This article is cited by

Comments

Commenting on this article is now closed.

  1. Darrel Ince

    We found the word limit that Nature has for Perspective articles prevented us from making some points. This coda contains them.

    OUR INTENT
    Our intent was not to criticise; indeed we have admiration for scientists who have to cope with the difficult problems we describe. One thesis of the article is that errors occur within many systems and does not arise from incompetence or lack of care. Developing a computer system is a complex process and problems will, almost invariably, occur. By providing the ability for code to be easily perused improvement will happen. This is the result detailed in both the boxes in the article: the Met Office data is more accurate, admittedly by a small amount, and because of feedback to developers the geophysical software was considerably improved.

    OTHER REASONS FOR CODE TRANSPARENCY
    Our message is simple: because of ambiguities in natural language, hardware, and floating point representations, together with programming errors, the code that supports some published scientific result must be released. The code represents an exact description of some computational process and is far superior to a written description, even one expressed in some formal notation.

    Also, given the size of datasets and the fact that the only way to deal with them is via code, it makes no sense to speak of the data and code separately. If you release one, you must release the other.
    There are, however, a number of other reasons for code release which, because of pressure of space, we were unable to explore.

    The first is that it enables researchers to build upon other research. A module that has applicability to a specific research area can be used by other researchers. Here we have the software analogue of researchers building upon others research results.

    The second is that released code provides excellent teaching material, both to the post-graduate and undergraduate levels. One of us (DCI) has taught programming for many years. He, along with many others, has discovered that after the initial teaching of the elements of a programming language many students benefit hugely from realistic source code examples.

    The third is connected with citizen science. Increasingly there are more and more projects that encourage non-institutional researchers to get involved in research, for example the Galaxy Zoo project. Often these projects involve data collection or classification. We believe that there is scope for some really ambitious projects which would involve citizen scientists, for example statisticians and software developers, to get involved (JG-C was one of them). For these to happen both code and data transparency is needed.

    The fourth is to do with improvement. JG-Cs work has meant that the error bounds for the data in a temperature database have been narrowed, albeit in a small way, leading to a greater accuracy. This is common within the open-source environment where many such systems, for example, the web server Apache, are highly reliable.

    EXCEPTIONS
    We may get some criticisms for including the caveat that there are some exceptions. If a researcher has persuaded his institution to apply for some patent or copyright protection (a long and quite expensive process) and there is evidence of sales then this caveat clearly has to stand. However, if the researcher claims, without evidence, that the software has commercial potential we feel that this should not hinder publication of the code.

    It is also worth mentioning some other reasons for not releasing code. The first is where damage can be done. For example a biological researcher may find a way of developing some virus that could wreck havoc with the world health system and a software component forms part of the research then this should not be released.

    The second is where the possession of both software and data could lead to someone being able to carry out some criminal activity or anti-social activity. DCI recently attended a presentation that described research that involved a system that was capable of identifying a certain criminal activity. The data could never be released; neither could the code, since possession of the analysis within the code would have lead to the crime being much less easily detected.

    CLARIFICATION
    The quantity 5000 execution years is used in the article. This represents the time over what would have been thousands of concurrent executions. The Merali reference can be found with the tag ?Computational Science? fronting it in the online edition of Nature. This is an internal web-based tag that is not part of the title of the paper and we have omitted it.

    CODE DISCLOSURE
    DI has carried out four research projects over the last ten years that involve software. One has been published and software is available; a second is in the process of refereeing, but the software is available; the third involved commercial data and software used in an automotive industry project and is subject to commercial agreement; the fourth is in the early stages of research and software will be published during the latter stages of the publication process. LH makes the code and data for all recent scientific work available as category 1 or 2 on his own web-site and for commercial code as category 3. JG-C makes all his non-commercial code available freely under open source license.

    A FIRST STEP
    What we have described in this article is just a first step. Much more needs to be done. We would have liked to devote more time to the Australian National Data Service in our article, but word limits prevailed. This, we regard as a terrific model for national or subject-specific archiving and our description of it as a meta-data repository and index is just the tip of the iceberg.

    THANKS
    Could we thank Nature for accepting this paper, particularly since the content was driven by criticism of a code availability statement made in an editorial in the journal.

  2. Wayne Williamson

    Coming from a computing background this make perfect sense. In order to reproduce a set of results it is much easier to use existing code(as the article states). It would still need someone to validate that the code is doing what its supposed to.

  3. Charles Warner

    Some very strong arguments for code availability, and a critical issue for evaluating the validity of research results.

    You may want to consider adding to your list of example repositories:

    DLMF: NIST Digital Library (http://dlmf.nist.gov/) a digital version of the Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, edited by Milton Abramowitz and Irene A. Stegun, which includes a link to their "Guide to Available Software", http://gams.nist.gov/.

    And, of course, there is netlib (http://netlib.org/).

    Both of these sites offer some very practical models for archiving source code, and researchers who draw on the codes included in these archives derive the benefit that this code hs been vetted by the larger community.

    Regarding packaging, there is a very interesting Open Source platform called "KNIME" (http://www.knime.org/) that, although designed as a "data mining" platform, offers a way to package documents, software, procedures, etc. in a single "project", and it is infinitely extendable through R or user-developed external programs. I have been using it to organize my own research projects lately- handy for keeping references, notes, and data all in one place. Possibly a good model for what you call "packaging"...
    Charles Warner

  4. Thomas Dent

    This is a rather strange article, it makes a fetish of 'reproducibility' as a computing-science requirement apparently detached from scientific content. Surely we should be more interested in the reproducibility of the science regardless of coding implementation, than in any narrower definition requiring agreement to so many decimal places between code outputs.

    Scientific codes are designed to answer scientific questions &#8211 defined without reference to any details of computing &#8211 to some specified accuracy. If the question is stated precisely and unambiguously enough, the answer ought to be independent of the code implementation to within the required accuracy. This is a different criterion from requiring computer-science 'reproducibility', meaning if I run the same algorithms on the same data I should get the same result &#8211 which can only detect errors in carrying out the algorithm. But scientific questions cannot be mapped one-to-one to computing algorithms.

    As scientists we would like to be confident that research results are not merely artefacts of our chosen analysis methods. The most thorough way to verify this is to analyze the same data with an independent code designed to answer the same scientific question. We cannot expect exactly the same answer, but we can demand that any deviations between results are small enough to attribute to insignificant calculational details.

    Publication of codes may or may not help towards this. It is very difficult to eliminate errors in any code longer than a thousand lines or so, doubly so when inspecting it from 'outside'. But comparing two independently written codes can be extremely informative, revealing pitfalls and hidden assumptions. Having the original code published will inevitably reduce the independence of any verification, as people will be tempted to copy or adapt aspects of it. In the worst case we might have two codes that gave similar answers, satisfying a computing science definition of reproducibility, but both containing the same scientific error, for example an invalid approximation.

    At the back of this is the amount of effort the community is expected to put in to checking analysis results. The best way to do this is by having two or more teams doing the same science but with independent codes. If this is too much time and effort, opening code to some level of public scrutiny is at least a decent second best.

  5. Thomas Dent

    This article completely misses the main point of reproducibility in science: a result should not depend on technical details of implementation of algorithms. If the analysis were re-implemented by different people in a different computing language the results should be comparable within uncertainties. The considerations of computing-science reproducibility discussed in the article are at best tangential to this.

    It's one thing to check that a code gives the claimed results when run as documented. Even if it passes all such computational tests it may still contain scientific mistakes (eg invalid approximations or assumptions) -- these would be more likely to be revealed by having an independent team write an independent analysis without even looking at the original code.

    What would be more useful is not releasing the source code, but instead giving a set of unit tests that any analysis ought to satisfy in order to answer the desired scientific questions. The article, despite its extraordinary length and verbosity, doesn't even mention 'scientific' unit tests of this kind.

  6. zaskia alies

    The case for open computer programs i love it
    pakar seo indo
    harga besi beton untuk kontraktor

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics