Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Erroneous analyses of interactions in neuroscience: a problem of significance

Nature Neuroscience volume 14, pages 1105–1107 (2011)Cite this article

Subjects

Abstract

In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to the full article PDF.

39,95 €

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Graphs illustrating the various types of situations in which the error of comparing significance levels occurs.
The alternative text for this image may have been generated using AI.

Similar content being viewed by others

References

  1. Gelman, A. & Stern, H. The difference between “significant” and “not significant” is not itself statistically significant. Am. Stat. 60, 328–331 (2006).

    Article  Google Scholar 

  2. Rosnow, R.J. & Rosenthal, R. Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44, 1276–1284 (1989).

    Article  Google Scholar 

  3. Loftus, G.R. & Masson, M.E.J. Using confidence intervals in within-subject designs. Psychon. Bull. Rev. 1, 476–490 (1994).

    Article  CAS  Google Scholar 

  4. Cumming, G., Fidler, F. & Vaux, D.L. Error bars in experimental biology. J. Cell Biol. 177, 7–11 (2007).

    Article  CAS  Google Scholar 

  5. Henson, R. What can functional neuroimaging tell the experimental psychologist? Q. J. Exp. Psychol. A 58, 193–233 (2005).

    Article  Google Scholar 

  6. Poldrack, R.A. et al. Guidelines for reporting an fMRI study. Neuroimage 40, 409–414 (2008).

    Article  Google Scholar 

  7. Rosenthal, R. & Gaito, J. The interpretation of levels of significance by psychological researchers. J. Psychol. 55, 33–38 (1963).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Leiden University, Leiden, The Netherlands

    Sander Nieuwenhuis

  2. Leiden Institute for Brain and Cognition, Leiden, The Netherlands

    Sander Nieuwenhuis

  3. Cognitive Science Center Amsterdam, University of Amsterdam, Amsterdam, The Netherlands

    Birte U Forstmann & Eric-Jan Wagenmakers

Authors
  1. Sander Nieuwenhuis
    View author publications

    Search author on:PubMed Google Scholar

  2. Birte U Forstmann
    View author publications

    Search author on:PubMed Google Scholar

  3. Eric-Jan Wagenmakers
    View author publications

    Search author on:PubMed Google Scholar

Contributions

S.N. conceived the project and made the figure. S.N., B.U.F. and E.-J.W. conducted the literature analyses and wrote the paper.

Corresponding author

Correspondence to Sander Nieuwenhuis.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nieuwenhuis, S., Forstmann, B. & Wagenmakers, EJ. Erroneous analyses of interactions in neuroscience: a problem of significance. Nat Neurosci 14, 1105–1107 (2011). https://doi.org/10.1038/nn.2886

Download citation

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/nn.2886

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics