Data have noise, so we can never say \u2018the effect is exactly <\/em>zero.\u2019 \u00a0We can only say \u2018the effect is basically<\/em> zero.\u2019 What we do is\u00a0draw a line close to zero and if we are confident the effect is below the line, we accept the null. In this post I describe 4 ways to draw the line, and then pit the top-2 against each other.<\/p>\n Way 1. Absolutely small \u201cThe impact of financial literacy on the average remittance frequency has a 95 percent confidence interval [\u22124.3%, +2.5%] \u2026. We consider this a relatively precise zero<\/u><\/strong> effect, ruling out large positive or negative effects of training\u201d (emphasis added) In much of behavioral science effects of any<\/em> size can be of theoretical interest, and sample sizes are too small to obtain tight confidence intervals, making this approach unviable in principle and in practice [1<\/a>].<\/p>\n Way 2. Undetectably Small We don\u2019t draw the line where we stop caring about effects. Say an original study with n=50 finds people can feel the future.\u00a0A replication with n=125 \u2018fails,\u2019 getting and effect estimate of d=0.01, p<\/em>=.94.\u00a0Data are noisy, so the confidence interval goes all the way up to d<\/em>=.2.\u00a0That\u2019s a respectably big feeling-the-future effect we are not ruling out. So we cannot say the effect is absolutely small. Way 3. Smaller than expected in general Hypothesis 1 (null): \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0The effect is exactly zero. When data clearly favor 1 more than 2, we accept the null.\u00a0The bigger the effects Hypothesis 2 includes, the further from zero we draw the line, the more likely we accept the null [3<\/a>].<\/p>\n The default Bayesian test, commonly used by Bayesian advocates in psychology, draws the line too far from zero (for my taste). Reasonably powered studies of moderately big effects wrongly accept the null of zero effect too often (see Colada[35<\/a>]) [4<\/a>].<\/p>\n Way 4. Smaller than expected this time Similar to Way 3 the bigger the effect seen in the original is, the bigger the effect we expect in the replication, and hence the further from zero we draw the line. Importantly, here the line moves based on what we observed in the original, not (only) on what we arbitrarily choose to consider reasonable to expect. The approach is the handsome cousin of testing if effect size differs between original and replication.<\/p>\n Small Telescope vs Expected This Time<\/em> (Way 2 vs Way 4) In Study 7 in the Science<\/em> paper, The Small Telescopes approach is irked by the small sample of the replication. Its wide\u00a0confidence interval includes effects as big as d<\/em> =1.14, giving the original >99% power. We cannot rule out detectable effects, the replication is inconclusive<\/span>.<\/p>\n The Bayesian observer, in contrast, draw a line quite far from zero after seeing the massive Original effect size. The line, indeed is at a remarkable\u00a0d<\/em>=.8. Replications with smaller effect size estimates, anything smaller than large, 'supports the null.' Because the replication is d<\/em>=.26, it strongly\u00a0supports the null<\/span>.<\/p>\n A hypothetical scenario where they disagree in the opposite direction (R Code<\/a>), The Small Telescopes approach asks if the replication rejects an effect big enough to be detectable by the original. Yes. d<\/em>=.1 cannot be studied with N=40. Null Accepted\u00a0<\/span>[7<\/a>].<\/p>\n Interestingly, that small N=40 pushes the Bayesian in the opposite direction. An original with N=40 changes very little her beliefs about the effect, so d=.1 in the replication is not that surprising \u00a0vs. the Original, but it is incompatible with d=0 given the large sample size, null rejected.<\/span><\/p>\n I find myself agreeing with the\u00a0Small Telescopes' line more than any other. But that's a matter of taste, not fact.<\/span><\/p>\n
\n![\"Drawing](\"https:\/\/datacolada.org\/wp-content\/uploads\/2015\/10\/Pic-accept-null-whiteboard-3.jpg\")
<\/a>We can draw the line via Bayes or via p<\/em>-values, it does not matter very much.\u00a0The line is what really matters. How far from zero is it? What moves it up and down?<\/p>\n
\n<\/strong>The oldest approach draws the line based on absolute size. Say, diets leading to losing less than 2 pounds have an effect of basically zero.\u00a0Economists do this often. For instance, a recent World Bank paper (.html<\/a>) reads<\/p>\n
\n<\/em>(Dictionary note. Remittance: immigrants sending money home).<\/em><\/p><\/blockquote>\n
\n<\/strong>In our first p<\/em>-curve paper with Joe and Leif (SSRN<\/a>), and in my \u201cSmall Telescopes\u201d paper on evaluating replications (.pdf<\/a>), we draw the line based on detectability.<\/p>\n
\nWe draw the line where we stop being able to detect them.<\/p>\n
\n![\"example\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/2015\/10\/example-1024x656.png\")
\nThe original study, with just n=50, however, is unable to detect that small an effect (it would have <18% power). So we accept the null, the null that the effect is either zero, or undetectably small by existing studies.<\/p>\n
\n<\/em><\/strong>Bayesian hypothesis testing runs a horse race between two hypotheses:<\/p>\n
\nHypothesis 2 (alternative): The effect is one of those moderately sized ones [2<\/a>].<\/p>\n
\n<\/em><\/strong>A new Bayesian approach to evaluate replications, by Verhagen and Wagenmakers (2014 .pdf<\/a>), pits a different Hypothesis 2 against the null.\u00a0Its Hypothesis 2 is what a Bayesian observer would predict for the replication after seeing the Original (with some\u00a0assumed prior).<\/p>\n
\n<\/strong>I compared the conclusions both approaches arrive at when applied to the 100 replications from that Science<\/em> paper.\u00a0The results are similar but far from equal, r<\/em> = .9 across all replications, and r<\/em> = .72 among n.s. <\/em>ones (R Code<\/a>). Focusing on situations where the two lead to opposite conclusions is useful to understand each better [5<\/a>],<\/span>[6<\/a>].<\/p>\n
\nThe Original estimated a monstrous d<\/em>=2.14 with N=99 participants total.
\nThe Replication estimated a small\u00a0\u00a0\u00a0 d<\/em>=0.26, with a miniscule N=14.<\/p>\n
\nOriginal.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 N=40, \u00a0 \u00a0 \u00a0 d=.7
\nReplication.\u00a0 N=5000, d=.1<\/p>\n![\"Wide](\"https:\/\/datacolada.org\/wp-content\/uploads\/2014\/02\/Wide-logo-300x145.jpg\"){kind=logo}
<\/p>\n
\n