{"id":868,"date":"2015-04-09T07:00:48","date_gmt":"2015-04-09T11:00:48","guid":{"rendered":"http:\/\/datacolada.org\/?p=868"},"modified":"2020-11-18T23:17:33","modified_gmt":"2020-11-19T04:17:33","slug":"35-the-default-bayesian-test-is-prejudiced-against-small-effects","status":"publish","type":"post","link":"https:\/\/datacolada.org\/35","title":{"rendered":"[35] The Default Bayesian Test is Prejudiced Against Small Effects"},"content":{"rendered":"

When considering any statistical tool I think it is useful to answer the following two practical questions:<\/p>\n

1. \u201cDoes it give reasonable answers in realistic circumstances?\u201d
\n2. \u201cDoes it answer a question I am interested in?\u201d<\/p>\n

In this post I explain why, for me<\/em>, when it comes to the default Bayesian test that's starting to pop up in some psychology publications, the answer to both questions is \u201cno<\/strong>.\u201d<\/span><\/p>\n

The Bayesian test
\n<\/strong>The Bayesian approach to testing hypotheses is neat and compelling. In principle. [1<\/a>]\n

The p<\/em>-value assesses only how incompatible the data are with the null hypothesis.\u00a0The Bayesian approach, in contrast, assesses the relative<\/em> compatibility of the data with a null vs an alternative hypothesis.<\/p>\n

The devil is in choosing that alternative. \u00a0If the effect is not zero, what is it?<\/p>\n

Bayesian advocates in psychology have proposed using a \u201cdefault\u201d alternative (Rouder et al 1999, .pdf<\/a>). This default is used in the online (.html<\/a>) and R based (.html<\/a>)\u00a0Bayes factor calculators.\u00a0The original papers do warn attentive readers that the default can be replaced with alternatives informed by expertise or beliefs (see especially Dienes 2011 .pdf<\/a>), but most researchers leave the default unchanged. [2<\/a>]\n

This post is written with that majority of default following researchers in mind.\u00a0I explain why, for me, when running the default Bayesian test, the answer to Questions 1 & 2 is \"no\" .<\/span><\/span><\/p>\n

Question 1. \u201cDoes it give reasonable answers in realistic circumstances?\u201d
\n<\/strong>No. It is prejudiced against small effects<\/strong><\/p>\n

The null hypothesis is that the effect size (henceforth d<\/em>) is zero.\u00a0Ho<\/span>: d<\/em> = 0.\u00a0What's the alternative hypothesis? It can be whatever we want it to be, say, Ha<\/span>: d\u00a0<\/em>= .5. We would then ask: are the data more compatible with d\u00a0<\/em>= 0 or are they more compatible with d\u00a0<\/em>= .5?<\/p>\n

The default alternative hypothesis used in the Bayesian test is a bit more complicated. It is a distribution, so more like\u00a0Ha<\/span>:\u00a0d~N(0,1). So we ask if the data are more compatible with zero or with d~N(0,1). [3<\/a>]\n

That the alternative is a distribution makes it difficult to think about the test intuitively. \u00a0Let's not worry about that. The key thing for us is that that\u00a0default is prejudiced against small effects.<\/p>\n

Intuitively (but not literally), that default means the\u00a0Bayesian test ends up asking: \u201cis the effect zero, or is it biggish<\/em>?\u201d When the effect is neither, when it\u2019s small, the Bayesian test ends up concluding (erroneously) it's zero. [4<\/a>]\n

Demo 1. Power at 50%<\/span><\/em><\/p>\n

Let's see how the test behaves as the effect size get smaller (R Code<\/a>)\"Fig1\"<\/a>The Bayesian test erroneously supports the null about 5% of the time when the effect is biggish<\/em>, d=.64, but it does so five times more frequently when it is smallish<\/em>, d=.28. \u00a0The smaller the effect (for studies with\u00a0a given level of power), the more likely we are to dismiss its existence. \u00a0We are prejudiced against small effects. [5<\/a>]
\n<\/span><\/em><\/p>\n

Note how as sample gets larger the test becomes more confident (smaller white area) and more wrong (larger red area).<\/span><\/p>\n

Demo 2. Facebook
\n<\/span><\/em>For a more tangible example consider the Facebook experiment (.html<\/a>) that found that seeing images of friends who voted (see panel a<\/strong> below) increased voting by\u00a00.39% (panel b<\/strong>).\"Facebook3\"<\/a>While the null of a zero effect is rejected (p<\/em>=.02) and hence the entire\u00a0confidence interval for the effect is above zero, [6<\/a>] the Bayesian test concludes VERY strongly in favor of the null, 35:1. (R Code<\/a>)<\/p>\n

Prejudiced against (in this case very) small effects.<\/p>\n

Question 2. \u201cDoes it answer a\u00a0question I am interested in?\u201d
\n<\/strong>No. I am not interested in how well data support one elegant\u00a0distribution.<\/strong><\/span><\/p>\n

\u00a0When people run a Bayesian test they like writing things like
\n\"The data support the null<\/em>.\"<\/span><\/p>\n

But that\u2019s not quite right. What they actually ought to write is
\n\"The data support the null more than they support one mathematically elegant\u00a0alternative hypothesis I compared it to\"<\/span><\/em><\/p>\n

Saying a Bayesian test \u201csupports the null\u201d in absolute terms seems as fallacious to me as interpreting the p-value\u00a0as the probability that the null is false.<\/p>\n

We are constantly reminded that:
\nP(D|H0<\/sub>)\u2260P(H0<\/sub>)
\nThe probability of the data given the null is not the probability of the null<\/p>\n

But let\u2019s not forget that:
\nP(H0<\/sub>|D) \/ P(H1<\/sub>|D)\u00a0 \u2260 P(H0<\/sub>)
\nThe relative probability of the null over one mathematically elegant alternative is not the probability of the null either.<\/p>\n

Because I am not interested in the distribution designated as the alternative hypothesis, I am not interested in how well the data support it. The default Bayesian test does not answer a question I would ask.<\/p>\n

\"Wide<\/p>\n

 <\/p>\n

\n

Feedback\u00a0from Bayesian advocates:<\/strong><\/span>
\n I shared an early draft of this post with three Bayesian advocates. I asked for feedback and invited them to comment.<\/span><\/p>\n

1.\u00a0Andrew Gelman\u00a0<\/a>\u00a0<\/strong>Expressed \"100% agreement<\/em>\" with my argument but\u00a0thought I should make it clearer this is not the only Bayesian approach, e.g., he writes \"You can spend your entire life doing Bayesian inference without ever computing these Bayesian Factors.<\/em>\" I made several edits in response to his suggestions, including changing\u00a0the\u00a0title.<\/span><\/p>\n

2.\u00a0<\/strong>Jeff Rouder<\/strong>\u00a0<\/a>\u00a0Provided additional feedback and also wrote a formal reply (.html<\/a>).\u00a0He begins highlighting the importance of comparing p<\/em>-values and Bayesian Factors when -as is the case in reality- we don't know if the effect does or does not exist, and the paramount importance for science of subjecting specific predictions to data analysis\u00a0(again, full reply: .html<\/a>)<\/span><\/p>\n

3.\u00a0EJ<\/a> Wagenmakers<\/a>\u00a0<\/span><\/strong>Provided feedback on terminology, the poetic response that follows, and a more in-depth critique of confidence intervals (.pdf<\/a>)<\/span><\/p>\n

In a desert of incoherent frequentist testing there blooms a Bayesian flower. You may not think it is a perfect flower. Its color may not appeal to you, and it may even have a thorn. But it is a flower, in the middle of a desert. Instead of critiquing the color of the flower, or the prickliness of its thorn, you might consider planting your own flower — with a different color, and perhaps without the thorn. Then everybody can benefit.\u201d<\/span><\/p>\n

\"Sunbaked<\/a><\/p><\/blockquote>\n

\n

Subscribe to Blog via Email<\/h2>\n\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t\t

Enter your email address to subscribe to this blog and receive notifications of new posts by email.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t