{"id":1112,"date":"2016-01-14T08:50:49","date_gmt":"2016-01-14T13:50:49","guid":{"rendered":"http:\/\/datacolada.org\/?p=1112"},"modified":"2020-02-11T23:06:02","modified_gmt":"2020-02-12T04:06:02","slug":"45-ambitious","status":"publish","type":"post","link":"https:\/\/datacolada.org\/45","title":{"rendered":"[45] Ambitious P-Hacking and P-Curve 4.0"},"content":{"rendered":"

In this post, we first consider how plausible it is for\u00a0researchers to engage in more ambitious p<\/em>-hacking (i.e., past the nominal significance level of p<.05). Then, we describe how we have modified p<\/em>-curve (see app 4.0<\/a>) to deal with this possibility.<\/p>\n

Ambitious p<\/em>-hacking is hard.<\/strong>
\nIn \u201cFalse-Positive Psychology\u201d (SSRN<\/a>), we simulated the consequences of four (at the time acceptable) forms of p-hacking. We found that the probability of finding a statistically significant result (p<.05) skyrocketed from the nominal 5% to 61%.\"f1\"<\/a><\/p>\n

For a recently published paper, \"Better P<\/em>-Curves\" (.pdf<\/a>), we modified those simulations to see how hard it would be for p<\/em>-hackers to keep going past .05. We found that p<\/em>-hacking needs to increase exponentially to get smaller and smaller p<\/em>-values. For instance, once a nonexistent effect has been p<\/em>-hacked to p<\/em><.05, a researcher would need to attempt nine times as many analyses to achieve p<\/em><.01.<\/p>\n

\"F2\"<\/a><\/p>\n

Moreover, as Panel B shows, because there is a limited number of alternative analyses one can do (96 in our simulations), ambitious p<\/em>-hacking often fails.[1<\/a>]\n

P<\/em>-Curve and Ambitious\u00a0p<\/em>-hacking<\/strong>
\nP-<\/em>curve is a tool that allows you to diagnose the evidential value of a set of statistically significant findings. It is simple: you plot the significant p<\/em>-values of the statistical tests of interest to the original researchers, and you look at its shape. If your p<\/em>-curve is significantly right-skewed, then the literature you are examining has evidential value. If it\u2019s significantly flat or left-skewed, then it does not.<\/p>\n

In the absence of p<\/em>-hacking, there is, by definition, a 5% chance of mistakenly observing a significantly right-skewed p<\/em>-curve if one is in fact examining a literature full of nonexistent effects. Thus, p<\/em>-curve\u2019s false-positive rate is 5%.<\/p>\n

However, when researchers p<\/em>-hack trying to get p<\/em><.05, that probability drops <\/em>quite a bit, because p<\/em>-hacking causes p<\/em>-curve to be left<\/em>-skewed in expectation, making it harder to (mistakenly) observe a right-skew. Thus, literatures studying nonexistent effects through p<\/em>-hacking have less than a 5% chance of obtaining a right-skewed p<\/em>-curve.<\/p>\n

But if researchers get ambitious and keep p<\/em>-hacking past .05, the barely significant results start disappearing and so p<\/em>-curve starts having a spurious right-skew. Intuitively, the ambitious p<\/em>-hacker will eliminate the .04s and push past to get more .03s or .02s. The resulting\u00a0p<\/em>-curve starts to look artificially good.<\/p>\n

Updated p<\/em>-curve app, 4.0 (htm<\/a>), is robust to ambitious p-hacking
\n<\/strong>In \u201cBetter P<\/em>-Curves\u201d (.pdf<\/a>) we introduced a new test for evidential value that is much more robust to ambitious p<\/em>-hacking. The new app incorporates it (it also computes confidence intervals for power estimates, among many other improvements, see summary (.htm<\/a>)).<\/p>\n

The new test focuses on the \u201chalf p<\/em>-curve,\u201d the distribution of p<\/em>-values that are p<\/em><.025. On the one hand, because half p<\/em>-curve does not include barely significant results, it has a lower probability of mistaking ambitious p<\/em>-hacking for evidential value. On the other hand, dropping observations makes the half p<\/em>-curve less powerful, so it has a higher chance of failing to recognize actual evidential value.<\/p>\n

Fortunately, by combining<\/em> the full and half p<\/em>-curves into a single analysis, we obtain inferences that are robust to ambitious p<\/em>-hacking with minimal loss of power.<\/p>\n

The new test of evidential value:
\nA set of studies is said to contain evidential value if either<\/u> the half p-curve has a p<.05 right-skew test, or<\/u> both the full and half p-curves have p<.1 right-skew tests.\u00a0<\/span><\/em>[2<\/a>]<\/span><\/span><\/p>\n

In the figure below we compare the performance of this new combination test with that of the full p<\/em>-curve alone (the \u201cold\u201d test). The top three panels show that both tests are similarly powered to detect true effects. Only when original research is underpowered at 33% is the difference noticeable, and even then it seems acceptable. With just 5 p<\/em>-values the new test still has more power than the underlying studies do.<\/p>\n

\"f3\"<\/a><\/p>\n

The bottom panels show that moderately ambitious p<\/em>-hacking fully invalidates the \u201cold\u201d test, but the new test is unaffected by it.[3<\/a>]\n

We believe that these revisions to p-curve, incorporated in the updated app (.html<\/a>), make it much harder to falsely conclude that a set of ambitiously p<\/em>-hacked results contains evidential value. As a consequence, the incentives to ambitiously p<\/em>-hack are even lower than they were before.<\/p>\n

\"Wide<\/p>\n

\n

Subscribe to Blog via Email<\/h2>\n\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t\t

Enter your email address to subscribe to this blog and receive notifications of new posts by email.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t