{"id":1531,"date":"2017-04-12T07:00:15","date_gmt":"2017-04-12T11:00:15","guid":{"rendered":"http:\/\/datacolada.org\/?p=1531"},"modified":"2024-11-13T02:27:31","modified_gmt":"2024-11-13T07:27:31","slug":"59-pet-peese-is-not-like-homeopathy","status":"publish","type":"post","link":"https:\/\/datacolada.org\/59","title":{"rendered":"[59] PET-PEESE Is Not Like Homeopathy"},"content":{"rendered":"

PET-PEESE is a meta-analytical tool that seeks to correct for publication bias. In a footnote in my previous post (.htm<\/a>), I referred to is as the homeopathy of meta-analysis. That was unfair and inaccurate.<\/p>\n

Unfair <\/em>because, in the style of our President, I just called PET-PEESE a name instead of describing what I believed was wrong with it. I deviated from one of my rules for 'menschplaining' (.htm<\/a>): \"Don't label, describe.\"<\/p>\n

Inaccurate <\/em>because skeptics of homeopathy merely propose that it is ineffective, not harmful. But my argument is not that PET-PEESE is merely ineffective, I believe it is also harmful. It doesn't just fail to correct for publication bias, it adds<\/em> substantial bias where none exists.<\/p>\n

note: A few hours after this blog went live, James Pustejovsky (.htm<\/a><\/span>) identified a typo in the R Code which affects some results. I have already updated the code and figures below. (I archived the original post: .htm<\/a><\/span>).<\/span>
\n<\/span><\/p>\n

PET-PEESE in a NUT-SHELL
\n<\/strong>Tom Stanley (.htm<\/a>), later joined by Hristos Doucouliagos, developed PET-PEESE in various papers that have each accumulated 100-400 Google cites (.pdf<\/a> | .htm<\/a>). The procedure consists of running a meta-regression: a regression in which studies are the unit of analysis, with effect size as the dependent variable and its variance as the key predictor [1<\/a>]. The clever insight by Stanley & Doucouliagos is that the intercept of this regression is the effect we would expect in the absence of noise, thus, our estimate of the -publication bias corrected- true effect [2<\/a>].<\/p>\n

PET-PEESE in Psychology<\/strong>
\nPET-PEESE was developed with the meta-analysis of economics papers in mind (regressions with non-standardized effects). It is possible that some of the problems identified here, considering meta-analyses of standardized effect sizes, Cohen's d, do not extend to such settings [3<\/a>].<\/p>\n

Psychologists have started using PET-PEESE recently. For instance, in meta-analyses about religious primes (.pdf<\/a>), working memory training (.htm<\/a>), and personality of computer wizzes (.htm)<\/a>. Probably the most famous example is Carter et al.'s meta-analysis of ego depletion, published in JEP:G (.htm<\/a>).<\/p>\n

In this post I share simulation results that suggest we should not treat PET-PEESE estimates, at least of psychological research, very seriously. It arrives at wholly invalid estimates under too many plausible circumstances. Statistical tools need to be generally valid, or at least valid under predictable circumstances. PET-PEESE, to my understanding, is neither [4<\/a>].<\/p>\n

Results
\n<\/strong>Let's start with a baseline case for which PET-PEESE does OK: there is no publication bias, every study examines the exact same effect size, and sample sizes are distributed uniformly between n=12 and n=120 per cell. Below we see that when the true effect is d=0, PET-PEESE correctly estimates it as d\u0302=0, and as d gets larger, d\u0302 gets larger (R Code<\/a>).<\/p>\n

\"\"<\/a><\/p>\n

About 2 years ago, Will Gervais evaluated PET-PEESE in a thorough blog post (.htm<\/a>) (which I have cited in papers a few times). He found that in the presence of publication bias PET-PEESE did not perform well, but that in the absence of publication bias it at least did not make things worse. The simulations depicted above are not that different from his.<\/p>\n

Recently, however, and by happenstance, I realized that Gervais got lucky with the simulations (or I guess PET-PEESE got lucky) [5<\/a>]. If we deviate slightly from some\u00a0 of the specifics of the ideal scenario in any of several directions, PET-PEESE no longer performs well even in the absence of publication bias.<\/p>\n

For example, imagine that sample sizes don't go all the way to up n=120 per cell; instead, they go up to only n=50 per cell (as is commonly the case with lab studies) [6<\/a>]:<\/p>\n

\"\"<\/a><\/p>\n

A more surprisingly consequential assumption involves the symmetry of sample sizes across studies. Whether there are more small than large n studies, or vice versa, PET PEESE's performance suffers quite a bit. For example, if sample sizes look like this:<\/p>\n

\"\"<\/a><\/p>\n

then PET-PEESE looks like this:<\/p>\n

\"\"<\/a>
\nMicro-appendix<\/span><\/span><\/strong>
\n 1) It looks worse<\/span> <\/em>if there are more big n than small n studies (.png<\/a><\/span>). <\/span>
\n 2) Even if studies have n=50 to n=120, there is noticeable bias if n<\/em> is skewed across studies (.png<\/a>) <\/span>
\n<\/span><\/p>\n

It's likely, I believe, for real meta-analyses to have skewed n distributions. e.g., this is what it looked like in that ego depletion paper (note: it plots total N, not per-cell):<\/p>\n

\"\"<\/a><\/p>\n

So far we have assumed all studies have the exact same effect size, say all studies in the d=.4 bin are exactly d=.4. In real life different studies have different effects. For example, a meta-analysis of ego-depletion may include studies with stronger and weaker manipulations that lead to, say, d=.5 and d=.3 respectively. On average<\/em> the effect may be d=.4, but it moves around. Let's see what happens if across studies the effect size has a standard deviation of SD=.2.<\/p>\n

\"\"<\/a><\/p>\n

Micro-appendix<\/span><\/span><\/strong>
\n 3) If big n studies are more common than small ns: .<\/span>png
\n<\/a><\/span>4) If n=12 to n=120 instead of just n=50, .png<\/a><\/span><\/span><\/p>\n

Most troubling scenario
\n<\/strong>Finally, here is what happens when there is publication bias (only observe p<\/em><.05)<\/p>\n

\"\"<\/a>
\nMicro-appendix<\/span><\/span><\/strong>
\n With publication bias,
\n5) If n<\/em> goes up to n=120: .png<\/a>
\n6) If n is uniform n=12 to n=50 .png<\/a>
\n7) If d is homogeneous, sd(d)=0 .png<\/a> <\/span><\/span><\/p>\n

It does not seem prudent to rely on PET-PEESE, in any way, for analyzing psychological research. It's an invalid tool under too many scenarios.<\/p>\n

\"Wide<\/p>\n

\n

Author feedback.
\n<\/strong>Our policy is to share early drafts of our post with authors whose work we discuss. I shared this post with the creators of PET-PEESE, and also with others familiar with it: Will Gervais, Daniel Lakens, Joe Hilgard, Evan Carter, Mike McCullough and Bob Reed. Their feedback helped me identify an important error in my R Code, avoid some statements that seemed unfair, and become aware of the recent SPPS paper by Tom Stanley (see footnote 4). During this process I also learned, to my dismay, that people seem to believe -incorrectly-<\/strong> that p<\/em>-curve is invalidated under heterogeneity of effect size. A future post will discuss this issue, impatient readers can check out our p-curve papers, especially Figure 1 in our first paper (here<\/a><\/span>) and Figure S2 in our second (here<\/a><\/span>), which already address it; but evidently insufficiently compellingly.
\n<\/span><\/p>\n

Last but not least, everyone I contacted was offered an opportunity to reply within this post. Both Tom Stanley (.pdf<\/span><\/a>), and Joe Hilgard (.pdf<\/a><\/span>) did.
\n<\/span><\/p>\n

Subscribe to Blog via Email<\/h2>\n\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t\t

Enter your email address to subscribe to this blog and receive notifications of new posts by email.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t