The hostility in the debate has obscured an interesting empirical question. This post aims to\u00a0answer that interesting empirical question. [1<\/a>]\n Ceiling effect The sole issue of contention regarding the data (.xlsx<\/a>), is that nearly twice as many\u00a0respondents gave the highest possible answer in the replication as in the original study (about 41% vs about 23%). \u00a0In a forthcoming commentary (.html<\/a>), the original author proposes a \"ceiling effect\" <\/em>explanation: it is hard to increase something that is already very high.<\/p>\n I re-analyzed the original and replication data to assess this sensible\u00a0concern. The experiments The new analysis It consists of going beyond comparing means, examining instead all observations.The stylized figures below give the intuition. They plot the cumulative percentage of observations for each value of the dependent variable.<\/p>\n The first shows an effect across the board: there is a gap between the curves throughout. Let's now look at real data. Study 1 first: [3<\/a>]\n Study 2 is more compelling, In the Original the effect is largest\u00a0in the 4-6 range.\u00a0In the Replication about 60% of the data is in that range, far from the ceiling of 7. But still there is no gap between the lines.<\/p>\n Ceiling analysis by original author Dividing something by increasingly larger numbers leads to increasingly smaller ratios, with or without a ceiling. Imagine\u00a0the effect were\u00a0constant, completely unaffected by ceiling effects. Say a 1 point increase in the morality scale in every<\/em> scenario. This constant effect would be a smaller % in scenarios with a larger baseline; going from 2 to 3 is a 50% increase, whereas going from 9 to 10 only 11%. [4<\/a>]\n If a store-owner gives you $5 off any item, buying\u00a0a $25 calculator gets you a 20% discount, buying\u00a0a $100 jacket gets you only a 5% discount. But there is no ceiling, you are getting $5 in both cases.<\/p>\n To eliminate the arithmetic confound, I redid this\u00a0analysis with effect size defined as the difference of means, rather than %, and there was no association between effect size and share of answers at boundary across scenarios\u00a0(see calculations,\u00a0.xlsx<\/a>).<\/p>\n Ceiling analysis by replicators Dropping observations at\u00a0the boundary lowers power whether there is a ceiling effect or not, by a lot<\/em>. \u00a0In simulations, I saw drops of 30% and more, say from\u00a050% to 20% power (R Code<\/a>). So not getting an effect this way does not support the absence of a ceiling effect problem.<\/span><\/p>\n Tobit Authors' response The replication authors, Brent Donnellan<\/a>, Felix Cheung<\/a> and David Johnson suggested minor modifications to analyses and writing. They are reflected in the version you just read.<\/span><\/p>\n
\n<\/strong>The replication (.html<\/a>) was pre-registered; it was evaluated and approved by peers, including the original authors, before being run. The predicted effect was not obtained, in two separate replication studies.<\/p>\n
\nMy read is that the evidence is greatly inconsistent with the<\/span> ceiling effect<\/em>\u00a0explanation.<\/span><\/p>\n
\n<\/strong>In the original paper (.html<\/a>), participants rated six \"dilemmas\" involving moral judgments (e.g., How wrong is it to keep money found in a lost wallet?<\/em>). These judgments were predicted to become less harsh for\u00a0people primed with cleanliness (Study 1) or who just washed their hands (Study 2).<\/p>\n
\n<\/strong>In a paper with\u00a0Joe and Leif (SSRN<\/a>),\u00a0we showed\u00a0that a prominent failure to replicate in economics was invalidated by a ceiling effect. I use the same key analysis here. [2<\/a>]\n
\nThe third shows the absence of an effect: the curves perfectly overlap.<\/p>\n![\"Example](\"https:\/\/datacolada.org\/wp-content\/uploads\/2014\/06\/Example-Figure.png\")
<\/a>The middle figure captures what a ceiling effect looks like. All\u00a0values above 2 were brought down to 2 so the lines overlap there, but below the ceiling the gap is still easy to notice.<\/p>\n![\"Ori1\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/2014\/06\/Ori1-300x218.png\")
<\/a>\u00a0 ![\"Rep1\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/2014\/06\/Rep1-300x218.png\")
<\/a>
\nIt is easy to spot the effect in the original data.
\nIt is just as easy\u00a0to spot the absence of an effect in the replication.<\/p>\n
\n![\"Ori2\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/2014\/05\/Ori22.png\")
<\/a>\u00a0![\"Rep2\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/2014\/05\/Rep22.png\")
<\/a><\/p>\n
\n<\/strong>In her forthcoming commentary (.pdf<\/a>), effect size is computed as a percentage <\/em>and shown\u00a0to be smaller in scenarios\u00a0with higher baseline levels\u00a0(see her Figure 1<\/a>). This is interpreted as evidence of a ceiling effect.
\nI don't think that's right.<\/span><\/p>\n
\n<\/strong>In their rejoinder (.pdf<\/a>), the replicators counter by dropping all observations at the ceiling and showing the results are still not significant.
\nI don't think that's right either.<\/span><\/p>\n
\n<\/strong>To formally take ceiling effects into account one can use the Tobit model (common in economics for censored data, see Wikipedia<\/a>). A feature of this approach is that it allows analyzing the data at the scenario level, where the ceiling effect would actually be happening. I run Tobits on all datasets. The replications still had tiny effect sizes (<1\/20th size of original), with p-values>.8 (STATA code<\/a>). [5<\/a>]\n![\"Wide](\"https:\/\/datacolada.org\/wp-content\/uploads\/2014\/02\/Wide-logo-300x145.jpg\"){kind=logo}
<\/p>\n
\n<\/strong>Our policy at DataColada is to give drafts of our post to authors whose work we cover before posting, asking for feedback and providing an opportunity to comment. This causes delays (see footnote 1) but avoids misunderstandings.<\/p>\n