{"id":2860,"date":"2018-12-05T07:00:17","date_gmt":"2018-12-05T12:00:17","guid":{"rendered":"http:\/\/datacolada.org\/?p=2860"},"modified":"2020-11-18T23:15:28","modified_gmt":"2020-11-19T04:15:28","slug":"74-in-press-at-psychological-science-a-new-nudge-supported-by-implausible-data","status":"publish","type":"post","link":"https:\/\/datacolada.org\/74","title":{"rendered":"[74] In Press at Psychological Science: A New 'Nudge' Supported by Implausible Data"},"content":{"rendered":"

Today Psychological Science<\/em> issued a Corrigendum (.htm<\/a>) and an expression of concern (htm<\/a>) for a paper originally posted online in May 2018 (.htm<\/a>). This post will spell out the data irregularities we uncovered that eventually led to the two postings from the journal today. We are not convinced that those postings are sufficient.<\/p>\n

It is important to say at the outset we have not identified who is responsible for the problems. In the correction, for example, the authors themselves make clear that they \"do not have an explanation\" for some peculiarities, in part because many other people handled the data between collection and reporting. This post is therefore not about who caused the problems [1<\/a>].<\/p>\n

R Code<\/a> to reproduce all calculations and figures.<\/p>\n

Background<\/strong>
\nThe history of the correction starts back in May, in a Shanghai journal club discussion Leif participated in while on sabbatical in China. Puzzled by a few oddities, four members of the group \u2013 Frank Yu (.htm<\/a>), Leif, and two other anonymous researchers \u2013 went on to consider the original data posted by the authors (.htm<\/a>) and identified several patterns that were objectively, instead of merely intuitively, problematic.<\/p>\n

Most notably, the posted data has two classic markers of data implausibility:<\/p>\n

(i) anomalous distribution of last digits, and<\/p>\n

(ii) means are excessively similar.<\/p>\n

Leif and his team first went to Uri for his independent assessment of the data, who concurred that the problems looked significant and added new analyses. Then, back in June, they contacted Steve Lindsay (.html<\/a>), the editor of Psychological Science.<\/em>\u00a0 In consultation with the editor, the authors then wrote a correction. We deemed this correction to be insufficient and we drafted a blog post. We shared it with the authors and the editor. They asked us to wait while they considered our arguments further. We promised we would, and we did.\u00a0Eventually they wrote an expression of concern, to be published alongside the Corrigendum, and they shared it with us. Today, six months after we first contacted the editor, we publish this post, in part because these responses (1) seem insufficient given the gravity of the irregularities, and (2) do not convey the irregularities clearly enough for readers to understand their gravity.<\/p>\n

The basic design in the original paper.<\/strong>
\nLi (htm<\/a>), Sun (htm<\/a>), & Chen, report three field experiments showing that the Decoy Effect \u2013 a classic finding from decision research [2<\/a>] \u2013 can be used as a nudge to increase the use of hand\u2011sanitizer by food factory workers.<\/p>\n

In the experiments, the authors manipulate the set of sanitizer dispensers available, and measure the amount of sanitizer used, by weighing the dispensers at the end of each day. There is one observation per worker-day.<\/p>\n

For example, in Experiment 1, some workers only had a spray dispenser, while others had two dispensers, both the spray dispenser and a squeeze-bottle:<\/p>\n

\"\"<\/a>
\nThe authors postulated that the squeeze-bottle sanitizer was objectively inferior to the spray, and that it would serve as a decoy. Thus, the authors predicted that workers would be more likely to use the spray dispenser when it was next to the squeeze bottle than when it was the only dispenser available [3<\/a>].<\/p>\n

Original results: Huge effects.
\n<\/strong>Across three studies, the presence of a decoy dispenser increased the use of the spray dispenser by more than 1\u00a0standard deviation on average (d<\/em> = 1.06).\u00a0That's a large effect. Notably, in Study 2, only one<\/span> participant in the control condition increased sanitizer use more than the participant who increased the least<\/span> in the treatment. Almost non-overlapping distributions.<\/p>\n

Problem 1. Inconsistency in scale precision.<\/strong>
\nThe original article indicated that the experimenters used \"an electronic scale accurate to 5 grams\"<\/em> (p.4). Such a scale could measure 15 grams, or 20 grams, but not 17 grams. Contradicting this description, the posted data has many observations (8.4% of them) that were not multiples of 5.<\/p>\n

The correction states that scales accurate to 1, 2 and 3 grams may have been used sometimes, instead of scales precise to 5 grams (We do not believe scales precise to 3 grams exist). \u00a0[4<\/a>].<\/p>\n

Problem 2. Last digit in Experiment 1
\n<\/strong>But there is another odd thing about the data purportedly obtained with the more precise scales. The problem involves the frequency of the last digit in the number of grams (by last digit we mean, for example, the 8, in 2018<\/u>).<\/p>\n

In particular, the problem with those observations draws on the generalization of something called \u201cBenford\u2019s Law\u201d, \u00a0which tells us the last digit should be distributed (nearly) uniformly: there should be just about as many workers using sanitizer amounts that end in 3 grams (e.g, 23 or 43 grams), as in 4 (e.g., 24 or 44 grams), etc. But as we see below, the data looks nothing like the uniform distribution. (If you are not familiar with Benford's law, read this footnote: [5<\/a>]).<\/p>\n

\"\"<\/a>
\n<\/strong>Fig 1<\/strong>. Histogram for last digit in Study 1 <\/em>[6<\/a>].<\/p>\n

About this problem, the expression of concern reads:<\/p>\n

\"\"<\/p>\n

This speculated behavior, one scale precise to 5 grams used in the morning, another precise to 1 or 2 grams in the afternoon, or vice versa, cannot explain the posted data. A uniform distribution of last digits is anyway expected, not the bizarre prevalence of 3s and 7s that we see (R Code<\/a>).<\/p>\n

Problem 3. Last digit in Experiment 3.
\n<\/strong>Let's look at the last digit again. In this study sanitizer use was measured for 80 participants over 40 days, and with a scale sensitive to the 100th<\/sup> of a gram. The expectation of the last digit being uniformly distributed here is more obvious.<\/p>\n

\"\"<\/a>
\nFig 2.<\/strong> Last digit for Study 3<\/em><\/p>\n

To appreciate how implausible Fig 2<\/strong> is, consider that it implies, for example, that workers would be 3 times as likely to use 45.56<\/u> grams of sanitizer, as they would be to use 45.53<\/u> grams\u00a0[7<\/a>].<\/p>\n

About this problem, the expression of concern reads:
\n\"\"
\nProblem 4. Implausibly similar means in Experiment 2<\/strong>
\nIn Experiment 2, sanitizer use was measured daily for 40 participants for 40 days (20 days of baseline, 20 of treatment), all with a scale sensitive to 100th<\/sup> of a gram.<\/p>\n

Recall that the manipulation was done at the room level. This figure, which was in the original article, shows the daily average use of sanitizer across the two rooms.<\/p>\n

\"\"<\/a>
\nTreatment started on day 21. In days 1-20 the two rooms had extraordinarily similar means. Average sanitizer usage differed, on average, by just .19 grams across rooms. Moreover, across days, average sanitizer use was correlated at r<\/em> = .94 across rooms.<\/p>\n

To quantify how surprisingly similar the conditions were in the \"before treatment\" period, we conducted the following resampling test: we shuffle all 40 participants into two new groups of 20 (keeping all observations per worker fixed). We then compute daily means for each of the two groups ('rooms'). We did this one million times and asked \"How often do simulation results look as extreme as the paper's?\" The answer is \"almost never\":<\/p>\n

\"\"<\/a>
\nSo, for example, the figure on the right shows that the correlation between means is on average about r=.7 rather than the r=.94 that's reported in the paper. Only 96 times in a million, would we expect it to be .94 or higher.<\/p>\n

We don't think readers of the expression of concern would come away with sufficient information to appreciate the impossibility we shared with the authors and editor; all it says about it is:<\/p>\n

\"\"<\/p>\n

Problem 5. Last digit\u2026 in Experiment 2<\/strong><\/p>\n

While less visually striking than for Experiments 1 and 3, the last digit is not distributed uniform in this experiment either, with N=1600, a scale precise to 1\/100th<\/sup> of gram, rejects the uniform null with: \u03c72<\/sup>(9)\u00a0=\u00a043.45, p<\/em><.0001; see histogram .png<\/a> [8<\/a>].<\/p>\n

Summary.
\n<\/strong>We appreciate that the authors acknowledge some of the problems we brought to their attention and further that they cannot assuage concerns because the data collection and management necessarily occurred at such a remove. On the other hand, as readers we are at a loss. Three experiments show unambiguous signs that there are problems with all of the reported data. How can we read the paper and interpret differences across conditions as meaningful while discounting the problems as otherwise meaningless? We think that it might be warranted to take the opposite view and see meaning in the long list of problems and therefore seeing the differences across conditions as meaningless.<\/p>\n

We should maintain a very high burden of proof to conclude that any individual tampered with data.<\/p>\n

But the burden of proof for dataset<\/em> concerns should be considerably lower. We do not need to know the source of contamination in order to lose trust in the data.<\/p>\n

Even after the correction, and the clarifications of the Expression of Concern, we still believe that these data do not deserve the trust of Psychological Science<\/em> readers.<\/p>\n

\"Wide<\/p>\n

\n

Author feedback<\/strong>
\nOur policy (.htm<\/a>) is to share drafts of blog posts that discuss someone else\u2019s work with them to solicit feedback. As mentioned above we contacted the authors and editor of Psych Science. They provided feedback on wording and asked that we wait while they revised the correction, which we did (for over 6 months).<\/span><\/p>\n

Just before posting they gave us another round of suggestions and then Meng Li\u00a0(htm<\/a>) wrote a separate piece (.htm<\/a>).<\/span><\/p>\n

When all is said and done, the original authors have not yet provided benign mechanisms that could have generated the data they reported (neither the last digit pattern, nor the excessive similarity of means).<\/span><\/p>\n

Subscribe to Blog via Email<\/h2>\n\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t\t

Enter your email address to subscribe to this blog and receive notifications of new posts by email.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t