{"id":994,"date":"2015-06-29T07:00:13","date_gmt":"2015-06-29T11:00:13","guid":{"rendered":"http:\/\/datacolada.org\/?p=994"},"modified":"2020-02-11T22:49:09","modified_gmt":"2020-02-12T03:49:09","slug":"40-reducing-fraud-in-science","status":"publish","type":"post","link":"https:\/\/datacolada.org\/40","title":{"rendered":"[40] Reducing Fraud in Science"},"content":{"rendered":"

Fraud in science is often attributed to incentives: we reward sexy-results\u2192fraud happens. The solution, the argument goes, is to reward other things. \u00a0In this post I counter-argue, proposing three alternative solutions.<\/span><\/p>\n

Problems with the Change the Incentives<\/em> solution.
\n<\/strong>First, even if rewarding sexy-results caused fraud, it does not follow we should stop rewarding sexy-results. We should pit costs vs benefits. Asking questions with the most upside is beneficial.<\/span><\/p>\n

Second, if we started rewarding unsexy stuff, a likely consequence is fabricateurs continuing to fake, now just unsexy stuff. \u00a0Fabricateurs want the lifestyle of successful scientists. [1<\/a>] Changing incentives involves making our lifestyle less appealing. (Finally, a benefit to committee meetings).\u00a0<\/span><\/p>\n

Third, the evidence for \u201cliking sexy\u2192fraud\u201d is just not there. Like real research, most fake research is not<\/em> sexy. Life-long fabricateur Diederik Stapel mostly published dry experiments with \u201cfindings\u201d in line with the rest of the literature. That we attend to and remember the sexy<\/em> fake studies is diagnostic of what we pay attention to, not what causes fraud. \u00a0<\/span><\/p>\n

The evidence that incentives causes fraud comes primarily from self-reports, with fabricateurs saying \"the incentives made me do it\" (see e.g., Tijdink et al\u00a0.pdf<\/a>; or Stapel interviews).\u00a0\u00a0To me, the guilty saying \u201cit\u2019s not\u00a0my<\/u><\/em>\u00a0fault\u201d seems like weak evidence. What else could they say?<\/span>
\n \u201cI realized I was\u00a0not cut-out for this; it was either faking some science or getting a job with less status\u201d<\/em><\/span>
\n \u201cI am kind of a psychopath, I had fun tricking everyone\u201d
\n\u201cA voice in my head told me to do it\u201d<\/em><\/span><\/p>\n

Similarly weak, to me, is the observation that fraud is more prevalent in top journals; we find fraud where we look for it. Fabricateurs faking articles that don\u2019t get read don\u2019t get caught\u2026.<\/span><\/p>\n

It's good for universities\u00a0to ignore quantity<\/em> of papers when hiring and promoting, good for journals to publish interesting questions with inconclusive answers. But that won't help with fraud.<\/span><\/p>\n

Solution 1. Retract without asking \u201care the data fake?\u201d
\n<\/strong>We have a high bar for retracting articles, and a higher bar for accusing people of fraud.\u00a0<\/span>The latter makes sense. The former does not.<\/span><\/p>\n

Retracting is not such a big deal, it just says \"we no longer have confidence in the evidence.\u201d\u00a0<\/span><\/p>\n

So many things can go wrong when collecting, analyzing and reporting data that this should be a relatively routine occurrence even in the absence of fraud. An accidental killing\u00a0may not land the killer in prison, but the victim goes 6 ft under regardless. I'd propose a \u00a0retraction doctrine like:<\/span><\/p>\n

If something is discovered that would lead reasonable experts to believe the results did not originate in a study performed as described in a published paper, or to conclude the study was conducted with excessive sloppiness, the journal should retract the paper. \u00a0\u00a0<\/span><\/p><\/blockquote>\n

Example 1.<\/u> Analyses indicate published results\u00a0are implausible for a study conducted as described (e.g., excessive linearity, implausibly similar means, or a covariate is impossibly imbalanced across conditions).\u00a0Retract.<\/span><\/em><\/p>\n

Example 2.<\/u> Authors of a paper published in a journal that requires data sharing upon request, when asked for it, indicate to have \u201clost the data\u201d. \u00a0Retract. [2<\/a>]<\/em><\/span><\/p>\n

Example 3.<\/u> Comparing original materials with posted data reveals important inconsistencies (e.g., scales ranges are 1-11 in the data but 1-7 in the original). Retract.<\/em><\/span><\/p>\n

When journals reject original submissions it is not their job to figure out why the authors run an uninteresting study or executed it poorly. They just reject it.<\/span><\/p>\n

When journals lose confidence in the data behind a published article it is not their job to figure out why the authors published data whose confidence was eventually lost. They should just retract it.<\/span><\/p>\n

Employers, funders, and co-authors can worry about why an author published untrustworthy data.\u00a0<\/span><\/p>\n

Solution 2. Show receipts
\n<\/strong>Penn, my employer, reimburses me for expenses incurred at conferences.\"receipt\"<\/a><\/span><\/span><\/p>\n

However, I don\u2019t get to just say \u201chey, I bought some tacos in that Kansas City conference, please deposit $6.16 onto my checking account.\u201d I need receipts.\u00a0 They trust me, but there is a paper trail in case of need.<\/span><\/p>\n

When I submit the work I presented in Kansas City to a journal, in contrast, I do just say \u201chey, I collected the data this or that way.\u201d No receipts.<\/span><\/p>\n

The recent Science<\/em> retraction, with\u00a0canvassers & gay marriage, is a great example for the value of receipts. The statistical evidence \u00a0suggested\u00a0something was off, but the receipts-like paper trail helped a lot\u00a0<\/span><\/p>\n

Author<\/em>: \u201cso and so run the survey with such and such company\u201d
\n<\/span>Sleuths<\/em>: \"hello such and such company, can we talk with so and so about this survey you guys run?\"
\n<\/span>Such and such company: <\/em>\u201cwe don\u2019t know any so and so, and we don\u2019t have the capability to run the survey.\u201d<\/span><\/p>\n

Authors should provide as much documentation about how they run their science as they do about what they eat at conferences: where exactly was the study run, at what time and day, which research assistant run it (with contact information), how exactly were participants paid, etc.<\/span><\/p><\/blockquote>\n

We will trust everything researchers say. Until the need to verify arises.<\/span><\/p>\n

Solution 3. Post data, materials and code
\n<\/strong>Had the raw data not been available, the recent Science<\/em> retraction would probably not have happened. Stapel would probably not have gotten caught. The cases against Sanna and Smeesters would not have moved forward. \u00a0To borrow from a recent paper with Joe and Leif:<\/span><\/p>\n

Journals that do not increase data and materials posting requirements for publications are causally, if not morally, responsible for the continued contamination of the scientific record with fraud and sloppiness. \u00a0<\/span><\/p><\/blockquote>\n

\"Wide<\/p>\n

\n

Feedback from Ivan Oransky, co-founder of Retraction Watch
\n<\/strong>Ivan co-wrote an editorial in the New \u00a0York Times<\/em> on changing the incentives to reduce fraud (.html<\/a>). I reached out to him to get feedback. He directed me to some papers on the evidence of incentives and fraud. I was unaware of, but also unpersuaded by, that evidence. This prompted to\u00a0add the last paragraph in the incentives section (where I am skeptical of that evidence). \u00a0<\/span>Despite our different takes on the role of rewarding sexy-findings on fraud, Ivan\u00a0is on board with the three non-incentive solutions proposed here. \u00a0I thank Ivan for the prompt response and useful feedback. (and for Retraction Watch!)<\/span><\/p>\n

\n

Subscribe to Blog via Email<\/h2>\n\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t\t

Enter your email address to subscribe to this blog and receive notifications of new posts by email.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t