We did not choose this meta-analysis because it is atypical or especially problematic. We chose it because we believe it is typical and representatively problematic. Indeed, our critique is aimed at the enterprise of meta-analysis rather than at these particular meta-analysts, who, as far as we can tell, were simply doing what meta-analysts are instructed to do.<\/em><\/span><\/p>\n Authors of a recent PNAS article meta-analyzed 447 effects of \u201cnudging\u201d on behavior (.htm<\/a>). In the abstract, the authors report \u201cthat choice architecture interventions [i.e., nudges] overall promote behavior change with a small to medium effect size of Cohen\u2019s d = 0.43\u201d, and that \u201cthe effectiveness of choice architecture interventions varies significantly as a function of technique and domain\u201d.<\/span><\/p>\n This paper has gotten a fair bit of attention, both because of its claims about the average effects of nudges, and because letters published in PNAS, responding to this article, have proposed that the overall average effect may be much smaller \u2013 perhaps as low as zero \u2013 when correcting for publication bias (letter 1<\/a> | letter 2<\/a>). Indeed, letter 1 is titled, \u201cNo evidence for nudging after adjusting for publication bias.\u201d<\/span><\/p>\n In this post, we walk through a few studies included in this meta-analysis, and in so doing illustrate the problems with meta-analytic averages. We argue that these problems render meaningless both the published averages as well as the (ostensibly) bias-corrected versions of those averages.<\/span><\/p>\n Combining Incommensurable Results Obviously, we did not approach this task by reading all ~200 studies. In the interest of (our) time and to maximize informativeness, we decided in advance to focus on the extremes: the biggest and smallest effect size estimates within each category. Do these studies provide valid answers to the question of interest to the meta-analyst? Does it make sense to combine them? Let\u2019s take a look.<\/span><\/p>\n Reminders 1. The Smallest Reminder Effect: Something Is The \u201cDish of The Day\u201d in the State of Denmark The key measure was whether students chose the vegetarian option.<\/span><\/p>\n The original authors reported the results from each country separately, and thus so did the meta-analysts. This most negative effect of reminders in the meta-analysis (d = -.1223) represents the result that emerged in Denmark (N = 84), when only 4 out of 37 (10.8%) students in the intervention condition chose the vegetarian option vs. 7 out of 47 (14.9%) in the control condition [3<\/a>], [4<\/a>].<\/span><\/p>\n Note that it is not clear that this study should be included in the meta-analysis, since, as emphasized in the original article, students were not randomly assigned to condition, rendering these results correlational.<\/span><\/p>\n 2. The Largest Reminder Effect: Reminding (Whites) To Go To Sleep All participants wore a sleep monitor to measure sleep duration.<\/span><\/p>\n In the overall analysis, the authors found no effect of the text message reminders on sleep duration (p<\/em> = .471). But then they found a significant interaction with race\/ethnicity: reminders significantly increased the sleep hours of the 20 (non-Hispanic) Whites in the sample (d = 1.18, p<\/em> = .028), and had no significant effect on other ethnic groups (p <\/em>= .094, opposite direction).<\/span><\/p>\n The meta-analysis includes only the p <\u00a0 .05 result for (non-Hispanic) Whites, not the p > .05 result for the whole sample, and not the p > .05 result for minority students. In so doing, the meta-analysis introduced a form of publication bias \u2013 selectively including only the analyses that were statistically significant \u2013 that was not present in the original study [5<\/a>].<\/span><\/p>\n At this point, you may want to pause to consider some questions. Do you think these studies represent valid tests of the effects of reminders? Do you think it makes sense to average these studies together? What would that average tell you?<\/span><\/p>\n Food Choices The Smallest Food Choice Effect: Location, Location, Location (of the Risotto Primavera) This study came from an article published in 2018 in the journal Appetite <\/em>(.htm<\/a>). In that study, 750 participants on Prolific were asked to \u201cimagine they were catching up with a friend in a nice restaurant during the week\u201d (p. 193), and to select from a menu that contained 2 vegetarian and 6 non-vegetarian dinner options. The dependent variable was whether they selected a vegetarian option.<\/span><\/p>\n The researchers randomly assigned participants to one of four variations of the menu. There was a \u201cControl Menu\u201d and three modified menus. The negative effect at issue here involved a comparison between the \u201cControl Menu\u201d and a \u201cVegetarian Menu\u201d, for which \u201cthe vegetarian options are placed in a separate section of the menu\u201d (p. 192). Let\u2019s look at those two menus (the yellow highlighting is ours):<\/span><\/p>\n The Appetite <\/em>authors included this manipulation because they noticed that some restaurants group all vegetarian options together, and the authors reportedly had the intuition that this may decrease <\/em>demand for vegetarian options [6<\/a>]. Since what they observed was what they expected, it is not obvious that the effect here should be negative (d = -.24) rather than positive (d = +<\/strong>.24)\u2026 [7<\/a>].<\/span><\/p>\n (As a separate issue, there is a confound in the design here. See footnote: [8<\/a>].)<\/span><\/p>\n Regardless, what this study finds is that altering the menu in this particular way produced a small(ish) (positive or negative) effect on food choices.<\/span><\/p>\n The Largest Food Choice Effect: Eating More When There Is More To Eat As the authors report, \u201cOn 10 days over 5 months, we covertly recorded the food intake of customers who purchased a baked pasta entr\u00e9e from a serving line at lunch. On 5 of the days, the portion size of the entr\u00e9e was the standard (100%) portion, and on 5 different days, the size was increased to 150% of the standard portion.\u201d The customers and those who worked in the cafeteria did not know the portions were varied, and the price was the same regardless of portion size. The finding was, um, not subtle: People ate more calories of pasta when they were given more pasta, d = 3.08.<\/span><\/p>\n Imagine that these were the only two food choice studies in the meta-analysis. Do you think it makes sense to average across them, and to conclude that the average effect of nudges on food choices is d = (3.08-.24)\/2 = 1.42?<\/span><\/p>\n Conclusion #1: Meaningless Means What you wouldn\u2019t do is consult an average of a bunch of different effects involving a bunch of different manipulations and a bunch of different measures and a bunch of a different contexts. And why not? Because that average is meaningless.<\/span><\/p>\n Conclusion #2: What Is \u201cThe\u201d Effect of Nudges? In the commentary claiming, in its title, that there is \u201cno evidence\u201d for the effectiveness for nudges, the authors do acknowledge that \u201csome nudges might be effective\u201d, but mostly they emphasize that after correcting for publication bias the average effect of nudging on behavior is indistinguishable from zero. Really? Surely, many of the things we call \u201cnudges\u201d have large effects on behavior. Giving people more to eat can exert a gigantic influence on how much they eat, just as interventions that make it easier to perform a behavior often make people much more likely to engage in that behavior. For example, people are much more likely to be organ donors when they are defaulted into becoming organ donors [9<\/a>]. For the average effect to be zero, you\u2019d either need to dispute the reality of these effects, or you\u2019d need to observe nudges that backfire in a way that offsets that effects. This doesn\u2019t seem plausible to us.<\/span><\/p>\n The commentary authors also claim that nudges occurring within the finance domain have an effect size of zero <\/em>and that there is no heterogeneity within this domain<\/em>. Think about what this means. It means that any nudge that tries to influence financial decisions must have an effect size of zero. So a nudge that defaults people into a particular kind of auto insurance (J Risk Unc. 1993; .htm<\/a>) has the same true effect of zero(!) as a nudge that reminds people that they wrote an honor code to pay off a loan that they were already defaulted into re-paying (JEBO 2017, .htm<\/a>). Again, this doesn\u2019t seem plausible. Much more plausible is that those publication bias corrections don\u2019t work (see DataColada [30<\/a>],[58<\/a>],[59<\/a>])\u00a0and that some nudges work more than others [10<\/a>].<\/span><\/p>\n It is also worth thinking about publication bias in this domain. Yes, you will get traditional publication bias, where researchers fail to report analyses and studies that find no effect of (usually subtle) nudges. But you could imagine also getting an opposite form of publication bias, where researchers only study or publish nudges that have surprising effects, because the effects of large nudges are too obvious. For example, many researchers may not run studies investigating the effects of defaults on behavior, because those effects are already known.<\/span><\/p>\n In sum, we believe that many nudges undoubtedly exert real and meaningful effects on behavior. But you won\u2019t learn that \u2013 or which ones \u2013 by computing a bunch of averages, or adjusting those averages for publication bias. Instead, you have to read the studies and do some thinking.<\/span><\/p>\n Coda Author feedback: 1) The authors of the meta-analysis provided valuable feedback that helped us improve the accuracy of this post. <\/span><\/p>\n 2) Sz\u00e1szi et al., authors of the commentary entitled, \u201cNo reason to expect large and consistent effects of nudge interventions,\u201d provided the following response: (.pdf<\/a>). Andrew Gelman, another author of that commentary, wrote the following response: (.pdf<\/a>).\u00a0<\/span> <\/span><\/p>\n 3) Max Maier, an author of the commentary entitled, \u201cNo evidence for nudging after adjusting for publication bias,\u201d provided a 180-word response (.pdf<\/a>), as <\/span>well as <\/span>a link to a longer blog post they wrote in response: (.htm<\/a>). <\/span>We believe their response<\/span> incorrectly equates a meaningless mean – what we<\/em> are worried about – with the mean of a variable with high variance (what meta-analysts' tools model). To see the difference between meaningless means and heterogeneous means, imagine computing the average height of three structures in a city block. One structure is 2 stories high, another 2 meters high, and a third is 2 Taylor Swifts high. The average structure is 2 units high, SD=0. That mean is homogeneous but meaningless. In contrast, the average height of all members of Taylor Swift's family, measured in inches, is meaningful but heterogeneous.<\/span><\/span><\/p>\n 4) Armando Perez-Cueto, author of the \u201cDish-of-the-day\u201d study, provided helpful feedback on the post and helpful details about that study. Dario Krpan, author of the Vegetarian menus study, provided helpful feedback on the post. And Barbara Rolls, author of the portion size study, provided helpful feedback on the post as well as this response:<\/span><\/p>\n My lab does not work on nudging so I am surprised that our careful demonstration that portion size affects intake outside a lab would be included in such a meta-analysis. There was no nudging involved in the study. Covertly in the context of a cafeteria, the pasta dish was either bigger or smaller.\u00a0 There was no choice except to order pasta rather than something else, and there was no nudge.\u00a0<\/em><\/span><\/p>\n Like you, I share concerns about such meta-analyses and systematic reviews. Prescriptions of what is to be included miss the nuances and do not adequately account for study quality (or in this case even relevance to the question under review).\u00a0<\/em><\/span><\/p><\/blockquote>\n No other authors responded to our request for feedback.<\/span><\/p>\n
\n
\n<\/strong>In this post, we focus our discussion on the fact that this meta-analysis averages across incommensurable results. That doesn\u2019t mean that the authors ignored differences across studies. On the contrary, they conducted two separate moderator analyses, one that breaks down the results by nudge type (e.g., defaults vs. reminders) and one that breaks them down by domain (e.g., finance vs. food). In this section, we review studies within <\/em>two of these categories, essentially asking whether it makes sense to average within a subset of studies investigating reminders or a subset of studies investigating food decisions.<\/span><\/p>\n
\n<\/u><\/strong>The meta-analysts defined a reminder as anything that increases \"the attentional salience of desirable behavior to overcome inattention due to information overload\u201d [1<\/a>]. Figure 4 from the PNAS paper, reprinted below, shows that the average effect of reminders is d = .29. Let\u2019s review the studies generating the smallest and largest effect sizes and consider whether it makes sense to average across them.
\n<\/span><\/p>\n![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/Mertens-et-al.-Nudges-By-Intervention-Figure-Highlighted.png\")
<\/span><\/p>\n
\n<\/em><\/strong>In a study published in 2020 in the European Journal of Nutrition (.htm<\/a>), researchers reported a quasi-experiment conducted in four countries: Denmark, France, Italy, and the UK . Student groups were assigned, but not randomly<\/em> assigned, to one of two foodservice conditions [2<\/a>]. As shown in Figure 1 of the original article, the key intervention was whether the vegetarian option was labeled as the \u201cdish of the day\u201d or not:<\/span><\/p>\n![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/Veggie-Balls-Dish-Of-The-Day-Figure.png\")
<\/span><\/p>\n
\n<\/em><\/strong>A study published in 2017 in Sleep Health<\/em> (.htm<\/a>) randomly assigned high school adolescents (N = 46) to either receive twice-daily text reminders to go to sleep on time (e.g., \u201cBedtime goals for tonight: Dim light at 9:30pm. Try getting into bed at 10:30pm.\u201d) or to receive no reminders about sleep.<\/span><\/p>\n
\n<\/u><\/strong>In this section, we look at studies investigating nudges of food choices, because the abstract identified those as having the biggest effect (an average of d = .65). Specifically, the abstract reports that \u201cFood choices are particularly responsive to choice architecture interventions, with effect sizes up to 2.5 times larger than those in other behavioral domains.\u201d But does it make sense to average across studies investigating food nudges?<\/span><\/p>\n
\n<\/em><\/strong>Although the authors report that nudges targeting food choices are the most effective, some of the food nudges in the meta-analysis did not work. Indeed, the least successful food nudge involves a moderately negative effect<\/em>, indicating that the nudge backfired <\/em>to the tune of d = -.24. In other words, the manipulation caused a nontrivial effect in the wrong direction<\/em>. Let\u2019s try to understand what happened.<\/span><\/p>\n![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/Menu.png\")
<\/span><\/p>\n
\n<\/em><\/strong>The biggest positive effect of a nudge on food choices comes from a 2004 paper published in Obesity Research <\/em>(.htm<\/a>). This study, which was conducted at \u201ca public cafeteria-style restaurant on a university campus\u201d, asked the question: Do people eat more food when they are given more food?<\/span><\/p>\n
\n<\/strong>Imagine someone tells you that they are planning to implement a reminder or a food-domain nudge, and then asks you to forecast how effective it would be, would you just say d = .29 or d = .65 and walk away? Of course not. What you\u2019d do is to start asking questions. Things like, \u201cWhat kind of nudge are you going to use?\u201d and \u201cWhat is the behavior you are trying to nudge?\u201d and \u201cWhat is the context?\u201d Basically, you\u2019d say, \u201cTell me exactly<\/em> what you are planning to do.\u201d And based on your knowledge of the literature, as well as logic and experience, you\u2019d say something like, \u201cWell, the best studies investigating the effects of text message reminders show a positive but small effect\u201d or \u201cGiving people a lot less to eat is likely to get them to eat a lot less, at least in the near term.\u201d<\/span><\/p>\n
\n<\/strong>What are the implications for the debate about the effect size of nudges?<\/span><\/p>\n
\n<\/strong>In this post, we reviewed only four effects from a meta-analysis of 447, but you just need four to see the problems [11<\/a>]. In some cases, the studies are not valid tests of the research question. Moreover, regardless of their validity, averaging across them generates a number that has no discernible meaning.<\/span><\/p>\n![\"Wide](\"https:\/\/datacolada.org\/wp-content\/uploads\/2014\/02\/Wide-logo-300x145.jpg\"){kind=logo}
<\/span><\/p>\n
\n<\/strong>Our policy (.htm<\/a>) is to share drafts of blog posts with authors whose work we discuss, in order to solicit suggestions for things we should change prior to posting, and to invite them to write a response that we link to at the end of the post. We contacted (i) the authors of the PNAS meta-analysis, (ii) those of the two PNAS letters, and of the individual studies we use as examples.
\n<\/span><\/p>\n
\n