(Illustration by Chad Hagen)
In 2003, the David and Lucile Packard Foundation funded a 10-year initiative to advance policies that would support high-quality preschool for California’s children. Knowing that the advocacy strategy would unfold without a predictable script, the foundation’s Children, Families, and Communities program team hired external evaluators, led by one of this article’s coauthors (Julia Coffman), to help them track progress and support their learning and decision-making. The funders and evaluators worked together at the start to develop the evaluation questions and to shape the methodology.
As the initiative and evaluation launched, everything seemed to be going according to plan—until the evaluators submitted an interim report in 2007. At the meeting to discuss the report, the program director began with a revealing statement: “Thank you for this; it’s well written and nicely packaged. But honestly, it doesn’t tell me anything I didn’t know already. I don’t know what to do with it.”
Variations of this scene play out regularly in foundation meeting rooms. Why is it so hard to apply findings from evaluation, research, and other data-driven projects to philanthropic decisions? Why does evidence so often seem to generate more questions than answers? How can social sector leaders better bridge the disconnect between knowledge and action?
Because we in the social sector have been trained to believe that smart decisions thrive on evidence, it’s easy to assume that the act of gathering evidence will make the difference between a poor decision and a wise one. In philanthropy, however, that assumption fails to hold time and again.
One of the most striking and consistent findings from research with foundations, policymakers, and nonprofits is that even when evidence is available, it is rarely used. A study of policy papers that the World Bank published between 2008 and 2012, for example, revealed that nearly a third of them had never once been downloaded, and more than 70 percent were accessed fewer than 100 times. Benchmarking research on evaluation in philanthropy has found that more than three-quarters of the foundations surveyed had difficulty commissioning evaluations that resulted in “useful lessons” or “meaningful insights” for the foundation, their grantees, or the field. Not surprisingly, therefore, more than 125 social sector leaders interviewed as part of Monitor Institute by Deloitte’s Reimagining Measurement initiative identified “more effectively putting decision-making at the center” as the sector’s top priority for the next decade.
Why Is Evidence Not Used?
Research on the use of evidence in philanthropy has noted a number of barriers to its greater adoption. One is that funding professionals and policymakers consistently report being stretched for time. A William and Flora Hewlett Foundation-commissioned report on knowledge dissemination among foundation staff noted that some respondents have so little bandwidth that they make a habit of deleting notifications about new studies before even checking them for their relevance.
Another frequently cited barrier is a lack of facility with data analysis. A study of more than 1,600 highly educated and competitively selected civil servants in Pakistan and India indicated that many nevertheless believe they lack adequate training to interpret and contextualize data, and, sure enough, most of them struggled to answer routine data analysis questions correctly when prompted.
An additional barrier is that using evidence may not always be at the top of decision makers’ minds. A literature review from Mathematica Policy Research noted that more urgent matters, including “political feasibility, personal priorities, public opinion, social implications, and budget constraints,” frequently crowd out consideration of research and evaluation findings.
Furthermore, overcoming these practical barriers of time, training, and motivation alone is not enough to erase the disconnect between evidence and the decision-making process. The gap is also the result of organizational design, misconceptions about the role and value of data, and a failure to prioritize decision-relevant questions. Informed by decision analysis literature, as well as our work with foundations and other social sector entities, we outline four of these less frequently acknowledged, but no less important, reasons why evidence so often falls short of its potential.
The problem is not always a lack of information. Since the inception of the measurement-and-evaluation function in philanthropy half a century ago, social sector leaders have been trying to promote evidence use in organizational settings. The most common strategy to date has been to increase the supply and methodological rigor of information available, filling gaps in knowledge where they exist. This strategy has led to an explosion of social sector research and evaluation, a proliferation of academic databases to catalog the results of this work, and numerous “What Works”-style clearinghouses synthesizing evidence on various topics. But even though evidence is more readily available today than at any previous time in human history, social sector leaders still make many of their toughest decisions without it.
What accounts for this gap between intention and action? One possibility is that social sector leaders simply have unrealistic expectations of the value that evidence is capable of providing in the decision-making process. Hackneyed calls for “data-driven” or “evidence-based” decision-making carry with them an implicit assumption that all of the other elements of an effective decision process are unimportant or easily implemented.
But decision analysis literature reveals a number of opportunities to improve decision-making that don’t directly involve evidence. For example, Stanford Graduate School of Business professor Chip Heath and his brother, Dan, a senior fellow at Duke University’s Center for the Advancement of Social Entrepreneurship, have proposed a four-step heuristic called the WRAP method to aid deliberation. (WRAP stands for “Widen Your Options, Reality-Test Your Assumptions, Attain Distance Before Deciding, and Prepare to Be Wrong.”) It offers suggestions for widening the number and variety of options under consideration, taking steps to combat common cognitive biases, and planning for multiple scenarios to hedge one’s bets. The Decision Quality framework developed by Strategic Decisions Group, a consulting firm cofounded by Stanford professor and decision analysis pioneer Ronald A. Howard, includes considerations such as “clarifying values and trade-offs” and “committing to action.”
Another element that the framework includes is “setting the right frame.” Arguably, no decisions are more important to a foundation or nonprofit and the communities it serves than selecting a mission, defining a scope of concern (e.g., a specific geographic area), and articulating the fundamental values of the organization, including who has authority to make decisions and to whom the organization is accountable. While evidence can play a role in these discussions—the effective altruism movement, for example, has made a point of finding and promoting “high-impact” causes based on its members’ analysis of need and opportunity—the determination of what evidence to use and believe unavoidably reflects choices of value systems and theories of knowledge.
These more holistic constructions of the decision-making process show that the role of evidence in good decision-making is often indirect or secondary. Indeed, neglecting any of the non-evidence-based dimensions of decision-making can easily destroy the value of an otherwise robust information-gathering effort.
We crave certainty that evidence doesn’t provide. Leaders often look to evidence to counter anxiety that comes from the uncertainty inherent in decision-making. But even the best designed and most rigorously executed evaluation or research study can’t ensure that decisions based on it will lead to the expected outcomes. For one, any methodology involving sampling of a population will come with statistical margins of error, not to mention potential doubts about data quality, sample representativeness, and relevance of the measures used. Application of evaluations to third-party programs cannot avoid the question of external validity or how relevant the results are when transposed outside the current context.
A more fundamental disconnect is also at play, one that does not receive nearly as much attention as the aforementioned issues. Research and evaluation are designed to answer questions about the past and present. Decisions, on the other hand, are predictions: They require us to imagine and compare the likelihood and desirability of disparate futures that might follow from the actions we take. And even the best possible evidence about the past or present can’t offer any guarantee that the situation will remain stable going forward.
All decisions are made in a cloud of uncertainty. While stretching the boundaries of our knowledge may be one of the most obvious ways to improve the quality of our decisions, doing so will have limited value until we have a strong understanding of what we don’t know and can’t know. In fact, acknowledging our uncertainty can even help us to make better use of evidence about what we do know, because it helps us to mitigate our biases and be more open to the information we have.
It’s easy to ask irrelevant questions. Questions drive everything from choices about design and methodology to the framing of the final knowledge product. If they are not the questions that users need answered, the evidence will not be used.
The challenge is that decision questions are fundamentally different from evaluation questions. Decision questions are normative and prescriptive and take the form of “What should we do in this situation?” By contrast, evaluation and research questions are descriptive and ask what happened and why.
If, for example, the decision question is “Should we renew this grant program?” an evaluation question such as “What proportion of people in our grantees’ target audience changed their behavior in response to our grantees’ activities?” might well provide useful context. But decision makers should not expect that an answer to the second question will provide an answer to the first. The evaluation question in this example explains nothing about what threshold of behavior change in the target audience is sufficient to proceed with the grant renewal. Nor does it address the cost involved in generating that behavior change, and it leaves unexamined any additional or related goals that may be important to the grant maker. Renewal decisions typically involve many such variables, and weighing the relative importance of each is a difficult analytical problem that consulting or creating a single piece of evidence rarely resolves.
Learning and decision-making are organizationally siloed. Evaluation and decision-making typically represent distinct steps in the management cycle, each involving unique practices, techniques, and considerations. As such, foundations often assign responsibility for evaluation and decision-making to different teams. Program teams develop and execute strategies and make grants, and monitoring, evaluation, and learning (MEL) teams track the progress of those strategies and share the findings with their program team colleagues. Foundations usually leave ultimate authority for decision-making to program teams or the executives who oversee them, and expect MEL staff to provide information but not participate in the decision-making process.
This division of responsibilities can fracture the relationship between the learning and decision-making functions of an organization. For example, an evaluation officer at the Colorado Health Foundation hired external evaluators to help the foundation understand how to structure grantmaking to improve the capacity of organizations to advocate for local policy change for healthier environments. Over time, the program officer chose to fund a wide variety of initiatives that broadly addressed social determinants of health—a departure from the initiative’s stated goal of focusing on healthy eating and active living. The foundation’s team was not proactive enough in communicating these changes with the external evaluators, who were then not aware of the shifts.
Nearly a year into the effort, the external evaluation contractors came back to the foundation to say that because of the grants it was actually making, the original evaluation questions didn’t fit with where the foundation was headed and the decisions it would need to make. A scramble to retrofit the evaluation ensued, but the foundation had already made some critical decisions about what to do next on this strategy, and ultimately the data already collected had lost much of their relevance to the foundation’s future direction. Unless evaluation teams and program teams are regularly communicating with each other and with any relevant consulting partners about key shifts in thinking, actions, or deadlines for decisions, monitoring and evaluation plans set up at the onset of an initiative will likely lose relevance over time.
Five Ideas to Use Evidence More Effectively
Even though evidence can contribute only partially to effective decision-making, it is still an important mechanism for improving social sector outcomes. Fortunately, a number of tools and tactics—rarely used effectively in philanthropy but with long track records in other professional communities—can help us become savvier consumers of information.
Try decision inventories. Teams routinely dive into evaluations and other analytical projects knowing how the exercise is thematically related to their work, but they rarely take the time to articulate the specific decisions the analysis is intended to inform. This happens partly because most organizations do not make a habit of cataloging the decisions they need to make. They shouldn’t be surprised, therefore, when, after they have completed a lengthy research or evaluation project, the information that they have collected isn’t as actionable as they originally imagined. Fortunately, a simple exercise called a “decision inventory” can help.
A decision inventory involves eliciting upcoming decision dilemmas from individual team members, then sorting the decisions into priority tiers in which more complex, uncertain, and higher-stakes decisions receive more attention. This exercise offers two advantages. First, it makes decisions explicit, rather than implicit. Second, it helps ensure that the level of resources devoted to analyzing each decision is proportional to that decision’s risks, opportunities, and difficulty.
While no universally adopted methodology for decision inventories exists, various decision analysis consultants, such as Strategic Decisions Group, have developed their own versions of the exercise. Building decision inventories around specific evidence-gathering projects is also possible. Imagine how the process might work, for example, for the evaluation of the Packard Foundation’s advocacy initiative, discussed at the beginning of this article. Based on the results to date, should that program be continued, revised, expanded, or ended? Does the strategy underlying the program’s design need to be adjusted? If so, how? What are the options on the table, and what are the relevant considerations for each path?
Rehearse decisions. Arguably, the clearest way to forge a stronger link between evidence and the decision-making process is to “rehearse” decisions, in the same vein as preparing for an important presentation or event through a rehearsal. Once upcoming decision dilemmas are clear to all, nothing prevents team members from breaking down and analyzing the decisions together. Rehearsing in this way accomplishes two things: First, by simulating the act of decision-making in advance, it helps to determine what information will be most useful when it’s time to make the real decision. Second, it creates a gap between the time when a team identifies decision-relevant questions and the time when it needs answers to those questions—a golden opportunity for an enterprising evaluator or MEL team to fill that gap.
Ambitious decision makers can go further by using decision trees or Monte Carlo simulation—a statistical method that imagines thousands of possible futures based on the factors identified as most important to the decision. These methods not only estimate the likelihood of various decision outcomes and their desirability but also help to home in on what specific additional information could most effectively reduce uncertainty about the right next step. For example, if the choice is whether or not to invest in a new impact venture fund, building a quantitative model for the decision might reveal that the recommendation hinges on the expected number of kilowatt-hours that the investment would divert from fossil fuel energy sources, and that narrowing the range of uncertainty about that estimate would increase confidence in the correct path. The discipline of decision analysis even offers a handy calculation to decide how much we would be willing to spend to gather that information: the expected value of information (EVI). This tool translates the benefits of the uncertainty reduction expected from new evidence into economic terms—conveniently, the very same translation decision makers must undertake when determining whether to commission, and how much to budget for, a research or evaluation project.
This way of approaching decisions has gained wide adoption in the energy and pharmaceutical industries. Decision scientist Douglas W. Hubbard, for example, has developed a method for modeling high-stakes investments across numerous contexts, including international development. In How to Measure Anything, Hubbard explains that many variables in a typical decision model have an information value of zero—not because the information wouldn’t be interesting to have, but because even if it were available, it wouldn’t change the recommendation. “The highest-value measurements almost always are a bit of a surprise to the client,” he writes, implying that measurement plans differ substantially from their existing practices—a phenomenon he coins the “measurement inversion.” Modeling our decisions can thus save significant time and money by deterring decision makers from chasing data that are unlikely to matter. Conversely, factors that do have a high information value can form the basis of powerful questions that virtually guarantee the relevance of any research or evaluation efforts that attempt to answer them.
Use strategic learning agendas as a bridge. Bridging the divide between retrospective evaluation questions and prospective decision questions can increase the odds that evaluation questions will yield answers that are useful for decision-making.
For example, consider a foundation that supports grantees to provide better access to mental health services in a community. The foundation is committed to this work but has developed a particular interest in figuring out how it can better address local disparities in access to mental health services. As the foundation prepares to decide how to do that, program staff want to use the findings from an evaluation of their existing portfolio to inform their thinking. They are likely to be disappointed when they do this, as that evaluation was not designed to look at how grantees addressed disparities in access. Instead, it examined whether and how existing grantees addressed access more broadly. While this is an important inquiry, it may not reveal anything about how the foundation might increase its ability to mitigate future disparities.
One tool that foundations use to link evidence more strongly to decisions is a learning agenda—a set of questions about what strategists need to know to do their work effectively, so that they achieve the intended results. Unfortunately, the learning agendas that foundations typically use tend to be exhaustive lists of study questions that represent the broadest scope of things that staff believe they need to learn about a location, a problem, a set of actors, a body of research, or potential solutions. For example, a foundation experimenting with different ways to support childcare providers might have learning questions that include: Which experiments show the most promise in terms of results? Which experiments can we scale, and how? How can we ensure that we sustain our progress? However, questions like these are rarely tethered to the specific decisions that the foundation needs to make and can be so overwhelming in their size and scope as to immobilize staff, confusing them about the direct process forward.
Two simple shifts in how learning agendas are organized can clarify the link between evidence gathering and decisions. First, learning agendas should be organized around forward-looking questions that derive from the decisions that need to be made in the future, as the strategy is implemented. This keeps the decision and possible future actions at the forefront, instead of as an afterthought. It also reminds decision makers and evaluators that an evaluation alone cannot answer the question about what to do next. Instead, evaluation is one among many informational inputs into a decision.
Second, a refined list of study questions to guide information- and evidence-gathering—whether landscape scans, evidence reviews, community/stakeholder feedback mechanisms, a comprehensive evaluation, or staff insights from experience—should flow from, and be tested for utility against, the overarching decision question. As decision makers rehearse the decision, they can refine these study questions to pinpoint what specific information they would need to uncover through evaluations, evidence reviews, stakeholder feedback, or other data-collection methods to inform the decision. The example (on page 53) illustrates the learning agenda architecture for the foundation, mentioned earlier, that is figuring out how to address local disparities in access to mental health services.
Some answers to the questions in this learning agenda may require formal research or evaluation. Predictive exercises, such as scenario planning or decision modeling, can answer other questions. And for some questions, less formal observation, conversations, and reflection might be sufficient. But every question relates to the decision that needs to be made. This approach to learning agendas helps to integrate information from a variety of sources into the decision-making process, disabusing decision makers and evaluators alike of the idea that a particular research or evaluation effort alone will provide all of the answers.
Ensure that support staff see their roles as critical thinkers. Most foundations hire program staff for their expertise in a particular subject or area. According to social and personality psychologists, all people have a set of “role identities” in their personal and professional lives that represent how they think about their responsibilities and what it means to be good at them. What we see as our primary role is important because it governs how we act.
Those who view their primary role as that of expert may think they should know the answers to most questions about their area of expertise, and that they should base their decision-making more on past experience and existing knowledge than on the acquisition of new data or the practice of challenging one’s own thinking. Compare this with those who view their primary role as that of a critical thinker. These people are more likely to question their assumptions, be open-minded about divergent views, and regularly acknowledge and look to expand the limits of their knowledge, instead of assuming they have it already.
Because it helps people to seek out and be open to incoming information, especially when it challenges their point of view, organizations that frame staff’s primary roles as those of critical thinkers, more than subject matter experts, have a better chance of bridging the disconnect between evaluation and decisions.
Some funders have been making this shift. The Colorado Health Foundation, for example, in its recent strategic overhaul, began working directly in local communities toward achieving health equity in the state. Its program staff shifted their roles and responsibilities from being experts to being critical thinkers and developing a deep knowledge of what was happening on the ground in communities. Program officers were reassigned to cover new issue areas and geographic regions; almost all of them left behind established expertise and relationships in particular fields of work. As a result, rather than being topical experts working primarily from their offices, program staff are expected to be in the field a minimum of 40 percent of their time, engaging with community members and gathering evidence about community needs, opportunities, and dynamics. They now see themselves as responsible more for gathering evidence and making sense of what that means for how the foundation supports communities than for simply providing answers.
Don’t treat decisions as handoffs. To achieve a better connection between evaluation and decisions, we recommend bringing together process experts (e.g., MEL staff and consultants) and content experts (program staff). Typically, evaluators, whether they are staff or consultants, provide information and hand off decisions to program staff. But research suggests that involving evaluators in decision-making could actually lead to better decisions.
Small groups—program teams in foundations—that discuss a situation and reach a conclusion by vote or consensus make most organizational judgments and decisions. But according to law professor Cass Sunstein and behavioral scientist Reid Hastie, group judgments via this method are not necessarily more accurate than individual judgments, and in some cases are even more prone to bias. For example, deliberative groups are more likely than individuals to fall prey to planning fallacy (the tendency to underestimate how long projects and initiatives will take), base-rate neglect (the tendency to pay too much attention to the specifics of a case, rather than to the general category to which it belongs), and framing effects (perceiving an outcome as better or worse depending on an arbitrary reference point). Sunstein and Hastie’s review of the research literature indicates that pressure for consensus tends to silence dissenting voices in group settings, causing participants to withhold relevant information or perspectives that would lead to a richer shared understanding of the decision problem.
To avoid such flawed judgments, make room in the decision-making process for alternative points of view. Sunstein and Hastie recommend that executives and other leaders with formal authority censor themselves early in decision-making meetings so as not to bias the rest of the group in favor of their views. Indeed, a range of literature supports the notion that what we might call “intentional facilitation” is crucial to promoting critical thinking in group settings. Such facilitation involves an explicit emphasis on critical thinking and strategic learning; employs efforts to heighten engagement, foster trust, and encourage diverse perspectives; and simulates realistic hypothetical situations and multiple potential outcomes. Because creating a space for reflection and learning is often already part of their mandates, MEL staff or consultants are ideally placed to play such a role in a deliberative group decision-making setting.
Applied Learnings
For the Children, Families, and Communities program team at the David and Lucile Packard Foundation, a lot has changed since 2007 to ensure that evaluation findings are more relevant and used for decision-making. The program team is applying these changes to its current 10-year initiative, Starting Smart and Strong, launched in 2014. The initiative seeks to ensure that all California children grow up healthy, confident, and ready to learn. It is a place-based and emergent strategy in which three communities are making their own decisions about how to use foundation resources to achieve school readiness.
First, the program team planned the initiative in developmental stages, anticipating the decisions it would need to make during each phase and not getting too far ahead of itself. During the first phase, the team decided primarily how to support community capacity to develop and execute a plan for improving how adult caregivers interact with and care for children. During the second phase, decisions were about how to support testing and experimentation in communities. In future phases, decisions will focus on how to support communities as they scale effective practices.
The foundation also decided to hire a developmental evaluation team from Engage R+D to support its decision-making. Developmental evaluation is well suited for social change strategies situated in complex or uncertain environments because it facilitates continuous data collection, feedback, and learning. Developmental evaluation has a natural focus on decisions; its goal is to help evaluation users determine what to do next.
The developmental evaluators help the program team by asking tough evaluative questions, uncovering untested assumptions about how change will occur, and collecting and interpreting evidence to inform broad strategic questions, as well as specific decision dilemmas. They are in regular contact with both the foundation and the grantees so that they stay up-to-date on what is happening and how they might need to shift their approach in response. The evaluation team has the flexibility it needs to adjust its plans regularly, making sure the evaluation work is aligned with the right decision questions at the right time. Evaluators match the resources spent on investigating a question with the value of information that investigation might be expected to provide.
Program staff now view their role as being partners with communities, rather than experts who drive community decisions. Because communities are leading the work in partnership with program staff, foundation leadership emphasizes the importance of program staff’s being critical learners and thinkers, welcoming divergent perspectives to the table, and working at a pace that allows them to make space for reflection in the work.
Program staff have also built learning habits and routines into their day-to-day work so that they can get and use evidence regularly, including but not limited to when evaluators supply it. For example, they are asking more pointed questions so that evidence generates links to the decisions they need to make.
Every year, program staff develop new learning agendas that focus specifically on decisions they need to make over the next 12 months. These include questions such as: What would it take for our grantees to plan for and begin to scale within each community? What would it take to develop practices around improving the quality, consistency, and use of data in communities? At the end of the year, they have an annual “look-back session” to discuss what they learned and how they will apply that information.
As a result of these shifts in how the program and evaluation team relate, the evaluation team’s work is connected to the program team’s and the communities’ ongoing questions and decisions. Evidence enters their strategy discussions regularly and in multiple forms, not just occasionally in written evaluation reports.
Having a robust evidence generation and learning practice is essential for strategies like this, which requires regular decisions about what to do next to meet long-term goals. Since 2014, the program team has made numerous strategic adjustments in response to what it has learned from evidence. For example, in 2018, it tested, challenged, and revised assumptions about the kinds of technical assistance communities would need to scale effective practices; that process led to changes in foundation decisions about how to support communities on scaling.
Building a habit of smart decision-making requires leadership commitment and collaboration, and individual employees or consultants are rarely in a position to make it happen on their own. For that reason, we recommend beginning with reforms inside one’s own sphere of influence. A CEO might take more diligent steps to model reflection and making space for opposing views in leadership meetings, while a program officer might have more success experimenting with decision inventories and rehearsing department-level decisions in advance. The most productive steps to take will look different, depending on one’s role in or outside the organization. The most important lesson to remember is that smart decision-making requires a collaborative culture in which the organization’s expectations, values, and experiences guide decision-making behavior. Just as we all have a role to play in making our workspaces safe, happy, productive, and equitable, we also have a role to play in ensuring they are wise.
Read more stories by Ian David Moss, Julia Coffman & Tanya Beer.