I confess that I am an experiment chauvinist – I look down on studies that are purely observational, studies that don’t manipulate anything. Where does my prejudice come from? One factor is that as a perceptual and cognitive psychologist, when I do science, I’m usually interested in the causes, or underlying mechanisms, of a phenomenon. For the phenomena that I’m interested in, typically one can easily do a controlled experiment that allows one to infer a cause of the phenomenon.
For many aspects of the universe that humans are interested in understanding, experiments are often not feasible, and sometimes wouldn’t even be appropriate to achieve the sort of knowledge researchers are after. Below, for a class I teach called “Good science, bad science”, I tried to get beyond my provincial experiment-centrism to explain to students the value of studies that make observations but don’t manipulate anything.
—
Most sciences advance through a combination of observational and experimental studies, often done by different researchers. For example, in medicine, treatments for diseases are usually best tested with experiments, for example with half of a group of patients randomly assigned to one treatment and half assigned to another. However, observational studies, where the researchers don’t actually manipulate anything, can also be critically important to advancing knowledge.
Decades of study of health records found that people who exercise more have less heart disease. Because this was an observational study, however, the lower heart disease rates in those who exercise a lot might have been due to confounding factors. Perhaps only people who start out without chronic diseases are able to exercise much, or perhaps people who live in rural areas with cleaner air are more likely to do outdoor activities, which often involve exercise. So it might have been that the reason for the lower incidence of heart disease is due to breathing cleaner air, not due to getting more exercise.
An experiment in which a random half of the study participants are assigned to exercise, and the other half don’t, helped resolve the debate, because the random assignment ensures that, on average, there will will be no confounding difference between the groups, such as living in a place with cleaner air. However, doing experiments with people is often both more difficult than only observing them, and also much more expensive. If everything goes well (e.g., those assigned to the exercise group actually do the exercises, and those assigned to the other group don’t), one may be able to safely conclude that exercise reduces the chance of heart disease. There are problems, however, of generalizing this to the real world, where very few may actually exercise as intensively, as frequently, or in the same way as those who followed the exercise protocol of the experiment.
Often, neither observational studies nor experiments fully answer a research question by themselves, but when they both point to the same conclusion, we can justifiably be very confident in that conclusion.
Some fields of research, such as astronomy, are almost entirely observational. For many thousands of years, people have speculated about what causes the motion of the stars and planets. Various hypotheses were invented, hypotheses which could never be tested with experiments, because people were never able to change the movements of astral bodies. However, by amassing a very large set of observations, people made progress by revising their theories so that they could explain more and more of the observations.
Tycho Brahe’s observatory, which also extended underground. Image: public domain.
Johannes Kepler was uncompromising in his quest to explain the precise observations of the movements of the planets that had been made by Tycho Brahe. In his dogged attempts to fit the data, Kepler came up with the idea that the planets followed elliptical orbits. This explained Brahe’s observations better than the circular-orbits version of heliocentrism, contributing to heliocentrism’s eventual triumph.
Johannes Kepler (1571—1630). Image: public domain.
In biology, ideas about the origins of plants and animals came about almost entirely through considering observations. From the meticulous records of a long line of European naturalists, Darwin knew of many thousands of observations regarding various plants and animals. When combined with his own observations during the voyage of the Beagle, including in then-remote (to Europeans) places like Australia, Darwin formulated his theory of “descent with modification”, now known simply as “the theory of evolution”.
An illustration by G.R. Waterhouse of a native rat that Darwin and he caught in southwest Australia and documented for European science. Image public domain.
The concept of reproducibility and replication, a focus of this class, can be more complicated for observational sciences than for the experimental sciences. If subsequent researchers wanted to confirm that Australia had a rat species that really looked the way that Waterhouse and Darwin had illustrated, they could go to southwest Australia and set out a trap with cheese as Darwin had done, but even if they put the trap in exactly the same location, they were unlikely to end up with the exact same rat in their trap and as the local population of those rats may have shifted locations, so they might not catch any. Because the world is always changing, it can be hard to know whether a difference in observations should cast much doubt on a previous study.
Sometimes one can build replication into the initial effort to make observations. For example, when an important event is predicted to occur in astronomy, researchers arrange for multiple telescopes around the globe to collect observations near-simultaneously. That way, if one telescope yields different results than the others, the researchers will know that they should investigate whether it was functioning correctly before trusting its observations.
Alex Holcombe's blog 