The common-sense theory of science is that we gain knowledge through the senses, that all our theories are either derived from or justified by sense data. This idea is called empiricism. It says that we know the Earth moves around the Sun because we can see it happening; cells exist because we can see them under a microscope; stars exist because we observe them through telescopes; and so forth.

But it is theory, not observation, that shapes our understanding of the world, much of which will forever remain unseen. Perhaps we will never explore the planets in other solar systems, or the stars in other galaxies, some of which are moving away from us so fast that we will never catch up to them again. We have not seen most of our own planet, having only explored part of its surface and some of the depths of its oceans. On top of that, there are still significant gaps in our understanding of the inside of the Earth.

There are other problems with empiricism too, since certain theoretical concepts preclude observation. At a cosmic scale, most of us will never explore the inside of a black hole, and anyone who did would never be able to relay his experience, being doomed to disappear into the centre after having crossed the Schwarzschild radius. Yet we know from theory that black holes have an interior. Feynman, in Surely You’re Joking, Mr Feynman!, gave a cute example of something else that is never seen: the inside of an object. Take a slab of metal. It has an inside. How do we know? We can cut the metal in two, demonstrating that it isn’t hollow. But in doing so, we create a new outside. What was previously the inside has become another surface, and is now visible. It takes conjecture to say that what is now the outside was once the inside.

There are many other examples of unseen things playing a key role in explaining the world. Long before atoms were directly observed, they provided a good theory of gases. Boltzmann used the then-unobserved atomic theory to derive properties of gases, including the ideal gas law, from more elementary assumptions about the motion of particles. It was only later that the atomic theory was tested experimentally. In particular, Einstein’s 1905 paper on Brownian motion provided a way to test the atomic theory. He proposed that the jiggling of particles like pollen on water, known as Brownian motion, was caused by unseen water molecules hitting them at random, and later experiments confirmed his predictions. Yet people like Boltzmann were right to use the atomic theory before it was ever tested, and similar people like Mach were wrong to criticise atomic theory for postulating atoms that could never be seen, not just because it turned out they could be seen, but because they already explained so much of the world around us.

A further problem for the empiricist is that false theories can make excellent predictions. Classical fluid theory, for instance, treats fluids as continuous substances that are smooth everywhere (or what is called differentiable, in maths jargon). It makes excellent predictions about pressure, turbulence, wave motion, and many other phenomena, except at atomic scales. One thing it fails to explain is droplet formation. As a droplet forms, a thin strand of liquid connects it to the main body. As the strand narrows, the liquid is no longer smooth, so the classical theory says the droplet cannot form (see image below). Yet obviously, droplets do form, as anyone who has been caught in the rain can attest. Classical fluid theory struggles to describe the final stages of droplet formation, where the neck becomes so thin that atomic effects dominate and the assumption of continuity breaks down.1

Still, classical fluid theory predicts many properties of water with high accuracy, because in most cases the atomic nature of the fluid does not matter. There are simply too many atoms for it to show. But when the system involves only a few atoms, as in the case of a thinning strand, the atomic theory becomes crucial. Hence, the theory is false, but we still use it to describe the motion of currents, vortices, and many other properties of liquids to this day.

A similar, more fundamental issue arises in particle physics. Quantum field theory has a problem: at very small length scales, the energy of a field becomes infinite. If you ask, “what is the energy of the field here?”, the exact answer is infinite. To get predictions out of the theory, physicists introduce a cut-off, a way of ignoring contributions from the very shortest length scales. This is one method of taming the theory and ensuring that its outputs remain finite. It works astonishingly well in practice, with countless experimental confirmations, including at CERN, and no known experiments that refute the theory. Yet from a theoretical point of view, it is problematic that we must impose an artificial boundary in order to get results. As with fluids, this is a case of a divergence, a prediction that certain quantities blow up to infinity. Just as atomic theory fixes the divergences in classical fluids, a future theory may resolve those in quantum field theory by telling us what actually happens at the smallest scales.

These issues are awkward for empiricists, who emphasise predictability. Quantum field theory has never failed a test, but it is almost certainly incomplete, because it says nothing about what happens at the smallest scales. So although it is superbly predictive, it must be false.

The lesson here is that observation and predictability are not the true measure of a scientific theory. What matters is its ability to explain the world. Some theories, like quantum field theory, are highly predictive and have never been refuted by experiment, yet we know they are incomplete. Others, like atomic theory in Boltzmann’s time, explained a great deal before they could ever be tested. A theory can be deeply explanatory and reliable even if it postulates things we cannot see, and conversely, a theory can be empirically successful yet still be in need of replacement.