How Science Works

a talk I gave on the Climate and Weather Livestream

The 2nd Weather and Climate Livestream is now underway. It is organized by a group of early-career researchers concerned about declining funding for the field and eager to demonstrate how weather and climate research makes a difference in our lives.

Last year’s event was a great success, drawing terrific press coverage including a piece in the NY Times. This year features a strong lineup of talks aimed at a general audience — I highly recommend tuning in.

link

I gave a talk yesterday about How Science Works in Three Steps. You can see the short presentation here (I’ve appended a transcript below):

And please tune into the livestream here. It runs through tomorrow at 6 pm ET.

Cleaned up transcript:

Thank you, and thanks to everyone tuning in. I want to talk about something I think a lot of people don’t really understand: how science actually works.

Arguments over any scientific topic — climate change, vaccines, you name it — revolve around competing claims. For climate change, the claims are: Is the Earth warming? Are humans to blame? What will the impacts be? Over the last 150 years, the scientific community has developed a remarkably robust method for sorting out which claims are true and which aren’t. That method is what I want to walk through.

So how does science generate knowledge? If you ask a high schooler, they’ll give you the version they learned in ninth grade: an individual scientist forms a hypothesis, runs an experiment, and draws a conclusion. That’s more or less what happens at the individual level, though in practice it’s more iterative — you design, test, and refine all at once, landing on a slightly different idea each time until you reach a conclusion about whatever hypothesis you’ve ended up with.

But this isn’t science. It’s only step one. Any individual scientist can make mistakes — scientists are human. Maybe there’s a bug in the code, maybe there’s an error in the experiment. Just because someone ran an experiment and reached a conclusion doesn’t mean it’s right, and scientists don’t treat a single result as settled truth.

Step two is peer review, the first layer of quality control. The scientist writes up what they did and submits it to a journal, which sends the paper to other experts in the field. Those experts judge whether the methods are sound, whether it properly references prior work, and whether the conclusions actually follow from the data. Ask any scientist and they’ll tell you peer review is miserable — the comments come back brutal, often harsh, and you have to revise and resubmit, sometimes multiple times, before an editor finally accepts the paper.

Peer review catches an immense amount of bad science, but it can’t catch all of it. Reviewers get sloppy or overworked, they miss obvious mistakes, and they can’t catch hidden errors — a bug in the code, a misread instrument. So a peer-reviewed paper making it into a journal doesn’t mean the paper is right. Nobody treats a single paper as the final word.

That’s where step three comes in: the crucible of science. Important results get retested by the broader community. If you publish something interesting, it gets dissected. Sometimes people try to reproduce your work directly but more often they test its implications. If someone claims X is true, and that implies that Y is also true, then scientists will go check Y. If a claim survives this gauntlet long enough, it becomes accepted. That’s how we generate knowledge.

Let me give an example: the ozone hole. In 1985 — and I doubt many people watching were around then, but I remember it well — Joe Farman and his colleagues published observations from Antarctica showing that a huge chunk of ozone was disappearing every year. This was shocking because most atmospheric chemists had expected ozone loss to occur at mid-latitudes and high altitudes, not over the South Pole.

When scientists see something they don’t understand, they rush to explain it, and several competing hypotheses quickly appeared in the peer-reviewed literature. One pointed to chlorine chemistry from chlorofluorocarbons. A second blamed nitrogen chemistry, possibly driven by solar proton events — meaning the sun was responsible. A third argued it wasn’t chemistry at all but dynamics: winds were carrying naturally low-ozone air from the lower atmosphere up into the stratosphere.

Then the community went to work testing which theory could survive the crucible. Aircraft flew into the ozone hole and found no nitrogen, ruling out the nitrogen theory. Measurements showed the air in the polar vortex was descending, not rising — which would bring high-ozone air down, eliminating the dynamics theory. Meanwhile, those same flights measured enormous amounts of chlorine, strongly correlated with exactly the regions where ozone was being destroyed. That became known as the “smoking gun” paper, and it established that chlorine chemistry was the cause.

Through this process, a consensus emerged. And it’s worth being clear that this consensus is organic. There’s no vote, no poll, no meeting where it gets decided. Scientists go to conferences, talk to one another, and independently arrived at the same answer. It’s the ultimate free market of ideas. After the consensus was established, the peer-reviewed literature doesn’t even mention the rival theories — it simply takes chlorine chemistry as given. That’s what a consensus looks like. It doesn’t mean every single scientist agrees, but it does mean there’s no longer any legitimate debate.

So where can you see what scientists actually think? It’s all in the peer-reviewed literature, which contains the entire historical record of the argument: the discovery, the competing theories, the debate, the testing, and finally the community coalescing around one answer. But there’s a lot of room for misrepresentation. You can dig up an old paper claiming the ozone hole was caused by dynamics and wave it around as if there still was a debate about the cause of the ozone hole — without mentioning that the idea was disproven long ago and that even its own authors no longer believe it.

There’s a bigger problem, though: the peer-reviewed literature is written in a language I call “nerd.” It’s extremely hard to read — honestly, some of it is hard for me to follow. It takes graduate students years to work through the literature in a single field, so we can’t expect policymakers or ordinary people to do it.

This is where scientific assessments come in. The most famous is the IPCC — the Intergovernmental Panel on Climate Change. The job of an assessment is to read and summarize the peer-reviewed literature, synthesizing what’s settled, what’s been disproven, and what remains genuinely debated.

To ensure the assessments accurately summarize the peer-reviewed literature, assessments are written by large teams, which are far less prone to bias than individuals and which dilute the influence of any one scientist with an axe to grind. The teams are deliberately assembled to avoid stacking the deck, and the reports themselves go through multiple rounds of peer review. That rigorous process is exactly why you can trust them and why people consider the IPCC to be the gold standard of what we know about the climate system. An assessment produced through a different process may well not be credible.

You can even watch the consensus evolve across assessments. Take the attribution of warming. The 1990 IPCC report said the observed warming was broadly consistent with model predictions but also comparable to natural variability — in other words, we don’t know whether humans are responsible. By 1995, it concluded that the balance of evidence suggested a discernible human influence — a sentence only a committee could write, and one that barely commits to anything. By 2001, most of the warming over the previous 50 years was judged “likely” due to greenhouse gases; by 2007, “very likely”; by 2013, “extremely likely.” The 2023 report says essentially the same as 2013 — there’s not much stronger language available.

So, to wrap up: science is our ultimate tool for deciphering the universe — the most successful method we’ve ever developed for determining what’s true and what isn’t. Our understanding is written into the peer-reviewed literature, so if you want to know what scientists think, that’s where to look. And for those of us who don’t speak nerd, scientific assessments are the Rosetta Stone — they translate nerd into English.

Let me open the floor to questions. Thank you.