Nobody knows the future of energy

coal collapsed, renewables boomed, and the forecasts mostly missed it

TCB update: We just passed 20,000 subscribers! Zeke and I are grateful to each of you for reading our Substack. Your support means a lot to us and we plan to keep publishing content that we hope you’ll find interesting and thought-provoking. Thank you for being part of our community, and we look forward to sharing more with you in the future.

I’ve long been struck by how hard it is to predict the evolution of our energy system, even a few years in advance, never mind 25 or 30 years. I still remember the “peak oil” craze in the mid 2000s, when people were telling me the end of oil was nigh. It sounded convincing right up until it turned out to be wrong.

In this post, let me show you how bad previous predictions have been for the electricity sector.

evolution of our energy system in 6 charts

Each plot below shows annual predictions of how a particular source of electricity will evolve as well as what actually happened. The data come from the Energy Information Administration and cover the U.S. electricity sector.

We’ll start with coal. In the first plot, the black line shows actual U.S. coal-fired electricity generation. The colored lines are predictions made each year since 2008.

In 2008, coal was expected to produce increasing amounts of electricity into the future. Instead, it immediately started to decline and it took until 2023 before the EIA began to predict a long-term decline in coal, despite the fact that coal had been declining for 15 years.

Natural gas, by contrast, has generated an increasing share of U.S. electricity. This is largely due to the tidal wave of cheap natural gas from fracking. The predictions, on the other hand, did not anticipate this.

The takeaway here is that predicting the evolution of our energy system is not just hard in the long run, e.g., thirty years from now, but it’s hard even in the short run.

If we combine coal and gas, the forecasts look better. This reflects the fact that natural gas was replacing coal, so that the overestimate for coal was cancelled to some extent by the underestimate for natural gas.

But even for the combined category, the forecasts vary widely.

Here’s solar (including both utility and residential solar):

And here’s wind:

For both energy sources, predictions before 2015 were really bad.

Across all energy sources, the 2023 and 2025 forecasts differ sharply from the 2026 forecast. The predictions made for 2023/2025 assume Biden’s Inflation Reduction Act, while 2026 predictions assume the reversal of those policies.

The difference between 2023/2025 and 2026 is an estimate of the role that politics plays in the future evolution of our electricity sector. Because we cannot confidently predict who will win future elections or what their policies will be, this is a very good reason why it’s so hard to predict the future of our energy system.

the cost of energy

Why is it so hard to predict the energy mix in our electricity system? One big reason is that it is hard to predict the future rate of innovation. We can see this in a plot of the cost of energy¹:

You can see that the price of wind and solar plummeted in the early 2010s, reflecting enormous innovation in the production of renewable energy. That was not predicted by most mainstream forecasts (as confirmed by predictions of wind and solar above).

There has also been a lot of innovation in fossil fuel production, most importantly hydraulic fracturing and horizontal drilling. These technologies drove down the cost of natural gas in the late 2000s and changed the economics of electricity generation almost overnight. Coal plants that had looked like safe long-term investments suddenly faced a cheaper competitor. Yet this, too, was largely missed. In the late 2000s, many utilities were still trying to build coal plants, unable to see that coal was entering a precipitous decline.

TXU didn’t see the end of coal coming. Most of these plants were never built.

And, as wind and solar costs fell, renewables began taking market share too. Coal was not beaten by a single technology; it was beaten by a sequence of technologies that forecasters failed to anticipate.

Based on economics, coal is now a stone-cold loser. Its remaining advantage is not cost, speed of construction, or flexibility. It is politics. The Trump Administration is forcing coal-fired plants to stay open and some recent reporting suggests these interventions are raising costs for consumers.

In the competition between solar, wind, and natural gas, solar and wind are the cheapest. The combination of low costs, short construction times, and natural gas’ price volatility gives wind and solar a huge market advantage, explaining their exponential growth. Yes, solar and wind are coming for natural gas.

The plot also shows the profound disadvantage nuclear faces. Nuclear energy costs nearly $200/MWh, around four times the cost of wind and solar. And it takes a decade or two to get it online. Without government mandates or heavy policy support, I believe there is little likelihood that we will see a nuclear renaissance.

what are the implications of this?

Much of the debate in climate policy centers on the cost, difficulty, and timeline for phasing out fossil fuels in order to achieve net zero. You constantly hear pundits and analysts throwing around eye-popping numbers, confidently claiming, e.g., that “it will cost XXX trillions of dollars to reach net zero in our economy by 2050.”

from McKinsey

But if the forecasting failures of the last twenty years have taught us anything, it’s this: we simply have no idea how much decarbonization will cost.

You should treat numbers like McKinsey’s estimate above as guesses. They could be right, but historically speaking, they probably aren’t. To summarize, here are the reasons why the true cost of reaching net zero remains so uncertain:

• We can’t predict the foundational energy mix: As the charts above show, our ability to forecast the trajectory of the electricity sector even a few years out is abysmal. If forecasters cannot accurately predict the baseline scenario (how much wind, solar, or natural gas will be on the grid), it seems unlikely they will be able to make accurate predictions of how much additional solar and wind will be needed in 2050 to reach net zero.

• Innovation shatters financial models: Long-term cost forecasts rely heavily on estimates of how fast innovation will occur. Such predictions are incredibly hard to make. Almost no one foresaw the exponential drop in the price of solar energy since the late 2000s, nor did experts predict the current plummeting costs of battery storage. Falling battery costs could reshape the electricity system.

• Geopolitics rewrites the math: External shocks can alter energy economics overnight. Few energy forecasts anticipated wars in Ukraine and Iran, both of which are going to have an enormous impact on our energy mix going forward.

Overall, the uncertainty in these long-term forecasts is enormous. And if history is any guide, the errors are not random. They usually point in the same direction: they overestimate the cost of the energy transition.

One reason is that traditional forecasting models tend to assume slow, steady technological progress. But energy technologies do not always improve that way. Solar, wind, batteries, and fracking all show that costs can change fast when conditions line up. Most models, which assume gradual change, will miss these breaks.

Another problem is that fossil fuels are often treated as stable, low-risk alternatives. They are not. Their prices can swing wildly, and their supply chains are exposed to wars, political instability, and global market shocks. Those costs are real and hard to predict, so they are left out of these estimates.

That is the central point: estimates of the cost of the energy transition should be treated as conditional guesses built on assumptions about technology, fuel prices, politics, and geopolitics, all of which have repeatedly surprised us.

The lesson of the last twenty years is not that the energy transition will be easy or cheap. Anyone claiming to know the cost decades in advance should be treated with skepticism.

Code to reproduce the plots can be found here.

related things

Is nuclear energy the answer? Nope.

Is renewable energy cheaper than fossil fuels? Yup.

An explanation of how renewable energy saves you money. It’s not that complicated unless you’re being paid to push fossil fuels. Then it’s very complicated.

If you’re looking for a new Substack on energy, check out Bright Spots by Jan Rosenow. This recent post discusses how renewables change the price of energy.

Do you want to see how each U.S. state’s mix of electricity has changed? Brendan Pierpont has you covered here.

1. I’m using levelized cost of energy (LCOE) as my measure of the cost to produce power from each source. I understand the limitations of LCOE, but for an energy developer, LCOE is the number that counts. Yes, wind and solar are intermittent, but that’s a grid problem. All that matters to the developer is which low-LCOE energy source they can build.