This is exactly the direction the conversation needs to go. We've spent a lot of time debating the energy cost of a single prompt, but agentic AI changes the unit of analysis. The better question may be: what outcome did that energy produce? Looking at prompts in isolation risks missing the much bigger systems story. Thanks for a thoughtful piece.
Thanks for this - this is a really valuable contribution which I will be sharing with others, and broadly agree with your analysis. You are right that both energy-per-token and tokens-per-task are both rapidly moving downwards for both hardware and software reasons - it is a very young technology, and there are still a lot of low hanging fruit for efficiency gains. I suspect that already your estimate will be on the high side, for this reason. Things are moving so quickly.
I would partly disagree on Jevons, though - people are quick to claim Jevons whenever there are efficiency gains alongside absolute increases in energy use, and it can be for other reasons. Jevons, strictly, is that efficiency gains result in reduced use of a resource by a service (in this case electricity and hardware), which reduces the cost of the service, allowing more use of it and more innovative ways to use it.
It is certainly true that AI is being used more, and there is extensive experimentation finding new ways to use it - but it does not appear to me that this is a result of resource cost reduction but simply the natural consequence of a brand-new already cheap-to-provide technology disseminating globally at a historically exceptional rapid pace (via the global internet). The fact that the US companies charge so much more than the Chinese companies shows that energy usage is a relatively small factor in their decision about price setting. Jevons may play some role, but at the moment the main driver of increased energy use is anticipatory investment in data centre capacity for both realistic and hype-driven reasons.
Interestingly, efficiency improvements may actually result in fewer data centres being required even while Jevons results in expanded use. This scenario will happen if most of us (including coders) find that 'good enough' AI meets our needs. Already, laptops are in the pipeline which can run ChatGPT models roughly as powerful as the frontier models of 18 months ago. Some high-value uses of 'super intelligence' (eg cybersecurity) may justify frontier model development, but for most of us what we already have is nearly good enough. In this scenario, your laptop will be powerful enough to produce those axolotl games. It wont be completely guilt free though. At least initially, such laptops will move towards gamer-spec levels of power consumption when at work - so energy use will increase at home.
Thats a fair point. My friend Jon Koomey always likes to point out how projections of exponential energy use in the early days of the internet never panned out because efficiency ultimately outpaced demand. Its just hard to know where things will land this time around!
That's true.. the speed of computation to cost has been accelerating blindingly quick. Wish they'd AI the solution to the current projected water and electrical use. Don't forget.. the more you get from chip brings more demand to do more. A 🐍 going for its tail
Thanks - fun to see a real analysis by a serious user! Since you touch on the related question toward the end: Google’s 2.7 GW data center being built N of Fort Collins CO promises to use natural gas via fuel cells, then to pump the exhaust stream (presumably pretty pure CO2) into their rock formation for permanent disposal, after using the heat to cool the project via absorption chillers. Is this do-able, and assuming it will actually happen, is there much of a reason (besides well- and delivery-methane leakage which could be drone-monitored and controlled in a sane regulatory environment) to dislike it for green reasons? I write a substack on green energy (including inherently safe small nuclear like Kairos should provide), and think Jade might have a good compromise here. -Bob Meyer
Zeke.. it's also important to remember that direct reductions in CO2 emissions will take none of the CO2 already added out of the atmosphere to potentially lower global temperatures. It does leave carbon in the ground .But more importantly those reductions will increasingly make transportation fuels less and less available as we move forward with the transition to renewables and EVs. In other words we can't even build these AI data centers much less feed eight billion people using electric transportation. What that must mean is that more CO2 will be added and little or none taken out. Because Mauna Loa CO2 correlates almost perfectly with population we can expect atmospheric CO2 to keep rising with no viable way to stop it or remove it. We must move towards innovative infrastructure to help us survive and adapt to extreme weather. And even that will also add CO2 going forward.
Great 👍 cost/query/ true cost post Zeke!I think they should build AI centers deep or under cool water ( like China in the sea). Also, have their own reservoirs to help cool. The data centers should be autonomous in water usage and electricity. Otherwise, we are f'd.
This is exactly the direction the conversation needs to go. We've spent a lot of time debating the energy cost of a single prompt, but agentic AI changes the unit of analysis. The better question may be: what outcome did that energy produce? Looking at prompts in isolation risks missing the much bigger systems story. Thanks for a thoughtful piece.
Thanks for this - this is a really valuable contribution which I will be sharing with others, and broadly agree with your analysis. You are right that both energy-per-token and tokens-per-task are both rapidly moving downwards for both hardware and software reasons - it is a very young technology, and there are still a lot of low hanging fruit for efficiency gains. I suspect that already your estimate will be on the high side, for this reason. Things are moving so quickly.
I would partly disagree on Jevons, though - people are quick to claim Jevons whenever there are efficiency gains alongside absolute increases in energy use, and it can be for other reasons. Jevons, strictly, is that efficiency gains result in reduced use of a resource by a service (in this case electricity and hardware), which reduces the cost of the service, allowing more use of it and more innovative ways to use it.
It is certainly true that AI is being used more, and there is extensive experimentation finding new ways to use it - but it does not appear to me that this is a result of resource cost reduction but simply the natural consequence of a brand-new already cheap-to-provide technology disseminating globally at a historically exceptional rapid pace (via the global internet). The fact that the US companies charge so much more than the Chinese companies shows that energy usage is a relatively small factor in their decision about price setting. Jevons may play some role, but at the moment the main driver of increased energy use is anticipatory investment in data centre capacity for both realistic and hype-driven reasons.
Interestingly, efficiency improvements may actually result in fewer data centres being required even while Jevons results in expanded use. This scenario will happen if most of us (including coders) find that 'good enough' AI meets our needs. Already, laptops are in the pipeline which can run ChatGPT models roughly as powerful as the frontier models of 18 months ago. Some high-value uses of 'super intelligence' (eg cybersecurity) may justify frontier model development, but for most of us what we already have is nearly good enough. In this scenario, your laptop will be powerful enough to produce those axolotl games. It wont be completely guilt free though. At least initially, such laptops will move towards gamer-spec levels of power consumption when at work - so energy use will increase at home.
Thats a fair point. My friend Jon Koomey always likes to point out how projections of exponential energy use in the early days of the internet never panned out because efficiency ultimately outpaced demand. Its just hard to know where things will land this time around!
That's true.. the speed of computation to cost has been accelerating blindingly quick. Wish they'd AI the solution to the current projected water and electrical use. Don't forget.. the more you get from chip brings more demand to do more. A 🐍 going for its tail
Thanks - fun to see a real analysis by a serious user! Since you touch on the related question toward the end: Google’s 2.7 GW data center being built N of Fort Collins CO promises to use natural gas via fuel cells, then to pump the exhaust stream (presumably pretty pure CO2) into their rock formation for permanent disposal, after using the heat to cool the project via absorption chillers. Is this do-able, and assuming it will actually happen, is there much of a reason (besides well- and delivery-methane leakage which could be drone-monitored and controlled in a sane regulatory environment) to dislike it for green reasons? I write a substack on green energy (including inherently safe small nuclear like Kairos should provide), and think Jade might have a good compromise here. -Bob Meyer
Zeke.. it's also important to remember that direct reductions in CO2 emissions will take none of the CO2 already added out of the atmosphere to potentially lower global temperatures. It does leave carbon in the ground .But more importantly those reductions will increasingly make transportation fuels less and less available as we move forward with the transition to renewables and EVs. In other words we can't even build these AI data centers much less feed eight billion people using electric transportation. What that must mean is that more CO2 will be added and little or none taken out. Because Mauna Loa CO2 correlates almost perfectly with population we can expect atmospheric CO2 to keep rising with no viable way to stop it or remove it. We must move towards innovative infrastructure to help us survive and adapt to extreme weather. And even that will also add CO2 going forward.
Great 👍 cost/query/ true cost post Zeke!I think they should build AI centers deep or under cool water ( like China in the sea). Also, have their own reservoirs to help cool. The data centers should be autonomous in water usage and electricity. Otherwise, we are f'd.