Aovi Deshpande, Research Engineer, Engineering Services

The AI boom has driven massive investment in datacenter infrastructure, which relies heavily on GPUs and CPUs to process large volumes of data for training and deploying large-scale models. These systems must deliver valuable outputs quickly and reliably to justify their substantial costs. However, training these models is both expensive and energy intensive. GPT-4 for example, reportedly required more than $100 million to train and consumed approximately 50 GW of electricity [1].

Researchers from the University of Michigan [2] quantified the real-world energy cost of running AI models.

  • Llama 3.1 (8 billion parameters, text generation): Approximately 114 joules per response (57 joules of computation plus cooling and system overhead)
  • Llama 3.1 (405 billion parameters, text generation): Approximately 6,706 joules per response
  • Stable Diffusion 3 Medium (2 billion parameters, image generation): Approximately 2,282 joules per response, increasing to roughly 4,402 joules when higher-quality settings are enabled

While larger models typically produce higher-quality outputs, their energy requirements increase dramatically. GPT-4, with an estimated parameter count exceeding one trillion, likely consumes more energy and has a greater need for heat removal. Nearly all of the electrical energy consumed by GPUs during AI processing is ultimately rejected as waste heat. Because computing operations do not permanently store energy, most of the power drawn by processors is converted into thermal energy that must be actively cooled or exhausted.

Most GPUs have maximum operating temperatures between 90 – 100°C. If temperatures rise too high, the GPU may slow itself down—a process known as thermal throttling—or shut down to prevent damage. Overheating is not only a risk to GPUs. Memory manufacturers also warn that high-bandwidth memory stacks face failure risks with temperatures reach above 120°C. This highlights that inadequate cooling or thermal management can lead to performance shortfalls. Although many chips include temperature sensors and can shut down when overheated, external cooling remains essential to prevent server-rack temperatures from rising.

Modern datacenters use several cooling technologies, including air cooling for racks, evaporative cooling using external heat exchangers and cooling towers, liquid cooling in which a coolant circulates through pipe networks around or within server assemblies, and hybrid cooling approaches. Direct contact and immersion cooling may be among the most effective cooling methods.

Immersion cooling works by submerging electronic modules in a dielectric fluid, which is continuously circulated through an external heat exchanger to maintain a stable operating temperature. This method is already widely used in Bitcoin mining and relies on direct fluid-to-component contact to remove heat rapidly from chip surfaces.

At its core, immersion cooling is a controlled boiling process that harnesses both sensible and latent heat. The coolant is typically introduced in a subcooled state to delay the onset of critical heat flux. As heat transfers into the fluid, nucleate boiling begins. If the fluid is sufficiently subcooled, the resulting microbubbles condense almost immediately in the cooler bulk liquid—a regime known as subcooled boiling. This allows heat to be removed efficiently through both sensible and latent heat transfer without generating excessive vapor or causing dryout before saturation.

However, two-phase flow introduces significant operational challenges. Flashing, micro-cavitation, vapor accumulation, and flow instabilities (e.g., wave oscillations or transitions to film boiling) can degrade performance. These challenges are especially pronounced in microchannel environments, where flow behavior is often highly unsteady. Research [3] found that, unlike macro-channels, micro-scale two-phase flows are dominated by instabilities such as vapor backflow into the inlet zones.

Practical deployment adds further complexity. Dielectric coolants often have low to moderate thermal conductivity, which can limit heat dissipation rates. Maintaining consistent heat flux while preventing fluid saturation requires precise control. The presence of phase change also complicates system design, because operators must continuously manage pressure, temperature, and recirculation flow rate.

Despite these challenges, optimized immersion cooling can deliver measurable efficiency gains. For example, a study conducted by an immersion cooling startup company in collaboration with UCLA's Samueli School of Engineering found that an engineered phase-change immersion system improved computational power efficiency by approximately 15% compared with conventional liquid cooling [4]. This suggests that the technology has strong potential when thermal and fluid dynamics are carefully managed [5].

Next-generation computing may involve quantum chips, which often require cryogenic temperatures, or quantum photonic chips that can operate at ambient temperatures. In either case, precise thermal management remains for a critical requirement for fully functional, uninterrupted computing systems.

This raises an important question:

Will heat transfer become a fundamental bottleneck for chip performance and datacenter efficiency?

The implications extend beyond individual server racks. Improving thermal efficiency directly reduces the energy required for electronic cooling, helping preserve grid capacity for other essential applications. As datacenter demand continues to grow, warnings about systemic grid strain are mounting. With rising energy consumption, maximizing efficiency in computing and cooling is no longer just an engineering optimization; it is becoming a grid-scale necessity.

Click here to read more articles in this series.

References

  1. J. O'Donnell and C. Crownhart, Climate change and energy: We did the math on AI's energy footprint. Here's the story you haven't heard, MIT Tech. Rev. (20 May 2025). Online. https://www.technologyreview.com/2025/05/20/1116327/ai-energy-usage-climate-footprint-big-tech/
  2. J. Chung, J. J. Ma, R. Wu, J. Liu, O. J. Kweon, Y. Xia, Z. Wu, and M. Chowdhury, The ML.ENERGY benchmark: Toward automated inference energy measurement and optimization, in Advances in Neural Information Processing Systems 38, 121191 – 121219 (2025). https://doi.org/10.52202/085713-3652
  3. S. Lee, V. S. Devahdhanush, and I. Mudawar, Investigation of subcooled and saturated boiling heat transfer mechanisms, instabilities, and transient flow regime maps for large length-to-diameter ratio micro-channel heat sinks, Intl. J. Heat Mass Trans. 123, 172 – 191 (2018).
  4. Ferveret, Inc., 15% compute efficiency doesn't sound like much. Until you do the math. (n.d.). Online. https://www.ferveret.com/news/15-compute-efficiency-doesnt-sound-like-much-until-you-do-the-math
  5. Z. Winn, Startup's nuclear-inspired cooling system could make data centers more sustainable, MIT News (10 June 2026). Online. https://news.mit.edu/2026/nuclear-inspired-cooling-system-ferveret-could-make-data-centers-more-sustainable-0610