Google's AI Infrastructure Chief, Amin Vahdat, on the Physics & Economics of Frontier AI

Google’s planned CapEx of more than $200 billion this year is largely aimed at data centers, which Vahdat says must be designed around AI hardware that changes faster and draws far more power than conventional facilities. He says racks draw hundreds of kilowatts today and could reach multiple megawatts by 2036, while hardware lifetimes may be about six years; Google’s seven- and eight-year-old TPUs still see significant use, but newer generations’ power efficiency can justify replacing systems as pods.

Vahdat argues that theoretical FLOPS do not capture delivered performance: at 100,000 accelerators, failures could occur multiple times a day or even an hour, disrupting synchronous workloads, while software issues can also reduce system output. Google therefore targets doubling serving token-generation capability every six months through a mix of hardware, model, and runtime improvements, with Vahdat emphasizing delivered goodput and intelligence per watt rather than chip metrics alone.

Vahdat says specialized chips make economic sense only when a workload is durable and demand is large enough to cover fixed costs, because specialization improves speed and power efficiency at the expense of flexibility. Google developed 8i for inference and 8T for training after estimating inference and serving could make up 30%–60% of demand over the chips’ lifetimes; he contrasts that with a workload estimated at 2%–5%, where specialization might not pay off even if a chip were twice as fast, and says GPUs and TPUs suit different workloads.

Vahdat says Google’s collaboration with DeepMind lets model requirements shape chip design, including possible architectural changes during development, while production work focuses on delivered intelligence or goodput per watt. That coordination operates within a five- or six-stage pipeline planned two to five years ahead, where teams generally avoid small 0.5%–1% changes near tapeout; the TPU’s core medium-level architecture has remained broadly stable since v1 even as its instruction set and primitives have been extended.

Vahdat expects long-horizon agents to generate model requests milliseconds apart, increasing the need for CPUs to reason, parse, retrieve context, and orchestrate prompts alongside accelerated compute, networking, and storage. Placing CPU racks beside GPU and TPU racks can complicate specialization, while separating them into different buildings may add hundreds of microseconds of network latency, potentially more with queuing.

Vahdat calls power the most fundamental data-center constraint and says Google plans gigawatt-scale supply with utilities years ahead; a one-gigawatt target at 99.99% or greater reliability could require building two gigawatts. Site size also depends on workload and power availability: Google uses facilities from tens of megawatts to nearly a gigawatt, with larger sites benefiting training through shorter network distances but raising single-point-of-failure concerns.