… Many types of hardware can perform inference, ranging from smartphone processors to beefy GPUs to custom-designed AI accelerators. But not all of them can run models equally well. Very large models would take ages to make predictions on, say, a laptop versus a cloud server with high-end AI chips. …
… The phase of AI where a model is actively generating responses has different computational requirements than training, and a new class of chips is being designed specifically for it. …
… Google’s strategy of co-engineering its full AI stack — from custom TPU chips through to models and agents — pays dividends in flops per watt more computation per unit of energy that a company buying off-the-shelf components simply can’t replicate, he suggested. “Running Gemini on TPUs is much more… …