AI Inference Chips Are the Semiconductor Product That Every Cloud Provider Is Now Designing Themselves

Why Hyperscalers Stopped Buying Off the Shelf

NVIDIA's dominance of the AI training chip market has been one of the most commercially significant technology market positions of the past decade. The H100 and subsequent H200 GPU architectures that NVIDIA has produced for AI model training represent the computational infrastructure on which virtually all major large language model development has been conducted, and NVIDIA's revenue and market capitalisation growth since 2022 reflects the commercial consequence of holding the essential hardware position in the most rapidly expanding segment of enterprise technology investment. The inference market, in which trained AI models are deployed to respond to user requests at scale rather than being trained in the first place, is the segment where NVIDIA's dominance is being most actively challenged by custom silicon that the hyperscale cloud providers are designing in-house. The commercial logic of this development is straightforward. Inference workloads differ from training workloads in their computational requirements, their latency constraints, and their cost sensitivity in ways that create space for purpose-built inference accelerators to outperform general-purpose training chips on the specific task of running trained models at scale. And at the volume of inference queries that Google, Amazon, Microsoft, and Meta are processing daily, the economics of custom silicon development are compelling.

The economics of custom AI inference chip development rest on the per-query cost reduction that purpose-built silicon achieves relative to general-purpose GPU inference. A chip designed specifically to run transformer model inference can eliminate the computational units that training requires but inference does not, optimise the memory architecture for the inference access patterns that differ from training, and implement the specific numerical precision and quantisation schemes that inference quality standards permit at reduced computational cost relative to training precision. The cumulative effect of these optimisations across billions of inference queries daily creates cost savings that justify the billions of dollars of chip design and manufacturing investment that custom silicon requires. Google's internal estimates of TPU cost savings relative to equivalent GPU inference have been cited as sufficient to recover the development cost of the TPU programme across the utilisation of a small number of data centres, providing the financial case that the hyperscale companies use to justify their semiconductor design programmes.

Google TPU and the First Mover Position

Google has the longest operational history of any hyperscaler in custom AI chip design through its Tensor Processing Unit programme, whose first version was deployed in Google's data centres in 2015 primarily for inference workloads. The progression from TPUv1 through TPUv4 and the most recent TPUv5 represents over a decade of sustained semiconductor design investment whose accumulated architecture learning has created a performance and efficiency advantage in Google-specific AI workloads that is difficult to replicate without equivalent depth of hardware-software co-optimisation. Google's TPUs serve both its internal AI applications, including Google Search's AI ranking, Google Translate, and the Gemini family of AI models, and its cloud customers through the TPU products available on Google Cloud Platform. The commercial availability of TPUs to external cloud customers creates a revenue source that partially offsets Google's semiconductor design investment, while the internal efficiency gains from deploying TPUs for its own AI inference constitute the primary financial justification for the programme.

AWS Trainium and Inferentia chips, developed by Amazon Web Services through its Annapurna Labs subsidiary, represent the hyperscaler silicon investment that has moved most aggressively from internal deployment to customer-facing cloud product. AWS's Inferentia chip, designed specifically for neural network inference, is available to AWS customers through Inf1 and Inf2 EC2 instances whose pricing is positioned as a cost-competitive alternative to GPU inference for AI applications whose model architecture and inference patterns are compatible with Inferentia's design. AWS's semiconductor programme reflects the strategic insight that cloud infrastructure cost is a primary competitive dimension in the market for AI developer tools, and that proprietary inference silicon whose cost efficiency advantage is passed to customers in the form of lower inference pricing creates the customer acquisition and retention benefit that justifies the chip development investment from a cloud business perspective beyond the internal cost saving alone.

Microsoft Maia and the Competitive Landscape

Microsoft's Maia AI accelerator chip, announced in 2023 and beginning deployment in Microsoft Azure data centres, represents the most recent major hyperscaler entry into custom AI silicon and the one whose commercial significance is amplified by Microsoft's deep investment in OpenAI's models that are among the highest-volume AI inference workloads in the cloud market. Maia's architecture is optimised for the specific computational patterns of large transformer models, reflecting Microsoft's advantage in having access to OpenAI's model architectures as a co-design input to the chip's design specifications. Meta's MTIA inference chip programme and ByteDance's custom inference chip investments demonstrate that the custom AI silicon trend extends beyond the three traditional hyperscalers to any large technology company running AI inference at a scale where the economics of custom silicon justify the development investment.

Top 10 Companies in AI Inference Chips Globally

1. NVIDIA: Dominant AI chip company whose H100 and H200 training chips have created the infrastructure standard for AI model development; its L40S and upcoming Blackwell inference-optimised architectures reflect its effort to defend its position in the inference market where custom hyperscaler silicon is most directly challenging its dominance in the GPU computing market.
2. Google (TPU): Longest-operating hyperscaler custom AI chip programme with TPUv5 deployed for Gemini inference at scale; its decade of TPU hardware-software co-design creates the inference efficiency advantage that Google's internal AI applications benefit from and that its cloud customers can access through Google Cloud's TPU product line.
3. AWS (Inferentia/Trainium): Custom AI chip programme with Inferentia2 for inference and Trainium2 for training available to external cloud customers; its pricing strategy that positions custom silicon inference below GPU inference cost on AWS creates the commercial incentive for AI developers to optimise their models for AWS custom hardware.
4. Microsoft (Maia): Azure AI accelerator chip optimised for large transformer model inference; its co-design with OpenAI's model architectures creates the hardware-model optimisation that general-purpose GPU inference cannot match for the specific computational patterns of GPT and related model families.
5. Groq: AI inference chip startup whose Language Processing Unit architecture achieves industry-leading inference throughput for transformer models through its deterministic execution approach; its inference-as-a-service commercial model provides access to its LPU hardware without requiring customers to build or manage custom silicon infrastructure.
6. Cerebras Systems: Wafer-scale engine developer whose entire silicon wafer forms a single processor for AI computation; its architecture eliminates the inter-chip communication bottleneck that limits inference throughput in multi-chip GPU clusters and creates the performance advantage for very large model inference that its wafer-scale integration provides.
7. Graphcore (Softbank): UK AI processor company whose Intelligence Processing Unit architecture is designed for AI compute patterns; its acquisition by SoftBank provides the capital and distribution access that its technology development required to compete with the scale advantages that hyperscaler custom silicon programmes bring to the inference market.
8. SambaNova Systems: AI inference platform company whose reconfigurable dataflow architecture provides the flexibility to optimise inference execution for different model types without fixed hardware specialisation; its enterprise AI inference systems target the enterprise customers who are running inference at scales below hyperscaler volumes but above the point where GPU inference cost is acceptable.
9. Etched: AI inference startup designing chips hardcoded for transformer model inference at the architecture level; its hardware specialisation beyond GPU flexibility to pure transformer execution creates the maximum possible inference efficiency for the model architecture that dominates current AI applications at the cost of inflexibility to non-transformer model types.
10. Untether AI: Canadian AI inference chip startup whose at-memory compute architecture moves computation to where data is stored rather than moving data to computation units; its memory-centric architecture addresses the memory bandwidth bottleneck that limits inference throughput in conventional processor architectures and that the transformer model's attention mechanism makes particularly acute.