Predicting the future AI infra stack - AI-Infra Overview PART-3

This is the third part of the series. Part 1 defined AI infrastructure as the implementation layer that maps models onto hardware under a service objective. Part 2 showed how model work, context, output, and agent loops multiply demand. Part 4 treats edge intelligence as its own hardware, economics, and model-quality problem.

This part looks forward from the supply side. The prediction is simple: AI infrastructure will become more heterogeneous, and more hardware will be designed around LLM-specific bottlenecks rather than generic FLOPs.

General-purpose GPUs will remain central, but the winning system increasingly looks like a coordinated package: CPUs for orchestration, GPUs for dense math, tensor engines for narrow numerical formats, NPUs for local inference, HBM and SRAM for locality, scale-up fabrics for model parallelism, scale-out networks for cluster scheduling, and software that can place work across all of them.

The evidence falls into three constraints. Compute progress increasingly depends on narrower numerical contracts. Memory determines how much model and context can stay close to arithmetic. Communication determines whether many accelerators behave like one system.

1. Compute becomes specialized

Key insightFuture compute gains increasingly come from narrower numerical contracts and workload-specific engines, not transistor shrink alone.

Peak math is conditional

The H100 is a useful example. NVIDIA’s Hopper material highlights HBM3 bandwidth around 3 TB/s, a 50 MB L2 cache, Transformer Engine support, NVLink/NVSwitch scale-out, and low-precision tensor paths.¹ These are not just “more FLOPs.” They are area, power, and system-design choices that help specific workload shapes.

The GPU timeline shows the same pattern:

Representative figures make the jump visible. Tesla C870 was advertised at 518 GFLOP/s peak single precision in 2007.² C1060 reached 933 GFLOP/s in 2008.³ K20X reached 3.95 TFLOP/s single precision and 1.31 TFLOP/s double precision in 2012.⁴ P100 delivered 21.2 TFLOP/s FP16 in 2016, V100 delivered about 125-130 Tensor TFLOP/s, A100 reached 312 TFLOP/s dense FP16/BF16 Tensor Core performance, and H100 lists 1,979 TFLOP/s FP16/BF16 Tensor Core with sparsity, or half that without sparsity.⁵⁶⁷⁸

Blackwell continues the shift. NVIDIA’s DGX B200 system lists 144 PFLOP/s FP4 Tensor Core performance across eight Blackwell GPUs, roughly 18 PFLOP/s per GPU at the published system level, while Blackwell Ultra emphasizes 15 PFLOP/s dense NVFP4 per GPU.⁹¹⁰ NVIDIA’s preliminary Rubin specifications list 4 PFLOP/s dense FP16/BF16 and 50 PFLOP/s NVFP4 inference compute per GPU. Vera Rubin NVL144 CPX is framed around 8 exaFLOP/s of rack-scale AI performance for massive-context inference.¹¹¹²¹³

These figures are not an apples-to-apples speedup curve. The datatype, sparsity mode, memory system, and programming model all changed. That is the important part. GPU progress came from changing the numerical contract: CUDA, SIMT execution, HBM, NVLink, tensor cores, TF32, BF16, FP8, FP4, sparsity, and compiler/runtime support made model structure visible to hardware.

Three views of GPU economics

These are infrastructure proxies, not chip MSRPs or workload benchmarks. Dividing a DGX price by eight allocates CPUs, memory, storage, networking, and chassis cost to each GPU; the H100 and B200 values are launch-window estimates rather than NVIDIA-published standalone prices.¹⁷ Rental prices also include the provider’s host system, operations, capacity, and margin. That is why the current rental curve can move differently from launch purchase economics: in Lambda’s current catalog, A100 costs more per theoretical unit of dense compute than V100, then H100 and B200 resume the decline. Rubin appears only in peak compute because comparable purchase and rental prices are not yet public.

Epoch AI’s broader historical work reaches the same qualitative conclusion: GPU FLOP/s per dollar doubled roughly every 2.5 years across 2006-2021, and its newer AI hardware trend page estimates AI chip performance per dollar improving by about 37% per year across 2012-2025.¹⁹²⁰ Our World in Data republishes the same broad compute-per-dollar series as an interactive chart, adjusted for inflation.²¹

A BOM proxy shows where the dollars moved

Key insightArithmetic is only a small attributed share of accelerator manufacturing cost. H20 makes the memory tax especially visible: its 96 GB HBM subsystem likely represents well over half of module cost.

The H20 is the most directly relevant case for inference fleets using it today. Epoch AI’s component dataset assigns roughly USD 2.59 billion of median logic, CoWoS, HBM, and auxiliary spend to H20 production in Q3-Q4 2024. Reuters reported that NVIDIA shipped approximately one million H20s in 2024, implying a rough USD 2,600 manufacturing cost per module, with a deliberately broad USD 2,100-3,100 sensitivity range.²² On this basis, HBM contributes about 57.9%, CoWoS packaging 16.6%, all logic 15.1%, and module auxiliary components 10.4%.

The important operational point is that H20 is memory-rich but compute-restricted: it carries 96 GB of HBM3 and about 4 TB/s of memory bandwidth, while its exported compute configuration is far below H100.²³ Applying the same illustrative 60/15/25 logic-area allocation attributes about 9.1% of module cost to SM/ALU structures and 2.3% to on-chip SRAM. Those are manufacturing allocations, not utilization: disabled or restricted compute structures still consume die area and wafer cost. The module sold for roughly USD 12,000-15,000 in 2024, so procurement price also includes NVIDIA margin, channel margin, software value, and market conditions rather than mapping directly to BOM.²⁴

Epoch AI estimates a B200 module at roughly USD 5,700-7,300, centered near USD 6,400. Its central component estimates are USD 2,900 for 192 GB of physically packaged HBM3E, USD 1,100 for CoWoS-L packaging, USD 900 for two logic dies, USD 1,000 for packaging yield loss, and USD 480 for module power delivery, PCB, assembly, and testing.²⁵ Shipping B200 specifications expose 180 GB as usable GPU memory; the BOM model prices the physical HBM capacity. The component midpoints sum to USD 6,380, close to the model’s rounded headline.

The die itself is not sold as separate ALU and SRAM line items. The left chart therefore applies a deliberately round area allocation to the USD 900 logic-die cost. Under the 60/15/25 midpoint, arithmetic and SM structures contribute about 8.5% of module manufacturing cost, on-chip SRAM about 2.1%, and other die logic about 3.5%. A broad 50-70% compute and 10-20% SRAM sensitivity moves those first two shares only to roughly 7-10% and 1.4-2.8%. The exact floorplan is undisclosed; the architectural anchor is that B200 exposes about 126 MB of shared L2 plus large per-SM register and local-memory structures.²⁶

At rack scale, a 2026 Morgan Stanley estimate puts VR200 NVL72 at USD 7.80 million: USD 3.96 million for 72 Rubin GPUs, USD 2.00 million for memory, USD 720,000 for NVLink Switch and other networking chips, USD 180,000 for Vera CPUs, and the remainder for cooling, power, boards, substrates, passives, assembly, and other platform content.²⁷ That gives the quick rack-level ratio: GPU packages 50.7%, memory 25.7%, communication 9.2%, CPUs 2.3%, and platform/power/cooling 12.1%. NVIDIA’s public topology confirms the physical reason communication has its own bill: an NVL72 domain contains 18 compute trays, nine NVLink Switch trays, and 72 GPUs.²⁸

ALU manufacturing: narrow math buys more lanes

The ALU-level version of the story is simpler. Arithmetic got cheaper because accelerators stopped treating every operation as a wide general-purpose floating-point operation.

The area trend is the important signal: a 16-bit floating-point multiply-plus-add datapath that is about 3,000 um2 at 45nm becomes an order of magnitude smaller by 7nm, roughly 35x smaller by 3nm, and about 50x smaller at an N2-style next node under a pure logic-density scaling model. SMIC’s N+1 to N+3 branch shows how DUV multi-patterning and design-technology co-optimization can keep improving a 7nm-class platform, but SemiAnalysis finds that N+3 pays for its TSMC N6-class density in process complexity, efficiency, and control.⁴⁰

Huawei LogicFolding is a different scaling axis. Huawei describes it as shortening critical-path wiring, increasing density, and reaching its first commercial Kirin implementation in fall 2026; SemiAnalysis characterizes the approach as stacking active logic and recovering density through advanced packaging.⁴¹⁴⁰ It should not be converted into a smaller planar ALU area: package-footprint density can rise by stacking active layers even when each underlying die remains on a less-dense process.

The raw wafer-cost proxy falls less smoothly because advanced wafer prices rise sharply. N2 can fit more arithmetic, but a USD 30,000 wafer can erase much of the dollar-per-ALU gain versus 3nm. Real ALUs also need registers, operand routing, control, clocking, SRAM, verification margin, and yield. Tensor cores improve the economics further by amortizing control and data movement across matrix tiles instead of treating every multiply-add as an isolated scalar unit.

2. Memory sets the working set

Key insightModel and context capacity are constrained by where bytes live, how quickly they move, and how much each level of the memory hierarchy costs.

Memory and manufacturing show the same pattern. Compute can keep rising, but every token also needs bytes close to the math unit. The difficult part is that each level of memory optimizes a different constraint: on-chip SRAM is fast but area-expensive, HBM is bandwidth-rich but package-expensive, commodity DRAM is capacity-rich but far away, and advanced wafers are no longer getting cheap fast enough to hide the tradeoff.

3. Communication becomes topology

Key insightOnce a model outgrows one accelerator, interconnect topology becomes part of the computer and part of model performance.

The next bottleneck appears when one accelerator is not enough. Scaling out turns compute into a distributed system problem: GPUs must exchange gradients, activations, KV cache state, expert routes, pipeline bubbles, and scheduling metadata. A useful first-order model is the latency-bandwidth model:

Here alpha is the per-message latency cost and B_effective is achieved communication bandwidth after topology, protocol, contention, and collective implementation overhead. That term matters because modern training and inference are full of collectives:

There are two different network curves hiding behind the same word “interconnect”:

• Scale-up is the tightly coupled GPU domain inside a box or rack. NVLink / NVSwitch bandwidth is not sold like a generic switch port; it is bundled into GPU systems, board design, power delivery, and thermal design. The useful public metric is bandwidth per GPU or per rack-scale domain.
• Scale-out is the cluster fabric across nodes and racks. Ethernet and InfiniBand have visible port speeds, switch radix, optics, cables, NICs, and sometimes observable street prices. This is where a rough dollar-per-Gb/s proxy is possible.

The speed curve is steep. InfiniBand moved from 4x QDR at 32 Gb/s in the late 2000s, to EDR 100 Gb/s, HDR 200 Gb/s, NDR 400 Gb/s, and XDR 800 Gb/s. Ethernet followed the same broad shape: 40/100GbE was standardized in 2010, 200/400GbE in 2017, and 800GbE in 2024.^50515253 NVIDIA’s current Quantum-X800 documentation lists 72-port and 144-port XDR systems at 800 Gb/s per port, up to 115.2 Tb/s of maximum throughput for the 4U system.⁵⁴

The punchline is subtle: network silicon has delivered a large cost-per-bit improvement, but AI clusters keep spending the savings. Higher port speed enables larger all-reduce domains, more tensor-parallel shards, more MoE all-to-all traffic, and more disaggregated serving. The value of better AI infrastructure is therefore not just “buy faster switches.” It is reducing bytes moved, placing bytes closer to compute, overlapping collectives with kernels, and choosing parallelism plans that turn expensive network bandwidth into useful tokens.

This is why “chips are slowing down” is not only a FLOP story. It is a locality and communication story. When model weights, activations, KV cache, and tool-use context grow, the system pays for bytes in several currencies: SRAM area, HBM dollars, HBM bandwidth, interconnect bandwidth, synchronization time, package complexity, wafer cost, and energy. Good AI infrastructure wins by spending fewer bytes, reusing them closer to compute, and making expensive memory and network bandwidth do useful work more often.

Edge becomes a separate placement frontier

Key insightEdge systems must optimize useful local intelligence within a product-level memory, bandwidth, thermal, power, and price envelope.

Edge hardware changes the accounting unit. A phone, AI PC, or robotics box is bounded by shared memory, bandwidth, thermals, battery life, and a product-level price rather than by rack power and HBM alone. Peak TOPS is useful, but it does not tell us how large a model fits, how quickly its weights can be streamed during decoding, or how much benchmark quality survives quantization.

Part 4 follows this edge intelligence envelope directly: how mobile, AI PC, and AI-box hardware evolved; how their product-level dollars per advertised performance changed; how edge-fit language models improved; and what those trends imply for local agents.

Prediction: heterogeneity becomes the default

Key insightThe future AI system is a coordinated heterogeneous package whose software places each workload on the right compute, memory, and communication resource.

The accounting unit connects the evidence:

token cost ~= math + memory movement + communication + scheduling + retries

The next AI infra frontier is not simply a larger cluster. It is a tighter control loop between algorithms, model architecture, serving systems, kernels, compilers, memory hierarchy, interconnect, edge devices, and chips.

The practical direction is heterogeneous and increasingly LLM-specific:

• More dedicated tensor formats and matrix engines, because narrow numerical contracts buy more useful math per watt and per dollar.
• More memory-aware architectures, because context, KV cache, retrieval, and multimodal state make bytes as important as FLOPs.
• More rack-scale and cluster-scale co-design, because scale-up and scale-out communication now shape model design.
• More edge/cloud partitioning, because not every inference should cross the network and not every local device can host the whole model.
• More compiler/runtime responsibility, because specialized hardware only matters when the software stack can expose locality, regularity, and parallelism.

The teams that win will not only have better models or better hardware. They will have better translation between the two: less work per useful result, better placement for every stage, and higher utilization of every expensive byte and arithmetic lane.

References