Predicting the future AI infra stack - AI-Infra Overview PART-3

This is the third part of the series. Part 1 defined AI infrastructure as the implementation layer that maps models onto hardware under a service objective. Part 2 showed how model work, context, output, and agent loops multiply demand. Part 4 treats edge intelligence as its own hardware, economics, and model-quality problem.

This part looks forward from the supply side.

Key insightAI infrastructure will become more heterogeneous, and more hardware will be designed around LLM-specific bottlenecks rather than generic FLOPs.

General-purpose GPUs will remain central, but the winning system increasingly looks like a coordinated package: CPUs for orchestration, GPUs for dense math, tensor engines for narrow numerical formats, NPUs for local inference, HBM and SRAM for locality, scale-up fabrics for model parallelism, scale-out networks for cluster scheduling, and software that can place work across all of them.

The evidence falls into three constraints. Compute progress increasingly depends on narrower numerical contracts. Memory determines how much model and context can stay close to arithmetic. Communication determines whether many accelerators behave like one system.

1. Compute becomes specialized

Key insightPeak dense compute keeps rising and dollars per peak compute keep falling, but the gains increasingly come from narrower numerical contracts and workload-specific engines rather than transistor shrink alone.

Peak math is conditional

The H100 is a useful example. NVIDIA’s Hopper material highlights HBM3 bandwidth around 3 TB/s, a 50 MB L2 cache, Transformer Engine support, NVLink/NVSwitch scale-out, and low-precision tensor paths.¹ These are not just “more FLOPs.” They are area, power, and system-design choices that help specific workload shapes.

The GPU timeline shows the same pattern:

Representative figures make the jump visible. Tesla C870 was advertised at 518 GFLOP/s peak single precision in 2007.² C1060 reached 933 GFLOP/s in 2008.³ K20X reached 3.95 TFLOP/s single precision and 1.31 TFLOP/s double precision in 2012.⁴ P100 delivered 21.2 TFLOP/s FP16 in 2016, V100 delivered about 125-130 Tensor TFLOP/s, A100 reached 312 TFLOP/s dense FP16/BF16 Tensor Core performance, and H100 lists 1,979 TFLOP/s FP16/BF16 Tensor Core with sparsity, or half that without sparsity.⁵⁶⁷⁸

Blackwell continues the shift. NVIDIA’s DGX B200 system lists 144 PFLOP/s FP4 Tensor Core performance across eight Blackwell GPUs, roughly 18 PFLOP/s per GPU at the published system level, while Blackwell Ultra emphasizes 15 PFLOP/s dense NVFP4 per GPU.⁹¹⁰ NVIDIA’s preliminary Rubin specifications list 4 PFLOP/s dense FP16/BF16 and 50 PFLOP/s NVFP4 inference compute per GPU. Vera Rubin NVL144 CPX is framed around 8 exaFLOP/s of rack-scale AI performance for massive-context inference.¹¹¹²¹³

These figures are not an apples-to-apples speedup curve. The datatype, sparsity mode, memory system, and programming model all changed. That is the important part. GPU progress came from changing the numerical contract: CUDA, SIMT execution, HBM, NVLink, tensor cores, TF32, BF16, FP8, FP4, sparsity, and compiler/runtime support made model structure visible to hardware.

What the three curves measure

These are infrastructure proxies, not chip MSRPs or workload benchmarks. Dividing a DGX price by eight allocates CPUs, memory, storage, networking, and chassis cost to each GPU; the H100 and B200 values are launch-window estimates rather than NVIDIA-published standalone prices.¹⁷ Rental prices also include the provider’s host system, operations, capacity, and margin. That is why the current rental curve can move differently from launch purchase economics: in Lambda’s current catalog, A100 costs more per theoretical unit of dense compute than V100, then H100 and B200 resume the decline. Rubin appears only in peak compute because comparable purchase and rental prices are not yet public.

Epoch AI’s broader historical work reaches the same qualitative conclusion: GPU FLOP/s per dollar doubled roughly every 2.5 years across 2006-2021, and its newer AI hardware trend page estimates AI chip performance per dollar improving by about 37% per year across 2012-2025.¹⁹²⁰ Our World in Data republishes the same broad compute-per-dollar series as an interactive chart, adjusted for inflation.²¹

A rack-to-die BOM proxy shows where the dollars moved

Key insightRead accelerator economics from the outside in. The rack first pays for GPUs, memory, communication, CPUs, power, and cooling; only then does it make sense to ask what sits inside one accelerator module and one logic die.

A 2026 Morgan Stanley estimate puts VR200 NVL72 at USD 7.80 million: USD 3.96 million for the GPU line, USD 2.00 million for the separate Memory line, USD 720,000 for NVLink Switch and other networking chips, USD 180,000 for Vera CPUs, and the remainder for cooling, power, boards, substrates, passives, assembly, and other platform content.²² The memory discussion explicitly includes Rubin HBM4 alongside Vera LPDDR5X/SoCAMM and NAND storage, so the GPU line must not be read as a complete HBM-inclusive physical package. The resulting top-level ratio is GPU line excluding separately accounted memory 50.7%, memory 25.7%, communication 9.2%, CPUs 2.3%, and platform/power/cooling 12.1%. NVIDIA’s public topology confirms why communication has its own bill: an NVL72 domain contains 18 compute trays, nine NVLink Switch trays, and 72 GPUs.²³

Accelerator packages: HBM and packaging dominate

Key insightInside an accelerator module, the logic die is not the majority cost. HBM and advanced packaging together account for roughly three quarters of both the H20 and B200 manufacturing proxies.

The H20 proxy divides approximately USD 2.59 billion of Q3-Q4 2024 component spend by roughly one million 2024 shipments, implying about USD 2,600 per module, with a broad USD 2,100-3,100 sensitivity range.²⁴ Its top-level split is 57.9% HBM3, 16.6% CoWoS-S, 15.1% logic, and 10.4% auxiliary components. This fits the product’s role: H20 retains 96 GB of HBM3 and about 4 TB/s of bandwidth while its exported compute configuration is far below H100.²⁵ Its roughly USD 12,000-15,000 2024 sale price additionally reflected margin, software value, channel costs, and market conditions.²⁶

Epoch AI estimates a B200 module at roughly USD 5,700-7,300, centered near USD 6,400. Its central inputs are USD 2,900 for 192 GB of physically packaged HBM3E, USD 1,100 for CoWoS-L, USD 900 for two logic dies, USD 1,000 for packaging yield loss, and USD 480 for power delivery, PCB, assembly, and testing.²⁷ The top-level split is therefore 45.5% HBM3E, 32.9% packaging and yield, 14.1% logic, and 7.5% auxiliary components. Shipping B200 specifications expose 180 GB as usable memory; the model prices the physical 192 GB capacity.

Logic dies: a floorplan explains the logic line

Key insightA logic die is not an ALU slab. Compute regions share silicon with local SRAM and control, while cache, memory controllers, die-to-die links, NVLink, PCIe, and fabric consume substantial visible area.

For GH100, the model assigns approximately 59.9% to complete SM regions, 8.9% to standalone L2, 11.8% to memory and link I/O, and 19.5% to other fabric and uncore.²⁸ Applying those shares to H20’s USD 393 logic line attributes about USD 235, USD 35, USD 46, and USD 77 respectively. Disabled or restricted execution units still occupy physical die area and incur wafer cost.

For Blackwell, tracing NVIDIA’s annotated dual-reticle image gives approximately 39.8% GPC/SM regions, 14.3% standalone L2, 11.2% HBM controllers, 20.9% NV-HBI/NVLink/PCIe regions, and 13.8% other uncore.²⁹ Applied only to B200’s USD 900 logic line, those areas attribute about USD 358, USD 129, USD 101, USD 188, and USD 124. B200 also exposes about 126 MB of shared L2 plus large per-SM register, tensor-memory, and local-memory structures.³⁰

Rubin’s official annotated image gives approximately 29.3% GPC/SM regions, 8.9% standalone L2, 14.2% HBM controllers, 27.3% NV-HBI/NVLink/PCIe regions, and 20.3% other uncore.³¹ No disclosed bare-die cost exists. Mechanically applying area to the analyst’s USD 55K GPU line would produce USD 16.1K, USD 4.9K, USD 7.8K, USD 15.0K, and USD 11.2K, but those are only attribution values. The analyst accounts for HBM4 elsewhere, while the GPU line may still include logic packaging, test, margin, and commercial value.

Die logic cost: arithmetic density versus wafer price

Key insightFP16 arithmetic occupies far less silicon than it did at 45nm, but rising leading-edge wafer prices make dollars per ALU fall more slowly than area per ALU.

The physical hierarchy starts inside the logic die. A normalized FP16 multiply-plus-add datapath shrinks from about 3,000 um2 at 45nm to roughly 206 um2 at 7nm and 87 um2 at 3nm. Lower precision and matrix engines can push useful compute density further, but only when software and the memory system keep those lanes fed.³²³³

2. Memory cost: on-die SRAM to package HBM

Key insightMoving outward from the die buys capacity, but costs latency and energy; HBM buys back bandwidth at a high package-level price.

On-die SRAM keeps reused data closest to the arithmetic, but each additional MB consumes expensive logic area. HBM holds far more model state and KV cache, yet it adds stacked memory, interposers, controllers, and package complexity. AI infrastructure bridges the two with tiling, fusion, quantization, paging, and cache placement.

3. Communication cost: package to rack to cluster

Key insightEvery step away from the die expands the compute domain, but adds more latency, contention, topology, and equipment cost.

The software-visible communication cost can be approximated with one latency-bandwidth model:

Here alpha is latency per message and B_effective is achieved bandwidth after topology, protocol, contention, and collective overhead. Tensor parallelism, MoE routing, and disaggregated serving increase either the message count, the bytes moved, or both. NVIDIA’s scale-up bandwidth per GPU rises from 900 GB/s on Hopper to 1.8 TB/s on Blackwell and 3.6 TB/s on Rubin; NVL72 aggregate bandwidth rises from 130 TB/s to 260 TB/s between Blackwell and Rubin.⁵¹⁵²

Cluster fabric cost: dollars per bandwidth

Key insightSwitch bandwidth became much cheaper, but AI systems spend the gain on larger communication domains and more distributed inference.

InfiniBand progressed from QDR through EDR, HDR, NDR, and XDR; Ethernet moved from 40/100GbE to 200/400GbE and then 800GbE.^53545556 Faster links do not automatically lower the cluster network bill: larger collective domains, expert routing, and KV-cache movement consume the added bandwidth.

Across all three levels, infrastructure creates value by moving fewer bytes, keeping reusable state near compute, overlapping communication with kernels, and choosing a parallelism plan that converts costly bandwidth into useful tokens.

Edge becomes a separate placement frontier

Key insightEdge systems must optimize useful local intelligence within a product-level memory, bandwidth, thermal, power, and price envelope.

Edge hardware changes the accounting unit. A phone, AI PC, or robotics box is bounded by shared memory, bandwidth, thermals, battery life, and a product-level price rather than by rack power and HBM alone. Peak TOPS is useful, but it does not tell us how large a model fits, how quickly its weights can be streamed during decoding, or how much benchmark quality survives quantization.

Part 4 follows this edge intelligence envelope directly: how mobile, AI PC, and AI-box hardware evolved; how their product-level dollars per advertised performance changed; how edge-fit language models improved; and what those trends imply for local agents.

Prediction: heterogeneity becomes the default

Key insightThe future AI system is a coordinated heterogeneous package whose software places each workload on the right compute, memory, and communication resource.

The accounting unit connects the evidence:

token cost ~= math + memory movement + communication + scheduling + retries

The next AI infra frontier is not simply a larger cluster. It is a tighter control loop between algorithms, model architecture, serving systems, kernels, compilers, memory hierarchy, interconnect, edge devices, and chips.

The practical direction is heterogeneous and increasingly LLM-specific:

• More dedicated tensor formats and matrix engines, because narrow numerical contracts buy more useful math per watt and per dollar.
• More memory-aware architectures, because context, KV cache, retrieval, and multimodal state make bytes as important as FLOPs.
• More rack-scale and cluster-scale co-design, because scale-up and scale-out communication now shape model design.
• More edge/cloud partitioning, because not every inference should cross the network and not every local device can host the whole model.
• More compiler/runtime responsibility, because specialized hardware only matters when the software stack can expose locality, regularity, and parallelism.

The teams that win will not only have better models or better hardware. They will have better translation between the two: less work per useful result, better placement for every stage, and higher utilization of every expensive byte and arithmetic lane.

References