From conventional infra to agentic infra - AI-Infra Overview PART-1

This is the first part of a four-part series organized around one claim: AI infrastructure is the implementation bridge between upstream workload design and underlying hardware reality.

The series:

1. History: how model execution and hardware specialization co-evolved, and how infrastructure kept them connected.
2. Problem: why model size, context, output length, and agent request volume scale faster than hardware economics.
3. Future: why the next phase points toward heterogeneous computing and more dedicated LLM hardware.
4. Edge: how local hardware economics and small-model progress determine the intelligence that can stay on-device.

Infra is the bridge

Key insightAI infrastructure implements a model on hardware under a service objective.

AI infrastructure is neither the model nor the silicon. It is the implementation layer that makes a model execute efficiently and reliably on a physical machine.

This diagram gives a compact review framework for any AI-infrastructure technique:

1. Model contract: which operators, tensor shapes, numerical formats, parameters, activations, context, or agent state must execute?
2. Hardware budget: which compute engines, memory levels, links, storage, power, and placement constraints are available?
3. Infrastructure mapping: how should the system compile, partition, place, move, cache, batch, schedule, isolate, and recover that work?
4. Service objective: did the mapping improve useful quality, latency, throughput, cost, energy, or reliability?

Model and hardware design still feed back into each other. A memory limit can motivate quantization, sparsity, or a smaller active parameter set; a model’s attention or routing pattern can motivate new kernels, cache layouts, fabrics, or accelerators. Tokens remain useful for accounting, but they are not the goal by themselves. The real target is useful model execution per dollar, watt, and second, within the required quality and reliability envelope.

From conventional infra to agentic infra

Key insightAcross four eras, AI infrastructure kept the same job while changing its mapping strategy as models and hardware co-evolved.

AI infrastructure has one durable job: run a model on hardware under a service objective. The model supplies an execution graph and state; the hardware supplies compute units, memory hierarchy, interconnect, storage, and power. Infrastructure decides how to map one onto the other without violating latency, throughput, reliability, or cost targets.

Compact models fit commodity infrastructure

In the CPU-centric era, production models were commonly compact classifiers, rankers, trees, or manually assembled feature pipelines. Their inference usually fit inside a conventional request path: load features from DRAM, run bounded computation on a CPU, and return a score. Infrastructure work concentrated on data pipelines, feature consistency, request scheduling, replication, and availability. Google Translate is a useful product anchor: its 2006 launch used statistical machine learning and large language-statistics pipelines rather than an end-to-end neural model.¹ The model was important, but it rarely dictated the shape of the datacenter.

Dense neural networks made accelerators part of the system

Deep learning changed the hardware contract. AlexNet paired a 60-million-parameter convolutional network with a GPU implementation and split the model across two GTX 580 GPUs because device memory was a binding constraint.² Dense tensor operations rewarded massively parallel processors; device memory, host-to-device copies, kernel launch overhead, and gradient synchronization became first-class systems concerns. Distributed training frameworks and accelerator-aware runtimes emerged to keep GPUs fed and to coordinate replicas.

Hardware then began specializing for the model. Google’s first production TPU, deployed in 2015, centered a 65,536-element 8-bit matrix multiply unit and 28 MiB of on-chip memory around neural-network inference.³ User-facing products made the shift visible: Google Photos launched in 2015 with automatic organization and search, while Google Translate moved eight language pairs to Neural Machine Translation in 2016.⁴ This is the recurring pattern: once a model family exposes a stable and valuable computation structure, infrastructure and silicon reorganize around it.

Transformers turned memory and communication into the bottleneck

The Transformer replaced recurrence with attention, making training more parallel while tying work and memory traffic to sequence length.⁵ By GPT-3, a single autoregressive model had reached 175 billion parameters,⁶ far beyond one accelerator’s memory. Infrastructure had to shard weights and activations across devices, combine data, tensor, and pipeline parallelism, and treat the interconnect as part of the computer.

Inference added a different execution pattern: prefill processes the prompt in parallel, while decode repeatedly generates one token and reads a growing KV cache. The systems response therefore moved beyond generic GPU scheduling. FlashAttention reduces traffic between HBM and on-chip SRAM,⁷ while PagedAttention applies virtual-memory-style paging to dynamic KV-cache allocation.⁸ These are infrastructure techniques derived directly from the model’s memory behavior and the hardware’s hierarchy.

The product transition was equally legible. The GPT-3-backed OpenAI API exposed a general-purpose text interface in 2020, and ChatGPT turned a GPT-3.5-series model into a conversational product in November 2022.⁹ Both depended on a serving layer that hid distributed model execution behind a simple request interface.

Agents make model execution a stateful workflow

Agentic systems do not replace the model-hardware mapping; they wrap it in a larger loop. ReAct demonstrated the core pattern by interleaving model-generated reasoning with actions against external environments.¹⁰ OpenAI Codex is a representative product: one delegated software task runs in its own cloud environment, where the agent can inspect a repository, edit code, execute tests, and propose a pull request.¹¹ A production agent generalizes that loop with retrieval, tool calls, code execution, tests, retries, and durable state. Different steps may belong on different resources: large-model decoding on GPUs or TPUs, embeddings or small models on NPUs, control logic on CPUs, and working memory in caches, databases, or object storage.

The infrastructure unit is therefore no longer only a request, experiment, training step, or token. It is a stateful execution graph whose model calls must be placed on heterogeneous hardware. Scheduling still matters, but it is one mechanism inside a broader mapping problem that includes compilation, numerical precision, parallelism, cache placement, communication, recovery, and tool isolation. The service objective also expands from tokens per second toward useful completed work per dollar, watt, and second.

That change creates the central problem for Part 2: both the work inside one goal and the number of machine-generated requests behind that goal can grow. Part 3 then asks what hardware and infrastructure become necessary when compute, memory, communication, and edge constraints all evolve at different rates.