LLM inference economics emerges from a coupled system: weights must fit, prompts must be prefetched, KV state persists, and decode produces one token per sequence at a time. Change the workload and watch the bottleneck move.
The moving blocks are requests. Cyan motion represents parallel prompt processing, yellow storage is KV state, and pink pulses represent autoregressive output tokens.
Weights usually dominate static memory. Quantization reduces bytes per parameter, but real formats add scales, zero points, padding, and runtime buffers. The simulator shows the ideal payload so the assumption remains visible.
Each active sequence stores keys and values for past tokens at every layer. GQA and MQA reduce KV heads, shrinking cache memory without changing weight count.
Processes prompt tokens in parallel. Longer prompts increase time to first token, although prefix reuse can skip repeated work when cache identity and permissions are safe.
Generates one next token for each active sequence per iteration. Small batches underutilize hardware; huge batches increase queueing and per-request latency.
Hourly hardware cost continues during idle gaps. Higher utilization spreads fixed time across more useful tokens, but saturation creates waiting and tail-latency risk.
Add weight payload, active KV cache, activations, runtime workspaces, and fragmentation. If the workload does not fit, throughput is irrelevant until parallelism or memory is changed.
Capacity boundaryDominated by queueing and prompt prefill. Interactive agents often value low TTFT because users perceive silence before streaming more strongly than small differences later.
ResponsivenessTotal generated tokens per second across the batch. It is not the same as per-user speed: aggregate throughput can rise while each request waits longer.
Fleet efficiencyEstimated occupied accelerator time divided across concurrent work. Actual price includes redundancy, networking, orchestration, storage, engineering, and margin.
Economic outputServing optimization is rarely one trick. Every change modifies quality, memory, latency, throughput, operational complexity, or all five.
Choose precision and parallelism with accuracy tests, not memory targets alone. Leave headroom for cache, temporary buffers, allocator fragmentation, and bursts.
Retrieve only relevant evidence, summarize durable history, and cap low-value context. Prefix caching helps repeated prompts, but cache keys must include model, adapter, policy, and tenant boundaries.
Continuous batching admits new requests as others finish, reducing padding and idle slots. Scheduler policy decides the tradeoff between throughput and latency fairness.
Average latency hides queue spikes. Track TTFT and inter-token latency percentiles by prompt length, output length, batch occupancy, model version, and hardware pool.
Each decode step may read a large fraction of model weights to produce relatively few tokens. Arithmetic units can wait on data movement, especially at small batch sizes.
No. Ideal weight bytes are four times smaller, but kernels, dequantization, KV precision, activation memory, bandwidth, batching, quality constraints, and hardware support determine realized gains.
More sequences share each weight read and expose more parallel work. The benefit eventually saturates, while queueing and memory consumption continue to grow.
Yes, if cache identity ignores tenant, permissions, policy, model configuration, or sensitive prompt differences. Reuse must preserve the same isolation rules as ordinary inference.
Explore SuperPowers workflows that turn explicit goals into inspectable agent work, with reusable computer state and complete website generation.
Explore SuperPowersComputer-use cacheWebsite-building agent