A company spends thousands of GPU-hours training a model, then deploys it to answer customer requests. The same accelerator model might be used in both stages, but the purchasing decision changes. Training needs to finish a learning task; inference needs to deliver useful responses when users ask for them.

A training job can often wait in a queue before it begins. A customer waiting for a reply may have a latency budget measured in seconds. Those different obligations affect memory, scheduling and cost.

Training has to store the learning process

Training performs a forward calculation, computes gradients and updates parameters. Beyond weights, memory may hold saved activations, gradients, optimizer state and temporary tensors. PyTorch's training-memory analysis breaks down these allocations and shows why optimizer state can be substantial.

Consider a simplified model with one billion parameters, FP32 parameters and gradients plus two FP32 Adam moment tensors. Those four groups alone occupy roughly 16 GB in decimal units. This excludes activations, temporary buffers and additional copies. Real mixed-precision, distributed and optimizer implementations can change the footprint substantially.

Activation checkpointing saves fewer intermediate values and recomputes some during backward execution. PyTorch's explanation describes the trade-off between memory and additional computation. It can make an otherwise oversized training job fit, but the result may take longer.

Larger allocations may finish sooner while increasing total GPU-hours. Compare the cost of reaching the required model quality, including any failed runs and recovery work, rather than assuming lower hourly rates always yield cheaper training.

Inference has two important phases

For a decoder-based language model, prefill processes the prompt and builds attention state. Decode produces subsequent tokens. The memory and compute demands can be different in the two phases.

Microsoft Research's Splitwise study characterized prompt processing as more compute-intensive and token generation as more memory-intensive in its evaluated workloads. The researchers explored placing the phases on separate machines. The result is workload-specific, not a universal recommendation to separate prefill from decode.

Serving also needs room for the active KV cache, as explained in our cache-memory article. Long prompts and concurrent conversations can consume substantial memory even though the model weights never change.

Throughput is not the same as goodput

Batching lets a server process requests together and share some work, potentially raising total throughput. But a request waiting for a batch may exceed its latency target. An overloaded service can generate a high total number of tokens while disappointing individual users.

The DistServe research at USENIX OSDI evaluates serving goodput against time-to-first-token and time-per-output-token requirements. It also considers the communication cost when the two phases are separated.

For an illustrative comparison, assume two servers each cost $2 per hour. One completes 1,000 requests in an hour but only 800 meet the service's latency and quality criteria. The cost per acceptable request is $0.0025. The second completes 900 requests and 890 meet those criteria, for about $0.00225 each. The second has lower raw throughput yet lower cost per acceptable request.

The example makes no claim about a real benchmark. It shows why completed useful work is the denominator that matters when customers buy a responsive service.

Capacity planning needs the actual demand curve

A training cluster may be busy for a long scheduled run, then release its resources. A production inference system may hold spare capacity ready for evening traffic spikes or unexpected long prompts. Daily average utilization does not reveal the queueing and memory pressure at peak time.

Input length, output length, request arrival rate and model quality all affect cost. A comparison per million generated tokens is useful only if the output is equivalent and other charges are accounted for. A cheap model with a lower task-success rate may require retries or additional processing.

For a hosted inference comparison, record the first-token latency, output token rate, tail latency, aggregate throughput and number of requests meeting the target. Include idle and standby capacity in the bill. For training, record time to the specified quality target, complete allocation cost and recovery overhead.

MLPerf inference scenarios provide a useful framework for interpreting different benchmark workloads. The point is to buy the resources needed for the actual obligation, whether that is a completed training run or a responsive model service.