A benchmark chart says one AI server generates more output than another. Before using the result to choose hardware, look for the scenario. A machine processing an entire batch at once may achieve excellent throughput while behaving differently when users submit requests throughout the day.

MLPerf Inference is useful precisely because its rules specify what is being measured. The details matter more than the largest number in the chart.

Offline and Server answer different questions

The MLCommons inference rules define workloads and how queries are presented to the system. Offline measures throughput with the samples supplied together. Server models a stream of arrivals and applies the latency constraints specified for the benchmark. SingleStream and MultiStream have their own request patterns and metrics.

For a batch job, maximum completed samples per second may be the relevant result. For an interactive service, responsiveness is part of the product. A larger batch might increase total throughput while making some users wait longer.

Language-model benchmarks can also distinguish time to first token from the rate of subsequent generation. The latency and quality requirements are part of the comparison. Check the actual model and scenario rules for the release in question rather than assuming a generic Server result uses a particular token-latency threshold.

One row describes a complete system

The MLPerf Inference benchmark paper explains the rationale for measuring combinations of hardware and software. A result belongs to the submitted system, not to the accelerator model in isolation.

Suppose an eight-GPU system delivers 16,000 hypothetical output tokens per second in a specified benchmark. Dividing by eight gives 2,000 tokens per second per installed GPU as an accounting measure. It is not a measured result for a single-GPU server. The model may be partitioned across those devices or benefit from a batch size that would not fit on one.

The submission guidelines explain how to inspect the implementation and submitted artifacts. Those details help explain the effects of accelerators, CPUs, software and execution settings.

Match the division and benchmark version

MLCommons' September 2026 Inference v6.1 analysis describes the distinction between Closed and Open submissions. Closed results are constrained by reference-model equivalence and the division's rules. Open allows changes to the model and preprocessing under its own requirements.

A different model or task-quality target can be useful in practice, but its speed cannot demonstrate that an unchanged model got faster. Compare the same model, division, benchmark version and scenario before making that claim. Also check the accuracy target and whether the submitted system is available for the intended purchase.

Benchmark versions change. A result from one release does not automatically remain comparable with a later release when the model, rules or workload change.

Use measured whole-system power

The MLPerf Datacenter results documentation specifies that accompanying power measurements use AC power for the complete system, measured at the wall during the benchmark. Server and Offline results can include system power; stream scenarios can report energy per stream. A GPU TDP is not a substitute for that measurement.

For illustration, a system delivering 10,000 output tokens per second while drawing a measured 2 kW would produce five tokens per joule at that test point. If a second system produced 12,000 tokens per second at 3 kW, it would deliver four tokens per joule. The second has higher throughput but lower energy efficiency on those hypothetical figures. The calculation only makes sense for comparable tasks and quality requirements.

That metric also excludes many costs of a production service, such as spare capacity, networking, storage and facility overhead. It is a system measurement, not the final hosting invoice.

Before relying on a published result

The MLCommons results-messaging guidelines identify information that should accompany a benchmark claim. In practice, record the model, version, scenario, division, quality target, system configuration and accelerator count. Confirm whether the result is verified and whether any power figure belongs to the same benchmark.

Then ask whether the benchmark resembles the planned service. If the application handles longer prompts or tighter response-time requirements, the result may be useful for shortlisting hardware without settling the purchase. Our articles on GPU memory constraints and training and inference economics cover the workload-specific factors that remain.