A server with twice the advertised FLOPS does not necessarily finish an AI job in half the time. The difference might disappear in a memory-bound kernel, or the faster accelerator might spend much of its time waiting for data from the CPU. Before comparing chips, establish what the application actually needs to do.
Where CPUs, GPUs and TPUs fit
CPUs handle general-purpose work well: application logic, irregular operations, request processing and much of the preparation that happens before a model executes. A small inference model may also run economically on CPU hardware already available to the application. Intel's CPU inference overview discusses how different stages of inference have different requirements. It is a useful starting point, not evidence that CPU inference always wins on cost.
GPUs provide many parallel execution resources and specialized matrix hardware. They are a natural choice for large, regular numerical workloads, particularly when the software can keep those resources busy. Google's TPU architecture documentation describes a more specialized system built around matrix multiplication units, memory and accelerator interconnects. A TPU workload also depends on its compilation and execution stack.
The important comparison is the whole application. A recommendation model that needs extensive CPU-side preprocessing has a different profile from a large language model with long batches of matrix operations. Neither should inherit the purchasing decision made for the other.
Read the units on the specification sheet
FLOPS measure floating-point operations per second. They are meaningful only when the precision and kind of arithmetic are specified. AMD's MI300X specifications, for instance, distinguish ordinary FP64 operations from matrix FP64 performance and list separate dense and sparse lower-precision figures.
A sparse FP8 number is not directly comparable with a dense FP16 number. Sparsity accelerates particular supported patterns; an application must actually meet those conditions. Lower precision also needs to preserve the quality required by the model. A comparison between two processors should use the same numerical task or explain how the different representations affect the result.
The bottleneck might be memory
The roofline model gives a useful way to reason about this. If a kernel performs 10 floating-point operations for every byte fetched from memory and the device has a peak bandwidth of 2 TB/s, its ideal memory-limited rate is 20 TFLOPS. A theoretical compute ceiling of 100 TFLOPS would not remove that memory limit. Real performance will be lower if the kernel cannot approach the quoted bandwidth.
The numbers are illustrative, not measurements of a particular accelerator. NVIDIA's GPU performance background explains arithmetic intensity and the effects of insufficient parallelism. Small workloads may not occupy a large GPU effectively; large workloads may instead saturate memory or communication links.
Capacity comes first. If the model cannot fit in the intended allocation, the plan may require quantization, offloading or several devices. Each changes the data path. See how memory capacity and bandwidth differ and the role of GPU interconnects for the consequences.
The software stack can decide the purchase
A processor's theoretical ability to execute an operation does not guarantee that a particular framework has an efficient implementation for it. AMD's ROCm compatibility matrix illustrates the dependencies between accelerator hardware, operating systems and supported framework versions.
Before migrating a workload, account for custom kernels, compiler behavior, libraries and any numerical differences that need validation. A cheaper rental can lose its advantage if the application needs extensive porting or a different model-serving implementation. Conversely, there is little reason to move a working application to a more popular platform unless the move solves a real problem.
Compare cost per result
Suppose one rental costs $2 per hour and finishes an accepted job in six hours. A second costs $3 per hour but needs only three hours. The respective compute bills are $12 and $9, assuming each price covers the entire allocation. Additional storage, transfer and engineering costs could change that ranking.
For interactive inference, the equivalent comparison is cost per successful request within the required latency target. Include the arrival pattern, input and output sizes, quality evaluation and complete hardware allocation. For batch processing, elapsed time and total device-hours may be enough to narrow the options.
Those are the figures worth asking a hardware supplier to demonstrate. Peak FLOPS help explain a result; they cannot replace the result.