A model stored with four-bit weights may fit on a GPU that cannot hold its sixteen-bit equivalent. That is a useful result. It does not mean the entire model now performs four-bit arithmetic, or that it will generate tokens four times faster.
The distinction starts with the numerical representation and ends with what the runtime actually executes.
FP8 and FP4 are not single formats
A floating-point encoding uses bits for sign, exponent and significand. Changing the allocation changes numerical range and precision. The FP8 Formats for Deep Learning paper describes E4M3 and E5M2, which use different exponent and significand sizes. E4M3 provides more precision within a smaller range; E5M2 trades some precision for range.
Four-bit floating point has an even smaller bit budget. The Open Compute Project MX specification defines formats that share scale information across groups of values. A four-bit integer checkpoint, a microscaled FP4 tensor and NVIDIA's NVFP4 recipe are not interchangeable just because each uses four-bit elements.
NVIDIA's Transformer Engine precision primer describes conventional FP8, MXFP8 and NVFP4 scaling. These recipes differ in block structure and scale representation. Some operations also accumulate or execute in wider types.
The raw memory calculation is only a starting point
Take a hypothetical model with 8 billion parameters. If every parameter occupies two bytes, raw weight storage is 16 GB (about 14.9 GiB). With one byte per parameter it falls to 8 GB; at four bits per parameter it is 4 GB before overhead.
The actual checkpoint also needs scale information, possibly zero points, tensor metadata and any layers left in wider formats. Runtime memory includes temporary tensors, activation storage and the KV cache. A 4 GB weight file therefore does not establish that an inference server fits inside a 4 GB device.
Scale granularity affects both cost and accuracy. With a shared scale for a large block, extreme values can reduce the useful resolution available to smaller values. Smaller groups may better represent the data but require more scale information. The format and kernel determine the trade-off.
Weight-only quantization is not the same as low-precision execution
AWQ is a weight-only quantization method that uses activation information to guide the representation of weights. A model using such a method can store low-bit weights while performing substantial computation with wider representations.
That can reduce memory traffic and improve throughput, especially when reading model weights dominates the execution. It may also add unpacking or dequantization work. Without an appropriate kernel, storing fewer bits does not guarantee faster arithmetic.
Hardware acceleration is format-specific. Consult the Transformer Engine support matrix for the actual library version and GPU compute capability. Its documented distinctions within the Blackwell family illustrate why a claim about a GPU generation is not enough to determine supported recipes. Other inference engines may implement a different set of kernels.
Measure model quality and latency separately
Suppose a service spends 40% of its response time in a matrix operation. Even if a lower-precision kernel cuts that portion in half, the total latency falls from 100 hypothetical units to 80. The end-to-end speedup is 1.25 times, not two times. That simplified example assumes every other part of the execution stays unchanged.
Memory savings may have a different benefit: more concurrent requests could fit on the same card. The vLLM team's FP8 KV-cache analysis examines this separate use of low precision and its quality and performance trade-offs. Weight quantization and cache quantization should be reported independently.
For a fair model comparison, evaluate both versions against the same task-quality criteria and representative inputs. Report the checkpoint format, activation and cache precision, runtime version, accelerator and request distribution. Compare first-token latency, generation speed and aggregate throughput under a realistic load.
A peak FP4 performance figure describes supported arithmetic. The purchase decision needs evidence about the model you plan to serve. GPU memory capacity and bandwidth describes the other constraints that remain after the representation becomes smaller.