A GPU with 80 GB of memory may run a model that cannot fit on a 48 GB card. That alone does not tell you which GPU generates tokens faster for a model that fits on both. Capacity answers whether the data fits; bandwidth influences how quickly the processor can move it.

Start with the memory budget

A dense 70-billion-parameter model with two bytes per parameter needs 140 billion bytes for weights alone, about 130.4 GiB. That does not include temporary tensors, runtime allocations or the KV cache used by language-model serving. Two nominally 80 GB devices might have enough combined capacity for those weights, but the model still needs an appropriate partitioning strategy and sufficient space on each device.

With training, the footprint can include gradients, saved activations and optimizer state. Inference usually avoids those allocations but can accumulate substantial request state. A four-bit checkpoint reduces weight storage without necessarily reducing the cache or all runtime buffers.

Product variants matter. NVIDIA's H100 specifications distinguish SXM and NVL configurations, including memory and bandwidth. AMD's MI300X specifications list 192 GB of HBM3 and 5.3 TB/s peak local memory bandwidth. These are specific hardware properties, not benchmark results.

Bandwidth sets a limit on data movement

Arithmetic intensity is the number of operations performed per byte transferred. AMD's HIP performance optimization documentation explains this using the roofline model. A workload that reuses data many times can be compute-bound; a workload that repeatedly loads large tensors with little computation can be memory-bound.

For a rough illustration, reading 100 GB of model weights from a memory subsystem rated at 2 TB/s takes at least 50 milliseconds if the entire transfer reaches peak bandwidth. If a single decoding step had to read that full amount, the weight-transfer limit alone would correspond to about 20 steps per second. Real kernels may achieve less bandwidth, reuse cached data or move additional tensors. Multiple requests in a batch can also share the weight reads, so this is not a general tokens-per-second prediction.

This is why a large arithmetic figure is not enough to explain decoding performance. On some workloads, faster memory access is more useful than additional compute capacity.

Batching changes how memory is used

Serving several requests together allows a runtime to reuse model weights across a batch. The additional requests need memory for their own active state. A larger batch can raise aggregate throughput while making an individual response slower or consuming too much memory.

The PagedAttention paper addresses allocation fragmentation and duplicated KV-cache storage. Better allocation can increase the number of requests a fixed GPU memory budget supports. It does not increase the physical memory bandwidth.

For a hosted service, ask for both aggregate output and per-request latency under a representative request distribution. A benchmark with one short conversation is a poor substitute for a service handling many long prompts. See how long contexts affect KV-cache memory for the actual storage arithmetic.

Multiple GPUs are not one ordinary memory bank

Adding the memory capacity of several accelerators gives an inventory total, not an automatically shared address space for an application. Tensor or pipeline parallelism distributes work and state according to the execution strategy. Communication across devices adds another constraint.

Local HBM bandwidth, PCIe bandwidth and GPU-to-GPU interconnect bandwidth describe different paths. If a model requires frequent communication between cards, the speed of those exchanges can become as important as memory bandwidth on each card. PCIe, NVLink and scale-out networking explains how those paths differ.

Use a benchmark that resembles the service

For a model that already fits with sufficient runtime headroom, compare measured first-token latency, output token rate and concurrent capacity. Record the precision, input and output lengths, batch size and complete device allocation.

If the model does not fit, a larger-memory GPU might be worth more than a card with higher peak bandwidth because it avoids an expensive distribution or offloading strategy. Once it fits, the decisive constraint may change. That is why the memory budget and measured workload results belong alongside the hardware specification.