A training job that takes eight hours on one GPU will not necessarily take one hour on eight. The devices may finish their individual calculations quickly and then wait while gradients or other tensors are exchanged. The network connecting those devices becomes part of the computation.

Three different paths through an AI system

PCIe connects a host to GPUs, network adapters and other peripherals through root complexes and switches. On a multi-GPU server, two GPUs may communicate through a nearby switch or across a CPU socket boundary. Peer access depends on the platform and configuration, not simply on the presence of PCIe.

NVLink provides a separate accelerator interconnect on supported NVIDIA hardware. Scale-up communication is no longer confined to a conventional server chassis. The DGX Grace Blackwell networking guide describes a rack-wide NVLink domain with a scale-out InfiniBand compute network between racks. It also separates storage, management and external networking.

Scale-out networks connect larger groups of devices or servers. Their behavior depends on the topology, switch capacity, oversubscription and traffic patterns. An Ethernet or InfiniBand port speed describes one link, not the throughput of the whole cluster.

The division matters when renting several GPUs. Are they in the same high-bandwidth accelerator domain, in separate servers connected by a fast fabric or distributed across an ordinary shared network? The GPU model alone cannot answer that.

Compare bandwidth using the same direction

The PCI-SIG PCIe 6.0 FAQ describes a 64 GT/s signalling rate and up to 256 GB/s bidirectional bandwidth for an x16 connection. The two-direction figure corresponds to approximately 128 GB/s per direction at the nominal link level, not 256 GB/s for one application transfer. Protocol overhead reduces useful payload throughput.

A 400 Gbit/s network port has a nominal line rate of 50 GB/s in one direction before overhead, using decimal units. It is not directly comparable to a PCIe or NVLink aggregate quoted across both directions or many links.

Read the specification closely: per-link, per-GPU and switch-wide bandwidth are different metrics. A switch's aggregate capacity cannot be assigned to each device individually. End-to-end throughput also depends on the slowest relevant link and competing traffic.

Distributed jobs use collectives

NVIDIA's NCCL overview describes collective operations such as AllReduce, AllGather, ReduceScatter and AlltoAll. AllReduce can combine gradients across workers, while other operations gather or redistribute model state. The communication pattern depends on the training or inference strategy.

For a simplified ring AllReduce over eight participants, reducing and distributing a 1 GB tensor moves approximately 1.75 GB per participant across the ring over its two phases. The calculation is 2 × (8 − 1) / 8 × 1 GB. Real implementations can choose other algorithms and topologies; the example illustrates why the tensor size alone does not describe all communication.

Small, frequent exchanges can be sensitive to latency. Larger transfers may be limited by bandwidth. Operations that overlap communication with computation can hide some waiting, but synchronization still matters when one worker falls behind.

Direct GPU access has prerequisites

NCCL's GPU troubleshooting documentation describes GPU peer access and GPU-to-network-adapter communication. Platform settings, virtualization and the exposed device topology can determine whether direct paths work.

An advertised fast NIC does not guarantee that a guest can use GPUDirect RDMA. A GPU allocation may have a high-speed interconnect on the physical host but expose different capabilities to the VM. Ask for topology and collective performance measurements for the actual allocation, not just a link specification.

Scaling and the compute bill

Suppose a job requires eight hours on one GPU. On four GPUs it completes in three hours. The speedup is 2.67 times and the scaling efficiency relative to ideal four-way scaling is about 67%. It consumes 12 GPU-hours instead of eight. At the same per-GPU rate, it costs 50% more in accelerator time, though finishing five hours earlier might be valuable.

The useful benchmark measures the complete application across the intended devices and identifies the network used. Compare elapsed time, aggregate GPU-hours and the result quality. That puts the interconnect in the purchasing decision without mistaking peak bandwidth for model performance. For the other hardware constraints, see choosing an AI accelerator beyond peak FLOPS.