NCCL backend contract

Butter’s optional wh-butter-cuda/nccl feature provides same-stream F32 all-reduce for Butter-owned CUDA tensors. It is a backend primitive, not full tensor-parallel model wiring.

Deployment preflight

Multi-rank startup is fail-closed unless every process has this bounded RoCE failure policy:

export NCCL_IB_TIMEOUT=10
export NCCL_IB_RETRY_CNT=1

NCCL’s defaults did not report a lost RoCE peer inside Butter’s five-second failure bound. The final merged-pin gate surfaced ncclRemoteError in 523.321 ms on the two-host GB10 pair. They are required even when another transport is expected, so an unintended RoCE fallback cannot silently weaken the failure contract.

The process bootstrap owns cross-rank agreement before communicator creation. It must compare:

• world size, rank assignment, collective shape, and collective sequence;
• the integer returned by nccl_runtime_version();
• a SHA-256 digest of the exact libnccl.so.2 deployment artifact.

The backend verifies NCCL’s actual rank and CUDA device after initialization. It cannot discover which filesystem path the dynamic loader selected, so the launcher must hash that resolved library and pass the digest through its bootstrap metadata. The model-free receipt uses BUTTER_NCCL_LIBRARY_SHA256 for this launcher-owned value.

For example:

export NCCL_LIBRARY=/lib/aarch64-linux-gnu/libnccl.so.2
export BUTTER_NCCL_LIBRARY_SHA256="$(sha256sum "$NCCL_LIBRARY" | awk '{print $1}')"
ldd ./nccl_allreduce | grep libnccl

Both ranks must also use the same transport selection. The validated GB10 RoCE gate used:

export NCCL_SOCKET_IFNAME=enp1s0f1np1
export NCCL_IB_HCA=rocep1s0f1
export NCCL_NET=IB
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,NET

The INFO log must name Using network IB and via NET/IB/0 on both ranks. An absent external libnccl-net.so is acceptable when NCCL reports its built-in IB provider; a socket fallback is not an equivalent performance gate.

Ordering and lifecycle

Butter currently admits one physical CUDA device per process, so all NCCL communicators share one process-global submission lane, including communicators created through separate CudaDevice wrappers. Collectives enqueue on Iron’s ordered stream between producer and consumer kernels. The hot path does not query NCCL async status after every collective; callers poll check_async_error() at request/sample boundaries or use wait_for_async_error() while handling a suspected peer loss.

CUDA graph capture is rejected until capture-safe communicator ownership is designed and tested. Call abort() when its completion/error matters. Dropping an active communicator transfers cleanup to a detached abort thread so Rust destruction does not wait on a failed transport or panic on a poisoned mutex.

World-size one is a no-NCCL bypass that still advances the collective sequence.