Performance

此内容尚不支持你的语言。

The perf pass closed the gap between the correctness-first early implementation and what the M-series GPU can actually deliver on single-stream decode. This page captures the last-measured numbers, what each wave changed, and where the remaining headroom is.

Status note. The headline numbers below are the Wave 1 baseline (last sweep: 2026-05) — they were taken before the AURA-performance, chunked-prefill, and post-bisect kernel-substitution work landed. A fresh benchmark sweep is queued once the AURA compressed-domain default + per-attention-layer decodeMulti work settles. Tracking and timeline in planning/roadmap.md.

The Wave 1 headline numbers are single-stream decode tokens/sec on Apple M1 Max, measured at batch 1 with ~32-token prompts and maxNewTokens = 64. The concurrent-serving section uses a newer workload and reports aggregate client-observed throughput. Do not compare the two metrics as if they were the same benchmark.

Headline numbers (Wave 1 baseline, 2026-05)

Model	Quant	Pre-perf baseline	Wave 1 result	Speedup
Llama 3.2 1B	bf16	5.45 tok/s	64.6 tok/s	11.9×
Qwen 3 4B	bf16	5.0 tok/s	28.0 tok/s	5.6×
Qwen 3 4B	8-bit	4.7 tok/s	27.5 tok/s	5.9×
Qwen 3 4B	6-bit	4.2 tok/s	26.1 tok/s	6.2×
Qwen 3 4B	5-bit	4.0 tok/s	25.4 tok/s	6.4×
Qwen 3 4B	4-bit	5.0 tok/s	29.8 tok/s	6.0×
Qwen 3 4B	3-bit	3.6 tok/s	24.1 tok/s	6.7×

Llama 3.2 1B sees the biggest win because it’s small enough to be encoder-bound pre-perf — eliminating the per-layer commit + waitUntilCompleted was almost pure profit.

Concurrent serving (2026-08)

Dense Qwen 3.8 27B 4-bit with AURA KV was measured with resident sessions, isolated prompts, 128 maximum completion tokens, temperature 0.7, top-p 0.9, and an 8192-token context allocation. Completion throughput is the aggregate number of generated tokens divided by client-observed wall time across all clients. It is not isolated kernel decode throughput.

Concurrency	Aggregate completion tok/s
1	17.68
2	21.48
4	28.78
8	46.76
16	62.61

A repeat measured 44.41 tok/s at concurrency 8 and 64.23 tok/s at concurrency 16. The two concurrency 16 runs average 63.42 tok/s. Post-TTFT throughput was 64.71 and 64.38 tok/s at concurrency 16, TTFT p95 stayed below 2.8 seconds, and both sweeps reported zero foreign-lane and zero empty responses.

Compatible equal-sized prefill chunks share a dense model pass while retaining request-owned recurrent and KV state. These medians are client-observed uncached prompt throughput from three trials per point on the same model.

Workload	Concurrency	Uncached prompt tok/s
short prompt	4	347.03
short prompt	8	390.64
about 1K prompt tokens	2	598.65
about 1K prompt tokens	4	569.86
about 1K prompt tokens	8	552.37

The short concurrency 2 case stays serial because coalescing regressed that measured workload by 4.3%. The roughly 1K-token concurrency 8 case splits into two concurrency 4 batches to remain within the scheduler token bound. Current fusion covers dense Qwen without MTP and equal-sized ready chunks. Other model families retain bounded concurrent sessions but do not claim these fused throughput numbers.

What each perf wave changed

The perf pass was sequenced into two waves; this is what each one bought.

Wave 1 — encoder-bound wins

The pre-perf dispatch loop looked roughly like:

for each layer:
  rms_norm                      ──┐
  Q/K/V projections               │   ~30-70 MTLCommandBuffers
  rope                            │   per token, each with its own
  KV append (CPU memcpy)        ──┤   commit + waitUntilCompleted.
  sdpa_decode                     │
  ...                           ──┘
final rms_norm + lm-head + argmax (CPU readback of logits)

Wave 1 collapsed this:

kv_cache_update Metal kernel. Replaces the CPU memcpy + mid-layer sync with a GPU kernel that appends K/V into the per-layer cache buffer. No CPU↔GPU traffic during a layer.
Single MTLCommandBuffer per token. All layers + LM head + sampling enqueue onto one command buffer; one commit + waitUntilCompleted per token (down from ~30-70).
GPU-side argmax. Logits never leave the GPU; the kernel writes a single uint32 (the sampled token id). 4 bytes cross CPU↔GPU per token instead of vocab_size × 4.
forwardSample(...) on LanguageModel. Single entry point that runs the forward pass + GPU argmax in one trip; the Generate.swift loop uses it exclusively now.

Wave 1 alone took Llama 3.2 1B from 5.45 to ~50 tok/s and Qwen 3 4B from ~5 to ~22 tok/s.

Wave 2 — kernel-bound wins

Once the encoder overhead was gone, the GPU kernels themselves became the bottleneck:

Cooperative-thread gemv. Replaces the naive one-thread-per- output-row gemv with the strided-reduce-dot pattern (one threadgroup per row, simd_sum reduction across the in_dim axis). Brought bf16 / fp16 matvec close to the M-series memory bandwidth ceiling.
Multi-row RMSNorm. Qwen 3’s per-head q_norm / k_norm was dispatching one rmsNorm per head (32 + 8 = 40 launches per layer × 36 layers = 1440 launches per token). Replaced with a single multi-row dispatch — 36 launches per token total. Significant on Qwen 3, no-op on Llama.
Sub-group split per-pack dequant_gemv. For 3 / 5 / 6-bit (byte-packed widths), assign one SIMD subgroup per pack within a row. Closes the perf gap between byte-packed widths and the uint32-aligned 4-bit / 8-bit. Without this, 3-bit was 30% slower than 4-bit; after, it’s within 20%.
RoPE → KernelMode::Grid3D. Previously Elementwise (1D), forced redundant program-id math. Grid3D matches the (seq, n_heads, head_dim) shape directly.

Wave 2 lifted Qwen 3 4B 4-bit from ~22 to 29.8 tok/s and brought the byte-packed widths within striking distance of the uint32-aligned ones.

Where the remaining headroom is

The main outstanding gaps vs MLX’s hand-tuned fused kernels are:

Fused RMSNorm + gemv. MLX folds the RMSNorm scale into the pre-matmul rescale of the next gemv. We dispatch them separately. Expected ~10-15% on the hot QKV path.
Fused QKV projection. MLX dispatches one gemv for Q, K, V concatenated, instead of three. Expected ~5-10%.
Online-softmax SDPA decode. Our sdpa_decode is correct but not yet using the simdgroup-cooperative online-softmax pattern from iron-bench. Largest remaining headroom on long-context decode.
Argument-buffers / ICB dispatch modes. Pre-bind weights into a per-layer argument buffer (or pre-record the entire forward pass via Indirect Command Buffers), so per-token only the activations + KV offset get bound. ~5× fewer setBuffer calls. Pulled in once profiles show encoding cost still matters after AURA + chunked-prefill land.
Autotuner. Per-shape selection of (tile_dims, threads, unroll, simd_matrix, async_copy). Tracked in planning/plan.md.

How to reproduce

make test                               # full suite, includes PerfTests
swift test --filter PerfTests           # just perf

PerfTests are deterministic for a fixed seed but their timing is machine-dependent; the recorded tok/s thresholds in CI are gentle floors (regression detection, not hard targets).

For a manual sanity check:

butter --model mlx-community/Qwen3.5-0.8B-MLX-4bit --prompt "Once upon a time" --max-tokens 64
butter --model mlx-community/Qwen3-4B-4bit --prompt "Once upon a time" --max-tokens 64

The CLI prints prompt: N tokens (Xs prefill) + generated: N tokens in Ys (Z tok/s) at the end of each run.

Wave 1 methodology notes

Single-stream decode. No batching, no speculative decoding.
Greedy. Greedy-argmax sampling on the GPU. No temperature / top-k / top-p in the hot path.
bf16 / fp16. Inputs and accumulators per the kernel; final reduction in fp32.
Cold prefill. Caches start empty; the prefill column is the first-N-token slow loop, the decode column is everything after.
tok/s = generated_tokens / decode_time_s. Excludes prefill, excludes load. Same convention as mlx-lm and llama.cpp.
No prewarm in the timing. The PSO cache is populated by LoadOptions.prewarm = true during Model.load (default on); the per-token timer starts after that.