Testing

此内容尚不支持你的语言。

Butter tests in three layers — kernel-wrapper correctness, Swift unit tests, and per-model integration tests — using Swift Testing (@Suite / @Test / #expect), not XCTest. CI gates on ≥ 80 % line coverage of the Swift surface plus a green integration sweep.

Running tests

For the Rust backend’s portable CUDA selector scripts and compile-only setup, see Prepare a CUDA selector campaign.

Go through make — do not run bare swift test. Each ModelIntegrationTests suite downloads a multi-GB checkpoint; an unconstrained parallel run loads several at once and OOMs the box (and can pin the GPU). The make targets cap parallelism correctly.

make test-unit          # ButterTests + IronSwiftTests — fast, parallel-safe
make test-unit-serial   # same unit surface, serialized for RST and race isolation
make test-integration   # ModelIntegrationTests — serialized (--num-workers 1)
make test               # both in sequence — the full local CI gate
make coverage           # unit-suite line coverage (≥ 80 %)
make test-stress        # production cap, uncapped parallelism — run after dispatch changes

make test runs make regenerate-kernels first, so you never test against stale kernels.

Use make test-unit-serial when investigating shared-state races, running mutation tests, or validating on a memory-constrained machine. It keeps the production command-buffer cap but disables cross-test concurrency. The regular unit gate remains the faster production-parity signal.

In a dedicated Butter worktree, point kernel regeneration at an Iron checkout whose HEAD matches .iron-revision when it is not adjacent to the worktree. The Makefile fails before deleting or regenerating artifacts when the revisions differ:

IRON_DIR=/path/to/iron make test-unit-serial

Kernel authors intentionally testing changes on top of another Iron revision must opt in explicitly:

IRON_DIR=/path/to/iron IRON_ALLOW_UNPINNED=1 make regenerate-kernels

Filtering to one suite or test

To iterate on a single suite, run swift test directly but keep the memory cap — --parallel --num-workers 1 loads one model at a time:

swift test --parallel --num-workers 1 --filter Qwen3TextIntegrationTests
swift test --parallel --num-workers 1 --filter ModelKVCacheMatrixIntegrationTests
swift test --filter OpsTests          # unit suite — fast, no cap needed

--filter matches a regex against suite + test names.

Test layout

Tests/
  IronSwiftTests/      One file per kernel wrapper — numerical
                            correctness vs a CPU reference across
                            fp32 / fp16 / bf16. Plus
                            KernelManifestSmokeTests.
  ButterTests/                Mirrors Sources/ButterSwift/ — every source file
                            has a sibling test. Audio/, Benchmark/,
                            Generation/, KVCache/, Loader/, Models/,
                            Ops/, Stats/, Telemetry/, Vision/ are the
                            top-level groups.
  ModelIntegrationTests/    Per-family end-to-end checkpoint runs,
                            grouped by modality (Text/, Vision/,
                            Audio/Omni/, Audio/STT/, Audio/STS/,
                            Audio/TTS/, Audio/VAD/), plus the
                            cross-cutting suites (see below).
  Helpers/                  CommonTestHelpers (`loadModel`,
                            `expectCoherentOutput`, `ModelLoadLock`),
                            TextTestHelpers, VisionTestHelpers,
                            AudioTestHelpers, RunAndWait.
  Resources/                Test inputs (dog.jpeg, cat.mp4, audio
                            clips, … shared via the helpers).

There are no golden fixtures. Cross-implementation token-parity vs mlx-lm proved to be a measure of rounding-mode alignment, not correctness — it was dropped. Numerical correctness now comes from the iron-side per-kernel GPU-correctness tests (compared to a naive CPU oracle); the Butter integration tests assert that the model pipeline produces coherent text.

Integration testing

Every model family has a Tests/ModelIntegrationTests/<Family>IntegrationTests.swift that downloads the smallest published checkpoint from mlx-community, greedy-decodes, and asserts expectCoherentOutput(...) (token-count floor, no degenerate repeat run, minimum token diversity). A checkpoint that can’t be fetched (offline, gated repo) prints a skip line and passes — integration tests never hard-fail on a missing download.

Cross-cutting suites:

ModelKVCacheMatrixIntegrationTests — the model family × weight bitwidth × KV-cache scheme cross-product.
Quantized{3,4,5,6,8}bitIntegrationTests — the weight-bitwidth ladder.
ModelDeterminismIntegrationTests — temp = 0 greedy decode is stable across runs.
ModelInspectionIntegrationTests — butter inspect end-to-end against a representative model from each family (verifies every family wired its InspectTap hooks).
SlidingWindowIntegrationTests — sliding-window KV eviction composes correctly across cache schemes.

Not every model runs by default — env-gated tests

The largest checkpoints are too heavy (or too slow) for the routine gate, so they are gated behind environment variables. With the var unset the test skips (and passes); set it to opt in:

Env var	Unlocks
`BUTTER_BUILD_MACHINE`	The heavy generation checks — `GPTOSSIntegrationTests` (~20B MoE), the Gemma 4 31B / 26B-A4B decode in `Gemma4TextIntegrationTests` (load + shape checks still run unconditionally), and every non-smallest cell of `ModelKVCacheMatrixIntegrationTests`. Intended for a dedicated build machine.
`BUTTER_MATRIX_FAMILY=<family>`	Restricts `ModelKVCacheMatrixIntegrationTests` to one family’s row (e.g. `BUTTER_MATRIX_FAMILY=Gemma4`) — fast targeted re-runs.

# Run the whole matrix incl. env-gated cells, on a build machine:
BUTTER_BUILD_MACHINE=1 swift test --parallel --num-workers 1 \
    --filter ModelKVCacheMatrixIntegrationTests

# Re-run just the Llama row of the matrix:
BUTTER_MATRIX_FAMILY=Llama swift test --parallel --num-workers 1 \
    --filter ModelKVCacheMatrixIntegrationTests

The default make test-integration runs only the always-on cells: the smallest checkpoint per family.

Writing a test

import Testing
@testable import ButterSwift

@Suite("Ops.add")
struct OpsAddTests {
    @Test("elementwise add matches a CPU reference")
    func addMatchesCPU() {
        let a = Tensor.empty(shape: [4], dtype: .f32)
        a.copyIn(from: [1, 2, 3, 4])
        // … dispatch, then:
        #expect(out.toArray(as: Float.self) == [2, 4, 6, 8])
    }
}

A model integration test loads through ModelLoadLock.shared (serializes the multi-GB load across suites) and asserts coherence. The canonical text-model pattern uses the loadModel(_:) helper from Tests/Helpers/CommonTestHelpers.swift — it wraps ModelLoadLock.shared.loadSerially { … } and fails the test on load failure instead of silently skipping:

import Testing
@testable import ButterSwift

@Suite("Qwen3 Text Integration", .serialized)
struct Qwen3TextIntegrationTests {
    @Test("load + greedy generate produces coherent output")
    func loadAndGenerate() async throws {
        let m = try await loadModel("mlx-community/Qwen3-1.7B-4bit")
        let r = try await m.generate(
            prompt: "Once upon a time, in a quiet village",
            parameters: GenerationParameters(maxTokens: 200, temperature: 0))
        expectCoherentOutput(r.generatedTokens, label: "Qwen3-1.7B")
    }
}

Audio + VL families typically need a typed cast after loading. The pattern is to resolve the snapshot through ModelLocator() first, then build the typed model directly so the test can call its modality-specific API (codec decode, vision-tower preprocess, …):

@Suite("FishSpeech Integration", .serialized)
struct FishSpeechIntegrationTests {
    private static let repoId = "fishaudio/openaudio-s1-mini"

    @Test("synthesises a coherent waveform")
    func synthesise() async throws {
        let dir = try await ModelLoadLock.shared.loadSerially {
            try await ModelLocator().resolve(idOrPath: Self.repoId)
        }
        let model = try await Model.load(dir.path)
        // … call the typed audio API on `model.audio`
    }
}

Both patterns serialize through the same lock so the multi-GB downloads / GPU footprint stay capped regardless of how many integration suites the runner picks up.

A non-trivial kernel lands with a paired iron GPU-correctness test in the same commit (see iron docs/testing.md).

CI

CI runs on Apple Silicon: the unit gate, then the serialized integration gate (matching make test-integration), and uploads the coverage report — a PR that drops Swift-surface coverage below the threshold fails.

What this Swift test guide doesn’t cover

Property / fuzz testing — revisit post-v0.2.
GPU mocking — all tests run real Metal dispatches.
Cross-implementation token parity — dropped (see above).
Rust CUDA / Linux — first-class Butter surfaces with their own Cargo tests under rust/crates/backends/wh-butter-cuda/tests; they are not run by the Swift make test target documented on this page.
Multi-GPU / multi-node — not implemented yet. A CUDA backend does not by itself provide rank placement or collectives.