Testing

이 콘텐츠는 아직 번역되지 않았습니다.

How kernels and codegen are verified, what runs where, how to write a test, and — importantly — the gaps in the test infrastructure that let bugs through silently.

The test layers

Correctness is checked in layers, each catching what the layer above cannot:

Layer	Catches	Where it lives	Runs in CI?
DSL / codegen unit tests	Pass correctness, body-parser arms, IR variants, emit paths; `trybuild` compile-fail fixtures	`crates/wh-iron-codegen`, `wh-iron-core`, `wh-iron-macros`	✅
Test-inventory witness	Silent registration shrinkage, dead stripping, renamed cases, dtype/tolerance drift	`crates/wh-iron-std/tests/kernel_test_inventory.rs`	✅ (macOS runner)
Test-inventory drift check	A file declaring `#[test]`, `#[test_kernel]`, or `#[bench]` with no audit row in `docs/test-inventory-status.tsv`, and a stale rendered `docs/test-inventory.md`	`make check-test-inventory` / `scripts/render-test-inventory.sh`	✅ (Lint lane)
MSL snapshots (`insta`)	Codegen output drift — a reviewable text diff in the PR	`crates/wh-iron-codegen/tests/msl_snapshots.rs`	✅
GPU correctness	Numeric disagreement vs a naive CPU oracle, on a real Metal device	`crates/wh-iron-std/tests/<kernel>_gpu_correctness.rs`	✅ (macOS runner)
Backend-specific GPU corpus	Target lowering, packed layout, launch geometry, graph behavior, and device-only boundary cases	`crates/wh-iron-runtime/tests/`, `crates/wh-iron-std/tests/`, plus consuming-engine integration tests	Target hardware; not every backend runs in general CI
MLX side-by-side (bench)	Throughput + numeric parity vs the upstream MLX kernel	`iron bench`	local-only (needs an MLX checkout)

No single layer is sufficient. The unit tests never touch a GPU; snapshots pin whatever the codegen emits, including wrong output; a target compiler only checks that source is accepted. A real-device oracle for every production backend is the floor. Metal CI does not prove CUDA, HIP, or Vulkan execution, and a backend kernel test does not prove model composition. See the gaps section below.

The inventory witness pins the sorted source, test name, dtype, and tolerance for every registered declarative GPU case. An intentional corpus change updates its case count and digest in review. The witness proves registration identity, not execution; iron test still owns device dispatch, skip accounting, and the CPU-oracle result.

Every GPU correctness test above dispatches one kernel at a time on an otherwise-idle device — a condition under which pool and chain-ordering defects stay invisible. crates/wh-iron-std/tests/metal_buffer_pool_stress_gpu.rs (part of the “Backend-specific GPU corpus” row) targets that gap: it runs a background-load Context on its own thread while asserting that (1) a multi-pass fused dispatch chain’s HazardTrackingModeUntracked buffers stay correctly ordered across passes under concurrent GPU load, (2) pooled buffers never hand back a stale payload, and (3) the per-device buffer pool stays coherent across multiple Contexts and threads, including a Context moved between threads mid-run. See issue #192.

Running tests — what runs where

make test        # whole workspace: codegen, runtime, GPU correctness (GPU on a Mac)
make clippy      # lint, -D warnings
make fmt-check   # formatting
make typos       # spell-check
make coverage    # HTML coverage report (needs cargo-llvm-cov)
make bench       # MLX side-by-side benchmark suite (macOS + Metal only)

Per-kernel, via cargo directly (these are the documented exceptions to “always use make”):

# One kernel's GPU correctness test:
cargo test -p wh-iron-std --test <kernel>_gpu_correctness

# One kernel's perf bench (the #[ignore]'d companion test):
cargo test --release -p wh-iron-std --test <kernel>_gpu_correctness -- --ignored --nocapture

CI vs local

Job	Workflow	What it runs
`typos` / `clippy` / tests	`.github/workflows/check.yml`	spell-check, lint `-D warnings`, `cargo test --workspace` (Ubuntu — no GPU)
build / test / bench	`.github/workflows/iron.yml`	`iron build`, `iron test` (GPU correctness vs CPU oracle), `iron bench` — on a macOS GPU runner
coverage	`.github/workflows/coverage.yml`	`cargo llvm-cov --workspace --codecov` on macOS, uploads to Codecov; runs on pushes touching `crates/`, `Cargo.*`, `rust-toolchain.toml`, `.github/configs/codecov.yml`
PR title	`.github/workflows/pr.yml`	validates the conventional-commit format
labels	`.github/workflows/auto-label.yml`	release-notes labels from the PR-title prefix

The DSL / codegen / GPU-correctness layers all run in CI — including on a macOS runner with a real GPU.
iron bench benches the wh-iron kernels by default; the MLX side-by-side A/B is opt-in via iron bench --mlx (it needs an MLX checkout, so the default CI bench runs wh-iron-only).

macOS runner environment

The macOS CI jobs (iron.yml build/test/bench, plus coverage.yml and release.yml) run on the macos-26 GitHub-hosted runner. To see exactly what is installed (Xcode versions, SDKs, CLI tools), consult the image manifest: runner-images → macos-26-arm64 readme.

NAX / MPP kernels need the macOS 26.5+ Metal toolchain. The cooperative-tensor kernels (mpp::tensor_ops::matmul2d) compile via the Metal toolchain that ships with the OS. Older versions either reject the dynamic cooperative-tensor allocation (“unsupported deferred-static-alloca-size”) or ship MPP headers without the get_left_input_cooperative_tensor / get_right_input_cooperative_tensor accessors. This is the runtime newLibraryWithSource compiler and system headers, which are the OS’s — selecting a newer Xcode (DEVELOPER_DIR) does not change it (that only swaps the offline metal compiler). The macos-26 runner image is currently behind those APIs, so iron test skips any cooperative-tensor kernel whose pipeline won’t build (Kernel::requires_cooperative_tensors() → [SKIP]), reporting them as skipped rather than failed. They still compile + get correctness-tested wherever the toolchain supports them, and CI automatically starts exercising them when the runner OS catches up. Numeric mismatches and unrelated compiler errors still fail; only the known capability diagnostics skip.

Writing tests

Every non-trivial kernel ships a GPU correctness test — same commit

The test runs the kernel on a real Metal device and compares against a naive CPU reference computed in f32. Shared helpers (ramp, dtype pack/unpack, max_abs_diff, naive_*) live in crates/wh-iron-std/tests/common/mod.rs.

#![cfg(target_os = "macos")]
mod common;
use common::{ramp, pack_bytes, unpack_bytes, max_abs_diff};
use wh_iron_runtime::Context;

#[test]
fn my_kernel_matches_naive_cpu_reference_f32() {
    // 1. Build small synthetic inputs (ramp / deterministic pattern).
    // 2. Compute a naive CPU reference in f32.
    // 3. Pack to bytes, populate the buffer map, dispatch via
    //    Context::dispatch_with_grid(&kernel, &buffers, &constexprs,
    //                                 grid_xyz, threadgroup_xyz).
    // 4. Unpack the output buffer; assert max_abs_diff < 1e-4.
}

#[test]
#[ignore = "perf bench, run via --ignored --nocapture"]
fn my_kernel_perf_bench_f32() {
    // 20 warmup + 100 measure iterations; report median GPU µs + GB/s.
}

The naive CPU reference is the contract. If kernel and reference disagree, decide which is wrong before merging — don’t loosen the tolerance to pass.

New: declarative `#[test_kernel]` / `#[bench]` (additive, opt-in)

Alongside the hand-written tests/*_gpu_correctness.rs files, a kernel can now declare its correctness test and benchmark next to the kernel with the #[test_kernel] / #[bench] attributes. This is being introduced additively — the legacy tests/*_gpu_correctness.rs files keep working unchanged, and during migration a kernel can carry both so old and new are A/B-compared on the same IR. crates/wh-iron-std/src/mlx/arange.rs is the first kernel ported; use it as the template.

use wh-iron::kernel;

#[kernel]
pub fn iron_arange<T>(out: Tensor<T>, start: Tensor<T>, step: Tensor<T>, #[constexpr] n: u32) { /* … */ }

pub mod kernel_tests {
    use wh-iron::{test::*, test_kernel};
    use super::iron_arange;
    use crate::utils::{pack_f32, scalar_bytes};

    fn setup(start: f32, step: f32, n: usize, dt: DType) -> TestSetup {
        let expected: Vec<f32> = (0..n).map(|i| start + i as f32 * step).collect(); // CPU oracle in f32
        TestSetup::new(iron_arange::kernel_ir_for(dt))
            .input(TestBuffer::from_vec("out", vec![0u8; n * dt.size_bytes()], dt))
            .input(TestBuffer::from_vec("start", scalar_bytes(start, dt), dt))
            .input(TestBuffer::from_vec("step", scalar_bytes(step, dt), dt))
            .constexpr("n", n as u32)
            .expect(TestBuffer::from_vec("out", pack_f32(&expected, dt), dt))
            .grid_1d(n, 256)
    }

    #[test_kernel(dtypes = [f32, f16, bf16], tol = 1e-6)]
    fn test_iron_arange_ascending(dt: DType) -> TestSetup { setup(0.0, 0.5, 64, dt) }

    // Per-dtype tolerances (order matches `dtypes`): f32, f16, bf16.
    #[test_kernel(dtypes = [f32, f16, bf16], tol = [1e-6, 1e-2, 1e-1])]
    fn test_iron_arange_fractional_step(dt: DType) -> TestSetup { setup(0.0, 0.1, 64, dt) }
}

pub mod kernel_benches {
    use wh-iron::{bench, test::*};
    use super::iron_arange;
    use crate::utils::scalar_bytes;

    #[bench(dtypes = [f32, f16, bf16])]
    fn bench_arange(dt: DType) -> BenchSetup {
        let n = 64 * 1024 * 1024usize;
        BenchSetup::new(iron_arange::kernel_ir_for(dt))
            .buffer(BenchBuffer::zeros("out", n, dt).output())
            .buffer(BenchBuffer::from_vec("start", scalar_bytes(0.0, dt), dt))
            .buffer(BenchBuffer::from_vec("step", scalar_bytes(1.0, dt), dt))
            .constexpr("n", n as u32)
            .grid_1d(n, 256)
            .bytes_moved((n * dt.size_bytes()) as u64)
    }
}

Notes:

Buffers bind by name (matching the kernel parameter names); ordering of .buffer()/.input() calls doesn’t matter. #[constexpr] scalars are passed as little-endian uniform buffers, same as the hand-written tests.
The CPU oracle is the same contract as above — compute expected in f32, let the runner pack to the dtype and diff within tolerance. tol accepts a scalar, a per-dtype array, or a { f32: …, f16: … } table.
Run them with iron test [-f <filter>] and iron bench [-f <filter>]; the new benches render in the same table as legacy rows. The tests/kernel_tests_harness.rs cargo bridge runs every #[test_kernel] under cargo test so the new path is part of the commit gate without iron test.
This in-process runner is deliberately simple (it re-dispatches per iteration rather than reusing the legacy GpuRunner’s resident-buffer + DVFS-pinning path), so new-syntax bench GB/s currently reads lower than the legacy rows — fidelity is a follow-up, correctness is not affected.

MSL snapshots for new emit paths

A new DSL primitive, fusion pattern, or dtype path also lands an insta fixture in crates/wh-iron-codegen/tests/msl_snapshots.rs — a hand-built kernel run through MslGenerator, with the full MSL pinned via assert_snapshot!. Any future codegen change then surfaces as a reviewable text diff. Refresh intentional changes with cargo insta review (interactive) or cargo insta test --accept.

Fixtures exist to exercise distinct emit paths, not to be exhaustive — add one when a new path lands that the existing snapshots don’t cover.

Optional-buffer placeholders under shader validation

A kernel whose optional buffer is documented as “never read when the flag is off — any 1-element buffer satisfies the binding” is tested at two layers, because the ordinary correctness tests bind full-size buffers for the disabled case and cannot see an out-of-bounds read:

Emitted-code check — an in-source #[cfg(test)] runs the kernel through MslGenerator and asserts, via crate::utils::msl_check::assert_indexed_loads_guarded_by, that every buffer[...] load sits directly inside an if whose condition derives from the flag (see the select hazard in Developing).
Device check — crates/wh-iron-std/tests/optional_buffer_placeholder_gpu.rs binds the one-element placeholder with the feature off and enough heads / timesteps to index past it, and re-executes each case in a child process with MTL_SHADER_VALIDATION=1 and MTL_SHADER_VALIDATION_REPORT_TO_STDERR=1. The validation layer bounds-checks every device-memory access against the bound buffer’s length and prints Invalid device load … buffer: <unnamed>, length:4 on a fault (the read itself returns 0 under the default zerofill fail mode, so the dispatch still completes — the report is the signal). The parent fails the case on any report; a canary case re-enables the feature against the same placeholder and requires the report, so the harness cannot pass vacuously. The child process is needed because MetalValidation(1) reads the variables before the first Metal device is created, and because the instrumentation lowers each PSO’s thread cap (the reason make test-validate is a diagnostic, not the default gate) — the cases dispatch 256-thread groups for the reduction kernels. The prefill fixtures bound only the four output scratch allocations to those eight simdgroups (288 floats each instead of the production capacity of 1056); otherwise the validation layer’s extra memory exceeds a 32 KiB device limit. All indexing, arithmetic, optional-buffer guards and shader instrumentation remain intact. A separate GPU test compares the bounded scratch bit-for-bit with unmodified production IR, and both prefill widths have actual enabled-sink invalid-read canaries. The report detector also recognizes the newer Invalid metal usage: diagnostic family without matching validation-enabled banners.

The structural buffer-binding lint resolves the literal zipped NAME variants used by conditional-buffer families. It checks each exported row’s own buffer set, so a dynamic decode requires state while its by-value sibling does not. Parser regressions run with every lint invocation; unsupported conditional syntax fails the check instead of dropping a required binding.

Coverage

make coverage (or ./scripts/coverage.sh) produces an HTML report at target/llvm-cov/html/index.html; ./scripts/coverage.sh summary prints the per-file table CI emits. Per-crate floors live in .github/configs/codecov.yml:

Crate	Floor
`wh-iron-macros`	92%
`wh-iron-codegen` / `wh-iron-core`	90%
`wh-iron-runtime`	85%
`wh-iron-cli`	80%
`wh-iron-std`	line-coverage exempt — gated by bench-correctness instead
`wh-iron` (facade)	excluded

wh-iron-std’s iron/ and mlx/ kernel-body files are excluded from the line-coverage denominator: the #[kernel] proc-macro consumes the body at compile time, the Rust body never executes, so line coverage on them is structurally meaningless. Their correctness is gated by GPU correctness tests and bench equivalence instead — not by line coverage.

⚠️ Gaps in the test infrastructure

These are the holes a bug can slip through. Know them; close them when you can.

⚠️ A wrong kernel can pass every check except a GPU correctness test

A kernel that emits an empty body — from an inner macro_rules! or from a codegen pass dropping a loop body (see Developing → kernel-authoring hazards) — produces all-zeros output. That output:

passes xcrun metal — an empty body is valid MSL;
passes iron build --emit smoke — same reason;
passes MSL-snapshot drift checks — the snapshot just pins the wrong-but-stable empty body;
passes a loose integration test if its tolerance absorbs the noise.

It fails only when actual GPU output is compared to an expected value. That is the GPU correctness test, and nothing else. This is exactly how a family of quantized-gather kernels shipped silently broken until a correctness test was added. Do not rely on the smoke build or snapshots to catch a broken kernel.

⚠️ Not every kernel has a GPU correctness test yet

Coverage of crates/wh-iron-std/tests/ is incomplete — some kernels have a bench row but no correctness test, and some have neither. A kernel with no correctness test has no automated proof it computes the right answer. When you touch such a kernel, add the test; when you add a kernel, add it in the same commit.

⚠️ Perf numbers can be harness artifacts

A bench number is only meaningful if the harness measures the kernel and not its own overhead. A latency that doesn’t scale with input size is the tell. See Developing → kernel-authoring hazards (“too flat to be physical”) for the resident-buffer and GPU-clock-warmup fixes.