Rapid Software Testing in Iron

This guide applies Rapid Software Testing (RST) to Iron’s kernel DSL, compiler, code generators, runtimes, and GPU kernels. It is written so a maintainer or coding agent can enter a change with little project history and still produce reviewable evidence.

RST is an investigation method, not a synonym for adding cases. A good session names the risk, the oracle, the evidence, and the limits of the investigation.

Agent contract

1. Fetch the shared branch and record the exact commit under test.
2. State the affected backend and whether target hardware is available.
3. Trace both code reachability and test reachability.
4. Build a thin coverage model around the diff.
5. Name an oracle for each risk before implementing the check.
6. Validate new test tools against a known-bad case.
7. Keep static findings separate from executed failures.
8. Establish correctness before performance.
9. Report skips, unsupported capabilities, and uncovered backends.
10. Name why the session stopped. A green command alone is not enough.

Context intake

Copy and complete this block before a change-triggered or bug-hunt session:

mission:                 # What decision will this evidence support?
who_matters:             # Kernel author, Butter maintainer, backend user
change_under_test:       # Branch, commit, files, operation family
workspace_freshness:     # fetch result and relation to origin/dev
layer:                   # DSL, IR, pass, codegen, runtime, kernel, registry
backend:                 # Metal, CUDA, HIP, Vulkan, or backend-neutral
hardware_and_toolchain:  # Device, OS, driver, compiler versions
consumer:                # Butter path or another actual caller
constraints:             # Hardware, time, permissions, CI, memory
oracles_available:       # CPU reference, snapshot, ABI, round trip, receipt
testability_limits:      # State or behavior not controllable/observable
out_of_scope:            # Explicit omissions and why
danger:                  # GPU pin, OOB, host OOM, destructive generated output

If the change is paired with Butter, record the exact Butter commit and the Iron revision it resolves. An isolated Iron pass does not prove the consumer binds or dispatches the new contract.

Choose the session shape that matches the mission:

Session	Use it when
Change-triggered	A diff or PR defines the risk surface; this is the default development session
Bug hunt	A symptom exists and the work is reproduction, reduction, boundary discovery, and cause isolation
Coverage sweep	The question is breadth across a named compiler, kernel, backend, or consumer model
Nondeterministic	Runs disagree and the distribution, environment, ordering, or cache state must be characterized

Build the coverage model

Coverage is extent relative to this model, not the number of tests run. The table adapts SFDIPOT (Structure, Function, Data, Interfaces, Platform, Operations, and Time) to Iron’s source-to-device pipeline.

Area	Questions
DSL surface	Which types, operations, attributes, constexprs, and variants can express the change?
IR and passes	Which values, blocks, effects, aliases, and ordering constraints can be changed or dropped?
Code generation	Which backend emitters, dtypes, vector widths, layouts, and special values differ?
Registration	Is the kernel or test in the inventory, emitted artifact, and selected filter?
Runtime	Are buffers bound by the correct names, offsets, mutability, alignment, and lifetime?
Launch	Do grid, threadgroup, shared memory, stream, and graph contracts match the kernel?
Arithmetic	Which reductions, rounding points, accumulation dtypes, codecs, and state updates define correctness?
Consumer	Is the operation called by Butter with the tested shapes and feature/backend selection?
Time	What changes at cold compile, warm replay, boundary shapes, concurrency, and long-running state?

For every coverage claim, ask two separate reachability questions:

1. Does this test execute in the command or CI job being cited?
2. Does this kernel or code path execute in the consuming engine?

A function named like a test is not proof that it is registered. A compiled kernel beside used siblings is not proof that it has a caller.

Use these quality lenses to find risks the diff does not advertise: capability, reliability, usability, security, scalability, performance, installability, compatibility, supportability, testability, maintainability, and portability. They are guidewords, not a completion checklist.

Choose techniques that can surprise the implementation: function, domain boundaries, stress, flow over time, scenario, claims, user, risk, and automated high-volume checks. A happy-path compile is a demonstration, not a strong kernel or compiler investigation.

Rank risks by change type

Change	High-value risks
DSL or macro	Valid Rust lowers to an empty/wrong body, variant omitted, output mutability lost, unsupported syntax silently ignored
IR pass	Entry block skipped, nested block skipped, effect reordered, loop header survives body removal, nondeterministic iteration changes source
Backend emitter	Dtype or special value differs, alignment assumption is false, unsupported feature accepted, target source compiles but computes incorrectly
Runtime binding	Name/index/offset mismatch, tensor view offset ignored, byte count overflows, lifetime outlives context, graph captures unstable pointers
Kernel arithmetic	Wrong reduction topology, scale axis, packed-code behavior, ragged tail, tie rule, recurrence state, or output rounding point
Launch geometry	Wrong thread/threadgroup interpretation, insufficient shared memory, OOB access, deadlock, occupancy cliff
Optimization	Isolated win is harness overhead, candidate path did not execute, register spill or lost parallelism erases traffic savings

Write charters as hunts. “Run the CUDA corpus” is a command. “Find a packed tail or view-offset case where CUDA disagrees with the independent codec” is a charter.

Oracles and what they prove

Oracle	Proves	Does not prove
Parser/IR unit assertion	Intended internal structure exists	Backend output or execution
Generated-source snapshot	Text drift is visible and reviewable	Numerical correctness
Target compiler	Source is accepted	Kernel body is useful or correct
Naive CPU result	Arithmetic for the tested domain	Model-level composition or performance
Codec/format round trip	Packed layout and special-code contract	Matrix kernel launch and accumulation
ABI/binding assertion	Declared names, types, and mutability	Butter supplies the right buffers
Real-device GPU comparison	Backend execution matches the oracle	Other hardware or untested shapes
Butter integration	Consumer composition works for that model/path	Independent kernel arithmetic for every case
Benchmark receipt	Performance under one controlled regime	Correctness or transfer to another regime

Use exact comparison when the contract is exact. For numerical paths, report a tuple that can expose magnitude and localized errors: max absolute error, max relative error, cosine similarity, and scale/norm behavior. Cosine alone is blind to uniform scaling.

Calibrate tolerances from the intended dtype, storage round trip, and algorithm. Do not widen a threshold merely to make a candidate pass.

Validate the checker

Every new sweep, linter, oracle, or harness must fail a known positive before a clean result is trusted. Useful Iron mutations include:

• scale output uniformly while preserving cosine;
• swap two buffer names or use a nonzero tensor-view offset;
• remove a store or leave a loop body empty;
• alter one packed code or block scale;
• launch the wrong threadgroup size;
• poison the rejected recurrence state;
• remove a registry entry or exclude the test filter;
• force an unsupported capability skip.

Restore the mutation before committing and record the assertion or guard that caught it.

Evidence required by change class

New or changed kernel

Minimum evidence:

1. dispatch/layout contract in the kernel documentation;
2. independent CPU oracle with boundary and adversarial inputs;
3. dtype/format/shape cases relevant to the shipped caller;
4. real-device execution on every claimed production backend;
5. known-bad mutation demonstrating oracle sensitivity;
6. benchmark at the production shape if performance is claimed;
7. Butter integration or explicit statement that consumer wiring is pending.

The correctness test and benchmark belong with the kernel in the same change.

DSL, IR, or codegen change

Minimum evidence:

1. focused unit test for the new transformation;
2. negative or compile-fail case when misuse is possible;
3. snapshot for each newly distinct emission path;
4. full traversal check for entry and nested blocks;
5. generated-source compile check;
6. real-device round trip through a probe or production kernel;
7. deterministic output when map/set iteration can affect emission.

Snapshots pin output; they cannot declare it correct.

Runtime or backend change

Minimum evidence:

1. capability and unsupported-path behavior;
2. binding name, dtype, byte length, alignment, and view-offset cases;
3. checked size arithmetic before byte conversion;
4. context, stream, event, graph, and buffer lifetime tests as applicable;
5. target-hardware smoke plus numerical corpus;
6. repeated or concurrent execution when cache/lifetime state is shared;
7. consuming-engine path proof.

Optimization

Correctness gates precede timing. A kernel candidate must preserve the defined arithmetic contract; a compiler candidate must preserve all affected emitted paths; a runtime candidate must preserve lifetime and ordering.

Measure the complete operation at the exact production dtype and shape. Then measure the consuming model. Register count, occupancy, traffic, and launch count explain a result; latency decides it.

Test layers and commands

# Workspace and static gates
make fmt-check
make clippy
make test
make typos

# One GPU correctness test
cargo test -p wh-iron-std --test <kernel>_gpu_correctness

# Declarative kernel corpus
make iron test

# Emit and compile all generated Metal kernels
iron build --emit all -o /tmp/iron-smoke

# One ignored performance companion
cargo test --release -p wh-iron-std \
  --test <kernel>_gpu_correctness -- --ignored --nocapture

Backend-specific corpora must run on matching hardware. A general CI pass that skips hardware jobs is honest only when the skip is reported as uncovered.

Use Metal shader validation as a targeted OOB diagnostic, not a universal gate:

make test-validate

Validation instrumentation changes resource limits and can false-fail kernels near the device threadgroup ceiling.

Campaign lessons to preserve

The 2026 complete-inventory audit established several repository-specific rules that are now part of the method:

• Keep file classifications in docs/test-inventory-status.tsv. Regeneration must fail on missing, duplicate, invalid, or stale rows; a renderer that silently resets reviewed rows to queued is itself a reporting defect. The Lint lane now runs make check-test-inventory, so a PR that adds a test or benchmark file must add its status row and regenerate docs/test-inventory.md in the same PR.
• A hardware test may skip by default, but deliberate selection must fail closed when the device, feature, fixture, or threshold is absent. A silent early return is not skip accounting and cannot support a coverage claim.
• The default workspace does not establish CUDA, HIP, or Vulkan execution. Feature-gated compilation and a matching real-device lane are separate evidence obligations.
• Generated-resource evidence must cover every shipped input to the consumer, including live-compile source as well as the metallib and manifest. Record the exact dependency revision used to generate or consume those artifacts.
• C ABI types are target properties. In particular, do not assume Rust c_char is signed or that as u8 expresses a byte-preserving contract; use the native byte representation and exercise the helper independently.
• A test label can create jobs only after prerequisite jobs finish. Inspect the workflow graph, requested runner labels, assignment, and queue state before concluding that the label did not trigger CI.
• Do not share a Cargo target directory between audit worktrees that have the same package identities. Cross-worktree fingerprint reuse can present stale binaries as fresh execution evidence.
• Aggregate numerical metrics require finite-value checks first. A maximum or cosine fold can otherwise conceal NaN or infinity instead of rejecting it.
• A timing-only declaration remains blocked until it has a correctness owner, an explicit campaign gate, and a threshold or retained baseline receipt.

Findings from the 2026 inventory audit

The generated test inventory now carries a fail-closed boundary for every blocked file. The audited tree contains 159 strong and 275 blocked files. Those 275 rows are fully accounted for by 266 performance campaigns without pinned baselines, 8 non-local backend-device campaigns, and 1 real-model dump fixture. All 248 blocked files under crates/wh-iron-std/src contain benchmark declarations; they are not hidden ordinary unit-test gaps.

The final repair sessions established these product and harness contracts:

• iron init scaffold reachability is exercised end to end, while the memory-heavy Linux lane is assigned to a runner with an honest budget.
• Typed output bindings reject misaligned resident buffers before dispatch.
• A missing unresolved host input is an error, while an explicitly supplied zero-length buffer remains valid when runtime scalar metadata proves the logical tensor is empty. Treating both cases as the same invariant broke the valid zero-token MoE plan and was rejected by the full Metal suite.
• Shape products and row-major strides use checked arithmetic; dynamic output capacity, short strided metadata, and dimensions wider than the u32 ABI fail explicitly instead of panicking, wrapping, or dispatching ambiguous state.
• The two resident chained-SDPA comparisons documented as Apple8+ are invoked explicitly on the known M2 CI runner with nextest’s ignored-only selection, while remaining ignored on Apple7-class machines.

The excluded BM64 ragged-MoE binary was also replayed directly: deterministic f32 and f16 ragged cases passed with no bad rows. Its remaining ignored test requires /tmp/bm64dump captured from the real model, so that file remains the single external-fixture boundary rather than being promoted on synthetic evidence alone.

Performance receipt

commit:                 # Iron and consuming Butter revisions
backend_and_hardware:   # device, OS, driver/compiler
kernel_and_shape:       # exact op, dtype, dimensions, launch geometry
correctness_oracle:     # result established before timing
path_proof:             # trace/marker proving candidate selection
buffers:                # resident/uploaded, bytes moved, view offsets
warmup_and_trials:      # warmup, iterations, process trials
timer:                  # GPU/event or justified end-to-end clock
baseline:               # current shippable path, same harness
result:                 # latency, dispersion, throughput/utilization
resources:              # registers, spills, shared memory, occupancy if relevant
consumer_result:        # end-to-end model effect or explicit untested status

A flat latency curve across materially different sizes is a harness warning. Confirm resident inputs, clock warmup, synchronization boundaries, and actual candidate execution before publishing it.

Reporting

Tag every finding:

• observed: executed and witnessed;
• static: code would behave this way if reached;
• hypothesis: plausible, not established;
• refuted: evidence rejected the concern or optimization.

End with:

1. the product story: what Iron does and what is wrong;
2. the testing story: commands, oracles, mutations, hardware, and exclusions;
3. the quality-of-testing story: weak checks, skips, untested consumers, and how much confidence the evidence supports.

Stop because the mission is answered, the time box is exhausted, evidence has flatlined under meaningful variation, the session is blocked, or a new charter is now more valuable. Name the decision and the next useful charter.

Agent handoff template

### Mission and exact revisions

### Layer, backend, hardware, and consumer

### Coverage model and exclusions

### Ranked risks

| Risk | Oracle | Evidence | Status |
|---|---|---|---|

### Known-bad mutation

### Commands and receipts

### Findings

Tag each observed, static, hypothesis, or refuted.

### Quality of testing

### Stopping decision and next charter

Related guides

• Testing for existing layers, CI coverage, and known gaps.
• Kernel style guide for the required kernel/test/bench shape.
• Architecture for source-to-runtime ownership.
• Rust efficiency for implementation and optimization rules.
• Qwen3.8 kernel TDD for a worked hybrid path.