Rapid Software Testing in Butter

This guide turns Rapid Software Testing (RST) into a working method for Butter. It is written for maintainers and coding agents. Use it when changing model execution, loading, caching, scheduling, serving, or a hardware backend.

RST is not a larger test checklist. It is a way to investigate risk, state what would reveal a problem, and report how strong the resulting evidence is.

Agent contract

Before changing code or claiming a result:

1. Fetch the shared branch and record the commit under test.
2. State the mission, affected user, danger, constraints, and exclusions.
3. Build a coverage model from the current code and call graph.
4. Name an oracle for each important risk before writing the check.
5. Validate any new harness against a known-bad case.
6. Distinguish static analysis from executed behavior.
7. Confirm that the test ran, the backend path ran, and the code is reachable.
8. Establish correctness before measuring speed.
9. Report uncovered areas and weak oracles.
10. Name the stopping reason. “Tests passed” is not a stopping reason.

Evidence overrides expectation. If the code, trace, or output contradicts the task description, report the contradiction with the evidence.

Start with context

Copy this block into the issue, session note, or working document. Unknowns remain risks; do not fill them with guesses.

mission:                 # What decision will this testing inform?
who_matters:             # User, operator, or maintainer who bears the failure
change_under_test:       # Branch, commit, diff, model, and backend
workspace_freshness:     # fetch result and relation to origin/dev
execution_path:          # Swift or Rust; Metal, CUDA, or Vulkan
checkpoint:              # repository/path, revision, format, quantization
hardware:                # device, memory, OS, driver/toolchain
constraints:             # time, model access, disk, memory, CI availability
oracles_available:       # Existing references, invariants, traces, receipts
testability_limits:      # State or behavior that cannot yet be controlled/seen
out_of_scope:            # Explicit omissions and why
danger:                  # OOM, GPU pin, corrupt cache, destructive operation

For paired Butter/Iron work, record both revisions. A Butter result against a different Iron revision does not validate the proposed pair.

Choose the session shape that matches the mission:

Session	Use it when
Change-triggered	A diff or PR defines the risk surface; this is the default development session
Bug hunt	A symptom exists and the work is reproduction, reduction, boundary discovery, and cause isolation
Coverage sweep	The question is breadth across a named product model
Nondeterministic	Runs disagree and the distribution, environment, or ordering must be characterized

Build the Butter coverage model

Coverage is extent relative to a model, never a test count. The table below is Butter’s SFDIPOT model: Structure, Function, Data, Interfaces, Platform, Operations, and Time. Start with the parts touched by the change:

Area	Questions
Structure	Which Swift or Rust crates, model layers, ops, generated wrappers, and Iron kernels participate?
Function	Does the change affect load, tokenize, prefill, decode, sample, stream, abort, reset, or unload?
Data	Which checkpoint layouts, dtypes, shapes, prompts, cache depths, and malformed inputs matter?
Interfaces	Which model-to-op, op-to-device, Butter-to-Iron, CLI, HTTP, or FFI contracts change?
Platform	Which hardware, backend, compiler, driver, and memory model must execute it?
Operations	Does it work cold and warm, single-stream and concurrent, interrupted and restarted?
Time	What changes with prompt length, decode depth, ordering, cancellation, graph capture, or cache reuse?

A diff-scoped session may use a thin model. Say that it is thin and list the areas not covered.

Use these quality lenses to find risks the diff does not advertise: capability, reliability, usability, security, scalability, performance, installability, compatibility, supportability, testability, maintainability, and portability. They are guidewords, not boxes that must all be checked.

Choose techniques that can surprise the implementation: function, domain boundaries, stress, flow over time, scenario, claims, user, risk, and automated high-volume checks. A session that only repeats intended use is a demonstration, not a strong investigation.

Rank risks before generating cases

Use the change type to seed investigation, then refine it from the actual diff.

Change	High-value risks
Loader or quantization	Wrong tensor selected, packed bytes reinterpreted, scale axis wrong, partial matrix tested, unsupported format silently accepted
Model graph	Wrong operation order, wrong layer split, precision boundary drift, prompt/decode disagreement, unused checkpoint tensor
KV or recurrent state	Off-by-one position, rejected state published, reset leakage, sliding-window overwrite, cache sized from the wrong logical limit
Scheduler or serving	Starvation, cancellation after admission, cross-request contamination, resource leak, queue order drift, hidden serialization
Backend dispatch	Stub or fallback executed, wrong grid, wrong binding offset, unsupported capability treated as success, stale generated kernel
Optimization	Candidate path did not execute, fingerprint changed, memory moved elsewhere, warmup artifact, microbench win lost end to end

Every charter should name a problem to hunt. “Exercise Qwen decode” is a feature tour. “Find a state publication error after a partial speculative accept” is a test charter.

Choose an oracle per risk

An oracle is how a problem will be recognized and where that expectation came from. Use the narrowest oracle that can detect the named failure.

Oracle	Appropriate use	Limitation
Exact bytes or bits	Codec round trips, layout, deterministic state, known exact paths	May reject an intentional numerical change
Naive CPU calculation	One operation or small model component	Must be independent of the implementation under test
Tensor error tuple	Numeric kernels: max absolute, max relative, cosine, norm/scale	Aggregate similarity alone can hide localized or scale errors
Fixed-prompt token fingerprint	Autoregressive trajectory and reduction-order stability	Covers only the named model, prompt, settings, and depth
State invariant	KV length, GDN/SSM state, slot ownership, rollback/commit	Requires state to be observable at the boundary under test
Lifecycle invariant	Allocation, cancellation, cleanup, fairness, queue progress	A green output does not establish lifecycle correctness
Coherent output	First-light smoke only	Not a numerical or state oracle
Performance receipt	Regression or optimization decision	Valid only after correctness and harness validation

Do not invent expected tokens from memory. Record the source of a fingerprint or regenerate it from a trusted baseline in the same session.

Test the test

Before trusting a green checker, demonstrate that it fails on a known defect. Useful mutations include:

• change one packed nibble or scale;
• perturb output magnitude while preserving direction;
• poison a rejected state row;
• swap two binding offsets;
• force the fallback path;
• return an empty token stream;
• disable the feature or environment variable that enables the test.

Restore the mutation before committing. The receipt should state what was mutated and which assertion failed.

Execute the evidence ladder

Run only the relevant rungs, but do not imply that an unrun rung passed.

1. Static reachability: trace the call from public entry point through the selected backend. Mark findings static until executed.
2. Focused host checks: parser, shape, configuration, cache, sampler, and state-machine tests without a large checkpoint.
3. Component oracle: execute the changed op or layer against an independent reference and exercise boundary shapes.
4. Model composition: run prefill and multi-token decode with the real checkpoint and fixed settings.
5. State and lifecycle: reset, reuse, cancellation, partial commit, cache boundary, and concurrent-session checks as applicable.
6. Target hardware: execute each claimed backend on its real device. A Metal pass says nothing about CUDA or Vulkan execution.
7. Performance: compare a correct baseline and candidate with a validated harness and preserve both receipts.
8. Soak or stress: use when the risk involves time, accumulation, memory, concurrency, or nondeterminism.

Confirm execution, not merely success

Environment-gated checkpoint tests and unsupported-device skips can exit green. For each hardware result, record:

• the test name and command;
• whether it ran, skipped, or fell back;
• the resolved device and backend;
• the checkpoint revision and feature flags;
• a trace, counter, or path marker proving the candidate path executed.

A compiled file is not necessarily registered. A registered test is not necessarily selected. A selected test is not necessarily executed.

Project commands

Swift and Metal:

make test-unit
make test-unit-serial
make test-integration
make test-stress

Use the serial target for mutation checks, shared-state investigations, and memory-constrained sessions. Use make test-stress after scheduling or dispatch changes.

Rust, backend-neutral:

make test-rust
make test-rust-cli

Focused Rust example:

cd rust
RUST_TEST_THREADS=1 cargo test -p wh-butter-models qwen35

CUDA tests require the real feature and target hardware:

cd rust
RUST_TEST_THREADS=1 cargo test -p wh-butter-cuda \
  --features cuda --test qwen35_units -- --nocapture

Qwen3.8 NVFP4 CUDA receipts and component oracles additionally need -- --ignored plus BUTTER_QWEN35_NVFP4 pointing at the checkpoint directory. An unset fixture or missing NVIDIA device must fail after that explicit selection; it is not a green skip.

Checkpoint tests may require documented environment variables. An unset variable that causes a skip is not evidence about the model.

Lessons from the complete inventory audit

The 2026 full-inventory audit added several project-specific rules:

• Keep the per-file status map exhaustive and versioned. Regeneration must fail on a missing, duplicate, invalid, or stale classification; otherwise a newly added file silently returns to queued while the document looks complete.
• A silent early return is not a skip. Hardware, checkpoint, and performance campaigns must be explicitly ignored or enabled in the default lane, and must fail closed after deliberate selection when a prerequisite is absent.
• Feature-free workspace success is not backend evidence. Compile and execute the feature-specific target on its claimed platform, record the device, and treat a missing system loader or accelerator as a precise blocked boundary.
• A rejection test can still be hardware-dependent when it allocates a device before validation. Refactor validation to a host-only owner or classify the test as a hardware campaign.
• Generated artifacts must include sources consumed by live compilation, not only checked-in wrappers. Dependency pins must name the exact audited Iron revision, and FFI byte conversions must preserve platform-dependent c_char bit patterns.
• Check finiteness before aggregate numerical comparisons. NaN can make a floating-point aggregate appear harmless, while cosine-only checks can miss magnitude drift.
• Performance tests need an always-on correctness owner and either a committed threshold or a preserved baseline receipt. Print-only microbenchmarks are opt-in diagnostics, not regression gates.
• Production-shape naive CPU oracles can consume seven to twelve minutes per case. Keep a smaller independent correctness partition active and place the production sweep on an explicit hardware/time-budget lane.
• Fixture paths must be portable. Do not rely on a developer home directory or transient /var/folders data. When an opt-in fixture is requested, absence is a failure; when it is not requested, report an explicit skip.
• Test filters can miss suites whose filenames or generated names differ from the filter. Record executed and ignored counts, not merely the command exit.
• Catch-all dispatch fallbacks, metadata-only registration, and successful compilation can all create false-green evidence. Require a path marker, execution receipt, or a known-bad mutation that proves the intended owner ran.
• Use a separate Cargo target directory for each audit worktree. Shared target artifacts can make a stale dependency or feature build look current.
• CI label jobs appear only after their prerequisite jobs and may then wait for a matching self-hosted runner. Inspect the workflow graph, requested labels, and runner assignment before concluding that a test label failed to trigger.
• A cancelled self-hosted job can leave GitHub reporting an idle online runner while its broker listener remains backed off. Confirm the queue and runner diagnostic log; restart only the idle runner service when assignment has genuinely stopped.
• Budget the complete Rust workspace for evidence and cleanup, not merely the last test line. This audit required a 90-minute job budget after a green run reached 59:59 and was cancelled during final bookkeeping. Keep full-workspace artifacts in a per-run temporary target and clean it with always().
• When Cargo’s private Iron URL is rewritten to an authenticated checkout, that checkout must contain the exact revision pinned in rust/Cargo.toml. Prove this on a cold runner: a warm Cargo object cache can conceal a stale checkout SHA and create a false green.
• Resource/setup failures are not product failures. Preserve the distinction between an asserted test failure and a compiler process killed by the runner.

Findings from the 2026 inventory audit

The completed source-file inventory is generated in test-inventory.md. Its status ledger is fail-closed: every blocked row must name one of four evidence boundaries, and stale or missing rows make the renderer fail. The audited tree contains 306 strong and 215 blocked files. The blocked set is fully accounted for by 118 checkpoint or captured-fixture rows, 88 non-local backend-device rows, 8 performance rows without pinned baselines, and 1 two-device distributed row.

The final Apple GPU replay promoted eight previously blocked files:

• DeepSeekV4 prefill fallback passed four MPP comparisons and the same four tests with BUTTER_FORCE_NO_MPP=1.
• NemotronH batched prefill passed three dense, attention, and NVFP4 MoE comparisons against sequential decode paths.
• The focused Ops replay executed 40 tests and explicitly skipped 34 checkpoint/production-shape cases. Gated-delta WY, expert-gather levers, fused attention, and last-row prefill had no remaining skips.
• Explicit production-shape selection then passed Laguna MPP GEMM at seven real XS shapes and NAX2 at eight shapes against CPU references.

The replay also exposed a harness defect before any product assertion ran: local make regenerate-kernels accepted an arbitrary sibling Iron checkout and could generate Swift wrappers with a different ABI than Butter’s pinned CI revision. Butter now records that revision in .iron-revision, rejects a mismatch before deleting generated artifacts, and requires an explicit IRON_ALLOW_UNPINNED=1 override for intentional cross-revision kernel work. Treat this as a setup RED, not a product RED.

Performance sessions

Keep four clocks distinct:

• cold load and compile;
• time to first token;
• prefill tokens per second;
• post-first-token decode tokens per second.

For serving, also report per-stream and aggregate throughput. For speculative decode, include proposed tokens, accepted drafts, committed tokens, target forwards, and each phase time. Whole-request throughput is not decode speed.

A performance receipt includes:

commit:                 # Butter and Iron revisions
hardware:               # device, memory, power/clock condition
software:               # OS, driver, compiler/toolchain
checkpoint:             # exact revision and format
mode:                   # prefill/decode, graph/eager, speculative/plain
shape:                  # prompt, output, context, batch/concurrency
correctness_oracle:     # result established before timing
warmup_and_trials:      # warmup count, trial count, summary statistic
baseline:               # same-binary or otherwise justified comparison
result:                 # median plus dispersion and raw receipt location
path_proof:             # evidence that the candidate executed

Retain negative results. They prevent the same plausible but losing idea from being repeated.

Reporting and stopping

End the session with three short stories:

1. Product: what was observed and what problems remain.
2. Testing: what was run, with which oracles, mutations, and environments.
3. Quality of testing: what was shallow, skipped, blocked, noisy, or based only on static analysis.

Use one of these finding labels:

• observed: the behavior was executed and witnessed;
• static: the code would produce the stated behavior if the path is reached;
• hypothesis: plausible and worth a charter, not yet established;
• refuted: the proposed risk or optimization lost to evidence.

Reasonable stopping decisions include mission accomplished, time box consumed, blocked by missing hardware or checkpoint, flatline after meaningful variation, or change of charter. State which one applies and what would justify resuming.

Handoff template for an agent

### Mission

### Context and exact revisions

### Coverage model and exclusions

### Ranked risks

| Risk | Oracle | Test or observation | Status |
|---|---|---|---|

### Known-bad mutation used to validate the harness

### Commands and receipts

### Findings

Tag each finding observed, static, hypothesis, or refuted.

### Quality of testing

### Stopping decision and next charter

Related guides

• Testing for the existing Swift and model test layout.
• Benchmarking for Butter benchmark fields and methods.
• Architecture for ownership and request lifecycle.
• Rust efficiency for optimization rules and evidence.
• Qwen3.8 architecture for a worked hybrid-state and speculative path.