Rust Efficiency in Butter

This guide defines how to improve the Rust engine without trading away model correctness, backend portability, or maintainability. It covers host/runtime work in Butter and the boundary where kernel work moves to Iron.

Agent quick contract

1. Profile the production path before editing it.
2. Record a correct baseline from the same commit family and hardware.
3. State whether the cost is host, transfer, dispatch, kernel, or memory.
4. Optimize the measured bottleneck, not a proxy such as launch count.
5. Preserve precision, reduction, layout, and state-publication contracts.
6. Keep allocations and synchronization out of steady-state decode.
7. Prove that the candidate path executed.
8. Re-run component, trajectory, and end-to-end performance gates.
9. Keep the change only when the production clock improves outside noise.
10. Record losing experiments and why they lost.

Read Rapid Software Testing in Butter before making a performance claim.

Know which layer owns the cost

Cost	Butter action	Iron action
Model graph or duplicated operation	Change model/ops composition	None unless a new primitive is needed
Allocation, cache, queue, lock, or host sampling	Change Butter runtime	None
Host/device copy or synchronization	Change device/session ownership	Add a device primitive only if required
Wrong dispatch or backend selection	Fix semantic op dispatch	Fix runtime launch contract if it is shared
Generated kernel arithmetic or layout	Specify the semantic contract	Change DSL, codegen, or kernel
Vendor matrix tactic or graph lifetime	Select and retain it in the backend	Expose only the narrow runtime capability needed

Do not hide an engine policy inside a kernel. Do not duplicate kernel math in Butter when Iron is the owner.

Measure the right path

Prefill and decode are different workloads:

• Prefill has many rows, can amortize dispatch, and often benefits from matrix-oriented batching and chunking.
• Decode usually has one row, repeatedly reads weights, mutates KV or recurrent state, and is sensitive to synchronization and graph stability.
• Speculative verify is a small fixed-width batch with transactional state. It is neither ordinary prefill nor scalar decode.

Report the production clock that consumes the change. A faster isolated op does not establish faster model decode. A lower dispatch count does not establish a faster GPU path.

Host hot-path rules

Allocate during construction, not replay

Prefer:

• Vec::with_capacity when cardinality is known;
• reusable session workspaces for stable tensor shapes;
• a measured SmallVec capacity for genuinely small bounded collections;
• stack bytes or typed scalars for tiny parameter uploads;
• buffer pools keyed by size, dtype, device, and lifetime class.

Avoid:

• format!, String, Vec, or map construction per token;
• cloning large descriptor collections to satisfy an ownership shortcut;
• allocating outputs inside a CUDA graph capture or replay path;
• a global workspace shared by sessions with independent lifetimes.

Choose a SmallVec inline capacity from observed modal occupancy, not generous headroom. Oversized inline storage raises move and stack costs everywhere.

Use ownership to express lifetime

• Use Rc only when the value cannot cross threads; otherwise use Arc.
• Keep graph-captured buffers owned by the graph/session arena that fixes their addresses.
• Keep cache slots, recurrent state, and abort state request-scoped.
• Do not store raw device pointers beyond the lifetime of their owning buffer or context.
• Bind registration or external-library handles to both resource and context lifetime. A detached cached handle is unsafe.

Keep lock scope narrow

parking_lot is appropriate for short internal critical sections already used in this workspace. Changing lock type does not repair a bad ownership model.

• Never hold a model, scheduler, or cache lock across GPU synchronization.
• Keep compilation and descriptor-cache single-flight work outside request state where possible.
• Measure contention with concurrent requests; uncontended microbenchmarks do not prove serving behavior.

Pick data structures by contract

• Pre-size maps and sets in analysis or setup code when an upper bound is available.
• Fast internal hashing is suitable for trusted numeric IDs and fingerprints.
• Preserve ordered maps where iteration order affects generated source, bindings, cache keys, logs, or deterministic receipts.
• Compute a stable content fingerprint before placing it in a fast in-process map. Map hashing and persistent identity are different jobs.

Make invariants explicit

Use names and types to make state transitions clear. Prefer let-else for early rejection and expect("specific invariant") at truly infallible sites. Comments should explain constraints, numerical boundaries, lifetime rules, or benchmark methodology. Do not paraphrase the next statement.

Device-resident decode

The steady-state goal is one prepared request state and a bounded sequence of device work per token.

• Keep token, position, cache length, and reusable descriptors resident when the backend contract allows it.
• Download the minimum decision value needed by the API. Greedy sampling does not need a full vocabulary transfer merely to obtain one token.
• Separate streaming or stop-condition synchronization from full-logit paths needed by probabilistic sampling.
• Reuse quantized activations only when every consumer expects the same exact representation.
• Preserve stable pointers and branch decisions across graph capture/replay.
• Resolve graph keys from every baked pointer, shape, dtype, tactic, and branch.
• Give independent sessions independent graph and workspace lifetimes.

Removing a synchronization can be numerically correct and still be below model noise. Retain it only when a production receipt shows value or when it repairs a separate correctness/liveness problem.

Numerical boundaries are part of the API

For autoregressive inference, a small component difference can change a later token. Before timing a candidate, establish:

1. local tensor error with max absolute, max relative, cosine, and scale/norm;
2. exact or documented-tolerance state parity for KV, GDN, SSM, and convolution;
3. fixed-prompt multi-token fingerprint or an explicitly approved non-exact mode;
4. reset, reuse, and partial-commit behavior where state is mutable.

Do not loosen a tolerance because a faster path nearly passes. Identify whether the reference, candidate, storage round-trip, reduction order, or dtype boundary defines the intended contract.

Packed low-precision formats include their legacy special-code behavior, scale-axis convention, rounding point, and output dtype. Matching the nominal format is insufficient.

Fusion and batching decisions

Favor vertical producer-to-consumer fusion when it removes an intermediate write/read and preserves the producer’s numerical boundary. Be skeptical of horizontal fusion between independent operations.

Before fusion, answer:

• Is an intermediate consumed immediately and only a small number of times?
• Are producer and consumer launch shapes compatible?
• Will live values raise registers or lower occupancy?
• Does the unfused baseline already share an encoder, stream, or graph?
• Does fusion change reduction order or the final rounding point?
• Does it remove actual traffic, or only a launch visible in a trace?

Batch only work with compatible numerical and lifetime contracts. Prefill batches, decode rows, speculative verify rows, and requests from different sessions may require different tactics.

Memory efficiency

Track logical and physical capacity separately:

• KV capacity comes from the cache policy, not automatically from maximum model context.
• Sliding-window layers should not allocate full-context physical storage.
• Draft, target, attention, and recurrent states have different lifetimes and should not inherit one another’s maximum allocation.
• Demand-sized session buffers need a tested growth and reuse policy.
• Unified-memory free-space numbers are not allocated-VRAM measurements.

When a full model does not fit safely, use a small synthetic checkpoint to measure per-layer and per-token scaling. Validate the synthetic instrument on a known allocation before extrapolating.

Build profiles

Butter intentionally keeps aggressive whole-program optimization in the bench profile:

[profile.bench]
inherits = "release"
lto = "fat"
codegen-units = 1
debug = true

Use the bench profile for runtime measurements. Do not move these settings into the general release profile without measuring downstream build cost. Do not compare a debug baseline to a bench candidate.

Optimization loop

1. Baseline: fixed commit, hardware, checkpoint, prompt/shape, settings, warmup, trials, and correctness receipt.
2. Profile: separate load, prefill, decode, sampling, transfers, synchronization, graph replay, and memory.
3. Hypothesis: name one mechanism and an expected bound on total impact.
4. Microgate: test the exact production dtype and shape. Reject quickly if the candidate cannot beat the current path outside noise.
5. Component gate: apply the independent numerical and state oracle.
6. Model gate: fixed multi-token trajectory, reset/reuse, and the affected concurrency or context rung.
7. Production clock: same-binary A/B where possible, multiple warm trials, median plus dispersion.
8. Decision: keep, revise, or record as a negative result.

The expected total gain must respect the measured phase share. Optimizing a one-percent phase cannot produce a ten-percent end-to-end win.

Review checklist

• The changed code is on the measured production path.
• Baseline and candidate use the same hardware, checkpoint, shape, and build profile.
• The candidate performs no steady-state allocation or avoidable host sync.
• Buffer, graph, and state lifetimes are explicit.
• Deterministic ordering is preserved where externally observable.
• Component numerical and state oracles pass.
• Multi-token trajectory and reset/reuse gates pass.
• Target hardware execution is proven, not skipped or emulated.
• End-to-end prefill/decode/serving clocks improve outside variance.
• Memory peak and concurrency behavior did not regress.
• Losing alternatives and test limitations are recorded.

Project commands

# Backend-neutral Rust tests
make test-rust

# CLI and serving control-plane tests
make test-rust-cli

# Build a measurement binary using the workspace bench profile
cd rust
cargo build --profile bench -p wh-butter-cli

Run backend-specific tests on the matching hardware and feature set. Use the model’s test/runbook for checkpoint environment variables and receipt fields.

Related guides

• Rapid Software Testing in Butter.
• Testing.
• Benchmarking.
• Performance.
• Architecture.
• Qwen3.8 architecture.