Architecture

Butter is the inference engine. Iron is the kernel and device layer beneath it. Butter has two maintained hosts: Swift for the Apple-native product and Rust for CUDA and portable GPU backends. The hosts share architectural contracts, but they do not share every implementation or capability.

Read this page in order if you are new to the project. It starts at the Butter/Iron boundary, follows one request through the engine, then explains the four execution graphs, state ownership, and the evidence required for a change.

1. The Butter and Iron boundary

application or serving client
              |
              v
Butter host
  model config and checkpoint interpretation
  tokenizer and chat template
  request admission and scheduling
  prefill, decode, sampling, and streaming
  KV and recurrent-state lifecycle
              |
              v
semantic operations
  projection, normalization, attention, recurrence,
  quantization, cache update, selection
              |
              v
Iron
  kernel math and layout contracts
  IR passes and target lowering
  launch geometry and backend runtime
              |
              v
Metal | CUDA | HIP/ROCm | Vulkan

The ownership rule is simple:

• Butter owns model semantics. It decides which tensors form a layer, which layer runs at a position, which state belongs to a request, and when state is committed or discarded.
• Iron owns device semantics. It defines what a kernel reads and writes, how dtypes and packed layouts are interpreted, and how the operation reaches a target GPU.
• The boundary is a contract, not an implementation detail. A kernel can be correct by itself and still be wrong in a model because Butter supplied the wrong shape, offset, stride, state version, or execution order. Production changes therefore need evidence on both sides.

2. The two Butter hosts

Host	Primary use	Kernel path	Current scope
Swift Butter	Apple-native applications on Mac, iPhone, and iPad	Precompiled Metal libraries and generated Swift bindings	Broad text, vision, and audio model catalog
Rust Butter	Serving and systems work on NVIDIA CUDA, plus portable backend development	Iron runtime plus backend-specific matrix and graph operations	Text, hybrid, MoE, quantized, speculative, and concurrent serving workloads

Rust also has Metal and Vulkan device crates. Iron executes HIP/ROCm kernels, but Butter does not yet expose a first-class ROCm host adapter. Backend support is admitted per model and checkpoint. Shared IR support alone is not a full-model readiness claim.

3. From kernel source to a running operation

The source pipeline is shared until target lowering. Compilation and loading then follow the backend contract.

Rust #[kernel] definition
          |
          v
Iron IR and backend-independent passes
          |
          +----------------+----------------+----------------+
          |                |                |                |
          v                v                v                v
         MSL            CUDA C++          HIP C++          SPIR-V
          |                |                |                |
          v                v                v                v
    metallib at build   NVRTC to PTX    hipRTC object    Vulkan module
          |                |                |                |
          +----------------+----------------+----------------+
                           |
                           v
                 Butter semantic operation

On Apple, the package ships a precompiled Metal library and generated Swift wrappers. On CUDA, the Rust device compiles emitted CUDA through NVRTC, loads the resulting module, and may select native matrix-library or AOT helpers for qualified shapes. HIP and Vulkan use their own runtime modules and narrower validated inventories.

Kernel identity matters. A useful result identifies the selected operation, backend, dtype, shape, generated artifact, and fallback route. A silent fallback is not equivalent evidence.

4. One request through the engine

Every request follows the same lifecycle even when its model graph differs.

model id or local checkpoint
          |
          v
resolve files and parse config
          |
          v
load immutable weights and build the model graph
          |
          v
tokenize and render the prompt
          |
          v
admit request-owned cache and workspace state
          |
          v
prefill prompt chunks
          |
          v
sample first token
          |
          v
decode one step or verify a speculative block
          |
          v
sample, stream, stop-check, repeat
          |
          v
publish usage and release request state

Load

The loader resolves a local directory or immutable model snapshot, parses config.json, validates tensor names and shapes, maps the checkpoint format, and constructs the layer schedule. Loader metadata is executable architecture: head counts, layer classes, quantization schemes, sliding-window rules, and tokenizer/template identity all affect runtime behavior.

Weights are shared and immutable after load. Request state is allocated later, under admission control, rather than being hidden inside the shared model.

Prefill

Prefill ingests many known prompt tokens. Its important dimension is token width, so it favors matrix-shaped operations, chunking, and compatible batches across requests. Chunk size trades prompt throughput against time available for ready decode work.

Prefill publishes state only after the chunk succeeds. A failed or cancelled chunk must not leave a partially visible KV cache or recurrent state.

Decode

Autoregressive decode consumes one newly chosen token per request. It is often limited by weight traffic and launch overhead rather than arithmetic. The hot path keeps intermediate tensors and selection on the device when the sampling contract allows it.

Independent requests may share a compatible model pass, but their logical position, cache, recurrent state, stop policy, and output stream remain independent. The scheduler is the sole owner of accelerator submission.

Finish

EOS, a stop sequence, cancellation, a token limit, or an error resolves the request. Cleanup has to release admission, cache leases, graph-stable storage, and any queued scheduler work exactly once.

5. The four execution graphs

There is no single universal transformer loop in Butter. The loader selects a graph from checkpoint metadata, and that graph owns its state contract.

Dense attention

token embedding
  -> repeat for each layer:
       normalize
       Q/K/V projections
       position transform
       append K/V
       attention
       output projection and residual
       normalize
       gate/up projections
       activation and down projection
       residual
  -> final normalize
  -> language-model head
  -> sampling

Attention state is a KV prefix. Decode appends one logical position; batched prefill appends a range. Physical cache layout is chosen for the reader kernel, not merely for convenient writes.

Some dense families (Gemma 3 / 4, GPT-OSS, Spark-X2.5) still use this graph but schedule sliding-window and full attention from layer_types. Sliding layers bound KV with .window eviction; full layers stay unbounded up to LoadOptions.maxContextLength (or the advertised max_position_embeddings). Spark also picks a per-layer RoPE recipe and uses exact erf-GELU (Ops.geluErf) in the MLP.

Hybrid attention and recurrence

Hybrid models interleave attention with GDN or SSM layers. Attention layers own KV state. Recurrent layers own matrices and convolution history. Both state types advance at the same logical token boundary.

layer schedule from model metadata
       |
       +-- attention layer -> KV read/write
       |
       +-- GDN layer       -> recurrent matrix + conv state
       |
       +-- SSM layer       -> SSM state + conv state
       |
       +-- shared dense or routed FFN

This distinction is critical during prompt chunking and speculative verification. Truncating KV length does not restore a recurrent matrix.

Mixture of experts

A routed layer adds selection and sparse expert execution:

hidden row
  -> router logits
  -> stable top-k experts and weights
  -> gather or group rows by expert
  -> expert gate/up/down work
  -> weighted reduction
  -> residual

The route includes capacity, ordering, accumulation precision, and empty-expert behavior. A fast expert kernel does not prove the router-to-reduction graph.

Speculative verification

A speculative path proposes several tokens, verifies them with the target model, commits the accepted prefix plus the target correction token, and discards the rejected tail.

committed target state
          |
          v
draft proposal block
          |
          v
target verification over all rows
          |
          v
acceptance plan
       /       \
      v         v
commit chosen   discard rejected
KV/recurrent    KV/recurrent rows
state           and draft state

Verification is a transaction. Target KV, recurrent state, token history, draft state, and public sequence length must resolve together. The target model remains the correctness authority even when draft arithmetic uses a different precision contract.

For the current Qwen3.8 terminology and tests, start with planning/qwen38-architecture-guide.md.

6. State and ownership

State	Owner	Lifetime	May be shared?
Model weights and immutable lookup tables	Loaded model	Model residency	Yes
Kernel modules, pipelines, and qualified plans	Device/runtime cache	Process or device lifetime	Yes, when the cache key is complete
KV cache	Request session	Prompt through completion	No mutable sharing
GDN/SSM and convolution state	Request session	Prompt through completion	No mutable sharing
Sampling and stop state	Request session	Completion	No
Activation workspace	Scheduler/session lease	One operation or captured graph	Only through explicit non-overlapping leases
Captured graph storage	Captured session/shape	Graph lifetime	Only when pointer stability and lane ownership are proven

Three rules prevent most state bugs:

1. Share immutable model data; isolate mutable sequence data.
2. Publish a new logical state only after all device writes that define it are ordered and complete.
3. Treat position, cache length, recurrent version, and token history as one commit boundary.

7. Prefill and decode are separate performance problems

Property	Prefill	Decode
Known token width	Many prompt tokens	One selected token per stream
Common limiting factor	Compute, tiling, chunk policy, batch formation	Weight traffic, launch count, synchronization, cache reads
Primary user metric	TTFT and prompt tok/s	Inter-token latency and tok/s per stream
Serving metric	Batched prompt throughput and queue delay	Per-stream latency plus aggregate output tok/s
Typical optimization	Wider matrix work and compatible batching	Exact small-row kernels, graph replay, device-resident state

Do not combine the two clocks into one headline without labeling it as whole-request throughput. A performance result records prompt tokens, output tokens, context depth, concurrency, sampling mode, warmup, repetitions, and whether the clock includes TTFT.

8. How a change becomes production evidence

Evidence is layered because each layer catches a different failure.

1. Pure logic and codegen: configuration parsing, shape arithmetic, IR passes, generated source, and selection predicates.
2. Kernel oracle: exact dtype, layout, offset, stride, tail, and production shape against an independent reference.
3. Model composition: checkpoint loading, layer schedule, eager/captured parity, token or declared numeric policy, and state progression.
4. Serving transaction: cancellation, concurrent sessions, cache ownership, speculative commit/abort, and cleanup.
5. Performance promotion: same workload and binary, warm repetitions, component and end-to-end clocks, correctness fingerprint, and rollback.

A microbenchmark can nominate a route. It cannot promote the route by itself. Rejected experiments remain useful when they preserve the failed correctness gate, measured ceiling, and condition for reconsideration.

9. Where to read and edit

Task	Swift host	Rust host	Iron
Model graph	Sources/ButterSwift/Models/	rust/crates/wh-butter-models/	Kernel primitives only
Loading	Sources/ButterSwift/Loader/	rust/crates/wh-butter-loader/	Packed layout helpers where shared
Generation	Sources/ButterSwift/Generation/	rust/crates/wh-butter/src/generate.rs	Selection/cache kernels
Serving and scheduling	Swift service surfaces	rust/crates/wh-butter-cli/	Device submission primitives
Semantic operation dispatch	Sources/IronSwift/ bindings	rust/crates/wh-butter-ops/	crates/wh-iron-std/ and runtime
Device adapter	Metal host types	rust/crates/backends/	crates/wh-iron-runtime/

Repository layout changes over time. Search for the named type or operation before assuming a path is current.

10. Next reading

• Rust engine for the Rust crate boundary and serving model.
• Iron architecture for source-to-shader lowering and runtime dispatch.
• Testing for Swift gates and the Rust/CUDA boundary.
• Benchmarking for metric definitions and report shape.
• Qwen3.8 architecture guide for the current dense and Flash-Next tracks.
• Detailed Swift/Metal diagrams for the Apple-host implementation.