Skip to content
The Waffuru waffle markWaffuru Docs

Architecture

Butter is the inference engine. Iron is the kernel and device layer beneath it. Butter has two maintained hosts: Swift for the Apple-native product and Rust for CUDA and portable GPU backends. The hosts share architectural contracts, but they do not share every implementation or capability.

Read this page in order if you are new to the project. It starts at the Butter/Iron boundary, follows one request through the engine, then explains the four execution graphs, state ownership, and the evidence required for a change.

1. The Butter and Iron boundary

Section titled “1. The Butter and Iron boundary”
application or serving client
              |
              v
Butter host
  model config and checkpoint interpretation
  tokenizer and chat template
  request admission and scheduling
  prefill, decode, sampling, and streaming
  KV and recurrent-state lifecycle
              |
              v
semantic operations
  projection, normalization, attention, recurrence,
  quantization, cache update, selection
              |
              v
Iron
  kernel math and layout contracts
  IR passes and target lowering
  launch geometry and backend runtime
              |
              v
Metal | CUDA | HIP/ROCm | Vulkan

The ownership rule is simple:

  • Butter owns model semantics. It decides which tensors form a layer, which layer runs at a position, which state belongs to a request, and when state is committed or discarded.
  • Iron owns device semantics. It defines what a kernel reads and writes, how dtypes and packed layouts are interpreted, and how the operation reaches a target GPU.
  • The boundary is a contract, not an implementation detail. A kernel can be correct by itself and still be wrong in a model because Butter supplied the wrong shape, offset, stride, state version, or execution order. Production changes therefore need evidence on both sides.

2. The two Butter hosts

Section titled “2. The two Butter hosts”
Host Primary use Kernel path Current scope
Swift Butter Apple-native applications on Mac, iPhone, and iPad Precompiled Metal libraries and generated Swift bindings Broad text, vision, and audio model catalog
Rust Butter Serving and systems work on NVIDIA CUDA, plus portable backend development Iron runtime plus backend-specific matrix and graph operations Text, hybrid, MoE, quantized, speculative, and concurrent serving workloads

Rust also has Metal and Vulkan device crates. Iron executes HIP/ROCm kernels, but Butter does not yet expose a first-class ROCm host adapter. Backend support is admitted per model and checkpoint. Shared IR support alone is not a full-model readiness claim.

3. From kernel source to a running operation

Section titled “3. From kernel source to a running operation”

The source pipeline is shared until target lowering. Compilation and loading then follow the backend contract.

Rust #[kernel] definition
          |
          v
Iron IR and backend-independent passes
          |
          +----------------+----------------+----------------+
          |                |                |                |
          v                v                v                v
         MSL            CUDA C++          HIP C++          SPIR-V
          |                |                |                |
          v                v                v                v
    metallib at build   NVRTC to PTX    hipRTC object    Vulkan module
          |                |                |                |
          +----------------+----------------+----------------+
                           |
                           v
                 Butter semantic operation

On Apple, the package ships a precompiled Metal library and generated Swift wrappers. On CUDA, the Rust device compiles emitted CUDA through NVRTC, loads the resulting module, and may select native matrix-library or AOT helpers for qualified shapes. HIP and Vulkan use their own runtime modules and narrower validated inventories.

Kernel identity matters. A useful result identifies the selected operation, backend, dtype, shape, generated artifact, and fallback route. A silent fallback is not equivalent evidence.

4. One request through the engine

Section titled “4. One request through the engine”

Every request follows the same lifecycle even when its model graph differs.

model id or local checkpoint
          |
          v
resolve files and parse config
          |
          v
load immutable weights and build the model graph
          |
          v
tokenize and render the prompt
          |
          v
admit request-owned cache and workspace state
          |
          v
prefill prompt chunks
          |
          v
sample first token
          |
          v
decode one step or verify a speculative block
          |
          v
sample, stream, stop-check, repeat
          |
          v
publish usage and release request state

Load

Section titled “Load”

The loader resolves a local directory or immutable model snapshot, parses config.json, validates tensor names and shapes, maps the checkpoint format, and constructs the layer schedule. Loader metadata is executable architecture: head counts, layer classes, quantization schemes, sliding-window rules, and tokenizer/template identity all affect runtime behavior.

Weights are shared and immutable after load. Request state is allocated later, under admission control, rather than being hidden inside the shared model.

Prefill

Section titled “Prefill”

Prefill ingests many known prompt tokens. Its important dimension is token width, so it favors matrix-shaped operations, chunking, and compatible batches across requests. Chunk size trades prompt throughput against time available for ready decode work.

Prefill publishes state only after the chunk succeeds. A failed or cancelled chunk must not leave a partially visible KV cache or recurrent state.

Decode

Section titled “Decode”

Autoregressive decode consumes one newly chosen token per request. It is often limited by weight traffic and launch overhead rather than arithmetic. The hot path keeps intermediate tensors and selection on the device when the sampling contract allows it.

Independent requests may share a compatible model pass, but their logical position, cache, recurrent state, stop policy, and output stream remain independent. The scheduler is the sole owner of accelerator submission.

Finish

Section titled “Finish”

EOS, a stop sequence, cancellation, a token limit, or an error resolves the request. Cleanup has to release admission, cache leases, graph-stable storage, and any queued scheduler work exactly once.

5. The four execution graphs

Section titled “5. The four execution graphs”

There is no single universal transformer loop in Butter. The loader selects a graph from checkpoint metadata, and that graph owns its state contract.

Dense attention

Section titled “Dense attention”
token embedding
  -> repeat for each layer:
       normalize
       Q/K/V projections
       position transform
       append K/V
       attention
       output projection and residual
       normalize
       gate/up projections
       activation and down projection
       residual
  -> final normalize
  -> language-model head
  -> sampling

Attention state is a KV prefix. Decode appends one logical position; batched prefill appends a range. Physical cache layout is chosen for the reader kernel, not merely for convenient writes.

Some dense families (Gemma 3 / 4, GPT-OSS, Spark-X2.5) still use this graph but schedule sliding-window and full attention from layer_types. Sliding layers bound KV with .window eviction; full layers stay unbounded up to LoadOptions.maxContextLength (or the advertised max_position_embeddings). Spark also picks a per-layer RoPE recipe and uses exact erf-GELU (Ops.geluErf) in the MLP.

Hybrid attention and recurrence

Section titled “Hybrid attention and recurrence”

Hybrid models interleave attention with GDN or SSM layers. Attention layers own KV state. Recurrent layers own matrices and convolution history. Both state types advance at the same logical token boundary.

layer schedule from model metadata
       |
       +-- attention layer -> KV read/write
       |
       +-- GDN layer       -> recurrent matrix + conv state
       |
       +-- SSM layer       -> SSM state + conv state
       |
       +-- shared dense or routed FFN

This distinction is critical during prompt chunking and speculative verification. Truncating KV length does not restore a recurrent matrix.

Mixture of experts

Section titled “Mixture of experts”

A routed layer adds selection and sparse expert execution:

hidden row
  -> router logits
  -> stable top-k experts and weights
  -> gather or group rows by expert
  -> expert gate/up/down work
  -> weighted reduction
  -> residual

The route includes capacity, ordering, accumulation precision, and empty-expert behavior. A fast expert kernel does not prove the router-to-reduction graph.

Speculative verification

Section titled “Speculative verification”

A speculative path proposes several tokens, verifies them with the target model, commits the accepted prefix plus the target correction token, and discards the rejected tail.

committed target state
          |
          v
draft proposal block
          |
          v
target verification over all rows
          |
          v
acceptance plan
       /       \
      v         v
commit chosen   discard rejected
KV/recurrent    KV/recurrent rows
state           and draft state

Verification is a transaction. Target KV, recurrent state, token history, draft state, and public sequence length must resolve together. The target model remains the correctness authority even when draft arithmetic uses a different precision contract.

For the current Qwen3.8 terminology and tests, start with planning/qwen38-architecture-guide.md.

6. State and ownership

Section titled “6. State and ownership”
State Owner Lifetime May be shared?
Model weights and immutable lookup tables Loaded model Model residency Yes
Kernel modules, pipelines, and qualified plans Device/runtime cache Process or device lifetime Yes, when the cache key is complete
KV cache Request session Prompt through completion No mutable sharing
GDN/SSM and convolution state Request session Prompt through completion No mutable sharing
Sampling and stop state Request session Completion No
Activation workspace Scheduler/session lease One operation or captured graph Only through explicit non-overlapping leases
Captured graph storage Captured session/shape Graph lifetime Only when pointer stability and lane ownership are proven

Three rules prevent most state bugs:

  1. Share immutable model data; isolate mutable sequence data.
  2. Publish a new logical state only after all device writes that define it are ordered and complete.
  3. Treat position, cache length, recurrent version, and token history as one commit boundary.

7. Prefill and decode are separate performance problems

Section titled “7. Prefill and decode are separate performance problems”
Property Prefill Decode
Known token width Many prompt tokens One selected token per stream
Common limiting factor Compute, tiling, chunk policy, batch formation Weight traffic, launch count, synchronization, cache reads
Primary user metric TTFT and prompt tok/s Inter-token latency and tok/s per stream
Serving metric Batched prompt throughput and queue delay Per-stream latency plus aggregate output tok/s
Typical optimization Wider matrix work and compatible batching Exact small-row kernels, graph replay, device-resident state

Do not combine the two clocks into one headline without labeling it as whole-request throughput. A performance result records prompt tokens, output tokens, context depth, concurrency, sampling mode, warmup, repetitions, and whether the clock includes TTFT.

8. How a change becomes production evidence

Section titled “8. How a change becomes production evidence”

Evidence is layered because each layer catches a different failure.

  1. Pure logic and codegen: configuration parsing, shape arithmetic, IR passes, generated source, and selection predicates.
  2. Kernel oracle: exact dtype, layout, offset, stride, tail, and production shape against an independent reference.
  3. Model composition: checkpoint loading, layer schedule, eager/captured parity, token or declared numeric policy, and state progression.
  4. Serving transaction: cancellation, concurrent sessions, cache ownership, speculative commit/abort, and cleanup.
  5. Performance promotion: same workload and binary, warm repetitions, component and end-to-end clocks, correctness fingerprint, and rollback.

A microbenchmark can nominate a route. It cannot promote the route by itself. Rejected experiments remain useful when they preserve the failed correctness gate, measured ceiling, and condition for reconsideration.

9. Where to read and edit

Section titled “9. Where to read and edit”
Task Swift host Rust host Iron
Model graph Sources/ButterSwift/Models/ rust/crates/wh-butter-models/ Kernel primitives only
Loading Sources/ButterSwift/Loader/ rust/crates/wh-butter-loader/ Packed layout helpers where shared
Generation Sources/ButterSwift/Generation/ rust/crates/wh-butter/src/generate.rs Selection/cache kernels
Serving and scheduling Swift service surfaces rust/crates/wh-butter-cli/ Device submission primitives
Semantic operation dispatch Sources/IronSwift/ bindings rust/crates/wh-butter-ops/ crates/wh-iron-std/ and runtime
Device adapter Metal host types rust/crates/backends/ crates/wh-iron-runtime/

Repository layout changes over time. Search for the named type or operation before assuming a path is current.

10. Next reading

Section titled “10. Next reading”