Butter

Table of contents for the Butter documentation. The top-level README is the curated landing page. Start with the path that matches what you are trying to understand; the reference list below remains available when you need one page directly.

Recommended reading paths

Understand the inference engine

1. Architecture for the Butter/Iron boundary, request lifecycle, execution graphs, state ownership, and evidence ladder.
2. Rust engine for the CUDA and portable-backend crate layout and concurrent-serving model.
3. Qwen3.8 architecture guide for a worked hybrid, mixed-precision, and speculative model path.
4. Qwen3.8 decode performance next steps for the clean-Spark correctness, quality, profiling, and promotion sequence.
5. Benchmarking for the clocks and receipt fields used to judge an optimization.

Use Swift Butter

1. Installation.
2. Quick start.
3. Models and generation parameters.
4. Streaming or serving.

Add or optimize a model

1. Architecture.
2. Rapid Software Testing for risk, oracle, coverage, mutation, and evidence rules.
3. Developing.
4. Adding a model.
5. Testing, then benchmarking.
6. Rust efficiency before changing the Rust runtime or a hot path.

• Packaged-product guide — user workflow, four commands, files on disk, operator configuration, native archives, and the deferred website installer.
• Packaged-product RST strategy — release identity, automation, exploratory charters, capacity/Aura matrices, and WH-003 gates.

Getting started

• Installation — SwiftPM / Xcode setup, platform requirements, sibling-iron checkout.
• Quick start — generate text in 5 lines.
• Using the CLI — build the butter binary and run it via swift run, the built path, or a PATH symlink.
• Serve — butter serve, Completions HTTP, one resident checkpoint, prefix reuse and abort.
• Architecture: both Butter hosts, the Iron boundary, request lifecycle, dense/hybrid/MoE/speculative graphs, state ownership, and evidence.
• Models — the supported architecture families (dense text, SSM / GDN / conv+attention hybrids, MoE, vision-language, audio, neural codecs), per-family known gaps, and adding a new family.

Cross-cutting topics

• GenerationParameters reference — every generation knob, per-family defaults table, the three call shapes (default, with-override, custom).
• Streaming — generateStream(...), GenerationChunk shape, cancellation, why streaming is the primitive over which buffered generate(...) is built.
• Chat templates — ChatMessage + ChatTemplateOptions, enableThinking / reasoningEffort hooks, per-family quirks (Qwen 3 / DeepSeek-R1 / GPT-OSS / Gemma).
• KV cache — the raw fp16 / bf16 cache, GPU-side kv_cache_update kernel, and what’s coming (affine, AURA, SSM/GDN).
• Quantization — mlx-format coverage (3 / 4 / 5 / 6 / 8-bit), packing layout, sub-group split dispatch.
• Performance — current tok/s numbers per model, what each Phase 4 wave got us, where the remaining headroom is.
• Observability — --stats (per-phase memory, TTFT, KV cache, wired ticket), --debug (subsystem-tagged stderr logs), --profiling (wallclock + os_signpost), perplexity / think-vs-gen split helpers.
• Benchmarking — butter bench --method <name> + --ref-model for KLD, per-day markdown + JSON sidecar reports (mlx-swift-lm-compatible row schema).
• Capabilities & lifecycle — the Capability enum, LoadOptions, ModelLifecycleEvent stream.

Local development

• Developing in Butter — repo layout, the make workflow, regenerating kernels.
• Adding a model — porting a new architecture from a reference implementation.
• Testing — running tests, the expectCoherentOutput integration model, coverage targets.
• Test inventory — the reproducible per-file audit queue for Swift and Rust declarations and their execution lanes.
• Rapid Software Testing — the Butter-specific session method for risks, oracles, reachability, known-bad mutations, and honest evidence handoffs.
• Rust efficiency — project rules for Rust allocation, ownership, synchronization, graph stability, numerical gates, and production performance receipts.
• NCCL backend contract — same-stream all-reduce, cross-rank preflight, bounded RoCE failure settings, and lifecycle rules.
• Spark-backed CI — opt-in NVIDIA Spark reservations for development, PR tests, scheduled benchmarks, and Deckard-safe capacity sharing.
• Publishing — dev → main release flow, release notes, and how the docs site rebuilds.

See also

• Top-level README — project landing page.
• planning/plan.md — phased build-out, what ships when.
• planning/architecture.md — longer-form architecture diagrams.
• planning/qwen38-architecture-guide.md: current Qwen3.8 graph, ownership, state, tests, and document map.

How these docs get published

The user-facing site at https://thewafflehaus.github.io/butter-website/ is built from the markdown in this repo (documentation/*.md, README.md, planning/architecture.md, planning/roadmap.md) by a separate site repo, thewafflehaus/butter-website. The site fetches Butter’s markdown at build time — there’s no manual copy step.

The published site always builds against a real, immutable Butter release tag — never main HEAD. So unreleased doc changes that land on this repo’s main are intentionally invisible to the published site until the next release.

When the site rebuilds

Trigger	What happens
A new release is published on this repo	.github/workflows/notify-docs.yml calls workflow_dispatch on butter-website’s deploy.yml, passing the release tag, name, body, and url as inputs. butter-website pins its Butter checkout to that tag, renders the release body as the Changelog page, updates the version label in the site title + hero, then deploys.
Push to main on butter-website	Site source changed (CSS, layout, new page). butter-website rebuilds against Butter’s latest published release (via gh release view).
Manual dispatch on either repo	Same — butter-website always builds against the latest release (or against a specific tag if you pass --field butter_tag=...).

The token

The cross-repo dispatch needs a WEBSITE_DISPATCH_TOKEN secret on this repo (Butter). Use a fine-grained PAT scoped to only thewafflehaus/butter-website with Actions: Read and write plus the implicit Contents: Read-only + Metadata: Read-only. The token does not need Contents: write — the dispatch uses workflow_dispatch (not the older repository_dispatch) so the worst a leaked token could do is spam-trigger the deploy workflow or cancel runs; it can’t modify butter-website’s repo contents.

Releasing → publishing flow

1. Land doc changes on dev alongside the code changes they describe.
2. Open a PR dev → main titled Release vX.Y.Z. Wait for CI; merge.
3. Trigger the Release workflow from the GitHub UI (Actions → Release → Run workflow on main). Pick the right bump_type + prerelease_tag; it runs tests, then scripts/release.sh tags the merge commit and gh release create --generate-notes publishes the release.
4. The release: published event automatically fires notify-docs.yml; the site rebuilds within a minute or two and the Changelog gets a new section from the release body.

You can also kick a rebuild manually without cutting a release:

# Re-publish against the latest release (e.g. you want to verify the
# site build before / between releases).
gh workflow run deploy.yml --repo thewafflehaus/butter-website

# Force a rebuild against a specific past release.
gh workflow run notify-docs.yml --repo waffuruai/butter --field tag=v0.1.0

# Rebuild butter-website against an arbitrary Butter tag directly (skips
# the Butter notify hop — handy for debugging the site itself).
gh workflow run deploy.yml --repo thewafflehaus/butter-website \
  --field butter_tag=v0.1.0

Previewing unreleased doc changes locally

The published site won’t show unreleased docs, but the local Astro dev server can build against your Butter working tree:

git clone https://github.com/thewafflehaus/butter-website ../butter-website
cd ../butter-website
pnpm install
BUTTER_REPO_PATH=$(pwd)/../Butter pnpm dev   # → http://localhost:4321

make docs from this repo prints the same commands if the ../butter-website checkout exists.