Changelog

Release notes for Butter, sourced from published releases on GitHub.

v0.1.0

Released 2026-05-17. View on GitHub →

First tagged release. FFAI is an Apple Silicon LLM inference library built on pre-compiled Metal kernels — single command buffer per token, single 4-byte CPU↔GPU round-trip, no JIT, no Python, no MLX runtime dependency.

Highlights

• Two dense LLM families end-to-end — Llama 3.x and Qwen 3, both running real HuggingFace checkpoints across bf16 / 3 / 4 / 5 / 6 / 8 bit quantizations from mlx-community.
• Mamba 2 dense decode (Phase 5e) — SSM + conv-cache layer with constant-memory recurrent state. 144 decode tok/s on mlx-community/mamba2-130m.
• Affine-quantized KV cache — --kv-cache affine4 / affine8 via shared working-buffer pattern. ~45% / 70% KV memory vs raw bf16 at modest tok/s cost.
• GPU-only sampling path — temperature, top-K, top-P, min-P, repetition penalty, seeded reproducible draws. Three execution paths: greedy-GPU, gpu-categorical (fused softmax + categorical kernel), cpu-sample (filter-bearing path).
• Streaming-first API — generateStream(prompt:parameters:) yields one chunk per token; buffered generate(...) is a collector over the same stream.
• Pre-compiled metallib — kernels are generated from the metaltile Rust DSL at build time via tile build --emit all, shipped as a single kernels.metallib resource. Zero JIT latency on first model load.
• Observability built in — --stats (per-phase GPU memory + KV bytes + wired ticket), --debug (subsystem-tagged stderr), --profiling N (wallclock + os_signpost), Perplexity + KL-divergence helpers.
• Benchmarking — ffai bench --method <name> mirroring mlx-swift-lm’s method matrix; per-day markdown + JSON sidecar reports.

Performance baseline (Apple M1 Max, this release)

Model	Quant	Decode tok/s	Peak GPU
unsloth/Llama-3.2-1B	bf16	31.86	6.30 GB
mlx-community/Qwen3-1.7B-bf16	bf16	23.06	7.60 GB
mlx-community/Qwen3-1.7B-8bit	8bit	24.32	6.10 GB
mlx-community/Qwen3-1.7B-6bit	6bit	19.47	5.70 GB
mlx-community/Qwen3-1.7B-5bit	5bit	18.99	5.50 GB
mlx-community/Qwen3-1.7B-4bit	4bit	25.95	5.30 GB
mlx-community/Qwen3-1.7B-3bit	3bit	20.24	5.10 GB
mlx-community/mamba2-130m	bf16	144.43	0.27 GB

What’s in this release

Inference

• ✨ Mamba 2 dense (Mamba2Model, Mamba2Layer, Mamba2LayerCache). Single-cmdbuf decode: RMSNorm → in_proj → conv1d_causal_step + SiLU → softplus(dt) → ssm_step → D·x skip → SiLU(z) gate → mixer norm → out_proj → residual. Verified against mlx-community/mamba2-130m.
• ✨ Affine-quantized KV cache (AffineQuantizedKVCache) — int4 + int8 with one shared working-buffer pair across every layer’s cache. CLI: --kv-cache affine4 / affine8. ~45% / 70% memory savings vs raw at small decode-tok/s cost.
• ✨ LayerCacheProtocol parent so SSM caches don’t need no-op attention methods. KVCacheProtocol is a sub-protocol that adds the attention surface. LanguageModel.makeKVCache → makeLayerCaches.
• ✨ Full sampling pipeline — temperature / top-K / top-P / min-P / repetition penalty / --seed, three execution paths (greedy-GPU, gpu-categorical, cpu-sample), per-family forwardSampleCategorical fusion on Llama + Qwen3.
• ✨ Streaming + chat templates — generateStream, swift-transformers chat template integration, enableThinking / reasoningEffort knobs for Qwen3, custom HF cache directory via LoadOptions.cacheDirectory.

Kernel + build pipeline

• 🚀 Migrated to metaltile CLI — tile build --emit msl,metallib,swift,ir,all replaces the standalone metaltile-emit bin. Emit pipeline now lives in metaltile-codegen::emit as a reusable library module.
• 🚀 78 kernels ported into metaltile-std/src/ops/ covering softplus, generic ffai_argmax<T>, softmax_categorical_sample, kv_cache_update, int4/int8 KV quantize + dequant, ssm_step + conv1d_causal_step, int3/4/5/6/8 dequant_gemv + dequant_gather, ffai_gather, ffai_rope_llama, ffai_sdpa_decode.
• 🚀 Ops.swift migrated to upstream kernel names where signatures matched: add → vector_add, mul / silu / softplus / rms_norm / gemv → mt_*. ffai_* prefix kept for kernels with FFAI-specific surface.

Observability + benchmarking

• ✨ --stats / --debug / --profiling flags on the CLI. PhaseMemoryTracker for prefill vs decode memory deltas. os_signpost integration. Perplexity + KL-divergence helpers (Perplexity.compute(...), Perplexity.klDivergence(...)).
• ✨ ffai bench --method <name> — simple, summarization, wikitext2 shipped; niah, multi-turn, tool-calling, ngram-*, vision stubbed for later. mlx-swift-lm-compatible report row schema; KLD comparison via --ref-model.

Docs + contributor surface

• 📚 Docs site at https://thewafflehaus.github.io/ffai-website/ (Astro + Starlight + Tailwind). Builds against a real FFAI release tag — never main HEAD. Rebuilds automatically when a new FFAI release is published.
• 📚 documentation/ covers installation, quickstart, using-the-cli, architecture, models, KV cache, quantization, performance, observability, benchmarking, capabilities, chat templates, streaming, generation parameters, plus the developer guides under documentation/developing/.
• 📚 CONTRIBUTING.md + structured GitHub issue / PR templates with mandatory type + AI-assistance disclosure checkboxes.
• 📚 New documentation/developing/publishing.md walks through the release workflow end-to-end.

CI + release infrastructure

• 🔧 release.yml workflow + scripts/release.sh (adapted from mlx-swift-lm). Manual workflow_dispatch from main: runs full unit + integration suite, auto-bumps FFAI.version to match the new tag, creates release/<tag> branch + annotated tag, fires gh release create --generate-notes.
• 🔧 notify-docs.yml fires on release: published → workflow_dispatch against ffai-website (least-privilege Actions: write token, not Contents: write).
• 🔧 ci.yml runs unit suite (~30s) on every push + PR for main and dev; integration suite runs at release time only, serialized to one model in memory at a time so the 7 GB runner doesn’t OOM-thrash.

Install

.package(url: "https://github.com/thewafflehaus/FFAI", from: "0.1.0")

Or the CLI:

git clone https://github.com/thewafflehaus/FFAI
cd FFAI
git clone https://github.com/thewafflehaus/metaltile ../metaltile
./scripts/setup-dev.sh
.build/release/ffai generate -m mlx-community/Qwen3-1.7B-bf16 -p "Once upon a time"

See documentation/ for the full setup.

What’s next (Phase 5e+, Phase 6)

Per planning/roadmap.md:

• Mamba 2 hybrid families (NemotronH / GraniteMoeHybrid / FalconH1).
• Qwen 3.5 hybrid (GDN + attention).
• TurboQuant KV cache (Phase 5d).
• Vision multi-modal (Qwen 2.5-VL / 3.5-VL).

Acknowledgements

• 0xClandestine/metaltile — the Rust kernel DSL FFAI’s metallib is built from.
• The mlx-community for the quantized checkpoints FFAI’s integration suite consumes.
• The swift-transformers team for the tokenizer + chat template plumbing.

---

Full Changelog: https://github.com/thewafflehaus/FFAI/commits/v0.1.0