Serve

butter serve is an OpenAI-compatible HTTP server. One process holds one loaded checkpoint and a bounded pool of independent generation sessions. Completions clients talk to it over HTTP/1.1. In-process generate() is unchanged.

On Apple Silicon the Swift CLI loads HuggingFace / MLX checkpoints. On other platforms the Rust CLI loads a GGUF. The HTTP contract is the same.

Start

Apple (Swift):

butter serve --model mlx-community/Qwen3.5-0.8B-MLX-4bit

Elsewhere (Rust, GGUF path required):

butter serve --model /path/to/model.gguf

Default bind is 127.0.0.1:8000. Override with --host and --port.

butter serve --model <id-or-path> --host 0.0.0.0 --port 8000

The process prints the routes when it is listening:

GET  /health
GET  /v1/models
POST /v1/chat/completions
POST /v1/messages

POST /v1/chat/completions is the product path. Point a Completions client at http://127.0.0.1:8000/v1.

Discovery

GET /v1/models lists the resident checkpoint: id plus max_model_len / context_length when known.

GET /health returns ok.

Completions

POST /v1/chat/completions accepts a JSON body with messages. Optional fields the server honors:

Field	Notes
model	Resident id, or a different resolvable id to swap (see below).
stream	SSE chunks when true.
stream_options.include_usage	Extra usage-only chunk after finish_reason.
max_tokens / max_completion_tokens	Generation cap.
temperature, top_p, seed	Sampling.
tools	Function tools; leaked markup is recovered as structured tool_calls.
chat_template_kwargs.enable_thinking	Thinking on/off when the template supports it.

Authorization headers are ignored (no auth). Unknown JSON fields are ignored.

Streaming

SSE data: lines share one completion id. Content deltas come first. A recovered tool call is two tool_calls deltas: the first carries id / name, later deltas are arguments only. Loop on presence of delta.tool_calls, not only finish_reason.

The finish chunk always has a string finish_reason (stop, tool_calls, or length). When include_usage is set, a usage-only chunk (empty choices) follows with prompt_tokens, completion_tokens, total_tokens, and cached_tokens. Then data: [DONE].

Closing the TCP connection mid-prefill or mid-decode aborts GPU work. Leftover prefill chunks do not run.

Prefix reuse

usage.cached_tokens is the matched prefix length (nPast) of the last prompt against the live KV. Overlapping prompts and tool rounds that resend the full messages (stable system + tools stem) reuse that prefix. It is not a planted constant.

A tool round that forks the conversation rebuilds the cache. Recurrent hybrid layers cannot rewind.

Tools

If the model emits <tool_call>…</tool_call> JSON or XML markup, the server strips think blocks, recovers structured tool_calls, and does not leave raw markup in content. Non-stream responses put tool_calls on the assistant message with finish_reason: "tool_calls".

One resident model

The process keeps a single loaded checkpoint and a bounded session pool. Each active request owns its session, cache state, cancellation flag, sampling state, and memory reservation.

• Completions model equal to the resident id: no reload. An idle session with the longest matching prompt prefix is leased first.
• A different resolvable id: abort in-flight generation, drop every old session cache, load the new checkpoint, then serve. After a successful swap the machine holds one model, not two.
• Unknown or unloadable id: 404 (or 400). The previous resident stays on GET /v1/models. No hang, no half-loaded process.

KV is not written to disk. After a swap the client resends full messages; the new model prefills from scratch.

Boot loads --model before accepting traffic. A Completions call can still swap the resident later.

Flags

Flag	Meaning
--model	HuggingFace id or local path (Swift). GGUF path (Rust). Required.
--host	Bind address. Default 127.0.0.1.
--port	Bind port. Default 8000.
--parallel	Maximum active generation sessions. Default 1; excess clients wait in FIFO order.
--enable-thinking	Default chat_template_kwargs.enable_thinking when the client omits it.
--reasoning-effort	low / medium / high when the client omits it.
--max-context	Size the KV for this many positions.
--kv-cache	raw, affine8, affine4, aura, or an auraNvM recipe. Affine/AURA are Apple-only; Rust logs and uses raw.
--aura-decode-path	compressed (default) or dequant-mirror. Ignored unless --kv-cache is AURA.
--kv-window-size	FIFO KV window. Unset = unbounded up to --max-context.
--kv-window-keep	Attention-sink slots. Needs --kv-window-size.

Concurrent requests

The session pool bounds active owners and queues excess clients in FIFO order. Closing a queued connection removes only that waiter. Closing an active connection cancels only that request at the next bounded work unit, while sibling requests continue.

Admission reserves prompt plus requested decode capacity before cache allocation. The reservation is reconciled after installation and released on completion, cancellation, or failure. Cache capacity grows from request demand rather than allocating the model maximum context for every session.

Compatible ready dense Qwen requests can share equal-sized prefill chunks and decode model passes. Recurrent state, attention KV, filters, token history, random draws, and result order remain request-owned. MoE, MTP, ragged prefill chunks, and other model families retain bounded independent sessions but do not use this fused path.

Concurrency benchmark

With a server already running, sweep synchronized request waves without loading a second model:

butter serve-bench --model <resident-id> --concurrency 1,2,4,8 \
  --max-tokens 128 --temperature 0.7 --top-p 0.9

The command isolates each lane and wave with a unique prompt marker. It reports prompt, cached, uncached, and completion token totals, plus marker-leakage, empty-response, and invalid-UTF-8 counts. Use --shared-prompt only when prefix reuse is the workload under test, and inspect the cached-token column before comparing levels. The wall-clock completion and uncached-prompt columns both include the full request lifetime; neither is an isolated kernel rate.

The post-TTFT completion rate removes one first token per response and measures the remaining tokens from the wave’s earliest first output through its latest completion. This separates the initial prefill window from aggregate decode without claiming per-token timing that the event stream does not expose.

Pass --long-prompt <text> and optionally --long-prompt-lanes <count> to mix long-prompt and standard lanes in each synchronized wave. Mixed reports include TTFT p50/p95, request p95, and class-specific TTFT p95. TTFT begins at the first non-empty content or structured tool delta; role metadata does not count.

Library (Apple)

HTTP is optional. An Apple app can keep a Model and call generate() / generateStream() in-process, or wrap a loaded model in ButterServeBackend + ServeServer if it wants the same HTTP surface.

let model = try await Model.load("mlx-community/Qwen3.5-0.8B-MLX-4bit")
let backend = ButterServeBackend(
    model: model,
    modelId: "my-model",
    maximumConcurrentRequests: 8)
let server = ServeServer(host: "127.0.0.1", port: 8000, backend: backend)
try await server.runForever()

ResidentServeBackend wraps that for Completions-driven swap with a loader closure.

Not this server

No /v1/responses. No multi-checkpoint LRU. No disk session snapshot. One-shot generate() stays the library entrypoint for a single prompt.