Chat Templates

Most modern chat / instruct models ship with a Jinja chat template in their tokenizer_config.json. Butter calls into swift-transformers’ Tokenizer.applyChatTemplate(...) to render those templates; you pass typed ChatMessage values + a typed ChatTemplateOptions and Butter threads the right variables into the Jinja context.

The plain Model.generate(prompt:...) API takes a raw string — no chat template applied — which means you are responsible for rendering. Use the messages: overloads instead when working with chat / instruct models.

Buffered

let messages: [ChatMessage] = [
    .init(role: .system, content: "You are concise."),
    .init(role: .user,   content: "Why is the sky blue?"),
]

let result = try await model.generate(messages: messages)
print(result.text)

Streaming

let stream = try model.generateStream(messages: messages)
for try await chunk in stream {
    print(chunk.text, terminator: "")
}

Same chunk shape as the prompt: streaming variant — see streaming.md.

ChatMessage

public struct ChatMessage: Sendable, Equatable {
    public enum Role: String { case system, user, assistant, tool }
    public var role: Role
    public var content: String
    public var thinking: String?   // re-emit reasoning trace in multi-turn
}

The thinking field is for multi-turn conversations where the prior assistant turn included a thinking segment that the template wants to re-emit (Qwen 3 / DeepSeek-R1 do this).

ChatTemplateOptions

public struct ChatTemplateOptions: Sendable {
    public var addGenerationPrompt: Bool        // = true
    public var enableThinking: Bool             // = false
    public var reasoningEffort: ReasoningEffort?// = nil  (.low | .medium | .high)
    public var maxLength: Int?                  // = nil
    public var truncation: Bool                 // = false
    public var extraContext: [String: any Sendable]
}

Field	Maps to	When to set
addGenerationPrompt	template’s “now generate the assistant reply” suffix	true (default) for the typical chat-completion case; false when scoring an existing assistant turn (e.g. perplexity over a fixed conversation).
enableThinking	enable_thinking Jinja variable	true to turn on the model’s reasoning mode (Qwen 3 emits <think>...</think> blocks). Harmless on templates that don’t reference the variable.
reasoningEffort	reasoning_effort Jinja variable	GPT-OSS Harmony reasoning levels (low / medium / high).
maxLength / truncation	swift-transformers’ template-side truncation	Hard cap on the templated token count. truncation: false (default) throws on overflow; true truncates leading turns.
extraContext	additional Jinja variables	Anything else the template reads.

Format quirks

The template does the per-family rendering. We pass typed inputs through the well-known variable names; the rest is in the model’s tokenizer_config.json. Specific behaviours:

Family	Notes
Qwen 3	enable_thinking: true → <think>...</think> block before the answer. Pair with ThinkingSplit for per-segment stats.
DeepSeek-R1	Same <think>...</think> convention as Qwen 3.
GPT-OSS (Harmony)	reasoning_effort: "high" (etc.) → analysis + final channel structure. The GPT-OSS family ships; pair with the ThinkingSplit.harmony scanner to partition the analysis + final channels.
Gemma 3 / 4	<channel|reasoning|> markers when reasoning is enabled. The Gemma 3 / 4 family files ship; the ThinkingSplit scanner partitions the reasoning segment.
Spark-X2.5	Chat-template-critical. Greedy raw generate(prompt:) of an open story prompt loops on official Spark-MLX as well as Butter. Use generate(messages:) so the Jinja template wraps System / User / Bot. Default template thinking is on (enable_thinking defaults true in the checkpoint); set ChatTemplateOptions(enableThinking: true) to match. The assistant prefix is `<
Llama 3 instruct	Standard chat template, no reasoning hooks.
Tools / function calling	Templates that read tools render correctly via swift-transformers’ applyChatTemplate(...) — but Butter doesn’t yet expose a typed Swift surface for tool-call args / results. See Tool calling below.

Errors

public enum ChatTemplateError: Error {
    case noTemplateOnTokenizer  // tokenizer_config.json had no chat_template
    case renderFailed(any Error) // wraps the underlying Jinja error
}

noTemplateOnTokenizer typically means you’ve loaded a base (non-chat) checkpoint and should either pass a raw prompt via generate(prompt:), or use a different checkpoint (e.g. *-Instruct).

Rendering without generating

For testing / debugging the templated input, render to token ids without running the model:

let ids = try model.renderChatTemplate(
    messages: messages,
    options: ChatTemplateOptions(enableThinking: true)
)
print(ids)
print(model.tokenizer.decode(tokens: ids, skipSpecialTokens: false))

Tool calling

The Jinja chat templates we render via swift-transformers’ AutoTokenizer.applyChatTemplate(...) already support the tools Jinja variable that most modern instruct checkpoints reference — Qwen 3, Llama 3.1+, Granite, Mistral, GPT-OSS Harmony, and friends all ship templates that render a tool / function-spec preamble into the prompt when the variable is set. What ships today is the rendering side: pass a tools list through ChatTemplateOptions.extraContext["tools"] = [...] (or whatever variable name the template reads) and the rendered prompt will include the tool preamble exactly as the upstream template defines it.

What’s not yet wired is the typed Swift surface on Butter’s side: ChatMessage doesn’t carry toolCalls / toolResults fields, and there’s no per-family tool-call parser that takes a stream of generated tokens and produces a typed [ToolCall(name:arguments:)] value. The model can emit a tool call (the template tells it how); Butter just hands you the raw tokens / text and lets you parse them yourself today.

The remaining work is tracked in the current planning/plan.md: per-family scanners (similar shape to the Thinking-vs-Generation Split scanners), the typed ChatMessage extension, and a .toolCalling capability bit so callers can branch on whether a loaded model supports it.

See also

• Quickstart — prompt: vs messages: decision.
• Streaming — both overloads support streaming.
• Observability § Thinking vs Generation Split — what enable_thinking: true enables on the stats side.