Capabilities & Lifecycle

このコンテンツはまだ日本語訳がありません。

Butter models declare what they can do via Capability, the user picks what to enable at load time via LoadOptions, and the model exposes its load progress + hot capability changes via an AsyncStream<ModelLifecycleEvent>.

The vision-language families (Gemma 3/4-VL, Qwen 2/2.5/3-VL, Qwen3-VL-MoE, MiniCPM-V, SmolVLM2, Nemotron-VLM, Idefics3, GlmOcr, FastVLM, Mistral3, Paligemma) and audio families exercise it end-to-end.

The `Capability` enum

public enum Capability: String, Sendable, Hashable, CaseIterable, Codable {
    case textIn
    case textOut
    case imageIn
    case videoIn
    case audioIn
    case audioOut
    case toolCalling
    case thinking
    case reasoningLevel
}

Capability	Today
`.textIn` / `.textOut`	✅ Always on for LLMs.
`.imageIn`	✅ Image input — image-only VL models (Gemma 3/4-VL, Nemotron-VL, Idefics3, GlmOcr, FastVLM, Mistral3, Paligemma, …). Video-capable families add `.videoIn` separately.
`.videoIn`	✅ Video input — video-capable VL families (Qwen 2-VL, Qwen 2.5-VL, Qwen 3-VL, MiniCPM-V, SmolVLM2). Always declared together with `.imageIn`.
`.audioIn`	✅ Whisper STT + SenseVoice STT + Qwen-Omni audio-in.
`.audioOut`	✅ Kokoro TTS (iSTFTNet vocoder tail).
`.toolCalling`	Not declared by any family.
`.thinking`	Model emits a chain-of-thought trace (Qwen 3 thinking, DeepSeek-R1, etc.).
`.reasoningLevel`	Model honours a user-tunable reasoning-effort dial (GPT-OSS-20B, …).

Convenience sets:

Capability.textOnly       // [.textIn, .textOut]
Capability.textWithTools  // [.textIn, .textOut, .toolCalling]
Capability.speechToText   // [.audioIn, .textOut]   — Whisper, SenseVoice
Capability.textToSpeech   // [.textIn, .audioOut]   — Kokoro
Capability.omniAudio      // [.textIn, .audioIn, .textOut] — Qwen-Omni

Audio models

Audio models do not route through Model / ModelRegistry (which describe a text-in / text-out causal decoder). They load through AudioModelRegistry, which inspects config.json, picks the family (Whisper STT, SenseVoice STT, Kokoro TTS, Qwen-Omni), and reports the audio Capability set:

let loaded = try AudioModelRegistry.load(directory: dir)
switch loaded {
case .whisper(let m):    ...   // .speechToText
case .senseVoice(let m): ...   // .speechToText
case .kokoro(let m):     ...   // .textToSpeech
case .qwenOmni(let m):   ...   // .omniAudio
}

AudioModelRegistry.capabilities(forConfigAt:) reports the capability set without loading weights — useful for a model picker.

What each family declares

Family	`availableCapabilities`
`Llama.LlamaDense`	`[.textIn, .textOut]`
`Qwen3.Qwen3Dense`	`[.textIn, .textOut]`
`Mamba2.Mamba2Dense`	`[.textIn, .textOut]`
`FalconH1.FalconH1Hybrid`	`[.textIn, .textOut]`
`NemotronH.NemotronHHybrid`	`[.textIn, .textOut]`
`NemotronDiffusion.NemotronDiffusionDense`	`[.textIn, .textOut]`
`Granite4.Granite4Hybrid`	`[.textIn, .textOut]`
`Jamba.JambaHybrid`	`[.textIn, .textOut]`
`LFM2.LFM2Dense` / `LFM2MoE`	`[.textIn, .textOut]`
`Qwen35.Qwen35Hybrid`	`[.textIn, .textOut]`
`Gemma4.Gemma4Dense` / `Gemma4E` / `Gemma4MoE`	`[.textIn, .textOut]`
`GPTOSS.GPTOSSMoEVariant`	`[.textIn, .textOut]`

When a family adds a capability (e.g. the VL families add .imageIn, video-capable VL families add both .imageIn and .videoIn), the family file declares it and the loader allocates the corresponding subnet only if the user opts in.

`LoadOptions`

let model = try await Model.load(
    "mlx-community/Qwen3.5-0.8B-MLX-4bit",
    options: LoadOptions(
        capabilities: [.textIn, .textOut],
        kvCache: .raw,
        dispatchMode: .eager,
        prewarm: true,
        lazyCapabilities: true,
        revision: "main"
    )
)

Field	Default	Notes
`capabilities`	`Capability.textOnly`	What to load. `textIn` + `textOut` are always implicitly on. Disabled modalities skip weight allocation.
`kvCache`	`.raw`	Cache compression scheme — see kv-cache.md.
`dispatchMode`	`.eager`	Standard `MTLComputeCommandEncoder` per kernel. `.argumentBuffers` / `.icb` deferred.
`prewarm`	`true`	Run one no-op forward to compile PSOs before the first user-visible decode.
`lazyCapabilities`	`true`	Allow runtime `enable(_:)` / `disable(_:)` after load.
`revision`	`"main"`	HF branch / tag / commit.

Inspecting a loaded model

let model = try await Model.load("mlx-community/Qwen3-4B-4bit")

print(model.availableCapabilities)   // what the family supports
print(model.enabledCapabilities)     // what you opted into
print(model.config.modelType)        // "qwen3"
print(model.modelDirectory)          // resolved local snapshot

If you ask for a capability the family doesn’t expose, the loader throws ModelError.capabilityNotAvailable(.imageIn).

Lifecycle states

ModelLifecycleState:
  idle → downloading(Progress) → loading(LoadProgress)
       → loaded → ready
       (or failed(Error) at any stage)

Model.events is an AsyncStream<ModelLifecycleEvent> that emits each transition. The stream is multi-consumer-safe and finishes when the Model is deinitialized.

let model = try await Model.load("mlx-community/Qwen3.5-0.8B-MLX-4bit")

Task {
    for await event in model.events {
        switch event.state {
        case .downloading(let progress): print("downloading \(progress.fractionCompleted)")
        case .loading(let p):            print("loading \(p)")
        case .loaded:                    print("weights resident")
        case .ready:                     print("ready to generate")
        case .failed(let err):           print("failed: \(err)")
        default:                         break
        }
    }
}

print(model.currentState)  // sync snapshot — typically .ready by the time load() returns

currentState is a thread-safe snapshot of the latest emitted event. The stream is the source of truth for fine-grained progress.

Hot capability changes

try await model.enable(.imageIn)    // mmaps vision weights, builds encoder, prewarms
// ... use the model with images ...
try await model.disable(.imageIn)   // releases MTLBuffers, frees GPU residency

Each call emits per-capability lifecycle events through the same events stream. If lazyCapabilities = false was passed at load time, both calls throw — capabilities are then frozen at the load-time set.