Capabilities & Lifecycle

Questi contenuti non sono ancora disponibili nella tua lingua.

Butter models declare what they can do via Capability, the user picks what to enable at load time via LoadOptions, and the model exposes its load progress + hot capability changes via an AsyncStream<ModelLifecycleEvent>.

The vision-language families (Gemma 3/4-VL, Qwen 2/2.5/3-VL, Qwen3-VL-MoE, MiniCPM-V, SmolVLM2, Nemotron-VLM, Idefics3, GlmOcr, FastVLM, Mistral3, Paligemma) and audio families exercise it end-to-end.

The `Capability` enum

public enum Capability: String, Sendable, Hashable, CaseIterable, Codable {
    case textIn
    case textOut
    case imageIn
    case videoIn
    case audioIn
    case audioOut
    case toolCalling
    case thinking
    case reasoningLevel
}

Capability	Today
`.textIn` / `.textOut`	✅ Always on for LLMs.
`.imageIn`	✅ Image input — image-only VL models (Gemma 3/4-VL, Nemotron-VL, Idefics3, GlmOcr, FastVLM, Mistral3, Paligemma, …). Video-capable families add `.videoIn` separately.
`.videoIn`	✅ Video input — video-capable VL families (Qwen 2-VL, Qwen 2.5-VL, Qwen 3-VL, MiniCPM-V, SmolVLM2). Always declared together with `.imageIn`.
`.audioIn`	✅ Whisper STT + SenseVoice STT + Qwen-Omni audio-in.
`.audioOut`	✅ Kokoro TTS (iSTFTNet vocoder tail).
`.toolCalling`	Not declared by any family.
`.thinking`	Model emits a chain-of-thought trace (Qwen 3 thinking, DeepSeek-R1, etc.).
`.reasoningLevel`	Model honours a user-tunable reasoning-effort dial (GPT-OSS-20B, …).

Convenience sets:

Capability.textOnly       // [.textIn, .textOut]
Capability.textWithTools  // [.textIn, .textOut, .toolCalling]
Capability.speechToText   // [.audioIn, .textOut]   — Whisper, SenseVoice
Capability.textToSpeech   // [.textIn, .audioOut]   — Kokoro
Capability.omniAudio      // [.textIn, .audioIn, .textOut] — Qwen-Omni

Audio models

Audio models do not route through Model / ModelRegistry (which describe a text-in / text-out causal decoder). They load through AudioModelRegistry, which inspects config.json, picks the family (Whisper STT, SenseVoice STT, Kokoro TTS, Qwen-Omni), and reports the audio Capability set:

let loaded = try AudioModelRegistry.load(directory: dir)
switch loaded {
case .whisper(let m):    ...   // .speechToText
case .senseVoice(let m): ...   // .speechToText
case .kokoro(let m):     ...   // .textToSpeech
case .qwenOmni(let m):   ...   // .omniAudio
}

AudioModelRegistry.capabilities(forConfigAt:) reports the capability set without loading weights — useful for a model picker.

What each family declares

Family	`availableCapabilities`
`Llama.LlamaDense`	`[.textIn, .textOut]`
`Qwen3.Qwen3Dense`	`[.textIn, .textOut]`
`Mamba2.Mamba2Dense`	`[.textIn, .textOut]`
`FalconH1.FalconH1Hybrid`	`[.textIn, .textOut]`
`NemotronH.NemotronHHybrid`	`[.textIn, .textOut]`
`NemotronDiffusion.NemotronDiffusionDense`	`[.textIn, .textOut]`
`Granite4.Granite4Hybrid`	`[.textIn, .textOut]`
`Jamba.JambaHybrid`	`[.textIn, .textOut]`
`LFM2.LFM2Dense` / `LFM2MoE`	`[.textIn, .textOut]`
`Qwen35.Qwen35Hybrid`	`[.textIn, .textOut]`
`Gemma4.Gemma4Dense` / `Gemma4E` / `Gemma4MoE`	`[.textIn, .textOut]`
`GPTOSS.GPTOSSMoEVariant`	`[.textIn, .textOut]`

When a family adds a capability (e.g. the VL families add .imageIn, video-capable VL families add both .imageIn and .videoIn), the family file declares it and the loader allocates the corresponding subnet only if the user opts in.

`LoadOptions`

let model = try await Model.load(
    "mlx-community/Qwen3.5-0.8B-MLX-4bit",
    options: LoadOptions(
        capabilities: [.textIn, .textOut],
        kvCache: .raw,
        dispatchMode: .eager,
        prewarm: true,
        lazyCapabilities: true,
        revision: "main"
    )
)

Field	Default	Notes
`capabilities`	`Capability.textOnly`	What to load. `textIn` + `textOut` are always implicitly on. Disabled modalities skip weight allocation.
`kvCache`	`.raw`	Cache compression scheme — see kv-cache.md.
`dispatchMode`	`.eager`	Standard `MTLComputeCommandEncoder` per kernel. `.argumentBuffers` / `.icb` deferred.
`prewarm`	`true`	Run one no-op forward to compile PSOs before the first user-visible decode.
`lazyCapabilities`	`true`	Allow runtime `enable(_:)` / `disable(_:)` after load.
`revision`	`"main"`	HF branch / tag / commit.

Inspecting a loaded model

let model = try await Model.load("mlx-community/Qwen3-4B-4bit")

print(model.availableCapabilities)   // what the family supports
print(model.enabledCapabilities)     // what you opted into
print(model.config.modelType)        // "qwen3"
print(model.modelDirectory)          // resolved local snapshot

If you ask for a capability the family doesn’t expose, the loader throws ModelError.capabilityNotAvailable(.imageIn).

Lifecycle states

ModelLifecycleState:
  idle → downloading(Progress) → loading(LoadProgress)
       → loaded → ready
       (or failed(Error) at any stage)

Model.events is an AsyncStream<ModelLifecycleEvent> that emits each transition. The stream is multi-consumer-safe and finishes when the Model is deinitialized.

let model = try await Model.load("mlx-community/Qwen3.5-0.8B-MLX-4bit")

Task {
    for await event in model.events {
        switch event.state {
        case .downloading(let progress): print("downloading \(progress.fractionCompleted)")
        case .loading(let p):            print("loading \(p)")
        case .loaded:                    print("weights resident")
        case .ready:                     print("ready to generate")
        case .failed(let err):           print("failed: \(err)")
        default:                         break
        }
    }
}

print(model.currentState)  // sync snapshot — typically .ready by the time load() returns

currentState is a thread-safe snapshot of the latest emitted event. The stream is the source of truth for fine-grained progress.

Hot capability changes

try await model.enable(.imageIn)    // mmaps vision weights, builds encoder, prewarms
// ... use the model with images ...
try await model.disable(.imageIn)   // releases MTLBuffers, frees GPU residency

Each call emits per-capability lifecycle events through the same events stream. If lazyCapabilities = false was passed at load time, both calls throw — capabilities are then frozen at the load-time set.