# Slots (hal0 docs) **URL:** \ **Category:** Concepts\ **Created:** [August 22, 2026, 6:49am UTC](https://forum.hal0.dev/t/slots-hal0-docs/28 "2026-08-22T06:49:33Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![system](https://forum.hal0.dev/user_avatar/forum.hal0.dev/system/32/37_2.png) [@system](https://forum.hal0.dev/u/system)\ **Post date:** [August 22, 2026, 6:49am UTC](https://forum.hal0.dev/t/slots-hal0-docs/28/1 "2026-08-22T06:49:33Z")
A **slot** is hal0’s unit of served inference: one named place that runs one model. Behind every slot is a single podman container, and around every slot is a lifecycle the platform manages for you. Slots are what the dispatcher routes to and what the GPU arbiter coordinates. A slot owns _where_ a model runs — device, GPU layers, threads, runner — and names the model and the profile it runs with. The profile supplies the tune; a slot can carry the six sparse overrides described below. See Who owns what for the ownership boundary. ## Per-slot tune overrides A slot can carry a sparse `[tune]` table over its profile: `ctx_size`, `enable_thinking`, `reasoning_effort`, `reasoning_budget`, `vision` and `speculation`. Missing keys inherit the profile. This applies to LLM slots outside the NPU lane. Models still own file facts and profiles own free flags. For example, serve the same file in a fast slot with an 8192-token context and a full slot with a larger window, both inheriting one profile. A dedicated memory-extraction slot can use thinking off and low effort; validate JSON conformance with the actual model and runner. Use the drawer’s Tune block, `hal0 slot edit`, or the config API. Thinking changes apply to new requests; other rows require restart. Profile flags `--reasoning on|off` pin thinking even over a hand-written override. ## One slot, one container Each slot is backed by a systemd template unit named `hal0-slot@.service`, generated as a Podman **Quadlet** unit (a `.container` file of typed `[Container]` keys — `Image=`, `AddDevice=`, `Volume=`, `PublishPort=`, and so on — rather than a hand-assembled `podman run` command line). Podman’s systemd generator turns that into the `[Unit]`/`[Service]` skeleton and the actual `podman run` invocation. Quadlet auto-removes any prior container of the same name on start, so there’s no more `--replace` / `ExecStartPre rm` dance to guard against a stale container name. That launches exactly one container — `hal0-slot-` — serving one model on a loopback port, publishing its port on `127.0.0.1` only. **`SlotManager`** owns this. It renders the unit, starts and stops it, tracks state, and registers the running container as an internal upstream so the dispatcher can forward to it. A slot’s TOML supplies its name, port, hardware grid, a reference to a [profile](https://forum.hal0.dev/t/capabilities-profiles/25) for the tune, and the model it runs; the model itself, with the facts about its file, comes from the registry. > [@](#): > > **Note** > > The container runtime is Podman only. Quadlet is Podman’s declarative unit > generator and has no Docker equivalent, so Docker is unsupported — the old > Docker fallback has been removed. ## Who owns what: slot, model, profile A launch is described by three objects, and every setting has exactly one owner: | Object | Holds | | --- | --- | | **Model** (registry row) | Facts about its file: path and file set, size, quantization, architecture, native context window, chat template, multimodal projector (`mmproj`), whether the file has MTP heads, and whether its chat template works with `--jinja`. | | **Profile** | Base tune: free llama-server `flags` (KV-cache types, batch size, sampling, reasoning, `--parallel`), `ctx_size`, `enable_thinking`, `reasoning_effort`, `reasoning_budget`, `vision` (`auto` or `off`), `speculation` (`auto`, `off` or `ngram`), and an optional `runner`. | | **Slot** | Placement and lifecycle: `device`, `gpu_index`, `n_gpu_layers`, `threads`, `binary` (a key into the `RUNNER_IMAGES` registry — the dashboard’s slot drawer calls this the slot’s **Runtime** ), the `image_pin` escape hatch, and the port. Plus which model and which profile it runs. This is the “hardware grid”: `[device · NGL · THREADS · BINARY]`. | A slot carries no free llama-server flags. It may override context, thinking, effort, budget, vision and speculation under `[tune]`. For different sampling or KV-cache flags, make separate profiles and point each slot at its own: ```sh hal0 profile create agent-32k --from agent --ctx-size 32768 hal0 slot edit scratch --profile agent-32k ``` Every image hal0 ships is a `RUNNER_IMAGES` registry entry — the default AMD GPU runtime (`standard`, titled **Standard** , the only entry allowed both a ROCm and a Vulkan lane) plus the CPU, NPU, CUDA and single-purpose speech and image runtimes, each pinned in `manifest.json`. `image_pin` is a debug-only override (a custom build, an A/B test, a rollback) and never appears as a pickable Runtime option. See [Capabilities & profiles](https://forum.hal0.dev/t/capabilities-profiles/25) for the full Runtime picker and how a profile can carry one. The split is enforced at every write path. Sparse `[tune]` overrides are the six exceptions. A slot write carrying a legacy tuning key outside that table — `[model].context_size` / `ctx_size`, `parallel`, `chat_template`, `workers`, `mtp`, `speculation`, `enable_thinking`, `vision` or `[server].extra_args` — is rejected with `slot.tuning_key_denied`, whose message says the key belongs on a profile. The guard runs the other way too: a profile’s flags may not set what the slot owns (`--n-gpu-layers`/`-ngl`, `--device`/`-dev`, `--threads`/`-t`), what hal0 manages (`--model`, `--host`, `--port`, `--alias`, `--ctx-size`), or a fact about the model file (`--jinja`, `--chat-template*`, `--mmproj`, `--no-mmproj`). A slot’s `[server].env` may not set `LLAMA_ARG_*` variables either, because llama.cpp reads those as launch options (`slot.llama_env_denied`). The full list of refused flags is in [Edit configuration](https://forum.hal0.dev/t/edit-configuration/32#launch-flags-are-screened-on-the-profile). > [@](#): > > **Caution** > > This partition is a hard invariant, but its enforcement has a real track > record of needing to be retrofitted onto _new_ write paths after the fact — > the stacks-apply engine, in particular, bypassed it more than once before a > single shared gate (`guard_slot_write_payload`) was made the one pipeline > every writer (`SlotManager.create`, `SlotManager.update_config`, and stacks > apply) routes through. If you’re adding a new way to write a slot config, > route it through that shared gate rather than hand-rolling a merge. A slot file that still carries a tuning key (a file written before profile-centric config, or edited by hand) fails to load with `slot.unconverted` and the slot does not launch. `hal0 config validate` names the file and the keys. Delete the keys from the slot file, put their values on a profile (`hal0 profile create`), and name it with `profile = ""`. A slot with no `profile` runs its type’s seed profile (`embedding` and `reranking` carry their mode flag; an `llm` slot runs `chat`). A slot that names a profile that doesn’t exist refuses to launch with `slot.profile_missing`, rather than quietly running some other tune. ### How the launch command is assembled A llama.cpp slot’s launch argv is built from four layers, lowest precedence first, then deduplicated last-wins per flag: 1. **base** — `--host`, `--port`, `--model`, `--alias` and `--ctx-size`, always present and managed by hal0. 2. **model** — facts about the file: `--chat-template-file`, `--mmproj` (only when the model has a projector and the profile’s `vision` is `auto`), `--jinja` (added when the runner supports it, unless the model’s `jinja` fact is `false` or the profile is an embedding or reranking one), and any family quirks. 3. **profile** — the draft bundle for the speculation kind in effect, then the profile’s `flags`. A profile’s own `--spec-draft-*` values follow the bundle, so they win. 4. **slot** — `-ngl`, `--threads`, `--metrics`, and, when a draft type is in effect, the draft device derived from the slot’s own device. `GET /api/slots/{name}/resolved` returns the resolved argv with each flag labelled `base`, `model`, `profile` or `slot`, and the slot drawer’s **Resolved command** panel shows the same thing. See [Capabilities & profiles](https://forum.hal0.dev/t/capabilities-profiles/25#flag-precedence-how-a-launch-command-is-assembled). ### The context window comes from the profile The window a slot launches with is its profile’s `ctx_size`. A profile with no `ctx_size` runs the model’s native window, capped at 32768; if the model reports no native window, it runs 8192. Slot views report the result as `ctx_effective` and where it came from as `ctx_source` (`profile`, `native` or `fallback`), and the slot drawer shows both read-only under the Profile row. Change the window by editing the profile (`hal0 profile edit --ctx-size N`) or by pointing the slot at another profile; slots running the same profile pick up a new `ctx_size` at their next restart. An FLM (NPU) slot reads the same `ctx_size`; with none set it runs 8192. The shipped seed profiles set `ctx_size` where a slot needs more than the default: `agent`, `utility` and `brain` carry 65536 (the bundled Hermes agent needs at least 64,000 tokens), and `embedding` and `reranking` carry 4096. When a box can’t afford a seed’s window, the installer writes a custom profile named `-ctx` and points the slot at it. ## Pinning a slot to one GPU On a multi-GPU host, a slot’s `gpu_index` (an integer, `0` and up) pins it to a single card instead of the default “all GPUs” behaviour. The env or device mapping it emits is backend-specific and is derived from the slot’s own `device`, never from a profile’s fields: | Device | What `gpu_index` does | | --- | --- | | `gpu-rocm` | Sets `HIP_VISIBLE_DEVICES` and `ROCR_VISIBLE_DEVICES` to the index. | | `gpu-vulkan` | Sets `GGML_VK_VISIBLE_DEVICES` to the index. | | `gpu-cuda` | Maps only that GPU in via CDI (`--device nvidia.com/gpu=`) and sets `CUDA_VISIBLE_DEVICES=0` inside the container, since the CDI mapping already exposes just the one card as ordinal 0. | Leaving `gpu_index` unset (the default) targets every GPU the backend can see, unchanged from single-GPU behaviour. Any explicit `[server].env` entry you set still wins over the derived visibility variables — they’re merged underneath, never over. ## The lifecycle state machine A slot moves through an explicit set of states. The `SlotState` values are also the wire representation streamed to the dashboard, so what you see is a real transition, not a systemd snapshot: _The slot lifecycle transitions visualized._ | State | Meaning | | --- | --- | | `offline` | Not running; no active unit. | | `pulling` | Model files are downloading or being verified. | | `starting` | The unit has started; waiting for the container to come up. | | `warming` | Container is up; the model is loading into the GPU/GTT pool. | | `ready` | Health probe passed; ready to serve. For a `type = "llm"` slot, this also requires passing an output-sanity probe — a known-answer completion that catches a backend serving garbage while `/health` reads green. See the [Slot lifecycle reference](https://forum.hal0.dev/t/slot-lifecycle-reference/58) for the gate’s mechanics. | | `serving` | A request is actively in flight. | | `idle` | Up but not serving — either warm-but-quiet, or up with no loadable model. | | `unloading` | Graceful shutdown in progress. | | `error` | Failed; details in state.json and the journal. | Transitions are constrained — only legal edges are allowed — and persisted atomically so a dashboard reader never observes a half-written state. The **dispatchable “ready set”** is `ready`, `serving`, and `idle`: those are the states a request can be routed to. Any other state means the slot is mid-lifecycle, and the dispatcher returns a structured `slot.loading` 503 with a `Retry-After` hint rather than failing with a raw connection error. > [@](#): > > **Tip** > > `idle` is two situations the dashboard renders the same way: a slot that was > ready but has had no traffic past its idle timeout (a candidate for unload), or > a process that started but has no model it can actually serve. Both are > distinct from `ready` so routing never picks a slot that can’t fulfil a request. ### Serving and idling When the dispatcher forwards a request to a slot, it wraps the call in the slot’s **serving** state: the first concurrent request moves the slot `ready`/`idle` → `serving`, and the last one to finish moves it back to `ready`. For a streamed response, the slot stays `serving` until the stream drains. A background idle monitor demotes a quiet `ready` slot to `idle` after its idle timeout (300 seconds by default), so the dashboard can distinguish “warm and working” from “warm but quiet.” ### The crash-loop breaker A slot that keeps failing to load isn’t retried at request cadence. After a load failure, `SlotManager` throttles further attempts with an exponential backoff; once a slot racks up five consecutive failures it is **parked** — loads are refused outright until a cooldown elapses, at which point a single **half-open** trial load runs. Success closes the breaker; failure re-parks it at a longer cooldown. A manual load, restart, or config edit resets the breaker immediately regardless of the clock. The dashboard surfaces the live state as a chip on the slot card and in the slot drawer; see the [Slot lifecycle reference](https://forum.hal0.dev/t/slot-lifecycle-reference/58) for the exact timing. ## Single-flight dispatch Two safety mechanisms keep concurrent traffic from colliding: - **Per-slot lock.** Load, unload, and restart for a given slot serialize behind one lock, so two requests can never race to start or swap the same slot’s container. - **Coalesced prefetch.** When the dispatcher needs to fetch a cold upstream’s model list, identical concurrent fetches share a single execution — a hundred simultaneous requests for the same uncached model trigger one upstream call, not a hundred. They all receive the same result (or the same error). The effect is that a burst of requests for a model that isn’t loaded yet produces exactly one load, and everyone waits on that load rather than starting their own. ## Keeping the launch command honest across updates A slot’s systemd unit bakes its launch argv **at load time** — updating hal0 changes the code that would render a slot’s command, but not the unit file that already rendered it. Left alone, a slot that was running before an update keeps running the pre-update flags through `systemctl restart`, crash-restarts, and even reboots, until something forces a fresh render. hal0 closes most of that gap automatically: after an update finishes (and during `install.sh`’s update flow — now the single user-facing entry point for both fresh installs and updates), it re-renders every existing slot’s unit file through current code and issues one `daemon-reload`. Crucially, it **never bounces a running unit** to do this — a slot that’s already serving keeps serving on its old argv; the fresh command only takes effect the next time that slot starts, whether that’s a manual restart, a crash-restart, or a reboot. A slot whose fresh render is byte-identical to what’s on disk is skipped as a no-op, and a per-slot render failure is logged and skipped rather than aborting the update. That still leaves a window — a slot that hasn’t restarted since the update is running old argv and there’s no way to tell just by looking at it. The dashboard closes that window with a **resolved-command drift indicator** : each slot card compares the container’s actual, live `podman inspect` argv against a fresh render from current config across the flags that matter most (context size, model, alias, batch/ubatch size) and flags a mismatch. Seeing drift on a slot you haven’t touched is the cue that it’s still running an older command and could use a restart. ## The GPU arbiter The reference hardware has a single iGPU sharing one unified memory pool, so two GPU workloads can’t both hold their weights at once. The **GPU arbiter** makes that exclusivity explicit. It sorts GPU-backed container slots into two groups — an **llm** group (GPU chat/embedding slots) and an **img** group (the ComfyUI image-generation slot) — and only one group may hold the GPU’s memory at a time. NPU and CPU slots are never arbitrated; they don’t contend for the iGPU. When the GPU is in image mode, the arbiter does not dispatch to an llm-group slot at once. A request for one waits for the image work to end and for the reload, up to 120 seconds in all. Only when image mode is pinned, ComfyUI’s queue cannot be read, or the wait runs out does the request get a structured `gpu.image_mode` 503 (with a `Retry-After`) instead of failing silently. Image mode lasts only while image generation runs: a `/v1/images/generations` request from before the switch until its render returns (Open WebUI and the omni loop use it too), or a render in ComfyUI’s queue from its editor. The image container itself stays resident — what’s exclusive is the GPU **memory** , not the container. About ten seconds after the last render, the arbiter frees the diffusion models and reloads the LLM slots it had stopped; an LLM request that arrives sooner starts that reload at once and waits for it rather than getting the 503. A render is never cut short: while one is running or queued, the reload waits. Pinning image mode (`POST /api/comfyui/pin`, or `pin` on the switchover) holds it until you unpin it. > [@](#): > > **Caution** > > The single NPU LLM anchor is exclusive too, but for a different reason: the AMD > XDNA hardware context admits exactly one NPU LLM at a time. hal0 enforces this — > enabling a second `device=npu` chat slot is rejected. ## Dedicated embed and rerank lanes `embed` and `rerank` each get their own tuned profile and their own podman container — they are never combined on one llama-server instance. That’s a hard constraint, not a style choice: passing both `--embedding` and `--reranking` to the same llama-server yields all-zero scores. Reranking is provisioned as its own slot, internally named `rerank`, distinct from the `embed` slot it’s paired with in the capability catalogue. Both lanes are tuned for how pooled scoring actually works: `-ub 8192` matches their batch size because a pooled embedding or rerank pass has to run the whole input in one physical micro-batch — a smaller ubatch truncates or fails on longer inputs. ## Speculation: the model carries the draft, the profile decides Speculative decoding (a drafter, or the model’s own MTP draft heads) takes two inputs, one from the model and one from the profile: - **The model’s facts: a paired drafter, and whether the file has MTP heads.** A drafter is a separate file paired like the projector (see [Drafters](https://forum.hal0.dev/t/pull-and-register-models/39#drafters)). For the heads, an explicit `mtp` fact on the model row wins in either direction; left unset, hal0 falls back to the registry’s `mtp` tag when present. There is no filename- or id-based sniff — an uncurated pull with neither the tag nor an explicit fact is simply not MTP-eligible. The dashboard’s model drawer records this as the **MTP heads** file fact; it is not a switch for whether a slot drafts. - **The profile’s choice: `speculation = "auto"`, `"off"` or `"ngram"`.** `auto` (the default) drafts with the paired drafter when the runner can run it, else with the heads when the model has them and the runner supports MTP; `off` never speculates; `ngram` uses n-gram lookup and needs no file. This field used to be `mtp`; `hal0 update` rewrites it once, and an API or CLI write that still sends `mtp` is refused with a 422 naming `speculation`. A slot has no MTP setting of its own (`mtp` and `speculation` are among the keys `slot.tuning_key_denied` refuses). To turn MTP off for one slot, give it a profile with `speculation = "off"` (`hal0 profile create --from --speculation off`). Under `auto`, a model without MTP heads simply doesn’t draft; hal0 logs `mtp.auto_off_model_ineligible` rather than launching a draft that would crash llama-server (`context type MTP requested but model doesn't contain MTP layers`). When speculation is in effect, hal0 adds a tuned draft-flag bundle for the kind in use to the profile layer of the launch command, and the draft device comes from the slot’s own device. A profile’s own `--spec-draft-*` flags override the bundle’s values. ## Seeded slots and aliases Every install starts with a set of **seeded slots** so the common capabilities have a home before you create anything: `flm`, `tts`, `rerank`, `utility`, `img`, `agent`, `brain`, `qwen3tts`, `coder`, and `embed`. When the NPU runtime (FastFlowLM) is present, two more shadow slots are seeded: `flm-stt` and `flm-embed`, riding the NPU chat anchor’s own `flm` slot. Seeded slots are reserved — you can’t create a conflicting name, and deleting one requires `--force` (see [Manage slots](https://forum.hal0.dev/t/manage-slots-hal0-docs/38)), which an install or update reconcile may later re-seed. Each seeded slot names a seed profile that carries its tune: `agent` runs `agent`, `utility` runs `utility`, `coder` runs `coding`, `brain` runs `brain`, `embed` runs `embedding`, `rerank` runs `reranking`, `flm` runs `flm`, `tts` runs `kokoro`, `qwen3tts` runs `qwen3-tts`, and `img` runs `comfyui`. The two canonical LLM slots are **`agent`** (the capable default — the dispatch fallback anchor every routing chain ends in) and **`utility`** (the cheap helper, seeded on every install so tasks like memory extraction always have a cheap local model to target). The slot name is the routing key. Each slot carries a **type** (`llm`, `embedding`, `reranking`, `transcription`, `tts`, or `image`) and a **device** preference (`gpu-rocm`, `gpu-vulkan`, `gpu-cuda`, `cpu`, or `npu` — `gpu-cuda` is the experimental path for non-Strix-Halo NVIDIA hosts). hal0 keeps one back-compat **alias** so older callers keep working: `embed-rerank` (the rerank capability slot’s briefly-used old name) resolves to `rerank`. Aliases are never written to disk and never appear in slot listings — they’re a transparent translation at dispatch time. Addressing a slot by its name (e.g. `model: "agent"`) is the stable way to pin a co-resident model: the name doesn’t change when you swap the underlying model file. hal0 also resolves `hal0/`-style virtual model names — `hal0/agent`, `hal0/utility`, `hal0/npu`, and the same pattern for any custom `type: llm` slot (`hal0/research`, say) — to whichever slot is actually loaded. As of v1.0 these canonical names are no longer advertised as separate entries in `/v1/models`: each enabled LLM slot’s own alias already covers its listing, so a `hal0/` request is resolved on demand at dispatch time instead of being double-listed. ## The NPU trio The AMD XDNA NPU is driven by FastFlowLM (FLM), and one detail shapes how hal0 uses it: a **single `flm serve` process serves three modalities at once** — chat, speech-to-text, and embeddings. hal0 calls this the **NPU trio**. See [Strix Halo](https://forum.hal0.dev/t/strix-halo-hal0-docs/30) for the hardware reason this exists. ## Stacks — declarative slot bundles A **stack** is a preconfigured slot + profile + model bundle you can apply atomically. See [Stacks](https://forum.hal0.dev/t/stacks-hal0-docs/29) for the full model — the TOML shape, apply semantics, and an honest maturity read. ## Where to go next - [Capabilities & profiles](https://forum.hal0.dev/t/capabilities-profiles/25): How a slot’s profile shapes its tune, and why hardware no longer lives there. - [Stacks](https://forum.hal0.dev/t/stacks-hal0-docs/29): Applying a whole set of slots as one named, portable unit. - [Architecture](https://forum.hal0.dev/t/architecture-hal0-docs/24): How slots sit under the dispatcher and API. - [Strix Halo](https://forum.hal0.dev/t/strix-halo-hal0-docs/30): The unified-memory hardware the arbiter coordinates.