# Slot lifecycle reference **URL:** \ **Category:** Reference\ **Created:** [August 22, 2026, 6:50am UTC](https://forum.hal0.dev/t/slot-lifecycle-reference/58 "2026-08-22T06:50:33Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![system](https://forum.hal0.dev/user_avatar/forum.hal0.dev/system/32/37_2.png) [@system](https://forum.hal0.dev/u/system)\ **Post date:** [August 22, 2026, 6:50am UTC](https://forum.hal0.dev/t/slot-lifecycle-reference/58/1 "2026-08-22T06:50:33Z")
The canonical state machine lives in `hal0.slots.state` and is used by `SlotManager`, the dashboard SSE stream, and the `state.json` persistence layer. ## States | State | Meaning | | --- | --- | | `OFFLINE` | Not running. No systemd unit active. | | `PULLING` | Model files are being downloaded or verified. systemd unit not yet started. | | `STARTING` | systemd unit has been started; waiting for the container to come up. | | `WARMING` | Container is up; the health probe is returning non-ready responses while the model loads into VRAM/GTT. | | `READY` | Passed the full health probe (non-empty `/v1/models` plus a sentinel completion) **and** , for a `type = "llm"` slot, the output-sanity probe below. Ready to serve. | | `SERVING` | An inference request is actively in-flight on this slot. | | `IDLE` | Reached two ways, both render identically on the wire: `warming → idle` (process up but `/v1/models` is empty — launched with `--model ""` or a missing file; routers must not treat this as ready), or `ready → idle` (no request for longer than the idle timeout — eviction candidate). | | `UNLOADING` | Graceful shutdown in progress. systemd stop issued; waiting for the container to exit. | | `ERROR` | Failed. Details in `state.json` and journald. A deliberate stop is not a failure: rendered units declare `SuccessExitStatus=143`, a plain SIGTERM stop on an older unit is recognised, and a cached `ERROR` converges to `OFFLINE` once the unit shows a clean stop or `systemctl reset-failed`. A crashed or crash-looping unit keeps its `ERROR`. | ```auto offline → pulling → starting → warming → ready ←──┐ │ ↑ │ │ ↓ │ └──→ idle ←──serving │ ↓ unloading → offline ↑ error ``` ## Legal transitions `LEGAL_TRANSITIONS` is the authoritative map `SlotManager` enforces: | From | Legal next states | | --- | --- | | `OFFLINE` | `PULLING`, `STARTING` | | `PULLING` | `STARTING`, `ERROR`, `OFFLINE` | | `STARTING` | `WARMING`, `ERROR`, `OFFLINE` | | `WARMING` | `READY`, `IDLE`, `ERROR`, `OFFLINE` | | `READY` | `SERVING`, `IDLE`, `UNLOADING`, `ERROR` | | `SERVING` | `READY`, `IDLE`, `ERROR` | | `IDLE` | `SERVING`, `UNLOADING`, `READY` | | `UNLOADING` | `OFFLINE`, `ERROR` | | `ERROR` | `OFFLINE`, `PULLING`, `STARTING` (re-loadable, subject to the crash-loop breaker) | ## Dispatchable states `DISPATCHABLE_STATES = {READY, SERVING, IDLE}` — the single source of truth used by `SlotManager`, the GPU arbiter, stack-apply, the Prometheus renderer, and the slot-view aggregator to decide whether a slot can take a request right now. ## Selection ranking When more than one candidate slot binds the same model, `_SELECTION_ORDER` ranks them: 1. `DISPATCHABLE_STATES` (ready/serving/idle) — answers now. 2. `{WARMING, STARTING, PULLING}` — will answer shortly. 3. `{OFFLINE, UNLOADING}` — needs a cold load. 4. `{ERROR}` — last resort (already failed once). Unknown states rank as `ERROR`. ## Typed errors Each is an HTTP-status-bearing error under the `slot.*` namespace: | Error | HTTP status | Meaning | | --- | --- | --- | | `SlotNotFound` | 404 | — | | `IllegalSlotTransition` | 409 | Attempted transition not in `LEGAL_TRANSITIONS`. | | `SlotNotReady` | 503 | — | | `SlotSpawnFailed` | 500 | — | | `SlotHealthFailed` | 503 | — | | `SlotOutputSanityFailed` | 500 | The server is healthy but its output is not language — see Output-sanity gate below. | | `SlotCrashLooping` | 503 | Reload blocked during exponential backoff, or after too many consecutive failures — see Crash-loop breaker below. Carries `retry_after_s`. | | `SlotTerminateTimeout` | 504 | Stop didn’t return in time — the request unblocks anyway. | | `SeamTimeout` (`slot.seam_timeout`) | 504 | A bounded `systemctl`/podman seam call (Quadlet write, `daemon-reload`, `reset-failed`, the spawning `systemctl restart`) did not return within its budget. `details` carries `command`, `budget_s` and `last_state`; the slot lands in `ERROR` with the same message. | | `SlotConfigError` | 400 | — | | `slot.tuning_key_denied` | 400 | A slot write carried a key that belongs on the profile. | | `slot.profile_missing` | 422 | The slot names a profile the catalog does not have; the launch is refused. | | `NpuExclusivityViolation` | 409 | Only one `device=npu, type=llm` slot with a bound model is allowed at a time (AMD XDNA hardware context limit). | | `SlotPinned` | 409 | A pinned slot needs `?force=true` to unload or delete. | Health-probe resolution on timeout: unresolved health checks resolve to `WARMING`, not `ERROR` — an ambiguous outcome is treated as “still loading,” not “failed.” > **Note: Client-side lifecycle timeouts** > > The CLI and the MCP admin bridge derive their read timeout for `load` / `unload` > / `restart` / `swap` from the server’s own worst-case budget > (`hal0.slot_lifecycle_budget`) rather than a fixed number — a load charges the > health poll, the output-sanity gate’s CPU-lane budget, and up to three > in-series eviction unloads, all with a 1.15x overhead margin. A restart or swap > charges both an unload and a load. This is why a slow (e.g. CPU-only) box’s > load can legitimately take several minutes before a client gives up. ## Output-sanity gate `/health` proves the runner bound its port and mapped its weights. It does not prove the backend turns those weights into coherent text — a GPU backend that mis-computes answers `/health`, streams tokens and closes the SSE stream cleanly while emitting garbage. So a `type = "llm"` slot on the container/llama-server lane runs one last check before it is published as `READY`: a single greedy completion whose answer is known (`The capital of France is` → `Paris`), once per load, while the slot still holds its lock and is not yet dispatchable. A wrong answer on the raw `/completion` endpoint buys one retry through `/v1/chat/completions` (with room for a reasoning model’s `` block) before the load is refused. A **wrong answer** is terminal for that load: the slot lands in `ERROR` with a message naming the probe, the expected token and what the model actually said, its unit is stopped (best-effort — a stop that does not return is reported in the same message), and it will **not** be re-adopted as `READY` by a later `/api/slots` poll or by the boot-time reconcile. A passing gate on a subsequent load is what clears the verdict. A **probe that never answers** is treated differently. A timeout says the answer did not arrive inside a wall clock, not that the answer was wrong, so the slot resolves to `WARMING` — the same policy the health wait already applies to its own ambiguous outcome. Nothing is published as `READY`, nothing is stamped, the unit is left alone, and the next load re-runs the gate. Slow hardware is judged on its answer; the budget widens to match on a `device = "cpu"` slot, which decodes far slower than the accelerated lane. > [@](#): > > **Note** > > “Inconclusive, so let it through” remains the one thing the gate never does: an > unanswered probe can only produce `WARMING`, never `READY`. Embedding, reranking, TTS, STT and image slots are never gated (they cannot serve a chat completion), and the FLM/NPU lane keeps its own `verify_inference` sentinel. Escape hatches, both default-ON: | Scope | How | | --- | --- | | Whole box | `HAL0_SLOT_OUTPUT_SANITY=0` in the api environment | | One slot | `output_sanity = false` in the slot TOML, or via `PUT /api/slots/{name}/config` | ## Crash-loop breaker A slot in `ERROR` is freely re-loadable, but not at request cadence. `SlotManager` tracks consecutive load failures per slot (`hal0.slots.manager`) and throttles further `load()` calls through three states: | State | When | Next retry | | --- | --- | --- | | `backoff` | 1–4 consecutive failures | Exponential: `min(5s * 2^(failures-1), 300s)` | | `parked` | 5 or more consecutive failures | `min(300s * 2^(failures-5), 3600s)` — loads are refused outright | | `half-open` | The current backoff or parked cooldown has elapsed | One trial load runs; success closes the breaker, a failed trial re-parks it at the next cooldown step | A load the breaker refuses raises `SlotCrashLooping` (503, see above) with **no** state transition — no `STARTING` event, no journal row, no `systemctl` call — and carries `retry_after_s` for the response’s `Retry-After` header. Any operator action (a manual load, a restart, or a config edit) resets the breaker immediately, independent of the cooldown clock; a load that converges healthy clears it too. The manager stamps the live view onto every slot snapshot as `metadata.breaker = {state, failures, retry_after_s}` while the breaker is non-closed, and omits the key entirely once it’s closed — so the field’s presence is the render signal. The dashboard shows this as a chip on both the slot card and the Edit-slot drawer’s state strip: amber “retry in Ns” while backing off, red “parked · N failures” once parked, and a neutral “trial pending” during the half-open window. ## Container image status `image_status` on a slot snapshot is one of `present`, `pulling`, `missing`, `unknown`, or `not-configured` (`hal0.slot_view.image_status_for`). Only `unknown` renders a dashboard chip: it means hal0 could not read the container image store at all — the rootful `hal0-podman-ro` seam returned an error, the sudoers grant is absent, or the read probe timed out — or could not resolve the slot’s image in the first place (an unreadable `profiles.toml`; the log carries `reason=resolve-error`) — which is a different claim from `missing` (podman was asked and said the image isn’t there). `present` and `not-configured` are the quiet, correct states, and `pulling` already has its own progress bar. Seeing the “image ?” chip is the cue to run `hal0 doctor all` and check the `seams` row, then the hal0-api journal for `slot_view.image_probe_failed` (a `reason=resolve-error` there means the image could not be resolved, not that the seam is broken). There is no standalone `hal0 doctor seams` command. A slot or its container that was stopped cleanly no longer paints as `crashed` on either the slot card or the dashboard’s slot list — only a unit that actually failed (`state = "error"` or a `crashed` container status) gets the red treatment; a graceful stop renders the same neutral “stopped” state as any other offline slot. ## Persistence `SlotStateRecord` (`name`, `state`, `model_id`, `port`, `updated_at`, `message`, `extra`) is written atomically (tempfile + `fsync` + `os.replace`) to `/var/lib/hal0/slots//state.json`, where `` is the slot’s name (or, after `migrate-id-keying`, its stable numeric id — see `hal0.slots.layout`), so SSE readers never observe a torn write. ## Self-managed providers `SELF_MANAGED_PROVIDERS = {kokoro, qwen3tts, moonshine, vibevoice}` — these providers serve a baked-in model, so no explicit `model_id` is needed to dispatch to them (`provider_requires_model()` returns `False` for this set). This set must stay in sync with the dashboard’s own `SELF_MANAGED_PROVIDERS` constant. ## Activation There is **no `enabled` field** on `SlotConfig`. A non-empty `model.default` is the activation signal — it is what makes a slot routable and loadable. Boot autostart is a separate, explicit setting: `autoload`. The generated Quadlet unit carries `[Install] WantedBy=hal0.target` **only when `autoload = true`** , and that stanza is the only thing that starts a slot at boot. With `autoload = false` the unit still exists and is start-able (load/swap use `systemctl restart`, which never consults `[Install]`) — nothing just pulls it up on reboot. Binding a model no longer implies a boot start. Slots written before the field existed (key absent on disk) migrate to `autoload = true` when they have a bound model, and `false` otherwise, so an upgrade preserves the old behaviour until you toggle it. Slots created through `POST /api/slots` always persist an explicit value, defaulting to `false`. Editing `autoload` via `PUT /api/slots/{name}/config` re-renders the on-disk unit, so the change takes effect from the next boot without a manual reload. ## Eviction priority Every non-pinned slot is an eviction candidate. `priority` (int, `0–100`, default `50`) orders the victims for both memory-pressure eviction (`SlotReaper.pressure_evict_once`) and pre-load eviction (`preload_evict`): the **lowest priority is unloaded first** , with least-recently-used as the tie-break inside a priority tier. `priority = 100` is not a pin — it is merely evicted last; `pinned = true` is what exempts a slot entirely, as do the default-pinned anchors (`agent` and friends). A slot with a request in flight is never a victim, and only resident slots (`READY`/`IDLE`) are considered at all. > [@](#): > > **Caution** > > The old `lru = true` eviction opt-in is retired. The eviction sweep ignores the key (with a > one-time `slot.lru_flag_retired` warning per slot), > and a slot file that still carries it fails the strict slot load with the replacement named — use > `priority`/`pinned` instead. Slots that never set `lru` are now evictable where they > previously were not. ## What a slot write may carry A slot holds placement (device, layers, threads, runner), its model and its profile. The profile holds every tune; the model row holds facts about its file. Every write of a slot file goes through `guard_slot_write_payload` (`hal0.slots.config_write`): `SlotManager.create()` runs it on the new slot, and `SlotManager.update_config()` and stacks-apply run it through `reconcile_and_guard_slot_config`. The slot HTTP routes (`POST /api/slots`, `PUT /api/slots/{name}/config`) also check the body in `api.routes.slots` before anything else, so their “belongs on a profile” answer comes before a generic unknown-key error. They refuse: - **A tuning key** (`[model].context_size` / `ctx_size`, `[server].extra_args`, `parallel`, `chat_template`, `workers`, `mtp`, `speculation`, `enable_thinking`, `vision`) with a 400 `slot.tuning_key_denied` (“belongs on a profile”). Set it on the slot’s profile (`PUT /api/profiles/{name}`) or pick another profile. - **A llama-server option in `[server].env`** (`LLAMA_ARG_*`, `LLAMA_API_KEY`, `HF_TOKEN`, `MTMD_BACKEND_DEVICE`) with a 400 `slot.llama_env_denied`. llama-server reads each of them as a launch option: a `LLAMA_ARG_*` tune goes in the profile’s `flags` (when that flag is allowed there), and the other three set options hal0 owns (see [Config schema](https://forum.hal0.dev/t/config-schema-reference/51)). - **A key removed from the slot schema** (for example `enabled`) with a 400 `slot.removed_key_denied`. To rebind a slot’s model, send `PUT /api/slots/{name}/config` with `{"model": {"default": ""}}`, or use `slot_edit` or `model_swap` over MCP. `PATCH /api/slots/{name}/defaults` no longer exists. A slot file on disk that still carries a tuning key is refused at launch with `slot.unconverted`, and the typed slot-config loader rejects it too (`hal0 config validate` names the file and the keys), until you delete the keys from the file and put their values on a profile. `GET /api/slots` reports such a slot as `conversion_blocked` (`reason`, `fix`). ## Launch composition and effective context A llama-server slot’s command is built from four layers, lowest precedence first, then deduplicated last-wins: `base` (`--host`, `--port`, `--model`, `--alias`, `--ctx-size`), `model` (family quirks, `--jinja`, `--chat-template-file`, `--mmproj` when the profile’s `vision` is `auto`, and `--spec-type` with `--spec-draft-model` when a drafter or MTP heads are in effect), `profile` (the draft bundle for that kind, then the profile’s `flags`) and `slot` (`--device` (`ROCm0`, `Vulkan0`, `CUDA0` or `none`) from the slot’s device, `-ngl`, `--threads`, `--metrics`, and, only for embedded MTP heads, `--spec-draft-device` from the slot’s device; a paired drafter file runs on the same devices as the model). `GET /api/slots/{name}/resolved` returns the deduplicated `argv` and, for each surviving flag, the layer that set it. A slot with no llama-server command (or one that cannot launch) answers `argv: null` with an empty `provenance`. `GET /api/slots` and `GET /api/slots/{name}` carry `ctx_effective` (tokens) and `ctx_source` (`profile`, `native` or `fallback`) for `llm`, `embedding` and `reranking` slots. Both come from the same resolver the launch path uses. The slot views do not return `ctx_max`, `chat_template`, `rope_freq_base`, `workers`, `parallel` or `llamacpp_args`. For the CLI surface that drives slot lifecycle transitions directly, see the [CLI reference](https://forum.hal0.dev/t/hal0-cli-reference/50), `hal0 slot`.