Slot lifecycle reference

[system]

The canonical state machine lives in hal0.slots.state and is used by SlotManager,
the dashboard SSE stream, and the state.json persistence layer.

States

State	Meaning
OFFLINE	Not running. No systemd unit active.
PULLING	Model files are being downloaded or verified. systemd unit not yet started.
STARTING	systemd unit has been started; waiting for the container to come up.
WARMING	Container is up; the health probe is returning non-ready responses while the model loads into VRAM/GTT.
READY	Passed the full health probe (non-empty /v1/models plus a sentinel completion) and, for a type = "llm" slot, the output-sanity probe below. Ready to serve.
SERVING	An inference request is actively in-flight on this slot.
IDLE	Reached two ways, both render identically on the wire: warming → idle (process up but /v1/models is empty — launched with --model "" or a missing file; routers must not treat this as ready), or ready → idle (no request for longer than the idle timeout — eviction candidate).
UNLOADING	Graceful shutdown in progress. systemd stop issued; waiting for the container to exit.
ERROR	Failed. Details in state.json and journald. A deliberate stop is not a failure: rendered units declare SuccessExitStatus=143, a plain SIGTERM stop on an older unit is recognised, and a cached ERROR converges to OFFLINE once the unit shows a clean stop or systemctl reset-failed. A crashed or crash-looping unit keeps its ERROR.

offline → pulling → starting → warming → ready ←──┐
                                   │      ↑       │
                                   │      ↓       │
                                   └──→ idle ←──serving
                                          │
                                          ↓
                                      unloading → offline
                                          ↑
                                        error

Legal transitions

LEGAL_TRANSITIONS is the authoritative map SlotManager enforces:

From	Legal next states
OFFLINE	PULLING, STARTING
PULLING	STARTING, ERROR, OFFLINE
STARTING	WARMING, ERROR, OFFLINE
WARMING	READY, IDLE, ERROR, OFFLINE
READY	SERVING, IDLE, UNLOADING, ERROR
SERVING	READY, IDLE, ERROR
IDLE	SERVING, UNLOADING, READY
UNLOADING	OFFLINE, ERROR
ERROR	OFFLINE, PULLING, STARTING (re-loadable, subject to the crash-loop breaker)

Dispatchable states

DISPATCHABLE_STATES = {READY, SERVING, IDLE} — the single source of truth used by
SlotManager, the GPU arbiter, stack-apply, the Prometheus renderer, and the
slot-view aggregator to decide whether a slot can take a request right now.

Selection ranking

When more than one candidate slot binds the same model, _SELECTION_ORDER ranks them:

1. DISPATCHABLE_STATES (ready/serving/idle) — answers now.
2. {WARMING, STARTING, PULLING} — will answer shortly.
3. {OFFLINE, UNLOADING} — needs a cold load.
4. {ERROR} — last resort (already failed once).

Unknown states rank as ERROR.

Typed errors

Each is an HTTP-status-bearing error under the slot.* namespace:

Error	HTTP status	Meaning
SlotNotFound	404	—
IllegalSlotTransition	409	Attempted transition not in LEGAL_TRANSITIONS.
SlotNotReady	503	—
SlotSpawnFailed	500	—
SlotHealthFailed	503	—
SlotOutputSanityFailed	500	The server is healthy but its output is not language — see Output-sanity gate below.
SlotCrashLooping	503	Reload blocked during exponential backoff, or after too many consecutive failures — see Crash-loop breaker below. Carries retry_after_s.
SlotTerminateTimeout	504	Stop didn’t return in time — the request unblocks anyway.
SeamTimeout (slot.seam_timeout)	504	A bounded systemctl/podman seam call (Quadlet write, daemon-reload, reset-failed, the spawning systemctl restart) did not return within its budget. details carries command, budget_s and last_state; the slot lands in ERROR with the same message.
SlotConfigError	400	—
slot.tuning_key_denied	400	A slot write carried a key that belongs on the profile.
slot.profile_missing	422	The slot names a profile the catalog does not have; the launch is refused.
NpuExclusivityViolation	409	Only one device=npu, type=llm slot with a bound model is allowed at a time (AMD XDNA hardware context limit).
SlotPinned	409	A pinned slot needs ?force=true to unload or delete.

Health-probe resolution on timeout: unresolved health checks resolve to WARMING, not
ERROR — an ambiguous outcome is treated as “still loading,” not “failed.”

Note: Client-side lifecycle timeouts

The CLI and the MCP admin bridge derive their read timeout for load / unload
/ restart / swap from the server’s own worst-case budget
(hal0.slot_lifecycle_budget) rather than a fixed number — a load charges the
health poll, the output-sanity gate’s CPU-lane budget, and up to three
in-series eviction unloads, all with a 1.15x overhead margin. A restart or swap
charges both an unload and a load. This is why a slow (e.g. CPU-only) box’s
load can legitimately take several minutes before a client gives up.

Output-sanity gate

/health proves the runner bound its port and mapped its weights. It does not prove the
backend turns those weights into coherent text — a GPU backend that mis-computes answers
/health, streams tokens and closes the SSE stream cleanly while emitting garbage.

So a type = "llm" slot on the container/llama-server lane runs one last check before it
is published as READY: a single greedy completion whose answer is known
(The capital of France is → Paris), once per load, while the slot still holds its lock
and is not yet dispatchable. A wrong answer on the raw /completion endpoint buys one
retry through /v1/chat/completions (with room for a reasoning model’s <think> block)
before the load is refused.

A wrong answer is terminal for that load: the slot lands in ERROR with a message
naming the probe, the expected token and what the model actually said, its unit is stopped
(best-effort — a stop that does not return is reported in the same message), and it will
not be re-adopted as READY by a later /api/slots poll or by the boot-time
reconcile. A passing gate on a subsequent load is what clears the verdict.

A probe that never answers is treated differently. A timeout says the answer did not
arrive inside a wall clock, not that the answer was wrong, so the slot resolves to
WARMING — the same policy the health wait already applies to its own ambiguous outcome.
Nothing is published as READY, nothing is stamped, the unit is left alone, and the next
load re-runs the gate. Slow hardware is judged on its answer; the budget widens to match on
a device = "cpu" slot, which decodes far slower than the accelerated lane.

Note

“Inconclusive, so let it through” remains the one thing the gate never does: an
unanswered probe can only produce WARMING, never READY.

Embedding, reranking, TTS, STT and image slots are never gated (they cannot serve a chat
completion), and the FLM/NPU lane keeps its own verify_inference sentinel.

Escape hatches, both default-ON:

Scope	How
Whole box	HAL0_SLOT_OUTPUT_SANITY=0 in the api environment
One slot	output_sanity = false in the slot TOML, or via PUT /api/slots/{name}/config

Crash-loop breaker

A slot in ERROR is freely re-loadable, but not at request cadence. SlotManager
tracks consecutive load failures per slot (hal0.slots.manager) and throttles
further load() calls through three states:

State	When	Next retry
backoff	1–4 consecutive failures	Exponential: min(5s * 2^(failures-1), 300s)
parked	5 or more consecutive failures	min(300s * 2^(failures-5), 3600s) — loads are refused outright
half-open	The current backoff or parked cooldown has elapsed	One trial load runs; success closes the breaker, a failed trial re-parks it at the next cooldown step

A load the breaker refuses raises SlotCrashLooping (503, see above) with no
state transition — no STARTING event, no journal row, no systemctl call — and
carries retry_after_s for the response’s Retry-After header. Any operator
action (a manual load, a restart, or a config edit) resets the breaker
immediately, independent of the cooldown clock; a load that converges healthy
clears it too.

The manager stamps the live view onto every slot snapshot as
metadata.breaker = {state, failures, retry_after_s} while the breaker is
non-closed, and omits the key entirely once it’s closed — so the field’s
presence is the render signal. The dashboard shows this as a chip on both the
slot card and the Edit-slot drawer’s state strip: amber “retry in Ns” while
backing off, red “parked · N failures” once parked, and a neutral “trial
pending” during the half-open window.

Container image status

image_status on a slot snapshot is one of present, pulling, missing,
unknown, or not-configured (hal0.slot_view.image_status_for). Only
unknown renders a dashboard chip: it means hal0 could not read the container
image store at all — the rootful hal0-podman-ro seam returned an error, the
sudoers grant is absent, or the read probe timed out — or could not resolve the
slot’s image in the first place (an unreadable profiles.toml; the log carries
reason=resolve-error) — which is a different
claim from missing (podman was asked and said the image isn’t there).
present and not-configured are the quiet, correct states, and pulling
already has its own progress bar. Seeing the “image ?” chip is the cue to run
hal0 doctor all and check the seams row, then the hal0-api journal for
slot_view.image_probe_failed (a reason=resolve-error there means the image
could not be resolved, not that the seam is broken). There is no standalone
hal0 doctor seams command.

A slot or its container that was stopped cleanly no longer paints as crashed
on either the slot card or the dashboard’s slot list — only a unit that
actually failed (state = "error" or a crashed container status) gets the
red treatment; a graceful stop renders the same neutral “stopped” state as any
other offline slot.

Persistence

SlotStateRecord (name, state, model_id, port, updated_at, message,
extra) is written atomically (tempfile + fsync + os.replace) to
/var/lib/hal0/slots/<stem>/state.json, where <stem> is the slot’s name
(or, after migrate-id-keying, its stable numeric id — see
hal0.slots.layout), so SSE readers never observe a torn write.

Self-managed providers

SELF_MANAGED_PROVIDERS = {kokoro, qwen3tts, moonshine, vibevoice} — these providers
serve a baked-in model, so no explicit model_id is needed to dispatch to them
(provider_requires_model() returns False for this set). This set must stay in sync
with the dashboard’s own SELF_MANAGED_PROVIDERS constant.

Activation

There is no enabled field on SlotConfig. A non-empty model.default is the
activation signal — it is what makes a slot routable and loadable.

Boot autostart is a separate, explicit setting: autoload. The generated Quadlet unit
carries [Install] WantedBy=hal0.target only when autoload = true, and that stanza
is the only thing that starts a slot at boot. With autoload = false the unit still
exists and is start-able (load/swap use systemctl restart, which never consults
[Install]) — nothing just pulls it up on reboot. Binding a model no longer implies a
boot start.

Slots written before the field existed (key absent on disk) migrate to
autoload = true when they have a bound model, and false otherwise, so an upgrade
preserves the old behaviour until you toggle it. Slots created through
POST /api/slots always persist an explicit value, defaulting to false.

Editing autoload via PUT /api/slots/{name}/config re-renders the on-disk unit, so the
change takes effect from the next boot without a manual reload.

Eviction priority

Every non-pinned slot is an eviction candidate. priority (int, 0–100, default 50)
orders the victims for both memory-pressure eviction (SlotReaper.pressure_evict_once)
and pre-load eviction (preload_evict): the lowest priority is unloaded first, with
least-recently-used as the tie-break inside a priority tier. priority = 100 is not a
pin — it is merely evicted last; pinned = true is what exempts a slot entirely, as do
the default-pinned anchors (agent and friends). A slot with a request in flight is
never a victim, and only resident slots (READY/IDLE) are considered at all.

Caution

The old lru = true eviction opt-in is retired. The eviction sweep ignores the key (with a
one-time slot.lru_flag_retired warning per slot),
and a slot file that still carries it fails the strict slot load with the replacement named — use
priority/pinned instead. Slots that never set lru are now evictable where they
previously were not.

What a slot write may carry

A slot holds placement (device, layers, threads, runner), its model and its profile. The profile holds every tune; the model row holds facts about its file.

Every write of a slot file goes through guard_slot_write_payload (hal0.slots.config_write): SlotManager.create() runs it on the new slot, and SlotManager.update_config() and stacks-apply run it through reconcile_and_guard_slot_config. The slot HTTP routes (POST /api/slots, PUT /api/slots/{name}/config) also check the body in api.routes.slots before anything else, so their “belongs on a profile” answer comes before a generic unknown-key error. They refuse:

• A tuning key ([model].context_size / ctx_size, [server].extra_args, parallel, chat_template, workers, mtp, speculation, enable_thinking, vision) with a 400 slot.tuning_key_denied (“belongs on a profile”). Set it on the slot’s profile (PUT /api/profiles/{name}) or pick another profile.
• A llama-server option in [server].env (LLAMA_ARG_*, LLAMA_API_KEY, HF_TOKEN, MTMD_BACKEND_DEVICE) with a 400 slot.llama_env_denied. llama-server reads each of them as a launch option: a LLAMA_ARG_* tune goes in the profile’s flags (when that flag is allowed there), and the other three set options hal0 owns (see Config schema).
• A key removed from the slot schema (for example enabled) with a 400 slot.removed_key_denied.

To rebind a slot’s model, send PUT /api/slots/{name}/config with {"model": {"default": "<model id>"}}, or use slot_edit or model_swap over MCP. PATCH /api/slots/{name}/defaults no longer exists.

A slot file on disk that still carries a tuning key is refused at launch with slot.unconverted, and the typed slot-config loader rejects it too (hal0 config validate names the file and the keys), until you delete the keys from the file and put their values on a profile. GET /api/slots reports such a slot as conversion_blocked (reason, fix).

Launch composition and effective context

A llama-server slot’s command is built from four layers, lowest precedence first, then deduplicated last-wins: base (--host, --port, --model, --alias, --ctx-size), model (family quirks, --jinja, --chat-template-file, --mmproj when the profile’s vision is auto, and --spec-type with --spec-draft-model when a drafter or MTP heads are in effect), profile (the draft bundle for that kind, then the profile’s flags) and slot (--device (ROCm0, Vulkan0, CUDA0 or none) from the slot’s device, -ngl, --threads, --metrics, and, only for embedded MTP heads, --spec-draft-device from the slot’s device; a paired drafter file runs on the same devices as the model). GET /api/slots/{name}/resolved returns the deduplicated argv and, for each surviving flag, the layer that set it. A slot with no llama-server command (or one that cannot launch) answers argv: null with an empty provenance.

GET /api/slots and GET /api/slots/{name} carry ctx_effective (tokens) and ctx_source (profile, native or fallback) for llm, embedding and reranking slots. Both come from the same resolver the launch path uses. The slot views do not return ctx_max, chat_template, rope_freq_base, workers, parallel or llamacpp_args.

For the CLI surface that drives slot lifecycle transitions directly, see the
CLI reference, hal0 slot.