Slots (hal0 docs)

[system]

A slot is hal0’s unit of served inference: one named place that runs one
model. Behind every slot is a single podman container, and around every slot is
a lifecycle the platform manages for you. Slots are what the dispatcher routes
to and what the GPU arbiter coordinates. A slot owns where a model runs —
device, GPU layers, threads, runner — and names the model and the profile it
runs with. The profile supplies the tune; a slot can carry the six sparse
overrides described below. See Who owns what for the ownership boundary.

Per-slot tune overrides

A slot can carry a sparse [tune] table over its profile: ctx_size,
enable_thinking, reasoning_effort, reasoning_budget, vision and
speculation. Missing keys inherit the profile. This applies to LLM slots
outside the NPU lane. Models still own file facts and profiles own free flags.

For example, serve the same file in a fast slot with an 8192-token context
and a full slot with a larger window, both inheriting one profile. A dedicated
memory-extraction slot can use thinking off and low effort; validate JSON
conformance with the actual model and runner.

Use the drawer’s Tune block, hal0 slot edit, or the config API. Thinking
changes apply to new requests; other rows require restart. Profile flags
--reasoning on|off pin thinking even over a hand-written override.

One slot, one container

Each slot is backed by a systemd template unit named
hal0-slot@<name>.service, generated as a Podman Quadlet unit (a
.container file of typed [Container] keys — Image=, AddDevice=,
Volume=, PublishPort=, and so on — rather than a hand-assembled podman run command line). Podman’s systemd generator turns that into the
[Unit]/[Service] skeleton and the actual podman run invocation.
Quadlet auto-removes any prior container of the same name on start, so
there’s no more --replace / ExecStartPre rm dance to guard against a
stale container name. That launches exactly one container —
hal0-slot-<name> — serving one model on a loopback port, publishing its
port on 127.0.0.1 only.

SlotManager owns this. It renders the unit, starts and stops it, tracks
state, and registers the running container as an internal upstream so the
dispatcher can forward to it. A slot’s TOML supplies its name, port, hardware
grid, a reference to a profile
for the tune, and the model it runs; the model itself, with the facts about
its file, comes from the registry.

Note

The container runtime is Podman only. Quadlet is Podman’s declarative unit
generator and has no Docker equivalent, so Docker is unsupported — the old
Docker fallback has been removed.

Who owns what: slot, model, profile

A launch is described by three objects, and every setting has exactly one
owner:

Object	Holds
Model (registry row)	Facts about its file: path and file set, size, quantization, architecture, native context window, chat template, multimodal projector (mmproj), whether the file has MTP heads, and whether its chat template works with --jinja.
Profile	Base tune: free llama-server flags (KV-cache types, batch size, sampling, reasoning, --parallel), ctx_size, enable_thinking, reasoning_effort, reasoning_budget, vision (auto or off), speculation (auto, off or ngram), and an optional runner.
Slot	Placement and lifecycle: device, gpu_index, n_gpu_layers, threads, binary (a key into the RUNNER_IMAGES registry — the dashboard’s slot drawer calls this the slot’s Runtime), the image_pin escape hatch, and the port. Plus which model and which profile it runs. This is the “hardware grid”: [device · NGL · THREADS · BINARY].

A slot carries no free llama-server flags. It may override context, thinking,
effort, budget, vision and speculation under [tune]. For different sampling
or KV-cache flags, make separate profiles and point each slot at its own:

hal0 profile create agent-32k --from agent --ctx-size 32768
hal0 slot edit scratch --profile agent-32k

Every image hal0 ships is a RUNNER_IMAGES registry entry — the default
AMD GPU runtime (standard, titled Standard, the only entry allowed both
a ROCm and a Vulkan lane) plus the CPU, NPU, CUDA and single-purpose speech
and image runtimes, each pinned in manifest.json.
image_pin is a debug-only override (a custom build, an A/B test, a
rollback) and never appears as a pickable Runtime option. See Capabilities
& profiles for the full Runtime
picker and how a profile can carry one.

The split is enforced at every write path. Sparse [tune] overrides are the
six exceptions. A slot write carrying a legacy tuning key outside that table —
[model].context_size / ctx_size, parallel,
chat_template, workers, mtp, speculation, enable_thinking, vision or
[server].extra_args — is rejected with slot.tuning_key_denied, whose
message says the key belongs on a profile. The guard runs the other way too:
a profile’s flags may not set what the slot owns (--n-gpu-layers/-ngl,
--device/-dev, --threads/-t), what hal0 manages (--model, --host,
--port, --alias, --ctx-size), or a fact about the model file
(--jinja, --chat-template*, --mmproj, --no-mmproj). A slot’s
[server].env may not set LLAMA_ARG_* variables either, because llama.cpp
reads those as launch options (slot.llama_env_denied). The full list of
refused flags is in Edit configuration.

Caution

This partition is a hard invariant, but its enforcement has a real track
record of needing to be retrofitted onto new write paths after the fact —
the stacks-apply engine, in particular, bypassed it more than once before a
single shared gate (guard_slot_write_payload) was made the one pipeline
every writer (SlotManager.create, SlotManager.update_config, and stacks
apply) routes through. If you’re adding a new way to write a slot config,
route it through that shared gate rather than hand-rolling a merge.

A slot file that still carries a tuning key (a file written before
profile-centric config, or edited by hand) fails to load with
slot.unconverted and the slot does not launch. hal0 config validate names
the file and the keys. Delete the keys from the slot file, put their values on
a profile (hal0 profile create), and name it with profile = "<name>".

A slot with no profile runs its type’s seed profile (embedding and
reranking carry their mode flag; an llm slot runs chat). A slot that
names a profile that doesn’t exist refuses to launch with
slot.profile_missing, rather than quietly running some other tune.

How the launch command is assembled

A llama.cpp slot’s launch argv is built from four layers, lowest precedence
first, then deduplicated last-wins per flag:

1. base — --host, --port, --model, --alias and --ctx-size,
   always present and managed by hal0.
2. model — facts about the file: --chat-template-file, --mmproj
   (only when the model has a projector and the profile’s vision is
   auto), --jinja (added when the runner supports it, unless the model’s
   jinja fact is false or the profile is an embedding or reranking one),
   and any family quirks.
3. profile — the draft bundle for the speculation kind in effect, then
   the profile’s flags. A profile’s own --spec-draft-* values follow the
   bundle, so they win.
4. slot — -ngl, --threads, --metrics, and, when a draft type is in
   effect, the draft device derived from the slot’s own device.

GET /api/slots/{name}/resolved returns the resolved argv with each flag
labelled base, model, profile or slot, and the slot drawer’s
Resolved command panel shows the same thing. See Capabilities &
profiles.

The context window comes from the profile

The window a slot launches with is its profile’s ctx_size. A profile with
no ctx_size runs the model’s native window, capped at 32768; if the model
reports no native window, it runs 8192. Slot views report the result as
ctx_effective and where it came from as ctx_source (profile, native
or fallback), and the slot drawer shows both read-only under the Profile
row. Change the window by editing the profile (hal0 profile edit <name> --ctx-size N) or by pointing the slot at another profile; slots running the
same profile pick up a new ctx_size at their next restart. An FLM (NPU) slot
reads the same ctx_size; with none set it runs 8192.

The shipped seed profiles set ctx_size where a slot needs more than the
default: agent, utility and brain carry 65536 (the bundled Hermes agent
needs at least 64,000 tokens), and embedding and reranking carry 4096.
When a box can’t afford a seed’s window, the installer writes a custom
profile named <seed>-ctx<N> and points the slot at it.

Pinning a slot to one GPU

On a multi-GPU host, a slot’s gpu_index (an integer, 0 and up) pins it to a
single card instead of the default “all GPUs” behaviour. The env or device
mapping it emits is backend-specific and is derived from the slot’s own
device, never from a profile’s fields:

Device	What gpu_index does
gpu-rocm	Sets HIP_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES to the index.
gpu-vulkan	Sets GGML_VK_VISIBLE_DEVICES to the index.
gpu-cuda	Maps only that GPU in via CDI (--device nvidia.com/gpu=<n>) and sets CUDA_VISIBLE_DEVICES=0 inside the container, since the CDI mapping already exposes just the one card as ordinal 0.

Leaving gpu_index unset (the default) targets every GPU the backend can see,
unchanged from single-GPU behaviour. Any explicit [server].env entry you set
still wins over the derived visibility variables — they’re merged underneath,
never over.

The lifecycle state machine

A slot moves through an explicit set of states. The SlotState values are also
the wire representation streamed to the dashboard, so what you see is a real
transition, not a systemd snapshot:

The slot lifecycle transitions visualized.

State	Meaning
offline	Not running; no active unit.
pulling	Model files are downloading or being verified.
starting	The unit has started; waiting for the container to come up.
warming	Container is up; the model is loading into the GPU/GTT pool.
ready	Health probe passed; ready to serve. For a type = "llm" slot, this also requires passing an output-sanity probe — a known-answer completion that catches a backend serving garbage while /health reads green. See the Slot lifecycle reference for the gate’s mechanics.
serving	A request is actively in flight.
idle	Up but not serving — either warm-but-quiet, or up with no loadable model.
unloading	Graceful shutdown in progress.
error	Failed; details in state.json and the journal.

Transitions are constrained — only legal edges are allowed — and persisted
atomically so a dashboard reader never observes a half-written state. The
dispatchable “ready set” is ready, serving, and idle: those are the
states a request can be routed to. Any other state means the slot is
mid-lifecycle, and the dispatcher returns a structured slot.loading 503 with a
Retry-After hint rather than failing with a raw connection error.

Tip

idle is two situations the dashboard renders the same way: a slot that was
ready but has had no traffic past its idle timeout (a candidate for unload), or
a process that started but has no model it can actually serve. Both are
distinct from ready so routing never picks a slot that can’t fulfil a request.

Serving and idling

When the dispatcher forwards a request to a slot, it wraps the call in the
slot’s serving state: the first concurrent request moves the slot
ready/idle → serving, and the last one to finish moves it back to ready.
For a streamed response, the slot stays serving until the stream drains. A
background idle monitor demotes a quiet ready slot to idle after its idle
timeout (300 seconds by default), so the dashboard can distinguish “warm and
working” from “warm but quiet.”

The crash-loop breaker

A slot that keeps failing to load isn’t retried at request cadence. After a
load failure, SlotManager throttles further attempts with an exponential
backoff; once a slot racks up five consecutive failures it is parked —
loads are refused outright until a cooldown elapses, at which point a single
half-open trial load runs. Success closes the breaker; failure re-parks it
at a longer cooldown. A manual load, restart, or config edit resets the
breaker immediately regardless of the clock. The dashboard surfaces the live
state as a chip on the slot card and in the slot drawer; see the Slot
lifecycle reference for the exact timing.

Single-flight dispatch

Two safety mechanisms keep concurrent traffic from colliding:

• Per-slot lock. Load, unload, and restart for a given slot serialize behind
  one lock, so two requests can never race to start or swap the same slot’s
  container.
• Coalesced prefetch. When the dispatcher needs to fetch a cold upstream’s
  model list, identical concurrent fetches share a single execution — a
  hundred simultaneous requests for the same uncached model trigger one upstream
  call, not a hundred. They all receive the same result (or the same error).

The effect is that a burst of requests for a model that isn’t loaded yet
produces exactly one load, and everyone waits on that load rather than starting
their own.

Keeping the launch command honest across updates

A slot’s systemd unit bakes its launch argv at load time — updating hal0
changes the code that would render a slot’s command, but not the unit file
that already rendered it. Left alone, a slot that was running before an
update keeps running the pre-update flags through systemctl restart,
crash-restarts, and even reboots, until something forces a fresh render.

hal0 closes most of that gap automatically: after an update finishes (and
during install.sh’s update flow — now the single user-facing entry point for
both fresh installs and updates), it re-renders every existing slot’s unit
file through current code and issues one daemon-reload. Crucially, it
never bounces a running unit to do this — a slot that’s already serving
keeps serving on its old argv; the fresh command only takes effect the next
time that slot starts, whether that’s a manual restart, a crash-restart, or a
reboot. A slot whose fresh render is byte-identical to what’s on disk is
skipped as a no-op, and a per-slot render failure is logged and skipped rather
than aborting the update.

That still leaves a window — a slot that hasn’t restarted since the update
is running old argv and there’s no way to tell just by looking at it. The
dashboard closes that window with a resolved-command drift indicator:
each slot card compares the container’s actual, live podman inspect argv
against a fresh render from current config across the flags that matter most
(context size, model, alias, batch/ubatch size) and flags a mismatch. Seeing
drift on a slot you haven’t touched is the cue that it’s still running an
older command and could use a restart.

The GPU arbiter

The reference hardware has a single iGPU sharing one unified memory pool, so two
GPU workloads can’t both hold their weights at once. The GPU arbiter makes
that exclusivity explicit. It sorts GPU-backed container slots into two groups —
an llm group (GPU chat/embedding slots) and an img group (the ComfyUI
image-generation slot) — and only one group may hold the GPU’s memory at a time.
NPU and CPU slots are never arbitrated; they don’t contend for the iGPU.

When the GPU is in image mode, the arbiter does not dispatch to an llm-group
slot at once. A request for one waits for the image work to end and for the
reload, up to 120 seconds in all. Only when image mode is pinned, ComfyUI’s queue
cannot be read, or the wait runs out does the request get a structured
gpu.image_mode 503 (with a Retry-After) instead of failing silently.

Image mode lasts only while image generation runs: a /v1/images/generations
request from before the switch until its render returns (Open WebUI and the omni
loop use it too), or a render in ComfyUI’s queue from its editor. The image
container itself stays resident — what’s exclusive is the GPU memory, not the
container. About ten seconds after the last render, the arbiter frees the diffusion
models and reloads the LLM slots it had stopped; an LLM request that arrives
sooner starts that reload at once and waits for it rather than getting the 503.
A render is never cut short: while one is running or queued, the reload waits.
Pinning image mode (POST /api/comfyui/pin, or pin on the switchover) holds it
until you unpin it.

Caution

The single NPU LLM anchor is exclusive too, but for a different reason: the AMD
XDNA hardware context admits exactly one NPU LLM at a time. hal0 enforces this —
enabling a second device=npu chat slot is rejected.

Dedicated embed and rerank lanes

embed and rerank each get their own tuned profile and their own
podman container — they are never combined on one llama-server instance.
That’s a hard constraint, not a style choice: passing both --embedding and
--reranking to the same llama-server yields all-zero scores. Reranking is
provisioned as its own slot, internally named rerank, distinct from
the embed slot it’s paired with in the capability catalogue.

Both lanes are tuned for how pooled scoring actually works: -ub 8192
matches their batch size because a pooled embedding or rerank pass has to run
the whole input in one physical micro-batch — a smaller ubatch truncates or
fails on longer inputs.

Speculation: the model carries the draft, the profile decides

Speculative decoding (a drafter, or the model’s own MTP draft heads) takes two
inputs, one from the model and one from the profile:

• The model’s facts: a paired drafter, and whether the file has MTP
  heads. A drafter is a separate file paired like the projector (see
  Drafters). For the heads, an explicit mtp
  fact on the model row wins in either direction; left unset, hal0 falls
  back to the registry’s mtp tag when present. There is no filename- or
  id-based sniff — an uncurated pull with neither the tag nor an explicit
  fact is simply not MTP-eligible. The dashboard’s model drawer records this
  as the MTP heads file fact; it is not a switch for whether a slot
  drafts.
• The profile’s choice: speculation = "auto", "off" or "ngram".
  auto (the default) drafts with the paired drafter when the runner can run
  it, else with the heads when the model has them and the runner supports MTP;
  off never speculates; ngram uses n-gram lookup and needs no file.
  This field used to be mtp; hal0 update rewrites it once, and an API
  or CLI write that still sends mtp is refused with a 422 naming
  speculation.

A slot has no MTP setting of its own (mtp and speculation are among the
keys slot.tuning_key_denied refuses). To turn MTP off for one slot, give it a
profile with speculation = "off" (hal0 profile create <name> --from <profile> --speculation off). Under auto, a model without MTP heads simply doesn’t draft;
hal0 logs mtp.auto_off_model_ineligible rather than launching a draft that
would crash llama-server (context type MTP requested but model doesn't contain MTP layers).

When speculation is in effect, hal0 adds a tuned draft-flag bundle for the kind
in use to the profile layer of the launch command, and the draft device comes from the slot’s own
device. A profile’s own --spec-draft-* flags override the bundle’s values.

Seeded slots and aliases

Every install starts with a set of seeded slots so the common capabilities
have a home before you create anything:

flm, tts, rerank, utility, img, agent, brain, qwen3tts,
coder, and embed.

When the NPU runtime (FastFlowLM) is present, two more shadow slots are
seeded: flm-stt and flm-embed, riding the NPU chat anchor’s own flm
slot. Seeded slots are reserved — you can’t create a conflicting name, and
deleting one requires --force (see Manage slots),
which an install or update reconcile may later re-seed.

Each seeded slot names a seed profile that carries its tune: agent runs
agent, utility runs utility, coder runs coding, brain runs
brain, embed runs embedding, rerank runs reranking, flm runs
flm, tts runs kokoro, qwen3tts runs qwen3-tts, and img runs
comfyui.

The two canonical LLM slots are agent (the capable default — the
dispatch fallback anchor every routing chain ends in) and utility (the
cheap helper, seeded on every install so tasks like memory extraction always
have a cheap local model to target). The slot name is the routing key.

Each slot carries a type (llm, embedding, reranking, transcription,
tts, or image) and a device preference (gpu-rocm, gpu-vulkan,
gpu-cuda, cpu, or npu — gpu-cuda is the experimental path for
non-Strix-Halo NVIDIA hosts). hal0 keeps one back-compat alias so older
callers keep working: embed-rerank (the rerank capability slot’s
briefly-used old name) resolves to rerank.
Aliases are never written to disk and never appear in slot listings —
they’re a transparent translation at dispatch time.

Addressing a slot by its name (e.g. model: "agent") is the stable way to pin
a co-resident model: the name doesn’t change when you swap the underlying
model file. hal0 also resolves hal0/<slot>-style virtual model names —
hal0/agent, hal0/utility, hal0/npu, and the same pattern for any
custom type: llm slot (hal0/research, say) — to whichever slot is
actually loaded. As of v1.0 these canonical names are no longer advertised
as separate entries in /v1/models: each enabled LLM slot’s own alias
already covers its listing, so a hal0/<slot> request is resolved on
demand at dispatch time instead of being double-listed.

The NPU trio

The AMD XDNA NPU is driven by FastFlowLM (FLM), and one detail shapes how hal0
uses it: a single flm serve process serves three modalities at once —
chat, speech-to-text, and embeddings. hal0 calls this the NPU trio. See
Strix Halo for the hardware reason this exists.

Stacks — declarative slot bundles

A stack is a preconfigured slot + profile + model bundle you can apply
atomically. See Stacks for the full model — the TOML
shape, apply semantics, and an honest maturity read.

Where to go next

• Capabilities & profiles: How a slot’s profile shapes its tune, and why hardware no longer lives there.
• Stacks: Applying a whole set of slots as one named, portable unit.
• Architecture: How slots sit under the dispatcher and API.
• Strix Halo: The unified-memory hardware the arbiter coordinates.