hal0 turns a Linux box into a polished, OpenAI-compatible inference
appliance. It is built for the AMD Ryzen AI Max+ 395 (“Strix Halo”) —
iGPU + XDNA NPU + 128 GB of unified memory — and falls back to portable
hardware parsers on every other host.
Every inference workload runs as its own podman container, supervised
by a hal0-slot@<name>.service systemd unit. A single control plane,
hal0-api on port 8080, owns the slot state machines, dispatches
OpenAI-compatible /v1/* requests to the right slot, and serves the
dashboard. There is no shared inference daemon to babysit.
Each capability — utility, embed, rerank, stt, tts, img, vision,
agent, and the NPU-conditional flm-stt / flm-embed slots — runs in its
own dedicated podman container. agent is the capable default/fallback
anchor and utility is the cheap helper used for background jobs like
memory extraction.
Hardware support
hal0 is first-class on AMD Strix Halo — the Ryzen AI Max+ 395 (128 GB) and
the 385 / 390 (64 GB) — with AMD RX 7000-series GPUs also supported. NVIDIA
RTX 30/40/50 (via CDI + the NVIDIA Container Toolkit) is experimental, and
CPU-only x86_64 hosts work as a fallback. There is no minimum-spec gate —
backend selection degrades to CPU rather than refusing to run.
What you get
- OpenAI-compatible
/v1/*API — chat, completions, embeddings,
rerank (/v1/rerankings, plus the/v1/rerankalias for
llama.cpp/Cohere-style clients), audio transcription, speech, and image
generation. Point any OpenAI SDK athttp://localhost:8080/v1and go. - Hardware-aware slots — each capability gets its own container,
image, flags, port, and lifecycle, sharing only the GPU through an
arbiter. - Profiles — the whole tune for a slot: llama-server flags, context
window, thinking default, vision and MTP (seeds includechat,agent,
utility,coding,embeddingandreranking). A slot names a profile;
to run one model two ways, make two profiles. - Stacks — named, exportable bundles of slot + profile + model
assignments (built-insaber,forge, andpi) that you can load,
clone, and re-apply as a single declarative unit. - Hardware-anchored install —
install.shprobes your hardware and
provisions a coherent set of slots, models, and extensions in one pass —
no separate setup wizard to run afterward. - Prewired chat UI — OpenWebUI on port
3001, zero config. - Signed self-update — cosign-verified tarballs with one-flag
rollback.
The hal0 dashboard — slot status, hardware metrics, and quick actions.
Start here
- Install hal0: One-line install: fetch, verify, unpack, and bring up the API.
- Load your first model: Pull a model from Hugging Face and assign it to a slot.
- Send your first chat: Chat from OpenWebUI or curl the OpenAI-compatible endpoint.
- Verify the install: The six-step ‘did it work’ checklist: health, models, first chat, dashboard, logs, doctor.
- Install on bare metal: Run the installer directly on a Linux host, no container layer.
- Install on Proxmox: Quick-path CT script or the full GPU/NPU passthrough recipe.
- Install in an Incus container: Unprivileged system container on a bare-metal host: GPU/NPU passthrough, clean host, delete to uninstall.
- Install in a rootless podman box (opt-in): v1.8.0 opt-in: the whole install in one rootless podman container, GPU/NPU passthrough, a host shim, and an uninstall that removes it all.
- WSL (experimental): Run hal0 under Windows Subsystem for Linux, CPU-only.
- GPU drivers and memory: Kernel/firmware requirements, GTT sizing, and render-group gids.
Caution: Open on the LAN by default, no TLS
hal0-api binds 0.0.0.0:8080 and, until you configure an admin key,
treats every request as authenticated by network reachability — the right
posture for a trusted LAN appliance. hal0’s own auth is opt-in (see
Authentication). If hal0 is reachable from anywhere you
do not control, front it with a reverse proxy that owns TLS.