Services (hal0 docs)

[system]

The hal0 platform itself is two systemd units: hal0-api, the API
daemon everything else talks to, and hindsight-api, the memory engine
it delegates recall/extraction to. Both run as the unprivileged hal0 system
user and are managed with plain systemctl/journalctl.

Note

This page covers the two platform units by raw systemctl below, and —
further down, in Companion services — the four services in the
/api/services registry (OpenWebUI, ComfyUI, Hermes, Hindsight) that the
dashboard’s Services page controls. Per-model inference units
(hal0-slot@<name>.service) have their own lifecycle; see
Manage slots and the
slot lifecycle reference.

hal0-api.service

The API daemon — serves /api/*, /v1/*, and the dashboard’s static
assets. Generated at install time by installer/install.sh (not a static
file in the repo):

[Unit]
Description=hal0 API daemon
Documentation=https://github.com/hal0ai/hal0
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=hal0
Group=hal0
SupplementaryGroups=systemd-journal
WorkingDirectory=${API_WORKDIR}
EnvironmentFile=${API_ENV}
EnvironmentFile=-${ETC_DIR}/hermes-python.env
EnvironmentFile=-${HF_SECRETS_ENV}
ExecStart=${HAL0_BIN} serve --port ${HAL0_PORT}
Restart=on-failure
RestartSec=3
StandardOutput=journal
StandardError=journal
SyslogIdentifier=hal0-api

[Install]
WantedBy=multi-user.target

• Journal read — SupplementaryGroups=systemd-journal lets the daemon
  read the journal behind /api/logs and the MCP logs_tail tool. The group
  applies to this unit’s process tree (children such as podman and flm
  inherit it); the hal0 account is not added to it, so
  hal0-agent@ units do not start with it. Under ADR-0002 the agent shares
  the hal0 uid and can influence hal0-api, so this is a scoping measure,
  not an isolation boundary. The installer omits the line on a host that has
  no systemd-journal group. hal0 update does not rewrite this unit:
  re-run the installer once after updating to 1.4.0. hal0 doctor all warns
  (API journal read) until it has been. Until then the Logs page’s slot
  channel and the slot log drawer show a notice naming the cause and the fix (a
  degraded frame, held open with keepalives) instead of reconnecting in a
  loop, the one-shot slot log tail carries the same hint, and
  hal0 slot logs --follow prints the reason and returns.
• User/group — hal0:hal0. Root-only operations (writing per-slot
  units, daemon-reload, restarting itself during an update) go through a
  narrow sudo -n wrapper (hal0-systemctl) rather than running the daemon
  as root. Every wrapper verb that takes a payload on stdin allow-lists that
  payload on the root side and writes its own validated reconstruction, so a
  hand-crafted unit body cannot smuggle a host-side ExecStartPre= or
  User=root past it. Use hal0-systemctl check-quadlet < unit (or
  check-dropin <gateway|hindsight> < body) to see why something was refused;
  both validate and echo without writing anything. Note the limit of that
  boundary: slots run under rootful podman and a slot’s [Container] section
  legitimately mounts host paths and passes through devices, so an attacker who
  owns the hal0 account can still author a privileged container. The wrapper
  is a check on the unit file, not a sandbox around slot content.
• WorkingDirectory — the current symlink in a production install, so
  hal0 update’s atomic swap moves it without rewriting the unit.
• ExecStart — no --host flag. The bind host comes from
  HAL0_BIND_HOST inside EnvironmentFile, so the unit and the hal0 serve
  CLI can never disagree about it.
• Port — 8080 by default (HAL0_PORT in api.env), binds 0.0.0.0
  unconditionally.

Config files it reads

File	Purpose
/etc/hal0/hal0.toml	Main config — includes the [security] table (see Authentication).
/etc/hal0/api.env	Required systemd EnvironmentFile, mode 0600 — port, log level, network settings, provider tokens, and (post-rotation) HAL0_ADMIN_KEY/HAL0_CLIENT_KEY.
/etc/hal0/hermes-python.env	Optional — Hermes interpreter policy (HAL0_HERMES_PYTHON=, see below).
/var/lib/hal0/secrets/hal0-api.env	Optional, root:root 0600 — HF_TOKEN and similar secrets.
/etc/hal0/upstreams.toml	External LLM upstream definitions.

Hermes interpreter policy

The Hermes venv runs on exactly one Python minor, which the pinned Hermes commit decides: the newest of 3.14 and 3.12 that its requires-python admits. The current pin, v0.22.4 (4d043038), declares >=3.11,<3.15, so Hermes runs on a uv-managed CPython 3.14 under /var/lib/hal0/hermes/python/cpython-3.14*, with its dependencies installed from upstream’s hash-locked uv.lock. Before 1.6.0 the pin capped below 3.14 and Hermes ran on 3.12; the first hal0 update that carries the new pin builds the 3.14 release next to the old one, switches, and rewrites /etc/hal0/hermes-python.env (the old 3.12 value is discarded with a hermes-python-discarded warning). When a build needs an interpreter for that minor, hal0 checks these in order:

1. HAL0_HERMES_PYTHON, the operator’s override, from the environment and then from /etc/hal0/hermes-python.env. A value that is not the target minor is discarded with a warning, and the interpreter is re-probed. A value whose interpreter links an SQLite older than 3.51.3 is still used, with a warning.
2. A uv-managed CPython under /var/lib/hal0/hermes/python, fetched by the uv that ships in hal0’s venv (uv python install --managed-python). It must link SQLite 3.51.3 or newer.
3. A system python<minor> (python3.14 for the current pin) that links SQLite 3.51.3 or newer, used when uv cannot fetch one. Ubuntu 26.04’s python3.14 links SQLite 3.46.1, below the floor where Hermes forces DELETE journal mode, so it is never used.

hal0’s own venv interpreter is never used for Hermes.

hal0 writes the interpreter it chose back to /etc/hal0/hermes-python.env. hal0 doctor reports the SQLite version of the live Hermes interpreter, and warns below 3.51.3.

The uv that does the fetching ships in hal0’s own venv (pinned to 0.13.4.x); the installer no longer installs it separately. uv’s cache lives at /var/lib/hal0/hermes/.cache/uv. A host with no network cannot build the first 3.14 release: the build fails (status build_failed) and the box keeps running its current Hermes. Rolling hal0 back to 1.5.x after the 3.14 build leaves the Hermes build failing (Hermes Python must be exactly 3.12) until you update again; do not edit hermes-python.env to 3.12 on 1.5.x.

Hermes releases

Every Hermes build (install, hal0 agent reprovision hermes --repair and hal0 update (including hal0 update component hermes --target)) produces an immutable release under /var/lib/hal0/hermes/releases/<id>/ with a release.json. hal0 checks it (imports, hermes --help, an MCP probe) before the current symlink moves, and /var/lib/hal0/venvs/hermes points at current/venv. Activating a release stops hal0-agent@hermes and the gateway, switches, runs hermes config migrate, writes sessions.auto_prune: false and the persona prompt into config.yaml, and restarts them. If the re-check fails, hal0 switches back to previous, restores config.yaml and reports activation_failed. The first update to 1.6.0 therefore causes a short Hermes outage. Hermes keeps session history: sessions.auto_prune is set to false, where Hermes 0.22 would otherwise delete ended sessions after 90 idle days. The directory tree is in Paths & files.

Health endpoints

Endpoint	Auth	Checks
GET /api/health	open	Shallow liveness — returns immediately, no slot/upstream/disk work. What the installer and the systemd watchdog poll.
GET /api/health/system	open	Deep health — disk headroom on state/config roots (disk_state, disk_config: ok, free_mb, total_mb, floor_mb, path), slot-manager responsiveness, errored-slot detection. Always 200, with {"status": "ok"|"degraded", "checks": {...}}.
GET /api/status	client	Dashboard summary poll (hardware, slots, memory-degraded flag).

hindsight-api.service

The memory engine — extraction, consolidation, reflection, and recall. A
static unit file shipped at installer/systemd/hindsight-api.service:

[Unit]
Description=hal0 shared Hindsight memory engine (platform brain)
After=network-online.target hal0-api.service
Wants=network-online.target

[Service]
Type=simple
User=hal0
Group=hal0
WorkingDirectory=/var/lib/hal0/memory/hindsight

Environment=HOME=/var/lib/hal0/memory/hindsight
Environment=HF_HOME=/var/lib/hal0/memory/hindsight/hf-cache

Environment=HINDSIGHT_API_LLM_PROVIDER=openai
Environment=HINDSIGHT_API_LLM_BASE_URL=http://127.0.0.1:8080/v1
Environment=HINDSIGHT_API_LLM_MODEL=hal0/utility
Environment=HINDSIGHT_API_LLM_API_KEY=hal0-local-noauth
EnvironmentFile=-/etc/hal0/hindsight-llm.env
Environment=HINDSIGHT_API_LLM_TIMEOUT=300
Environment=HINDSIGHT_API_SKIP_LLM_VERIFICATION=true

Environment=HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU=true
Environment=HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=true

ExecStart=/var/lib/hal0/memory/hindsight/.venv/bin/hindsight-api --host 127.0.0.1 --port 9177

Restart=on-failure
RestartSec=3
TimeoutStartSec=120

[Install]
WantedBy=multi-user.target

• Port — 9177, loopback only (--host 127.0.0.1). Not LAN-exposed;
  hal0-api is the thing that talks to it.
• Dependency direction is inverted from what the ordering suggests —
  After=... hal0-api.service is soft ordering only (an unknown unit name is
  tolerated), but the real dependency runs the other way: hindsight routes
  its own extraction/consolidation/reflection LLM calls back to hal0-api’s
  OpenAI-compatible endpoint (HINDSIGHT_API_LLM_BASE_URL=http://127.0.0.1:8080/v1).
  hindsight-api is a consumer of hal0-api, not the reverse.
• No TOML config — every setting is an inline Environment= line in the
  unit itself, with one exception: EnvironmentFile=-/etc/hal0/hindsight-llm.env
  (the leading - means “ignore if absent”) overrides the inline
  HINDSIGHT_API_LLM_API_KEY=hal0-local-noauth placeholder with the real
  client key. installer/install.sh writes that file at install time from
  HAL0_CLIENT_KEY/HAL0_ADMIN_KEY in /etc/hal0/api.env, and hal0 auth rotate keeps it fresh afterward. A hal0-owned drop-in,
  hindsight-api.service.d/extraction-model.conf, overrides the shipped unit
  without editing it. Its model line is always HINDSIGHT_API_LLM_MODEL=hal0/memory,
  the virtual id hal0’s /v1 resolves on every call to the local
  [memory.graph].extraction_slot or to the upstream model in
  extraction_model. Choosing a slot or an upstream model (Engine model on
  Settings → Data → Memory, or hal0 memory graph enable --slot|--model|--local)
  is therefore a config write that applies at once and never restarts the
  engine. The drop-in also carries HINDSIGHT_API_LLM_TIMEOUT and the extraction
  limits (HINDSIGHT_API_RETAIN_LLM_MAX_CONCURRENT,
  HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS,
  HINDSIGHT_API_RETAIN_LLM_MAX_RETRIES, HINDSIGHT_API_WORKER_MAX_RETRIES,
  HINDSIGHT_API_WORKER_TASK_RETRY_BACKOFF_SECONDS, and the derived
  HINDSIGHT_API_RETAIN_WALL_TIMEOUT). Changing one of those
  rewrites the file and restarts hindsight-api, and the first save after
  upgrading from a release that pinned a slot id there repairs the drop-in the
  same way. That drop-in lives under /etc/systemd/system
  and hal0-api runs unprivileged, so the write and the follow-up restart both
  route through the hal0-systemctl wrapper
  (write-hindsight-dropin + svc-restart hindsight). If the wrapper or its
  sudoers grant is missing, the API’s response carries a non-null
  propagation.error and the engine keeps the drop-in it had —
  check hal0 doctor for the seam rows. The wrapper allow-lists the drop-in
  body on the root side: only # comments, a single [Service] header and
  Environment= assignments of the seven keys above are accepted
  (HINDSIGHT_API_LLM_MODEL must be a hal0/<name> value), so a
  hand-crafted fragment (ExecStart=, User=, any other directive) is
  rejected rather than written. Run
  hal0-systemctl check-dropin hindsight < body to see why a body was refused;
  it validates and echoes without writing anything.
• Overrides in hindsight-llm.env win. systemd applies EnvironmentFile=
  after every Environment= line in the unit and its drop-ins. A
  HINDSIGHT_API_LLM_PROVIDER, HINDSIGHT_API_LLM_BASE_URL or
  HINDSIGHT_API_LLM_MODEL line in /etc/hal0/hindsight-llm.env therefore beats
  hal0/memory and whatever you picked under Engine model: hal0 reports the
  slot or model you chose while the engine runs the override. The Engine
  running row on Settings → Data → Memory (highlighted when an override applies),
  hal0 memory graph status and engine_llm in GET /api/memory/graph/status
  name the provider and model hindsight-api actually loads, read from the
  running engine process; when hal0-api can read neither the process nor the
  file, they say cannot verify instead of claiming no override. Remove the lines and
  restart hindsight-api to hand the choice back to hal0. The installer (on a box
  that has a key) and hal0 auth rotate rewrite the file with the API key line
  only, which drops any override you added.
• Data root — embedded Postgres at /var/lib/hal0/memory/hindsight/.pg0,
  pinned via the unit’s HOME= env (Hindsight has no dedicated data-dir
  variable of its own).

Config files it reads

File	Purpose
/etc/hal0/hindsight-llm.env	Optional EnvironmentFile, - prefixed (ignored if absent) — supplies the real HINDSIGHT_API_LLM_API_KEY, written by installer/install.sh and kept fresh by hal0 auth rotate. A HINDSIGHT_API_LLM_PROVIDER, _BASE_URL or _MODEL line here overrides hal0’s engine target.
hindsight-api.service.d/extraction-model.conf	hal0-owned systemd drop-in under /etc/systemd/system — sets HINDSIGHT_API_LLM_MODEL=hal0/memory, HINDSIGHT_API_LLM_TIMEOUT and the extraction limits; written when the timeout or a limit changes (hal0 memory graph enable, the Memory settings page) or the file is stale.

Health endpoint

GET http://127.0.0.1:9177/health — polled by the installer after start (up
to 40 × 3s) and by hal0 doctor verify’s memory-engine check. If
unreachable while the memory feature is enabled, hal0’s own health report
degrades to WARN "engine enabled but :9177 unreachable" rather than
failing hard.

Starting, stopping, and inspecting

Same commands for both units:

systemctl status  hal0-api
systemctl start   hal0-api
systemctl stop    hal0-api
systemctl restart hal0-api
systemctl enable  hal0-api        # already enabled by the installer

journalctl -u hal0-api -f         # follow
journalctl -u hal0-api -n 40      # last 40 lines — the recovery recipe install.sh itself uses

Swap hal0-api for hindsight-api for the memory engine. hal0-api’s own
self-restart path (the updater) shells out to the same systemctl restart
via the hal0-systemctl wrapper, so a manual restart and an update-triggered
one behave identically.

Log access over HTTP

Both units’ logs are also available without shell access, proxied from
journald with secret-redaction applied:

GET /api/logs?unit=hal0-api&n=200[&since=...&level=...]
GET /api/logs/stream?unit=hal0-api&level=...     # SSE tail

Swap unit=hindsight-api for the memory engine’s logs. Both routes are
admin-tier, and both read only hal0’s own units: hal0.target, any
hal0-* unit (hal0-api, hal0-slot@<id>, hal0-agent@<id>,
hal0-openwebui, hal0-bench*, …), hindsight-api and hermes-gateway.
Any other unit is rejected with a 400 system.logs_error.

Companion services

The dashboard’s Services page manages four companion services from a single
registry — hal0.services.registry.SERVICES in
src/hal0/services/registry.py. One ServiceDef per service names its
owning unit, its probe strategy, and which lifecycle verbs it permits:

Service	Purpose	Unit	Allowed actions
OpenWebUI	Chat UI companion (podman container, LAN :3001)	hal0-openwebui.service	start, stop, restart, enable, disable
ComfyUI	Image generation engine (runs as the seeded img slot container, LAN :8188 by default; [network].comfyui_listen = "loopback" fences it to the box)	[email protected]	start, restart (no stop)

| Hermes | Bundled agent + its own dashboard (loopback :9119) | [email protected] | start, stop, restart, enable, disable |
| Hindsight | Memory engine — this is the same unit documented above | hindsight-api.service | start, stop, restart, enable, disable |

The Open WebUI card also shows an Agents chip with the Hermes profiles wired into the picker, taken from openwebui_wiring_status.

Hermes systemd drop-in

hal0 ships its Hermes unit settings as
/etc/systemd/system/[email protected]/10-hal0.conf. Put your own
changes in a separate drop-in with systemctl edit --drop-in=local hal0-agent@hermes (it lands as local.conf and sorts after 10-hal0.conf, so
your values win). Earlier releases shipped this file as override.conf, the
name systemctl edit uses by default. Re-running the installer deletes an
override.conf that still matches a hal0-shipped version. An override.conf
you edited is kept and the installer warns, naming the hal0-owned directives it
repeats. Trim it to your own lines. A drop-in carrying ConfigurationDirectory=
is renamed to <name>.hal0-disabled, because that directive crash-loops the
unit (status 241); the file is kept, not deleted.

Hermes gateway and linger

The Hermes gateway runs as hermes-gateway.service, a system unit with
User=hal0. hal0 update renders the unit from hal0’s own template through
the root verb hal0-update hermes-gateway once the new release is live: it
orders the gateway after user@<uid>.service, turns systemd linger on for
hal0, and restarts the gateway only if it was running. It never enables,
starts or stops a gateway you disabled. Linger is needed because Hermes’s
scheduled and kanban workers start through systemd-run --user under that
user’s manager; without it kanban workers never start. hal0 doctor all
shows the linger state on its hal0_linger row, and if the verb fails
hal0 update prints sudo hal0 agent install hermes. hal0 agent uninstall hermes and installer/uninstall.sh run loginctl disable-linger hal0. A
gateway the operator disabled or masked is left off by the installer,
hal0 agent install hermes and hal0 update.

From Hermes v0.22.3 the gateway daemon owns chat turns and the session store. The dashboard’s default Chat tab still runs its turns in the dashboard process and needs no gateway. A chat opened for another profile (such as hal0-brain) and the classic hermes chat CLI are gateway clients that attach to hermes-gateway.service. hal0 keeps the unit file installed even when you turn the gateway off (it only ever runs systemctl disable --now, never removes the file), because an installed unit in any state is what stops a client from starting a gateway of its own. hal0 doctor has a Hermes gateway unit row (hermes-gateway-unit) that fails when the file is missing or a gateway run --idle-exit process is running, and the hermes-api row warns rather than fails while the gateway is off. To chat in another profile, run the gateway with sudo systemctl enable --now hermes-gateway. During hal0 update, a gateway unit whose ExecStart, ExecStop or ExecStopPost is not hal0’s launcher (for example one written by hermes gateway install outside hal0) is rewritten to hal0’s launcher while the gateway is stopped, before the new release starts.

Hindsight’s entry here isn’t a second thing to learn: the Services page’s
buttons for it run through the identical unit_action → hal0-systemctl
path already described in hindsight-api.service above.

Start, Stop, Restart

The Services page renders one button per allowed action
(ui/src/dash/services.jsx:100-124): Restart whenever it’s allowed, Start
when the unit isn’t currently active, Stop only while it is. Clicking one
calls POST /api/services/{id}/action with {"action": "start"|"stop"|"restart"},
which hal0.services.systemd.unit_action() runs against the unit — the
same verb allow-list and hal0-systemctl wrapper described above for
hal0-api and hindsight-api.

Stop carries a browser confirmation — “Stop <name>? Clients using it will
lose access until it is started again.” — because stopping OpenWebUI,
Hermes, or Hindsight takes down something other browsers or agents may be
using right now. It’s a client-side prompt only: it isn’t enforced by the
API, so a direct POST bypasses it. Restart shows no such prompt.

ComfyUI has no Stop button at all — the registry grants it only start
and restart. Its “start” is not a raw systemctl start: the route
substitutes the GPU-arbiter switchover (_comfyui_start,
src/hal0/api/routes/services.py:187-208, delegating to the same
comfyui_switchover handler the FirstRun repair flow uses) so resident LLM
slots get drained before the iGPU is handed to ComfyUI. Stopping is
arbiter-managed too, per its card hint: “stop is GPU-arbiter managed —
switch back to inference in the Image-Gen pane”.

The registry also allows enable/disable for OpenWebUI, Hermes, and
Hindsight, but the Services page renders no buttons for those two verbs —
only POST /api/services/{id}/action with {"action": "enable"|"disable"}
reaches them.

There’s no hal0 service command group, and for OpenWebUI and Hermes
specifically there’s no CLI equivalent for start/stop/restart. hal0 app list and hal0 agent list/hal0 agent status show a service’s systemd
enabled/active state, but neither hal0 app nor hal0 agent has a
start/stop/restart subcommand — for those two services, the dashboard
buttons, or a direct systemctl {start|stop|restart} <unit> as root (same
mechanism as the platform units above), are the only ways to drive these
verbs today. ComfyUI is the exception: its unit is [email protected],
the same seeded img slot the Slots page manages, so hal0 slot restart img, hal0 slot load img, and hal0 slot unload img reach it from the
CLI.

Health

Two endpoints report health, for different consumers:

• GET /api/services — the Services page’s own source. Every entry
  carries up (bool), a probe-specific detail string, and the raw
  systemd unit_state (active_state/sub_state/unit_file_state/since).
  The page’s status pill is green up when up is true, red failed
  when unit_state.active_state is "failed", grey down for any other
  managed service, and a plain grey — for an unmanaged one
  (ui/src/dash/services.jsx:67).
• GET /api/services/health — a stable summary over the same service
  catalog (OpenWebUI, ComfyUI, Hermes, Hindsight) that feeds the Overview
  page’s Services widget. It collapses the same probes into
  state: "up" | "stopped" | "down", splitting “not up” into a deliberate
  stopped (unit inactive — comes back on demand) from down (unit
  failed, or reachable-but-unhealthy).

Each service is probed differently: OpenWebUI is a loopback
GET http://127.0.0.1:3001/health (2xx = up); ComfyUI is its own
/system_stats + /queue (reachable = up, with a live “N running / M
queued” job count); Hermes and Hindsight both fall back to systemd
active_state == "active". Every probe is fail-soft by design — a probe or
systemd failure degrades to up=false with an honest detail, never a
fabricated “up” and never a 500. A service with no wired probe would report
up: false, state: "down", detail: "unmonitored", though none of the four
registered services hits that path today. The summary’s probes run
concurrently, so one hung or raising probe reports its own row as down
without dropping the rest.

mDNS discovery toggle

The DISCOVERY (mDNS) card above the service grid
(DiscoveryCard, ui/src/dash/services.jsx:200-241, toggle button at
:214-219) toggles POST /api/services/mdns with
{"advertise": true|false}. Turning it on writes one avahi service-group
file — /etc/avahi/services/hal0-addon-<id>.service — per service that is
both mDNS-capable and LAN-published: OpenWebUI (:3001) and, by default,
ComfyUI’s editor (:8188), so other machines reach them at
http://<host>.local:<port>. Hermes and Hindsight listen on the box’s
loopback, so they’re never mdns-capable. ComfyUI’s LAN publication is the
[network].comfyui_listen key in hal0.toml (ADR-0041):
"lan" (the default) binds the LAN and makes it advertisable; "loopback"
fences :8188 to the box, and the card then neither lists nor announces it
(its link answers only on the box itself or through an SSH tunnel). A change
applies at the next hal0 slot restart img. avahi-daemon inotify-watches that
directory, so the announcement takes effect immediately — no reload, no unit
restart. Turning it off removes every hal0-addon-*.service file. A ComfyUI
announcement written while it was on the LAN stays after you switch to
"loopback" until you turn the toggle on or off again; either removes it.

The toggle never touches hal0.service, the base dashboard announcement
the installer writes separately — that’s why “Dashboard announced as
<host>.local” can be true even when the addon toggle is off. If
avahi-daemon isn’t active, the button is disabled and the card tells the
operator to install/enable it or add a static hosts entry.

See also

• Authentication — /etc/hal0/api.env, the
  EnvironmentFile hal0-api.service loads, is where the admin/client keys
  described there actually live.