# Services (hal0 docs)

**URL:** <https://forum.hal0.dev/t/services-hal0-docs/45>\
**Category:** Operate\
**Created:** [August 22, 2026, 6:50am UTC](https://forum.hal0.dev/t/services-hal0-docs/45 "2026-08-22T06:50:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![system](https://forum.hal0.dev/user_avatar/forum.hal0.dev/system/32/37_2.png) [@system](https://forum.hal0.dev/u/system)\
**Post date:** [August 22, 2026, 6:50am UTC](https://forum.hal0.dev/t/services-hal0-docs/45/1 "2026-08-22T06:50:07Z")

</div>

The hal0 platform itself is two systemd units: **`hal0-api`** , the API  
daemon everything else talks to, and **`hindsight-api`** , the memory engine  
it delegates recall/extraction to. Both run as the unprivileged `hal0` system  
user and are managed with plain `systemctl`/`journalctl`.

> [@](#):
>
> **Note**
> 
> This page covers the two platform units by raw `systemctl` below, and —  
> further down, in Companion services — the four services in the  
> `/api/services` registry (OpenWebUI, ComfyUI, Hermes, Hindsight) that the  
> dashboard’s Services page controls. Per-model inference units  
> (`hal0-slot@<name>.service`) have their own lifecycle; see  
> [Manage slots](https://forum.hal0.dev/t/manage-slots-hal0-docs/38) and the  
> [slot lifecycle reference](https://forum.hal0.dev/t/slot-lifecycle-reference/58).

## hal0-api.service

The API daemon — serves `/api/*`, `/v1/*`, and the dashboard’s static  
assets. Generated at install time by `installer/install.sh` (not a static  
file in the repo):

```ini
[Unit]
Description=hal0 API daemon
Documentation=https://github.com/hal0ai/hal0
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=hal0
Group=hal0
SupplementaryGroups=systemd-journal
WorkingDirectory=${API_WORKDIR}
EnvironmentFile=${API_ENV}
EnvironmentFile=-${ETC_DIR}/hermes-python.env
EnvironmentFile=-${HF_SECRETS_ENV}
ExecStart=${HAL0_BIN} serve --port ${HAL0_PORT}
Restart=on-failure
RestartSec=3
StandardOutput=journal
StandardError=journal
SyslogIdentifier=hal0-api

[Install]
WantedBy=multi-user.target

```

- **Journal read** — `SupplementaryGroups=systemd-journal` lets the daemon  
read the journal behind `/api/logs` and the MCP `logs_tail` tool. The group  
applies to this unit’s process tree (children such as podman and flm  
inherit it); the `hal0` account is not added to it, so  
`hal0-agent@` units do not start with it. Under ADR-0002 the agent shares  
the `hal0` uid and can influence `hal0-api`, so this is a scoping measure,  
not an isolation boundary. The installer omits the line on a host that has  
no `systemd-journal` group. `hal0 update` does not rewrite this unit:  
re-run the installer once after updating to 1.4.0. `hal0 doctor all` warns  
(`API journal read`) until it has been. Until then the Logs page’s slot  
channel and the slot log drawer show a notice naming the cause and the fix (a  
`degraded` frame, held open with keepalives) instead of reconnecting in a  
loop, the one-shot slot log tail carries the same hint, and  
`hal0 slot logs --follow` prints the reason and returns.
- **User/group** — `hal0:hal0`. Root-only operations (writing per-slot  
units, `daemon-reload`, restarting itself during an update) go through a  
narrow `sudo -n` wrapper (`hal0-systemctl`) rather than running the daemon  
as root. Every wrapper verb that takes a payload on stdin allow-lists that  
payload on the root side and writes its own validated reconstruction, so a  
hand-crafted unit body cannot smuggle a host-side `ExecStartPre=` or  
`User=root` past it. Use `hal0-systemctl check-quadlet < unit` (or  
`check-dropin <gateway|hindsight> < body`) to see why something was refused;  
both validate and echo without writing anything. Note the limit of that  
boundary: slots run under rootful podman and a slot’s `[Container]` section  
legitimately mounts host paths and passes through devices, so an attacker who  
owns the `hal0` account can still author a privileged container. The wrapper  
is a check on the unit _file_, not a sandbox around slot content.
- **WorkingDirectory** — the `current` symlink in a production install, so  
`hal0 update`’s atomic swap moves it without rewriting the unit.
- **ExecStart** — no `--host` flag. The bind host comes from  
`HAL0_BIND_HOST` inside `EnvironmentFile`, so the unit and the `hal0 serve`  
CLI can never disagree about it.
- **Port** — `8080` by default (`HAL0_PORT` in `api.env`), binds `0.0.0.0`  
unconditionally.

### Config files it reads

| File | Purpose |
| --- | --- |
| `/etc/hal0/hal0.toml` | Main config — includes the `[security]` table (see [Authentication](https://forum.hal0.dev/t/authentication-hal0-docs/43)). |
| `/etc/hal0/api.env` | Required systemd `EnvironmentFile`, mode `0600` — port, log level, network settings, provider tokens, and (post-rotation) `HAL0_ADMIN_KEY`/`HAL0_CLIENT_KEY`. |
| `/etc/hal0/hermes-python.env` | Optional — Hermes interpreter policy (`HAL0_HERMES_PYTHON=`, see below). |
| `/var/lib/hal0/secrets/hal0-api.env` | Optional, `root:root` `0600` — `HF_TOKEN` and similar secrets. |
| `/etc/hal0/upstreams.toml` | External LLM upstream definitions. |

### Hermes interpreter policy

The Hermes venv runs on exactly one Python minor, which the pinned Hermes commit decides: the newest of 3.14 and 3.12 that its `requires-python` admits. The current pin, v0.22.4 (`4d043038`), declares `>=3.11,<3.15`, so Hermes runs on a uv-managed CPython 3.14 under `/var/lib/hal0/hermes/python/cpython-3.14*`, with its dependencies installed from upstream’s hash-locked `uv.lock`. Before 1.6.0 the pin capped below 3.14 and Hermes ran on 3.12; the first `hal0 update` that carries the new pin builds the 3.14 release next to the old one, switches, and rewrites `/etc/hal0/hermes-python.env` (the old 3.12 value is discarded with a `hermes-python-discarded` warning). When a build needs an interpreter for that minor, hal0 checks these in order:

1. `HAL0_HERMES_PYTHON`, the operator’s override, from the environment and then from `/etc/hal0/hermes-python.env`. A value that is not the target minor is discarded with a warning, and the interpreter is re-probed. A value whose interpreter links an SQLite older than 3.51.3 is still used, with a warning.
2. A uv-managed CPython under `/var/lib/hal0/hermes/python`, fetched by the uv that ships in hal0’s venv (`uv python install --managed-python`). It must link SQLite 3.51.3 or newer.
3. A system `python<minor>` (python3.14 for the current pin) that links SQLite 3.51.3 or newer, used when uv cannot fetch one. Ubuntu 26.04’s `python3.14` links SQLite 3.46.1, below the floor where Hermes forces DELETE journal mode, so it is never used.

hal0’s own venv interpreter is never used for Hermes.

hal0 writes the interpreter it chose back to `/etc/hal0/hermes-python.env`. `hal0 doctor` reports the SQLite version of the live Hermes interpreter, and warns below 3.51.3.

The uv that does the fetching ships in hal0’s own venv (pinned to 0.13.4.x); the installer no longer installs it separately. uv’s cache lives at `/var/lib/hal0/hermes/.cache/uv`. A host with no network cannot build the first 3.14 release: the build fails (status `build_failed`) and the box keeps running its current Hermes. Rolling hal0 back to 1.5.x after the 3.14 build leaves the Hermes build failing (`Hermes Python must be exactly 3.12`) until you update again; do not edit `hermes-python.env` to 3.12 on 1.5.x.

### Hermes releases

Every Hermes build (install, `hal0 agent reprovision hermes --repair` and `hal0 update` (including `hal0 update component hermes --target`)) produces an immutable release under `/var/lib/hal0/hermes/releases/<id>/` with a `release.json`. hal0 checks it (imports, `hermes --help`, an MCP probe) before the `current` symlink moves, and `/var/lib/hal0/venvs/hermes` points at `current/venv`. Activating a release stops `hal0-agent@hermes` and the gateway, switches, runs `hermes config migrate`, writes `sessions.auto_prune: false` and the persona prompt into `config.yaml`, and restarts them. If the re-check fails, hal0 switches back to `previous`, restores `config.yaml` and reports `activation_failed`. The first update to 1.6.0 therefore causes a short Hermes outage. Hermes keeps session history: `sessions.auto_prune` is set to `false`, where Hermes 0.22 would otherwise delete ended sessions after 90 idle days. The directory tree is in [Paths & files](https://forum.hal0.dev/t/paths-files-reference/56#hermes-release-tree).

### Health endpoints

| Endpoint | Auth | Checks |
| --- | --- | --- |
| `GET /api/health` | open | Shallow liveness — returns immediately, no slot/upstream/disk work. What the installer and the systemd watchdog poll. |
| `GET /api/health/system` | open | Deep health — disk headroom on state/config roots (`disk_state`, `disk_config`: `ok`, `free_mb`, `total_mb`, `floor_mb`, `path`), slot-manager responsiveness, errored-slot detection. Always `200`, with `{"status": "ok"|"degraded", "checks": {...}}`. |
| `GET /api/status` | client | Dashboard summary poll (hardware, slots, memory-degraded flag). |

## hindsight-api.service

The memory engine — extraction, consolidation, reflection, and recall. A  
static unit file shipped at `installer/systemd/hindsight-api.service`:

```ini
[Unit]
Description=hal0 shared Hindsight memory engine (platform brain)
After=network-online.target hal0-api.service
Wants=network-online.target

[Service]
Type=simple
User=hal0
Group=hal0
WorkingDirectory=/var/lib/hal0/memory/hindsight

Environment=HOME=/var/lib/hal0/memory/hindsight
Environment=HF_HOME=/var/lib/hal0/memory/hindsight/hf-cache

Environment=HINDSIGHT_API_LLM_PROVIDER=openai
Environment=HINDSIGHT_API_LLM_BASE_URL=http://127.0.0.1:8080/v1
Environment=HINDSIGHT_API_LLM_MODEL=hal0/utility
Environment=HINDSIGHT_API_LLM_API_KEY=hal0-local-noauth
EnvironmentFile=-/etc/hal0/hindsight-llm.env
Environment=HINDSIGHT_API_LLM_TIMEOUT=300
Environment=HINDSIGHT_API_SKIP_LLM_VERIFICATION=true

Environment=HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU=true
Environment=HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=true

ExecStart=/var/lib/hal0/memory/hindsight/.venv/bin/hindsight-api --host 127.0.0.1 --port 9177

Restart=on-failure
RestartSec=3
TimeoutStartSec=120

[Install]
WantedBy=multi-user.target

```

- **Port** — `9177`, **loopback only** (`--host 127.0.0.1`). Not LAN-exposed;  
hal0-api is the thing that talks to it.
- **Dependency direction is inverted from what the ordering suggests** —  
`After=... hal0-api.service` is soft ordering only (an unknown unit name is  
tolerated), but the real dependency runs the other way: hindsight routes  
its own extraction/consolidation/reflection LLM calls back _to_ hal0-api’s  
OpenAI-compatible endpoint (`HINDSIGHT_API_LLM_BASE_URL=http://127.0.0.1:8080/v1`).  
hindsight-api is a consumer of hal0-api, not the reverse.
- **No TOML config** — every setting is an inline `Environment=` line in the  
unit itself, with one exception: `EnvironmentFile=-/etc/hal0/hindsight-llm.env`  
(the leading `-` means “ignore if absent”) overrides the inline  
`HINDSIGHT_API_LLM_API_KEY=hal0-local-noauth` placeholder with the real  
client key. `installer/install.sh` writes that file at install time from  
`HAL0_CLIENT_KEY`/`HAL0_ADMIN_KEY` in `/etc/hal0/api.env`, and `hal0 auth rotate` keeps it fresh afterward. A hal0-owned drop-in,  
`hindsight-api.service.d/extraction-model.conf`, overrides the shipped unit  
without editing it. Its model line is always `HINDSIGHT_API_LLM_MODEL=hal0/memory`,  
the virtual id hal0’s `/v1` resolves on every call to the local  
`[memory.graph].extraction_slot` or to the upstream model in  
`extraction_model`. Choosing a slot or an upstream model ( **Engine model** on  
Settings → Data → Memory, or `hal0 memory graph enable --slot|--model|--local`)  
is therefore a config write that applies at once and never restarts the  
engine. The drop-in also carries `HINDSIGHT_API_LLM_TIMEOUT` and the extraction  
limits (`HINDSIGHT_API_RETAIN_LLM_MAX_CONCURRENT`,  
`HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS`,  
`HINDSIGHT_API_RETAIN_LLM_MAX_RETRIES`, `HINDSIGHT_API_WORKER_MAX_RETRIES`,  
`HINDSIGHT_API_WORKER_TASK_RETRY_BACKOFF_SECONDS`, and the derived  
`HINDSIGHT_API_RETAIN_WALL_TIMEOUT`). Changing one of those  
rewrites the file and restarts hindsight-api, and the first save after  
upgrading from a release that pinned a slot id there repairs the drop-in the  
same way. That drop-in lives under `/etc/systemd/system`  
and hal0-api runs unprivileged, so the write and the follow-up restart both  
route through the `hal0-systemctl` wrapper  
(`write-hindsight-dropin` + `svc-restart hindsight`). If the wrapper or its  
sudoers grant is missing, the API’s response carries a non-null  
`propagation.error` and the engine keeps the drop-in it had —  
check `hal0 doctor` for the seam rows. The wrapper allow-lists the drop-in  
body on the root side: only `#` comments, a single `[Service]` header and  
`Environment=` assignments of the seven keys above are accepted  
(`HINDSIGHT_API_LLM_MODEL` must be a `hal0/<name>` value), so a  
hand-crafted fragment (`ExecStart=`, `User=`, any other directive) is  
rejected rather than written. Run  
`hal0-systemctl check-dropin hindsight < body` to see why a body was refused;  
it validates and echoes without writing anything.
- **Overrides in `hindsight-llm.env` win.** systemd applies `EnvironmentFile=`  
after every `Environment=` line in the unit and its drop-ins. A  
`HINDSIGHT_API_LLM_PROVIDER`, `HINDSIGHT_API_LLM_BASE_URL` or  
`HINDSIGHT_API_LLM_MODEL` line in `/etc/hal0/hindsight-llm.env` therefore beats  
`hal0/memory` and whatever you picked under **Engine model** : hal0 reports the  
slot or model you chose while the engine runs the override. The **Engine  
running** row on Settings → Data → Memory (highlighted when an override applies),  
`hal0 memory graph status` and `engine_llm` in `GET /api/memory/graph/status`  
name the provider and model hindsight-api actually loads, read from the  
running engine process; when hal0-api can read neither the process nor the  
file, they say `cannot verify` instead of claiming no override. Remove the lines and  
restart hindsight-api to hand the choice back to hal0. The installer (on a box  
that has a key) and `hal0 auth rotate` rewrite the file with the API key line  
only, which drops any override you added.
- **Data root** — embedded Postgres at `/var/lib/hal0/memory/hindsight/.pg0`,  
pinned via the unit’s `HOME=` env (Hindsight has no dedicated data-dir  
variable of its own).

### Config files it reads

| File | Purpose |
| --- | --- |
| `/etc/hal0/hindsight-llm.env` | Optional `EnvironmentFile`, `-` prefixed (ignored if absent) — supplies the real `HINDSIGHT_API_LLM_API_KEY`, written by `installer/install.sh` and kept fresh by `hal0 auth rotate`. A `HINDSIGHT_API_LLM_PROVIDER`, `_BASE_URL` or `_MODEL` line here overrides hal0’s engine target. |
| `hindsight-api.service.d/extraction-model.conf` | hal0-owned systemd drop-in under `/etc/systemd/system` — sets `HINDSIGHT_API_LLM_MODEL=hal0/memory`, `HINDSIGHT_API_LLM_TIMEOUT` and the extraction limits; written when the timeout or a limit changes (`hal0 memory graph enable`, the Memory settings page) or the file is stale. |

### Health endpoint

`GET http://127.0.0.1:9177/health` — polled by the installer after start (up  
to 40 × 3s) and by `hal0 doctor verify`’s memory-engine check. If  
unreachable while the memory feature is enabled, hal0’s own health report  
degrades to `WARN "engine enabled but :9177 unreachable"` rather than  
failing hard.

## Starting, stopping, and inspecting

Same commands for both units:

```sh
systemctl status hal0-api
systemctl start hal0-api
systemctl stop hal0-api
systemctl restart hal0-api
systemctl enable hal0-api # already enabled by the installer

journalctl -u hal0-api -f # follow
journalctl -u hal0-api -n 40 # last 40 lines — the recovery recipe install.sh itself uses

```

Swap `hal0-api` for `hindsight-api` for the memory engine. `hal0-api`’s own  
self-restart path (the updater) shells out to the same `systemctl restart`  
via the `hal0-systemctl` wrapper, so a manual restart and an update-triggered  
one behave identically.

### Log access over HTTP

Both units’ logs are also available without shell access, proxied from  
journald with secret-redaction applied:

```auto
GET /api/logs?unit=hal0-api&n=200[&since=...&level=...]
GET /api/logs/stream?unit=hal0-api&level=... # SSE tail

```

Swap `unit=hindsight-api` for the memory engine’s logs. Both routes are  
admin-tier, and both read only hal0’s own units: `hal0.target`, any  
`hal0-*` unit (`hal0-api`, `hal0-slot@<id>`, `hal0-agent@<id>`,  
`hal0-openwebui`, `hal0-bench*`, …), `hindsight-api` and `hermes-gateway`.  
Any other unit is rejected with a 400 `system.logs_error`.

## Companion services

The dashboard’s Services page manages four companion services from a single  
registry — `hal0.services.registry.SERVICES` in  
`src/hal0/services/registry.py`. One `ServiceDef` per service names its  
owning unit, its probe strategy, and which lifecycle verbs it permits:

| Service | Purpose | Unit | Allowed actions |
| --- | --- | --- | --- |
| OpenWebUI | Chat UI companion (podman container, LAN `:3001`) | `hal0-openwebui.service` | start, stop, restart, enable, disable |
| ComfyUI | Image generation engine (runs as the seeded `img` slot container, LAN `:8188` by default; `[network].comfyui_listen = "loopback"` fences it to the box) | `hal0-slot@img.service` | start, restart (no stop) |

| Hermes | Bundled agent + its own dashboard (loopback `:9119`) | `hal0-agent@hermes.service` | start, stop, restart, enable, disable |  
| Hindsight | Memory engine — this is the same unit documented above | `hindsight-api.service` | start, stop, restart, enable, disable |

The Open WebUI card also shows an Agents chip with the Hermes profiles wired into the picker, taken from `openwebui_wiring_status`.

#### Hermes systemd drop-in

hal0 ships its Hermes unit settings as  
`/etc/systemd/system/hal0-agent@hermes.service.d/10-hal0.conf`. Put your own  
changes in a separate drop-in with `systemctl edit --drop-in=local hal0-agent@hermes` (it lands as `local.conf` and sorts after `10-hal0.conf`, so  
your values win). Earlier releases shipped this file as `override.conf`, the  
name `systemctl edit` uses by default. Re-running the installer deletes an  
`override.conf` that still matches a hal0-shipped version. An `override.conf`  
you edited is kept and the installer warns, naming the hal0-owned directives it  
repeats. Trim it to your own lines. A drop-in carrying `ConfigurationDirectory=`  
is renamed to `<name>.hal0-disabled`, because that directive crash-loops the  
unit (status 241); the file is kept, not deleted.

#### Hermes gateway and linger

The Hermes gateway runs as `hermes-gateway.service`, a system unit with  
`User=hal0`. `hal0 update` renders the unit from hal0’s own template through  
the root verb `hal0-update hermes-gateway` once the new release is live: it  
orders the gateway after `user@<uid>.service`, turns systemd linger on for  
`hal0`, and restarts the gateway only if it was running. It never enables,  
starts or stops a gateway you disabled. Linger is needed because Hermes’s  
scheduled and kanban workers start through `systemd-run --user` under that  
user’s manager; without it kanban workers never start. `hal0 doctor all`  
shows the linger state on its `hal0_linger` row, and if the verb fails  
`hal0 update` prints `sudo hal0 agent install hermes`. `hal0 agent uninstall hermes` and `installer/uninstall.sh` run `loginctl disable-linger hal0`. A  
gateway the operator disabled or masked is left off by the installer,  
`hal0 agent install hermes` and `hal0 update`.

From Hermes v0.22.3 the gateway daemon owns chat turns and the session store. The dashboard’s default Chat tab still runs its turns in the dashboard process and needs no gateway. A chat opened for another profile (such as `hal0-brain`) and the classic `hermes chat` CLI are gateway clients that attach to `hermes-gateway.service`. hal0 keeps the unit file installed even when you turn the gateway off (it only ever runs `systemctl disable --now`, never removes the file), because an installed unit in any state is what stops a client from starting a gateway of its own. `hal0 doctor` has a `Hermes gateway unit` row (`hermes-gateway-unit`) that fails when the file is missing or a `gateway run --idle-exit` process is running, and the `hermes-api` row warns rather than fails while the gateway is off. To chat in another profile, run the gateway with `sudo systemctl enable --now hermes-gateway`. During `hal0 update`, a gateway unit whose `ExecStart`, `ExecStop` or `ExecStopPost` is not hal0’s launcher (for example one written by `hermes gateway install` outside hal0) is rewritten to hal0’s launcher while the gateway is stopped, before the new release starts.

Hindsight’s entry here isn’t a second thing to learn: the Services page’s  
buttons for it run through the identical `unit_action` → `hal0-systemctl`  
path already described in hindsight-api.service above.

### Start, Stop, Restart

The Services page renders one button per allowed action  
(`ui/src/dash/services.jsx:100-124`): Restart whenever it’s allowed, Start  
when the unit isn’t currently active, Stop only while it is. Clicking one  
calls `POST /api/services/{id}/action` with `{"action": "start"|"stop"|"restart"}`,  
which `hal0.services.systemd.unit_action()` runs against the unit — the  
same verb allow-list and `hal0-systemctl` wrapper described above for  
hal0-api and hindsight-api.

Stop carries a browser confirmation — “Stop `<name>`? Clients using it will  
lose access until it is started again.” — because stopping OpenWebUI,  
Hermes, or Hindsight takes down something other browsers or agents may be  
using right now. It’s a client-side prompt only: it isn’t enforced by the  
API, so a direct `POST` bypasses it. Restart shows no such prompt.

ComfyUI has no Stop button at all — the registry grants it only `start`  
and `restart`. Its “start” is not a raw `systemctl start`: the route  
substitutes the GPU-arbiter switchover (`_comfyui_start`,  
`src/hal0/api/routes/services.py:187-208`, delegating to the same  
`comfyui_switchover` handler the FirstRun repair flow uses) so resident LLM  
slots get drained before the iGPU is handed to ComfyUI. Stopping is  
arbiter-managed too, per its card hint: “stop is GPU-arbiter managed —  
switch back to inference in the Image-Gen pane”.

The registry also allows `enable`/`disable` for OpenWebUI, Hermes, and  
Hindsight, but the Services page renders no buttons for those two verbs —  
only `POST /api/services/{id}/action` with `{"action": "enable"|"disable"}`  
reaches them.

There’s no `hal0 service` command group, and for OpenWebUI and Hermes  
specifically there’s no CLI equivalent for start/stop/restart. `hal0 app list` and `hal0 agent list`/`hal0 agent status` show a service’s systemd  
enabled/active state, but neither `hal0 app` nor `hal0 agent` has a  
start/stop/restart subcommand — for those two services, the dashboard  
buttons, or a direct `systemctl {start|stop|restart} <unit>` as root (same  
mechanism as the platform units above), are the only ways to drive these  
verbs today. ComfyUI is the exception: its unit is `hal0-slot@img.service`,  
the same seeded `img` slot the Slots page manages, so `hal0 slot restart img`, `hal0 slot load img`, and `hal0 slot unload img` reach it from the  
CLI.

### Health

Two endpoints report health, for different consumers:

- `GET /api/services` — the Services page’s own source. Every entry  
carries `up` (bool), a probe-specific `detail` string, and the raw  
systemd `unit_state` (`active_state`/`sub_state`/`unit_file_state`/`since`).  
The page’s status pill is green **up** when `up` is true, red **failed**  
when `unit_state.active_state` is `"failed"`, grey **down** for any other  
managed service, and a plain grey **—** for an unmanaged one  
(`ui/src/dash/services.jsx:67`).
- `GET /api/services/health` — a stable summary over the same service  
catalog (OpenWebUI, ComfyUI, Hermes, Hindsight) that feeds the Overview  
page’s Services widget. It collapses the same probes into  
`state: "up" | "stopped" | "down"`, splitting “not up” into a deliberate  
`stopped` (unit `inactive` — comes back on demand) from `down` (unit  
`failed`, or reachable-but-unhealthy).

Each service is probed differently: OpenWebUI is a loopback  
`GET http://127.0.0.1:3001/health` (2xx = up); ComfyUI is its own  
`/system_stats` + `/queue` (reachable = up, with a live “N running / M  
queued” job count); Hermes and Hindsight both fall back to systemd  
`active_state == "active"`. Every probe is fail-soft by design — a probe or  
systemd failure degrades to `up=false` with an honest detail, never a  
fabricated “up” and never a 500. A service with no wired probe would report  
`up: false, state: "down", detail: "unmonitored"`, though none of the four  
registered services hits that path today. The summary’s probes run  
concurrently, so one hung or raising probe reports its own row as `down`  
without dropping the rest.

### mDNS discovery toggle

The DISCOVERY (mDNS) card above the service grid  
(`DiscoveryCard`, `ui/src/dash/services.jsx:200-241`, toggle button at  
`:214-219`) toggles `POST /api/services/mdns` with  
`{"advertise": true|false}`. Turning it on writes one avahi service-group  
file — `/etc/avahi/services/hal0-addon-<id>.service` — per service that is  
both mDNS-capable and LAN-published: OpenWebUI (`:3001`) and, by default,  
ComfyUI’s editor (`:8188`), so other machines reach them at  
`http://<host>.local:<port>`. Hermes and Hindsight listen on the box’s  
loopback, so they’re never mdns-capable. ComfyUI’s LAN publication is the  
`[network].comfyui_listen` key in `hal0.toml` ([ADR-0041](https://github.com/Hal0ai/hal0/blob/main/docs/adr/0041-the-comfyui-editor-answers-on-the-lan.md)):  
`"lan"` (the default) binds the LAN and makes it advertisable; `"loopback"`  
fences `:8188` to the box, and the card then neither lists nor announces it  
(its link answers only on the box itself or through an SSH tunnel). A change  
applies at the next `hal0 slot restart img`. avahi-daemon inotify-watches that  
directory, so the announcement takes effect immediately — no reload, no unit  
restart. Turning it off removes every `hal0-addon-*.service` file. A ComfyUI  
announcement written while it was on the LAN stays after you switch to  
`"loopback"` until you turn the toggle on or off again; either removes it.

The toggle never touches `hal0.service`, the base dashboard announcement  
the installer writes separately — that’s why “Dashboard announced as  
`<host>.local`” can be true even when the addon toggle is off. If  
avahi-daemon isn’t active, the button is disabled and the card tells the  
operator to install/enable it or add a static hosts entry.

## See also

- [Authentication](https://forum.hal0.dev/t/authentication-hal0-docs/43) — `/etc/hal0/api.env`, the  
`EnvironmentFile` `hal0-api.service` loads, is where the admin/client keys  
described there actually live.