All roots are resolved by hal0.config.paths and are HAL0_HOME-aware — see
Environment variables. Production layout (with
HAL0_HOME unset) follows the FHS:
| Root | Path | Purpose |
|---|---|---|
usr_lib() |
/usr/lib/hal0/current |
Code — a symlink to the currently active versioned install dir. |
lib() |
/usr/lib/hal0 |
Shipped read-only tree: bin/ wrappers, versions/, hooks. |
etc() |
/etc/hal0 |
User-editable config, preserved across updates. |
var_lib() |
/var/lib/hal0 |
Mutable runtime state, preserved across updates. |
var_log() |
/var/log/hal0 |
Optional supplementary log files — journald is the primary log sink. |
With HAL0_HOME=$X set (dev installs, tests): usr_lib = $X/usr-lib/hal0/current,
lib = $X/usr-lib/hal0, etc = $X/etc/hal0, var_lib = $X/var-lib/hal0,
var_log = $X/var-log/hal0.
/etc/hal0/ layout
/etc/hal0/
├── hal0.toml # main config
├── api.env # 0600, systemd EnvironmentFile — secrets/keys
├── openwebui.env
├── hardware.json # probe snapshot
├── manifest.json # release manifest, image digest pins
├── profiles.toml # optional — seeds apply if absent
├── stacks.toml # optional
├── slots/ # one <name>.toml per slot
├── agents/<name>.toml # per-agent MCP allowlist/config
├── hermes-python.env # HAL0_HERMES_PYTHON=<interpreter> hal0 chose for the Hermes venv; optional operator override
├── hindsight-llm.env # root:hal0 (0640 from the installer, 0600 after hal0 auth rotate), optional EnvironmentFile of hindsight-api: HINDSIGHT_API_LLM_API_KEY, kept fresh by hal0 auth rotate; a PROVIDER, BASE_URL or MODEL line here overrides the Engine model choice
├── mcp-servers/<id>.toml # 0600, one record per user-installed MCP server
├── oauth-providers.toml # OAuth provider registry, seeded once from shipped defaults
└── *.lock # advisory RMW lock siblings
api.env is written with mode 0600 — it carries the admin/client bearer keys and
other secrets. See Config schema for every file’s
key layout.
/var/lib/hal0/ layout
/var/lib/hal0/
├── hal0.db # primary SQLite — registry, PortAuthority, metrics, runtime state
├── activity.db(+wal/shm) # audit trail SQLite
├── .first-run.lock # 0600, single-use OTP for first-run claim
├── .first_run_done # first-run completion sentinel
├── registry/
│ └── ignored-paths.json # model paths the discovery scan skips (deleted models' kept files)
├── backups/ # 0600 tar.gz archives written before a one-shot conversion: runners-<ts>.tar.gz, and hal0-pre-015-<ts>.db, a 0600 copy of hal0.db taken before schema migration 015 drops retired model tuning (an older box may also hold profiles-<ts>.tar.gz and the profile-centric*.json records from the 1.4.0 conversion; nothing reads them now and they can be deleted)
├── models/ # default model cache (unless overridden — see below)
├── slots/<name>/ # per-slot working dir
├── agents/
│ ├── secret.bin # 0600 HMAC secret for dashboard session cookies
│ └── <name>/workspace/ # per-agent sandbox root (chroot/bind-mount target)
├── stacks/state.json # active-stack pointer + content hash
├── mcp/hermes-managed.json # MCP server ids hal0 itself joined into Hermes config
├── venvs/hermes -> /var/lib/hal0/hermes/current/venv # fixed symlink; a real directory only before the first migration
├── hermes/ # Hermes release tree (hal0:hal0), see below
├── python/ # legacy uv interpreters from before the release layout; root-owned on older installs, never written by builds
├── openwebui/ # OpenWebUI state
└── .config/flm/models/ # default FLM (NPU) model cache
Hermes release tree
Hermes is installed as one release directory per build, behind a current
symlink (ADR-0032, ADR-0034). Only current moves when a release is
activated; the agent unit, the hermes wrapper and the gateway unit reach
the venv through /var/lib/hal0/venvs/hermes or /var/lib/hal0/hermes/current/.
/var/lib/hal0/hermes/
├── .lock # flock held by every build, activation and prune; holds the holder's pid, caller and start time
├── mirror.git/ # blobless bare mirror of NousResearch/hermes-agent, fetched by sha with tags
├── releases/<sha12>-<UTC stamp>/
│ ├── src/ # git worktree of mirror.git, detached at the vetted sha
│ ├── venv/ # editable install against src/
│ └── release.json # ref, describe, python, lock, install mode, platform_extras, web_dist, hal0 version; written last
├── releases/legacy-<sha12>/ # the pre-migration 3.12 venv, moved here on the first migration (no src/)
├── python/ # uv-managed interpreters for Hermes builds (UV_PYTHON_INSTALL_DIR)
├── .cache/uv/ # uv cache for Hermes builds (UV_CACHE_DIR)
├── current -> releases/<id> # the live release
└── previous -> releases/<id> # the automatic flip-back target
Builds run as the hal0 user, so uv’s interpreters and cache live inside
this hal0-owned tree. The older /var/lib/hal0/python is root-owned on
existing installs; builds never write it, and it is removed only by a root
path (installer/uninstall.sh, or the root prelude of hal0 agent reprovision once nothing uses the interpreter in it).
A release without release.json is incomplete and is deleted by the next
build. Every successful activation keeps current and previous and deletes
every other release, except one that /etc/systemd/system/hermes-gateway.service
still names. Each activation also writes the venv.pin stamp under
/var/lib/hal0/state/agents/hermes/ from release.json. The same directory
holds dashboard-session-token (0600, hal0), the stable dashboard session
token hal0-agent@hermes mints on its first start and exports to the Hermes
dashboard (hal0 agent dashboard-token). Steady state is two
releases (a few hundred MB each), the mirror (a blobless partial clone,
about 200 MB at the current pin) and the uv cache.
Model store resolution
The default model cache is /var/lib/hal0/models. Resolution order:
HAL0_MODEL_STOREenv var, if set.[models].store(or the deprecated[models].pull_root) inhal0.toml, via
effective_store()./var/lib/hal0/models(themodels_dir()default).
[models].roots (default: just the effective store) plus the effective store itself,
deduplicated, form the full scan set (scan_roots()).
FLM (NPU) store
The FLM store resolves from HAL0_FLM_MODELS_DIR, then [models].flm_store, then
/var/lib/hal0/.config/flm/models (config.paths.flm_models_dir()). The NPU slot
bind-mounts the resolved store. The host flm binary always writes to
$HOME/.config/flm/models, so when the store is elsewhere hal0 replaces that path with a
symlink to the store before each pull. .config and .config/flm must be owned by
hal0 for that to work. If the link can’t be made, the pull fails with
model.flm_store_unlinked. sudo hal0 doctor perms --fix hands .config and
.config/flm to hal0. Then, running as hal0, it moves models stranded in the default
dir into the store and creates the link. Last, it gives the store 1000:hal0 mode 2775
(the FLM container’s uid), but only if hal0 owns it and the path ends in flm/models. It follows no
symlink as root. When the same name exists in both directories it stops and reports it;
it never deletes either copy. A store path hal0 can’t create has to be created by hand
(install -d -o 1000 -g hal0 -m 2775 <store>).
DEFAULT_MODEL_STORE = /mnt/ai-models is a historic convention for an external
NFS/fast-disk mount used as a slot container bind target — it’s not a resolver default,
just a documented convention some installs still use.
Other notable paths
| Path | Purpose |
|---|---|
~/.local/share/hal0/models |
XDG-style per-user model store — a UI display option only, not a resolver default. |
$PWD/.hal0ai (or $HAL0_PREFIX) |
Dev-mode install root, used by hal0 uninstall --dev. |
$HERMES_HOME (default ~/.hermes) |
Hermes agent home directory. |
Lock files
hal0 uses a generic sibling-lock pattern (hal0.config.locking.file_lock()):
<target>.lock next to any config file, advisory fcntl.flock.
| Pattern | Purpose |
|---|---|
/etc/hal0/*.lock |
Advisory read-modify-write locks on config writes. |
/var/lib/hal0/*.lock |
Advisory read-modify-write locks on state writes. |
<slots_dir>.lock |
Slot-config directory-level lock. |
<record>.lock |
Per-record lock for MCP installed.py read-modify-write. |
Temp file prefixes (atomic-write pattern)
hal0 writes config and state files via tempfile.mkstemp in the same directory,
fsync, then os.replace — so readers never see a torn write. Recognizable prefixes;
every one of these is transient — created, then either unlinked or renamed away,
never left on disk as a final artifact:
| Prefix | Used by |
|---|---|
.hal0-writeprobe-{pid} |
Write-permission probe (hal0.agents.hermes_provision.path_is_writable) — touched then immediately unlinked in the same call, to test whether an ancestor directory is actually writable (not just os.access-writable, which lies under SELinux/ACLs/NFS root-squash). Never renamed into place. |
.hal0-gpu-arbiter-*.tmp |
GPU arbiter state. |
.hal0-state-*.tmp |
Slot state / ID-keying migration. |
.hal0-stack-state-*.tmp |
Stack state. |
.hal0-env-*.tmp |
Atomic env-file writer (config/env.py). |
Persistent markers
These are not temp files — each is written once and left in place as the
final artifact, checked again on a later run. Don’t confuse them with the
atomic-write temp files above.
| Marker | Used by |
|---|---|
.hal0-managed |
Stamped into a claimed HERMES_HOME/agent data dir the moment hal0 takes ownership of it (hal0.agents.hermes_provision). It persists for the life of the install: hal0 agent uninstall rmtrees an agent home only when this marker is present, refusing to touch a directory (e.g. a user’s pre-existing ~/.hermes) that lacks it — see _safe_to_remove_data_dir at src/hal0/agents/manager.py:507 (called from manager.py:466). |
.hal0-build-stamp |
Content hash of the ui/ source tree, written under the served UI dist dir (${UI_DIST}/.hal0-build-stamp) right after a successful npm run build (installer/install.sh:1109). Persists across installs — the installer diffs it against the current tree hash on the next run and skips the rebuild only on an exact match. |
Portable export envelope formats
| Extension | Purpose |
|---|---|
.hal0profile.json |
Portable profile export/import envelope. |
.hal0stack.json |
Portable stack export/import envelope. |
For who writes each of these files, when they get rewritten, and whether a hand
edit survives the next write, see Generated & rewritten files.