Speech to text and text to speech

system · August 22, 2026, 6:50am

hal0 exposes two voice directions through OpenAI-compatible endpoints:

Speech-to-text (POST /v1/audio/transcriptions) — served from the
single canonical stt slot, which can run either engine: Moonshine
(ONNX) on CPU, or whisper-v3:turbo on the XDNA NPU, co-loaded with the
chat model in one FLM process (the “NPU trio”: chat + STT + embed).
Choosing a device swaps which engine the stt slot runs — there’s no
separate slot to configure.
Text-to-speech (POST /v1/audio/speech) — served from the single
canonical tts slot, which can run either engine: Kokoro-82M ONNX
on CPU, or Qwen3-TTS on GPU (ROCm, native gfx1151). Choosing a device
swaps which provider the tts slot runs — there’s no separate slot to
configure.

Both are children of the voice capability: voice.stt maps to the
stt slot and voice.tts maps to the tts slot. Both children are
now device-keyed engine switches with the same shape. Before this
redesign, voice.stt could only ever run backend=npu — any host
without the XDNA NPU had zero speech-to-text. Reinstating Moonshine as a
CPU-side STT engine and keying stt on device the same way tts
already was closed that gap (see ADR-0001, “Reinstate Moonshine as the
CPU STT engine”, docs/adr/0001-moonshine-cpu-stt-reinstatement.md in
the repository — an internal design record, not published to these
docs).

Enable the voice capability

hal0 capabilities set voice stt --model <your-stt-model>
hal0 capabilities set voice tts --model <your-tts-model>

Or over the API, which accepts any subset of { backend, provider, model, enabled }:

Speech-to-text (CPU, Moonshine)

curl -X POST http://localhost:8080/api/capabilities/voice/stt \
  -H 'Content-Type: application/json' \
  -d '{"enabled": true, "backend": "cpu", "provider": "moonshine"}'

Speech-to-text (NPU, FLM)

curl -X POST http://localhost:8080/api/capabilities/voice/stt \
  -H 'Content-Type: application/json' \
  -d '{"enabled": true, "backend": "npu", "provider": "flm"}'

Text-to-speech (CPU, Kokoro)

curl -X POST http://localhost:8080/api/capabilities/voice/tts \
  -H 'Content-Type: application/json' \
  -d '{"enabled": true, "backend": "cpu", "provider": "kokoro"}'

Text-to-speech (GPU, Qwen3-TTS)

curl -X POST http://localhost:8080/api/capabilities/voice/tts \
  -H 'Content-Type: application/json' \
  -d '{"enabled": true, "backend": "gpu-rocm", "provider": "qwen3tts"}'

The `voice.stt` engine switch

Device	Provider	Runs on
`cpu`	`moonshine`	CPU (ONNX)
`npu`	`flm`	XDNA NPU, coresident with chat + embed in the FLM trio

There is deliberately no GPU row — hal0 ships no GPU STT engine. This is a
two-engine, two-device special case, not a general provider selector: if
a second CPU (or GPU) STT engine ever lands, device alone stops being
enough to disambiguate and provider must become a first-class part of
capability selection — this future-work trigger is recorded in ADR-0001
(docs/adr/0001-moonshine-cpu-stt-reinstatement.md, repo-only).

The `voice.tts` engine switch

Device	Provider	Runs on
`cpu`	`kokoro`	CPU (ONNX)
`gpu-rocm`	`qwen3tts`	Strix Halo iGPU (ROCm, native gfx1151)

Selecting a device rewrites which provider the tts (or stt) slot
loads — the slot itself doesn’t move. Qwen3-TTS and Moonshine weights are
operator-staged (not pulled through the hal0 registry; whisper-v3:turbo on
the FLM/NPU path resolves through FastFlowLM’s own model handling instead).
Moonshine’s weights preflight at slot spawn — a missing or empty bundle
fails loudly, by name (slot.weights_missing), rather than starting a
container that 500s on the first request.

Kokoro weights

Kokoro’s weights come from the registry: hal0 model pull kokoro-v1
installs the ONNX model and the voices pack into the model store, at
<store>/tts/kokoro-v1/kokoro-v1.0.onnx and
<store>/tts/kokoro-v1/voices-v1.0.bin. <store> is the configured
[models].store, /var/lib/hal0/models on a fresh install. The tts slot
mounts the store read-only and hal0 passes that directory as the runner’s
--model_path, so the kokoro profile carries no path.

If either file is missing, the slot refuses to start with
slot.weights_missing and names the command to run
(hal0 model pull kokoro-v1). It never falls back to downloading the
weights at start-up, so an offline box runs Kokoro once the bundle is
pulled.

On upgrade to 1.4.0, hal0 update pulls the bundle once if a slot or
capability selection uses Kokoro and the store does not have it yet. This
also replaces the single model.gguf an earlier pull left there (the ONNX
file under the wrong name, with no voices pack). On a box without internet
access the update still completes; run hal0 model pull kokoro-v1 once it
is online. Until then the tts slot’s error names the same command.

Staging the Moonshine weights

The bundle is a directory the loader reads directly — it resolves
encoder_model.ort (or .onnx) plus decoder_model_merged.* and the
tokenizer inside whatever directory it is handed. Two consequences:

Stage a non-streaming variant. The base-en bundle has the file
names the loader wants. The streaming bundles (small-streaming-en,
medium-streaming-en) ship a different file set (encoder.ort,
frontend.ort, adapter.ort) belonging to the Moonshine streaming SDK,
which this image does not consume. Pointing the slot at one is rejected
at spawn with slot.weights_missing naming the streaming case.
Point --model_path anywhere sensible. hal0 walks the tree to find
the directory holding the encoder, so both …/base-en and
…/base-en/quantized/base-en work. If the resolved path is outside the
model store root, the slot mounts it read-only at the same absolute path
(plus the symlink target, if any) so the container can read it.

Verify staging before enabling the capability:

hal0 doctor all --json | jq '.[] | select(.key == "stt-weights")'

A pass names the staged path; a fail names exactly what is missing.

How the CPU STT path works (Moonshine)

Enabling voice.stt with backend=cpu spawns the stt slot’s own
container from the moonshine runner’s image (ghcr.io/hal0ai/hal0-runner-moonshine, MoonshineProvider),
distinct from the NPU path below — it’s a standalone process, not a
co-loaded FLM role. The container wraps an OpenAI-compatible FastAPI
server (POST /v1/audio/transcriptions, GET /v1/models, GET /health).
It also exposes WS /v1/audio/stream for live PCM16 (16 kHz mono)
streaming transcription — this endpoint is in-container only and is
not routed by the hal0 dispatcher; it isn’t part of the public /v1/*
surface. hal0 binds 0.0.0.0:8080 with no built-in auth by design
(LAN-trust posture); voice endpoints accept file uploads on that
unauthenticated bind.

How the NPU STT path works

Enabling voice.stt with backend=npu does not spawn a standalone
process — it drives the FLM trio, one flm serve anchor process serving
chat, transcription, and embeddings together. The orchestrator toggles
the anchor’s [npu].asr flag and writes a type=transcription slot
record for dispatch gating. The anchor is not auto-restarted — the
response carries pending_reload: true, and the change takes effect once
you reload the FLM anchor yourself (hal0 slot restart <anchor-slot>).
NPU transcription requires the FLM chat anchor to already be loaded.

Transcribe audio (speech to text)

curl -X POST http://localhost:8080/v1/audio/transcriptions \
  -F '[email protected]' \
  -F 'model=<your-stt-model>'

The model form field is required; omitting it returns 400
(request.missing_model).

Synthesize speech (text to speech)

curl -X POST http://localhost:8080/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<your-tts-model>",
    "input": "Hello from hal0.",
    "voice": "<voice-id>"
  }' \
  --output speech.wav

model is required. voice, speed, and response_format are
optional — see TTS request defaults below.

TTS request defaults

The tts slot can carry persisted request defaults that
/v1/audio/speech seeds into a request whenever the body omits the
matching field — an explicit value in the request always wins:

default_voice — voice id injected when the request omits voice.
default_speed — playback speed (0.25–4.0) injected when the
request omits speed.
default_response_format — mp3 / wav / opus / flac / pcm,
injected when the request omits response_format.

These are set from the dashboard’s Settings → Voice panel and take effect
immediately — no container restart. The dashboard’s voice picker reads a
live voice list from GET /api/slots/tts/voices; if the tts slot is
cold or unreachable, it falls back to a built-in seed list.