← Selected work

Sovereign

A sovereign AI stack: your models, your machine, one endpoint — private by construction, hot-swapped on consumer silicon.

Stack: llama.cpp · llama-swap · ROCm · GGUF · Python Status: Public repo, daily-driver Links: github.com/achagani/local-ai

Product shot — status, one OpenAI-compatible endpoint, model hot-swap in under 4 seconds (recreated from the running stack).

01The problem

Local AI usually arrives as a pile of scripts. A web UI here, a launch script there, a folder of model weights nobody has pruned since March. Every app that wants an OpenAI-compatible endpoint gets its own port and its own mental model; switching models means quitting, waiting, restarting. Nothing fails loudly when a config drifts or a weight file goes missing. It works — the way a workshop works. The goal with Sovereign was the same stack behaving like an appliance: one endpoint, one service, models that swap while everything keeps running.

02The approach

Three layers with a strict division of labor. A custom llama.cpp build does the inference. llama-swap owns the public API and the engine lifecycle. A Python launcher resolves configuration and weights, then executes the selected native engine. Everything downstream — GUIs, agents, editor plugins — talks to one OpenAI-compatible endpoint at 127.0.0.1:1234/v1 and never learns what sits behind it.

Profiles are the unit of configuration: a balanced 27B Qwen profile with thinking on, a fast variant with thinking off, a 12B Gemma for lighter work, an experimental long-context flash profile. The model field in the API request selects the profile; llama-swap's exclusive groups handle the swap. One profile is resident at a time — the four Qwen "slots" mean four concurrent requests to one model, not four loaded models. The CLI refuses a conflicting load without --replace; the GUI and API can trigger automatic swapping on demand.

Config is layered TOML: common defaults, then an inherited profile, then a per-platform mapping, merged in a fixed order. The install script verifies published archive SHA-256s for llama-swap and the stock runtime, and records the custom engine's hashes in a runtime manifest. A systemd user service starts the manager at login with no model preloaded — RAM is only spent when a model is actually asked for.

03Trade-offs

GGUF quantization — quality versus fit. The 27B profile runs in ROCmFP4 with a draft model for speculative decoding: a 27B-class assistant that fits in consumer memory. Quantization is a lossy choice made deliberately, per profile, and the balanced/fast split exists because the same weights serve different quality-latency postures with a single toggle.

llama-swap — freed idle RAM versus cold-start latency. One resident model means a switch costs a reload. The mitigation is semantic: unload waits for busy slots to go idle before releasing (with --force to skip), and on-demand requests transparently reload an unloaded model rather than failing.

Consumer unified memory — the Strix Halo's 128 GB makes model configurations that would need a server rack possible at consumer prices. Prefill measures around 1,200–1,400 tok/s on this hardware: not datacenter throughput, but enough that a 27B model feels immediate for the agentic workloads that dominate daily use.

Staying local — no rate limits, no subscriptions, no third party seeing the prompts. The cost is access: this stack will never be the frontier. It is tuned to be good enough, always on, and yours.

04Under the hood

The paranoia is in the details. A verify command checks weights against catalog hashes and records destination path, size, mtime, and hash in a verification file. Storage roots carry volume markers — a sentinel file that catches the wrong drive mounted at a reused path. Hot reload of config is deliberately disabled: engine startup rejects drift rather than combining old manager settings with new flags, and show-config prints resolved paths and a sample argv before you commit to a change. Unloading is documented as not an atomic admission barrier — the README says so out loud, because a system you trust has to tell you where its edges are. A unittest suite and a smoke test against a loaded model gate every change.