Chat, retrieval over your own documents, code execution, voice and research agents — served from a machine you control. No third-party model API sits on the serving path.
Every capability listed here is built and covered by tests. Anything not yet built is named as such, further down.
Conversations persist, older turns compact into summaries, and durable facts you state once are recalled later. You can read, edit and delete everything it remembers.
Upload a file and ask about it. Retrieval fuses vector similarity with keyword search, and answers cite the passages they used so you can check them.
Write Python and execute it in a sandbox with no network and a hard timeout — the same sandbox the model uses, so what you see is what it sees.
Speech-to-text and text-to-speech run on this machine. Your microphone audio never leaves the host — which is the point of self-hosting a thing that listens.
Plans sub-questions, searches, reads real pages and answers with sources. Bounded by design: capped steps, capped reads, and a wall-clock deadline.
Every request is logged with its route, sources and tool calls, with personal data stripped at write time. The log is the audit trail and your future training set.
One environment variable chooses where tokens come from: an in-process stub for development, llama.cpp on this machine, or vLLM on a GPU you rent. Nothing above that layer knows which it got, so upgrading the model is a config change and an eval run.
Figures from this machine — a 2019 Intel MacBook Pro with no CUDA, running CPU-only. A GPU backend is a different order of magnitude; these are the honest numbers for the hardware in front of you.
| Model | Decode | First token | Eval score |
|---|---|---|---|
| Llama 3.2 1B, Q4_K_M | 6.0 tok/s | 1.6 s | 73% |
| Llama 3.2 3B, Q4_K_M | 3.0 tok/s | 5.8 s | 63% |
| Gateway overhead alone | 113 req/s | 86 ms p50 | — |
The eval suite is 16 chat cases plus 13 capability cases. That is too small to rank two models against each other — the gap above is a two-case swing — but large enough to catch a regression. Not yet built: image generation, vision input, and custom personas.
