Architecture overview
This page is the map. It exists so that when you go into Lohar, Thermal states, Networking, or Decisions, you already have a sense of where each fits.
If you're using bhatti, you don't need this page. If you're evaluating it for production or thinking about contributing, this is the right starting point.
The shape
bhatti has one Go host daemon, one bhatti-vmm helper per running
sandbox, and lohar inside each VM. The daemon never links the VMM;
it controls it out of band over a Unix socket and talks to lohar over
vsock. Only owners running networked sandboxes also get a
bhatti-netd gateway; new sandboxes have no NIC by default.
bhatti is a single Go binary that does many jobs depending on how
you invoke it:
bhatti serve— the daemon (HTTP API, krucible engine, thermal manager, public proxy, all in one process)bhatti create,bhatti exec,bhatti shell, ... — the CLI client, talking to the daemon's HTTP APIbhatti user create,bhatti admin status— admin commands that read the daemon's SQLite database directly (root only)
Two more binaries ship in the release bundle and are launched by the daemon, never by you:
bhatti-vmm— the cgo helper that is the VM. It links krucible (bhatti's fork of libkrun) and callskrun_start_enter, at which point the process becomes the guest and blocks for the life of the VM. The daemon controls it from outside over a Unix-socket control channel.bhatti-netd— a per-owner userspace gVisor gateway for networked sandboxes. It enforces each sandbox's outbound policy and optional secret grants. Sibling traffic is denied unless the sending sandbox opts in. No network device means no netd for that sandbox.
lohar runs as PID 1 inside every microVM. It handles exec, file
operations, PTY sessions, and pretends to be systemctl and
journalctl for in-guest callers. The full story is in
Lohar.
┌─ Host ──────────────────────────────────────────────────────────────────────┐
│ │
│ ┌─ bhatti daemon (bhatti serve, single pure-Go process) ───────────────┐ │
│ │ │ │
│ │ REST / WS API Engine (krucible) Store (SQLite, WAL mode) │ │
│ │ api.sock create / destroy sandboxes, users, secrets, │ │
│ │ public proxy* stop / start templates, volumes, events │ │
│ │ thermal manager pause / resume │ │
│ │ snapshot / fork │ │
│ │ exec, shell, files │ │
│ │ │ │
│ └───────────┬───────────────────────────────────┬───────────────────────┘ │
│ │ │ │
│ control socket (UDS) unix socket (only with NIC) │
│ PAUSE/RESUME/STATUS/SAVE virtio-net │
│ │ │ │
│ ▼ ▼ │
│ ┌─ bhatti-vmm × N (krucible/libkrun VMM) ──┐ ┌─ bhatti-netd × owners ─┐ │
│ │ │ │ optional per-owner │ │
│ │ KVM (Linux) / HVF (macOS) │◄──┤ gVisor gateway │ │
│ │ qcow2 CoW root (/dev/vda) │ │ per-sandbox policy │ │
│ │ boot config via vsock volumes … │ │ grants; siblings opt-in│ │
│ │ │ └────────────────────────┘ │
│ │ ┌─ lohar (PID 1) ─────────────────┐ │ │
│ │ │ vsock ── control plane │ │ │
│ │ │ (exec, files, PTY) │ │ │
│ │ │ vsock ── forward plane │ │ │
│ │ │ sessions scrollback │ │ │
│ │ │ systemctl shim journald │ │ │
│ │ └─────────────────────────────────┘ │ │
│ └───────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
The API defaults to an authenticated Unix socket at
<data_dir>/api.sock (the installer gives the bhatti group access).
Outside domain mode, TCP is opt-in via an explicit server listen:
and is plaintext — do not expose it to an untrusted network. For remote
clients, configure a TLS domain or SSH
to the host. Public HTTPS aliases need domain/TLS configuration;
without domain mode, public_proxy_listen can instead serve a separate
unauthenticated path-based HTTP proxy for development/testing. Neither
is the bearer-authenticated control API proxy. Domain mode ignores
listen and public_proxy_listen. The VMM control socket, agent vsock
and netd gateway are internal.
Host resource placement
On Linux with systemd placement enabled, the daemon and its helpers no longer share one service cgroup. The helpers live in a separate hierarchy:
bhatti.slice
└── bhatti-<owner>.slice
├── bhatti-<owner>-<engineID>.scope bhatti-vmm, one per sandbox
└── bhatti-<owner>-netd.scope bhatti-netd, when networking is used
<owner> is a stable hash of the owner key, not a display name;
<engineID> identifies the sandbox VM. CPU, memory and I/O accounting
are enabled on the slices and scopes. This is placement and accounting,
not resource limits: no memory ceiling, CPU quota or I/O cap is set.
The top slice's CPU weight is 100; each gateway weighs 1000 against a
VM's default 100 under contention. Weights affect relative CPU sharing,
not how much CPU a helper may use when the host is idle.
A new helper waits behind a launch gate until the daemon verifies its scope membership and saves the placement. If placement fails on a host using systemd mode, the launch fails instead of running unaccounted. Recovery is different: an older, unplaced helper can be moved into a scope, but its earlier usage remains charged elsewhere (partial accounting). If a recovered guest cannot be placed, it is kept running and logged as unaccounted, not killed.
The cgroups setting defaults to
auto: use systemd where available, otherwise report none with a
reason. It requires systemd as Linux PID 1, unified cgroup v2 and
permission to create units over the system bus. macOS uses per-process
accounting with auto, not this hierarchy. /health
reports the effective mode.
Four decisions that shape the rest
These aren't all the choices in the codebase, but they're the ones you can't understand the rest of the docs without.
1. The daemon is a single Go process
REST/WS API, krucible engine, thermal manager, public proxy,
SQLite store, rate limiter — all in one process. No microservices,
no message queue, no separate workers. The daemon coordinates the
per-sandbox bhatti-vmm and optional per-owner bhatti-netd helpers, but it
owns all the state and all the decisions. State lives in SQLite
(see Where state lives).
This is deliberately one host and one daemon for the control plane: no message queue or orchestrator is required to manage your own sandboxes on a box.
2. We own the VMM, and drive it out of band
The daemon never links libkrun. Linking it wouldn't work anyway:
libkrun's entry call, krun_start_enter, becomes the VM and blocks
for its whole life — a daemon that called it would stop being a
daemon. So the VMM lives in a separate process, bhatti-vmm, and the
daemon drives it from outside.
That out-of-band channel is a tiny Unix-socket control socket. The daemon writes a verb, the VMM acts, and answers:
PAUSE -> OK | ERR <reason> (freeze the vCPUs)
RESUME -> OK | ERR <reason> (thaw them)
STATUS -> OK running | OK paused
SAVE <dir> -> OK | ERR <reason> (save state; leave paused)
The warm thermal transition uses PAUSE/RESUME, keeping memory in
host RAM. Cold thermal transition and explicit stop power off instead:
the disk is retained, not running processes or /tmp; start
boots from disk and reruns --init. An explicit memory snapshot
uses SAVE <dir> on a running VM to save RAM, devices and vCPU state;
<dir> must be a new absolute directory. The daemon copies disks and
volumes while the VM remains paused, then sends RESUME — no separate
PAUSE before SAVE. snapshot resume loads a snapshot into a new
VM identity, while create --from <sandbox> forks an existing
sandbox's memory/disk state. See
the engine page for the mechanisms.
What I get from owning the VMM: pause, resume, snapshot, restore, and fork exist at all. Upstream libkrun does not ship this set of VM lifecycle primitives, so bhatti maintains the fork (krucible) rather than wrapping the stock VMM. The cost is maintaining that hypervisor fork.
3. No jailer — confine the host helpers
There is no Firecracker-style jailer in v2. Every sandbox has a real
VM boundary — KVM on Linux, HVF on Apple Silicon — with its own
guest kernel. On Linux, v2.4 added no_new_privs and Landlock
confinement to bhatti-vmm; v2.5 added a seccomp deny-list. A root
daemon assigns each sandbox's helper its own unprivileged UID; with
a non-root daemon, helpers keep the daemon's user. Root-launched
helpers drop all capabilities unless virtio-fs mounts require retained
filesystem capabilities. Landlock depends on kernel support: without
it, mount-free helpers warn and run without filesystem confinement,
while virtio-fs mounts are refused. These Linux controls are not
enabled on macOS. The optional Linux bhatti-netd also drops root
when started as root and uses Landlock when available. These controls
limit what a compromised helper can access; they are not a promise of
hostile multi-tenant isolation.
A no-NIC guest has no guest-network path to the host. When networking
is enabled, netd authenticates each VM link and enforces its egress
policy. Host addresses are denied except for configured daemon TCP
service listener address/port pairs when the remaining policy permits
them; private daemon-service addresses need an explicit CIDR allowance,
and API authentication still applies. Other host ports, gateway and
other-owner addresses stay blocked; same-owner siblings need
--allow-siblings. No host TAP, bridge or iptables rule is involved.
Host mounts are separately limited by mount_roots.
See Networking and
Network & egress.
4. Per-VM mutex with capture-and-release
Concurrency is the part that bites every "single Go process" design eventually. bhatti's pattern is small but load-bearing.
Each VM struct has two mutexes
(pkg/engine/krucible/engine.go):
launchMu serializes lifecycle transitions (launch / start / stop /
pause / resume / destroy) so a burst of concurrent wake-on-request
calls can't double-spawn the helper; mu guards field reads and
writes. The lock order is always launchMu before mu, never the
reverse — read-only Status/List take only mu and must not block
behind an in-flight transition. The engine-level sync.RWMutex only
protects the map of VMs.
The pattern every operation follows:
vm.mu.Lock() // 1. take the lock
if vm.Status != "running" { // 2. check invariants
vm.mu.Unlock()
return errNotRunning
}
ag := vm.Agent // 3. capture references we need
vm.mu.Unlock() // 4. release the lock immediately
return ag.Exec(ctx, cmd) // 5. do the slow work outside the lock
The key move is steps 3-4: copy the references, drop the lock,
then make the network call. The lock protects access to vm.Agent
itself, not any operation that uses it.
The reason this matters: a Shell or Tunnel call lasts as long as
the user's session — minutes, hours. If those calls held the lock
for their duration, the thermal manager couldn't pause any VM
(because every cycle iterates the VM map and tries to read each VM's
thermal state). One slow shell would freeze idle pause / cold
snapshot for everything else.
The safety argument: vm.Agent is only replaced during a launch
(cold restore), and that path takes launchMu and then mu to do
it. So if a Shell goroutine grabs vm.Agent, releases the lock, and
then a restore runs to completion and replaces vm.Agent with a new
client — the Shell goroutine still has a valid reference to the old
agent's vsock connection. That connection is good until the
bhatti-vmm process is killed, which happens only on Stop() or
Destroy(). By that point the connection is closed, the Shell sees a
read error, and cleans up.
So: long-running operations don't block the system, and they don't explode when state is updated underneath them — they just see the underlying connection close and exit cleanly. The pattern is in every method on the krucible engine that touches an agent.
Where state lives
State lives in SQLite and per-sandbox on-disk VMM state (disk overlays, recovery metadata and checkpoint bundles) under the data directory.
SQLite. One database file at /var/lib/bhatti/state.db on a
default server install. Tables:
sandboxes— id, name, owner, status, engine IDusers— name, hashed API key (SHA-256), per-user limitssecrets— name, encrypted value (age), scoped per usertemplates,volumes,publish_rulesevents— audit logtask_progress— async tasks (image pull)
WAL mode is on so background workers (the thermal manager, port scanner, metrics snapshotter) don't block HTTP request handlers and vice versa.
Per-sandbox. Under the data directory's sandboxes/<id>/, the
qcow2 copy-on-write root overlay and attached volumes preserve files;
the guest's boot configuration comes over vsock, not a persistent
config drive. Recovery metadata (state.json) records the helper
PID and sockets. Warm sandboxes keep VM state in host RAM; thermal
cold and explicit stop instead keep the disks while powering off.
A named memory snapshot separately
preserves memory, disks and copies of attached volumes.
The root overlay is a qcow2 copy-on-write image stacked over a shared read-only base. CoW lives at the image-format layer rather than requiring btrfs or host reflinks; this is how the same VM model works on Linux filesystems and APFS. v1 needed host filesystem cloning. See Storage for the disk and page-cache story.
Other disk locations under /var/lib/bhatti/:
runtime/— the relocatable v2 runtime bundle, re-laid on every install/update:bin/(bhatti-vmm,bhatti-netd),lib/(libkrun + symlinks), andkernel/(the lean external kernel)images/— read-only base rootfs templates (rootfs-minimal-arm64.ext4etc.), CoW-overlaid per sandboxvolumes/<user_id>/— standalone persistent volumes (created bybhatti volume create, attachable to any sandbox as/dev/vdc+)snapshots/<user_id>/<name>/— named snapshot artifacts; memory snapshots include RAM/device state, root-disk and volume copies, and the source network policy.--type filesystemcaptures only the root disk, not attached volumes or network policy, and restores with a cold boot.snapshot exportpackages artifacts for another compatible host; use--include-basewhen the destination lacks the required base image. See Moving between hosts.age.key— the X25519 private key for encrypted stored secrets and sandbox CA keys. Created on the first encryption operation, not necessarily at daemon startup. Back this up with the database; without it encrypted material is unrecoverable.
There are no jails/, firecracker.sock, or persistent boot-config
drive under a v2 sandbox directory — those belonged to the older
layout. Boot settings are sent over vsock rather than being placed in
a snapshot-exportable disk.
Recovery on startup
When the daemon starts — fresh boot, after a crash or after a service restart (systemd on Linux, launchd on macOS) — helper processes from the previous run may still be alive. Recovery inspects their state and re-adopts running helpers when possible.
Recovery is driven by the per-sandbox state.json, not the
database. Each sandbox's file records everything needed to re-adopt
or re-launch it: the helper PID, the socket paths, the base spec, the
bundle dir, and (for the net backend) the owner's bhatti-netd key
and subnet index. bhatti-vmm and bhatti-netd are spawned
detached from the daemon's process group, so they survive a daemon
restart or crash.
On startup the engine globs every sandboxes/*/state.json and, for
each one:
- Adopts a live helper. If the recorded PID is still alive and its agent answers a probe over vsock, the sandbox is adopted as-is — running, no restart, sessions intact.
- Recovers a dead helper from retained state. The persisted
sandbox disk remains, but RAM and running processes are gone unless
an explicit compatible checkpoint bundle exists to restore.
A powered-off sandbox starts by booting from disk; its
--initcommand runs again.
The engine re-adopts a surviving bhatti-netd only for owners whose
networked sandboxes require one. A no-NIC sandbox has no gateway to
reconnect. Recovery writes reconciled status back to state.json.
There is no system-wide device cleanup step, because there are no
host TAP devices, bridges, or iptables rules to leave behind — the
network lives entirely inside bhatti-netd. That whole class of
"stale plumbing after a crash" problem doesn't exist in v2.
What's not on this page
This was the map. The depth is in the topic pages:
- Lohar — the agent inside every VM — PID 1, the systemctl shim, sessions, file ops
- Thermal states — hot / warm / cold, pause/resume, snapshot mechanics
- Networking — the
bhatti-netdgVisor gateway, policed egress, sibling reachability, vsock forward - Wire protocol — frame format, all frame types, atomic writes
- The krucible engine — the control
socket, the
krun_*boot sequence, block-root qcow2, the lean kernel, and why bhatti owns the VMM - Decisions & learnings — the bugs we paid for, in narrative form