Skip to content
[bhatti]

Architecture overview

This page is the map. It exists so that when you go into Lohar, Thermal states, Networking, or Decisions, you already have a sense of where each fits.

If you're using bhatti, you don't need this page. If you're evaluating it for production or thinking about contributing, this is the right starting point.

The shape

Bhatti is a host daemon plus three helper processes per running sandbox. The daemon (bhatti serve) is pure Go and never links the VMM. Per sandbox it spawns bhatti-vmm (the microVM itself) and, per owner, bhatti-netd (the network gateway); inside each VM, lohar runs as PID 1. The daemon drives bhatti-vmm out-of-band over a control socket and talks to lohar over vsock with a small binary protocol.

bhatti is a single Go binary that does many jobs depending on how you invoke it:

  • bhatti serve — the daemon (HTTP API, krucible engine, thermal manager, public proxy, all in one process)
  • bhatti create, bhatti exec, bhatti shell, ... — the CLI client, talking to the daemon's HTTP API
  • bhatti user create, bhatti admin status — admin commands that read the daemon's SQLite database directly (root only)

Two more binaries ship in the release bundle and are launched by the daemon, never by you:

  • bhatti-vmm — the cgo helper that is the VM. It links krucible (bhatti's fork of libkrun) and calls krun_start_enter, at which point the process becomes the guest and blocks for the life of the VM. The daemon controls it from outside over a Unix-socket control channel.
  • bhatti-netd — a per-owner userspace gVisor network gateway. It polices egress, isolates the guest from the host, and makes same-owner sandboxes reachable — with no host TAP devices, bridges, or iptables rules. On by default.

lohar runs as PID 1 inside every microVM. It handles exec, file operations, PTY sessions, and pretends to be systemctl and journalctl for in-guest callers. The full story is in Lohar.

┌─ Host ──────────────────────────────────────────────────────────────────────┐
│                                                                             │
│  ┌─ bhatti daemon  (bhatti serve, single pure-Go process) ───────────────┐  │
│  │                                                                       │  │
│  │   REST / WS API    Engine (krucible)     Store (SQLite, WAL mode)     │  │
│  │     :8080            create / destroy     sandboxes, users, secrets,  │  │
│  │     public proxy     stop / start         templates, volumes, events  │  │
│  │     thermal manager  pause / resume                                   │  │
│  │                      snapshot / fork                                  │  │
│  │                      exec, shell, files                               │  │
│  │                                                                       │  │
│  └───────────┬───────────────────────────────────┬───────────────────────┘  │
│              │                                   │                          │
│     control socket (UDS)               unix socket (per owner)              │
│     PAUSE/RESUME/STATUS/SNAPSHOT              virtio-net                     │
│              │                                   │                          │
│              ▼                                   ▼                          │
│  ┌─ bhatti-vmm × N  (krucible/libkrun VMM) ──┐   ┌─ bhatti-netd × owners ─┐  │
│  │                                          │   │                        │  │
│  │   KVM (Linux) / HVF (macOS)              │◄──┤  gVisor userspace      │  │
│  │   qcow2 CoW root (/dev/vda)              │   │  netstack gateway      │  │
│  │   config drive (/dev/vdb)   volumes …    │   │  policed egress        │  │
│  │                                          │   │  sibling reachability  │  │
│  │   ┌─ lohar (PID 1) ─────────────────┐    │   └────────────────────────┘  │
│  │   │   vsock  ── control plane       │    │                               │
│  │   │           (exec, files, PTY)    │    │                               │
│  │   │   vsock  ── forward plane       │    │                               │
│  │   │   sessions   scrollback         │    │                               │
│  │   │   systemctl shim   journald     │    │                               │
│  │   └─────────────────────────────────┘    │                               │
│  └───────────────────────────────────────────┘                             │
└─────────────────────────────────────────────────────────────────────────────┘

The daemon's HTTP API is the only public surface. Everything below it — the control socket, the vsock agent protocol, the netd gateway — is internal. A client outside the host talks only to :8080 (or :443 in domain mode).

Four decisions that shape the rest

These aren't all the choices in the codebase, but they're the ones you can't understand the rest of the docs without.

1. The daemon is a single Go process

REST/WS API, krucible engine, thermal manager, public proxy, SQLite store, rate limiter — all in one process. No microservices, no message queue, no separate workers. The daemon coordinates the per-sandbox bhatti-vmm and per-owner bhatti-netd helpers, but it owns all the state and all the decisions. State lives in SQLite (see Where state lives).

This is a deliberately small box. One binary you can scp. One log to read. One process to restart. Whatever growth bhatti does, it won't be by adding more processes to the control plane — it'll be by getting better at the work the single process already does.

The cost is real: one bhatti daemon manages the sandboxes on one machine. There's no clustering inside bhatti and no plans for it. If you need to scale across hosts, the way I'd do it (and the way I'm planning to, eventually) is to put a thin proxy in front of several bhatti daemons that records which daemon a sandbox was created on and routes subsequent requests for that sandbox — including wake-on-request — to the same daemon. Each bhatti stays single-process; the proxy is the only piece that knows about the fleet. That's still outside bhatti itself.

2. We own the VMM, and drive it out of band

The daemon never links libkrun. Linking it wouldn't work anyway: libkrun's entry call, krun_start_enter, becomes the VM and blocks for its whole life — a daemon that called it would stop being a daemon. So the VMM lives in a separate process, bhatti-vmm, and the daemon drives it from outside.

That out-of-band channel is a tiny Unix-socket control socket. The daemon writes a verb, the VMM acts, and answers:

PAUSE            -> OK paused | ERR <reason>   (freeze the vCPUs)
RESUME           -> OK running | ERR <reason>  (thaw them)
STATUS           -> OK <state>                 (running|pausing|paused|resuming)
SNAPSHOT <dir>   -> OK                         (write RAM + device + vCPU state)

Everything the thermal manager and snapshot code needs is built from these primitives. A warm pause is PAUSE; a warm resume is RESUME. A checkpoint (a memory snapshot) is PAUSE, SNAPSHOT <dir>, RESUME — captured without stopping the sandbox. A restore is a fresh bhatti-vmm launched with the bundle directory, so libkrun reloads memory.img instead of cold-booting. A fork (create --from) is a checkpoint into a throwaway bundle immediately restored into a new identity. See the engine page for the full sequence.

What I get from owning the VMM: pause, resume, snapshot, restore, and fork exist at all. They're the whole point of bhatti's thermal model, and they are permanently off upstream libkrun's roadmap — so bhatti maintains the fork (krucible) rather than wrap a stock VMM. The cost is that I carry a hypervisor fork; the engine page is the argument for why that's worth it.

3. No jailer — the VMM runs as an ordinary host process

There is no jailer in v2. bhatti-vmm runs as a normal host process under the daemon's user. Isolation comes from three places that don't require a chroot wrapper:

  • The hardware boundary. Each sandbox is a real VM behind KVM (Linux) or HVF (Apple Silicon). The guest kernel is not the host kernel; a process in the guest is not a process on the host.
  • The network gateway. bhatti-netd is a userspace gVisor netstack, so the guest never touches host network devices — no TAP, no bridge, no iptables to get wrong. Egress is policed and same-owner siblings are routed; the host is not on the guest's network.
  • The storage boundary. The VMM only opens the files for its own sandbox — a qcow2 CoW root overlay, a config drive, and any attached volumes — not the wider filesystem.

Hardening the host-process boundary further — dropping to an unprivileged UID, a seccomp filter, a private mount namespace for hostile multi-tenant on Linux — is planned as a separate Track J capability. It is future work, not the current path: today the VMM is an ordinary process, and that is the honest description of what runs. On macOS the shipped bhatti-vmm is Developer-ID signed and notarized with the com.apple.security.hypervisor entitlement, so HVF works without root at all.

4. Per-VM mutex with capture-and-release

Concurrency is the part that bites every "single Go process" design eventually. Bhatti's pattern is small but load-bearing.

Each VM struct has two mutexes (pkg/engine/krucible/engine.go): launchMu serializes lifecycle transitions (launch / start / stop / pause / resume / destroy) so a burst of concurrent wake-on-request calls can't double-spawn the helper; mu guards field reads and writes. The lock order is always launchMu before mu, never the reverse — read-only Status/List take only mu and must not block behind an in-flight transition. The engine-level sync.RWMutex only protects the map of VMs.

The pattern every operation follows:

vm.mu.Lock()                       // 1. take the lock
if vm.Status != "running" {        // 2. check invariants
    vm.mu.Unlock()
    return errNotRunning
}
ag := vm.Agent                     // 3. capture references we need
vm.mu.Unlock()                     // 4. release the lock immediately
return ag.Exec(ctx, cmd)           // 5. do the slow work outside the lock

The key move is steps 3-4: copy the references, drop the lock, then make the network call. The lock protects access to vm.Agent itself, not any operation that uses it.

The reason this matters: a Shell or Tunnel call lasts as long as the user's session — minutes, hours. If those calls held the lock for their duration, the thermal manager couldn't pause any VM (because every cycle iterates the VM map and tries to read each VM's thermal state). One slow shell would freeze idle pause / cold snapshot for everything else.

The safety argument: vm.Agent is only replaced during a launch (cold restore), and that path takes launchMu and then mu to do it. So if a Shell goroutine grabs vm.Agent, releases the lock, and then a restore runs to completion and replaces vm.Agent with a new client — the Shell goroutine still has a valid reference to the old agent's vsock connection. That connection is good until the bhatti-vmm process is killed, which happens only on Stop() or Destroy(). By that point the connection is closed, the Shell sees a read error, and cleans up.

So: long-running operations don't block the system, and they don't explode when state is updated underneath them — they just see the underlying connection close and exit cleanly. The pattern is in every method on the krucible engine that touches an agent.

Where state lives

State is in two places: SQLite, on disk; and per-sandbox VMM state (disk overlays, config drive, cold-snapshot bundle) on disk under the data directory.

SQLite. One database file at /var/lib/bhatti/state.db on a default server install. Tables:

  • sandboxes — id, name, owner, status, engine ID
  • users — name, hashed API key (SHA-256), per-user limits
  • secrets — name, encrypted value (age), scoped per user
  • templates, volumes, publish_rules
  • events — audit log
  • task_progress — async tasks (image pull)

WAL mode is on so background workers (the thermal manager, port scanner, metrics snapshotter) don't block HTTP request handlers and vice versa.

Per-sandbox. Under /var/lib/bhatti/sandboxes/<id>/:

<CoW root overlay>   qcow2 copy-on-write overlay over the base image
config drive         ~1 MB ext4 (/dev/vdb): hostname, env, secrets,
                     files, per-sandbox token
state.json           recovery metadata — helper PID, socket paths,
                     bundle ref; this is what recovery reads
(when cold)          the cold-snapshot bundle: memory image + VM
                     state + a copy of the root overlay + any
                     attached volumes + manifest.json

The root overlay is a qcow2 copy-on-write image stacked over a shared read-only base. CoW lives at the image-format layer, so it's filesystem-independent — ext4, XFS, btrfs, APFS all give instant create and base-sharing with ~0.5% overhead. There's no btrfs or reflink requirement; that was v1. See Storage for the overhead breakdown and the page-cache story for cold wake.

Other disk locations under /var/lib/bhatti/:

  • runtime/ — the relocatable v2 runtime bundle, re-laid on every install/update: bin/ (bhatti-vmm, bhatti-netd), lib/ (libkrun + symlinks), and kernel/ (the lean external kernel)
  • images/ — read-only base rootfs templates (rootfs-minimal-arm64.ext4 etc.), CoW-overlaid per sandbox
  • volumes/<user_id>/ — standalone persistent volumes (created by bhatti volume create, attachable to any sandbox as /dev/vdc+)
  • snapshots/<user_id>/<name>/ — named snapshots from bhatti snapshot create (memory + disk artifacts, fully self-contained)
  • age.key — the X25519 private key used to encrypt user secrets. It's generated the first time anyone calls bhatti secret set, not on daemon startup (pkg/secrets/age.go:13-37). Back this up. If you lose it, every encrypted secret on the server is unrecoverable.

There are no jails/ and no firecracker.sock under a sandbox dir — those were v1. The config drive is 1 MB because ext4's minimum filesystem size is 1 MB; the contents (a few hundred bytes of JSON describing hostname, env vars, volume mounts, plus the token) would fit in less, but the filesystem can't.

Recovery on startup

When the bhatti daemon starts — fresh boot, after a crash, after a systemctl restart — the helper processes from the previous run may or may not still be alive. The daemon's job at startup is to look at every sandbox and decide, for each one, what state it's really in.

Recovery is driven by the per-sandbox state.json, not the database. Each sandbox's file records everything needed to re-adopt or re-launch it: the helper PID, the socket paths, the base spec, the bundle dir, and (for the net backend) the owner's bhatti-netd key and subnet index. bhatti-vmm and bhatti-netd are spawned detached from the daemon's process group, so they survive a daemon restart or crash.

On startup the engine globs every sandboxes/*/state.json and, for each one:

  1. Adopts a live helper. If the recorded PID is still alive and its agent answers a probe over vsock, the sandbox is adopted as-is — running, no restart, sessions intact.
  2. Marks a dead-but-restorable one cold. If the helper is gone but a cold-snapshot bundle exists on disk, the sandbox is marked stopped/cold — the next request cold-restores it from the bundle (a sub-second operation).
  3. Marks the rest stopped. A dead helper with no bundle (crashed before its first snapshot, RAM gone but the qcow2 root persists) needs an explicit Start, which cold-boots it fresh.

Then the engine re-adopts each owner's shared bhatti-netd gateway from its persisted record rather than respawning onto a socket it still holds, and writes the reconciled status back to state.json.

There is no system-wide device cleanup step, because there are no host TAP devices, bridges, or iptables rules to leave behind — the network lives entirely inside bhatti-netd. That whole class of "stale plumbing after a crash" problem doesn't exist in v2.

What's not on this page

This was the map. The depth is in the topic pages: