Skip to content

Architecture overview

This page is the map. It exists so that when you go into Lohar, Thermal states, Networking, or Decisions, you already have a sense of where each fits.

If you're using bhatti, you don't need this page. If you're evaluating it for production or thinking about contributing, this is the right starting point.

The shape

bhatti has one Go host daemon, one bhatti-vmm helper per running sandbox, and lohar inside each VM. The daemon never links the VMM; it controls it out of band over a Unix socket and talks to lohar over vsock. Only owners running networked sandboxes also get a bhatti-netd gateway; new sandboxes have no NIC by default.

bhatti is a single Go binary that does many jobs depending on how you invoke it:

  • bhatti serve — the daemon (HTTP API, krucible engine, thermal manager, public proxy, all in one process)
  • bhatti create, bhatti exec, bhatti shell, ... — the CLI client, talking to the daemon's HTTP API
  • bhatti user create, bhatti admin status — admin commands that read the daemon's SQLite database directly (root only)

Two more binaries ship in the release bundle and are launched by the daemon, never by you:

  • bhatti-vmm — the cgo helper that is the VM. It links krucible (bhatti's fork of libkrun) and calls krun_start_enter, at which point the process becomes the guest and blocks for the life of the VM. The daemon controls it from outside over a Unix-socket control channel.
  • bhatti-netd — a per-owner userspace gVisor gateway for networked sandboxes. It enforces each sandbox's outbound policy and optional secret grants. Sibling traffic is denied unless the sending sandbox opts in. No network device means no netd for that sandbox.

lohar runs as PID 1 inside every microVM. It handles exec, file operations, PTY sessions, and pretends to be systemctl and journalctl for in-guest callers. The full story is in Lohar.

┌─ Host ──────────────────────────────────────────────────────────────────────┐
│                                                                             │
│  ┌─ bhatti daemon  (bhatti serve, single pure-Go process) ───────────────┐  │
│  │                                                                       │  │
│  │   REST / WS API    Engine (krucible)     Store (SQLite, WAL mode)     │  │
│  │   api.sock          create / destroy     sandboxes, users, secrets,  │  │
│  │   public proxy*     stop / start         templates, volumes, events  │  │
│  │     thermal manager  pause / resume                                   │  │
│  │                      snapshot / fork                                  │  │
│  │                      exec, shell, files                               │  │
│  │                                                                       │  │
│  └───────────┬───────────────────────────────────┬───────────────────────┘  │
│              │                                   │                          │
│     control socket (UDS)            unix socket (only with NIC)           │
│     PAUSE/RESUME/STATUS/SAVE                  virtio-net                  │
│              │                                   │                          │
│              ▼                                   ▼                          │
│  ┌─ bhatti-vmm × N  (krucible/libkrun VMM) ──┐   ┌─ bhatti-netd × owners ─┐  │
│  │                                          │   │  optional per-owner    │  │
│  │   KVM (Linux) / HVF (macOS)              │◄──┤  gVisor gateway        │  │
│  │   qcow2 CoW root (/dev/vda)              │   │  per-sandbox policy    │  │
│  │   boot config via vsock   volumes …      │   │  grants; siblings opt-in│ │
│  │                                          │   └────────────────────────┘  │
│  │   ┌─ lohar (PID 1) ─────────────────┐    │                               │
│  │   │   vsock  ── control plane       │    │                               │
│  │   │           (exec, files, PTY)    │    │                               │
│  │   │   vsock  ── forward plane       │    │                               │
│  │   │   sessions   scrollback         │    │                               │
│  │   │   systemctl shim   journald     │    │                               │
│  │   └─────────────────────────────────┘    │                               │
│  └───────────────────────────────────────────┘                             │
└─────────────────────────────────────────────────────────────────────────────┘

The API defaults to an authenticated Unix socket at <data_dir>/api.sock (the installer gives the bhatti group access). Outside domain mode, TCP is opt-in via an explicit server listen: and is plaintext — do not expose it to an untrusted network. For remote clients, configure a TLS domain or SSH to the host. Public HTTPS aliases need domain/TLS configuration; without domain mode, public_proxy_listen can instead serve a separate unauthenticated path-based HTTP proxy for development/testing. Neither is the bearer-authenticated control API proxy. Domain mode ignores listen and public_proxy_listen. The VMM control socket, agent vsock and netd gateway are internal.

Host resource placement

On Linux with systemd placement enabled, the daemon and its helpers no longer share one service cgroup. The helpers live in a separate hierarchy:

bhatti.slice
└── bhatti-<owner>.slice
    ├── bhatti-<owner>-<engineID>.scope   bhatti-vmm, one per sandbox
    └── bhatti-<owner>-netd.scope        bhatti-netd, when networking is used

<owner> is a stable hash of the owner key, not a display name; <engineID> identifies the sandbox VM. CPU, memory and I/O accounting are enabled on the slices and scopes. This is placement and accounting, not resource limits: no memory ceiling, CPU quota or I/O cap is set. The top slice's CPU weight is 100; each gateway weighs 1000 against a VM's default 100 under contention. Weights affect relative CPU sharing, not how much CPU a helper may use when the host is idle.

A new helper waits behind a launch gate until the daemon verifies its scope membership and saves the placement. If placement fails on a host using systemd mode, the launch fails instead of running unaccounted. Recovery is different: an older, unplaced helper can be moved into a scope, but its earlier usage remains charged elsewhere (partial accounting). If a recovered guest cannot be placed, it is kept running and logged as unaccounted, not killed.

The cgroups setting defaults to auto: use systemd where available, otherwise report none with a reason. It requires systemd as Linux PID 1, unified cgroup v2 and permission to create units over the system bus. macOS uses per-process accounting with auto, not this hierarchy. /health reports the effective mode.

Four decisions that shape the rest

These aren't all the choices in the codebase, but they're the ones you can't understand the rest of the docs without.

1. The daemon is a single Go process

REST/WS API, krucible engine, thermal manager, public proxy, SQLite store, rate limiter — all in one process. No microservices, no message queue, no separate workers. The daemon coordinates the per-sandbox bhatti-vmm and optional per-owner bhatti-netd helpers, but it owns all the state and all the decisions. State lives in SQLite (see Where state lives).

This is deliberately one host and one daemon for the control plane: no message queue or orchestrator is required to manage your own sandboxes on a box.

2. We own the VMM, and drive it out of band

The daemon never links libkrun. Linking it wouldn't work anyway: libkrun's entry call, krun_start_enter, becomes the VM and blocks for its whole life — a daemon that called it would stop being a daemon. So the VMM lives in a separate process, bhatti-vmm, and the daemon drives it from outside.

That out-of-band channel is a tiny Unix-socket control socket. The daemon writes a verb, the VMM acts, and answers:

PAUSE            -> OK | ERR <reason>         (freeze the vCPUs)
RESUME           -> OK | ERR <reason>         (thaw them)
STATUS           -> OK running | OK paused
SAVE <dir>       -> OK | ERR <reason>         (save state; leave paused)

The warm thermal transition uses PAUSE/RESUME, keeping memory in host RAM. Cold thermal transition and explicit stop power off instead: the disk is retained, not running processes or /tmp; start boots from disk and reruns --init. An explicit memory snapshot uses SAVE <dir> on a running VM to save RAM, devices and vCPU state; <dir> must be a new absolute directory. The daemon copies disks and volumes while the VM remains paused, then sends RESUME — no separate PAUSE before SAVE. snapshot resume loads a snapshot into a new VM identity, while create --from <sandbox> forks an existing sandbox's memory/disk state. See the engine page for the mechanisms.

What I get from owning the VMM: pause, resume, snapshot, restore, and fork exist at all. Upstream libkrun does not ship this set of VM lifecycle primitives, so bhatti maintains the fork (krucible) rather than wrapping the stock VMM. The cost is maintaining that hypervisor fork.

3. No jailer — confine the host helpers

There is no Firecracker-style jailer in v2. Every sandbox has a real VM boundary — KVM on Linux, HVF on Apple Silicon — with its own guest kernel. On Linux, v2.4 added no_new_privs and Landlock confinement to bhatti-vmm; v2.5 added a seccomp deny-list. A root daemon assigns each sandbox's helper its own unprivileged UID; with a non-root daemon, helpers keep the daemon's user. Root-launched helpers drop all capabilities unless virtio-fs mounts require retained filesystem capabilities. Landlock depends on kernel support: without it, mount-free helpers warn and run without filesystem confinement, while virtio-fs mounts are refused. These Linux controls are not enabled on macOS. The optional Linux bhatti-netd also drops root when started as root and uses Landlock when available. These controls limit what a compromised helper can access; they are not a promise of hostile multi-tenant isolation.

A no-NIC guest has no guest-network path to the host. When networking is enabled, netd authenticates each VM link and enforces its egress policy. Host addresses are denied except for configured daemon TCP service listener address/port pairs when the remaining policy permits them; private daemon-service addresses need an explicit CIDR allowance, and API authentication still applies. Other host ports, gateway and other-owner addresses stay blocked; same-owner siblings need --allow-siblings. No host TAP, bridge or iptables rule is involved. Host mounts are separately limited by mount_roots. See Networking and Network & egress.

4. Per-VM mutex with capture-and-release

Concurrency is the part that bites every "single Go process" design eventually. bhatti's pattern is small but load-bearing.

Each VM struct has two mutexes (pkg/engine/krucible/engine.go): launchMu serializes lifecycle transitions (launch / start / stop / pause / resume / destroy) so a burst of concurrent wake-on-request calls can't double-spawn the helper; mu guards field reads and writes. The lock order is always launchMu before mu, never the reverse — read-only Status/List take only mu and must not block behind an in-flight transition. The engine-level sync.RWMutex only protects the map of VMs.

The pattern every operation follows:

vm.mu.Lock()                       // 1. take the lock
if vm.Status != "running" {        // 2. check invariants
    vm.mu.Unlock()
    return errNotRunning
}
ag := vm.Agent                     // 3. capture references we need
vm.mu.Unlock()                     // 4. release the lock immediately
return ag.Exec(ctx, cmd)           // 5. do the slow work outside the lock

The key move is steps 3-4: copy the references, drop the lock, then make the network call. The lock protects access to vm.Agent itself, not any operation that uses it.

The reason this matters: a Shell or Tunnel call lasts as long as the user's session — minutes, hours. If those calls held the lock for their duration, the thermal manager couldn't pause any VM (because every cycle iterates the VM map and tries to read each VM's thermal state). One slow shell would freeze idle pause / cold snapshot for everything else.

The safety argument: vm.Agent is only replaced during a launch (cold restore), and that path takes launchMu and then mu to do it. So if a Shell goroutine grabs vm.Agent, releases the lock, and then a restore runs to completion and replaces vm.Agent with a new client — the Shell goroutine still has a valid reference to the old agent's vsock connection. That connection is good until the bhatti-vmm process is killed, which happens only on Stop() or Destroy(). By that point the connection is closed, the Shell sees a read error, and cleans up.

So: long-running operations don't block the system, and they don't explode when state is updated underneath them — they just see the underlying connection close and exit cleanly. The pattern is in every method on the krucible engine that touches an agent.

Where state lives

State lives in SQLite and per-sandbox on-disk VMM state (disk overlays, recovery metadata and checkpoint bundles) under the data directory.

SQLite. One database file at /var/lib/bhatti/state.db on a default server install. Tables:

  • sandboxes — id, name, owner, status, engine ID
  • users — name, hashed API key (SHA-256), per-user limits
  • secrets — name, encrypted value (age), scoped per user
  • templates, volumes, publish_rules
  • events — audit log
  • task_progress — async tasks (image pull)

WAL mode is on so background workers (the thermal manager, port scanner, metrics snapshotter) don't block HTTP request handlers and vice versa.

Per-sandbox. Under the data directory's sandboxes/<id>/, the qcow2 copy-on-write root overlay and attached volumes preserve files; the guest's boot configuration comes over vsock, not a persistent config drive. Recovery metadata (state.json) records the helper PID and sockets. Warm sandboxes keep VM state in host RAM; thermal cold and explicit stop instead keep the disks while powering off. A named memory snapshot separately preserves memory, disks and copies of attached volumes.

The root overlay is a qcow2 copy-on-write image stacked over a shared read-only base. CoW lives at the image-format layer rather than requiring btrfs or host reflinks; this is how the same VM model works on Linux filesystems and APFS. v1 needed host filesystem cloning. See Storage for the disk and page-cache story.

Other disk locations under /var/lib/bhatti/:

  • runtime/ — the relocatable v2 runtime bundle, re-laid on every install/update: bin/ (bhatti-vmm, bhatti-netd), lib/ (libkrun + symlinks), and kernel/ (the lean external kernel)
  • images/ — read-only base rootfs templates (rootfs-minimal-arm64.ext4 etc.), CoW-overlaid per sandbox
  • volumes/<user_id>/ — standalone persistent volumes (created by bhatti volume create, attachable to any sandbox as /dev/vdc+)
  • snapshots/<user_id>/<name>/ — named snapshot artifacts; memory snapshots include RAM/device state, root-disk and volume copies, and the source network policy. --type filesystem captures only the root disk, not attached volumes or network policy, and restores with a cold boot. snapshot export packages artifacts for another compatible host; use --include-base when the destination lacks the required base image. See Moving between hosts.
  • age.key — the X25519 private key for encrypted stored secrets and sandbox CA keys. Created on the first encryption operation, not necessarily at daemon startup. Back this up with the database; without it encrypted material is unrecoverable.

There are no jails/, firecracker.sock, or persistent boot-config drive under a v2 sandbox directory — those belonged to the older layout. Boot settings are sent over vsock rather than being placed in a snapshot-exportable disk.

Recovery on startup

When the daemon starts — fresh boot, after a crash or after a service restart (systemd on Linux, launchd on macOS) — helper processes from the previous run may still be alive. Recovery inspects their state and re-adopts running helpers when possible.

Recovery is driven by the per-sandbox state.json, not the database. Each sandbox's file records everything needed to re-adopt or re-launch it: the helper PID, the socket paths, the base spec, the bundle dir, and (for the net backend) the owner's bhatti-netd key and subnet index. bhatti-vmm and bhatti-netd are spawned detached from the daemon's process group, so they survive a daemon restart or crash.

On startup the engine globs every sandboxes/*/state.json and, for each one:

  1. Adopts a live helper. If the recorded PID is still alive and its agent answers a probe over vsock, the sandbox is adopted as-is — running, no restart, sessions intact.
  2. Recovers a dead helper from retained state. The persisted sandbox disk remains, but RAM and running processes are gone unless an explicit compatible checkpoint bundle exists to restore. A powered-off sandbox starts by booting from disk; its --init command runs again.

The engine re-adopts a surviving bhatti-netd only for owners whose networked sandboxes require one. A no-NIC sandbox has no gateway to reconnect. Recovery writes reconciled status back to state.json.

There is no system-wide device cleanup step, because there are no host TAP devices, bridges, or iptables rules to leave behind — the network lives entirely inside bhatti-netd. That whole class of "stale plumbing after a crash" problem doesn't exist in v2.

What's not on this page

This was the map. The depth is in the topic pages: