Skip to content
[bhatti]

The krucible engine: control socket, block-root boot, snapshots

This page is the lower-level companion to Architecture. It walks through what the krucible engine actually does to boot a VM, how the daemon drives the VMM from outside, what the storage substrate looks like, and why bhatti maintains its own hypervisor fork.

If you're using bhatti, you don't need this page. If you're hacking on the engine, extending it, or debugging a weird VMM-level failure, this is the reference.

krucible is bhatti's fork of libkrun — an in-process VMM library. bhatti extended it with pause/resume, snapshot/restore, a control socket, block-root boot, and a lean external kernel. It runs on KVM (Linux, /dev/kvm) and HVF (macOS, Apple Silicon).

Design decisions on this page

  1. The daemon never links the VMM. libkrun's entry call becomes the VM and blocks forever, so bhatti puts it in a separate process (bhatti-vmm) and drives it out-of-band over a control socket. See The control socket.
  2. We own the fork. Pause, resume, snapshot, restore, and fork are permanently off upstream libkrun's roadmap — and they're the whole point of bhatti's thermal model — so bhatti maintains krucible rather than wrap a stock VMM. See Why bhatti owns the VMM.
  3. Block-root, qcow2 copy-on-write. The guest boots from a qcow2 CoW overlay over a shared base image, so create is instant and filesystem-independent. See The block-root substrate.
  4. All snapshots are Full. There are no diff or dirty-page snapshots in krucible — a checkpoint always writes the whole memory image. The full reasoning is in Thermal states.

The control socket

The daemon and the VMM are two processes. bhatti-vmm links libkrun and calls krun_start_enter, which becomes the guest and only returns on a boot error — so the daemon can't call into it. Instead the daemon drives the running VM from outside over a tiny Unix-socket control socket (krun_set_control_socket in the fork).

The protocol is line-oriented text. From pkg/engine/krucible/control.go:

PAUSE            -> OK paused | ERR <reason>
RESUME           -> OK running | ERR <reason>
STATUS           -> OK <state>            (running|pausing|paused|resuming)
SNAPSHOT <dir>   -> OK                     (write RAM + device + vCPU state to <dir>)

Four verbs. Everything the thermal manager and the snapshot code needs is composed from them:

  • Warm pause / resume (Thermal states) is a bare PAUSE / RESUME — the vCPUs freeze, memory stays resident, resume is ~4 ms.
  • Checkpoint (a memory snapshot) is PAUSE, SNAPSHOT <dir>, RESUME — captured without stopping the sandbox. libkrun writes manifest.json, checkpoint.bin, and memory.img into <dir>.
  • Restore is not a control-socket verb at all: it's a fresh bhatti-vmm launched with krun_set_snapshot(<dir>), so libkrun reloads memory.img instead of cold-booting.
  • Fork (create --from) is a checkpoint into a throwaway bundle, immediately restored into a new sandbox identity, then the bundle is discarded. Instant, and the source is undisturbed.

The lock discipline around these transitions — launchMu before mu, capture-and-release for long-running agent calls — is on the Architecture page.

Boot sequence: what the VMM does

A bhatti create spawns bhatti-vmm with a VMSpec, and the helper walks a fixed sequence of krun_* calls before it becomes the VM. The order matters — disks must be added so they enumerate in the right order, devices must precede the port bridges. From cmd/vmm/main.go:

1.  krun_create_ctx()                     new VM context
2.  krun_set_vm_config(vcpus, mem_mib)     CPU + RAM

3.  krun_set_kernel(lean kernel, cmdline)  load our own kernel;
      cmdline carries root=/dev/vda init=/usr/local/bin/lohar
      (this makes krun_start_enter skip libkrunfw entirely)

4.  krun_set_root_disk2(root, QCOW2|RAW)   root overlay -> /dev/vda
5.  krun_add_disk("config", drive, rw)     config drive -> /dev/vdb
6.  krun_add_virtiofs3(...)   (optional)   live --mount host-dir binds
7.  krun_add_disk2(vol, fmt, ro)  × N      volumes -> /dev/vdc+

8.  krun_add_virtio_console_default(...)   console -> the daemon's log
9.  krun_add_vsock(features)               plain vsock (net backend on)
                                           or TSI-inet hijack (net off)
10. krun_add_net_unixstream(uds, mac, ...) virtio-net -> bhatti-netd
                                           (only with the net backend)
11. krun_add_vsock_port2(1024, ctrl-uds)   agent control plane bridge
    krun_add_vsock_port2(1025, fwd-uds)    agent forward plane bridge
12. krun_set_control_socket(ctrl-uds)      the PAUSE/RESUME/etc. socket
13. krun_set_snapshot(bundle)  (optional)  cold-restore instead of boot

14. krun_start_enter()                     THIS PROCESS BECOMES THE VM

A few of these are easy to miss but load-bearing:

  • Step 3 is the whole "lean kernel" story (below). Setting an external kernel makes libkrun skip its bundled libkrunfw; the cmdline supplies init=, so lohar boots as PID 1 directly. On the bundled-kernel path instead, krun_disable_implicit_init is called so libkrun doesn't inject its own /init.krun ahead of lohar.
  • Step 9's vsock is dual-purpose. With the network gateway on (the default), the guest's real networking goes over eth0 via bhatti-netd, so the vsock device is added plain and used only for the agent port bridges (step 11). With the legacy TSI backend, the same vsock carries KRUN_TSI_HIJACK_INET so the guest borrows the host netstack — see Networking.
  • Step 11 is how the daemon reaches lohar. The host dials the bridge UDS and libkrun forwards it to the guest port where lohar listens: 1024 for the control plane (exec, files, sessions), 1025 for the forward plane (port forwarding). This is plain vsock, not TCP over a host network device.

After krun_start_enter, the daemon polls the agent control bridge until lohar answers, then the create returns to the user. There is no ARP priming and no host-side IP plumbing — the guest reaches the network through bhatti-netd, and the daemon reaches the guest through vsock.

The block-root substrate

krucible boots from a block device, not a shared host directory. Each sandbox's root is a qcow2 copy-on-write overlay stacked over a single read-only base image (krun_set_root_disk2 with the qcow2 format; the overlay is provisioned by krun_create_disk_overlay, imago-backed, in the fork). The guest sees an ordinary ext4 at /dev/vda; libkrun translates qcow2 host-side.

Because CoW lives at the image-format layer, it's filesystem-independent — ext4, XFS, btrfs, APFS all give instant create and base-sharing, with roughly 0.5% overhead on agent workloads. There is no btrfs or reflink requirement; that was a v1 constraint. Setting KRUCIBLE_ROOT_RAW=1 opts a sandbox out of qcow2 onto a raw image (losing the instant-clone property on non-CoW filesystems).

The rest of the device set stacks on top in a fixed order:

  • /dev/vda — the qcow2 root overlay
  • /dev/vdb — the ~1 MB config drive (a raw ext4 lohar mounts read-only to read hostname, env, secrets, attached-volume specs, and the per-sandbox token)
  • /dev/vdc+ — attached block volumes, in attach order

A memory checkpoint or fork restores the whole device set, not just RAM: Checkpoint freezes each attached volume as an independent copy consistent with the paused VM and records it in manifest.json; restore clones each back and re-attaches it, so the restored RAM's view of its disks stays valid. See Storage for the format details.

The lean external kernel

krucible boots its own kernel instead of the one bundled inside libkrunfw. krun_set_kernel (step 3 above) hands libkrun an external Image (arm64) or vmlinux (x86_64) plus a full boot cmdline, and libkrun skips libkrunfw entirely.

The payoff is speed. It's the same 6.12.x kernel libkrunfw ships, but built with a lean config (roughly 1,000 options versus ~1,400) — just what a microVM guest needs. The result is about 2× faster cold-start: boot→agent measured at ~312 ms versus ~610 ms on HVF, validated cross-arch on HVF and on KVM (arm64 + x86_64). The kernel is ours — pinned, reproducible, built from the config under scripts/lean-kernel/ via scripts/build-lean-kernel.sh (see Contributing → the kernel for the build).

The lean kernel is opt-in-by-presence: point krucible_kernel_image at it (the install lays it under runtime/kernel/), or leave it unset and the daemon falls back to the bundled libkrunfw kernel. The external kernel is what enables the block-root path — it's the one that reads root=/dev/vda init=/usr/local/bin/lohar from the cmdline.

Why bhatti owns the VMM

bhatti forks libkrun rather than depending on it, and that's a deliberate, load-bearing choice.

The entire thermal model — warm pause/resume, cold snapshot/restore, and fork — depends on the VMM being able to freeze a running guest, serialize its RAM + device + vCPU state to disk, and reconstruct it later (in the same process, or a fresh one, or a new identity). Stock libkrun does none of that, and those capabilities are permanently off its upstream roadmap — it's designed to launch a workload and run it, not to checkpoint and clone live VMs. There is no config flag that turns them on.

So the capabilities live in the fork:

  • krun_set_control_socket — the PAUSE/RESUME/STATUS/SNAPSHOT channel the daemon drives from outside.
  • Snapshot / restore — SNAPSHOT <dir> and krun_set_snapshot(<dir>), including the GIC save/restore on arm64 KVM that makes cold-restore work there.
  • Warm-clock freeze — a paused guest's CLOCK_MONOTONIC must not jump forward by the pause duration on resume; the fork rewinds the guest-visible virtual counter.
  • Block-root + qcow2 — krun_set_root_disk2 and krun_create_disk_overlay, the CoW substrate above.
  • The external-kernel path — krun_set_kernel skipping libkrunfw.

The cost is real: bhatti carries a hypervisor fork (libkrucible, vendored as a git submodule; the gitlink SHA is the version-of-record), rebased onto upstream libkrun on its own cadence. The upside is that pause, resume, snapshot, restore, and fork exist at all — and they're not bolted on, they're first-class in the VMM. For an agent-sandbox platform whose whole economics rest on sub-second cold wake, that trade is the point.

Listening port discovery

When a request hits a sandbox proxy URL, the daemon needs to know which ports the guest has open. Rather than maintain a host-side registry that gets stale, bhatti just asks the guest over the agent:

// pkg/engine/krucible/agent.go
result, _ := ag.Exec(ctx, []string{"ss", "-tln", "--no-header"}, nil, "")
return parseSSOutput(result.Stdout), nil

This is on-demand: we don't poll. The proxy queries listening ports when a request arrives, and the cost is one exec (a few milliseconds) per query. Guest-side ss is fast and accurate — it queries the kernel directly rather than walking /proc/net/tcp.

The trade-off: this only works on hot VMs. The ListeningPorts call errors out for warm or cold sandboxes. The public proxy handles this by ensureHot-ing the sandbox first — if you publish a port and a request hits the cold URL, the proxy wakes the VM (a sub-second cold-restore), queries ports, checks that the requested port is in the list, then forwards.

Recovery: what gets persisted

The VMM doesn't survive a daemon restart on its own — a crashed or restarted daemon inherits helper processes that may or may not still be alive. krucible persists everything needed to re-adopt or re-launch each sandbox to a per-sandbox state.json, and the helpers are spawned detached from the daemon's process group so they outlive it.

The record captured per VM (pkg/engine/krucible/recovery.go):

  • id, name, user_id, status, thermal
  • sandbox_dir, sock_dir, bundle_dir (the cold-snapshot bundle), log_path
  • base_spec — the full spec to re-launch or cold-restore with
  • helper_pid — the bhatti-vmm PID, for adopt-by-PID
  • netd_key, subnet_idx — the owner's shared bhatti-netd gateway, so recovery re-adopts the running gateway instead of respawning onto a socket it still holds

On daemon startup the engine globs every sandboxes/*/state.json and, for each, decides:

  • Adopt if the recorded PID is still alive and its agent answers a vsock probe — the sandbox is running, no restart.
  • Cold if the helper is dead but a cold-snapshot bundle exists — marked stopped, ready to cold-restore from the bundle on the next request.
  • Stopped otherwise — a dead helper with no bundle (RAM gone, but the qcow2 root persists) needs an explicit Start to cold-boot fresh.

There's no system-wide device cleanup afterward, because there are no host TAP devices, bridges, or iptables rules to leave behind — the network lives entirely inside bhatti-netd.

Cold-snapshot bundle layout

A cold sandbox — or a named snapshot from bhatti snapshot create — has a self-contained bundle on disk. A memory snapshot writes:

<bundle-dir>/
├── manifest.json      arch/feature gate, config-drive + token, volumes, mounts
├── checkpoint.bin      VM + device + vCPU state (small)
├── memory.img          full guest RAM image (size of the VM's allocated RAM)
├── <root overlay>      the qcow2 root at snapshot time
└── vol0.img, vol1.img  frozen copies of attached volumes, if any

memory.img is the size of the VM's allocated RAM. A 4 GB VM that's been cold for an hour has a 4 GB memory.img on disk. checkpoint.bin is small — CPU registers, vCPU state, and device-model state.

manifest.json is what makes a snapshot portable and safe. It records the host architecture and feature set, so restore refuses a cross-arch bundle cleanly rather than crashing; and it lists the config drive, the per-sandbox token to re-enforce from RAM, and every attached volume, so a restore into a fresh sandbox — even after the original is destroyed — reconstructs the full device set. A filesystem-type snapshot is disk-only (no memory.img); it restores with a cold boot rather than a memory reload.

Where to go next

  • Architecture — the bird's-eye view that this page sits underneath
  • Thermal states — what the engine does for Stop/Start/Pause/Resume, and why all snapshots are Full
  • Networking — the bhatti-netd gVisor gateway, the vsock forward primitive, and TSI
  • Storage — the qcow2 CoW substrate, volumes, and the cost-by-filesystem story
  • Decisions & learnings — the own-the-fork choice and other engine-level lessons