Skip to content

The krucible engine: control socket, block-root boot, snapshots

bhatti runs one bhatti-vmm helper per sandbox. The Go daemon doesn't link the VMM: the helper loads libkrun, boots a Linux guest on KVM (Linux) or HVF (Apple Silicon), and accepts lifecycle commands over a local Unix control socket. That boundary lets the daemon manage many of its own agents without running a host TAP bridge or giving each sandbox a host-facing API port. The guest agent, lohar, uses a separate vsock bridge.

Control and boot

The helper's control loop accepts PAUSE, RESUME, STATUS, and SAVE <dir>. PAUSE stops the vCPUs while retaining memory in the helper. RESUME releases them. SAVE writes a VM checkpoint and leaves the helper paused; the engine handles the disk and attached-volume copies and then resumes the source after a named memory snapshot. This is the VMM-side mechanism behind explicit memory checkpoints, not a promise that bhatti stop or thermal cold keeps memory: both power off, retaining disks but discarding processes and /tmp.

The VMM reads a VMSpec, configures CPU/RAM and an external guest kernel, attaches the qcow2 root overlay as /dev/vda, and attaches optional persistent volumes as subsequent virtio-blk devices. The guest's boot configuration travels over vsock port 1026 rather than an always-present config.ext4 drive. Lohar listens on vsock 1024 for control (exec, files, sessions) and 1025 for forwarding; host Unix bridges carry those requests without guest TCP. See cmd/vmm/main.go and Lohar.

A guest NIC is optional. A fresh sandbox normally has no network device. If a network policy is enabled, the VMM authenticates a Unix-stream connection to its owner's bhatti-netd and adds eth0; netd is not a mandatory per-VM helper. The guest still reaches the host daemon over vsock if it has no NIC. Networking covers public egress, narrower allow rules, and separately opted-in sibling access.

On Linux, cgroups: auto places helpers in systemd scopes where the host supports it: bhatti.slice contains a per-owner slice, with one scope per VM and a sibling scope for that owner's netd. New helpers wait behind a launch gate until placement is verified. This enables CPU, memory and I/O accounting, not resource limits. See Host resource placement for names, CPU weights and recovery caveats; macOS uses per-process accounting instead.

Overlay, snapshot, fork

A sandbox's root is a qcow2 copy-on-write overlay over an immutable, content-addressed tier base. It does not require btrfs. Storage describes the base layout and disk cost. Persistent volumes are separate block images, not files inside that root overlay.

A default named memory snapshot records RAM/CPU/device state, the root overlay, attached-volume copies and network policy. The source VM keeps going. Resuming creates a new VM with private writable layers over the saved disks, including restored volumes; it does not attach the source sandbox's live volumes. snapshot create --type filesystem captures only the root disk and cold-boots a fresh sandbox without attached volumes or the source's network policy; preserve volume data separately. Neither type is an incremental memory diff. See Storage for disk-sharing costs and snapshot deletion dependencies.

bhatti create --from <sandbox> checkpoints a live sandbox, then normally layers the child over that frozen disk state, with private writable root and volume layers. To branch a named snapshot, use bhatti snapshot resume <snapshot> --name <new>. A live fork inherits the source network policy; the CLI refuses --net, --egress, --allow-host, --allow-cidr and --allow-siblings overrides, and the server rejects --mount and --secret-grant on a fork. See Fork & branch.

Snapshots can leave this host as a tar + zstd archive through bhatti snapshot export. Without --include-base, an importing host needs the same base content; include it if necessary. Import always checks architecture and base availability. A memory snapshot additionally needs the same host OS and compatible CPU features, so Mac↔Linux memory checkpoints cannot move; a root-disk-only filesystem snapshot on the same architecture can cross OS boundaries and cold-boot. Export refuses snapshots containing host-directory --mount binds. Server-side secret records and grant authorizations are not exported. Credentials already present in captured guest RAM or files remain included, including plaintext injected with --secret; treat archives as sensitive. This is checkpoint transfer, not live migration — Move sandboxes between hosts shows the workflow.

Recovery and durability

Per-sandbox disk state and recovery metadata survive a daemon restart; still-running v2.4+ helpers can be adopted when their process and agent remain reachable. After host power loss, disk can be recovered but guest memory cannot: don't mistake daemon adoption for persistence of a running process across a reboot. The v2.5.1 sync-before-pause/stop change, sandbox.unclean_stop event and opt-in exec --sync make write boundaries visible. Read Thermal states and Data & durability before relying on a stop as a save point.