Storage

bhatti's storage layout is one of the most load-bearing design decisions in the system and the least visible. Every bhatti create, every snapshot, every resume from cold leans on it. In v1 those assumptions lived in the filesystem — reflinks on btrfs, a real copy on ext4 — and the cost of a create swung by an order of magnitude depending on which one you had.

v2 moves copy-on-write down a layer, into the disk-image format itself. Each sandbox's root is a qcow2 overlay over a shared read-only base image. Because the CoW happens inside qcow2, not in the filesystem, it works the same on ext4, xfs, btrfs, and APFS — no btrfs requirement, no reflink, no cost cliff. This page covers what's actually on disk per sandbox, how the overlay stays cheap, and the one escape hatch (KRUCIBLE_ROOT_RAW) for when you don't want it.

What's on disk per sandbox

Under /var/lib/bhatti/sandboxes/<id>/:

root.qcow2          qcow2 CoW overlay over the shared base image (the sandbox's disk deltas)
config.ext4         ~1 MiB ext4 with hostname, env, secrets, file drops, per-sandbox token
vmspec.json         the VM spec the helper booted from
state.json          recovery metadata (helper PID, socket paths, bundle ref)
vmm.log             the bhatti-vmm helper's log
bundle/             the cold snapshot bundle (only while the sandbox is cold)

The cold bundle/ directory holds a self-contained snapshot:

memory.img          guest RAM image (equal to configured memory)
                    + VM/device + vCPU state
rootfs.qcow2        point-in-time copy of the root overlay
config.ext4         the config drive
<volume images>     any attached persistent volumes
manifest.json       bundle metadata (vCPUs, memory, arch/feature gate, proto_ver)

Plus shared resources elsewhere:

/var/lib/bhatti/images/rootfs-<tier>-<arch>.ext4 — read-only base rootfs templates. Every sandbox's root.qcow2 is a thin overlay over one of these; the base is never modified.
/var/lib/bhatti/volumes/<user_id>/ — standalone volumes (bhatti volume create), attached to one sandbox at a time.

From the guest's perspective these are all virtio-blk block devices. The root overlay is /dev/vda, the config drive is /dev/vdb, and attached volumes enumerate from /dev/vdc upward. On the host, root.qcow2's on-disk size is just the delta the sandbox has written — ls -lh shows the logical device size, du -h shows the real allocation. The gap is the whole point; see Disk usage on the images page for the walkthrough.

How the CoW root overlay works

bhatti create provisions the overlay through the VMM helper — the daemon is pure Go and never links libkrun, so it shells to bhatti-vmm create-overlay, which creates a qcow2 CoW node via krun_create_disk_overlay (libkrun's imago disk library — the same code that opens the image at boot) (pkg/engine/krucible/engine.go, cmd/vmm/main.go):

// Default: a qcow2 CoW root — instant + host-FS-independent (no
// reflink/btrfs). Raw is the opt-out (KRUCIBLE_ROOT_RAW=1).
rootImg = filepath.Join(sandboxDir, "root.qcow2")
e.createRootOverlayQcow2(rootImg, base)
baseSpec.RootDiskFormat = "qcow2"

A fresh overlay is metadata-only — a tiny qcow2 v3 header that records the backing image, never a copy of it. Creation is instant regardless of base size, and the read-only majority of every tier (/usr, /lib, package payloads) stays shared: one base file on disk backs every sandbox built from it, and the OS page cache holds one copy of those pages in RAM across all of them. The sandbox only allocates blocks when it writes — a config tweak, a log line, a package install. Typical overhead of the qcow2 indirection on agent/dev workloads is ~0.5%.

The key difference from v1: this is all filesystem-independent.

| Filesystem | Create-from-base | Base sharing | Notes | |---|---|---|---| | ext4 | instant (metadata overlay) | yes | no reflink needed | | xfs | instant | yes | — | | btrfs | instant | yes | optional transparent compression on top | | APFS (macOS) | instant | yes | Apple Silicon dev boxes |

There's no "what changes on ext4" penalty section any more, because nothing changes. The overlay is a qcow2 file; qcow2 does the CoW; the underlying filesystem just stores bytes.

Inside the guest the overlay is transparent. libkrun (imago) presents root.qcow2 as a plain raw ext4 at /dev/vda — the kernel command line gets root=/dev/vda and the guest never knows it's talking to a CoW overlay (cmd/vmm/main.go).

Volumes and the config drive

Standalone volumes (bhatti volume create) and the per-sandbox config drive are also block devices, attached to the VM as additional virtio-blk disks (pkg/engine/krucible/engine.go):

/dev/vdb — the config drive, a ~1 MiB ext4 image carrying the sandbox's hostname, environment, decrypted secrets, injected files, and its per-sandbox auth token. Read-only from the guest's side.
/dev/vdc and up — attached persistent volumes, in attach order. Each carries its guest mount path and filesystem in the config drive's volume spec. A volume image may itself be raw or a qcow2 CoW node; the VMM opens each with the right format.

Because volumes are separate block images, their data is independent of the root overlay: it survives even when a memory snapshot doesn't, and a snapshot/fork of a sandbox restores the whole device set (RAM

root overlay + attached volumes) as one consistent bundle.

Snapshots, forks, and saved images

Everything expensive in v1's storage layer was a cp --reflink. In v2 it's a qcow2 operation, and the three write paths all reuse the same overlay machinery:

bhatti stop / cold tier. The thermal manager pauses the VM at a quiesced boundary and writes the full bundle/ (memory image + VM state + a copy of root.qcow2 + config drive + volumes + manifest.json), then frees RAM. bhatti start restores it. See Thermal states → Warm → Cold.
Fork (bhatti create --from). An instant CoW clone: the new sandbox's root is a fresh qcow2 overlay over the parent's image, and its memory is cloned from the parent's snapshot. No copy of the base.
bhatti image save. Captures the running root as a bootable qcow2 image without stopping the sandbox — a quiesce (in-guest sync, pause, host flush, copy the overlay), yielding a qcow2 CoW node that itself still backs the shared base (pkg/engine/krucible/snapshot.go). Stamping a new sandbox from it is another instant overlay.

All snapshots are Full — v2 has no diff/incremental snapshot mode. The reasoning lives on the thermal page, Why all snapshots are Full. The bundle is portable across machines of the same architecture (manifest.json records an arch/feature gate; a cross-arch restore is refused).

Cold wake and the page cache

When a sandbox is in the Cold state, its bundle/memory.img is the only thing standing between "dead VM" and "fully restored guest." Restore re-launches a fresh bhatti-vmm helper that reads the memory image back into a new VM and resumes from the snapshot point. How fast that is depends heavily on whether memory.img is still in the host page cache.

Right after a snapshot is written, its pages are hot in the cache, so the restore read is nearly free. Thirty minutes later, under memory pressure from a busy host, the kernel may have evicted them — and the restore does a real disk read of the memory working set. This is filesystem-agnostic: it's the OS page cache doing its job, not anything qcow2 or btrfs specific.

In practice a cold restore is sub-second — ~380 ms measured end-to-end on Apple Silicon (M-series, NVMe) for a small sandbox. Slower storage (a Raspberry Pi 5's capped NVMe) is proportionally slower; the shape is the same. A warm resume, where the helper is still alive and the pages are still resident, is ~4 ms — no disk read at all. See Thermal states → What "cold wake" actually costs for the full breakdown.

Bhatti doesn't currently bias eviction of memory.img files in the page cache, or prefetch them ahead of a restore. Both are possible future optimizations, bounded by demand and not yet measured to clear a bar that matters; tracked for later.

No special filesystem needed

The headline for self-hosters: put /var/lib/bhatti on your normal root filesystem and you're done. ext4, xfs, btrfs, APFS — all give instant create-from-base and shared-base dedup, because CoW is in the qcow2 layer, not the filesystem.

btrfs and xfs still offer transparent compression on the base images and bundles if you mount them that way, and it can shrink the data dir meaningfully. But that's an optimization you opt into, not a prerequisite the way btrfs+reflink was in v1. If you're coming from a v1 install on a btrfs loopback, you don't need to carry it forward — v2 is a fresh install.

If you genuinely want the raw path — a plain ext4 root image instead of a qcow2 overlay, for native disk performance or to loop-mount the image on the host — set KRUCIBLE_ROOT_RAW=1. The daemon then clones the base into a raw root.img per sandbox instead of overlaying it (pkg/engine/krucible/engine.go). You trade instant, dedup'd creates for a full base copy per sandbox — the qcow2 default is the right choice for almost everyone.