Decisions & learnings

Every system has a paper trail of bad afternoons. This page is bhatti's.

The decisions on the other Under-the-Hood pages are summarised at the top of each — this page tells the story behind a few of them in more depth than fits on a how-it-works page. If you're evaluating bhatti for production, this is the page I'd read.

The order is roughly chronological with respect to when each problem hit me, not by importance.

The 1-second SYN retransmit (and why it's gone)

On the early Firecracker-based builds, bhatti create had a mysterious floor: about 1 second of every create was a single TCP SYN retransmission timeout. I spent a long time before I understood why, and the answer only makes sense once you know it was a TAP-networked, TCP-agent design.

The mechanism: after the VM started, the host immediately polled the guest's agent port with exec true to detect readiness. The guest kernel was still booting, so the SYN got no answer. Linux's default initial SYN RTO is 1 second (RFC 6298, net.ipv4.tcp_syn_retries); after that timer expired the host retransmitted, and by then the agent was up. A whole second, burned waiting for a guest kernel to boot, visible in every benchmark. Two hacks kept it from being worse — a pre-populated permanent ARP entry so the first probe didn't also pay an ARP-resolution second, and a stale-ARP flush before IP reuse.

krucible deletes the entire phenomenon rather than tuning it:

The transport is vsock, not TCP-over-TAP. The host reaches the guest agent through libkrun's vsock bridge (it dials a host Unix socket bridged to the guest's vsock port). There is no SYN, no three-way handshake to a not-yet-listening port, and no ARP resolution to get wrong — so no TAP device, no bridge, no permanent-ARP trick, and no stale-entry flush.
The lean external kernel boots to agent-ready in ~312 ms on Apple Silicon (HVF, measured — roughly 2× faster than libkrun's bundled kernel), so the window the old design spent retransmitting through barely exists.

The lesson that survived the rewrite: a mysterious constant floor in a latency number is almost always a timer, not your code. The 1-second SYN RTO was invisible in host-side instrumentation because the host work genuinely finished in ~200 ms — the extra second belonged to a kernel timer neither side controlled. Once you can name the timer, you can design it away.

The rory incident

April 2026. A user's persistent sandbox named rory came back from a restore with a corrupted virtio ring buffer. The agent was unreachable. The VM was wedged. We had to destroy it and lost the working state of the sandbox.

Three failures compounded into one bad outcome (docs/archive/PLAN-reliability.md, PLAN-snapshot-reliability-fixes.md):

Diff snapshot corruption. The memory image was a diff — only the pages the dirty-page bitmap said had changed — and that bitmap was incomplete. KVM tracks the pages the guest CPU touches, but a userspace VMM's device model writes guest memory outside KVM: a virtio device servicing the guest writes straight into ring buffers and DMA regions, and those writes never set a bit in KVM's bitmap. Restoring a diff snapshot then loads those pages from a stale base, and any in-flight virtio ring-buffer state gets clobbered.
No snapshot verification — we'd written a corrupt mem file and marked the snapshot as good without checking.
Volume attachment metadata wasn't persisted on snapshot resume — vm.Volumes was empty when a sandbox came back from a named snapshot, so even when we tried to recover, the volumes were unreachable.

The fixes:

No diff snapshots, ever. krucible only writes full snapshots — there is no dirty-page / incremental path left to get wrong. A full snapshot is a few hundred ms slower to write but correct in every case. See Why all snapshots are Full.
Verify the artifacts after every checkpoint. Wrong size, bad path, corrupted state — error out before marking the snapshot good, rather than discovering it on restore.
The device set travels with the snapshot. A memory snapshot (or a fork) reproduces every attached volume, not just RAM and the root overlay — so a restored VM's disk view is always consistent with the RAM it came back with.

The audit also found 17 other bugs — race conditions, missing defer-close, type assertions that would panic on certain SQLite inputs. The PLAN-reliability.md covers all of them. v0.7 was the release that fixed the entire batch.

The lesson I keep coming back to: the cost of correctness is much smaller than the cost of one lost user sandbox. Diff snapshots saved ~500 ms per snapshot. Losing rory cost a user a working state, me a week of debugging, and a release-line of compounding bug fixes. Not even close.

The systemd snapshot/restore problem

I had lohar running as a systemd-managed service for a while. Fresh boot worked. Snapshot/restore broke in a way that took a week to understand.

The setup: a regular Ubuntu rootfs with systemd as PID 1, lohar running as lohar.service (Type=simple, Restart=always). On a fresh boot, systemctl is-system-running returned running, lohar accepted connections, bhatti exec worked.

Then bhatti stop (snapshot to disk) and bhatti start (restore).

The restore looked successful: the restore returned ok, the guest kernel resumed, lohar's listener was still bound to port 1024 according to ss -tlnp from inside the VM. The first bhatti exec after restore would succeed. Every subsequent exec would hang forever.

What I observed:

The kernel-level TCP state was fine. Three-way handshake completed, packets visible in tcpdump.
The connection sat in the kernel's accept queue.
The Go runtime — sitting on top of epoll_wait — was not waking up. The goroutine blocked on Accept() never got scheduled.

I don't have a clean explanation for why this happens specifically when lohar is a child of systemd, and not when it's PID 1 itself. My guess: something about how systemd manages file descriptors, control groups, or its own epoll sets across PID 1 leaves the resumed Go process holding a poller in a state Go's runtime doesn't recover from. That's speculation, not diagnosis.

What I know empirically:

Reproduced across multiple VMM and guest-kernel versions.
Reproduced with Restart=no and Restart=always.
Lohar as PID 1 (no systemd) — never reproduced, in CI or production.

So lohar stayed PID 1, and we built a systemctl shim to satisfy the package machinery. That decision carried straight into v2: krucible boots lohar as PID 1 from /init.krun (block-root boot), so the systemd-child hazard never had a chance to reappear. If you've traced this further or think the diagnosis is wrong, please open an issue. The shim works, but real systemd would be less code to maintain.

Making vsock survive snapshot/restore

vsock is the natural host↔guest channel for an in-process VMM: libkrun exposes each guest vsock port as a host Unix socket, and the daemon dials it directly. It works perfectly during normal operation. The hazard is snapshot/restore — and it isn't in the transport, it's in the guest's Go runtime.

What happens with a naive listener: after a cold restore, the guest's Go runtime can come back with its netpoller (epoll) registrations in a state it never recovers from. A goroutine blocked in Accept() on a runtime-registered vsock socket simply never gets scheduled again — the host connects, the kernel completes the accept, and the guest never wakes to it. (This is the same failure shape as the systemd story above: a resumed Go poller that wedges.)

The fix lives in lohar, not the transport. lohar's vsock listener is a plain blocking socket with a one-second SO_RCVTIMEO, so accept4 returns EAGAIN on a kernel timer rather than parking on the Go netpoller. When an accept that should have returned in milliseconds instead takes seconds, lohar reads that as "we were just restored," tears the listener down, and recreates it fresh (cmd/lohar/net.go). A kernel-level timer keeps firing even when the Go runtime is stalled, so the guest always gets a chance to notice the resume and rebuild a working listener.

The lesson: the thing that breaks across snapshot/restore is rarely the device — it's the userspace runtime state layered on top of it. An earlier design reached for a different transport to dodge the problem; the durable fix was to stop trusting the runtime's poller across a resume and lean on a timer the kernel owns. As a public corroboration point that suspend/restore is genuinely hard here, SlicerVM dropped their suspend/restore feature in v0.1.108 (archive notes).

The Cloudflare Tunnel disconnect

A user reported their bhatti shell dropping silently when they ran a long-running command through api.bhatti.sh (docs/archive/PLAN-shell-sessions.md). Screenshot: no error, no "detached" message, no bash prompt — just a silent return to the shell on their Mac.

Root cause:

They ran bhatti shell rory, started hermes gateway (a daemon that prints a startup banner then waits for events).
Cloudflare Tunnel has a WebSocket idle timeout around 100 seconds. No traffic in either direction = connection killed.
The CLI's conn.ReadMessage() returned an error. The error path was silent — defer term.Restore() ran, terminal mode reset, but nothing got printed.
bhatti shell rory again created a new session. The old session was still alive inside the VM with hermes gateway running and scrollback accumulating, but there was no way to get back to it.

Three problems compounded:

The WebSocket layer didn't have ping/pong keepalives, so idle connections were getting killed by intermediaries.
The CLI didn't print anything on disconnect, so users didn't know what happened.
There was no session reattach. bhatti shell always created a new session.

The fix was a long PR (docs/archive/PLAN-shell-sessions.md) with a 12-bug inventory. The notable ones:

WebSocket ping/pong keepalives every 30 seconds, both client and server.
Concurrent WebSocket write race — the CLI's main loop and its resize-handling goroutine both wrote to the connection without coordination. Replaced with a single writer goroutine fed by a channel.
Session reattach — bhatti shell <name> now attaches to the most recent live session (if any) rather than always creating new.
Disconnect message — CLI prints [disconnected: <reason>; reconnect with bhatti shell <name>] on the way out.
Scrollback ring buffer thread safety — was being accessed concurrently without a mutex; reads could get torn.

The takeaway: production network conditions kill connections that look idle to intermediaries. If you're building over WebSocket, ping/pong is not optional. And the user-visible behavior on disconnect is part of the API — silently exiting is an outage from the user's point of view.

The public proxy rate-limiter recalibration

The first version of the public proxy used a per-alias token bucket with burst 100 and refill 200/min as the primary rate limiter. Logic: "each published URL gets its own quota."

Then a real Vite app with hot module reload generated 90+ requests on a single page load, exhausted the bucket on the first reload, and served 18 of 91 requests as 429s — a blank page on reload.

Documented in docs/archive/PLAN-public-proxy-ratelimit.md. Two things were wrong:

The aggregation level was wrong. Standard reverse-proxy practice (nginx limit_req_zone, Cloudflare, AWS WAF) is per-source-IP rate limiting. Per-destination is a secondary aggregate against distributed attacks, not the primary mechanism. Browsers HMRing are one source IP, not one URL.
The numbers were too low. 200 requests/min ≈ 3.3 req/s. A modern bundler-driven page exceeds that during a single load.

The fix: per-source-IP primary limit (much higher burst, much higher refill), per-alias as a secondary cap (very high — only catches runaway loops), global as a third (host-level safety net).

This is the kind of decision that's hard to get right in advance, because the number depends on the workload. We had reasonable defaults for an API server. The defaults were wrong for a frontend. The calibration came from one user with one Vite app.

Pure-Go SQLite

The choice was mattn/go-sqlite3 (CGo binding to the C SQLite library) vs modernc.org/sqlite (pure-Go translation of SQLite's C code).

The CGo version is faster and more compact. The pure-Go version is ~10% slower and the binary is ~3 MB larger. For metadata CRUD (sandboxes, users, secrets, events), the speed difference is irrelevant.

What matters: cross-compilation. Bhatti is built on a Mac and deployed to a Pi. With CGo, this requires a cross-compiler toolchain (aarch64-linux-gnu-gcc), careful library management, and different build commands per platform. It also means the binary is dynamically linked against libc and isn't fully portable across distros.

With pure-Go SQLite, GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build produces a static binary that runs on any Linux. Same binary works on Pi, Hetzner, Graviton, an Alpine container, an Ubuntu host, anything.

The 3 MB and 10% are bought, gladly, for the build simplicity.

Per-owner network isolation (no shared bridge)

The first version of bhatti had one Linux bridge — brbhatti0 — and all VMs across all users shared it on a single subnet. Simple to operate. Insecure once you actually have multiple tenants: at L2, alice's VM could ARP-scan the bridge and reach bob's VM directly, and iptables FORWARD rules were the only barrier. A misconfigured rule meant cross-tenant traffic.

v1 fixed the L2 leak with a bridge per user. v2 removes the L2 fabric entirely. Under krucible there are no host TAP devices, no bridges, and no iptables rules. Each guest's eth0 is a virtio-net link over a Unix socket to a per-owner bhatti-netd gateway — a userspace gVisor netstack that is the guest's router, DNS, and egress policer. One netd per owner: alice's VMs connect to alice's gateway, bob's to bob's, and there is no shared switch for either to scan. Same-owner siblings reach each other through their gateway (an L3-routed proxy), not over a shared L2 segment. See Networking.

The lesson, unchanged from v1: start with the security model you'll need, not the simplest one that works for now. v1 learned it the hard way — a single shared bridge, then a scramble to per-user bridges; v2 bakes per-owner isolation into the transport itself, so there is no shared fabric left to misconfigure.

Server-side file truncation

Coding agents read files. They almost always truncate — typically to the first 2000 lines or 50 KB. We discovered this by watching what clients actually did with our FILE_READ_RESP data: they'd ask for a 100 MB log file and immediately throw away 99.95% of it on the client side. We were paying full bandwidth for almost no information.

The fix is small. FILE_READ_REQ accepts offset, limit, max_bytes. Lohar reads line-by-line with bufio.Scanner and stops at whichever limit hits first. The response includes the full file size so the consumer knows whether content was truncated. See wire protocol — file read.

Performance: a truncated read on a 10 K-line file is about 4.5× faster at p50 than a full read of the same file. For larger files, proportionally bigger.

Why not head -n via exec? Fork-exec, pipe setup, shell argument parsing — all of that is overhead the file protocol path skips. A 1 KB file read takes about 472 µs in the benchmark; the equivalent exec is more like 1 ms. For a coding agent that does hundreds of file reads per task, this adds up.

The general lesson: the client knows what it wants; tell the server. Even when bandwidth is "free" inside a single host, the allocations, copies, and discards still cost.

Sessions everywhere

Most sandbox systems split exec and shell into different concepts. Bhatti unifies them: every TTY exec is a session. There's no separate "shell" code path. A shell is just a TTY exec of /bin/zsh or /bin/bash.

Three reasons this fell out of real product needs:

Init scripts need to be attachable. When you create a sandbox with --init "npm install && npm run build", that command runs as a session named init. You can attach to it from the host to watch progress. If exec weren't a session, the init output would be invisible from the host.
Shells need to survive proxy disconnects (the Cloudflare Tunnel story above).
Snapshot/restore needs to preserve interactive shells. A shell running a long-running command when the VM is snapshotted should be the same shell after restore. Sessions are the data structure that survives that round-trip.

Trade-off: every TTY session allocates a 64 KB scrollback buffer. With 100 concurrent sessions per VM, that's 6.4 MB. We cap at 20 sessions per VM (cmd/lohar/handler.go:50) which keeps it bounded.

When kasmvnc.service couldn't restart

The computer tier shipped in v1.11.9 — XFCE + KasmVNC + Chromium, all running as systemd units managed by the shim. First boot worked end-to-end (desktop renders, password generation runs once, screenshot works). Restarting kasmvnc.service did not. The unit transitioned to failed and the journal showed:

_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
Fatal server error:
Cannot establish any listening sockets - Make sure an X server isn't
already running

Xkasmvnc from the previous run was still alive, holding the abstract Unix socket @/tmp/.X11-unix/X99. No file-system cleanup frees an abstract socket — only killing the holder does.

systemctl stop kasmvnc should have killed it. The shim does cgroup.kill on the unit's cgroup, which the kernel translates into SIGKILL for every PID in cgroup.procs. Why didn't Xkasmvnc go down?

$ sudo cat /proc/$(pgrep -x Xkasmvnc)/cgroup
0::/

$ sudo cat /sys/fs/cgroup/system.slice/kasmvnc.service/cgroup.procs
(empty)

Xkasmvnc was in the root cgroup, not the unit's. cgroup.kill on the unit's cgroup had nothing to kill.

The shim's old startDaemon called cmd.Start() and then wrote the daemon's PID into the unit's cgroup.procs. The window between those two operations is microseconds, but Xkasmvnc uses daemon(3) — fork, parent exits, child detaches — during startup. The child fork happened during the window, before the placement write, so the child inherited whatever cgroup the parent was in (lohar's, i.e. root). Once the parent exited, the placement write moved only the dead parent's PID. The detached Xkasmvnc was forever in the wrong cgroup.

The same shape applies to any double-forking daemon: classic Apache, dbus-daemon in default mode, nginx without daemon off. v1.11.9 shipped the contract bug — every future tier author would have to audit whether their daemon double-forks during startup, and we'd have no way to enforce it without reading every package's source.

The fix is to remove the race entirely by placing the cgroup before any fork can happen. Real systemd does this with clone3(CLONE_INTO_CGROUP) — a single syscall that forks and places atomically. Go's runtime doesn't expose clone3; reproducing it means reimplementing syscall.ForkExec's post-fork setup (close-on-exec, signal-handler reset, credentials, working dir) under syscall.ForkLock. That's real surgery on code where bugs are silent and only surface under load.

The alternative is a thin one-purpose helper. The supervisor invokes lohar spawn instead of the daemon directly; spawn writes its own PID into <cgroup>/cgroup.procs and then execves into the daemon. execve preserves PID and cgroup membership, so the daemon — same PID throughout — is already in the unit's cgroup when it forks anything later. About 60 lines of plain Go (cmd/lohar/spawn.go) gets the same correctness guarantee as clone3 with code anyone can read in one sitting. The dispatch is on argv[1] (/proc/self/exe spawn ...) rather than the busybox-style symlink we use for systemctl and journalctl, because lohar spawn is a private supervisor primitive — not something a user should be wrapping their own commands with. clone3 stays open as a future migration if the extra execve ever shows up in profiling.

Shipped in v1.11.10. Every existing unit file works unchanged. The operator-visible difference: systemctl stop/restart now works for every kind of daemon, including the ones that fork during startup. See How services are spawned for the mechanism.

Owning the VMM (krucible)

bhatti's whole product is thermal states: sandboxes that pause, snapshot to disk, restore in a fraction of a second, and fork into instant copies. Those are VMM-level capabilities — pause/resume, checkpoint/restore, and memory-clone fork — and they are not on the upstream roadmap of the in-process VMM library we build on (libkrun). You can't bolt them on from the outside; they touch how the VMM saves and restores vCPU state, memory, and device state.

So we forked libkrun into krucible and added exactly those pieces: a control socket carrying PAUSE/RESUME/STATUS/CHECKPOINT/RESTORE/FORK, full-VM snapshot and restore, and a block-root boot path that lets lohar run as PID 1 from /init.krun. The bhatti daemon stays pure Go and never links libkrun — it spawns one bhatti-vmm helper per sandbox (the helper becomes the VM, since libkrun's entry call blocks for the VM's life) and drives it out-of-band over that control socket.

The lesson: if a capability is the product, own the layer it lives in. Waiting for upstream to grow pause/resume/fork would have meant not shipping the product. A fork is a maintenance cost — we rebase onto upstream libkrun on our own cadence — but it's the only way to control the thing the product is made of.

Why krucible over Firecracker

v1 ran on Firecracker. krucible replaces it for four reasons, each of which was awkward or impossible on the old engine:

One engine on the laptop and the server. krucible runs on KVM on Linux and HVF (Hypervisor.framework) on Apple Silicon, so the same sandbox model boots on a developer's Mac and on a Linux host. Firecracker is Linux/KVM only.
Live snapshot, pause/resume, and fork as first-class verbs, driven over the control socket — the thermal-state machinery the product is built on, owned end to end rather than pieced together around an external process.
Host-owned networking. The guest link is a virtio-net device to a per-owner bhatti-netd gateway; there are no TAP devices, bridges, or iptables rules on the host, and no per-user subnet to allocate. The network policy lives in a userspace gVisor netstack we control.
A zero-secret-on-disk posture. Per-sandbox env, secrets, and the agent token ride a small config drive that lohar mounts read-only, applies, then unmounts and removes — the secrets don't linger on a guest filesystem after boot.

Firecracker is a genuinely excellent VMM; the switch isn't a knock on it. It's that bhatti's requirements — cross-platform, snapshot/fork-native, host-owned networking — line up with a VMM we can extend, not one we drive from the outside.

qcow2 copy-on-write (dropping the btrfs dependency)

v1's fast create leaned on the host filesystem: a btrfs (or xfs-reflink) volume so a new sandbox's rootfs was a metadata-only reflink clone of a base image. It was fast, but it made the filesystem a deployment prerequisite — and on plain ext4 the same clone fell back to a full copy and create got noticeably slower.

v2 moves copy-on-write up into the disk image format: each sandbox gets a thin qcow2 overlay over a shared read-only base, and the CoW is the image format's job, not the filesystem's. Instant create and base-sharing now work identically on ext4, xfs, btrfs, and APFS, with ~0.5% overhead on agent workloads. There is nothing to pre-provision.

The lesson: push a requirement down a layer until it stops being the operator's problem. "You must run bhatti on btrfs" is a real barrier; "bhatti stores qcow2 files on whatever filesystem you already have" isn't a barrier at all.

No jailer yet (Track J)

On Firecracker, the VMM process ran under a jailer — a chroot, a dropped-privilege uid, cgroups, and a seccomp filter around the VM process — so a hypothetical VM escape landed in an empty jail rather than on the host. krucible does not have an equivalent yet: the daemon spawns bhatti-vmm directly (on Linux it runs as root because it needs /dev/kvm; on macOS HVF needs no root, but the helper is unsandboxed).

I'd rather be honest about this than paper over it. Jailing the helper for genuinely hostile multi-tenant use on Linux is planned work — internally "Track J" — not something that ships today. For single-user and dev use (the common case, and the whole story on a Mac laptop) it isn't load bearing; for hostile multi-tenant on shared Linux hardware, it's a gap you should know about.

What I'd do differently

Things I'd revisit if I were starting bhatti from scratch today:

Use a real init for the systemctl shim from the start. I'm reasonably happy with the shim (it's small, it covers the surface Debian's tools actually call), but I went through several rounds of "what about this directive too?" before it stabilized. Looking at it again, I'd start from a small supervisor like s6-rc and add a thin systemctl-compatible wrapper. The shim would still exist, but the service-supervisor code would be battle-tested rather than home- rolled.
Snapshot verification from day one. I shipped snapshots without verifying the artifacts, and the rory incident is what it cost. The verification logic is ~50 lines. Not having it from the start was a real mistake — and it's why v2 only ever writes full snapshots and checks them before marking them good.
Per-source-IP rate limiting on the public proxy from day one. Per-destination was the wrong default; "match what nginx does" is the boring-and-correct answer.
TLS for the agent protocol. The agent protocol authenticates with a 16-byte hex token, framed over vsock — the host dials the guest's socket through libkrun's bridge, so there's no network path another guest could reach it on, and the host always initiates (no certificate-pinning equivalent on the guest side). Still, TLS would give us defense in depth. Haven't done it yet because the threat model doesn't demand it on a single-host deployment, but it should be on the table for any real multi-host setup.

Where to go next

If you want to read more like this, the full archive is in docs/archive/ in the repo. Plans, post-mortems, write-ups on the Linux / KVM / cloud-init behaviors that drove our design choices (with cross-references to publicly documented incidents at other orchestrators in the space — SlicerVM, fly.io's Sprites — where their reports matched ours), and the migrations that got us to v1.7.

Architecture — how all the pieces fit together
Lohar — the agent inside every VM
Thermal states — pause / snapshot / restore in detail
Networking — the per-owner bhatti-netd gateway, egress policy, and host↔guest forwarding
Wire protocol — frames, ports, atomic writes