Skip to content
[bhatti]

Thermal states: hot, warm, cold

A sandbox you aren't using doesn't keep its CPUs running, doesn't keep its memory resident, and eventually doesn't show up as a process on the host at all. It costs you nothing while it's idle. The next request wakes it.

That's the headline. The rest of this page is how it works, and what happens when it doesn't.

If you haven't read it yet, the engine internals page is the right pre-read — this page leans on three concepts from it: vCPU pause/resume over the control socket, the memory checkpoint/restore bundle, and krucible's lazy-commit memory model. There's a one-paragraph recap of each below.

Background, in three paragraphs

vCPUs. Each sandbox is a bhatti-vmm helper process — a cgo wrapper around krucible (libkrun) that is the VM. The daemon drives it out-of-band over a control socket: a one-line PAUSE freezes the guest's vCPUs, RESUME unfreezes them, STATUS reports state (pkg/engine/krucible/control.go). "Paused" means the hypervisor stops running guest instructions; the helper process stays alive and the guest's memory stays mapped. The round-trip is single-digit milliseconds.

Checkpoints. A control-socket SNAPSHOT <dir> writes the VM's full memory image, its device/vCPU state, a copy of the qcow2 root overlay, the config drive, and any attached volumes into a self-contained bundle, plus a manifest.json. Restore is the inverse: re-launch a fresh helper pointed at the bundle, which loads memory and device state back and resumes from the checkpoint point. After a successful restore every guest process picks up exactly where it was, right down to in-flight TCP connections.

The memory model. krucible maps guest RAM MAP_PRIVATE | MAP_ANONYMOUS with lazy commit, so a VM's host memory footprint only ever counts the pages the guest has actually touched — not its configured ceiling. This is why a paused (warm) VM's resident memory naturally shrinks to its working set with no extra work, and why v2 needs none of v1's balloon-reclaim machinery (pkg/engine/krucible/thermal.go).

Design decisions on this page

  1. Three thermal states, not two. Hot/warm/cold gives us a cheap mid-point (vCPUs paused but memory still resident) so the common case of "you come back five minutes later" doesn't pay the cost of a checkpoint restore. See The three states.
  2. All snapshots are Full. v2 has no diff/incremental snapshot mode — every checkpoint writes every touched page. This is a deliberate correctness choice paid for by a real production incident. See Why all snapshots are Full.
  3. No balloon needed on hot→warm. Because krucible commits guest RAM lazily, a paused VM already only holds its touched pages resident — there's no separate step to "take memory back." See The memory model: lazy commit.
  4. Cold check uses host-side timing, not an agent query. vCPUs are paused while warm — the agent can't respond. Querying it would either time out or wake the VM. So we use the timestamp recorded when we transitioned to warm. See Why the cold check doesn't talk to the agent.
  5. Circuit breaker on a stuck VM. If the agent fails ten Activity queries in a row (~100 s of silence), the thermal manager force-pauses the VM rather than leaving it hot and unresponsive. See The circuit breaker.
  6. Three retries before marking unknown. Snapshot writes can fail on a transient I/O hiccup. We retry. Three failures in a row gets the sandbox marked unknown and surfaces an event. See Snapshot retries.

The three states

            idle 30 s                 idle 30 min
   Hot ────────────────────► Warm ──────────────────────────► Cold
    ▲                       │                             │
    │                       │                             │
    │    warm wake ~4 ms    │      cold wake (see below)  │
    └───────────────────────┴─────────────────────────────┘
                        any API request

| State | bhatti-vmm process | vCPUs | Host RAM | Resume to hot | |---|---|---|---|---| | Hot | alive | running | working set | — | | Warm | alive | paused | working set (still resident) | ~4 ms | | Cold | dead | — | 0 (checkpoint on disk) | sub-second (see below) |

Warm resume is a single RESUME over the control socket — the helper is alive, the guest pages are still mapped in host RAM, no disk read is needed. It's ~4 ms.

Cold resume has its own breakdown because it spans more than one thing: spawn a fresh bhatti-vmm helper, read the memory image from disk, restore the VM, wait for the agent to respond. How much of that disk read hits the page cache vs. storage depends on how recently the sandbox went cold and what else is happening on the host. See What "cold wake" actually costs below.

Slower hardware (a Raspberry Pi 5 with its capped NVMe) gets proportionally slower numbers across the board; the shape of the state machine is the same.

Transitions happen automatically. Every API call to a sandbox runs through the engine's wake path (Wake/EnsureHot) first. If the sandbox is warm it resumes in milliseconds. If cold, it restores the checkpoint, then your operation runs. From an API client's perspective, every sandbox is always "running" — the resume is just the first slice of latency on the first call after it went cold.

What "cold wake" actually costs

A cold restore re-launches the helper and reads the memory image back into a fresh VM. In practice this is sub-second — ~380 ms measured end-to-end on Apple Silicon (M-series, NVMe) for a small sandbox.

The dominant variable is the host page cache. Right after a checkpoint is written, the memory image is hot in the cache and the restore read is nearly free. Thirty minutes of memory pressure between checkpoint and resume is enough to evict most of those pages on a busy host, and then the read happens for real off disk. This is filesystem-agnostic — it's the OS page cache, not anything qcow2- or btrfs-specific. The wider context (base-image sharing, what's not yet optimized) is on the storage page: Storage → Cold wake and the page cache.

You can opt out per sandbox with bhatti create --keep-hot on creation, or bhatti edit <name> --keep-hot later. A keep-hot sandbox skips thermal management entirely (pkg/server/sandbox_handlers.go) and holds its full working set resident. This is for sandboxes that maintain external connections — a Slack WebSocket, a Discord gateway, a long-running training job that you don't want randomly suspended.

Hot → Warm

After 30 seconds with no API activity and no attached TTY sessions, the thermal manager pauses the VM (pkg/engine/krucible/thermal.go):

  1. PAUSE over the control socket. The vCPU pause is single-digit milliseconds.
  2. Record the pause time as the sandbox's last-activity timestamp. The warm-to-cold timer starts from now, not from your last exec.

There's no balloon step. Because krucible commits guest RAM lazily, the paused VM's resident footprint is already just the pages the guest has touched — the host doesn't need to ask for memory back. The bhatti-vmm helper is still alive; the guest's network link (its virtio-net socket to the owner's bhatti-netd gateway) stays connected. The vCPUs are simply frozen. Resuming is another RESUME call.

The "no attached TTY sessions" check matters. If someone is in bhatti shell right now, we don't pause — that's a person staring at a terminal. The attached-session count comes from the agent's Activity response (cmd/lohar/handler.go), which is the only authoritative source.

Warm → Cold

After 30 minutes warm, the manager takes a Full checkpoint to disk and kills the bhatti-vmm helper (pkg/engine/krucible/engine.go):

  1. PAUSE over the control socket (a no-op if already paused).
  2. SNAPSHOT <bundle-dir> — the helper streams the whole guest RAM, the device/vCPU state, a copy of the qcow2 root overlay, the config drive, and any attached volumes into the bundle, plus a manifest.json. The guest's page cache travels inside the memory image, so in-flight file writes are preserved across the round-trip without a separate flush. This is a generous-deadline operation (streaming a whole RAM image to disk).
  3. If the snapshot write fails — say the disk is out of space for the memory image — the guest is still paused, so we RESUME it rather than leave it frozen (a frozen guest hangs the next exec), then surface the error. The sandbox stays usable.
  4. On success, kill the helper. Its RAM is returned to the host, and the sandbox's on-disk footprint drops to the bundle.
  5. Update the database — sandbox is now stopped, thermal is cold.

There are no host TAP devices or bridges to tear down — networking is handled by the per-owner bhatti-netd gateway, which is shared across the owner's sandboxes and outlives any single VM. See Networking.

Why all snapshots are Full

A diff/incremental snapshot writes only the pages modified since the last one — for an idle VM that's a fraction of the memory and a fraction of the time. It's tempting. bhatti does not do it.

The reason is the rory incident (April 2026). A user's persistent sandbox came back after a restore with corrupted device state. The agent was unreachable, the VM was wedged, and we had to destroy it and lost the working state of the sandbox. The root cause was a diff snapshot whose dirty-page set was incomplete: the hypervisor's dirty-page tracking missed writes that happened in the VMM's own userspace device-model code, so restoring the diff loaded some pages from a stale base and clobbered in-flight device state. A diff snapshot is only ever as trustworthy as its dirty-page bitmap, and that bitmap had a blind spot.

The fix was to stop chasing diffs entirely. v2 has no diff snapshot mode — every checkpoint is a complete, self-consistent image of every touched page (pkg/engine/krucible/engine.go, pkg/engine/krucible/snapshot.go). The bundle is also verified before the sandbox is marked stopped — a corrupt checkpoint is worse than no checkpoint.

It's slower to write than a diff would be. On any modern host disk that's fine, and the trade is correctness for speed — which I'll take any day after losing rory. The full audit is in the decisions page.

Snapshot retries

Disks fail in transient ways: a brief I/O stall, a momentary contention spike. A snapshot that fails once usually succeeds on the second try. The thermal manager retries up to three times per cycle before marking the sandbox unknown (pkg/server/server.go):

attempt 1 fails → log warning, increment counter, try again next cycle
attempt 2 fails → log warning, increment counter, try again next cycle
attempt 3 fails → log error, mark sandbox unknown, record event

Each failure also records a thermal.snapshot_failed event with the attempt number and error, so an operator can see why (bhatti admin events --type thermal.snapshot_failed). The VM stays warm (alive, vCPUs paused) between retries — a failed snapshot never kills a live sandbox.

Why the cold check doesn't talk to the agent

Look at runThermalCycle and you'll see the warm→cold transition computed without an agent query. That's deliberate.

Agents can't respond when vCPUs are paused. A query would either time out — silently skipping the cold check — or trigger a vCPU resume to service the request, which defeats the entire reason we paused. So the cold check uses the timestamp the manager set when it transitioned the sandbox to warm.

It's a small detail. It took an afternoon to figure out why my warm sandboxes never went cold.

Cold → Hot

Any API call to a cold sandbox triggers the wake path, which calls Start to restore from the checkpoint bundle (pkg/engine/krucible/engine.go):

  1. Validate the bundle. If it's present and well-formed, restore from it; if it's missing or corrupt (a crashed or never-snapshotted sandbox whose RAM is gone but whose root overlay persists), fall back to a fresh cold boot. Recovery relies on this fallback for restart-safety.
  2. Ensure the owner's bhatti-netd gateway is listening on its net socket, spawning it if this is the owner's first live sandbox. Sibling sandboxes reuse the same gateway.
  3. Remove any stale control/forward sockets from the dead incarnation, then spawn a fresh bhatti-vmm helper pointed at the bundle. The helper loads the memory image and device/vCPU state back and resumes the guest from the checkpoint point.
  4. WaitReady — poll the guest agent over its vsock-backed control socket for up to 30 seconds. If lohar answers, the sandbox is hot. If it doesn't, the restore is treated as failed (see below).

The kernel and the guest's processes don't restart — they pick up exactly where they were, including open connections. A successful cold restore is sub-second (~380 ms on Apple Silicon); most of the cost on a cold cache is the memory-image read. See the cold-wake breakdown above and Storage → Cold wake and the page cache for the full picture.

When restore fails

If the helper can't load the bundle, or the agent doesn't respond within 30 seconds, the restore is aborted: the half-spawned helper is killed and the sandbox surfaces an error rather than sitting in a half-restored state (pkg/engine/krucible/engine.go).

Error: sandbox "dev" failed to restore: <reason> —
use 'bhatti start --force' to retry or destroy and recreate
(volume data is safe).

bhatti start --force <name> retries the restore. If the underlying issue (disk full, a truncated bundle) is fixed, this gets you back to running. If not, it fails again.

Volume data lives on separate block images attached as virtio-blk drives; they survive even when the memory checkpoint doesn't. So the fix is usually: detach volumes, destroy the sandbox, recreate, reattach.

The memory model: lazy commit

This is the part of thermal management most other systems bolt on a device for, and krucible gets for free. In v1 (Firecracker), a paused VM still held all of its configured RAM as host memory, so pausing alone reclaimed nothing — you had to inflate a virtio-balloon to hand pages back to the host, and deflate it on resume.

krucible doesn't have that problem. It maps the guest's RAM MAP_PRIVATE | MAP_ANONYMOUS, which the kernel commits lazily — a guest page only consumes host physical memory once the guest actually writes to it. A 1 GB VM whose working set is 200 MB only ever costs the host ~200 MB, hot or warm. Pausing the vCPUs freezes that working set in place; nothing needs to be "taken back."

Concretely, the balloon interface is a no-op on krucible — it's kept on the engine interface only for compatibility, and calling it does nothing (pkg/engine/krucible/thermal.go):

// BalloonSet is a no-op on krucible: libkrun maps guest RAM
// MAP_PRIVATE|MAP_ANONYMOUS (lazy commit), so a paused VM's host RSS
// already only counts touched pages.
func (e *Engine) BalloonSet(ctx context.Context, id string, amountMiB int64) error {
    return nil
}

For a self-hosted box with twenty sandboxes, this is the difference between "I can run twenty VMs" and "I can run many more" — density falls out of the memory model rather than a reclaim trick, and there's nothing to configure or tune.

The host-side activity cache

Every API call records time.Now() for the sandbox in the server's activity cache (pkg/server/server.go). The thermal cycle checks this before querying the agent at all. If the cached timestamp is within the warm timeout, the agent query is skipped.

Why bother? Because with 50 active sandboxes the cycle runs every ~10 seconds, and naively that's 50 agent round-trips every cycle — most of them returning "still active." The cache eliminates the queries for sandboxes that have had recent API traffic. Only genuinely-idle sandboxes get queried over vsock.

The activity cache is a heuristic, not the source of truth. The agent's own lastActivity (updated by every exec and stdin) is authoritative — but querying it costs a round-trip. So we use the cache as a fast-path negative ("definitely active, skip") and fall through to the agent for everything else.

The circuit breaker

If the agent fails to respond ten times in a row to Activity queries — about 100 seconds of silence — the manager force-pauses the VM (pkg/server/server.go):

const maxThermalFailures = 10

Pausing is a control-socket PAUSE; it doesn't need the agent. This catches the "agent is alive but stuck" case and stops a wedged VM from burning a vCPU indefinitely. A user sees their sandbox go to warm even though they never asked for it; their next request wakes it normally. If the agent is genuinely dead, the next restore will fail and they'll get the restore-failed error described above. keep_hot sandboxes are exempt.

The threshold of 10 is a heuristic:

  • ~10-second tick interval × 10 failures = ~100 seconds of unresponsiveness
  • A normally-loaded agent answers Activity in single-digit milliseconds
  • A real network or scheduling hiccup might lose 1–2 queries; not 10

If you want to monitor this, watch for thermal.force_pause events.

What you'll see in practice

A short field guide.

bhatti list shows cold next to your sandbox. That's normal. It's saving you memory. Run any command on it and it'll wake.

First request after a long pause is slow. That's a cold wake. Subsequent requests are normal speed.

Sandbox transitioned to warm even though I'm using it. The check requires no attached TTY and >30 s since the last API call. Background processes inside the VM don't count as activity from the manager's perspective. If you have a build running and want it to stay hot, set --keep-hot.

bhatti list shows unknown. Three snapshot retries failed. Look at the bhatti server logs and bhatti admin events --type thermal.snapshot_failed. You can usually bhatti start --force once the underlying issue (disk full, truncated bundle) is fixed.

Resume fails with "failed to restore". Volume data is safe; destroy and recreate. If this happens twice in a row on the same VM, file an issue — we'd like to see it.

Where to go next

  • How a sandbox boots — the cold start path in detail
  • Networking — the per-owner bhatti-netd gVisor gateway, and why open connections survive a checkpoint/restore
  • Storage — the qcow2 overlays and the checkpoint bundle on disk
  • Lohar — the agent that resumes inside the VM
  • Decisions & learnings — the rory incident in full, plus other paid-for knowledge