This is the full developer documentation for bhatti
# Quickstart
> Install bhatti on your own machine, create a sandbox, and run something inside it. About five minutes including the install download.
This is the path I recommend for everyone except remote-CLI users sharing someone else’s bhatti server. You’ll install the daemon on a machine you own — a Linux box with KVM **or** a Mac on Apple Silicon — and from that same box create your first sandbox and run something inside it.
If you don’t have a suitable box in front of you, the [Self-hosting](/docs/self-hosting/) page covers what you need (a Raspberry Pi 5, a Hetzner box, any cloud VM with nested virtualization, or an Apple Silicon Mac).
## Install
[Section titled “Install”](#install)
```bash
curl -fsSL bhatti.sh/install | sudo bash
```
That’s the whole install. The script:
1. Downloads one self-contained runtime bundle — the `bhatti` CLI + daemon, the `bhatti-vmm` VMM helper, the `bhatti-netd` gateway, the `libkrun` (krucible) library, the lean guest kernel — plus the minimal Ubuntu 24.04 rootfs.
2. Installs and starts the service (systemd on Linux, launchd on macOS).
3. Creates an `admin` user and writes its API key + endpoint to your user’s `~/.bhatti/config.yaml`. **You don’t need to run `bhatti setup` after.** The CLI on this box is already wired up.
The transcript ends with the admin API key printed once. Save it — you’ll want it later for adding teammates or driving the server from another machine.
## Create a sandbox
[Section titled “Create a sandbox”](#create-a-sandbox)
```bash
bhatti create --name dev
```
```text
sandbox/dev created (1 vCPU, 1024 MB)
Shell: bhatti shell dev
```
Defaults are 1 vCPU and 1024 MB. Override with `--cpus` and `--memory`. See [`bhatti create`](/docs/reference/cli/sandbox/create/) for every flag.
## Run a command
[Section titled “Run a command”](#run-a-command)
```bash
bhatti exec dev -- echo hello
# → hello
```
Anything after `--` runs verbatim inside the sandbox. The `--` is optional when there’s no ambiguity:
```bash
bhatti exec dev uname -a
```
## Open a shell
[Section titled “Open a shell”](#open-a-shell)
```bash
bhatti shell dev
```
Interactive PTY, full keyboard. Press `Ctrl+\` to detach — the shell keeps running. Reconnect with `bhatti shell dev` and the scrollback is replayed.
## Publish a port
[Section titled “Publish a port”](#publish-a-port)
If you want to reach a service running inside the sandbox from the internet:
```bash
bhatti exec dev -- bash -c 'python3 -m http.server 3000 &'
bhatti publish dev -p 3000 -a my-app
# → https://my-app.bhatti.sh
```
That URL is public. The sandbox can be cold and the URL still works — the first request restores it in well under a second, then serves. See [Networking](/docs/under-the-hood/networking/) for aliases, custom domains, and how the wake-then-serve proxy works.
## Clean up
[Section titled “Clean up”](#clean-up)
```bash
bhatti destroy dev
```
Or `bhatti destroy dev -y` if you don’t want the confirmation prompt.
## What just happened
[Section titled “What just happened”](#what-just-happened)
1. **`bhatti create`** asked the daemon to boot a **krucible** microVM — a real Linux VM with its own kernel and filesystem, isolated by the CPU’s virtualization (KVM on Linux, HVF on macOS).
2. **`bhatti exec`** sent a command over the wire protocol to lohar, the guest agent running as PID 1 inside the VM.
3. **`bhatti shell`** opened a WebSocket and attached to a PTY session.
4. **`bhatti destroy`** stopped the VM, tore down its network gateway, and removed the sandbox’s copy-on-write overlay.
The sandbox was a full Linux environment, not a container. When idle, it would have paused itself (warm, memory still resident) and, after longer idle, snapshotted its memory to disk (cold) — then restored transparently on the next request.
## Next steps
[Section titled “Next steps”](#next-steps)
* [Concepts](/docs/concepts/) — sandboxes, thermal states, the binaries
* [Self-hosting](/docs/self-hosting/) — adding teammates, custom domains, backups, the rest of the operator path
* [`bhatti exec`](/docs/reference/cli/exec/exec/) — streaming, timeouts, detach
* [`bhatti create`](/docs/reference/cli/sandbox/create/) — every flag
***
### Driving a remote bhatti from your laptop
[Section titled “Driving a remote bhatti from your laptop”](#driving-a-remote-bhatti-from-your-laptop)
If someone else runs the bhatti daemon and you only need the CLI on your local machine — or you want to install bhatti on a server and drive it from your laptop — install the CLI without sudo and point it at the server:
```bash
# Install the CLI binary
curl -fsSL bhatti.sh/install | bash
# Configure the endpoint and key
bhatti setup --url https://your-server:8080 --token bht_abc...
# or interactively:
bhatti setup
```
`bhatti setup` accepts `--url` and `--token` for non-interactive configuration (agents, CI, provisioning scripts). Without flags, it prompts. See [`bhatti setup`](/docs/reference/cli/admin/setup/) for the full reference.
The server operator gets your API key by running `bhatti user create --name ` once, on the server, and sending you the key. See [Self-hosting → adding teammates](/docs/self-hosting/#adding-teammates).
# Concepts
> Sandboxes, thermal states, and the binaries — the mental model for bhatti.
## What a sandbox is
[Section titled “What a sandbox is”](#what-a-sandbox-is)
A sandbox is a real Linux microVM — its own kernel, its own filesystem, its own network stack. Not a container: process isolation is hardware-enforced by the CPU’s virtualization, via **KVM** on Linux and **HVF** (Hypervisor.framework) on Apple Silicon. bhatti boots it with **krucible**, its own VMM (a fork of [libkrun](https://github.com/containers/libkrun)) — so the same sandbox model runs on a Linux server and on a macOS laptop.
Sandboxes are created in well under a second and destroyed instantly. When idle, they’re automatically paused — and eventually their whole memory image is snapshotted to disk and the RAM is freed — then restored transparently on the next request.
## The binaries
[Section titled “The binaries”](#the-binaries)
**bhatti** runs on the host. It’s both the server daemon (`bhatti serve`) and the CLI client (`bhatti create`, `bhatti exec`, …). The daemon is pure Go and never links the VMM. Instead it spawns, per sandbox, a small helper:
* **bhatti-vmm** — the cgo helper that *is* the VM. It links krucible (libkrun) and boots the guest; the daemon drives it (pause, resume, snapshot, restore) out-of-band over a control socket, because the libkrun entry call blocks for the life of the VM.
* **bhatti-netd** — a per-owner network gateway (a userspace [gVisor](https://gvisor.dev/) netstack). It gives the guest policed egress, isolates it from the host, and lets same-owner sandboxes reach each other — with no host TAP devices, bridges, or iptables rules. It’s on by default.
**lohar** runs *inside* the VM as PID 1. It’s the guest agent — a static binary with zero dependencies (no systemd, no libc, no initramfs). It handles command execution, file operations, and PTY sessions, and talks to the host over vsock using a custom [wire protocol](/docs/under-the-hood/wire-protocol/).
The names: *bhatti* (भट्टी) means furnace. *lohar* (लोहार) means blacksmith.
## Thermal states
[Section titled “Thermal states”](#thermal-states)
Every sandbox is in one of three thermal states:
| State | What’s happening | Resume latency | Memory cost |
| -------- | -------------------------------------- | ----------------------------------------------- | ----------- |
| **Hot** | VM is running, vCPUs active | — | Full |
| **Warm** | vCPUs paused, memory still allocated | \~4 ms | Full |
| **Cold** | Memory snapshotted to disk, VM stopped | sub-second (\~380 ms measured on Apple Silicon) | Zero |
The thermal manager transitions sandboxes automatically based on activity: hot → warm after 30 s idle, warm → cold after 30 min. From the API consumer’s perspective, every sandbox appears to be running — a request to a cold sandbox triggers a transparent restore before the operation executes.
You can opt out with `keep_hot: true` on sandbox creation or update, which prevents automatic pausing. Useful for sandboxes maintaining persistent external connections.
## Architecture at a glance
[Section titled “Architecture at a glance”](#architecture-at-a-glance)
```plaintext
CLI / API client
│
▼ HTTP (TCP or unix socket)
┌──────────────┐
│ bhatti serve │ ← host daemon: REST API, thermal manager, proxy (pure Go)
└──────┬───────┘
│ spawns per sandbox / per owner, drives over a control socket
├───────────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ bhatti-vmm │ │ bhatti-netd │ ← krucible (libkrun) VMM + gVisor net gateway
└──────┬───────┘ └──────────────┘
│ vsock
▼
┌──────────────┐
│ lohar │ ← PID 1 inside the VM: exec, files, PTY sessions
└──────────────┘
```
For the full picture, see [Architecture Overview](/docs/under-the-hood/architecture/).
# API Reference
> Every HTTP endpoint with a runnable curl example, the response shape, and notable status codes.
The bhatti server exposes a REST API over HTTP. Every authenticated endpoint requires a Bearer token; unauthenticated endpoints are limited to `/health` and `/_shell/` (the embedded web-shell page).
This page is the canonical hand-curated reference. Each endpoint shows a runnable `curl` and the response shape. Anything not in the curl example uses the documented default.
## Base URL
[Section titled “Base URL”](#base-url)
| Mode | API base | Public proxy |
| ------------------------------------------------------ | --------------------------- | --------------------------------------------- |
| Default (`listen: :8080`) | `http://localhost:8080` | path-based at `/sandboxes//proxy//` |
| Path-based public proxy (`public_proxy_listen: :8443`) | `http://localhost:8080` | `http://:8443//` |
| Domain mode (`domain.api_host`, `domain.proxy_zone`) | `https://api.` | `https://.` |
## Authentication
[Section titled “Authentication”](#authentication)
All endpoints except `/health` and `/_shell/...` require a Bearer token in the `Authorization` header:
```http
Authorization: Bearer bht_abc123def456...
```
Tokens come from [`bhatti user create`](/docs/reference/cli/admin/user-create/). The CLI loads them from `~/.bhatti/config.yaml`.
**WebSocket endpoints use the same Bearer header.** There is no `?token=…` query-param auth — leaving the token out of URLs eliminates accidental logging in proxy access logs.
The web shell is the exception: the URL contains `#token=…` in the **fragment** (never sent to the server), and the embedded page extracts the token client-side and uses it as a Bearer header on the WebSocket connection.
## Response shape
[Section titled “Response shape”](#response-shape)
JSON for every endpoint. Errors use `{"error": "...", "request_id": "..."}`:
```json
{"error": "sandbox not found", "request_id": "req_a1b2c3d4"}
```
`request_id` is included on every error response; quote it when reporting bugs.
## Status codes
[Section titled “Status codes”](#status-codes)
| Code | Meaning |
| ----- | ----------------------------------------------------------------------------------------------- |
| `200` | OK. |
| `201` | Created. Used for resource-creation endpoints. |
| `202` | Accepted. Used for async tasks (image pull) — body contains a `task_id`. |
| `204` | No content. Used by `unpublish`. |
| `400` | Validation error. Body has `{"error": "..."}`. |
| `401` | Missing or invalid Bearer token. |
| `403` | Forbidden — typically a per-sandbox cap (CPU, memory) being exceeded. |
| `404` | Resource not found. |
| `409` | Conflict — name already exists, or volume is attached when an op needs it detached. |
| `413` | Request too large (file write > 100 MB). |
| `429` | Rate-limited or quota exceeded (`max-sandboxes`, `max-cpus`, etc). |
| `500` | Internal server error. |
| `501` | Endpoint exists but the server isn’t configured for it (e.g. backups without an `s3_*` config). |
| `502` | Bad gateway from the reverse proxy — the sandbox process isn’t listening on the requested port. |
## Response headers
[Section titled “Response headers”](#response-headers)
The server stamps every response with:
* `X-Bhatti-Version` — server version. CLI clients use this to detect server upgrades.
* `X-Bhatti-Min-CLI` — the minimum CLI version the server requires. Clients older than this should upgrade.
* `X-Bhatti-Existing: true` — set on `POST /sandboxes` when the request was idempotent (a sandbox with that name already existed).
## Async tasks
[Section titled “Async tasks”](#async-tasks)
Long-running operations like `POST /images/pull` return immediately with `202 Accepted` and a body like:
```json
{"task_id": "tsk_abc123", "status": "running"}
```
Poll `GET /tasks/:id` until `status` is `completed` or `failed`. The `progress` field, when present, contains a human-readable progress string the CLI can display.
***
# System
[Section titled “System”](#system)
## Health check
[Section titled “Health check”](#health-check)
```http
GET /health
```
No auth required. Lightweight check.
```bash
curl http://localhost:8080/health
```
```json
{"status": "ok", "uptime": "2h15m30s"}
```
***
# Sandboxes
[Section titled “Sandboxes”](#sandboxes)
## List sandboxes
[Section titled “List sandboxes”](#list-sandboxes)
```http
GET /sandboxes
```
Returns sandboxes owned by the authenticated user, enriched with thermal state and any published URLs. Listing does **not** wake cold sandboxes.
```bash
curl http://localhost:8080/sandboxes \
-H "Authorization: Bearer $TOKEN"
```
```json
[
{
"id": "a1b2c3d4",
"name": "dev",
"status": "running",
"ip": "",
"cpus": 2,
"memory_mb": 1024,
"image": "minimal",
"created_at": "2026-04-29T15:21:48Z",
"urls": ["https://dev-k3m9x2.bhatti.sh"]
}
]
```
## Create a sandbox
[Section titled “Create a sandbox”](#create-a-sandbox)
```http
POST /sandboxes
```
```bash
curl -X POST http://localhost:8080/sandboxes \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "dev",
"cpus": 2,
"memory_mb": 1024,
"env": {"NODE_ENV": "production"},
"init": "cd /workspace && npm install",
"image": "minimal",
"persistent_volumes": [
{"name": "workspace", "mount": "/workspace", "auto_create": false}
],
"secrets": ["OPENAI_KEY"],
"files": [
{"guest_path": "/etc/app/config.json", "content": "", "mode": "0644"}
]
}'
```
**Request fields:**
| Field | Type | Default | Notes |
| -------------------- | ------ | ---------- | ------------------------------------------------------------------------------------- |
| `name` | string | auto | Pattern: `[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}`. |
| `cpus` | float | `1` | Fractional values allowed. Capped at user’s `max_cpus_per_sandbox`. |
| `memory_mb` | int | `1024` | Capped at user’s `max_memory_mb_per_sandbox`. |
| `disk_size_mb` | int | image size | Resize the rootfs at create time. |
| `env` | object | — | Env vars baked into the config drive. |
| `init` | string | — | Boot-time script. Runs as a session named `init`. |
| `keep_hot` | bool | `false` | Disables the thermal manager for this sandbox. |
| `hugepages` | bool | `false` | 2 MB hugepages. Faster boot. |
| `image` | string | `minimal` | Image name from `GET /images`. |
| `template_id` | string | — | Create from a template. |
| `persistent_volumes` | array | — | `[{name, mount, auto_create, read_only, size_mb}]`. |
| `secrets` | array | — | Names of stored secrets to inject as env vars. **Ignored when `template_id` is set.** |
| `files` | array | — | `[{guest_path, content (base64), mode}]`. **Ignored when `template_id` is set.** |
| `new_volumes` | array | — | **Legacy** equivalent of `persistent_volumes`. The CLI no longer sends these. |
| `volumes` | array | — | **Legacy** equivalent. |
**Responses:**
* `201 Created` with the sandbox object.
* `200 OK` + `X-Bhatti-Existing: true` if a non-destroyed sandbox with that name already exists. Idempotent — safe to retry from scripts.
* `400` for invalid name / size / etc.
* `403` if `cpus` or `memory_mb` exceed the user’s per-sandbox cap.
* `429` if the user is at their `max-sandboxes` cap.
## Get a sandbox
[Section titled “Get a sandbox”](#get-a-sandbox)
```http
GET /sandboxes/:id
```
```bash
curl http://localhost:8080/sandboxes/dev \
-H "Authorization: Bearer $TOKEN"
```
Returns the full sandbox record. Both name and ID work as `:id`.
## Update a sandbox
[Section titled “Update a sandbox”](#update-a-sandbox)
```http
PATCH /sandboxes/:id
```
Currently only `keep_hot` is mutable. All other fields (cpus, memory, image) are immutable after creation — destroy and recreate.
```bash
curl -X PATCH http://localhost:8080/sandboxes/dev \
-H "Authorization: Bearer $TOKEN" \
-d '{"keep_hot": true}'
```
Setting `keep_hot: true` on a stopped sandbox immediately wakes it.
## Destroy a sandbox
[Section titled “Destroy a sandbox”](#destroy-a-sandbox)
```http
DELETE /sandboxes/:id
```
```bash
curl -X DELETE http://localhost:8080/sandboxes/dev \
-H "Authorization: Bearer $TOKEN"
```
```json
{"status": "destroyed"}
```
Persistent volumes attached to the sandbox are detached (not deleted). Published URLs are cleaned up.
## Stop a sandbox
[Section titled “Stop a sandbox”](#stop-a-sandbox)
```http
POST /sandboxes/:id/stop
```
Snapshot to disk and free memory. Resume with `POST /sandboxes/:id/start`. Stop takes a full memory snapshot; there are no diff snapshots.
```bash
curl -X POST http://localhost:8080/sandboxes/dev/stop \
-H "Authorization: Bearer $TOKEN"
```
## Start a sandbox
[Section titled “Start a sandbox”](#start-a-sandbox)
```http
POST /sandboxes/:id/start
```
Restore from snapshot.
```bash
curl -X POST http://localhost:8080/sandboxes/dev/start \
-H "Authorization: Bearer $TOKEN" \
-d '{"force": true}'
```
`force: true` retries even if the sandbox is in `unknown` state from a previously failed restore.
***
# Execution & shells
[Section titled “Execution & shells”](#execution--shells)
## Run a command
[Section titled “Run a command”](#run-a-command)
```http
POST /sandboxes/:id/exec
```
```bash
curl -X POST http://localhost:8080/sandboxes/dev/exec \
-H "Authorization: Bearer $TOKEN" \
-d '{"cmd": ["echo", "hello"]}'
```
```json
{"exit_code": 0, "stdout": "hello\n", "stderr": ""}
```
**Request fields:**
| Field | Type | Default | Notes |
| ------------- | ------ | ---------- | -------------------------------------------------------------------------- |
| `cmd` | array | (required) | Command and arguments. |
| `timeout_sec` | int | `300` | Max `86400` (24h). |
| `detach` | bool | `false` | Fire-and-forget. Returns `{pid, output_file, detached: true}` immediately. |
| `output_file` | string | auto | When `detach: true`, the guest path to redirect output to. |
This endpoint does **not** accept an `env` field. Env vars are baked into the sandbox’s config drive at creation time (`--env`, `--secret`). To pass per-call env vars, use the WebSocket exec path below.
### Streaming output
[Section titled “Streaming output”](#streaming-output)
Set `Accept: application/x-ndjson`:
```bash
curl -N -X POST http://localhost:8080/sandboxes/dev/exec \
-H "Authorization: Bearer $TOKEN" \
-H "Accept: application/x-ndjson" \
-d '{"cmd":["npm","install"]}'
```
```json
{"type":"stdout","data":"Installing dependencies...\n"}
{"type":"stderr","data":"npm warn deprecated...\n"}
{"type":"exit","exit_code":0}
```
Each line is flushed immediately. `type` is `stdout`, `stderr`, `exit`, or `error`.
Cold sandboxes wake automatically before executing.
## Interactive shell (WebSocket)
[Section titled “Interactive shell (WebSocket)”](#interactive-shell-websocket)
```http
GET /sandboxes/:id/ws
```
Upgrade to WebSocket. Authentication via `Authorization: Bearer ...` header on the upgrade request.
| Query param | Description |
| -------------- | ------------------------------------------------------ |
| `session=` | Reattach to an existing session by ID. |
| `new=true` | Force a new session even if one is currently attached. |
**Wire protocol:**
* **Server → client (binary):** raw terminal output.
* **Server → client (text):** initial `{"type":"session","session_id":"s1"}` message; thereafter only on session events.
* **Client → server (binary):** keystrokes (forwarded to the PTY).
* **Client → server (text JSON):** `{"type":"resize","rows":N,"cols":N}` or `{"cmd":[...], "env":{}, "max_idle_sec":N}` (first message of a non-attached session — this is where per-call env injection lives).
Detach by closing the connection. The session keeps running.
## List sessions
[Section titled “List sessions”](#list-sessions)
```http
GET /sandboxes/:id/sessions
```
```bash
curl http://localhost:8080/sandboxes/dev/sessions \
-H "Authorization: Bearer $TOKEN"
```
```json
[
{"session_id": "init", "argv": "npm install", "running": true, "attached": false, "created_at": 1761792000},
{"session_id": "s1", "argv": "/bin/zsh -li", "running": true, "attached": true, "created_at": 1761792100}
]
```
## List listening ports (one sandbox)
[Section titled “List listening ports (one sandbox)”](#list-listening-ports-one-sandbox)
```http
GET /sandboxes/:id/ports
```
```bash
curl http://localhost:8080/sandboxes/dev/ports \
-H "Authorization: Bearer $TOKEN"
```
```json
[
{"sandbox_id": "a1b2c3d4", "container_port": 3000, "proxy_url": "/sandboxes/a1b2c3d4/proxy/3000/"}
]
```
The `proxy_url` is path-relative; prepend the API base URL for a full reachable URL.
## List listening ports (all sandboxes)
[Section titled “List listening ports (all sandboxes)”](#list-listening-ports-all-sandboxes)
```http
GET /ports
```
Same row shape as above; covers every sandbox the authenticated user owns.
***
# Files
[Section titled “Files”](#files)
All file operations target the same path with query parameters.
## Read a file
[Section titled “Read a file”](#read-a-file)
```http
GET /sandboxes/:id/files?path=/path
```
Returns raw file content with `Content-Type: application/octet-stream`. The `X-File-Size` header always carries the total size (so clients can detect truncation).
```bash
curl "http://localhost:8080/sandboxes/dev/files?path=/workspace/app.js" \
-H "Authorization: Bearer $TOKEN" \
-o app.js
```
**Server-side truncation:**
```bash
curl "http://localhost:8080/sandboxes/dev/files?path=/var/log/agent.log&offset=1&limit=2000&max_bytes=51200" \
-H "Authorization: Bearer $TOKEN"
```
| Param | Type | Description |
| ----------- | ------ | ------------------------------------ |
| `path` | string | Absolute path inside the sandbox. |
| `offset` | int | 1-indexed line number to start from. |
| `limit` | int | Max lines to return. |
| `max_bytes` | int | Max bytes to return. |
Whichever limit hits first stops the read.
## List a directory
[Section titled “List a directory”](#list-a-directory)
```http
GET /sandboxes/:id/files?path=/dir&ls=true
```
```bash
curl "http://localhost:8080/sandboxes/dev/files?path=/workspace&ls=true" \
-H "Authorization: Bearer $TOKEN"
```
```json
[
{"name": "app.js", "size": 1234, "mode": "0644", "is_dir": false, "mtime": 1761792345},
{"name": "node_modules", "size": 4096, "mode": "0755", "is_dir": true, "mtime": 1761792500}
]
```
Capped at 10 000 entries; a sentinel row indicates truncation if hit.
## Stat a file
[Section titled “Stat a file”](#stat-a-file)
```http
HEAD /sandboxes/:id/files?path=/path
```
```bash
curl -I "http://localhost:8080/sandboxes/dev/files?path=/workspace/app.js" \
-H "Authorization: Bearer $TOKEN"
```
Response headers:
| Header | Description |
| -------------- | ------------------------ |
| `X-File-Size` | Size in bytes. |
| `X-File-Mode` | Octal mode, e.g. `0644`. |
| `X-File-IsDir` | `true` or `false`. |
## Write a file
[Section titled “Write a file”](#write-a-file)
```http
PUT /sandboxes/:id/files?path=/path[&mode=0644]
```
Body is the raw content. `Content-Length` is required (chunked transfer-encoding is rejected).
```bash
curl -X PUT "http://localhost:8080/sandboxes/dev/files?path=/workspace/app.js&mode=0644" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Length: 25" \
--data-binary 'console.log("hello world")'
```
```json
{"status": "ok"}
```
Writes are atomic (temp file + fsync + rename). 100 MB hard cap.
***
# Networking
[Section titled “Networking”](#networking)
## Reverse proxy
[Section titled “Reverse proxy”](#reverse-proxy)
```http
ANY /sandboxes/:id/proxy/:port/*path
```
Tunnels HTTP and WebSocket through the engine into the sandbox. The path after `/proxy//` is forwarded verbatim to `localhost:` inside the VM. Bearer auth required.
Useful for development. For public, auth-free URLs, use [`publish`](#publish-a-port).
```bash
curl http://localhost:8080/sandboxes/dev/proxy/3000/api/users \
-H "Authorization: Bearer $TOKEN"
```
`502` is returned if no process is listening on the port.
## Publish a port
[Section titled “Publish a port”](#publish-a-port)
```http
POST /sandboxes/:id/publish
```
```bash
curl -X POST http://localhost:8080/sandboxes/dev/publish \
-H "Authorization: Bearer $TOKEN" \
-d '{"port": 3000, "alias": "my-app"}'
```
```json
{
"id": "pub_a1b2c3d4",
"sandbox_id": "a1b2c3d4",
"port": 3000,
"alias": "my-app",
"url": "https://my-app.bhatti.sh",
"created_at": "2026-04-30T17:00:00Z"
}
```
If `alias` is omitted, an alias is auto-generated as `-<6-hex>`. `409` is returned on alias collision.
The `url` value depends on the server config:
| Server config | URL format |
| --------------------------------- | -------------------------------------------------------------------------------------------- |
| `domain.proxy_zone: bhatti.sh` | `https://my-app.bhatti.sh` |
| `public_proxy_listen: :8443` only | `http://:8443/my-app/` |
| Neither | `(no public proxy configured) alias: my-app` — the CLI surfaces a hint to use the proxy URL. |
## List published ports
[Section titled “List published ports”](#list-published-ports)
```http
GET /sandboxes/:id/publish
```
Returns an array of publish rules with their generated URLs.
## Unpublish a port
[Section titled “Unpublish a port”](#unpublish-a-port)
```http
DELETE /sandboxes/:id/publish/:port
```
```bash
curl -X DELETE http://localhost:8080/sandboxes/dev/publish/3000 \
-H "Authorization: Bearer $TOKEN"
```
`204 No Content`.
## Mint a shell token
[Section titled “Mint a shell token”](#mint-a-shell-token)
```http
POST /sandboxes/:id/shell-token
```
Generates a new web-shell token bound to this sandbox. Each call invalidates the previous token — only one is active at a time.
```bash
curl -X POST http://localhost:8080/sandboxes/dev/shell-token \
-H "Authorization: Bearer $TOKEN"
```
```json
{
"url": "https://api.bhatti.sh/_shell/a1b2c3d4#token=k3m9x2qr...",
"token": "k3m9x2qr..."
}
```
The token is in the URL **fragment** (`#token=...`); fragments are not sent to the server. The `_shell` page extracts the token client-side and uses it as a Bearer header on the WebSocket.
## Revoke a shell token
[Section titled “Revoke a shell token”](#revoke-a-shell-token)
```http
DELETE /sandboxes/:id/shell-token
```
The previously-active token (if any) is invalidated immediately.
***
# Templates
[Section titled “Templates”](#templates)
Templates are reusable sandbox blueprints. The CLI doesn’t expose them yet; reach for the API directly.
```http
GET /templates
POST /templates
GET /templates/:id
DELETE /templates/:id
```
```bash
curl -X POST http://localhost:8080/templates \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "node-dev",
"cpus": 2,
"memory_mb": 1024,
"image": "minimal",
"env": {"NODE_ENV": "development"},
"user_data": "cd /workspace && npm install",
"secrets": ["OPENAI_KEY"]
}'
```
Use `template_id` in `POST /sandboxes` to create a sandbox from a template. Template fields are defaults; per-request fields override them. **Note:** the template path silently drops request-side `secrets` and `files` arrays — set them on the template instead, or don’t use a template.
***
# Secrets
[Section titled “Secrets”](#secrets)
Encrypted at rest with [age](https://age-encryption.org/), scoped per user.
## List secrets
[Section titled “List secrets”](#list-secrets)
```http
GET /secrets
```
Returns an array of `{name, created_at, updated_at}`. Values are never returned.
## Create or update a secret
[Section titled “Create or update a secret”](#create-or-update-a-secret)
```http
POST /secrets
```
```bash
curl -X POST http://localhost:8080/secrets \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "OPENAI_KEY", "value": "sk-..."}'
```
Re-posting with an existing name updates the value silently.
## Delete a secret
[Section titled “Delete a secret”](#delete-a-secret)
```http
DELETE /secrets/:name
```
```bash
curl -X DELETE http://localhost:8080/secrets/OPENAI_KEY \
-H "Authorization: Bearer $TOKEN"
```
***
# Volumes
[Section titled “Volumes”](#volumes)
Persistent ext4 volumes. Several operations require the volume to be detached (no active sandbox attachments) and return `409` otherwise: `delete`, `resize`, `clone` (snapshot), `restore`.
## List volumes
[Section titled “List volumes”](#list-volumes)
```http
GET /volumes
```
Returns an array of volume records including current attachments.
## Create a volume
[Section titled “Create a volume”](#create-a-volume)
```http
POST /volumes
```
```bash
curl -X POST http://localhost:8080/volumes \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "workspace", "size_mb": 5120}'
```
`429` if the user is over their storage quota.
## Get a volume
[Section titled “Get a volume”](#get-a-volume)
```http
GET /volumes/:name
```
## Delete a volume
[Section titled “Delete a volume”](#delete-a-volume)
```http
DELETE /volumes/:name
```
`409` if attached.
## Resize a volume
[Section titled “Resize a volume”](#resize-a-volume)
```http
POST /volumes/:name/resize
```
```bash
curl -X POST http://localhost:8080/volumes/workspace/resize \
-H "Authorization: Bearer $TOKEN" \
-d '{"size_mb": 10240}'
```
`new_size > current_size` only — shrinking is not supported. `409` if attached.
## Clone a volume
[Section titled “Clone a volume”](#clone-a-volume)
```http
POST /volumes/:name/snapshot
```
Independent point-in-time copy. Source must be detached.
```bash
curl -X POST http://localhost:8080/volumes/workspace/snapshot \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "workspace-pre-upgrade"}'
```
***
# Backups
[Section titled “Backups”](#backups)
Volume backups to S3-compatible storage. Requires a `backup` block in the server config; otherwise every endpoint returns `501`.
## List backups for a volume
[Section titled “List backups for a volume”](#list-backups-for-a-volume)
```http
GET /volumes/:name/backups
```
## Trigger a backup
[Section titled “Trigger a backup”](#trigger-a-backup)
```http
POST /volumes/:name/backups
```
Compresses (zstd) and uploads. Does not require the volume to be detached, but writes in flight at backup time may produce an inconsistent snapshot — stop the sandbox first if consistency matters.
## Restore from a backup
[Section titled “Restore from a backup”](#restore-from-a-backup)
```http
POST /volumes/:name/backups/restore
```
```bash
curl -X POST http://localhost:8080/volumes/workspace/backups/restore \
-H "Authorization: Bearer $TOKEN" \
-d '{"backup_id": "bk_a1b2c3d4"}'
```
`409` if attached.
## Delete a backup
[Section titled “Delete a backup”](#delete-a-backup)
```http
DELETE /volumes/:name/backups/:backup_id
```
***
# Images
[Section titled “Images”](#images)
## List images
[Section titled “List images”](#list-images)
```http
GET /images
```
Returns user-owned, system tier, and shared images.
## Get an image
[Section titled “Get an image”](#get-an-image)
```http
GET /images/:name
```
## Delete an image
[Section titled “Delete an image”](#delete-an-image)
```http
DELETE /images/:name
```
System tier images can’t be deleted via this endpoint.
## Pull an OCI image
[Section titled “Pull an OCI image”](#pull-an-oci-image)
```http
POST /images/pull
```
Async — returns `202` with a task ID. Poll `/tasks/` until it completes.
```bash
curl -X POST http://localhost:8080/images/pull \
-H "Authorization: Bearer $TOKEN" \
-d '{"ref": "python:3.12", "name": "python-3.12"}'
```
```json
{"task_id": "tsk_abc123", "status": "running"}
```
Optional `auth: "user:token"` for simple HTTP basic auth. For real private registries with rotating credentials, prefer `image import` after a local `docker pull`.
If the same image was already pulled with the same content digest, returns `200` immediately with the existing record.
## Import an image
[Section titled “Import an image”](#import-an-image)
```http
POST /images/import?name=
```
Body is a `docker save`-style tarball streamed as `Content-Type: application/x-tar`. The CLI uses this for `bhatti image import`.
```bash
docker save python:3.12 | curl -X POST \
"http://localhost:8080/images/import?name=python-3.12" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/x-tar" \
--data-binary @-
```
```json
{"name": "python-3.12", "size_mb": 284}
```
## Save a sandbox as an image
[Section titled “Save a sandbox as an image”](#save-a-sandbox-as-an-image)
```http
POST /sandboxes/:id/save-image
```
```bash
curl -X POST http://localhost:8080/sandboxes/dev/save-image \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "my-stack"}'
```
The sandbox keeps running. Only the rootfs is captured; persistent volumes are not part of the image.
***
# Snapshots
[Section titled “Snapshots”](#snapshots)
Named VM snapshots — memory + CPU + disk + processes.
## List snapshots
[Section titled “List snapshots”](#list-snapshots)
```http
GET /snapshots
```
## Get a snapshot
[Section titled “Get a snapshot”](#get-a-snapshot)
```http
GET /snapshots/:name
```
## Delete a snapshot
[Section titled “Delete a snapshot”](#delete-a-snapshot)
```http
DELETE /snapshots/:name
```
## Create a snapshot
[Section titled “Create a snapshot”](#create-a-snapshot)
```http
POST /sandboxes/:id/checkpoint
```
```bash
curl -X POST http://localhost:8080/sandboxes/dev/checkpoint \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "pre-experiment"}'
```
The source sandbox keeps running.
## Resume from a snapshot
[Section titled “Resume from a snapshot”](#resume-from-a-snapshot)
```http
POST /snapshots/:name/resume
```
Creates a **new** sandbox from the snapshot.
```bash
curl -X POST http://localhost:8080/snapshots/pre-experiment/resume \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "dev-restored"}'
```
`name` is optional — auto-generated from the source sandbox name if omitted. Persistent volumes are not re-attached.
***
# Tasks
[Section titled “Tasks”](#tasks)
Async operations get a task ID; poll its status here.
## Get task status
[Section titled “Get task status”](#get-task-status)
```http
GET /tasks/:id
```
```json
{
"id": "tsk_abc123",
"status": "running",
"progress": "downloading layers (45 MB / 142 MB)",
"error": "",
"result": ""
}
```
`status` is `running`, `completed`, or `failed`.
## Cancel a task
[Section titled “Cancel a task”](#cancel-a-task)
```http
DELETE /tasks/:id
```
Best-effort cancel. Some operations are uncancellable past a certain point (e.g. a finalizing image conversion).
***
# Image sharing
[Section titled “Image sharing”](#image-sharing)
There’s no HTTP API for image sharing. It’s a local SQLite operation done with [`bhatti image share --user `](/docs/reference/cli/images/share/) on the server itself, using `--data-dir` to point at the data directory. This is an intentional restriction — image visibility is controlled at the database level, not via the API.
# Configuration
> Server and CLI config files, layered loading, environment variables, and the data directory layout.
bhatti has a single YAML config file. The same format is used by the daemon and the CLI client; the daemon ignores the client-only fields, the client ignores the daemon-only ones.
## Layered loading
[Section titled “Layered loading”](#layered-loading)
The config is loaded from multiple paths and merged. Higher rows override lower rows:
| Source | Loaded? | Used for |
| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------ |
| `$BHATTI_CONFIG` (path) | If set, **only** this is loaded — no merging. | Override everything; useful for tests and running a second daemon. |
| `/etc/bhatti/config.yaml` | First match wins. | Server settings — engine, runtime paths, listen address, data dir. |
| `~/.bhatti/config.yaml` | Always — fills in only `api_url` and `auth_token` if still empty. | CLI client credentials. |
This means a developer machine that’s also running a server can keep the server config in `/etc/bhatti/config.yaml` (no client credentials) and the CLI’s API key in `~/.bhatti/config.yaml` (without disturbing the server config). Both files are read, neither overwrites the other.
The deprecated location `/var/lib/bhatti/config.yaml` is still honoured as a fallback; the daemon prints a stderr warning when it loads from there. Migrate to `/etc/bhatti/config.yaml`.
## Environment variables
[Section titled “Environment variables”](#environment-variables)
| Variable | Used by | Description |
| --------------------- | ------- | --------------------------------------------------------------------- |
| `BHATTI_CONFIG` | both | Override the config file path. When set, layered loading is bypassed. |
| `BHATTI_LOG_LEVEL` | server | `debug`, `info` (default), `warn`, `error`. |
| `BHATTI_URL` | CLI | API endpoint. |
| `BHATTI_TOKEN` | CLI | API key. |
| `BHATTI_FORCE_STREAM` | CLI | Force NDJSON streaming output even when stdout isn’t a TTY. |
The CLI’s endpoint precedence is: `--flag` → env var (`BHATTI_URL`/`BHATTI_TOKEN`) → config file → the daemon’s local unix socket → built-in default. Env overrides the config file (12-factor: point an already-configured CLI at another daemon without editing `~/.bhatti/config.yaml`). When neither an endpoint flag nor `api_url`/`BHATTI_URL` is set, the CLI talks to the daemon’s local control socket (`/api.sock`), which is not reachable from inside a sandbox.
## Server config
[Section titled “Server config”](#server-config)
bhatti v2 runs the **krucible** engine (libkrun). The daemon itself is pure Go; it spawns a per-sandbox `bhatti-vmm` helper (the VMM) and a per-owner `bhatti-netd` gateway, both shipped in the release bundle under `/runtime/`. A typical server config (the one written by `curl -fsSL bhatti.sh/install | sudo bash`) looks like:
```yaml
engine: krucible
listen: :8080
data_dir: /var/lib/bhatti
# Secure per-owner network gateway (bhatti-netd) is ON by default: the guest is
# isolated from the host, egress is policed, same-owner siblings are reachable.
# Set 'krucible_net_backend: false' for the legacy shared-netstack (TSI) path.
krucible_vmm: /var/lib/bhatti/runtime/bin/bhatti-vmm
krucible_netd: /var/lib/bhatti/runtime/bin/bhatti-netd
krucible_libdir: /var/lib/bhatti/runtime/lib
krucible_kernel_image: /var/lib/bhatti/runtime/kernel/Image-lean-6.12.94-aarch64
krucible_base_image: /var/lib/bhatti/images/rootfs-minimal-arm64.ext4
```
### Top-level fields
[Section titled “Top-level fields”](#top-level-fields)
| Field | Default | Description |
| ----------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `engine` | `krucible` | Engine backend. v2 is `krucible` (libkrun). (v1 Firecracker lives on the `firecracker` branch.) |
| `listen` | `:8080` | Address for the TCP control API. Empty = unix socket only. |
| `api_socket` | `/api.sock` | Local control-API unix socket. Never reachable from a sandbox. |
| `data_dir` | `~/.bhatti` (CLI) or whatever the install script writes (typically `/var/lib/bhatti` on a server) | Root directory for state — DB, runtime bundle, images, sandboxes, volumes, snapshots. |
| `krucible_vmm` | next to the binary / `PATH` | Path to the `bhatti-vmm` helper (the per-sandbox VMM). |
| `krucible_netd` | next to the binary / `PATH` | Path to the `bhatti-netd` network gateway helper. |
| `krucible_libdir` | autodetect | Directory holding `libkrun` (and, on the non-lean path, `libkrunfw`). The install lays this under `runtime/lib`. |
| `krucible_kernel_image` | autodetect | The lean external guest kernel (`Image-lean-*` on arm64, `vmlinux-lean-*` on x86\_64). \~2× faster cold-start; boots the block-root path. Empty falls back to the bundled libkrunfw kernel. |
| `krucible_base_image` | — | Prebuilt ext4 base rootfs used when `bhatti create` is called without `--image`. Each sandbox gets a copy-on-write overlay over it. |
| `krucible_block_root` | implied when `krucible_base_image` is set | Boot from a CoW block image (required for the cold tier). |
| `krucible_net_backend` | `true` | Per-owner `bhatti-netd` gateway (policed egress, host isolation, sibling reachability). `false` = legacy TSI (guest can reach the host loopback — not recommended). |
| `krucible_socket_dir` | `/tmp/bhatti-kr` | Short directory for vsock/control unix sockets (paths must stay short — `sun_path` caps at 104/108 bytes). |
| `public_proxy_listen` | — | When set (e.g. `:8443`), the daemon exposes a path-based public proxy at this address. Skip this and use [domain mode](#domain-mode) for production. |
| `api_url` | — | CLI-only field. Put it in `~/.bhatti/config.yaml`, not `/etc/bhatti/config.yaml`. |
| `auth_token` | — | CLI-only. Same. |
| `domain` | — | Optional. Enables [domain mode](#domain-mode) — host-based routing + TLS. |
| `backup` | — | Optional. Enables [volume backups](#backup) to S3-compatible storage. |
There is no host TAP/bridge/iptables configuration and no jailer in v2 — krucible owns networking (via `bhatti-netd`/TSI) and boots the VMM as an ordinary host process. (The FC-era `firecracker_*` and `jail_*` fields belong to v1.)
### Domain mode
[Section titled “Domain mode”](#domain-mode)
For host-based routing with TLS — `https://api.` for the API, `https://.` for published sandboxes.
```yaml
domain:
api_host: api.bhatti.sh
proxy_zone: bhatti.sh
tls_cert: /etc/bhatti/wildcard.pem
tls_key: /etc/bhatti/wildcard-key.pem
```
| Field | Description |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `api_host` | Hostname for the API. Requests to this host go through normal Bearer auth. |
| `proxy_zone` | Zone for published sandboxes. `.` is the URL `bhatti publish` generates. |
| `tls_cert`, `tls_key` | Paths to a wildcard cert covering `*.` and ``. **Recommended.** |
| `acme_email` | Fallback: per-alias Let’s Encrypt certificates. **Rate-limited** to 50 new aliases per registered domain per week — fine for stable subdomains, fast to hit if you’re stamping out preview environments. |
When domain mode is on, the daemon listens on `:443` (TLS, both API and proxy by Host header), `:80` (ACME challenges + HTTPS redirect), and the local control socket for internal health checks. HTTP/2 is disabled on the TLS listener so WebSocket upgrades work behind a proxy.
You must set either `tls_cert`+`tls_key` or `acme_email`. A wildcard cert is the right answer for any setup that creates more than a handful of aliases per week. See [Custom domain](/docs/managing/custom-domain/) for the full walk-through.
### Backup
[Section titled “Backup”](#backup)
For volume backups to S3-compatible storage. Enables [`bhatti volume backup`](/docs/reference/cli/volumes/backup/), `restore`, `backup-list`, and `backup-delete`. Without this block, those endpoints return `501`.
```yaml
backup:
s3_endpoint: https://s3.eu-central-003.backblazeb2.com
s3_region: eu-central-003
s3_bucket: bhatti-backups
s3_access_key: ...
s3_secret_key: ...
schedule:
- volume: workspace
cron: "0 3 * * *"
retention: 7
```
| Field | Description |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `s3_endpoint` | S3-compatible endpoint URL. Backblaze B2, AWS S3, MinIO, R2, etc. |
| `s3_region` | Region. Required by AWS-compatible APIs even when irrelevant. |
| `s3_bucket` | Bucket name. Must already exist. |
| `s3_access_key`, `s3_secret_key` | Credentials with read/write/delete on the bucket. |
| `schedule` | Optional. Array of automatic backup schedules. Each entry has `volume`, `cron` (5-field cron expression), and `retention` (keep last N backups for that volume). |
Schedules run inside the daemon — no external cron required. Retention is enforced after each scheduled backup.
## CLI client config
[Section titled “CLI client config”](#cli-client-config)
Lives at `~/.bhatti/config.yaml`. `bhatti setup` writes it for you.
```yaml
api_url: https://api.bhatti.sh
auth_token: bht_abc123def456...
```
| Field | Description |
| ------------ | --------------------------------------------- |
| `api_url` | bhatti API endpoint. |
| `auth_token` | The user’s API key from `bhatti user create`. |
## Data directory layout
[Section titled “Data directory layout”](#data-directory-layout)
```plaintext
data_dir/
├── state.db SQLite database (WAL mode)
├── age.key Secret-encryption key (auto-generated on first secret set)
├── id_ed25519, id_ed25519.pub SSH keypair (auto-generated on first start; guest agent identity)
├── api.sock Local control-API unix socket
├── .latest-version Cache for `bhatti version`'s GitHub-release check (~/.bhatti only)
├── runtime/ Relocatable v2 runtime bundle (re-laid on every install/update)
│ ├── bin/ bhatti-vmm, bhatti-netd
│ ├── lib/ libkrun (the krucible VMM shared lib) + symlinks
│ └── kernel/ Image-lean-- (arm64) / vmlinux-lean-- (x86_64)
├── images/
│ └── rootfs--.ext4 Base rootfs images (CoW-overlaid per sandbox)
├── sandboxes//
│ ├── qcow2 overlay over the base image (the sandbox's disk deltas)
│ ├── config drive env, secrets, files, per-sandbox token (~1 MB ext4, /dev/vdb)
│ ├── state.json recovery metadata (helper PID, sockets, bundle ref)
│ └── (when cold) memory image + VM state + attached volumes + manifest.json
├── volumes// Standalone persistent volumes
└── snapshots/// Named-snapshot bundles (self-contained)
```
`` is `arm64` on aarch64 hosts, `amd64` on x86\_64. There are no `jails/` or `firecracker.sock` entries in v2 — those are v1/Firecracker.
## Validating a config
[Section titled “Validating a config”](#validating-a-config)
The daemon prints which file it loaded on startup:
```plaintext
config loaded path=/etc/bhatti/config.yaml
```
If the path is empty, no config file was found and built-in defaults are used. The CLI client doesn’t print this; check with `bhatti version` (which reports the loaded endpoint).
# Documentation
> Redirecting to Quickstart...
# Adding a Tier
> How to create a new rootfs tier for bhatti.
Tiers are pre-built ext4 root filesystem images. Each tier builds on `minimal` and adds specific tooling. The server auto-discovers tiers at startup by globbing `rootfs-*-{arch}.ext4` in the images directory.
## Steps
[Section titled “Steps”](#steps)
### 1. Create the tier script
[Section titled “1. Create the tier script”](#1-create-the-tier-script)
Add `scripts/tiers/.sh`. This runs inside a chroot during the build. It receives `$MOUNT`, `$ARCH`, `$DEB_ARCH`, `$AGENT`, and `$SCRIPT_DIR` as environment variables.
Most tiers source minimal first:
```bash
#!/bin/bash
set -euo pipefail
"$SCRIPT_DIR/tiers/minimal.sh"
chroot "$MOUNT" /bin/bash -c '
apt-get update -qq
apt-get install -y --no-install-recommends your-packages
apt-get clean
rm -rf /var/lib/apt/lists/*
'
```
### 2. Long-running daemons: ship them as systemd units
[Section titled “2. Long-running daemons: ship them as systemd units”](#2-long-running-daemons-ship-them-as-systemd-units)
If your tier needs a daemon running on boot — a database, a headless browser, a VNC gateway — ship it as a `.service` file and let lohar’s [`systemctl` shim](/docs/under-the-hood/lohar-the-blacksmith/) manage it. Don’t write `init.sh` (the legacy pattern); don’t fork the daemon yourself; don’t try to manage PIDs, restarts, or cgroups.
The canonical short example is `headless-chrome.service` from the browser tier:
```ini
[Unit]
Description=Headless Chromium with Chrome DevTools Protocol
After=network.target
[Service]
Type=simple
# Defaults; user-supplied values in /run/bhatti/config-env override.
Environment=CHROME_REMOTE_PORT=9222
Environment=CHROME_FLAGS=
EnvironmentFile=-/run/bhatti/config-env
ExecStart=/usr/local/bin/headless_shell --no-sandbox --disable-gpu \
--disable-dev-shm-usage \
--remote-debugging-port=${CHROME_REMOTE_PORT} \
--remote-debugging-address=0.0.0.0 \
$CHROME_FLAGS
Restart=on-failure
RestartSec=2s
[Install]
WantedBy=multi-user.target
```
The pattern:
* **`Type=simple`** for daemons that stay in the foreground (the common case). **`Type=notify`** for daemons that speak `sd_notify(3)` and signal `READY=1` once their socket is up (`dockerd`, `postgresql`). **`Type=oneshot`** with `RemainAfterExit=yes` for first-boot setup helpers (password generation, env-file writers).
* **`ExecStart=`** points at the binary directly. No `/bin/sh -c` wrapper. The shim already wraps your command in `/bin/sh -c "exec "` for variable expansion; nesting another shell is redundant at best.
* **`Restart=on-failure`** + **`RestartSec=2s`** is the right default for long-running daemons. The shim respects the standard `StartLimitBurst` / `StartLimitIntervalSec` so a crash loop gives up cleanly.
* **User-tunable knobs**: declare defaults via `Environment=` and add `EnvironmentFile=-/run/bhatti/config-env`. The leading `-` makes the file optional, so sandboxes with no `--env` flags still boot cleanly. Values passed at create time (`bhatti create --env CHROME_REMOTE_PORT=9333`) override the unit’s defaults via the config-env bridge that lohar materialises at boot ([`cmd/lohar/main.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go)).
Drop the unit into `/etc/systemd/system/.service` and symlink it into `multi-user.target.wants/` so lohar’s `startEnabledServices` picks it up at boot:
```bash
cat > "$MOUNT/etc/systemd/system/headless-chrome.service" << UNIT
... (unit body above) ...
UNIT
ln -sf /etc/systemd/system/headless-chrome.service \
"$MOUNT/etc/systemd/system/multi-user.target.wants/headless-chrome.service"
```
That’s all. The shim handles cgroup placement (race-free, see [How services are spawned](/docs/under-the-hood/lohar-the-blacksmith/#how-services-are-spawned)), pidfile creation, restart-on-failure, and `journalctl -u` log routing. Your unit just declares what to run.
#### Anti-patterns
[Section titled “Anti-patterns”](#anti-patterns)
Three mistakes will silently break your tier:
* **Don’t write `init.sh`.** The legacy pre-v1.11.3 pattern (one shell script in `/etc/bhatti/init.sh` that runs all your daemons in the background) is gone. Daemons started that way escape the shim’s lifecycle entirely — no `systemctl status`, no `journalctl -u`, no `Restart=`, no `cgroup.kill` on stop. Existing tiers were converted off `init.sh` in v1.11.7–v1.11.9.
* **Don’t wrap `ExecStart=` in `/bin/sh -c '...'` yourself.** The shim does it for you. An inner shell is redundant on a good day; on a bad day (a daemon that double-`fork`s during startup) it introduces a fork the placement helper has to chase. Use `Environment=` and `${VAR}` expansion in the unit file — systemd’s parser handles that natively.
* **Don’t manage cgroups, pidfiles, or restart loops yourself.** All three live in the shim once your unit is enabled. Writing your own `pkill` cleanup or pidfile parsing means you’re fighting the shim, not using it.
You don’t need to know about `lohar spawn` to author a unit. The helper sits upstream of `ExecStart=` and is invisible to the unit file — it’s how the shim gets the daemon into the right cgroup before any fork. If you find yourself reading [`cmd/lohar/spawn.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/spawn.go) you’ve probably gone too deep.
### 3. Add size default in `scripts/build-tier.sh`
[Section titled “3. Add size default in scripts/build-tier.sh”](#3-add-size-default-in-scriptsbuild-tiersh)
```bash
case "$TIER" in
minimal) SIZE_MB="${SIZE_MB:-512}" ;;
browser) SIZE_MB="${SIZE_MB:-2048}" ;;
new-tier) SIZE_MB="${SIZE_MB:-1024}" ;; # ← add this
*) echo "unknown tier: $TIER" >&2; exit 1 ;;
esac
```
### 4. Add to CI release matrix
[Section titled “4. Add to CI release matrix”](#4-add-to-ci-release-matrix)
In `.github/workflows/release.yml`:
```yaml
tier: [minimal, browser, docker, computer, new-tier]
```
### 5. Add to the install script
[Section titled “5. Add to the install script”](#5-add-to-the-install-script)
In `scripts/install.sh`, update the interactive tier prompt and the `ALL_KNOWN_TIERS` variable in `do_server_update()`.
### 6. Verify
[Section titled “6. Verify”](#6-verify)
The bats test suite (`scripts/install_test.bats`) validates that every tier in `scripts/tiers/` appears in all four places:
```bash
bats scripts/install_test.bats
```
If you miss a registration point, the tier consistency tests catch it.
## Checklist
[Section titled “Checklist”](#checklist)
```plaintext
[ ] scripts/tiers/.sh
[ ] scripts/tiers/.sh — .service unit(s) for long-running daemons,
symlinked into multi-user.target.wants/
[ ] scripts/build-tier.sh — SIZE_MB case
[ ] .github/workflows/release.yml — matrix.tier
[ ] scripts/install.sh — interactive menu
[ ] scripts/install.sh — ALL_KNOWN_TIERS
[ ] bats tests pass
```
# Kernel
> The lean external guest kernel — how it's configured, built, and loaded on the krucible boot path.
bhatti v2 boots guests on a custom-built **lean Linux kernel** — a minimal 6.12.94 with only the features a sandbox workload needs. It’s an *external* kernel: krucible loads it directly with `krun_set_kernel`, which bypasses the kernel bundled inside `libkrunfw`. Skipping libkrunfw’s driver/subsystem init is worth roughly **2× on cold-start** — boot→agent lands at \~312ms on HVF versus \~610ms with the stock bundled kernel.
## Building
[Section titled “Building”](#building)
The kernel is built reproducibly inside Docker so the macOS dev box and Linux CI produce the same binary:
```bash
# Build for the host architecture (default), output to dist/kernel/
scripts/build-lean-kernel.sh
# Or name the target arch explicitly
scripts/build-lean-kernel.sh aarch64
scripts/build-lean-kernel.sh x86_64
```
The script pulls the pinned kernel.org source (`KERNEL_VERSION`, default `6.12.94`), copies our in-repo config over it, runs `make olddefconfig`, and builds the image:
* **arm64** → `arch/arm64/boot/Image` → `dist/kernel/Image-lean--`
* **x86\_64** → `vmlinux` → `dist/kernel/vmlinux-lean--`
Override the version with `KERNEL_VERSION=` (must be a `linux-stable` tag). There’s no libkrunfw in this path — the external kernel replaces it entirely.
## How it’s loaded
[Section titled “How it’s loaded”](#how-its-loaded)
The daemon **autodetects** the lean kernel: if a `dist/kernel/*-lean-*` image exists it’s used automatically; if not, krucible falls back to the `libkrunfw` bundle (slower, but keeps things working before you’ve built one). You can also point the daemon at a specific image with the `krucible_kernel_image` config field. `bhatti-vmm` passes the image to `krun_set_kernel` along with an arch-aware kernel command line, and `krun_start_enter` then skips libkrunfw entirely.
## Key config choices
[Section titled “Key config choices”](#key-config-choices)
The config under `scripts/lean-kernel/` (`config-lean_aarch64`, `config-lean_x86_64`) is deliberately lean — \~998 `=y` symbols versus \~1433 in the libkrunfw-equivalent config. Fewer subsystems to initialize means a faster boot and a smaller footprint. It enables exactly what block-root microVM guests use:
* **virtio** — virtio-blk (root overlay, config drive, volumes), virtio-net (the netd link), virtio-vsock (host↔guest control), virtio-console, virtio-mmio, balloon
* **ext4** — filesystem for the root image, config drive, and volumes; plus **overlayfs** and **virtio-fs**
* **devtmpfs** — required for `/dev` device nodes (auto-mounted)
* **TTY/PTY** — for interactive shell sessions
* **block layer** — the loop device and block-integrity bits the CoW image stack relies on
Everything unnecessary for a sandbox is turned off: USB, sound, most filesystems, most network drivers, 9p, initrd/initramfs support (guests are block-root, so there’s no ramdisk boot), and hardware-specific drivers.
## Architecture support
[Section titled “Architecture support”](#architecture-support)
Two configs are maintained, one per target arch:
* `config-lean_aarch64` — Raspberry Pi 5, Graviton, Apple Silicon, and other arm64 hosts (produces `Image-lean-*`)
* `config-lean_x86_64` — x86\_64 bare metal and cloud instances (produces `vmlinux-lean-*`)
The build script selects the source config and the make target from the arch argument, and the daemon selects the matching image for the host architecture at runtime.
# Testing
> Test philosophy, categories, and how to run bhatti's ~25,000 lines of tests.
\~25,000 lines of tests across \~95 test files. Zero mocks for VM tests — all krucible integration tests run on real microVMs, on real KVM (Linux) or HVF (Apple Silicon).
## Philosophy
[Section titled “Philosophy”](#philosophy)
**Test at the real boundary.** The engine tests drive a real krucible VM — a real `bhatti-vmm`, a real lohar guest — not a mock. Mocks silently accept any input and hide broken behavior; a real boot doesn’t.
**Test the protocol without VMs.** lohar has a test mode (`runTestMode`) that listens on Unix sockets instead of vsock. The whole protocol/handler surface runs on macOS or Linux in seconds, without root or a hypervisor.
**Test performance against budgets.** Timing-sensitive paths (cold restore, warm resume, boot→agent) assert on a measured latency budget, not a comment — so a regression that doubles cold-restore time fails a test instead of quietly shipping.
## Running tests
[Section titled “Running tests”](#running-tests)
### Protocol and agent tests (macOS/Linux, no root)
[Section titled “Protocol and agent tests (macOS/Linux, no root)”](#protocol-and-agent-tests-macoslinux-no-root)
```bash
go test ./pkg/agent/proto/ # frame encoding, round-trips
go test ./cmd/lohar/ # agent handlers via Unix socket (test mode)
go test ./pkg/agent/ # host-side client
```
These need no hypervisor — they exercise lohar over Unix sockets and the wire protocol directly.
### krucible engine tests (real VMs)
[Section titled “krucible engine tests (real VMs)”](#krucible-engine-tests-real-vms)
The full VM suite is behind the `krucible` build tag and needs a hypervisor (KVM or HVF):
```bash
go test -tags krucible ./pkg/engine/krucible/ -count=1
```
On macOS (Apple Silicon), build the fork and the VMM helper first, then run the suite locally on HVF:
```bash
make krucible # build libkrun (our fork) + assemble the link prefix
make vmm # build & sign bhatti-vmm (cgo against libkrun)
go test -tags krucible ./pkg/engine/krucible/ -count=1
```
Build the lean guest kernel too (`scripts/build-lean-kernel.sh`) for the faster external-kernel boot path — the daemon autodetects it. If a hypervisor isn’t available, the VM suites self-skip via `hasHypervisor()`.
### Cross-engine behavior
[Section titled “Cross-engine behavior”](#cross-engine-behavior)
VMM-agnostic behavior assertions live in `pkg/engine/enginetest` — the core an engine must satisfy (status/list, exec exit codes and stdout, the file API). krucible’s tests wire these suites to a real engine, so the same behavior contract is checked against the running VMM rather than bespoke per-engine scripts.
## Continuous integration
[Section titled “Continuous integration”](#continuous-integration)
Two workflows cover the engine:
* **`ci.yml` → `krucible-build`** (GitHub-hosted, no `/dev/kvm`). Builds the Rust fork (`make krucible`), the cgo helper (`make vmm`), and the `-tags krucible` Go, then runs the pure-unit tests (bundle, capability, recovery, state). The VM suites self-skip here — there’s no hypervisor on a hosted runner — so this is the build/link + unit gate.
* **`krucible-integration.yml`** (self-hosted `arc-runner-set`, real KVM). Brings up libkrun + `bhatti-vmm` + `bhatti-netd`, builds lohar, boots a real release-tier rootfs, and runs the full `-tags krucible` suite on actual hardware (arm64 and x86). This is the canonical parity gate for v2, and it touches no host networking — netd/TSI keep everything in userspace.
## Test categories
[Section titled “Test categories”](#test-categories)
| Category | Location | Requires |
| ------------- | ------------------------------------- | ------------------------------- |
| Protocol | `pkg/agent/proto/` | Nothing |
| Agent (guest) | `cmd/lohar/` | Nothing (Unix-socket test mode) |
| Client | `pkg/agent/` | Nothing |
| Cross-engine | `pkg/engine/enginetest/` | Hypervisor (via krucible) |
| Engine | `pkg/engine/krucible/` | libkrun + KVM/HVF |
| Server | `pkg/engine/krucible/` | libkrun + KVM/HVF |
| CLI | `cmd/bhatti/` | libkrun + KVM/HVF |
| Recovery | `cmd/bhatti/`, `pkg/engine/krucible/` | Mixed (unit vs VM) |
### What each category covers
[Section titled “What each category covers”](#what-each-category-covers)
* **Protocol tests** — binary framing: round-trips, EOF handling, max frame size, concurrent writes
* **Agent tests** — exec (exit codes, env vars, kill, process groups), TTY sessions (detach/reattach, scrollback, resize), files (read/write/stat/ls, truncation, atomicity), and the PID-1 lohar shims (systemctl/journalctl over the in-guest Unix socket)
* **Client tests** — the host-side agent client against lohar in test mode
* **Cross-engine tests** — the VMM-agnostic behavior contract every engine must satisfy
* **Engine tests** — full VM lifecycle on a real krucible boot: create/exec/destroy, full snapshot/restore, fork (CoW memory clone), cold-tier restore, qcow2 overlays and mounts, volumes, and networking through `bhatti-netd` (egress policing, same-owner sibling reachability, vsock forward/tunnel)
* **Server tests** — the full daemon stack (HTTP API + store + thermal manager) over a real engine, including the wake-on-request path (exec against a cold sandbox transparently cold-restores it)
* **CLI tests** — end-to-end: create → exec → file → shell → destroy
* **Recovery tests** — daemon restart scenarios: adopting live helpers by PID/cmdline, crashed VMs, missing snapshots, orphaned state
# Custom Domain
> Set up a domain so bhatti publish creates real URLs like alias.yourdomain.com.
By default, `bhatti publish` generates URLs using the server’s IP or hostname. With a custom domain, published sandboxes get proper URLs like `my-app.yourdomain.com` with TLS.
## What you need
[Section titled “What you need”](#what-you-need)
* A domain (e.g. `yourdomain.com`)
* DNS access to create A and wildcard records
* Either a wildcard TLS certificate or an email for Let’s Encrypt
## DNS records
[Section titled “DNS records”](#dns-records)
Point your API hostname and a wildcard for published sandboxes to your server:
```plaintext
A api.yourdomain.com →
A *.yourdomain.com →
```
The wildcard record is what makes `my-app.yourdomain.com` work — every published alias becomes a subdomain.
## Server configuration
[Section titled “Server configuration”](#server-configuration)
Add the `domain` block to `/etc/bhatti/config.yaml`:
### Option A: Wildcard certificate (recommended)
[Section titled “Option A: Wildcard certificate (recommended)”](#option-a-wildcard-certificate-recommended)
If you have a wildcard cert (from your DNS provider, Cloudflare Origin CA, or `certbot --manual` with DNS challenge):
```yaml
domain:
api_host: api.yourdomain.com
proxy_zone: yourdomain.com
tls_cert: /etc/bhatti/wildcard.pem
tls_key: /etc/bhatti/wildcard-key.pem
```
### Option B: Let’s Encrypt (automatic per-alias certs)
[Section titled “Option B: Let’s Encrypt (automatic per-alias certs)”](#option-b-lets-encrypt-automatic-per-alias-certs)
If you don’t have a wildcard cert, bhatti can issue individual certificates via Let’s Encrypt:
```yaml
domain:
api_host: api.yourdomain.com
proxy_zone: yourdomain.com
acme_email: you@example.com
```
Caution
Let’s Encrypt has a rate limit of 50 new certificates per week per registered domain. If you create and publish many sandboxes rapidly (e.g. per-PR previews in CI), you’ll hit this limit. Use a wildcard cert for production preview environments.
## What changes in domain mode
[Section titled “What changes in domain mode”](#what-changes-in-domain-mode)
Without `domain` config, bhatti listens on `:8080` for everything.
With `domain` config, bhatti starts three listeners:
| Port | Purpose |
| ---------------- | ---------------------------------------------------------------- |
| `:443` | API (`api.yourdomain.com`) + proxy (`*.yourdomain.com`) with TLS |
| `:80` | ACME challenges + HTTPS redirect |
| `127.0.0.1:8080` | Internal API (localhost only — health checks, admin) |
Routing is by `Host` header: requests to `api.yourdomain.com` go through normal API auth. Requests to `anything-else.yourdomain.com` go to the public proxy (no auth — this is the published URL).
## Restart
[Section titled “Restart”](#restart)
```bash
sudo systemctl restart bhatti
```
Check the logs for:
```plaintext
bhatti listening (domain mode)
api: https://api.yourdomain.com
proxy: https://*.yourdomain.com
```
## Verify
[Section titled “Verify”](#verify)
```bash
bhatti create --name test
bhatti publish test -p 8080 -a hello
# → https://hello.yourdomain.com
```
## Updating CLI config
[Section titled “Updating CLI config”](#updating-cli-config)
Remote CLI users should update their endpoint:
```bash
bhatti setup
# API endpoint: https://api.yourdomain.com
# API key: ****
```
## Getting a wildcard certificate
[Section titled “Getting a wildcard certificate”](#getting-a-wildcard-certificate)
### Cloudflare (if your DNS is on Cloudflare)
[Section titled “Cloudflare (if your DNS is on Cloudflare)”](#cloudflare-if-your-dns-is-on-cloudflare)
Origin CA certificates are free and last 15 years. Create one in the Cloudflare dashboard → SSL/TLS → Origin Server → Create Certificate. Select `*.yourdomain.com` and `yourdomain.com`. Download the PEM files.
### certbot with DNS challenge
[Section titled “certbot with DNS challenge”](#certbot-with-dns-challenge)
```bash
sudo certbot certonly --manual --preferred-challenges dns \
-d "*.yourdomain.com" -d "yourdomain.com"
```
This asks you to create a DNS TXT record to prove domain ownership. After validation, certs are saved to `/etc/letsencrypt/live/yourdomain.com/`. Renew every 90 days.
### acme.sh (lightweight alternative)
[Section titled “acme.sh (lightweight alternative)”](#acmesh-lightweight-alternative)
```bash
acme.sh --issue -d "*.yourdomain.com" -d "yourdomain.com" --dns dns_cf
```
Supports automatic DNS API integration with most providers. See [acme.sh DNS API](https://github.com/acmesh-official/acme.sh/wiki/dnsapi).
# Images & Tiers
> Built-in rootfs tiers, custom-image workflows, and how image scoping and sharing work.
Sandbox root filesystems are ext4 images. bhatti ships pre-built Ubuntu 24.04 *tier* images out of the box and supports three ways to add your own: pull from a public OCI registry, import a Docker image, or save a configured sandbox as an image.
The full CLI surface is in [Images reference](/docs/reference/cli/images/). This page covers the tier landscape, the typical workflows, and how scoping/sharing work — none of which is in the per-command pages.
## Built-in tiers
[Section titled “Built-in tiers”](#built-in-tiers)
bhatti ships four built-in tiers. Each is a pre-built Ubuntu 24.04 rootfs that builds on top of `minimal`:
| Tier | Adds | Size |
| -------------------------------------------- | -------------------------------------- | -------: |
| [`minimal`](/docs/managing/tiers/) | (base) | \~200 MB |
| [`browser`](/docs/managing/tiers/browser/) | Chromium, Playwright, Node 22 | \~600 MB |
| [`docker`](/docs/managing/tiers/docker/) | Docker Engine, buildx, compose, binfmt | \~550 MB |
| [`computer`](/docs/managing/tiers/computer/) | XFCE, KasmVNC, Chromium, full desktop | \~1.5 GB |
```bash
bhatti create --name scraper --image browser
bhatti create --name ci --image docker
```
The server install prompts for one tier on first run; install more later with `sudo bhatti update --tiers all` (or a comma-separated list). Tiers are auto-discovered from `/images/rootfs--.ext4` — no hardcoded list.
The deep dive for each tier — how its long-running daemons are managed, what env knobs you can pass, sizing, troubleshooting — lives at [Tiers](/docs/managing/tiers/). The rest of this page is about images you build *on top* of those tiers.
## Building custom images
[Section titled “Building custom images”](#building-custom-images)
Three workflows depending on where the source lives.
### Pull from a public registry
[Section titled “Pull from a public registry”](#pull-from-a-public-registry)
```bash
bhatti image pull python:3.12
bhatti image pull node:22-slim --name node-22
```
Async, server-side. The CLI shows progress and exits when the image is ready.
### Import from local Docker
[Section titled “Import from local Docker”](#import-from-local-docker)
```bash
docker pull ghcr.io/myorg/private:latest # any auth Docker can do
bhatti image import ghcr.io/myorg/private:latest
```
The CLI runs `docker save` locally and streams the tar to the server. This is the recommended path for private registries — Docker handles your existing auth, bhatti just receives bytes.
For tarballs without Docker:
```bash
docker save ubuntu:24.04 > /tmp/ubuntu.tar
bhatti image import --tar /tmp/ubuntu.tar --name ubuntu-24
```
### Save a configured sandbox
[Section titled “Save a configured sandbox”](#save-a-configured-sandbox)
```bash
bhatti create --name build --image minimal
bhatti exec build -- apt-get update && apt-get install -y nodejs pnpm
bhatti exec build -- pnpm install -g some-tool
bhatti image save build --name node-stack
```
The captured image now stamps out new sandboxes with everything pre-installed:
```bash
bhatti create --name worker-1 --image node-stack
bhatti create --name worker-2 --image node-stack
```
Only the **rootfs** is captured — persistent volumes are not part of the image. If your stack puts work in a volume, snapshot the volume separately ([`bhatti volume clone`](/docs/reference/cli/volumes/clone/)) and re-attach when stamping.
## Disk usage
[Section titled “Disk usage”](#disk-usage)
A common question when you first look at `/var/lib/bhatti/`: *“how much disk does each sandbox actually cost?”* The short answer in v2 is **the bytes the sandbox has written since boot** — and it’s the same answer on any filesystem, because copy-on-write lives in the disk-image format, not in the filesystem.
A fresh sandbox dir looks like this:
```plaintext
/var/lib/bhatti/
images/
rootfs-computer-arm64.ext4 1.5G (logical) read-only base image
rootfs-minimal-arm64.ext4 ~200M
sandboxes//
root.qcow2 (qcow2 CoW overlay over the base — the sandbox's write deltas)
config.ext4 ~1M (logical) env, secrets, file drops
bundle/ (only while cold: memory image + VM state + overlay copy + volumes)
```
Each sandbox’s `root.qcow2` is a thin overlay *over* one of the base images — a metadata-only header at create time that only allocates blocks as the guest writes them. `ls -lh` shows the *logical* device size the guest sees; `du -h` shows what’s actually on disk. The gap is usually enormous, and it’s the whole reason many sandboxes off one base are affordable:
```bash
$ du -sh /var/lib/bhatti/sandboxes/abc123/
45M /var/lib/bhatti/sandboxes/abc123/
$ ls -lh /var/lib/bhatti/sandboxes/abc123/root.qcow2
-rw------- 1 root root 1.0G ... # logical device size, not real allocation
```
### The per-sandbox cost
[Section titled “The per-sandbox cost”](#the-per-sandbox-cost)
The marginal cost of adding a sandbox is the bytes it writes over the shared base — config tweaks, log lines, packages installed since boot — typically single-digit MiB until you do something heavy. The base image is stored once and shared by every sandbox’s overlay; the OS page cache likewise holds one copy of the read-only base pages in RAM across all of them. Overhead of the qcow2 indirection is \~0.5%.
This is **filesystem-independent**. ext4, xfs, btrfs, and APFS all get instant create-from-base and shared-base dedup — there’s no ext4 penalty and no btrfs requirement (that was v1). If you *want* to shrink the data dir further, btrfs and xfs can transparently compress the base images and cold bundles, but that’s an optional optimization, not a prerequisite. See [Self-hosting → Storage, no special filesystem needed](/docs/self-hosting/#storage--no-special-filesystem-needed) and [Storage](/docs/under-the-hood/storage/) for the full picture, including the `KRUCIBLE_ROOT_RAW=1` opt-out for a plain raw ext4 root.
## Scoping and sharing
[Section titled “Scoping and sharing”](#scoping-and-sharing)
Images you create are private to your user — other users can’t see them in `image list`, can’t reference them by name, can’t read the underlying file.
To share an image with a specific user (or list of users):
```bash
sudo bhatti image share my-image --user alice --user bob
```
This command operates directly on the local SQLite database, so it requires running on the server with DB access (`sudo`). It is not an HTTP API call. There’s no “share with everyone” mode — sharing is always with named users. Inspect current shares with `--list`; revoke with [`bhatti image unshare`](/docs/reference/cli/images/unshare/).
System tier images (`minimal`, `browser`, …) are visible to everyone on the server automatically; they live under `user_id=""` in the database.
## See also
[Section titled “See also”](#see-also)
* [Tiers](/docs/managing/tiers/) — the four built-in starting points, with per-tier deep dives
* [Images reference](/docs/reference/cli/images/) — every command
* [Adding a tier](/docs/contributing/adding-a-tier/) — build a new system tier
# Secrets
> How encryption, scoping, and env-var injection work for sandbox secrets.
Secrets are values you’d rather not put on a command line. bhatti stores them encrypted, scoped per user, and injects them as environment variables when a sandbox boots.
The CLI surface lives in [Secrets reference](/docs/reference/cli/secrets/). This page covers the *model* — what’s encrypted, what isn’t, and the rules for which env var wins when multiple sources set the same name.
## Encryption at rest
[Section titled “Encryption at rest”](#encryption-at-rest)
On first start the daemon generates an [age](https://age-encryption.org/) keypair at `/age.key`. Every secret you set is encrypted under this key. Plaintext is never written to disk on the server — not in the database, not in temp files.
The age key file is the only thing that decrypts your secrets. Back it up alongside the database; lose either and the secrets become unrecoverable.
## Scoping
[Section titled “Scoping”](#scoping)
Secrets are per-user. Other users can’t list, read, or reference them by name. There’s no shared / global secret store.
`secret list` returns names + timestamps but never the encrypted blob — once a value is set, you can replace it with `secret set NAME new-value` but you can’t read it back.
## How injection works
[Section titled “How injection works”](#how-injection-works)
When a sandbox is created with `--secret API_KEY`:
1. The server fetches the encrypted blob from the secrets table for that user.
2. It decrypts under the age key.
3. The plaintext is written into the sandbox’s config drive as part of the env-var bundle.
4. lohar reads the config drive at boot and exports `API_KEY=<
Nothing secret-related. Decryption happens exactly once, at sandbox *create*, when the daemon reads the ciphertext from SQLite, decrypts it with the age key on disk, and writes the plaintext into the config drive. From that moment on, the secret lives as bytes in guest RAM and is captured into the memory image (`memory.img`) on snapshot.
When a [cold sandbox wakes](/docs/under-the-hood/thermal-states/#cold--hot):
* The server does **not** re-read from SQLite.
* It does **not** load the age key.
* It does **not** rewrite the config drive.
* It restores the memory image into a fresh krucible VM and resumes. The env vars are already in the guest’s process memory.
The age key (`<data_dir>/age.key`) is only loaded on an explicit encrypt or decrypt call — `bhatti secret set` or `bhatti create --secret`. It is **not** kept in long-lived daemon memory.
This matters for latency: a cold wake doesn’t pay any secret-reconstruction cost. The 360 ms p50 number on the homepage is dominated by the memory-image disk read, not by anything secret-handling — see [Storage → Cold wake and the page cache](/docs/under-the-hood/storage/#cold-wake-and-the-page-cache).
### Threat model
[Section titled “Threat model”](#threat-model)
* **Stolen `state.db`**: safe (ciphertext only, no key).
* **Cross-tenant access**: safe (per-user scoping + auth).
* **Stolen `state.db` + `age.key` together**: not safe; both files live in `<data_dir>` and an attacker with filesystem access has everything.
* **Host root compromise**: not safe.
* **Memory-image exfiltration**: not safe; secrets are env vars in guest RAM and the memory image (`memory.img`) is unencrypted on disk.
The host root is the trust boundary, same as Vault-on-a-box. If you need to weaken that — wrap `age.key` in KMS/HSM. We do not currently support that; it’s an open design question.
## Env var priority inside a sandbox
[Section titled “Env var priority inside a sandbox”](#env-var-priority-inside-a-sandbox)
When a command runs, env vars resolve in this order — later wins:
```plaintext
1. defaults (PATH, HOME, TERM, LANG)
↓
2. secrets (--secret NAME at create time)
↓
3. sandbox env (--env K=V at create time)
```
There is **no** per-`bhatti exec` env override. The `POST /sandboxes/:id/exec` endpoint doesn’t accept an `env` field — anything passed there is silently ignored. To inject env vars at exec time, use the WebSocket exec path (which does take an `env` map on the initial JSON command spec). For most use cases, set them at create time and accept that they apply to every command.
## Caveat: `--template` ignores `--secret`
[Section titled “Caveat: --template ignores --secret”](#caveat---template-ignores---secret)
Creating a sandbox from a template silently drops request-side `--secret` and `--file` flags. Only the secrets pre-configured on the template are honoured. If you need request-time secrets, create directly without `--template`.
## See also
[Section titled “See also”](#see-also)
* [Secrets reference](/docs/reference/cli/secrets/) — `set`, `list`, `delete`
* [`bhatti create --secret`](/docs/reference/cli/sandbox/create/) — inject at sandbox boot
# Tiers
> Built-in rootfs tiers, the per-tier operational story, and which one to pick.
A *tier* is a pre-built Ubuntu 24.04 rootfs that ships as a starting point for sandboxes. bhatti has four. Each builds on `minimal` and adds tooling for a specific workload.
```bash
bhatti create --name dev --image docker
bhatti create --name scraper --image browser
bhatti create --name desk --image computer --cpus 2 --memory 4096
```
Pick the one closest to what you’re doing — every tier is a real Ubuntu, so you can always `apt-get install` more on top.
## The four tiers
[Section titled “The four tiers”](#the-four-tiers)
| Tier | Adds on top of `minimal` | Approx size | Typical use |
| ------------------------- | -------------------------------------- | ----------: | ---------------------------------------- |
| [`minimal`](#minimal) | (nothing — bare base) | \~200 MB | Custom builds, anything CLI-first |
| [`browser`](./browser/) | Chromium, Playwright, Node 22 | \~600 MB | Scraping, CDP automation, headless tests |
| [`docker`](./docker/) | Docker Engine, buildx, compose, binfmt | \~550 MB | Docker-in-VM, OCI builds, **multi-arch** |
| [`computer`](./computer/) | XFCE, KasmVNC, Chromium, full desktop | \~1.5 GB | Visual / agent-driven desktop sessions |
The server install prompts for one tier on first run. Install more later:
```bash
sudo bhatti update --tiers all
# or pick: --tiers docker,browser
```
The server auto-discovers any `rootfs-<tier>-<arch>.ext4` it finds in its images directory, so installing a tier is just a download — no config changes, no daemon restart logic to think about.
## Units shipped by each tier
[Section titled “Units shipped by each tier”](#units-shipped-by-each-tier)
What’s actually running inside each tier, post-boot. Every unit listed here is managed by lohar’s [`systemctl` shim](/docs/under-the-hood/lohar-the-blacksmith/) — no real systemd anywhere — and the same three commands (`systemctl status`, `journalctl -u`, `systemctl restart`) work the same way regardless of tier.
| Tier | Unit | `Type=` | What it does |
| ---------- | ---------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `minimal` | (none shipped) | — | Provides the shim infrastructure. Anything you `apt install` (openssh-server, postgresql, nginx) becomes shim-managed on first boot. |
| `browser` | `headless-chrome.service` | `simple` | `headless_shell` with `--remote-debugging-port=9222`. Long-running, no PIDFile, restarted on crash. |
| `docker` | `docker.service` | `notify` | `dockerd` from upstream `docker-ce`. Drop-in binds the socket directly (no `fd://` socket activation) and chmods it for the `lohar` user. |
| `computer` | `kasmvnc-firstboot.service` | `oneshot` | Generates a per-sandbox VNC password on first boot. `RemainAfterExit=yes` so dependent units’ `After=` clauses are satisfied. |
| `computer` | `kasmvnc.service` | `simple` | `Xkasmvnc` — X server + RFB→WebSocket gateway, the endpoint on `:6080`. After `kasmvnc-firstboot.service`. |
| `computer` | `xfce-session.service` | `simple` | XFCE desktop session. `Requires=kasmvnc.service` so killing kasmvnc cascades. |
| `computer` | `bhatti-display-env.service` | `oneshot` | Writes `DISPLAY=:99` into the env file `bhatti exec` reads, so `screenshot` and `xdotool` Just Work without per-call env. |
If a tier’s daemon crashes, the shim’s `Restart=on-failure` policy brings it back; you see the crash and the restart in `journalctl -u <unit>`. If you edit a config file, `systemctl restart <unit>` picks it up. Every unit in this table is `Restart=on-failure` (long-running daemons) or `oneshot` (firstboot helpers). The cgroup-placement mechanism that makes `systemctl stop` reliable for forking daemons like `Xkasmvnc` is documented at [How services are spawned](/docs/under-the-hood/lohar-the-blacksmith/#how-services-are-spawned).
## The common operator story
[Section titled “The common operator story”](#the-common-operator-story)
Every long-running process in every built-in tier is managed by lohar’s **`systemctl` shim** ([lohar internals](/docs/under-the-hood/lohar-the-blacksmith/)). The same three commands — `systemctl status`, `journalctl -u`, `systemctl restart` — work the same way regardless of which tier you started from:
```bash
bhatti exec dev -- systemctl status docker
bhatti exec dev -- journalctl -u docker -n 50
bhatti exec dev -- systemctl restart docker
```
If a managed daemon crashes, the shim’s `Restart=on-failure` policy brings it back; you see the crash in `journalctl`. If you edit a config file, `systemctl restart <unit>` picks it up. The shape is the same on every tier — that’s the point of having one shim instead of one bespoke init script per tier.
A few practical points the shim deliberately doesn’t try to replicate from real systemd, because they cost more than they’re worth in a microVM sandbox:
* **No socket activation.** Daemons that ship with `ExecStart=… -H fd://` need a drop-in to bind the socket directly. The docker tier does this for `docker.service`; you only need to know if you write your own units.
* **No D-Bus.** Daemons that default to the `systemd` cgroup driver (notably dockerd) need a config override to use `cgroupfs`. Again — the docker tier handles this; flagged only because the failure mode is opaque (`dial unix /run/systemd/private: …`) if you build a tier from scratch.
* **No `loginctl`, no `journald` binary format.** Single-user sandbox, plain-text per-unit logs in `/var/log/bhatti/`. `journalctl -u <unit>` reads those files; `-f` follows.
The deeper rationale is in [Decisions & learnings](/docs/under-the-hood/decisions/) under “Why no real systemd.”
## Sizing and resources
[Section titled “Sizing and resources”](#sizing-and-resources)
| Tier | Minimum sensible | Comfortable |
| ---------- | ---------------- | ----------- |
| `minimal` | 1 vCPU / 256 MB | 1 / 512 |
| `browser` | 1 / 1024 | 2 / 2048 |
| `docker` | 2 / 2048 | 4 / 4096 |
| `computer` | 2 / 2048 | 4 / 4096 |
KasmVNC’s encoder thread count is sized to `nproc - 1`, so `--cpus 2` is the practical floor for the computer tier (any less and the encoder, X server, XFCE, and Chromium all share one core). For the docker tier, anything memory-intensive (large `buildx` builds, multi-container compose stacks) wants 4+ GB.
## Minimal
[Section titled “Minimal”](#minimal)
The base. Bare Ubuntu 24.04 with curl, sudo, ca-certificates, fuse3, and the `systemd-resolved` apt pin so package installs don’t accidentally pull in a real systemd. lohar is PID 1.
Useful when:
* You’re going to install everything yourself anyway.
* You want the smallest possible rootfs for snapshot/restore latency.
* You’re building a custom image with [`bhatti image save`](/docs/reference/cli/images/save/) and want to start from clean.
The `lohar` user (uid 1000) has passwordless sudo. `bhatti exec` runs as this user. The shim’s `systemctl` / `journalctl` symlinks are wired in at this layer, so any service-aware package you install (openssh-server, postgresql, redis-server, nginx) Just Works under the shim.
```bash
bhatti create --name dev --image minimal
bhatti exec dev -- sudo apt-get update
bhatti exec dev -- sudo apt-get install -y postgresql redis-server
bhatti exec dev -- systemctl status postgresql redis-server
```
## Adding a tier
[Section titled “Adding a tier”](#adding-a-tier)
If your team has a stack that’s repeated often, you can ship a tier of your own. See [Adding a tier](/docs/contributing/adding-a-tier/) for the build script convention, CI matrix entry, and install-flow integration.
## See also
[Section titled “See also”](#see-also)
* [Images & custom builds](/docs/managing/images/) — pull from OCI, import from Docker, save a configured sandbox
* [Adding a tier](/docs/contributing/adding-a-tier/) — build a new system tier from scratch
* [Lohar: the agent inside every VM](/docs/under-the-hood/lohar-the-blacksmith/) — how the `systemctl` shim works
* [Decisions & learnings](/docs/under-the-hood/decisions/) — why no real systemd, what’s intentionally out of scope
# Browser tier
> Headless Chromium with the Chrome DevTools Protocol exposed on :9222.
The `browser` tier ships Chromium (via Playwright’s `headless_shell` build), Playwright itself, and Node 22. Chromium starts at boot with CDP exposed on `:9222` ready to drive — no display server, no X, no VNC. The right tier for scraping, programmatic CDP control, headless tests, and PDF / screenshot rendering.
```bash
bhatti create --name scraper --image browser --cpus 2 --memory 2048
bhatti exec scraper -- curl -s http://localhost:9222/json/version | jq .Browser
# "Chrome/<version>"
```
## What’s in it
[Section titled “What’s in it”](#whats-in-it)
| Component | Source |
| ---------------------------------- | ------------------------------------------------------------------- |
| Chromium (`headless_shell` binary) | Playwright’s pinned download (`~/.cache/ms-playwright/chromium-*/`) |
| Playwright | `npm install -g playwright` |
| Node 22 | NodeSource apt repo |
| Chromium runtime deps | `npx playwright install-deps chromium` |
Playwright’s `headless_shell` is used rather than full Chrome — it’s the dedicated headless build with reliable CDP. The full Chrome binary has known CDP-attach issues in headless mode that show up as silent stream hangs.
## How Chromium is started
[Section titled “How Chromium is started”](#how-chromium-is-started)
Managed by lohar’s `systemctl` shim as `headless-chrome.service` (since v1.11.9). The unit ships with these defaults:
```ini
[Service]
Type=simple
Environment=CHROME_REMOTE_PORT=9222
Environment=CHROME_FLAGS=
EnvironmentFile=-/run/bhatti/config-env
ExecStart=/root/.cache/ms-playwright/chromium_headless_shell-<rev>/chrome-linux/headless_shell \
--no-sandbox \
--disable-gpu \
--disable-dev-shm-usage \
--remote-debugging-port=${CHROME_REMOTE_PORT} \
--remote-debugging-address=0.0.0.0 \
$CHROME_FLAGS
Restart=on-failure
RestartSec=2s
```
The path to `headless_shell` is resolved at build time by the tier script (Playwright stamps a version-suffixed directory) and baked into the unit. Operator UX is the same as every other tier:
```bash
bhatti exec scraper -- systemctl status headless-chrome
bhatti exec scraper -- journalctl -u headless-chrome -n 50
bhatti exec scraper -- systemctl restart headless-chrome
```
If Chromium crashes, the shim restarts it after 2 s. The crash + restart show up in `journalctl -u headless-chrome`.
## Driving it from outside the sandbox
[Section titled “Driving it from outside the sandbox”](#driving-it-from-outside-the-sandbox)
CDP is HTTP + WebSocket. Publish the port and connect from anywhere:
```bash
bhatti publish scraper -p 9222
# → https://scraper-abc123.bhatti.sh
```
From your host, point any CDP client at the published URL. Playwright on the host:
```js
const browser = await chromium.connectOverCDP('https://scraper-abc123.bhatti.sh');
```
Puppeteer:
```js
const browser = await puppeteer.connect({
browserWSEndpoint: 'wss://scraper-abc123.bhatti.sh/devtools/browser/…'
});
```
You can fetch the WebSocket endpoint from `/json/version`:
```bash
curl -s https://scraper-abc123.bhatti.sh/json/version | jq -r .webSocketDebuggerUrl
```
## Driving it from inside the sandbox
[Section titled “Driving it from inside the sandbox”](#driving-it-from-inside-the-sandbox)
`bhatti exec` runs as uid 1000 with Node and Playwright already in `$PATH`. The fastest agent loop is to write your script as a file and exec it:
```bash
bhatti file write scraper /workspace/scrape.js < ./scrape.js
bhatti exec scraper -- node /workspace/scrape.js
```
The Playwright globals (`chromium`, `firefox`, `webkit`) all work, but only Chromium has a running browser to attach to via CDP. To launch fresh Chromium instances from your script (rather than the boot-time one), use `chromium.launch({ executablePath: '/root/.cache/ms-playwright/chromium-*/chrome-linux/headless_shell' })` — the boot-time instance is a convenience, not a requirement.
## Tunables
[Section titled “Tunables”](#tunables)
Set at create time via `bhatti create --env K=V,K=V`. The values flow into the unit through lohar’s [config-env bridge](/docs/under-the-hood/lohar-the-blacksmith/#config-env-bridge):
| Variable | Default | Effect |
| -------------------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `CHROME_REMOTE_PORT` | `9222` | CDP port to listen on |
| `CHROME_FLAGS` | `""` | Extra space-separated flags appended to ExecStart (`--user-agent=…`, `--proxy-server=…`, `--lang=…`, etc.) |
```bash
bhatti create --name scraper --image browser \
--env "CHROME_FLAGS=--user-agent=Mozilla/5.0 (compatible; bhatti-bot/1.0)"
```
To edit flags on a running sandbox, drop a unit override and restart:
```bash
bhatti exec scraper -- sudo mkdir -p /etc/systemd/system/headless-chrome.service.d
bhatti exec scraper -- sudo tee /etc/systemd/system/headless-chrome.service.d/custom.conf <<'EOF'
[Service]
Environment=CHROME_FLAGS=--proxy-server=http://my-proxy:3128
EOF
bhatti exec scraper -- sudo systemctl restart headless-chrome
```
## Sizing
[Section titled “Sizing”](#sizing)
| Workload | `--cpus` | `--memory` |
| ------------------------------ | -------: | ---------: |
| Single-page scrapes, CDP drive | 1 | 1024 |
| Concurrent Playwright tests | 2 | 2048 |
| PDF rendering at scale | 2–4 | 2048–4096 |
Chromium is memory-hungry — 512 MB will OOM on any non-trivial page. The 1 GB default minimum is a real floor.
## Troubleshooting
[Section titled “Troubleshooting”](#troubleshooting)
| Symptom | Cause | Fix |
| ----------------------------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `curl /json/version` returns connection refused | Chromium failed to start at boot | `systemctl status headless-chrome`; `journalctl -u headless-chrome` for the crash reason |
| Chromium crashed mid-session | crash recovered via `Restart=on-failure`, but flapping if it crashes repeatedly | `systemctl status headless-chrome` shows `(activating)` if currently restarting, `(failed)` if start-limit hit; `journalctl -u headless-chrome` for the underlying error |
| Pages OOM with `Aw, Snap!` | Memory too low | bump `--memory` to 2048+ |
| `chrome-sandbox` permission errors | `--no-sandbox` got dropped from a custom drop-in | check `/etc/systemd/system/headless-chrome.service.d/` — the upstream unit always passes `--no-sandbox` |
| CHROME\_FLAGS not picked up | `--env` syntax: must be comma-separated, not repeated `--env` flags | `bhatti create --env "K1=V1,K2=V2"` (single string, comma-separated) |
## See also
[Section titled “See also”](#see-also)
* [Tiers overview](/docs/managing/tiers/)
* [Docker tier](./docker/) — when you’d combine Docker + headless Chromium
* [Computer tier](./computer/) — when you need a *visible* browser (KasmVNC + Chromium UI)
* [`bhatti publish`](/docs/reference/cli/networking/publish/) — exposing :9222 publicly
# Computer tier
> A full XFCE desktop in a microVM, served over KasmVNC on a single web port.
The `computer` tier is a full graphical Linux desktop: KasmVNC + XFCE + Chromium, served over one HTTP/WebSocket port (`6080`) as a browser-based web client. The right tier for visual agent-driven work (a model that sees a real desktop), demos, and any scenario where “open a browser and look at it” is the interface.
```bash
bhatti create --name desk --image computer --cpus 2 --memory 4096 --disk-size 8192
bhatti publish desk -p 6080
bhatti exec desk -- vnc-creds # username + password
# Open the URL printed by `publish` in your browser, log in.
```
## Why `--cpus 2` is the practical floor
[Section titled “Why --cpus 2 is the practical floor”](#why---cpus-2-is-the-practical-floor)
KasmVNC’s encoder thread count is sized to the guest’s vCPUs (`nproc - 1`, leaving one core for the desktop itself). On `--cpus 1` the encoder, the X server, XFCE, and Chromium all share one core — expect <10 fps and laggy input. Two cores is usable; four is comfortable.
## What’s in it
[Section titled “What’s in it”](#whats-in-it)
| Component | Role |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **KasmVNC** (`Xkasmvnc`) | Combined X server + RFB → WebSocket gateway. Replaces the older Xvfb + x11vnc + noVNC + websockify stack with one binary, one port. |
| **XFCE** (`startxfce4`) | Lightweight desktop environment. Window manager, panel, file manager, app launcher. |
| **Chromium** | The browser. `chromium-browser <url>` wrapper applies sane flags (`--no-sandbox` because we’re in a VM, `--test-type` to suppress nags). |
| `xdotool`, `scrot`, `xdpyinfo` | Mouse/keyboard input, screenshots, display introspection. Used by the agent-helper scripts below. |
## Credentials
[Section titled “Credentials”](#credentials)
A random 16-character password is generated **on first boot of each sandbox** (not at image-bake time):
* Hashed into `/root/.kasmpasswd` for KasmVNC.
* Stored cleartext in `/root/.vnc/cleartext` (root-only, mode `0600`) for the `vnc-creds` helper.
* Each sandbox gets its own; the published rootfs image carries no shared secret.
* Snapshot/resume preserves the password (the file already exists; firstboot generation is skipped).
Retrieve them anytime:
```bash
bhatti exec desk -- vnc-creds # human-readable
bhatti exec desk -- vnc-creds --json # for scripts/agents
```
Caution
[`bhatti image save`](/docs/reference/cli/images/save/) on a running computer-tier sandbox **bakes the current password into the saved image**. Treat saved-from-running images like any other secret-bearing artifact. If you share such an image, anyone with it has VNC access to every sandbox stamped from it. The clean alternative is to delete `/root/.kasmpasswd` and `/root/.vnc/cleartext` before saving, so firstboot regen kicks in on every new sandbox.
## Tunables
[Section titled “Tunables”](#tunables)
Pass at create time with `--env` (comma-separated):
| Variable | Default | What it controls |
| ---------------- | ----------- | ------------------------------------------------------------------------- |
| `DISPLAY_WIDTH` | `1280` | X server geometry width |
| `DISPLAY_HEIGHT` | `720` | X server geometry height |
| `DISPLAY_DEPTH` | `24` | Colour depth (16 / 24 / 32) |
| `KASM_FRAMERATE` | `60` | Max frames/sec the encoder will emit (matches KasmVNC’s upstream default) |
| `KASM_THREADS` | `nproc - 1` | Encoder thread count; default leaves one vCPU for the desktop |
```bash
bhatti create --name desk --image computer --cpus 4 --memory 4096 \
--env "DISPLAY_WIDTH=1920,DISPLAY_HEIGHT=1080,KASM_FRAMERATE=30"
```
`KASM_FRAMERATE=30` is worth setting on low-bandwidth links — encoder load drops roughly linearly with framerate, and 30 fps is more than enough for desktop work.
## Beyond the env knobs
[Section titled “Beyond the env knobs”](#beyond-the-env-knobs)
KasmVNC has dozens of options bhatti deliberately doesn’t surface (dynamic quality bounds, video-mode thresholds, scaling algorithms, DLP/clipboard policy, etc.). For those, edit `/etc/kasmvnc/kasmvnc.yaml` inside the sandbox and reconnect. The upstream docs are authoritative:
* [Video rendering options](https://github.com/kasmtech/KasmVNC/wiki/Video-Rendering-Options)
* [Stats / control API](https://github.com/kasmtech/KasmVNC/wiki/API)
* [Browser-side tuning](https://github.com/kasmtech/KasmVNC/wiki/Browser-Support)
## Agent helpers
[Section titled “Agent helpers”](#agent-helpers)
Run any of these via `bhatti exec`. `DISPLAY=:99` is pre-set (lohar writes it to `/run/bhatti/env` so exec inherits it), so X-talking helpers just work.
| Helper | Purpose |
| -------------------------------- | ------------------------------------------------ |
| `vnc-creds [--json]` | Print the username and password for this sandbox |
| `screenshot [--base64]` | Capture the current display as PNG |
| `screen-size` | Print the current X resolution |
| `active-window` / `list-windows` | Inspect window state |
| `xdotool …` | Drive mouse/keyboard input |
| `chromium-browser <url>` | Launch Chromium with sane flags |
The typical agent loop:
```bash
bhatti exec desk -- chromium-browser https://example.com &
sleep 2
bhatti exec desk -- screenshot --base64 > frame.b64
bhatti exec desk -- xdotool key Return
bhatti exec desk -- xdotool type "hello world"
```
## How the desktop is started
[Section titled “How the desktop is started”](#how-the-desktop-is-started)
Managed by lohar’s `systemctl` shim as four units (since v1.11.9), replacing the monolithic `init.sh`:
| Unit | Type | Purpose |
| ---------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kasmvnc-firstboot.service` | `oneshot` | Per-sandbox password generation + dynamic `KASM_THREADS` default. `ConditionPathExists=!/root/.kasmpasswd` makes it idempotent on snapshot/resume. |
| `kasmvnc.service` | `simple` | `Xkasmvnc :99` — combined X server + RFB-over-WebSocket gateway. Listens on port 6080. `After=kasmvnc-firstboot.service`. `Restart=on-failure`. |
| `xfce-session.service` | `simple` | `startxfce4` desktop session. `After=` + `Requires=kasmvnc.service` so killing the X server cascades. `Restart=on-failure`. |
| `bhatti-display-env.service` | `oneshot` | Writes `DISPLAY=:99` to `/run/bhatti/env` so `bhatti exec` inherits it. |
Activation order at boot: `kasmvnc-firstboot` and `bhatti-display-env` start in the first wave (no `After=`), then `kasmvnc.service`, then `xfce-session.service`.
Operator UX:
```bash
bhatti exec desk -- systemctl status kasmvnc xfce-session
bhatti exec desk -- journalctl -u kasmvnc -n 50
bhatti exec desk -- systemctl restart kasmvnc # cascades: xfce restarts too
bhatti exec desk -- systemctl restart xfce-session # just the desktop, X server stays
```
### What’s deliberately not running
[Section titled “What’s deliberately not running”](#whats-deliberately-not-running)
The pre-v1.11.9 `init.sh` started a system `dbus-daemon`, an orphan session bus via `dbus-launch`, and `pulseaudio`. None of those are started in the systemd-unit model. The reasoning:
* **`dbus-daemon --system`** keeps long-lived inotify watches on `/etc/dbus-1/`, an epoll on its listening socket, and per-connection timers. microVM snapshot/restore doesn’t preserve all kernel-side poller state cleanly on ARM64 — the same reason lohar runs as PID 1 instead of real systemd ([Decisions & learnings](/docs/under-the-hood/decisions/)).
* **`dbus-launch`** in the original init.sh ran a session bus tied to the boot shell’s scope, leaked its address into the `startxfce4` env, and orphaned when init.sh exited. A bug, not a feature. Modern XFCE (4.16+) launches its own per-session bus on demand when it actually needs one.
* **`pulseaudio`** wasn’t connected to any sink and no documented agent flow uses audio.
The `dbus` and `dbus-x11` packages stay installed (libdbus links from XFCE / Chromium), they’re just not auto-started as a system bus. Practical fallout: a few `xfsettingsd` warnings on first XFCE start, Thunar’s D-Bus activation paths are skipped, Chromium’s notifications and password-manager features are unavailable. The desktop still loads.
If you need a system dbus for an app that depends on it, the narrowest fix is to scope it to xfce-session’s lifecycle rather than running at multi-user.target — so the snapshot-risk window is only “while xfce is up”:
```ini
# /etc/systemd/system/dbus-system.service (drop in via the sandbox)
[Unit]
Description=System bus for xfce
Before=xfce-session.service
[Service]
Type=forking
PIDFile=/run/dbus/pid
ExecStartPre=/bin/mkdir -p /run/dbus
ExecStart=/usr/bin/dbus-daemon --system --fork
```
Then `systemctl enable --now dbus-system.service`. We don’t ship this by default.
## Sizing
[Section titled “Sizing”](#sizing)
| Workload | `--cpus` | `--memory` |
| --------------------------------------------------- | -------: | ---------: |
| Light interactive use, agent loops with screenshots | 2 | 2048 |
| Browser + agent + occasional video | 4 | 4096 |
| Multiple Chromium tabs + multimedia | 4 | 4096–6144 |
| Heavy multimedia / many open apps | 6+ | 8192+ |
`--disk-size 8192` is the recommended floor — Chromium’s cache and any web content you accumulate during a session can easily hit 4–5 GB.
## Troubleshooting
[Section titled “Troubleshooting”](#troubleshooting)
| Symptom | Cause | Fix |
| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| KasmVNC web client loads, then shows a red `No password configured for VNC Auth` banner | `-SecurityTypes=None` flag missing from `Xkasmvnc` invocation; image is pre-fix | `sudo bhatti update --tiers computer` on the server |
| HTTP Basic auth challenge appears but the password is wrong | Sandbox was saved-from-image with stale creds | `bhatti exec desk -- sudo rm /root/.kasmpasswd && sudo /etc/bhatti/init.sh` to regen |
| Black screen, no XFCE panel | `startxfce4` failed (usually D-Bus). Check `journalctl -u bhatti` (host side) or `~/.xsession-errors` (guest) | Restart desktop: `bhatti exec desk -- /etc/bhatti/init.sh` |
| Mouse input lags by seconds | Encoder is CPU-starved | Bump `--cpus`; lower `KASM_FRAMERATE`; check `top` inside the sandbox |
| `xdotool` does nothing | `DISPLAY` not set in the exec environment | \`bhatti exec desk — env |
## See also
[Section titled “See also”](#see-also)
* [Tiers overview](/docs/managing/tiers/)
* [`bhatti publish`](/docs/reference/cli/networking/publish/) — exposing port 6080 with a public URL
* [`bhatti share`](/docs/reference/cli/networking/share/) — single-use authenticated session URLs (useful for VNC handoff)
* [Thermal states](/docs/under-the-hood/thermal-states/) — `--keep-hot` for active sessions, to prevent the VM from pausing under your feet
# Docker tier
> Run Docker Engine inside a microVM. Multi-arch buildx, compose, and the operator UX.
The `docker` tier ships Docker Engine, the buildx and compose plugins, containerd, and the kernel-side bits for cross-arch builds (`binfmt_misc` mounted at PID 1, qemu-user emulation available on demand). It’s the right starting point for anything that wants Docker-in-VM — CI runners, image builds, compose stacks, sandboxed `docker run` experiments.
```bash
bhatti create --name ci --image docker --cpus 4 --memory 4096
bhatti exec ci -- docker run hello-world
```
That’s the entire setup. There is no `docker setup`, no daemon to start by hand, no socket permission dance — the tier wires all of it at build time.
## What’s in it
[Section titled “What’s in it”](#whats-in-it)
| Component | Version | Source |
| --------------------------------------- | -------------- | -------------------------------------- |
| Docker Engine (`dockerd`, `docker` CLI) | 29.x | `docker-ce` from Docker’s apt repo |
| `containerd` | latest stable | `containerd.io` from Docker’s apt repo |
| `docker buildx` plugin | 0.34.x | `docker-buildx-plugin` |
| `docker compose` plugin | latest | `docker-compose-plugin` |
| `iptables-legacy` | from Ubuntu | configured as the default |
| `binfmt_misc` filesystem | kernel feature | mounted by lohar at PID 1 |
The custom bhatti kernel enables every flag dockerd needs at runtime — bridge networking, veth, overlayfs, netfilter conntrack and security tables, and `BINFMT_MISC` for cross-arch emulation. See [Kernel](/docs/contributing/kernel/) for the full list and rationale.
## How dockerd is managed
[Section titled “How dockerd is managed”](#how-dockerd-is-managed)
The tier doesn’t ship its own `docker.service`. It reuses the one that comes with the `docker-ce` package and customises two things via a drop-in at `/etc/systemd/system/docker.service.d/bhatti.conf`:
```ini
[Service]
# Replace upstream `-H fd://` (which requires systemd socket activation
# lohar's shim deliberately doesn't implement) with a direct socket bind.
ExecStart=
ExecStart=/usr/bin/dockerd -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock
# Pick up `bhatti create --env DOCKER_*=...` overrides.
EnvironmentFile=-/run/bhatti/config-env
# Widen the socket so `bhatti exec` (uid 1000, no supplementary groups)
# can reach it. Safe inside an isolated single-user microVM.
ExecStartPost=/bin/chmod 666 /var/run/docker.sock
Restart=on-failure
RestartSec=2s
```
And `/etc/docker/daemon.json`:
```json
{ "exec-opts": ["native.cgroupdriver=cgroupfs"] }
```
Both overrides exist because the shim is **not** real systemd. Two specific gaps, both deliberate per [Decisions & learnings](/docs/under-the-hood/decisions/):
> **Why `-H fd://` is replaced.** Real `docker.service` listens via a paired `docker.socket` unit; systemd binds the socket and hands the fd to dockerd. The shim doesn’t implement socket activation (“a lot of code for marginal benefit — services start fast in our VMs anyway”). The drop-in tells dockerd to bind the socket itself.
> **Why `cgroupfs` instead of `systemd`.** Docker 29 defaults to `native.cgroupdriver=systemd`, which talks to systemd over D-Bus during every container create. The microVM has no D-Bus (libpam-systemd is pinned out of the rootfs to keep snapshot/restore safe). Without the override, every `docker run` dies in `runc` with `dial unix /run/systemd/private: connect: no such file`.
You don’t need to think about any of this in normal use. The drop-in is in place from boot one.
## The operator UX
[Section titled “The operator UX”](#the-operator-ux)
```bash
bhatti exec dev -- systemctl status docker
# ● docker.service - Docker Application Container Engine
# Active: active (running, PID 310)
# …
bhatti exec dev -- journalctl -u docker -n 50
bhatti exec dev -- journalctl -u docker -f # follow
bhatti exec dev -- systemctl restart docker
bhatti exec dev -- systemctl is-active docker # exits 0 if active
bhatti exec dev -- systemctl is-failed docker # exits 0 if failed
```
If dockerd crashes, the shim restarts it (`Restart=on-failure`, 2 s back-off). The failure shows up in `journalctl -u docker`. There is no separate logs file you have to remember; everything daemon-level is in the journal.
`docker logs <container>` (per-container application logs) is unchanged from upstream Docker — those flow through containerd, not the shim.
## Environment knobs
[Section titled “Environment knobs”](#environment-knobs)
dockerd inherits anything you pass to `bhatti create --env`. The bridge happens through `/run/bhatti/config-env`, which lohar writes from the config drive at boot and the unit reads via `EnvironmentFile=`.
```bash
bhatti create --name reg --image docker \
--env "DOCKER_REGISTRY_MIRROR=https://mirror.example.com,DOCKER_OPTS=--debug"
```
`--env` takes a **single comma-separated list** (`K=V,K=V`), not a repeated flag.
Recognised env vars today:
| Variable | Effect |
| -------------------------------- | ---------------------------------------------------------------------- |
| Anything starting with `DOCKER_` | Visible in `dockerd`’s environment; useful for env-driven docker hooks |
The drop-in deliberately doesn’t translate env vars into dockerd flags. If you need to change daemon flags themselves, write a second drop-in or edit `/etc/docker/daemon.json` inside the sandbox and `systemctl restart docker`.
## Socket permissions
[Section titled “Socket permissions”](#socket-permissions)
`/var/run/docker.sock` is `0666` after boot — anyone in the sandbox can talk to dockerd. This is the world-writable workaround mentioned in the drop-in. The reasoning:
* `bhatti exec` runs as uid 1000 (the `lohar` user) without supplementary group membership, so the standard `docker` group ACL doesn’t apply.
* The VM is the security boundary, not the docker group. Inside, it’s a single-user sandbox.
* Hardening this further would require lohar to preserve supplementary groups across exec, which is a separate change that hasn’t shipped yet.
If you re-mount the rootfs read-write and tighten the socket yourself, expect to also write a unit that wraps `bhatti exec` with `sg docker -c '…'`.
## Multi-arch builds with buildx
[Section titled “Multi-arch builds with buildx”](#multi-arch-builds-with-buildx)
Same-arch builds are native and need no setup:
```bash
bhatti exec dev -- docker buildx build -t me/app:latest .
```
Cross-arch builds need one extra command — `tonistiigi/binfmt --install all` — to populate the kernel’s `binfmt_misc` handler table with qemu-user interpreters:
```bash
bhatti exec dev -- docker run --privileged --rm tonistiigi/binfmt --install all
bhatti exec dev -- docker buildx create --use --name xb --driver docker-container
bhatti exec dev -- docker buildx build --platform linux/amd64,linux/arm64 \
-t me/app:multi --push .
```
### What that one extra command actually does
[Section titled “What that one extra command actually does”](#what-that-one-extra-command-actually-does)
Three layers have to line up for cross-arch builds to work:
| Layer | Provided by | State on a fresh docker sandbox |
| -------------------------------------------------- | ----------------------------- | ------------------------------- |
| `CONFIG_BINFMT_MISC=y` in the kernel | bhatti kernel | ✅ on |
| `/proc/sys/fs/binfmt_misc` filesystem mounted | lohar at PID 1 | ✅ mounted |
| Per-arch handler registrations + qemu interpreters | `tonistiigi/binfmt` container | ❌ empty until you run it |
The kernel knows it could dispatch foreign-arch ELFs to a userspace interpreter, but it has no interpreters and no mapping yet. The `tonistiigi/binfmt` image carries pre-built static qemu binaries (`qemu-aarch64`, `qemu-arm`, `qemu-riscv64`, …) and uses the kernel’s **fix-binary (`F`) flag** when registering, which makes the kernel open the interpreter and keep an fd to it. After registration the container exits, the qemu binaries are gone, the kernel’s open fds keep working.
You run the command once per sandbox. Registrations survive snapshot/restore (they live in kernel state, not on disk).
### Performance: when emulation is fine, when it isn’t
[Section titled “Performance: when emulation is fine, when it isn’t”](#performance-when-emulation-is-fine-when-it-isnt)
qemu-user emulation is correct but slow — typically 5–20× slower than native for CPU-bound work. Reasonable rules of thumb:
| Workload | Cross-arch emulation | Native-per-arch |
| --------------------------------------------------------- | ------------------------- | --------------- |
| Single-image dev builds, occasional cross-arch validation | 👍 fine | overkill |
| CI pushing release images on every merge | 😬 painful (multi-minute) | 👍 right answer |
| Heavily CPU-bound layers (compilers, native deps, ML) | ❌ avoid | 👍 right answer |
| `apt-get install`-heavy layers (mostly disk + network) | 👌 acceptable | 👍 faster |
For real CI, run one bhatti sandbox per native arch and stitch them with buildx:
```bash
# Driver machine — orchestrates two remote builders.
docker buildx create --name xb --platform linux/amd64 ssh://user@amd64-sandbox-host
docker buildx create --append --name xb --platform linux/arm64 ssh://user@arm64-sandbox-host
docker buildx use xb
docker buildx build --platform linux/amd64,linux/arm64 --push -t me/app:x .
```
(Server-side requires either an SSH driver or a TCP-listening dockerd that you publish via `bhatti publish`.) Each platform builds natively on a matching host; buildx writes the multi-arch manifest. bhatti’s [warm-wake](/docs/under-the-hood/thermal-states/) (\~4 ms) makes this cheap to leave idle between runs.
## Compose
[Section titled “Compose”](#compose)
`docker-compose-plugin` is installed, so `docker compose up` works as you’d expect. There’s no extra wiring.
```bash
bhatti create --name stack --image docker --cpus 4 --memory 4096
bhatti file write stack /workspace/compose.yaml < ./compose.yaml
bhatti exec stack -- docker compose -f /workspace/compose.yaml up -d
bhatti publish stack -p 8080 # if compose exposes :8080 on the host
```
For long-running compose stacks, consider `--keep-hot` on `bhatti create` so the [thermal manager](/docs/under-the-hood/thermal-states/) doesn’t pause the VM during idle periods (TCP connections from the public preview URL survive pause, but new connections during pause cost a wake).
## Networking
[Section titled “Networking”](#networking)
The bhatti kernel ships `iptables-legacy` (not nft). Docker’s bridge driver expects either, and the tier sets the legacy alternatives at build time. If you install a package that switches the alternatives back to nft, dockerd will fail to set up bridge networking. To recover:
```bash
bhatti exec dev -- sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
bhatti exec dev -- sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
bhatti exec dev -- sudo systemctl restart docker
```
Inside the sandbox, dockerd creates its own `docker0` bridge. Container-to-container networking, port publishing inside the VM (`docker run -p 8080:80 ...`), and DNS for container names all work normally. The bhatti-level publish (`bhatti publish dev -p 8080`) targets the sandbox’s host-side port; if a container binds to `0.0.0.0:8080` inside, it’s reachable.
## Troubleshooting
[Section titled “Troubleshooting”](#troubleshooting)
| Symptom | Cause | Fix |
| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `docker run` hangs / `dial unix /run/systemd/private` in logs | daemon.json missing or wrong cgroup driver | `cat /etc/docker/daemon.json` should say `cgroupfs`; if not, write it and `systemctl restart docker` |
| `docker buildx: unknown command` | `docker-buildx-plugin` not installed (very old image) | `sudo apt-get install -y docker-buildx-plugin` or `sudo bhatti update --tiers docker` on the server |
| `tonistiigi/binfmt --install all` reports success but cross-arch builds fail | `/proc/sys/fs/binfmt_misc` not mounted (very old lohar) | Verify with \`mount |
| `permission denied` on `/var/run/docker.sock` from `bhatti exec` | socket mode regressed after `systemctl restart docker` and ExecStartPost didn’t fire | `systemctl status docker` to confirm active; if active but socket is `0660`, the drop-in is missing — check `/etc/systemd/system/docker.service.d/bhatti.conf` |
| `docker run …` fails with iptables errors | alternatives switched to nft | re-set to legacy (see [Networking](#networking)) |
| Cross-arch container runs but exits with `exec format error` | qemu handler not registered for that arch | re-run `tonistiigi/binfmt --install all`; check `ls /proc/sys/fs/binfmt_misc/` for `qemu-<arch>` |
| `docker.service` repeatedly entering “activating” | `dockerd` not sending `READY=1` (Type=notify) — usually a containerd issue | `systemctl status containerd`; check `journalctl -u containerd` |
## Resource sizing
[Section titled “Resource sizing”](#resource-sizing)
| Workload | `--cpus` | `--memory` |
| ------------------------------------------------------------- | -------: | ---------: |
| `docker run` of small images (`hello-world`, alpine tools) | 2 | 2048 |
| Light compose stacks (3–5 small services) | 2–4 | 2048–4096 |
| `buildx build` of a typical app image | 4 | 4096 |
| Cross-arch `buildx build` with qemu emulation | 4 | 4096–8192 |
| Heavy CI builds (large dependency graphs, native compilation) | 8 | 8192+ |
Memory matters more than CPU for most builds — overlayfs page cache, build context, and intermediate layers all live in the guest. dockerd itself runs comfortably in 256 MB; anything above that is your workload.
## See also
[Section titled “See also”](#see-also)
* [Tiers overview](/docs/managing/tiers/) — what each tier is for
* [Adding a tier](/docs/contributing/adding-a-tier/) — build your own
* [Kernel](/docs/contributing/kernel/) — flags enabled for Docker, why
* [Lohar: systemctl shim](/docs/under-the-hood/lohar-the-blacksmith/) — how unit files are managed without real systemd
* [Decisions & learnings](/docs/under-the-hood/decisions/) — why no socket activation, why no D-Bus
* [Thermal states](/docs/under-the-hood/thermal-states/) — `--keep-hot` and what idle behaviour to expect for long-running stacks
# Users & Auth
> Multi-tenant model — per-user API keys, resource caps, network isolation, sandbox scoping.
bhatti is multi-tenant by default. Every user gets:
* An API key (`bht_<64-hex>`).
* Their own network: each user’s sandboxes get a dedicated `bhatti-netd` gateway, so sandboxes from different users never share a network.
* Per-user caps on sandbox count, per-sandbox CPU, and per-sandbox memory.
* Their own secrets, volumes, snapshots, and images.
Users are created server-side, with `sudo bhatti user create`. The CLI commands all live in [Server & admin](/docs/reference/cli/admin/) — this page covers the model, not the surface.
You won’t usually run `user create` for yourself. The install script (`curl -fsSL bhatti.sh/install | sudo bash`) creates an `admin` user during setup and writes its API key to your `~/.bhatti/config.yaml`. The first manual `user create` you run is for adding a teammate — see [Self-hosting → adding teammates](/docs/self-hosting/#adding-teammates).
## How keys work
[Section titled “How keys work”](#how-keys-work)
The server stores only the SHA-256 hash of each API key. The plaintext is shown once at creation and once on rotation; if the user loses it, [`rotate-key`](/docs/reference/cli/admin/user-rotate-key/) is the only path back. There’s no recovery.
Auth is `Authorization: Bearer <token>` on every API request except `/health` and `/_shell/<id>`. WebSocket connections use the same header. There’s no `?token=…` query-param auth — keys never end up in URLs (and therefore not in proxy logs).
The web shell is the one place a token appears in a URL — embedded in the **fragment** (`#token=…`), which never leaves the browser. The page extracts it client-side and uses it as a Bearer header on the WebSocket.
## Resource caps
[Section titled “Resource caps”](#resource-caps)
Every user has three per-user limits:
| Limit | Meaning |
| --------------------------- | ------------------------------------------------------ |
| `max_sandboxes` | Concurrent (non-destroyed) sandboxes the user can own. |
| `max_cpus_per_sandbox` | Cap on `--cpus` at create time. |
| `max_memory_mb_per_sandbox` | Cap on `--memory` at create time. |
Lowering a cap doesn’t affect existing sandboxes. New ones are rejected with `403` if they exceed the cap, or `429` if the user is at their `max_sandboxes`.
Storage is not currently capped per-user — disk usage grows freely until the host fills up. Track per-user volume size with `bhatti volume list` and the relevant `--data-dir`.
## Sandbox scoping
[Section titled “Sandbox scoping”](#sandbox-scoping)
The API enforces strict per-user isolation:
* `GET /sandboxes` returns only sandboxes the authenticated user created.
* Looking up someone else’s sandbox by ID returns `404` (intentional — the existence isn’t leaked).
* The user can’t exec, file-read, or destroy a sandbox they don’t own.
This is consistent across every endpoint that takes a sandbox ID.
## Network isolation
[Section titled “Network isolation”](#network-isolation)
User isolation isn’t a host firewall — it’s enforced at two layers. The API layer scopes every query to the authenticated user (see [Sandbox scoping](#sandbox-scoping) above). And at the data plane, each user’s sandboxes get their own [`bhatti-netd`](/docs/under-the-hood/networking/) gateway — a per-owner userspace gVisor netstack — so cross-user traffic never shares a network in the first place. There are no host bridges or iptables rules to reason about.
A user’s own sandboxes *can* reach each other (e.g. an agent sandbox can curl a worker sandbox), because they share one gateway; that traffic is still mediated and policed by netd rather than flowing peer-to-peer.
The `subnet` index you may see in some CLI output is vestigial — it survives from the v1 per-user-bridge design and now just picks a private address octet for the gateway. It isn’t a Linux bridge and you don’t need to think about it.
## See also
[Section titled “See also”](#see-also)
* [`bhatti user create`](/docs/reference/cli/admin/user-create/), [`list`](/docs/reference/cli/admin/user-list/), [`rotate-key`](/docs/reference/cli/admin/user-rotate-key/), [`delete`](/docs/reference/cli/admin/user-delete/)
* [Networking](/docs/under-the-hood/networking/) — the per-owner `bhatti-netd` gateway, policed egress, and host isolation
# Volumes
> Persistent ext4 volumes — attach semantics, mount restrictions, and the detached-state requirement.
Persistent volumes are ext4 filesystems that survive sandbox destruction. Create one, attach it to a sandbox, destroy the sandbox — the data is still there.
The CLI surface is in [Volumes reference](/docs/reference/cli/volumes/). This page covers the rules — when you can attach, what mount paths are allowed, and which operations require the volume to be unattached.
## Attaching at create time
[Section titled “Attaching at create time”](#attaching-at-create-time)
Volumes are mounted by lohar during sandbox boot. Specify them with `--volume` (repeatable):
```bash
bhatti volume create --name workspace --size 5120
bhatti create --name dev --volume workspace:/workspace
```
The format is `<name>:<mount>[:ro]`:
```bash
bhatti create --name reader --volume workspace:/data:ro
```
`:ro` mounts read-only. Multiple sandboxes can mount the same volume read-only at the same time — useful for shared base data. Read-write attachment is exclusive: only one sandbox can have it as read-write at any moment.
You can attach more than one volume per sandbox by repeating `--volume`.
## Mount path restrictions
[Section titled “Mount path restrictions”](#mount-path-restrictions)
Mount paths must be **absolute**. The guest agent rejects anything else.
You also can’t shadow system paths. Specifically, the following are blocked:
```plaintext
/proc /sys /dev /etc /usr /bin /sbin /lib /lib64 /run /var/run
```
Use a fresh subtree like `/workspace`, `/data`, `/srv/<app>` — anywhere outside the system hierarchy.
## Detached state
[Section titled “Detached state”](#detached-state)
Four operations require the volume to be **detached** (no active sandbox attachments):
* [`bhatti volume delete`](/docs/reference/cli/volumes/delete/)
* [`bhatti volume resize`](/docs/reference/cli/volumes/resize/)
* [`bhatti volume clone`](/docs/reference/cli/volumes/clone/)
* [`bhatti volume restore`](/docs/reference/cli/volumes/restore/)
The server returns `409 Conflict` if any sandbox has the volume mounted. To detach, either destroy the sandbox or [`bhatti stop`](/docs/reference/cli/sandbox/stop/) it (stopping detaches volumes; starting reattaches).
[`bhatti volume backup`](/docs/reference/cli/volumes/backup/) is the exception — it works while the volume is attached. The trade-off is consistency: writes in flight at backup time may be captured mid-update. For a guaranteed-clean backup, stop the sandbox first.
## Backups
[Section titled “Backups”](#backups)
Volume backups go to S3-compatible storage (AWS, Backblaze B2, R2, MinIO, etc.). They require a `backup` block in the server config; without it, backup-related commands return `501`.
```bash
bhatti volume backup workspace
bhatti volume backup-list workspace
bhatti volume restore workspace --backup-id bk_a1b2c3d4
```
Schedule regular backups with retention via the `backup.schedule` config — see [Configuration](/docs/reference/config/#backup).
## See also
[Section titled “See also”](#see-also)
* [Volumes reference](/docs/reference/cli/volumes/) — every command
* [`bhatti create --volume`](/docs/reference/cli/sandbox/create/) — attach syntax
* [Configuration: backup](/docs/reference/config/#backup) — S3 setup
# CLI Reference
> Every bhatti command, global flags, environment variables, output formats, and exit codes.
`bhatti` is a single binary. `bhatti serve` runs the daemon; everything else is a CLI client that talks to the daemon’s HTTP API. All sandbox commands accept the sandbox name or ID interchangeably.
```bash
bhatti setup # configure endpoint + API key
bhatti create --name dev # create a sandbox
bhatti exec dev -- echo hello # run a command
bhatti shell dev # interactive shell (Ctrl+\ to detach)
bhatti destroy dev # clean up
```
## Operations
[Section titled “Operations”](#operations)
| Command | Syntax | Description |
| -------------------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------- |
| **Core** | | |
| [`create`](/docs/reference/cli/sandbox/create/) | `bhatti create [flags]` | Create a new sandbox VM |
| [`destroy`](/docs/reference/cli/sandbox/destroy/) | `bhatti destroy <sandbox> [-y]` | Destroy a sandbox (alias: `rm`) |
| [`edit`](/docs/reference/cli/sandbox/edit/) | `bhatti edit <sandbox> [--keep-hot \| --allow-cold]` | Update mutable settings |
| [`exec`](/docs/reference/cli/exec/exec/) | `bhatti exec <sandbox> [--] <command...>` | Run a command in a sandbox |
| [`inspect`](/docs/reference/cli/sandbox/inspect/) | `bhatti inspect <sandbox>` | Show sandbox details (alias: `info`) |
| [`list`](/docs/reference/cli/sandbox/list/) | `bhatti list [-o wide]` | List sandboxes (alias: `ls`) |
| [`ports`](/docs/reference/cli/exec/ports/) | `bhatti ports <sandbox>` | List listening ports inside a sandbox |
| [`ps`](/docs/reference/cli/exec/ps/) | `bhatti ps <sandbox>` | List active sessions in a sandbox |
| [`share`](/docs/reference/cli/networking/share/) | `bhatti share <sandbox> [--revoke]` | Generate a web shell URL |
| [`shell`](/docs/reference/cli/exec/shell/) | `bhatti shell <sandbox> [--new]` | Open an interactive shell (alias: `sh`) |
| [`start`](/docs/reference/cli/sandbox/start/) | `bhatti start <sandbox> [--force]` | Resume a stopped sandbox |
| [`stop`](/docs/reference/cli/sandbox/stop/) | `bhatti stop <sandbox>` | Snapshot and stop a sandbox |
| **Files** | | |
| [`file read`](/docs/reference/cli/files/read/) | `bhatti file read <sandbox> <path>` | Read a file from a sandbox |
| [`file write`](/docs/reference/cli/files/write/) | `bhatti file write <sandbox> <path>` | Write a file (reads from stdin) |
| [`file ls`](/docs/reference/cli/files/ls/) | `bhatti file ls <sandbox> <path>` | List directory contents |
| **Networking** | | |
| [`publish`](/docs/reference/cli/networking/publish/) | `bhatti publish <sandbox> -p <port> [-a <alias>] [--shell]` | Publish a port with a public URL |
| [`unpublish`](/docs/reference/cli/networking/unpublish/) | `bhatti unpublish <sandbox> -p <port>` | Remove a published port |
| **Images** | | |
| [`image list`](/docs/reference/cli/images/list/) | `bhatti image list` | List available rootfs images |
| [`image pull`](/docs/reference/cli/images/pull/) | `bhatti image pull <ref> [--name N] [--auth U:T]` | Pull from a public OCI registry |
| [`image import`](/docs/reference/cli/images/import/) | `bhatti image import <docker-ref> \| --tar <path> [--name N]` | Import a Docker image or tarball |
| [`image save`](/docs/reference/cli/images/save/) | `bhatti image save <sandbox> --name <image>` | Save a sandbox’s filesystem as an image |
| [`image delete`](/docs/reference/cli/images/delete/) | `bhatti image delete <name> [-y]` | Delete an image |
| [`image share`](/docs/reference/cli/images/share/) | `bhatti image share <name> --user <u>...` | Share an image with specific users |
| [`image unshare`](/docs/reference/cli/images/unshare/) | `bhatti image unshare <name> --user <u>...` | Revoke image access from users |
| **Volumes** | | |
| [`volume create`](/docs/reference/cli/volumes/create/) | `bhatti volume create --name <v> --size <MB>` | Create a persistent volume |
| [`volume list`](/docs/reference/cli/volumes/list/) | `bhatti volume list` | List volumes |
| [`volume delete`](/docs/reference/cli/volumes/delete/) | `bhatti volume delete <name> [-y]` | Delete a volume |
| [`volume resize`](/docs/reference/cli/volumes/resize/) | `bhatti volume resize <name> --size <MB>` | Resize a volume (grow only) |
| [`volume clone`](/docs/reference/cli/volumes/clone/) | `bhatti volume clone <src> --name <new>` | Clone a volume (point-in-time copy) |
| [`volume backup`](/docs/reference/cli/volumes/backup/) | `bhatti volume backup <name>` | Back up a volume to S3 |
| [`volume backup-list`](/docs/reference/cli/volumes/backup-list/) | `bhatti volume backup-list <name>` | List backups for a volume |
| [`volume restore`](/docs/reference/cli/volumes/restore/) | `bhatti volume restore <name> --backup-id <id>` | Restore a volume from a backup |
| [`volume backup-delete`](/docs/reference/cli/volumes/backup-delete/) | `bhatti volume backup-delete <name> <id> [-y]` | Delete a volume backup |
| **Secrets** | | |
| [`secret set`](/docs/reference/cli/secrets/set/) | `bhatti secret set <name> <value>` | Create or update a secret |
| [`secret list`](/docs/reference/cli/secrets/list/) | `bhatti secret list` | List secret names |
| [`secret delete`](/docs/reference/cli/secrets/delete/) | `bhatti secret delete <name> [-y]` | Delete a secret |
| **Snapshots** | | |
| [`snapshot create`](/docs/reference/cli/snapshots/create/) | `bhatti snapshot create <sandbox> --name <snap>` | Checkpoint a running sandbox |
| [`snapshot list`](/docs/reference/cli/snapshots/list/) | `bhatti snapshot list` | List snapshots |
| [`snapshot resume`](/docs/reference/cli/snapshots/resume/) | `bhatti snapshot resume <snap> [--name <new>]` | Resume from a snapshot into a new sandbox |
| [`snapshot delete`](/docs/reference/cli/snapshots/delete/) | `bhatti snapshot delete <name> [-y]` | Delete a snapshot |
| **Server & admin** | | |
| [`serve`](/docs/reference/cli/admin/serve/) | `bhatti serve` | Start the bhatti daemon |
| [`setup`](/docs/reference/cli/admin/setup/) | `bhatti setup` | Configure CLI endpoint and API key |
| [`version`](/docs/reference/cli/admin/version/) | `bhatti version` | Print version and check for updates |
| [`update`](/docs/reference/cli/admin/update/) | `bhatti update [--cli-only] [--tiers <list>]` | Update bhatti |
| [`completion`](/docs/reference/cli/admin/completion/) | `bhatti completion <bash\|zsh\|fish>` | Generate shell completion script |
| [`user create`](/docs/reference/cli/admin/user-create/) | `bhatti user create --name <u> [--max-...]` | Create a user (server-only) |
| [`user list`](/docs/reference/cli/admin/user-list/) | `bhatti user list` | List users (server-only) |
| [`user rotate-key`](/docs/reference/cli/admin/user-rotate-key/) | `bhatti user rotate-key <name>` | Rotate a user’s API key (server-only) |
| [`user delete`](/docs/reference/cli/admin/user-delete/) | `bhatti user delete <name> [-y]` | Delete a user (server-only) |
| [`admin status`](/docs/reference/cli/admin/admin-status/) | `bhatti admin status` | One-shot system overview (server-only) |
| [`admin events`](/docs/reference/cli/admin/admin-events/) | `bhatti admin events [--type T] [--since 24h]` | Query the event log (server-only) |
| [`admin metrics`](/docs/reference/cli/admin/admin-metrics/) | `bhatti admin metrics [--since 1h]` | Query metrics snapshots (server-only) |
## Global flags
[Section titled “Global flags”](#global-flags)
These flags are accepted by every command.
| Flag | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--url <url>` | API endpoint (overrides config and `BHATTI_URL`). |
| `--token <key>` | API key (overrides config and `BHATTI_TOKEN`). |
| `--data-dir <path>` | Path to the server’s data directory. Required for `bhatti user *` and `bhatti admin *` since those operate directly on the local SQLite database. |
| `--json` | Print machine-readable JSON instead of the default human format. Universally supported. |
| `--timing` | Print a per-request timing breakdown (DNS, connect, TLS, server, transfer, total) to stderr after the command runs. |
| `-h`, `--help` | Show help for any command. |
## Environment variables
[Section titled “Environment variables”](#environment-variables)
| Variable | Used by | Description |
| --------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `BHATTI_URL` | CLI | API endpoint. Falls back when `--url` is not set and the config file has no `api_url`. |
| `BHATTI_TOKEN` | CLI | API key. Same fallback semantics as `BHATTI_URL`. |
| `BHATTI_CONFIG` | CLI + server | Override path to the config file. When set, layered loading is bypassed and only this file is read. |
| `BHATTI_LOG_LEVEL` | server (`bhatti serve`) | `debug`, `info` (default), `warn`, `error`. |
| `BHATTI_FORCE_STREAM` | CLI (`bhatti exec`) | Set to `1` to force NDJSON streaming output even when stdout is not a TTY. Useful in CI logs where you want real-time progress. |
## Configuration precedence
[Section titled “Configuration precedence”](#configuration-precedence)
Per-call flag → config file → environment variable → built-in default.
```plaintext
--url <x> (highest)
api_url: <x> (in config file)
BHATTI_URL=<x> (env var)
http://localhost:8080 (default, lowest)
```
The config file is the primary source — `bhatti setup` writes it and most users never touch the rest. Environment variables are the fallback for CI pipelines and scripts.
See [Configuration](/docs/reference/config/) for the file format and layered loading rules (`/etc/bhatti/config.yaml` for system settings, `~/.bhatti/config.yaml` for client credentials).
## Output formats
[Section titled “Output formats”](#output-formats)
**Human (default).** Compact, TTY-friendly. `bhatti list` prints columns; `bhatti inspect` prints kubectl-`describe`-style details with live disk usage on running VMs.
**`--json`.** Universally supported. Drops all human formatting; emits the raw API response (for queries) or a small status object (for mutations).
**`bhatti list -o wide`.** Adds CPUs, memory, disk, and image columns to `list`. Equivalent to `kubectl get -o wide`.
**Streaming exec output.** `bhatti exec` auto-detects how to format its output:
* Stdout is a TTY → NDJSON streaming, line-by-line, flushed immediately.
* Stdout is a pipe or `--json` is set → buffered JSON with `{exit_code, stdout, stderr}`.
* Override with `BHATTI_FORCE_STREAM=1` to force streaming when piping.
## Exit codes
[Section titled “Exit codes”](#exit-codes)
| Code | Meaning |
| ----- | -------------------------------------------------------------------------------------------------------------------------------- |
| `0` | Success. |
| `1` | CLI or API error (see stderr for details and recovery hints). |
| `<n>` | `bhatti exec` only — the child process’s exit code is forwarded verbatim, so a script’s `$?` reflects the command’s true status. |
Destructive commands (`destroy`, `volume delete`, `image delete`, `snapshot delete`, `volume backup-delete`, `secret delete`, `user delete`) require confirmation. Pass `-y` / `--yes` to skip the prompt; without `-y` in a non-interactive shell, the command exits `1`.
## Shell completion
[Section titled “Shell completion”](#shell-completion)
```bash
bhatti completion bash > /etc/bash_completion.d/bhatti
bhatti completion zsh > "${fpath[1]}/_bhatti"
bhatti completion fish > ~/.config/fish/completions/bhatti.fish
```
Sandbox-name completion uses a local cache (`/tmp/bhatti-completions-<uid>`) updated by `create`, `destroy`, and `list`. It never hits the network — instant, works offline.
# Server & admin
> Daemon, configuration, version, updates, user management, observability.
Commands that run on the bhatti server itself, plus a couple that work from any client (`setup`, `version`, `completion`, `update`).
## Categories
[Section titled “Categories”](#categories)
| | |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Daemon** | [`serve`](/docs/reference/cli/admin/serve/) |
| **Client config** | [`setup`](/docs/reference/cli/admin/setup/), [`version`](/docs/reference/cli/admin/version/), [`completion`](/docs/reference/cli/admin/completion/) |
| **Updates** | [`update`](/docs/reference/cli/admin/update/) |
| **User management** | [`user create`](/docs/reference/cli/admin/user-create/), [`user list`](/docs/reference/cli/admin/user-list/), [`user rotate-key`](/docs/reference/cli/admin/user-rotate-key/), [`user delete`](/docs/reference/cli/admin/user-delete/) |
| **Observability** | [`admin status`](/docs/reference/cli/admin/admin-status/), [`admin events`](/docs/reference/cli/admin/admin-events/), [`admin metrics`](/docs/reference/cli/admin/admin-metrics/) |
## Server-only commands
[Section titled “Server-only commands”](#server-only-commands)
`user *` and `admin *` operate **directly on the server’s SQLite database** (`/var/lib/bhatti/state.db`). They aren’t HTTP API calls — they need filesystem access to the database. Run them on the server, typically with `sudo`.
`serve` runs the daemon. Also server-only by definition (requires KVM and root).
The remaining commands (`setup`, `version`, `update`, `completion`) work from any client.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
# Bootstrap a new user (server)
sudo bhatti user create --name alice --max-sandboxes 10
# Check overall health (server)
sudo bhatti admin status
# Find recent thermal pauses (server)
sudo bhatti admin events --type thermal --since 24h
# Update the CLI on a developer machine
bhatti update
# Update everything on the server (CLI, kernel, rootfs, lohar)
sudo bhatti update
```
# bhatti admin events
> Query the event log. Filter by type, user, sandbox, time range.
# bhatti admin events server only
Query the structured event log.
## Synopsis
```
sudo bhatti admin events
[--type <prefix>] [--user <name|id>] [--sandbox <name|id>]
[--since <range>] [--limit <n>] [--count]
```
## Description
Returns rows from the audit/observability event log stored in the local database. Each event captures a moment in the system: sandbox lifecycle changes, thermal transitions, snapshot operations, image pulls, auth failures, proxy errors, daemon start/shutdown, and more.
Default output is one row per event with timestamp, type, user, sandbox, and a per-event-type summary line. JSON output includes the full meta object.
`--type` is a **prefix match**, so `--type thermal` matches `thermal.pause`, `thermal.wake`, `thermal.snapshot`, etc.
`--since` accepts:
* Relative durations: `30s`, `15m`, `24h`, `7d`.
* Absolute dates: `2026-04-01` (UTC midnight), `2026-04-01T12:00:00`, full RFC3339.
`--user` and `--sandbox` accept either the name or the internal ID. Names are easier; IDs are stable across renames.
`--count` returns a number-only response — useful for alerting/scripting (`if [[ $(bhatti admin events --type auth.failed --since 5m --count) -gt 10 ]]; then ...`).
## Examples
```bash
# Last 50 events of any kind
sudo bhatti admin events
```
```bash
# All thermal events in the last day
sudo bhatti admin events --type thermal --since 24h
```
```bash
# Auth failures in the last hour, count only
sudo bhatti admin events --type auth.failed --since 1h --count
```
```text
3
```
```bash
# Everything Alice did this week
sudo bhatti admin events --user alice --since 7d
```
```bash
# Drill into one sandbox's history
sudo bhatti admin events --sandbox dev --since 24h
```
```bash
# JSON for piping into jq
sudo bhatti admin events --type sandbox.destroyed --since 7d --json \
| jq '[.[] | .meta.lifetime_s] | add / length' # avg sandbox lifetime
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--type <prefix>` | — | Filter by event type. Prefix match (`thermal` matches `thermal.*`). | | `--user <name\|id>` | — | Filter by user. Name or user ID. | | `--sandbox <name\|id>` | — | Filter by sandbox. Name or ID. | | `--since <range>` | — | Earliest event to include. Relative (`24h`, `7d`) or absolute (`2026-04-01`). | | `--limit <int>` | `50` | Maximum number of events to return. | | `--count` | `false` | Return only the count of matching events, not the rows. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti admin status`](/docs/reference/cli/admin/admin-status/) — shows the most recent \~10 events inline
* [`bhatti admin metrics`](/docs/reference/cli/admin/admin-metrics/) — counter-based view
# bhatti admin metrics
> Time-bucketed metrics — request rates, errors, thermal counts, host load.
# bhatti admin metrics server only
Query metrics snapshots over time.
## Synopsis
```
sudo bhatti admin metrics [--since <range>]
```
## Description
The server takes a metrics snapshot roughly every minute and persists it to the database (in `metrics_snapshots`). This command reads them back and prints a compact table — request rates, error counts, proxy traffic, cold wakes, sandbox state distribution, host load, available memory.
The output is **bucketed automatically** based on the time range:
\| Range | Bucket size | | ----- | ----------- | | < 2 hours | 1 minute | | 2–24 hours | 15 minutes | | > 24 hours | 1 hour |
Each row aggregates the snapshots in its bucket (sums for counters like `req/min`, last-value for gauges like `LOAD`).
`--since` accepts the same formats as [`admin events --since`](/docs/reference/cli/admin/admin-events/) — relative durations or absolute dates. The default is **1 hour**.
## Examples
```bash
# Last hour at 1-minute resolution (default)
sudo bhatti admin metrics
```
```text
TIME REQ/min ERR PROXY/min WAKES HOT WARM COLD LOAD MEM_AVAIL
14:21 47 0 8 0 4 6 7 0.4 19421 MB
14:20 52 1 9 1 4 5 8 0.5 19388 MB
14:19 44 0 7 0 4 4 9 0.5 19412 MB
...
```
```bash
# Last 12 hours, auto-bucketed to 15-minute intervals
sudo bhatti admin metrics --since 12h
```
```bash
# Last week, auto-bucketed to 1-hour intervals
sudo bhatti admin metrics --since 7d
```
```bash
# Machine-readable, raw 1-min snapshots for the last 30 minutes
sudo bhatti admin metrics --since 30m --json | jq '.[] | .api_requests'
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--since <range>` | `1h` | Earliest snapshot to include. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti admin status`](/docs/reference/cli/admin/admin-status/) — current totals
* [`bhatti admin events`](/docs/reference/cli/admin/admin-events/) — what happened, when
# bhatti admin status
> One-shot system overview — sandboxes, users, host load, recent events.
# bhatti admin status server only
One-shot system overview.
## Synopsis
```
sudo bhatti admin status
```
## Description
Reads the latest metrics snapshot and event log from the local SQLite database and prints a compact dashboard:
* **Version + update notice** — current server version, latest GitHub release if newer.
* **Host stats** — load average, memory in use vs. total.
* **Sandbox totals** — broken down by thermal state (hot/warm/cold).
* **User totals** — total users, currently-active users.
* **API + proxy counters** — total requests, errors, auth failures, cold-wake count.
* **Per-user breakdown** — sandbox count vs. limit, current running/stopped split.
* **Recent events** — last \~10 events from the audit log with a one-line summary each.
Reads database state directly; doesn't touch the engine. Safe to run while the daemon is busy.
## Examples
```bash
sudo bhatti admin status
```
```text
Bhatti v1.10.0
Host: 0.42 load, 12.4 / 32.0 GB memory
Sandboxes 17 total (4 hot, 6 warm, 7 cold)
Users 8 total, 5 active
API 184.2K requests, 423 errors, 12 auth failures
Proxy 47.1K requests, 89 errors, 156 cold wakes
USER SANDBOXES STATUS
alice 3/10 2 running, 1 stopped
bob 0/5 -
worker-bot 5/5 5 running
RECENT EVENTS
14:22:18 thermal.pause alice scraper idle=180s
14:21:45 sandbox.created bob test-1 cpus=2 mem=1024 image=minimal
14:20:11 proxy.error - staging alias=staging status=502
14:18:33 thermal.wake alice api from=cold 47ms
...
```
```bash
# Machine-readable
sudo bhatti admin status --json | jq .gauges.sandboxes_hot
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti admin events`](/docs/reference/cli/admin/admin-events/) — drill into the event log
* [`bhatti admin metrics`](/docs/reference/cli/admin/admin-metrics/) — time-series view
# bhatti completion
> Generate shell completion scripts for bash, zsh, or fish.
Generate a shell completion script.
## Synopsis
```
bhatti completion <bash | zsh | fish>
```
## Description
Prints the completion script for the requested shell to stdout. Save it to the right place for your shell, restart, and tab-completion is enabled.
**Sandbox-name completion** uses a local cache (`/tmp/bhatti-completions-<uid>`), populated by `create`, `destroy`, and `list`. It never hits the network, so completions are instant and work offline. The cache may be slightly stale right after another machine creates a sandbox; running `bhatti list` refreshes it.
## Examples
### bash
```bash
# System-wide
bhatti completion bash | sudo tee /etc/bash_completion.d/bhatti > /dev/null
# Or per-user (no sudo)
echo 'source <(bhatti completion bash)' >> ~/.bashrc
```
### zsh
```bash
bhatti completion zsh > "${fpath[1]}/_bhatti"
# Or simpler
echo 'source <(bhatti completion zsh)' >> ~/.zshrc
```
### fish
```bash
bhatti completion fish > ~/.config/fish/completions/bhatti.fish
```
## Options
This command takes no flags besides the positional shell argument.
## See also
* [`bhatti setup`](/docs/reference/cli/admin/setup/) — `setup` detects your shell and prints the right snippet automatically.
# bhatti serve
> Start the bhatti daemon on Linux (KVM) or macOS (HVF).
# bhatti serve server only
Start the bhatti daemon.
## Synopsis
```
sudo bhatti serve
```
## Description
Starts the HTTP API listener, the thermal manager, the public proxy (if domain mode or `public_proxy_listen` is configured), the port scanner, and the metrics + event recorders. Recovers any previously running sandboxes from the SQLite database on startup, adopting their live `bhatti-vmm` helpers by PID.
Requires:
* Linux with KVM (`/dev/kvm`), or macOS on Apple Silicon (HVF)
* Root on Linux (for `/dev/kvm`); macOS needs no root
* Config at `/etc/bhatti/config.yaml` (or `$BHATTI_CONFIG`)
* The runtime bundle (`bhatti-vmm`, `bhatti-netd`, `libkrun`, the lean kernel) and at least one rootfs image present at the paths in the config
Logs are emitted as structured JSON to stdout. Set `BHATTI_LOG_LEVEL=debug` for verbose logging. Most operators run this under systemd; the install script sets up the unit file at `/etc/systemd/system/bhatti.service`.
## Examples
```bash
# Foreground (development)
sudo bhatti serve
```
```bash
# Under systemd (production)
sudo systemctl start bhatti
sudo journalctl -u bhatti -f
```
```bash
# Verbose logs
BHATTI_LOG_LEVEL=debug sudo bhatti serve
```
## Options
This command takes no flags. Configuration is via `/etc/bhatti/config.yaml`. See [Configuration](/docs/reference/config/).
## Lifecycle
* **Startup:** load config → open store → initialise engine → recover VMs → start servers (API + optional public proxy + optional internal loopback).
* **Shutdown** (`SIGTERM` / `SIGINT`): drain HTTP, snapshot all running VMs to disk, close store, exit. Snapshots are restored on the next start.
## See also
* [Configuration](/docs/reference/config/)
* [Self-hosting](/docs/self-hosting/)
# bhatti setup
> Configure the CLI's API endpoint and key. Saves to ~/.bhatti/config.yaml. Interactive by default; --url and --token make it scriptable.
Configure the CLI's API endpoint and key.
## Synopsis
```
bhatti setup [--url URL] [--token KEY]
```
## Description
Writes the API endpoint and key to `~/.bhatti/config.yaml` and tests the connection by listing sandboxes. The auth test hits `GET /sandboxes` (an authenticated endpoint), so a bad key fails immediately instead of appearing to succeed.
With no flags, runs interactively — prompts for endpoint and key, masks the key as you type. With `--url` and `--token`, runs non-interactively. Use the latter form for agents, CI scripts, and provisioning tools.
You usually don't need to run this on the server itself: the install script (`curl -fsSL bhatti.sh/install | sudo bash`) creates the admin user and writes the config to your `$SUDO_USER`'s `~/.bhatti/config.yaml` automatically. Run `setup` on a different machine that you want to drive the server from, or to switch your local CLI to a different bhatti server.
## Examples
### Interactive (the default)
```bash
bhatti setup
```
```text
API endpoint [http://localhost:8080]: https://api.bhatti.sh
API key: <input is hidden>
Saved to /home/alice/.bhatti/config.yaml
Testing connection... ✓ authenticated (3 sandboxes)
Enable completions:
echo 'source <(bhatti completion zsh)' >> ~/.zshrc
```
### Non-interactive (agents, CI, provisioners)
```bash
bhatti setup --url https://api.bhatti.sh --token bht_abc123
```
```text
Saved to /home/alice/.bhatti/config.yaml
Testing connection... ✓ authenticated (3 sandboxes)
```
The completions hint is suppressed — there's no human reading the output anyway. If the connection test fails, the command exits non-zero so your provisioner can detect the failure.
## Options
``string required
API endpoint URL. Sets the prompt's default. With `--token` set too, the prompt is skipped entirely and `setup` writes immediately.
``string required
API key. With `--url` set too, the prompt is skipped and `setup` writes immediately.
The auth test always runs. If you need to write the config without contacting a server (pre-baking an image, for example), edit `~/.bhatti/config.yaml` directly — it's just two YAML keys.
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token` (the global versions of these — they override config but don't write to it; `setup`'s flags are the persisting form), `--json`, `--timing`, `--data-dir`.
## Configuration file
After `setup`, your config looks like:
```yaml
api_url: https://api.bhatti.sh
auth_token: bht_abc123def456...
```
At runtime, this can be overridden with `--url`, `--token`, `BHATTI_URL`, or `BHATTI_TOKEN`. See [Configuration precedence](/docs/reference/cli/#configuration-precedence).
## See also
* [`bhatti version`](/docs/reference/cli/admin/version/) — check connection without changing config
* [`bhatti user create`](/docs/reference/cli/admin/user-create/) — create a user and get an API key (server-side, run by the operator)
* [`bhatti completion`](/docs/reference/cli/admin/completion/) — shell completions
* [Configuration](/docs/reference/config/) — full file format
* [Self-hosting → adding teammates](/docs/self-hosting/#adding-teammates) — the full flow for setting someone up against a remote bhatti
# bhatti update
> Update bhatti. Auto-detects CLI vs server.
Update bhatti to the latest release.
## Synopsis
```
bhatti update # auto-detect: CLI on a client, full update on a server
sudo bhatti update # full server update (CLI, kernel, lohar, rootfs)
sudo bhatti update --tiers all # also pull additional rootfs tiers
bhatti update --cli-only # update just the CLI binary, even on a server
```
## Description
Downloads `https://bhatti.sh/install` and runs it. The installer auto-detects whether this is a server (config exists at `/etc/bhatti/config.yaml`) or a CLI-only host, and updates accordingly.
**On a CLI-only machine:** updates the `bhatti` binary at `/usr/local/bin/bhatti`. Doesn't need root unless `/usr/local/bin` isn't writable by the user.
**On a server:** updates everything — the CLI plus the runtime bundle (`bhatti-vmm`, `bhatti-netd`, `libkrun`, the lean kernel) and any installed rootfs tier images. Only changed components are re-downloaded; unchanged tiers and kernels are skipped.
**`--cli-only`** forces the binary-only update path even on a server. Useful when you want to test a CLI fix without touching kernel/rootfs.
**`--tiers`** takes a comma-separated list (or `all`) of additional rootfs tiers to install (`browser`, `docker`, `computer`). The default install only ships `minimal`; this is how you add more without re-running the full install command.
Server updates require root (`/var/lib/bhatti` and the runtime bundle are root-owned). The command refuses up front if it detects a server install but the user isn't root, rather than partly-completing then failing inside the install script.
## Examples
```bash
# CLI host
bhatti update
```
```bash
# Server, default behaviour (full update)
sudo bhatti update
```
```bash
# Server, also pull all tiers
sudo bhatti update --tiers all
```
```bash
# Server, only update the CLI binary
bhatti update --cli-only
```
```bash
# Server, add a single tier
sudo bhatti update --tiers browser
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--cli-only` | `false` | Update only the CLI binary, even on a server. | | `--tiers <list>` | — | Comma-separated rootfs tiers to install, or `all`. Server-only — has no effect on CLI hosts. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti version`](/docs/reference/cli/admin/version/) — check current version + latest release
* [Self-hosting](/docs/self-hosting/) — initial install
* [Updating](/docs/updating/) — upgrade and uninstall guide
# bhatti user create
> Create a user with API key and per-user resource limits.
# bhatti user create server only
Create a user with an API key and per-user resource limits.
## Synopsis
```
sudo bhatti user create --name <name>
[--max-sandboxes <n>] [--max-cpus <n>] [--max-memory <MB>]
```
## Description
Allocates a new user record in the local SQLite database. The install script already runs this once for `admin` during setup, so the first time you'd call `user create` yourself is when you want to add a teammate or a CI account.
Each user gets:
* A user ID (`usr_<8-hex>`).
* An **API key** (`bht_<64-hex>`) — printed once, then only its SHA-256 hash is stored. **Save the key immediately**; it can't be recovered. Use [`user rotate-key`](/docs/reference/cli/admin/user-rotate-key/) if it's lost.
* A **subnet index** — an integer assigned monotonically. It's a vestigial field carried over from v1; in v2, user isolation comes from API scoping and a per-owner `bhatti-netd` gateway, not a per-user bridge network. You don't need to think about it.
* Per-sandbox resource caps (`--max-cpus`, `--max-memory`) and a per-user sandbox count cap (`--max-sandboxes`).
Records the `user.created` event in the audit log.
This is a **server-only** command — it operates directly on `/var/lib/bhatti/state.db`. Runs without the daemon needing to be up, but typically as `sudo` because the data dir is root-owned.
## Examples
```bash
sudo bhatti user create --name alice
```
```text
User created:
ID: usr_a1b2c3d4
Name: alice
Subnet: 2
API key: bht_abc123def456...
This key will not be shown again. Save it now.
For remote CLI use:
bhatti setup --url https://your-server:8080 --token bht_abc123def456...
```
```bash
# Custom limits
sudo bhatti user create --name beefy \
--max-sandboxes 20 \
--max-cpus 8 \
--max-memory 16384
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | (required) | Username. Must be unique. | | `--max-sandboxes <int>` | `5` | Maximum concurrent sandboxes. | | `--max-cpus <int>` | `4` | Maximum vCPUs per sandbox (capped at create time). | | `--max-memory <int>` | `4096` | Maximum memory MB per sandbox (capped at create time). |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
`--data-dir` is honoured here when you need to point at a non-default data directory.
## See also
* [`bhatti user list`](/docs/reference/cli/admin/user-list/)
* [`bhatti user rotate-key`](/docs/reference/cli/admin/user-rotate-key/) — replace a lost or compromised key
* [Users & auth](/docs/managing/users/) — broader narrative on multi-tenant setup
# bhatti user delete
> Delete a user. Fails if the user has active sandboxes.
# bhatti user delete server only
Delete a user.
## Synopsis
```
sudo bhatti user delete <name> [-y]
```
## Description
Removes the user record. The user's API key stops working immediately.
**Fails if the user has active (non-destroyed) sandboxes.** Destroy them first:
```bash
sudo bhatti user list # find them via the user's sandbox count
# (server-side — to list a specific user's sandboxes you need the SQL or the user's API key)
```
Volumes, secrets, snapshots, and images owned by the user remain on disk after the user is deleted (orphaned). A future `--purge` mode could clean these up; for now, manual cleanup is required if you want to reclaim the space.
By default, prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
Records a `user.deleted` event in the audit log.
## Examples
```bash
sudo bhatti user delete alice
```
```text
Delete user "alice"? [y/N]: y
deleted user "alice"
```
```bash
# Active sandboxes still around
sudo bhatti user delete alice
```
```text
Error: user "alice" has 3 active sandbox(es)
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti user list`](/docs/reference/cli/admin/user-list/)
* [`bhatti user rotate-key`](/docs/reference/cli/admin/user-rotate-key/) — usually a better answer than delete
# bhatti user list
> List users with their resource limits and current sandbox counts.
# bhatti user list server only
List all users on the server.
## Synopsis
```
sudo bhatti user list
```
## Description
Lists all users with ID, name, current sandbox count vs. limit, per-sandbox CPU and memory caps, and the subnet index. Reads from the local SQLite database.
## Examples
```bash
sudo bhatti user list
```
```text
ID NAME SANDBOXES CPUS MEM SUBNET
usr_a1b2c3d4 alice 3/10 4 4096 2
usr_e5f6g7h8 bob 0/5 4 4096 3
usr_i9j0k1l2 worker-bot 5/5 2 2048 4
```
The `SANDBOXES` column is `current/limit` — `5/5` means at-cap. The `SUBNET` column is a vestigial per-user index carried over from v1 — in v2 it no longer maps to a bridge network (isolation is API scoping + the per-owner `bhatti-netd` gateway).
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti user create`](/docs/reference/cli/admin/user-create/)
* [`bhatti admin status`](/docs/reference/cli/admin/admin-status/) — also shows per-user sandbox breakdown
# bhatti user rotate-key
> Rotate a user's API key. Old key is immediately invalidated.
# bhatti user rotate-key server only
Rotate a user's API key.
## Synopsis
```
sudo bhatti user rotate-key <name>
```
## Description
Generates a new API key, stores its SHA-256 hash, and prints the plaintext **once**. The previous key is invalidated server-side immediately — any in-flight requests using the old key fail with `401`.
Use this when:
* A key may have leaked.
* A user has lost their key (the original is unrecoverable).
* You have a key-rotation policy.
The user's existing sandboxes, secrets, volumes, snapshots, and images are all preserved — only the auth token changes.
Records a `user.key_rotated` event in the audit log.
## Examples
```bash
sudo bhatti user rotate-key alice
```
```text
API key rotated for "alice"
New key: bht_xyz789...
The old key is immediately invalidated.
This key will not be shown again. Save it now.
```
After rotation, give the user the new key. They run `bhatti setup` to update their config.
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti user create`](/docs/reference/cli/admin/user-create/)
* [`bhatti setup`](/docs/reference/cli/admin/setup/) — what the user runs to install the new key
# bhatti version
> Print CLI version, server version, and check for updates.
Print CLI and server versions; check for updates.
## Synopsis
```
bhatti version
```
## Description
Prints:
* **CLI version** — embedded at build time via `-ldflags`. Shows `dev` for unreleased builds.
* **API endpoint** — currently configured from flags, env, or config.
* **Server version** — read from the `X-Bhatti-Version` response header on a probe to `GET /sandboxes`. Skipped if unreachable or unauthenticated.
* **Update notice** — if the CLI's GitHub release check (cached for 1 hour at `~/.bhatti/.latest-version`) finds a newer release, prints it.
The server-version probe also drives the `X-Bhatti-Min-CLI` warning that appears on every command — if your CLI is older than the server's minimum, every command emits a one-line stderr warning.
## Examples
```bash
bhatti version
```
```text
bhatti v1.10.0
api: https://api.bhatti.sh
server: v1.10.0
```
```bash
# When an update is available
bhatti version
```
```text
bhatti v1.9.0
api: https://api.bhatti.sh
server: v1.10.0
Update available: v1.9.0 → v1.10.0 (bhatti update)
```
```bash
# Machine-readable
bhatti version --json
```
```json
{
"version": "v1.10.0",
"api": "https://api.bhatti.sh",
"server_version": "v1.10.0",
"latest_version": "v1.10.0"
}
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti update`](/docs/reference/cli/admin/update/) — apply available updates
* [`bhatti admin status`](/docs/reference/cli/admin/admin-status/) — also prints version info on the server side
# Execution & shells
> Run commands, open shells, list active sessions, and discover listening ports inside sandboxes.
Commands that drive execution inside a running sandbox: one-shot commands, interactive shells, and visibility into what’s currently running and listening.
| Command | Description |
| ------------------------------------------------- | ------------------------------------------------------------------------------- |
| [`bhatti exec`](/docs/reference/cli/exec/exec/) | Run a command. Streaming or buffered output, optional timeout, optional detach. |
| [`bhatti shell`](/docs/reference/cli/exec/shell/) | Open an interactive shell with detach/reattach semantics (alias: `sh`). |
| [`bhatti ps`](/docs/reference/cli/exec/ps/) | List active sessions in a sandbox (init scripts, shells, detached execs). |
| [`bhatti ports`](/docs/reference/cli/exec/ports/) | List listening ports detected inside a sandbox. |
All commands wake cold sandboxes transparently — you don’t need to start the sandbox first.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
bhatti exec dev -- npm install # one-shot command
bhatti exec dev -- bash -c 'echo $API_KEY' # shell expansion via /bin/bash
bhatti exec dev --timeout 60 -- long-running.sh # cap runtime at 60s
bhatti exec dev --detach -- ./worker.sh # run in background, return PID
bhatti shell dev # interactive (Ctrl+\ to detach)
bhatti shell dev --new # don't reattach to a running session
bhatti ps dev # what's running inside?
bhatti ports dev # what's listening?
```
## See also
[Section titled “See also”](#see-also)
* [Sandbox commands](/docs/reference/cli/sandbox/) — create/list/destroy
* [Files](/docs/reference/cli/files/) — read and write files inside sandboxes
* [API: `POST /sandboxes/:id/exec`](/docs/reference/api/#run-a-command) — the underlying HTTP endpoint, streaming and detach modes
# bhatti exec
> Execute a command inside a sandbox. Streaming or buffered output, optional timeout, optional detach.
Execute a command inside a sandbox. The exit code of the child process is forwarded to the CLI's exit code. Sleeping sandboxes wake automatically.
## Synopsis
```
bhatti exec <sandbox> [--] <command...> [flags]
```
## Description
The `--` separator is optional — `bhatti exec dev echo hello` works just like `bhatti exec dev -- echo hello`. Use `--` only when the command contains flags that cobra would otherwise try to parse (e.g. `bhatti exec dev -- ls -la /tmp`).
Commands run as the unprivileged user `lohar` (uid 1000), not root. Use `sudo` inside the sandbox for root operations. Stdout and stderr are kept on separate streams; when output is captured, the result distinguishes them.
**Output mode auto-detects:**
* **Stdout is a TTY** → streaming NDJSON, line-by-line, flushed as the process emits it. You see `npm install`'s progress bar in real time.
* **Stdout is a pipe or `--json` is set** → buffered. The CLI prints a JSON object `{exit_code, stdout, stderr}` once the command finishes.
* **`BHATTI_FORCE_STREAM=1`** → forces streaming even when piping. Useful for CI logs that want progressive output.
**Cold sandboxes wake transparently.** If the sandbox is `cold` or `warm`, the server resumes it before executing — typically \~50ms for cold, sub-millisecond for warm. The wake is invisible to the caller; the command just takes a bit longer.
**Failures get recovery hints.** Common errors (sandbox not found, sandbox not running, name conflict, quota reached) are followed by a one-line suggestion in stderr. For example:
```text
404 Not Found: sandbox not found
Check sandbox name:
bhatti ls
```
## Examples
```bash
# Most basic
bhatti exec dev -- echo hello
```
```bash
# -- is optional
bhatti exec dev echo hello
```
```bash
# Pipe output to other tools — CLI switches to buffered mode automatically
bhatti exec dev -- cat /workspace/data.json | jq .name
```
```bash
# Shell expansion needs a shell — env vars in the *guest*, not in your terminal
bhatti exec dev -- bash -c 'echo $API_KEY'
```
```bash
# Cap a long-running command at 60s
bhatti exec dev --timeout 60 -- ./benchmark.sh
```
```bash
# Run in the background, return immediately with the guest PID
bhatti exec dev --detach -- ./worker.sh
```
```text
pid: 4218
output: /var/log/lohar/sessions/sess-7b3c.log
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--timeout <int>` | `300` | Exec timeout in seconds. The server kills the process group on timeout and returns exit code `137`. Hard max: `86400` (24 hours). | | `--detach` | `false` | Run the command in the background and return immediately with the guest PID and the path to the captured output file. The session continues until it finishes or is killed. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## Exit codes
\| Code | Meaning | | ---- | ------- | | `<n>` | The child process's exit code is forwarded verbatim. | | `137` | The child was killed (timeout exceeded, or session manually killed). | | `1` | CLI / API error before the command ran (sandbox not found, network, auth). |
## See also
* [`bhatti shell`](/docs/reference/cli/exec/shell/) — interactive shell with reattach
* [`bhatti ps`](/docs/reference/cli/exec/ps/) — see detached / running sessions
* [API: `POST /sandboxes/:id/exec`](/docs/reference/api/#run-a-command)
# bhatti ports
> List listening TCP ports detected inside a sandbox, with proxy URLs for each.
List the listening TCP ports detected inside a sandbox.
## Synopsis
```
bhatti ports <sandbox>
```
## Description
The host periodically scans each running sandbox for listening ports and caches them. `bhatti ports` reads that cache and prints one row per port. Each row also shows the path-based proxy URL — a developer convenience for hitting the port from the host without a domain or `bhatti publish`.
The proxy URL goes through the API server at `<api>/sandboxes/<id>/proxy/<port>/...`. It requires the same Bearer auth as any other API call — useful for poking at services during development. For public, auth-free URLs, use [`bhatti publish`](/docs/reference/cli/networking/publish/).
Cold sandboxes are not scanned (they have no listening processes by definition). To list ports for a sleeping sandbox, wake it first or just send a request that triggers a wake.
## Examples
```bash
bhatti ports dev
```
```text
PORT URL
3000 http://localhost:8080/sandboxes/dev/proxy/3000/
6379 http://localhost:8080/sandboxes/dev/proxy/6379/
```
```bash
# Quick test against a service inside the sandbox
bhatti ports dev --json | jq -r '.[] | .proxy_url' | head -1 | xargs curl -H "Authorization: Bearer $BHATTI_TOKEN"
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti publish`](/docs/reference/cli/networking/publish/) — public auth-free URL for a port
* [API: `GET /sandboxes/:id/ports`](/docs/reference/api/#list-listening-ports)
* [API: `ANY /sandboxes/:id/proxy/:port/*`](/docs/reference/api/#reverse-proxy)
# bhatti ps
> List active sessions inside a sandbox — init scripts, shells, detached execs.
List active sessions inside a sandbox.
## Synopsis
```
bhatti ps <sandbox>
```
## Description
A *session* is a process the guest agent is tracking — typically because it has a PTY, was started detached, or was launched as the sandbox's `--init` script. Active sessions persist across client disconnections and show up here regardless of whether anyone is attached.
The `init` script (from `bhatti create --init "..."`) appears as a session named `init`. You can attach to it with [`bhatti shell`](/docs/reference/cli/exec/shell/) (passing its session ID via the API) to watch progress.
Sessions without an attached client and without recent I/O are eventually garbage-collected by an idle timer. Active sessions (producing output or accepting input) are never killed by the timer.
## Examples
```bash
bhatti ps dev
```
```text
ID COMMAND RUNNING ATTACHED
init npm install && npm run dev true false
s1 /bin/zsh -li true true
s2 ./long-running-worker.sh true false
```
```bash
bhatti ps dev --json | jq '.[] | select(.attached == false) | .session_id'
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti shell`](/docs/reference/cli/exec/shell/) — open or reattach to an interactive session
* [`bhatti exec --detach`](/docs/reference/cli/exec/exec/) — start a session in the background
* [API: `GET /sandboxes/:id/sessions`](/docs/reference/api/#list-sessions)
# bhatti shell
> Open an interactive shell inside a sandbox. Detach with Ctrl+\, reattach later, scrollback preserved.
Open an interactive terminal inside a sandbox. Aliased as `bhatti sh`.
## Synopsis
```
bhatti shell <sandbox> [--new]
bhatti sh <sandbox>
```
## Description
Opens a WebSocket connection to the sandbox's guest agent and attaches a PTY session. The default shell is `/bin/zsh`. Terminal resize is forwarded to the guest.
**Detach with `Ctrl+\`.** The shell keeps running; output flows into a 64 KB scrollback ring buffer in the guest. Reconnect with `bhatti shell <sandbox>` and you'll see the recent scrollback followed by live I/O. Detach does **not** send `SIGHUP` — child processes survive.
**Single attached client per session.** If you reattach from a second machine, the first client is disconnected. Use `--new` to force a brand-new session instead of reattaching.
**Clean exit vs. lost connection.** When the shell exits cleanly (Ctrl+D, `exit`), the CLI returns silently. Any other disconnection (network drop, server restart) shows:
```text
connection lost (session s3f9 still running)
reconnect: bhatti shell dev
```
The session keeps running in the guest. Reattach with the same command to pick it back up.
## Examples
```bash
bhatti shell dev
```
```bash
# Force a new session, even if one is already attached
bhatti shell dev --new
```
```bash
# Detach scenario:
bhatti shell dev # attach, run things, hit Ctrl+\
# ...session keeps running in the guest...
bhatti shell dev # reattach later, scrollback replayed
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--new` | `false` | Don't reattach to an existing session. Start a fresh one. The previous session, if any, is left running. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti exec`](/docs/reference/cli/exec/exec/) — non-interactive command execution
* [`bhatti ps`](/docs/reference/cli/exec/ps/) — list active sessions including this shell
* [`bhatti share`](/docs/reference/cli/networking/share/) — browser-based shell URL
* [API: `GET /sandboxes/:id/ws`](/docs/reference/api/#interactive-shell-websocket)
# File operations
> Read, write, and list files inside sandboxes.
Move bytes in and out of a sandbox without opening a shell. Useful for scripted setup, fetching results, and inspecting state.
| Command | Description |
| ------------------------------------------------------- | ----------------------------------------------- |
| [`bhatti file read`](/docs/reference/cli/files/read/) | Read a file from a sandbox to stdout. |
| [`bhatti file write`](/docs/reference/cli/files/write/) | Write a file from stdin into a sandbox. Atomic. |
| [`bhatti file ls`](/docs/reference/cli/files/ls/) | List directory contents inside a sandbox. |
The CLI exposes the simplest forms — read whole files, write whole files. The underlying API supports partial reads (`offset`, `limit`, `max_bytes`), permission control (`mode`), and HEAD-style stat requests; reach for it directly when you need those. See the [API reference](/docs/reference/api/#files) for the full surface.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
bhatti file read dev /workspace/app.js
echo 'config' | bhatti file write dev /etc/config
bhatti file ls dev /workspace/
bhatti file read dev /workspace/data.json | jq .
```
## Limits
[Section titled “Limits”](#limits)
* Writes are capped at **100 MB** per request.
* Directory listings are capped at **10 000 entries**; a sentinel entry indicates truncation.
* Writes are atomic — temp file, fsync, rename. Concurrent readers never see a partial file.
# bhatti file ls
> List directory contents inside a sandbox.
List directory contents inside a sandbox.
## Synopsis
```
bhatti file ls <sandbox> <path>
```
## Description
Lists files and subdirectories at the given absolute path. The default human format prints one row per entry: a directory flag, mode, size in bytes, and name.
Listings are capped at **10 000 entries**. If the directory has more entries than that, a sentinel row indicates truncation. For listings with this many entries you almost certainly want to be running `find` inside the sandbox via [`bhatti exec`](/docs/reference/cli/exec/exec/) and processing the results there instead.
## Examples
```bash
bhatti file ls dev /workspace
```
```text
drwxr-xr-x 4096 node_modules
-rw-r--r-- 1234 app.js
-rw-r--r-- 256 package.json
-rw-r--r-- 512 README.md
```
```bash
# JSON for scripting
bhatti file ls dev /workspace --json | jq '.[] | select(.is_dir == false) | .name'
```
## Output fields (JSON)
```json
[
{"name": "app.js", "size": 1234, "mode": "0644", "is_dir": false, "mtime": 1761792345},
{"name": "node_modules", "size": 4096, "mode": "0755", "is_dir": true, "mtime": 1761792500}
]
```
`mtime` is a Unix timestamp in seconds.
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti file read`](/docs/reference/cli/files/read/) / [`bhatti file write`](/docs/reference/cli/files/write/)
* [API: `GET /sandboxes/:id/files?ls=true`](/docs/reference/api/#list-directory)
# bhatti file read
> Read a file from a sandbox to stdout.
Read a file from a sandbox and write its bytes to stdout.
## Synopsis
```
bhatti file read <sandbox> <path>
```
## Description
Streams the file's contents to stdout, byte-for-byte. The path must be absolute. Cold sandboxes wake transparently before the read.
The CLI always reads the whole file. The API supports partial reads (`offset`, `limit`, `max_bytes` query params) for tailing logs without transferring the entire file — call it directly when you need that. Every response includes an `X-File-Size` header so clients can detect truncation. See [API: files](/docs/reference/api/#files).
## Examples
```bash
bhatti file read dev /workspace/app.js
```
```bash
# Pipe to other tools
bhatti file read dev /workspace/data.json | jq .name
```
```bash
# Save to a local file
bhatti file read dev /var/log/agent.log > agent-$(date +%F).log
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti file write`](/docs/reference/cli/files/write/) — opposite direction
* [`bhatti file ls`](/docs/reference/cli/files/ls/) — list a directory
* [API: `GET /sandboxes/:id/files`](/docs/reference/api/#read-a-file)
# bhatti file write
> Write a file into a sandbox from stdin. Atomic, capped at 100 MB.
Write a file into a sandbox. Reads bytes from stdin and writes them to the absolute `<path>` inside the sandbox.
## Synopsis
```
bhatti file write <sandbox> <path>
```
## Description
Writes are atomic — the guest agent writes to a temp file, fsyncs it, then renames it over the target. Concurrent readers never see partial content.
Writes are capped at **100 MB** per request. For larger files, split into chunks or attach a [persistent volume](/docs/reference/cli/volumes/) instead.
The CLI doesn't expose `--mode` for setting the file permissions (defaults to `0644`). The underlying [API endpoint](/docs/reference/api/#write-a-file) accepts a `mode=0755` query parameter — use it directly when you need to set a specific mode (e.g. `0700` for keys, `0755` for executables).
## Examples
```bash
echo 'console.log("hello")' | bhatti file write dev /workspace/app.js
```
```bash
# From a local file
bhatti file write dev /workspace/config.json < ./config.json
```
```bash
# Generate and write inline
date | bhatti file write dev /tmp/last-deploy.txt
```
```bash
# Set executable bit via the API directly
local_token="$BHATTI_TOKEN"
curl -X PUT "http://localhost:8080/sandboxes/dev/files?path=/usr/local/bin/run.sh&mode=0755" \
-H "Authorization: Bearer $local_token" \
--data-binary @./run.sh
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti file read`](/docs/reference/cli/files/read/) — opposite direction
* [`bhatti create --file local:guest`](/docs/reference/cli/sandbox/create/) — inject a file at boot, no separate write call
* [API: `PUT /sandboxes/:id/files`](/docs/reference/api/#write-a-file)
# Images
> Manage rootfs images. Pull from public OCI registries, import from Docker, save sandboxes as reusable images.
Images are ext4 filesystem snapshots used as sandbox root filesystems. bhatti ships pre-built tier images (`minimal`, `browser`, `docker`, `computer`); you can also pull from public OCI registries, import from Docker, or save a running sandbox’s filesystem as an image.
| Command | Description |
| ------------------------------------------------------------- | ---------------------------------------------------------- |
| [`bhatti image list`](/docs/reference/cli/images/list/) | List available images. |
| [`bhatti image pull`](/docs/reference/cli/images/pull/) | Pull from a public OCI registry. Async; returns a task ID. |
| [`bhatti image import`](/docs/reference/cli/images/import/) | Import a local Docker image, or a tarball with `--tar`. |
| [`bhatti image save`](/docs/reference/cli/images/save/) | Save a running sandbox’s rootfs as a reusable image. |
| [`bhatti image delete`](/docs/reference/cli/images/delete/) | Delete an image. |
| [`bhatti image share`](/docs/reference/cli/images/share/) | Share an image with specific users (server-only). |
| [`bhatti image unshare`](/docs/reference/cli/images/unshare/) | Revoke image access from users (server-only). |
## Built-in tiers
[Section titled “Built-in tiers”](#built-in-tiers)
| Tier | What’s in it | Size |
| ---------- | --------------------------------- | -------- |
| `minimal` | Bare Ubuntu 24.04 + curl + fuse3 | \~200 MB |
| `browser` | + Chromium, Playwright, Node 22 | \~600 MB |
| `docker` | + Docker Engine | \~550 MB |
| `computer` | + XFCE desktop, KasmVNC, Chromium | \~1.5 GB |
```bash
bhatti create --name scraper --image browser
bhatti create --name ci --image docker
```
The server auto-discovers tiers from `/var/lib/bhatti/images/`. Install more with `sudo bhatti update --tiers all`.
## Image scoping
[Section titled “Image scoping”](#image-scoping)
Each image is owned by the user who created it (or by the system, for built-in tiers). Other users can’t see or use your custom images by default.
To share with a specific user: [`sudo bhatti image share <image> --user alice`](/docs/reference/cli/images/share/). The command operates directly on the local SQLite database, so it requires running on the server with DB access — it isn’t an API call you can make remotely.
Use `--list` to inspect current shares; pass `--user alice --user bob` to share with multiple users in one call. [`unshare`](/docs/reference/cli/images/unshare/) revokes access by user.
# bhatti image delete
> Delete an image. Cannot be undone.
Delete an image.
## Synopsis
```
bhatti image delete <name> [-y]
```
## Description
Removes the image record and the underlying ext4 file. **Cannot be undone.**
Sandboxes that were created from this image keep running — they have their own copy-on-write rootfs derived from the image at creation time. They don't reference the original after boot.
By default, `delete` prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
You cannot delete a system tier image (`minimal`, `browser`, etc.) — those are owned by `user_id=""` and only the operator can remove them by removing the file from `/var/lib/bhatti/images/`.
## Examples
```bash
bhatti image delete python-3.12
```
```text
Delete image "python-3.12"? [y/N]: y
deleted
```
```bash
# Non-interactive
bhatti image delete python-3.12 -y
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti image list`](/docs/reference/cli/images/list/)
* [API: `DELETE /images/:name`](/docs/reference/api/#delete-an-image)
# bhatti image import
> Import a local Docker image (via docker save) or a raw tarball.
Import an image from a local Docker daemon, or directly from a tarball without Docker.
## Synopsis
```
bhatti image import <docker-ref> [--name <string>]
bhatti image import --tar <path> --name <string>
```
## Description
Two modes:
**Docker mode** (default). The CLI runs `docker save <ref>` on your machine and streams the resulting tarball to the bhatti server, which converts it to an ext4 rootfs. Requires Docker installed locally. The image must already be in your local Docker daemon — pull it first with `docker pull` if needed.
**Tarball mode** (`--tar`). Skip Docker entirely. The CLI reads the file at `<path>` and streams it to the server. The tarball must be in the format produced by `docker save` (an OCI layout or v1 image manifest). `--name` is required in this mode since there's no ref to derive the name from.
If you don't pass `--name` in Docker mode, a name is derived from the ref the same way as `image pull`: `python:3.12` → `python-3.12`.
For private registries, this is the recommended path — `docker pull` handles authentication on your machine using your existing Docker credentials, then bhatti just receives the bytes:
```bash
docker pull ghcr.io/org/private:latest
bhatti image import ghcr.io/org/private:latest
```
## Examples
```bash
# Import from local Docker (name derived from ref)
bhatti image import python:3.12
```
```bash
# Private image: pull with Docker first, then import
docker pull ghcr.io/org/private:latest
bhatti image import ghcr.io/org/private:latest
```
```bash
# Locally-built image
docker build -t my-env .
bhatti image import my-env
```
```bash
# Custom name override
bhatti image import python:3.12 --name py312
```
```bash
# Raw tarball — no Docker needed
docker save ubuntu:24.04 -o /tmp/ubuntu.tar
bhatti image import --tar /tmp/ubuntu.tar --name ubuntu-24
```
## Failure modes
* **Docker not found**: install Docker, or use `--tar` with a tarball.
* **Image not in local Docker**: `docker pull <ref>` first, or use `--tar` with a saved tarball.
* **Server rejects the upload**: image is too large or malformed; check stderr for the server message.
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | derived from ref (Docker mode); required (`--tar` mode) | Image name to register on the server. | | `--tar <path>` | — | Skip Docker; stream the given tarball directly. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti image pull`](/docs/reference/cli/images/pull/) — server-side pull from public registries (no Docker needed)
* [`bhatti image save`](/docs/reference/cli/images/save/) — save a sandbox's filesystem as an image
* [API: `POST /images/import`](/docs/reference/api/#import-an-image)
# bhatti image list
> List all rootfs images available to the authenticated user.
List all rootfs images available to the authenticated user.
## Synopsis
```
bhatti image list
```
## Description
Lists images in three scopes:
* **System tiers** (`minimal`, `browser`, `docker`, `computer`) — visible to everyone.
* **Your own images** — created via `image pull`, `image import`, or `image save`.
* **Images shared with you** by another user.
## Examples
```bash
bhatti image list
```
```text
ID NAME SOURCE SIZE SCOPE
img_abc123 minimal built-in 186MB admin
img_def456 browser built-in 612MB admin
img_ghi789 python-3.12 oci:python:3.12 284MB user
img_jkl012 my-stack snapshot:dev 512MB user
```
```bash
# JSON for scripting
bhatti image list --json | jq '.[] | select(.user_id != "") | .name'
```
## Output fields (JSON)
```json
[
{"id": "img_abc123", "name": "minimal", "source": "built-in", "size_mb": 186, "user_id": ""},
{"id": "img_jkl012", "name": "my-stack", "source": "snapshot:dev", "size_mb": 512, "user_id": "usr_..."}
]
```
`user_id` is empty for system images; non-empty values denote ownership.
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti image pull`](/docs/reference/cli/images/pull/) / [`import`](/docs/reference/cli/images/import/) / [`save`](/docs/reference/cli/images/save/)
* [API: `GET /images`](/docs/reference/api/#list-images)
# bhatti image pull
> Pull a public OCI/Docker image and convert it to a bhatti rootfs.
Pull a public image from any OCI-compatible registry. The server fetches the image, flattens its layers, and writes them to an ext4 file usable as a sandbox rootfs.
## Synopsis
```
bhatti image pull <ref> [--name <string>] [--auth <user:token>]
```
## Description
The pull runs asynchronously on the server. The CLI returns a task ID, then polls `/tasks/<id>` every 2 seconds and prints progress until the task completes or fails. **Press `Ctrl+C` to detach** — the pull continues on the server. Check progress later with `bhatti image list` (the image appears once it's ready).
If you don't pass `--name`, a name is derived from the ref:
* `python:3.12` → `python-3.12`
* `node:22-slim` → `node-22-slim`
* `ghcr.io/myorg/sandbox-base:latest` → `sandbox-base-latest`
`--auth` enables a single-shot HTTP basic auth header (`user:token`). For real private registries with rotating credentials, prefer the [`image import`](/docs/reference/cli/images/import/) flow — `docker pull` handles auth on your machine, then bhatti just receives the tarball.
If the same image was previously pulled with the same content digest, the server returns immediately without re-downloading.
## Examples
```bash
bhatti image pull python:3.12
```
```text
pulling python:3.12 as "python-3.12" (task: tsk_abc123)
resolving manifest...
downloading layers (45 MB / 142 MB)...
flattening to ext4...
done
```
```bash
# Custom name
bhatti image pull node:22-slim --name node-22
# Detach: press Ctrl+C after seeing the task ID
bhatti image pull ubuntu:24.04
^C
Interrupted. Pull continues on server.
Check status: bhatti image list
```
```bash
# Use the pulled image
bhatti create --name py --image python-3.12
```
## Authentication
For private registries, `--auth` works for simple `user:token` schemes. If the pull fails with `unauthorized` / `403`, the CLI prints a hint suggesting you use `image import` instead:
```text
This image may require authentication.
Pull it with Docker locally, then import:
docker pull ghcr.io/org/private:latest
bhatti image import ghcr.io/org/private:latest
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | derived from ref | Image name to register on the server. | | `--auth <user:token>` | — | HTTP basic auth credentials for the registry. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti image import`](/docs/reference/cli/images/import/) — local Docker / tarball import (handles private registries via local auth)
* [`bhatti create --image`](/docs/reference/cli/sandbox/create/) — use the pulled image
* [API: `POST /images/pull`](/docs/reference/api/#pull-an-image)
* [API: `GET /tasks/:id`](/docs/reference/api/#poll-an-async-task)
# bhatti image save
> Save a running sandbox's rootfs as a reusable image.
Snapshot a running sandbox's filesystem and save it as a reusable image.
## Synopsis
```
bhatti image save <sandbox> --name <image-name>
```
## Description
Captures the current state of the sandbox's rootfs (every file written since boot) and writes it to a new ext4 image owned by the user. The sandbox keeps running; the save is read-only from its perspective.
Only the rootfs is captured — attached persistent volumes are **not** included. If your sandbox stores work in a volume, save the volume separately with [`bhatti volume clone`](/docs/reference/cli/volumes/clone/) and recreate the attachment when stamping out new sandboxes from the saved image.
The saved image is private to the user. To make it available to other users, use [`bhatti image share`](/docs/reference/cli/images/share/) (server-only).
## Examples
```bash
# Build a custom environment, then save it
bhatti create --name build --image minimal
bhatti exec build -- apt-get update && apt-get install -y nodejs
bhatti exec build -- npm install -g pnpm
bhatti image save build --name node-app
# saved "node-app" (512MB)
```
```bash
# Stamp out workers from the saved image
bhatti create --name worker-1 --image node-app
bhatti create --name worker-2 --image node-app
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | (required) | Name for the saved image. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume clone`](/docs/reference/cli/volumes/clone/) — for snapshotting volume data alongside an image
* [`bhatti snapshot create`](/docs/reference/cli/snapshots/create/) — full VM snapshot (memory + disk + processes), not just filesystem
* [API: `POST /sandboxes/:id/save-image`](/docs/reference/api/#save-a-sandbox-as-an-image)
# bhatti image share
> Share an image with specific users. Operates directly on the local SQLite database; requires DB access (server-only).
# bhatti image share server only
Share an image with one or more specific users.
## Synopsis
```
bhatti image share <image-name> --user <name> [--user <name>]...
bhatti image share <image-name> --list
```
## Description
Adds an entry to the image-share table for each `<user>` so that user can reference the image with `--image <name>` from their sandbox creation calls. Sharing is per-user; there's no "public" or "everyone" mode.
The command operates directly on the server's SQLite database — it isn't an HTTP API call. You must run it on the server, with access to `/var/lib/bhatti/state.db` (typically as `sudo`).
`--list` prints the users an image is currently shared with. `--unshare` is its own command, [`bhatti image unshare`](/docs/reference/cli/images/unshare/).
## Examples
```bash
# Share with two users
sudo bhatti image share spc-golden --user kowshik --user sumo
```
```text
shared "spc-golden" with: kowshik, sumo
```
```bash
# Inspect current shares
sudo bhatti image share spc-golden --list
```
```text
spc-golden shared with: kowshik, sumo
```
```bash
# Image not yet shared
sudo bhatti image share spc-golden --list
```
```text
spc-golden: not shared
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--user <name>` | — | User to share with. Repeatable. Required unless `--list`. | | `--list` | `false` | Print current shares for the image instead of adding new ones. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## Failure modes
* **Image not found**: bhatti can't see your image because of name typos or because it's owned by a different user. Check with `bhatti image list`.
* **User not found**: the `<name>` doesn't match any user. Check with `bhatti user list`.
* **No `--user`**: the command refuses to do anything without an explicit user list (or `--list`). There's no global "share with all" mode.
## See also
* [`bhatti image unshare`](/docs/reference/cli/images/unshare/) — revoke access
* [`bhatti user list`](/docs/reference/cli/admin/user-list/) — find usernames
# bhatti image unshare
> Revoke image access from specific users. Server-only.
# bhatti image unshare server only
Revoke an image's share entries for one or more users.
## Synopsis
```
bhatti image unshare <image-name> --user <name> [--user <name>]...
```
## Description
Removes share-table entries for the listed users. After unsharing, those users will no longer see the image in [`bhatti image list`](/docs/reference/cli/images/list/) and can't reference it by name in `bhatti create --image`.
Sandboxes the affected users already created from the image keep running — they have their own copy-on-write rootfs and don't reference the original.
Like [`image share`](/docs/reference/cli/images/share/), this operates directly on the local SQLite database. Requires DB access on the server (typically `sudo`).
## Examples
```bash
sudo bhatti image unshare spc-golden --user sumo
```
```text
unshared "spc-golden" from: sumo
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--user <name>` | (required) | User to revoke. Repeatable. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti image share`](/docs/reference/cli/images/share/) — add a share
* [`bhatti image share --list`](/docs/reference/cli/images/share/) — see who has access
# Networking & sharing
> Publish ports with public URLs, share browser-based shells.
Expose sandbox services to the world (or a teammate) without an API key on the receiver’s side.
| Command | Description |
| --------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| [`bhatti publish`](/docs/reference/cli/networking/publish/) | Publish a sandbox port at a public URL. Cold sandboxes wake on the first request. |
| [`bhatti unpublish`](/docs/reference/cli/networking/unpublish/) | Remove a published port. |
| [`bhatti share`](/docs/reference/cli/networking/share/) | Generate a one-shot URL that opens an interactive terminal in the browser. |
Both `publish` and `share` produce URLs that don’t require an API key. Treat them like any other secret — anyone with the URL has access.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
# Publish on a friendly subdomain
bhatti publish dev -p 3000 -a my-app
# → https://my-app.bhatti.sh
# Auto-generated alias (random suffix prevents URL guessing)
bhatti publish dev -p 3000
# → https://dev-k3m9x2.bhatti.sh
# Publish + give a teammate a browser shell in one call
bhatti publish dev -p 3000 -a demo --shell
# Share a sandbox shell over the web
bhatti share dev
# → https://api.bhatti.sh/_shell/<id>#token=...
# Revoke any active share link
bhatti share dev --revoke
# Tear down a published port
bhatti unpublish dev -p 3000
```
## Domain mode vs. path-based
[Section titled “Domain mode vs. path-based”](#domain-mode-vs-path-based)
Public URLs at `https://<alias>.<your-zone>` only work when the server is configured with a [custom domain](/docs/managing/custom-domain/). Without one, `bhatti publish` falls back to a path-based proxy:
```text
http://<api-host>/sandboxes/<id>/proxy/<port>/
```
That URL still works, but it requires Bearer auth. Useful for development; not suitable for sharing publicly. See [`bhatti publish`](/docs/reference/cli/networking/publish/) for the exact fallback behaviour.
# bhatti publish
> Publish a sandbox port with a public URL. Cold sandboxes wake on first request.
Publish a sandbox port at a public URL.
## Synopsis
```
bhatti publish <sandbox> -p <port> [-a <alias>] [--shell]
```
## Description
Creates a routing rule on the bhatti server: incoming requests to `https://<alias>.<proxy-zone>` get reverse-proxied to `localhost:<port>` inside the sandbox.
**Aliases.** With `-a my-app`, the URL is `https://my-app.<proxy-zone>` exactly. Without `-a`, an alias is generated from the sandbox name with a random suffix (`dev-k3m9x2`) to prevent URL guessing — useful for ephemeral previews.
**Cold sandboxes wake automatically.** The first request after a sandbox went cold takes \~50ms longer than usual; subsequent requests are full speed.
**No auth.** Published URLs are public. Anyone with the URL can reach the port. There's no per-URL access control — share carefully.
**HTTP and WebSocket.** Both work through the proxy. WebSocket idle timeout is 10 minutes; HTTP body cap is 50 MB per request.
**`--shell` combo.** Pass `--shell` to also generate a [web shell token](/docs/reference/cli/networking/share/) in the same call. Useful for "send a teammate a preview URL and a debug terminal in one click".
**Without a custom domain configured.** If the server isn't running in [domain mode](/docs/managing/custom-domain/), there's no `<alias>.<zone>` URL. The CLI prints a fallback hint:
```text
Publishing requires a custom domain.
Setup: https://bhatti.sh/docs/managing/custom-domain
In the meantime, use the proxy URL directly:
http://<api-host>/sandboxes/<id>/proxy/<port>/
```
The proxy URL works but requires Bearer auth, so it's developer-only.
## Examples
```bash
# Friendly subdomain
bhatti publish dev -p 3000 -a my-app
# Published: https://my-app.bhatti.sh
```
```bash
# Auto-generated alias
bhatti publish dev -p 3000
# Published: https://dev-k3m9x2.bhatti.sh
```
```bash
# Preview URL + browser shell, single call
bhatti publish dev -p 3000 -a demo --shell
# Published: https://demo.bhatti.sh
# Shell: https://api.bhatti.sh/_shell/<id>#token=...
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-p`, `--port <int>` | (required) | Port inside the sandbox to publish. | | `-a`, `--alias <string>` | auto-generated | Custom subdomain alias. Without this, an alias derived from the sandbox name with a random suffix is used. | | `--shell` | `false` | Also generate a [web shell URL](/docs/reference/cli/networking/share/) in the same call. The shell URL is printed alongside the published URL. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti unpublish`](/docs/reference/cli/networking/unpublish/) — remove a published port
* [`bhatti share`](/docs/reference/cli/networking/share/) — web shell URL on its own
* [Custom domain](/docs/managing/custom-domain/) — required for `<alias>.<zone>` URLs
* [API: `POST /sandboxes/:id/publish`](/docs/reference/api/#publish-a-port)
# bhatti share
> Generate a one-shot URL that opens an interactive terminal in the browser.
Generate a URL that opens an interactive terminal in the browser. No CLI install or API key needed on the receiver's side.
## Synopsis
```
bhatti share <sandbox> [--revoke]
```
## Description
Mints a fresh token bound to the sandbox and embeds it in the URL fragment (`#token=...`). The fragment is **not** sent to the server in HTTP request lines, so the token doesn't appear in access logs or referer headers — but anyone with the URL has shell access until the token is revoked.
**Each call invalidates the previous token.** Calling `bhatti share dev` again generates a new URL; the old one stops working. Only one share link is active at a time per sandbox.
**Scope.** A share token grants access *only* to the specific sandbox it was minted for. It is not interchangeable with the user's API key.
**Behaviour in the browser.** The page opens an xterm.js terminal connected via WebSocket; it's a full PTY session, identical to [`bhatti shell`](/docs/reference/cli/exec/shell/). Closing the tab detaches; the session keeps running and scrollback is preserved. Reopening the URL reattaches.
**Revoke** with `--revoke` — the token is deleted server-side and the URL stops working.
## Examples
```bash
bhatti share dev
```
```text
Shell: https://api.bhatti.sh/_shell/a1b2c3d4#token=k3m9x2qr...
```
```bash
# JSON for scripting
bhatti share dev --json
```
```json
{
"url": "https://api.bhatti.sh/_shell/a1b2c3d4#token=k3m9x2qr...",
"token": "k3m9x2qr..."
}
```
```bash
# Revoke
bhatti share dev --revoke
```
```text
Shell access revoked.
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--revoke` | `false` | Delete the active token for this sandbox. The shared URL stops working immediately. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti publish --shell`](/docs/reference/cli/networking/publish/) — preview URL + share URL in one call
* [`bhatti shell`](/docs/reference/cli/exec/shell/) — terminal-based interactive shell
* [`bhatti shell`](/docs/reference/cli/exec/shell/) — the CLI counterpart
* [API: `POST /sandboxes/:id/shell-token`](/docs/reference/api/#mint-a-shell-token)
# bhatti unpublish
> Remove a published port. The URL stops working immediately.
Remove a published port. The URL stops working immediately.
## Synopsis
```
bhatti unpublish <sandbox> -p <port>
```
## Description
Deletes the routing rule. New requests to the previously published URL get a 404 from the proxy. In-flight WebSocket connections continue until the client closes them.
Publish rules are also automatically cleaned up when a sandbox is destroyed — `unpublish` is only needed when you want to keep the sandbox but stop exposing the port.
## Examples
```bash
bhatti unpublish dev -p 3000
```
```text
Unpublished port 3000
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-p`, `--port <int>` | (required) | Port to unpublish. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti publish`](/docs/reference/cli/networking/publish/)
* [API: `DELETE /sandboxes/:id/publish/:port`](/docs/reference/api/#unpublish-a-port)
# Sandbox commands
> Lifecycle commands for sandbox VMs — create, list, inspect, edit, stop, start, destroy.
Sandboxes are isolated Linux VMs. These commands cover the full lifecycle: create them, look at them, change mutable settings, pause and resume them on disk, and tear them down.
| Command | Description |
| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| [`bhatti create`](/docs/reference/cli/sandbox/create/) | Create a new sandbox VM. Optional flags cover CPU, memory, env vars, init scripts, volumes, secrets, and file injection. |
| [`bhatti list`](/docs/reference/cli/sandbox/list/) | List your sandboxes (alias: `ls`). `-o wide` adds resource columns. |
| [`bhatti inspect`](/docs/reference/cli/sandbox/inspect/) | Show full details for one sandbox (alias: `info`) — kubectl-`describe` style. |
| [`bhatti edit`](/docs/reference/cli/sandbox/edit/) | Toggle mutable settings. Currently only `--keep-hot` / `--allow-cold`. |
| [`bhatti stop`](/docs/reference/cli/sandbox/stop/) | Snapshot to disk and free memory. |
| [`bhatti start`](/docs/reference/cli/sandbox/start/) | Resume from snapshot. Continues exactly where it left off. |
| [`bhatti destroy`](/docs/reference/cli/sandbox/destroy/) | Permanently destroy a sandbox (alias: `rm`). Volumes are detached, not deleted. |
All commands accept either the sandbox name or its ID. Names are easier; IDs are stable across renames.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
bhatti create --name dev --cpus 2 --memory 1024
bhatti ls -o wide
bhatti inspect dev
bhatti exec dev -- npm install
bhatti stop dev # save state, free memory
bhatti start dev # resume from where it stopped
bhatti destroy dev -y
```
In normal operation you don’t need `stop` and `start` — idle sandboxes are paused and resumed automatically. See [Thermal states](/docs/under-the-hood/thermal-states/) for the engineering. Use them to control thermal state explicitly when you know a sandbox will be idle for a while.
## See also
[Section titled “See also”](#see-also)
* [Execution & shells](/docs/reference/cli/exec/) — `exec`, `shell`, `ps`, `ports`
* [Files](/docs/reference/cli/files/) — read/write files inside a sandbox
* [Networking & sharing](/docs/reference/cli/networking/) — `publish`, `share`
* [API: `POST /sandboxes`](/docs/reference/api/#create-a-sandbox)
# bhatti create
> Create a new sandbox VM with custom resources, environment, init script, volumes, and secrets.
Create a new sandbox VM. Each sandbox is an isolated Linux environment with its own kernel, filesystem, and network interface — created in seconds, paused for free, resumed in microseconds.
## Synopsis
```
bhatti create [flags]
```
## Description
Boots a krucible microVM and registers it under the authenticated user. Returns once the guest agent is responding.
The created sandbox is in the `running` (hot) thermal state. Without `--keep-hot`, it pauses to *warm* after 30 seconds of inactivity and freezes to *cold* (memory snapshotted to disk, RAM freed) after 30 minutes. Any incoming request transparently wakes it.
`--volume` and `--secret` reference resources that must already exist (create them with [`bhatti volume create`](/docs/reference/cli/volumes/create/) and [`bhatti secret set`](/docs/reference/cli/secrets/set/)). `--file` reads a local file at submit time and writes its bytes to the guest path during boot — convenient for injecting config files without a volume.
When used with `--template`, request-side `--secret` and `--file` are merged into the template's defaults: secrets from both lists are resolved (request-side names that don't exist in your secret store return `400`); `--file` adds to the template (templates have no files of their own). For environment variables, `--secret` wins over `--env` for the same name.
`--cpus` accepts fractional values (e.g. `0.5`, `1.5`); the host scheduler runs vCPUs as time-shared threads, so fractional allocation maps to "you're entitled to \~N vCPU-seconds per second."
`--hugepages` uses 2MB hugepages for the VM's memory, which makes boot faster. Use it for short-lived sandboxes, not for long-lived agents.
The `--memory` server default is 1024 MB. The `--cpus` default is 1. Both are capped by the user's per-sandbox limits set via [`bhatti user create`](/docs/reference/cli/admin/user-create/).
## Examples
```bash
# Smallest possible sandbox
bhatti create --name dev
```
```bash
# Custom resources, env vars, and an init script
bhatti create --name api \
--cpus 2 --memory 2048 \
--env NODE_ENV=production,LOG_LEVEL=info \
--init "cd /workspace && npm install"
```
```bash
# Mount a persistent volume; create the volume first
bhatti volume create --name workspace --size 5120
bhatti create --name dev --volume workspace:/workspace
```
```bash
# Reference a stored secret as a guest env var
bhatti secret set OPENAI_KEY sk-...
bhatti create --name agent --secret OPENAI_KEY
```
```bash
# Inject a local config file at boot
bhatti create --name worker --file ./config.json:/etc/worker/config.json
```
```bash
# Use a non-default rootfs image (browser tier with Chromium + Playwright)
bhatti create --name scraper --image browser
```
```bash
# Autonomous agent — never paused, even when idle
bhatti create --name agent --init "hermes gateway" --keep-hot
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | auto-generated | Sandbox name. Must match `[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}`. | | `--cpus <float>` | `1` | Number of vCPUs. Fractional values allowed (e.g. `0.5`). Capped by per-sandbox limit. | | `--memory <int>` | `0` (server uses `1024`) | Memory in MB. `0` means use the server default. Capped by per-sandbox limit. | | `--disk-size <int>` | `0` (use image size) | Rootfs disk size in MB. `0` means inherit from the base image. | | `--env <K=V,K=V>` | — | Environment variables, comma-separated, applied to every command. | | `--init <string>` | — | Init script that runs at boot as an attachable session named `init`. Survives the boot path; visible in [`bhatti ps`](/docs/reference/cli/exec/ps/). | | `--keep-hot` | `false` | Disable thermal transitions. Use for sandboxes that maintain external connections (websockets, polling agents). | | `--hugepages` | `false` | Use 2MB hugepages. Faster boot. | | `--template <name>` | — | Create from a template. Template fields are defaults; flags here override them. | | `--image <name>` | — | Rootfs image. Built-in: `minimal`, `browser`, `docker`, `computer`. Or a name from [`bhatti image list`](/docs/reference/cli/images/list/). | | `--volume <spec>` | — | Attach a persistent volume. Format: `name:mount[:ro]`. Repeatable. | | `--secret <name>` | — | Reference a stored secret by name. Decrypted and exposed as an env var. Repeatable. | | `--file <local:guest>` | — | Read a local file and write it to `guest` inside the sandbox at boot. Repeatable. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## Output
Default output is a verbose summary:
```
sandbox/dev created (1 vCPU, 1024 MB)
Shell: bhatti shell dev
```
With `--json`, the full sandbox object is printed.
If a sandbox with the requested name already exists *and is identical*, the server returns the existing record and the CLI prints:
```
sandbox/dev unchanged (already exists)
```
This makes `create` safe to re-run from scripts.
## Exit codes
\| Code | Meaning | | ---- | ------- | | `0` | Sandbox created, or already existed with the same configuration. | | `1` | Validation error, quota exceeded (e.g. `max-sandboxes` reached), name conflict, or boot failure. |
## See also
* [`bhatti list`](/docs/reference/cli/sandbox/list/) — find your sandboxes
* [`bhatti exec`](/docs/reference/cli/exec/exec/) — run a command in the new sandbox
* [`bhatti shell`](/docs/reference/cli/exec/shell/) — open an interactive shell
* [`bhatti destroy`](/docs/reference/cli/sandbox/destroy/) — clean up
* [API: `POST /sandboxes`](/docs/reference/api/#create-a-sandbox)
# bhatti destroy
> Permanently destroy a sandbox and all its data.
Permanently destroy a sandbox. Aliased as `bhatti rm`.
## Synopsis
```
bhatti destroy <sandbox> [-y]
bhatti rm <sandbox> [-y]
```
## Description
Kills the `bhatti-vmm` process, removes the CoW rootfs overlay and snapshot files, and removes the sandbox record from the database. **This cannot be undone.**
Persistent volumes attached to the sandbox are detached but **not** deleted — their data survives. Use [`bhatti volume delete`](/docs/reference/cli/volumes/delete/) to remove a volume. Published URLs are automatically cleaned up.
By default, destroy prompts for confirmation. In a non-interactive shell without `-y`, the command refuses to proceed and exits `1`. Pass `-y` / `--yes` to skip the prompt.
## Examples
```bash
bhatti destroy dev
```
```text
Destroy sandbox "dev"? [y/N]: y
sandbox/dev destroyed
```
```bash
# Non-interactive (CI / scripts)
bhatti destroy dev -y
```
```bash
# Tear down everything you own
bhatti list --json | jq -r '.[].name' | xargs -I{} bhatti destroy {} -y
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip the interactive confirmation. Required for non-interactive use. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## Exit codes
\| Code | Meaning | | ---- | ------- | | `0` | Destroyed. | | `1` | Sandbox not found, or non-interactive shell without `-y`. |
## See also
* [`bhatti stop`](/docs/reference/cli/sandbox/stop/) — pause without destroying (data preserved)
* [`bhatti volume delete`](/docs/reference/cli/volumes/delete/) — detached volumes don't go with the sandbox
* [API: `DELETE /sandboxes/:id`](/docs/reference/api/#destroy-a-sandbox)
# bhatti edit
> Update mutable settings on a sandbox. Currently keep-hot only.
Update mutable settings on a running or stopped sandbox.
## Synopsis
```
bhatti edit <sandbox> --keep-hot
bhatti edit <sandbox> --allow-cold
```
## Description
Currently the only mutable setting is the thermal-management toggle. Resource fields like `cpus` and `memory_mb` are immutable after creation — destroy and re-create instead.
Setting `--keep-hot` on a stopped sandbox immediately wakes it, since "kept hot" implies running. Setting `--allow-cold` doesn't pause the sandbox; it just re-enables the thermal manager's automatic transitions.
`--keep-hot` and `--allow-cold` are mutually exclusive.
## Examples
```bash
# Pin an autonomous agent so it's never paused
bhatti edit my-agent --keep-hot
```
```bash
# Re-enable automatic thermal transitions
bhatti edit my-agent --allow-cold
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--keep-hot` | `false` | Disable thermal transitions. Wakes the sandbox if it was stopped. | | `--allow-cold` | `false` | Re-enable thermal transitions. Does not actively pause the sandbox. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti create --keep-hot`](/docs/reference/cli/sandbox/create/) — set this at creation
* [Thermal states](/docs/under-the-hood/thermal-states/) — how hot/warm/cold transitions work
* [API: `PATCH /sandboxes/:id`](/docs/reference/api/#update-a-sandbox)
# bhatti inspect
> Show full details for a single sandbox. Aliased as bhatti info.
Show full details for one sandbox in `kubectl describe`-style output. Aliased as `bhatti info`.
## Synopsis
```
bhatti inspect <sandbox>
bhatti info <sandbox>
```
## Description
Prints status, thermal state, image, creation/stop timestamps, resource allocation, network info, and `keep_hot`. For sandboxes in the `running` state, disk usage is queried live by running `df -m /` inside the guest — the row shows total / used / free in MB.
krucible sandboxes report no guest IP, so the `IP` line is empty — reach a sandbox through a published URL or the reverse proxy, not a guest address.
Stopped sandboxes show their static disk size; the live `df` query is skipped to avoid waking them.
Use `--json` to get the full sandbox record as JSON for scripting.
## Examples
```bash
bhatti inspect dev
```
```text
Name: dev
ID: a1b2c3d4
Status: running
Thermal: hot
Image: minimal
Created: 2026-04-29T15:21:48Z
Resources:
CPUs: 2
Memory: 1024 MB
Disk: 8192 MB (1247 MB used, 6945 MB free)
Network:
IP:
```
```bash
bhatti info agent --json | jq .keep_hot
```
```text
true
```
```bash
# Stopped sandbox — no live df, static disk size shown
bhatti inspect scraper
```
```text
Name: scraper
ID: e5f6g7h8
Status: stopped
Thermal: cold
Image: browser
Created: 2026-04-22T10:14:03Z
Stopped: 2026-04-29T08:55:12Z
Resources:
CPUs: 1
Memory: 2048 MB
Disk: — (stopped)
Network:
IP:
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti list`](/docs/reference/cli/sandbox/list/) — list all sandboxes
* [`bhatti edit`](/docs/reference/cli/sandbox/edit/) — change mutable settings
* [API: `GET /sandboxes/:id`](/docs/reference/api/#get-a-sandbox)
# bhatti list
> List sandboxes you own. Adds a wide format and JSON output.
List sandboxes owned by the authenticated user. Aliased as `bhatti ls`.
## Synopsis
```
bhatti list [-o wide]
bhatti ls [-o wide]
```
## Description
Returns one row per sandbox with status, thermal state, and the first published URL (if any). The default columns are tuned for fitting in a normal terminal; `-o wide` adds CPUs, memory, disk, and the rootfs image.
The `IP` column is empty under krucible: sandboxes have no host-visible guest IP — you reach a service inside one by publishing it (`bhatti publish`) or forwarding a host port to it (`bhatti forward`), not by dialing a guest address. The column is retained for output compatibility.
Listing does not wake cold sandboxes — the server reads from its database, not from the engines. The `THERMAL` column shows the current state (`hot`, `warm`, `cold`).
As a side effect, `list` refreshes the local sandbox-name completion cache (`/tmp/bhatti-completions-<uid>`). Tab-completing sandbox names will pick up new sandboxes after the first `list` call following a `create` from another machine.
## Examples
```bash
bhatti list
```
```text
NAME STATUS THERMAL IP
dev running hot
agent running cold
scraper stopped -
```
```bash
bhatti ls -o wide
```
```text
NAME STATUS THERMAL IP CPUS MEMORY DISK IMAGE
dev running hot 1 1024 0 minimal
agent running cold 2 4096 8192 browser
scraper stopped - 1 1024 0 minimal
```
```bash
# Machine-readable output
bhatti ls --json | jq '.[] | select(.status == "running") | .name'
```
```bash
# When at least one sandbox has a published URL, an extra URL column appears
bhatti list
```
```text
NAME STATUS THERMAL IP URL
api running hot https://api.bhatti.sh
dev running hot
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-o`, `--output <fmt>` | (default columns) | Output format. Currently only `wide` is supported. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti inspect`](/docs/reference/cli/sandbox/inspect/) — full details for a single sandbox
* [`bhatti ps`](/docs/reference/cli/exec/ps/) — sessions inside a sandbox
* [API: `GET /sandboxes`](/docs/reference/api/#list-sandboxes)
# bhatti start
> Resume a stopped sandbox from its snapshot.
Resume a stopped sandbox from its on-disk snapshot. The VM continues exactly where it left off — same processes, same memory, same network connections.
## Synopsis
```
bhatti start <sandbox> [--force]
```
## Description
Restores the VM from its on-disk snapshot bundle (`memory.img` + `checkpoint.bin` + `manifest.json`). Typical cold restore is sub-second (\~380ms measured on Apple Silicon); a warm sandbox — one whose memory is still resident — resumes in \~4ms.
If a previous restore failed (for example because the underlying rootfs file was corrupt or the VMM helper couldn't be launched), the sandbox enters the `unknown` state and `bhatti start` refuses to retry by default. Pass `--force` to retry — useful after fixing the underlying issue.
You normally don't need to call `start` explicitly. Any API request — `exec`, `file read`, an HTTP request to a published URL — wakes a cold sandbox transparently before executing.
## Examples
```bash
bhatti start dev
```
```text
sandbox/dev started
```
```bash
# Retry after a previously failed restore
bhatti start dev --force
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--force` | `false` | Retry start even if the sandbox is in an `unknown` state from a previously failed restore. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti stop`](/docs/reference/cli/sandbox/stop/) — snapshot and stop
* [Thermal states](/docs/under-the-hood/thermal-states/) — automatic resume
* [API: `POST /sandboxes/:id/start`](/docs/reference/api/#start-a-sandbox)
# bhatti stop
> Snapshot a sandbox to disk and free its memory.
Snapshot a sandbox to disk and free its memory. The VM keeps its identity and on-disk state; resume with [`bhatti start`](/docs/reference/cli/sandbox/start/).
## Synopsis
```
bhatti stop <sandbox>
```
## Description
Pauses the VM, dumps memory + CPU state to disk, then frees the host RAM. Every `stop` takes a full memory snapshot; there are no diff snapshots.
Stopped sandboxes use zero CPU and zero memory on the host. Their disk footprint includes the rootfs, attached volumes, and the snapshot bundle (`memory.img`, `checkpoint.bin`, `manifest.json`).
In normal operation you don't need this — idle sandboxes pause and resume automatically per the thermal manager. Use `bhatti stop` to control thermal state explicitly when you know a sandbox will be idle for a while and want to free memory immediately.
## Examples
```bash
bhatti stop dev
```
```text
sandbox/dev stopped
```
```bash
# Resume later
bhatti start dev
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti start`](/docs/reference/cli/sandbox/start/) — resume from snapshot
* [`bhatti snapshot create`](/docs/reference/cli/snapshots/create/) — named snapshot you can resume into a *new* sandbox
* [Thermal states](/docs/under-the-hood/thermal-states/) — automatic pause/resume
* [API: `POST /sandboxes/:id/stop`](/docs/reference/api/#stop-a-sandbox)
# Secrets
> Encrypted secrets injected into sandboxes as environment variables.
Secrets are key-value pairs encrypted at rest with [age](https://age-encryption.org/) and scoped to your user. Reference them by name when creating a sandbox; they’re decrypted server-side and injected as environment variables at boot.
| Command | Description |
| ------------------------------------------------------------- | ---------------------------------------------- |
| [`bhatti secret set`](/docs/reference/cli/secrets/set/) | Create or update a secret. |
| [`bhatti secret list`](/docs/reference/cli/secrets/list/) | List secret names (values are never returned). |
| [`bhatti secret delete`](/docs/reference/cli/secrets/delete/) | Delete a secret. |
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
# Store
bhatti secret set OPENAI_KEY sk-...
bhatti secret set DATABASE_URL "postgres://user:pass@db.internal/app"
# Reference at create
bhatti create --name agent --secret OPENAI_KEY --secret DATABASE_URL
# List names (no values)
bhatti secret list
# Update (set with the same name overwrites)
bhatti secret set OPENAI_KEY sk-...new-value
# Delete (no confirmation prompt — be careful)
bhatti secret delete OPENAI_KEY
```
## How secrets work
[Section titled “How secrets work”](#how-secrets-work)
1. **Encryption at rest.** Values are encrypted with an age key the server stores at `/var/lib/bhatti/age.key`. The key is generated on first server start. Plaintext is never written to disk.
2. **Scoped per user.** Each user has their own secrets. Other users can’t see them, list them, or reference them by name.
3. **Names only in `list`.** `bhatti secret list` returns metadata (name, created/updated timestamps). The encrypted value is never serialized to JSON output.
4. **Inject by name.** `bhatti create --secret API_KEY` decrypts `API_KEY` and adds it to the sandbox’s environment as `API_KEY=<plaintext>`.
5. **Update is idempotent.** Re-running `bhatti secret set NAME value` overwrites the previous value with no error.
## Env var priority inside a sandbox
[Section titled “Env var priority inside a sandbox”](#env-var-priority-inside-a-sandbox)
When a command runs inside a sandbox, env vars are resolved in this order (later wins):
```plaintext
defaults (PATH, TERM, HOME, LANG)
↓ overridden by
secrets (--secret on create)
↓ overridden by
sandbox env (--env on create)
```
There is **no** per-`bhatti exec` env override. The exec endpoint doesn’t accept an `env` field; the env passed in the request is silently ignored. To inject env vars, do it at create time.
## Caveat: `--template` ignores `--secret`
[Section titled “Caveat: --template ignores --secret”](#caveat---template-ignores---secret)
When creating from a template, the server uses only the template’s pre-configured secret list. Request-side `--secret` flags are silently dropped. See [`bhatti create`](/docs/reference/cli/sandbox/create/) for the workaround.
# bhatti secret delete
> Delete a secret. Cannot be undone.
Delete a secret.
## Synopsis
```
bhatti secret delete <name> [-y]
```
## Description
Removes the secret from the encrypted store. **Cannot be undone.**
By default, prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
Sandboxes that already have the secret resolved into their environment keep running with the value they got at boot. Future creates that reference the deleted secret will fail with `secret "<name>" not found`.
## Examples
```bash
bhatti secret delete OLD_KEY
```
```text
Delete secret "OLD_KEY"? [y/N]: y
deleted
```
```bash
# Non-interactive
bhatti secret delete OLD_KEY -y
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti secret list`](/docs/reference/cli/secrets/list/) — confirm the right name first
* [API: `DELETE /secrets/:name`](/docs/reference/api/#delete-a-secret)
# bhatti secret list
> List secret names. Values are never returned.
List your secrets by name. Values are never returned by the API.
## Synopsis
```
bhatti secret list
```
## Description
Returns metadata for each secret you own — name, created-at, updated-at — but never the encrypted value or the decrypted plaintext. Once a secret is set, the value can be replaced (with `bhatti secret set NAME new-value`) but not read back.
This is intentional. The encryption-at-rest model assumes the value is read by the server only when injecting into a sandbox; exposing it via the API would defeat the point.
## Examples
```bash
bhatti secret list
```
```text
OPENAI_KEY
DATABASE_URL
GITHUB_TOKEN
```
```bash
# JSON for scripting
bhatti secret list --json
```
```json
[
{"name": "OPENAI_KEY", "created_at": "2026-04-15T10:00:00Z", "updated_at": "2026-04-29T14:22:00Z"},
{"name": "DATABASE_URL", "created_at": "2026-04-22T09:15:00Z", "updated_at": "2026-04-22T09:15:00Z"}
]
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [API: `GET /secrets`](/docs/reference/api/#list-secrets)
# bhatti secret set
> Create or update an encrypted secret.
Create or update a secret.
## Synopsis
```
bhatti secret set <name> <value>
```
## Description
Encrypts the value with the server's age key and stores it under the given name, scoped to the authenticated user. Re-running with the same name overwrites the previous value with no error.
The CLI takes the value as a positional argument. **There's no stdin variant** — long or multiline values pass through your shell history. For sensitive values, use a shell that doesn't record (`HISTCONTROL=ignorespace` + leading space) or pipe through a temp variable that you `unset` afterwards.
## Examples
```bash
bhatti secret set OPENAI_KEY sk-abc123def456
```
```text
ok
```
```bash
# Update is the same call
bhatti secret set OPENAI_KEY sk-new-value
```
```bash
# Multiline / connection string
bhatti secret set DATABASE_URL "postgres://user:pass@db.internal:5432/app?sslmode=require"
```
```bash
# Avoid shell history (bash with HISTCONTROL=ignorespace, note leading space)
bhatti secret set OPENAI_KEY sk-abc123def456
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti create --secret`](/docs/reference/cli/sandbox/create/) — inject the secret into a sandbox at boot
* [API: `POST /secrets`](/docs/reference/api/#create-or-update-a-secret)
# Snapshots
> Named VM snapshots that capture memory, CPU, and disk. Resume into a new sandbox at any time.
A snapshot captures the entire VM state at a point in time — memory, CPU registers, disk, running processes, open file descriptors, network connections. Resume restores all of it, exactly. The new sandbox picks up where the original left off.
| Command | Description |
| ----------------------------------------------------------------- | ------------------------------------------ |
| [`bhatti snapshot create`](/docs/reference/cli/snapshots/create/) | Checkpoint a running sandbox. |
| [`bhatti snapshot list`](/docs/reference/cli/snapshots/list/) | List your snapshots. |
| [`bhatti snapshot resume`](/docs/reference/cli/snapshots/resume/) | Resume from a snapshot into a new sandbox. |
| [`bhatti snapshot delete`](/docs/reference/cli/snapshots/delete/) | Delete a snapshot. |
## How is this different from `bhatti stop`?
[Section titled “How is this different from bhatti stop?”](#how-is-this-different-from-bhatti-stop)
`bhatti stop` snapshots the *current* sandbox to disk and freezes it; `bhatti start` resumes that same sandbox. The sandbox doesn’t go away — it’s just paused.
`bhatti snapshot create` saves a **named** copy of the VM state. The original sandbox keeps running. `bhatti snapshot resume` later creates a **new** sandbox from that state, with a new name and new IP. You can resume from the same snapshot multiple times to stamp out copies.
| | `stop`/`start` | snapshot |
| -------------------------- | ----------------------------- | -------------------------------- |
| Original sandbox affected? | Yes — paused | No — keeps running |
| Result of resume | Same sandbox awake | A *new* sandbox |
| Multiple uses? | Once (it pairs with the stop) | Many — fork as often as you want |
| Captures volumes? | No (volumes are detached) | No (rootfs + memory only) |
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
# Checkpoint before a risky experiment
bhatti snapshot create dev --name dev-pre-experiment
# Fork: same VM state, multiple copies
bhatti snapshot resume dev-pre-experiment --name copy-1
bhatti snapshot resume dev-pre-experiment --name copy-2
# Clean up
bhatti snapshot delete dev-pre-experiment -y
```
## What’s *not* in a snapshot
[Section titled “What’s not in a snapshot”](#whats-not-in-a-snapshot)
* **Persistent volumes.** Snapshots only capture the rootfs and memory; volumes are detached when you resume into a new sandbox. Re-attach them on the resumed sandbox if you need them.
* **External state.** TCP connections to *other* hosts may be in an awkward state on resume — kernel state is preserved, but the remote peer doesn’t know the sandbox stopped and resumed.
# bhatti snapshot create
> Checkpoint a running sandbox to a named snapshot.
Checkpoint a running sandbox to a named snapshot.
## Synopsis
```
bhatti snapshot create <sandbox> --name <snapshot-name>
```
## Description
Captures memory, CPU registers, disk, and running processes from the running sandbox and saves them under `<snapshot-name>`. The source sandbox is briefly paused during the capture (a few seconds — all snapshots are full; there are no diff snapshots), then resumes.
The snapshot is stored on the server in the user's snapshot directory (`/var/lib/bhatti/snapshots/<user>/`). Resume into a new sandbox with [`bhatti snapshot resume`](/docs/reference/cli/snapshots/resume/) at any time.
`--name` is required. Names must match `[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}` — same rules as sandbox names.
## Examples
```bash
bhatti snapshot create dev --name pre-experiment
```
```text
checkpoint "pre-experiment" created (512MB)
```
```bash
# Re-running with the same snapshot name overwrites the previous one
bhatti snapshot create dev --name daily
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | (required) | Snapshot name. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti snapshot resume`](/docs/reference/cli/snapshots/resume/) — create a sandbox from this snapshot
* [`bhatti stop`](/docs/reference/cli/sandbox/stop/) — pause the original instead of forking
* [API: `POST /sandboxes/:id/checkpoint`](/docs/reference/api/#create-a-snapshot)
# bhatti snapshot delete
> Delete a snapshot. Cannot be undone.
Delete a named snapshot.
## Synopsis
```
bhatti snapshot delete <snapshot-name> [-y]
```
## Description
Removes the snapshot record and the on-disk snapshot files (`memory.img`, `checkpoint.bin`, `manifest.json`, copied rootfs). **Cannot be undone.**
By default, prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
Sandboxes that were previously resumed from this snapshot are unaffected — they have their own copy of the state and don't reference the original.
## Examples
```bash
bhatti snapshot delete pre-experiment
```
```text
Delete snapshot "pre-experiment"? [y/N]: y
deleted
```
```bash
# Non-interactive
bhatti snapshot delete pre-experiment -y
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [API: `DELETE /snapshots/:name`](/docs/reference/api/#delete-a-snapshot)
# bhatti snapshot list
> List named VM snapshots.
List your named VM snapshots.
## Synopsis
```
bhatti snapshot list
```
## Description
Lists snapshots owned by the authenticated user. Each row includes the snapshot ID, name, source sandbox name, and on-disk size in MB.
## Examples
```bash
bhatti snapshot list
```
```text
ID NAME SOURCE SIZE
snap_a1b2c3d4 pre-experiment dev 512MB
snap_e5f6g7h8 daily worker 1024MB
snap_i9j0k1l2 ready base 384MB
```
```bash
# JSON for scripting
bhatti snapshot list --json | jq '.[] | select(.source_sandbox == "base")'
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [API: `GET /snapshots`](/docs/reference/api/#list-snapshots)
# bhatti snapshot resume
> Resume a snapshot into a new sandbox. The original is unaffected.
Resume a named snapshot into a new sandbox.
## Synopsis
```
bhatti snapshot resume <snapshot-name> [--name <new-sandbox-name>]
```
## Description
Creates a brand-new sandbox by restoring the snapshot's memory, CPU, and disk state. Running processes pick up exactly where they left off; open file descriptors are still open; in-memory data structures are intact.
The new sandbox gets a new ID. **Persistent volumes are not re-attached** — re-attach them with the same `--volume` semantics if needed.
`--name` is optional. Without it, a name is generated from the snapshot's source sandbox plus a random suffix.
You can resume the same snapshot multiple times — each call produces a fresh sandbox with the same starting state. Useful for fork-and-experiment workflows.
## Examples
```bash
# Resume into a sandbox with a specific name
bhatti snapshot resume pre-experiment --name dev-restored
```
```text
abc123 dev-restored
```
```bash
# Auto-generated name (source sandbox + suffix)
bhatti snapshot resume pre-experiment
```
```bash
# Stamp out parallel copies for an A/B test
bhatti snapshot resume base --name worker-a
bhatti snapshot resume base --name worker-b
bhatti snapshot resume base --name worker-c
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | auto-generated | Name for the new sandbox. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti snapshot create`](/docs/reference/cli/snapshots/create/) — make the snapshot
* [`bhatti snapshot list`](/docs/reference/cli/snapshots/list/) — find snapshot names
* [API: `POST /snapshots/:name/resume`](/docs/reference/api/#resume-a-snapshot)
# Volumes
> Persistent ext4 volumes that survive sandbox destruction. Resize, clone, back up, restore.
Persistent volumes are ext4 filesystems that survive sandbox destruction. Create one, attach it at sandbox creation with `--volume name:/mount`, destroy the sandbox — the data stays.
| Command | Description |
| --------------------------------------------------------------------------- | ------------------------------------------------------------ |
| [`bhatti volume create`](/docs/reference/cli/volumes/create/) | Create a persistent volume. |
| [`bhatti volume list`](/docs/reference/cli/volumes/list/) | List your volumes and current attachments. |
| [`bhatti volume delete`](/docs/reference/cli/volumes/delete/) | Delete a volume. Requires it to be detached. |
| [`bhatti volume resize`](/docs/reference/cli/volumes/resize/) | Grow a volume. Cannot shrink. |
| [`bhatti volume clone`](/docs/reference/cli/volumes/clone/) | Point-in-time copy. Source must be detached. |
| [`bhatti volume backup`](/docs/reference/cli/volumes/backup/) | Back up to S3-compatible storage. Requires server S3 config. |
| [`bhatti volume backup-list`](/docs/reference/cli/volumes/backup-list/) | List backups for a volume. |
| [`bhatti volume restore`](/docs/reference/cli/volumes/restore/) | Restore from a backup. Volume must be detached. |
| [`bhatti volume backup-delete`](/docs/reference/cli/volumes/backup-delete/) | Delete a backup. |
## Detached state
[Section titled “Detached state”](#detached-state)
Four operations require the volume to be **detached** from any sandbox before they’ll run: `delete`, `resize`, `clone`, and `restore` all return a `409 Conflict` if the volume is currently attached.
Detach by either destroying the sandbox using the volume, or stopping it with [`bhatti stop`](/docs/reference/cli/sandbox/stop/) (which also detaches volumes).
`backup` does **not** check for attached state — it can run while the sandbox is mounted, but be aware the snapshot may be inconsistent if writes are in flight at backup time. Stop the sandbox first if you need a guaranteed-consistent backup.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
# Create + attach in one workflow
bhatti volume create --name workspace --size 5120
bhatti create --name dev --volume workspace:/workspace
# Grow on the fly
bhatti volume resize workspace --size 10240
# Snapshot before a risky operation
bhatti volume clone workspace --name workspace-pre-upgrade
# Off-site backup (requires `backup` config block on server)
bhatti volume backup workspace
bhatti volume backup-list workspace
```
## Backup configuration
[Section titled “Backup configuration”](#backup-configuration)
`backup`, `restore`, `backup-list`, and `backup-delete` require an S3-compatible storage backend configured on the server. Without it, they return `501 Not Implemented` with the message `backup not configured — add backup section to config.yaml`. See [Configuration: backup](/docs/reference/config/#backup).
# bhatti volume backup
> Back up a volume to S3-compatible storage. Requires server backup config.
Back up a persistent volume to S3-compatible storage.
## Synopsis
```
bhatti volume backup <volume-name>
```
## Description
Compresses (zstd) the volume's ext4 file and uploads it to the S3-compatible bucket configured on the server. Returns a backup record with an ID, size, and S3 key.
The volume **does not** need to be detached for this to run. However, if writes are in flight at backup time, the snapshot may be inconsistent. For a guaranteed-consistent backup, stop the sandbox first:
```bash
bhatti stop dev # detaches volume, flushes guest writes
bhatti volume backup workspace
bhatti start dev # resume
```
Requires a `backup` block in the server's `config.yaml`. Without it, the API returns `501` with `backup not configured — add backup section to config.yaml`. See [Configuration: backup](/docs/reference/config/#backup).
Backups are stored under the per-user S3 prefix configured by the server. Restore with [`bhatti volume restore`](/docs/reference/cli/volumes/restore/).
## Examples
```bash
bhatti volume backup workspace
```
```text
backup bk_a1b2c3d4 created (workspace, 5234567890 bytes)
```
```bash
# JSON for scripting (capture the ID for later restore)
backup_id=$(bhatti volume backup workspace --json | jq -r .id)
echo "backed up: $backup_id"
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume backup-list`](/docs/reference/cli/volumes/backup-list/) — see existing backups
* [`bhatti volume restore`](/docs/reference/cli/volumes/restore/) — recover from a backup
* [`bhatti volume clone`](/docs/reference/cli/volumes/clone/) — local point-in-time copy (no S3, no compression)
* [API: `POST /volumes/:name/backups`](/docs/reference/api/#backup-a-volume)
# bhatti volume backup-delete
> Delete a backup from S3 and the local backup list.
Delete a volume backup.
## Synopsis
```
bhatti volume backup-delete <volume-name> <backup-id> [-y]
```
## Description
Removes the backup record from the server and deletes the corresponding object in S3. Cannot be undone.
By default, prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
## Examples
```bash
bhatti volume backup-delete workspace bk_a1b2c3d4
```
```text
Delete backup bk_a1b2c3d4? [y/N]: y
deleted
```
```bash
# Non-interactive
bhatti volume backup-delete workspace bk_a1b2c3d4 -y
```
```bash
# Trim to last 7 backups
bhatti volume backup-list workspace --json \
| jq -r 'sort_by(.created_at) | reverse | .[7:][] | .id' \
| xargs -I{} bhatti volume backup-delete workspace {} -y
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume backup-list`](/docs/reference/cli/volumes/backup-list/)
* [API: `DELETE /volumes/:name/backups/:id`](/docs/reference/api/#delete-a-backup)
# bhatti volume backup-list
> List backups for a volume.
List backups for a volume.
## Synopsis
```
bhatti volume backup-list <volume-name>
```
## Description
Lists backups recorded server-side for the given volume. The backups themselves live in S3; this command shows the metadata (ID, size, creation time) so you can pick one to restore.
If there are no backups, prints `no backups`.
## Examples
```bash
bhatti volume backup-list workspace
```
```text
ID SIZE CREATED
bk_a1b2c3d4 4998 MB 2026-04-30T03:00:14Z
bk_e5f6g7h8 5021 MB 2026-04-29T03:00:11Z
bk_i9j0k1l2 4892 MB 2026-04-28T03:00:09Z
```
```bash
# Latest backup ID
bhatti volume backup-list workspace --json | jq -r 'sort_by(.created_at) | last | .id'
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume backup`](/docs/reference/cli/volumes/backup/)
* [API: `GET /volumes/:name/backups`](/docs/reference/api/#list-volume-backups)
# bhatti volume clone
> Make a point-in-time copy of a volume. Source must be detached.
Make a point-in-time copy of a persistent volume.
## Synopsis
```
bhatti volume clone <source-name> --name <new-name>
```
## Description
Server-side copy of the volume's ext4 file. The clone is independent — writes to the source after the clone do not affect the copy, and vice versa.
The source volume must be **detached** when you clone it. If any sandbox has it mounted, the server returns `409 Conflict`. Detach by stopping or destroying the sandbox using it.
Useful for:
* Snapshotting before a risky upgrade or migration.
* Forking a workspace for an experiment.
* Pre-warming a base volume before stamping out workers.
## Examples
```bash
# Snapshot before upgrade
bhatti stop dev # detaches workspace
bhatti volume clone workspace --name workspace-pre-upgrade
bhatti start dev
```
```text
Cloned workspace → workspace-pre-upgrade
```
```bash
# Fork for an experiment
bhatti volume clone workspace --name workspace-experiment
bhatti create --name experiment --volume workspace-experiment:/workspace
```
```bash
# Source still attached — fails
bhatti volume clone workspace --name copy
```
```text
409 Conflict: source volume is attached — detach before snapshotting
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | (required) | Name for the cloned volume. Must match the standard volume-name pattern. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume backup`](/docs/reference/cli/volumes/backup/) — off-site copy via S3 (works while attached, but may be inconsistent)
* [`bhatti stop`](/docs/reference/cli/sandbox/stop/) — detach by stopping the sandbox using it
* [API: `POST /volumes/:name/snapshot`](/docs/reference/api/#clone-a-volume)
# bhatti volume create
> Create a persistent ext4 volume. Survives sandbox destruction.
Create a persistent ext4 volume.
## Synopsis
```
bhatti volume create --name <name> --size <MB>
```
## Description
Allocates an ext4 filesystem image on the server, owned by the calling user. The volume is unattached at creation; mount it at sandbox creation time with [`--volume name:/mount`](/docs/reference/cli/sandbox/create/).
The volume survives sandbox destruction. Multiple sandboxes can attach the same volume sequentially, but only one sandbox can have it attached at a time (read-only mounts via `name:/mount:ro` are the only way to share live).
## Examples
```bash
bhatti volume create --name workspace --size 5120
```
```text
vol_a1b2c3d4 workspace 5120MB ready
```
```bash
# Attach at sandbox create
bhatti create --name dev --volume workspace:/workspace
```
```bash
# Attach read-only — multiple sandboxes can mount this simultaneously
bhatti create --name reader --volume workspace:/data:ro
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | (required) | Volume name. Must match `[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}`. | | `--size <int>` | (required) | Initial size in MB. Grow later with [`bhatti volume resize`](/docs/reference/cli/volumes/resize/). |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti create --volume`](/docs/reference/cli/sandbox/create/) — attach at boot
* [`bhatti volume resize`](/docs/reference/cli/volumes/resize/) — grow the volume
* [API: `POST /volumes`](/docs/reference/api/#create-a-volume)
# bhatti volume delete
> Delete a persistent volume. Cannot be undone. Volume must be detached.
Delete a persistent volume. **Cannot be undone.**
## Synopsis
```
bhatti volume delete <name> [-y]
```
## Description
Removes the volume record from the database and deletes the underlying ext4 file.
The volume must be **detached** before deleting — if any sandbox has it mounted, the server returns `409 Conflict`. Stop or destroy the sandbox first, then retry.
By default, prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
## Examples
```bash
bhatti volume delete workspace
```
```text
Delete volume "workspace"? [y/N]: y
deleted
```
```bash
# Non-interactive
bhatti volume delete workspace -y
```
```bash
# Volume still attached
bhatti volume delete workspace -y
```
```text
409 Conflict: volume "workspace" attached to sandbox "dev"
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume list`](/docs/reference/cli/volumes/list/) — find current attachments
* [API: `DELETE /volumes/:name`](/docs/reference/api/#delete-a-volume)
# bhatti volume list
> List persistent volumes you own.
List your persistent volumes.
## Synopsis
```
bhatti volume list
```
## Description
Lists volumes owned by the authenticated user. Each row includes ID, name, size, and current status.
The default output is compact. Use `--json` to get the full record including current attachments (which sandbox has it mounted, where, and read-only flag).
## Examples
```bash
bhatti volume list
```
```text
ID NAME SIZE STATUS
vol_a1b2c3d4 workspace 5120MB ready
vol_e5f6g7h8 data 20480MB ready
```
```bash
# Find which sandbox has a volume mounted
bhatti volume list --json | jq '.[] | {name, attachments}'
```
```json
{
"name": "workspace",
"attachments": [
{"sandbox_id": "abc123", "mount": "/workspace", "read_only": false}
]
}
```
## Options
This command takes only global flags. See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [API: `GET /volumes`](/docs/reference/api/#list-volumes)
# bhatti volume resize
> Grow a persistent volume. Cannot shrink.
Grow a persistent volume.
## Synopsis
```
bhatti volume resize <name> --size <new-MB>
```
## Description
Resizes the underlying ext4 image file to the new size. The new size must be **larger** than the current size — shrinking is not supported.
The volume must be **detached** before resizing. The server returns `409 Conflict` if any sandbox has it mounted. After the resize, attach it again and the new capacity is visible to the guest filesystem on next mount.
## Examples
```bash
# Was 5120 MB; double it
bhatti volume resize workspace --size 10240
```
```text
resized to 10240MB
```
```bash
# Trying to shrink fails
bhatti volume resize workspace --size 1024
```
```text
400 Bad Request: new size (1024MB) must be larger than current size (5120MB)
```
```bash
# Volume still attached
bhatti volume resize workspace --size 10240
```
```text
409 Conflict: volume is attached — detach before resizing
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--size <int>` | (required) | New size in MB. Must be larger than the current size. |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [API: `POST /volumes/:name/resize`](/docs/reference/api/#resize-a-volume)
# bhatti volume restore
> Restore a volume from a backup. Volume must be detached.
Restore a volume from a backup.
## Synopsis
```
bhatti volume restore <volume-name> --backup-id <id>
```
## Description
Downloads the backup from S3, decompresses it, and overwrites the volume's ext4 file with the backup's contents. **Existing data on the volume is replaced** — there's no merge.
The volume must be **detached** before restoring. The server returns `409 Conflict` if any sandbox has it mounted.
The backup ID comes from [`bhatti volume backup-list`](/docs/reference/cli/volumes/backup-list/). The restore is in-place: the volume keeps its name and any pending attachments resolve once it's done.
## Examples
```bash
# Find a backup, restore from it
bhatti stop dev
bhatti volume backup-list workspace
bhatti volume restore workspace --backup-id bk_a1b2c3d4
bhatti start dev
```
```text
volume restored from backup bk_a1b2c3d4
```
```bash
# Volume still attached
bhatti volume restore workspace --backup-id bk_a1b2c3d4
```
```text
409 Conflict: volume is attached to a sandbox — detach before restoring
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--backup-id <string>` | (required) | The backup to restore from. Get one from [`backup-list`](/docs/reference/cli/volumes/backup-list/). |
See [Global flags](/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume backup-list`](/docs/reference/cli/volumes/backup-list/) — find a backup to restore from
* [`bhatti volume clone`](/docs/reference/cli/volumes/clone/) — make a clone before restoring, just in case
* [API: `POST /volumes/:name/backups/restore`](/docs/reference/api/#restore-a-volume)
# Self-Hosting
> Install the bhatti server on your hardware, add teammates, set a custom domain, back up the data directory.
The [Quickstart](/docs/quickstart/) covers the install in a few lines and gets you to your first sandbox. This page is the longer version — what’s actually happening on your server, how to add teammates, and the operational details I’ve picked up running bhatti on real hardware.
## Requirements
[Section titled “Requirements”](#requirements)
bhatti v2 (krucible) self-hosts on two platforms:
* **Linux with KVM** — `/dev/kvm` must exist and be readable. The daemon runs under systemd (as root by default; it needs KVM). Any aarch64 or x86\_64 box works.
* **macOS on Apple Silicon** — HVF (Hypervisor.framework). No root, no KVM; the shipped `bhatti-vmm` is Developer-ID signed and notarized, carries the hypervisor entitlement, and runs under launchd. Intel Macs are not supported.
1 GB+ RAM; NVMe recommended for snapshot performance.
I run on two Raspberry Pi 5s with NVMe HATs (home, integration tests), a Hetzner box, and my own Mac laptop. Anything in that ballpark or stronger works.
## Storage — no special filesystem needed
[Section titled “Storage — no special filesystem needed”](#storage--no-special-filesystem-needed)
Unlike v1, **v2 has no btrfs/reflink requirement.** Copy-on-write lives at the disk-image-format layer: each sandbox gets a thin **qcow2 overlay** over a shared read-only base image, so create-from-image is instant and one base is shared by many sandboxes’ overlays — on **ext4, xfs, btrfs, and APFS alike**. The overhead is \~0.5% on typical agent/dev workloads. There’s nothing to pre-provision; `/var/lib/bhatti` on your normal root filesystem is fine.
(btrfs/xfs still give you transparent compression on the base images if you want it, but it’s an optimization, not a prerequisite. If you have a v1 install on a btrfs loopback, you don’t need to carry it forward — v2 is a fresh install, not an in-place upgrade.)
## Install
[Section titled “Install”](#install)
```bash
curl -fsSL bhatti.sh/install | sudo bash
```
The script will:
1. **Detect your platform** (`linux`/`darwin`, `arm64`/`amd64`).
2. **Prompt for a rootfs tier** (or pass `--tier <name>` to skip the prompt; see below).
3. **Download one self-contained runtime bundle** — the `bhatti` CLI + daemon, the `bhatti-vmm` VMM helper, the `bhatti-netd` gateway, the `libkrun` (krucible) shared library, and the lean guest kernel — plus the rootfs tier. It’s laid down as a relocatable prefix under `<data_dir>/runtime/`; the binaries resolve `libkrun` via a relative rpath, so there’s no system-library pollution.
4. **Install a service** — a systemd unit at `/etc/systemd/system/bhatti.service` on Linux, or a LaunchDaemon at `/Library/LaunchDaemons/sh.bhatti.plist` on macOS — and start the daemon.
5. **Create an `admin` user**, save its API key to the invoking user’s `~/.bhatti/config.yaml` (so the CLI on the same box works without an explicit `bhatti setup`).
6. **Print the admin API key once** — save it. Anyone with this key can do anything on this server.
### Rootfs tiers
[Section titled “Rootfs tiers”](#rootfs-tiers)
| Tier | What’s in it | Size |
| ---------- | --------------------------------------- | -------- |
| `minimal` | Bare Ubuntu 24.04 | \~200 MB |
| `browser` | + Chromium, Playwright, Node 22 | \~600 MB |
| `docker` | + Docker Engine | \~550 MB |
| `computer` | + Full desktop: XFCE, KasmVNC, Chromium | \~1.5 GB |
You pick one tier as the *default* for `bhatti create` (used when you don’t pass `--image`). Other tiers are still installable later and can be selected per sandbox with `--image <tier>`.
### Non-interactive install
[Section titled “Non-interactive install”](#non-interactive-install)
For Ansible / packer / CI, skip the tier prompt with flags. The `bash -s -- ...` syntax passes flags through `curl | bash`:
```bash
# Specific tier as the default
curl -fsSL bhatti.sh/install | sudo bash -s -- --tier browser
# Default minimal, plus install every other tier
curl -fsSL bhatti.sh/install | sudo bash -s -- --tiers all
# Default docker, plus also install browser
curl -fsSL bhatti.sh/install | sudo bash -s -- --tier docker --tiers browser
```
`--tier` (singular) sets the *default* tier — what `bhatti create` uses when no `--image` is passed. `--tiers` (plural, comma-separated list or `all`) installs additional tiers alongside the default.
## Verifying the install
[Section titled “Verifying the install”](#verifying-the-install)
From the same box where you ran the install, no setup needed:
```bash
bhatti create --name test
bhatti exec test -- uname -a
bhatti destroy test
```
If the create succeeds, the daemon is healthy, the VMM has hypervisor access (KVM or HVF), the agent is reachable, and the CLI’s config was written correctly.
## Adding teammates
[Section titled “Adding teammates”](#adding-teammates)
The remote-CLI flow. On the server:
```bash
sudo bhatti user create --name alice --max-sandboxes 5 --max-cpus 4 --max-memory 4096
# → API key: bht_... (shown once, save it)
```
Send Alice the key over a secure channel (1Password, Signal, encrypted email — not Slack, not GitHub Issues). On her machine:
```bash
# CLI only, no sudo
curl -fsSL bhatti.sh/install | bash
# Wire it up. Either interactive or via flags.
bhatti setup --url https://your-server:8080 --token bht_...
# or:
bhatti setup
```
Now Alice can `bhatti create`, `bhatti exec`, etc. — but only against sandboxes she creates. Each user is isolated at the API layer (scoped queries), and each user’s sandboxes get their own `bhatti-netd` gateway so cross-user traffic never shares a network — see [Networking](/docs/under-the-hood/networking/).
For driving bhatti from agents, CI, or provisioning scripts, use the non-interactive form: `bhatti setup --url ... --token ...`. The auth test always runs and the command exits non-zero on failure, so your provisioner picks up bad credentials immediately.
## Custom domain (optional but recommended)
[Section titled “Custom domain (optional but recommended)”](#custom-domain-optional-but-recommended)
By default `bhatti publish` generates URLs at `<alias>.<your-server-ip>.nip.io` or similar — fine for testing, ugly for sharing. To get URLs like `my-app.yourdomain.com` with TLS, see [Custom domain](/docs/managing/custom-domain/).
## Backups
[Section titled “Backups”](#backups)
What to back up:
* `<data_dir>/state.db` — every sandbox/user/secret/template/volume row. The daemon writes WAL, so back up `state.db` + `state.db-wal` + `state.db-shm` together, or run `sqlite3 .backup` for a clean copy.
* `<data_dir>/age.key` — **the encryption key for every secret.** Lose it and every encrypted secret on the server is unrecoverable. Treat it like a TLS private key.
* `<data_dir>/volumes/` — standalone volumes (the ones created with `bhatti volume create`). Real ext4 images.
* Per-sandbox files under `<data_dir>/sandboxes/` are usually not worth backing up — sandboxes are reproducible from images and init scripts. For state you care about, take a named snapshot (`bhatti snapshot create <name>`), which writes a self-contained bundle under `<data_dir>/snapshots/`.
You do **not** need to back up `<data_dir>/runtime/` — it’s the relocatable binary/library/kernel bundle, re-laid on every install and update, not user data.
For S3-compatible volume backups, see [Volumes → backups](/docs/managing/volumes/#backups).
## Updating
[Section titled “Updating”](#updating)
```bash
sudo bhatti update # refresh the whole runtime bundle + tiers
sudo bhatti update --tiers all # also pull additional rootfs tiers
```
Or just re-run the install:
```bash
curl -fsSL bhatti.sh/install | sudo bash
```
The install script is idempotent — it skips components that haven’t changed and updates the rest. Note that a `bhatti update` within v2 is safe; crossing from a v1 (Firecracker) install is **blocked** (different VMM, different on-disk layout) — install v2 fresh instead.
## Uninstalling
[Section titled “Uninstalling”](#uninstalling)
```bash
# Remove binaries + service, keep <data_dir> so you can reinstall
curl -fsSL bhatti.sh/uninstall | sudo bash
# Remove everything, including all sandbox state, volumes, and age.key
curl -fsSL bhatti.sh/uninstall | sudo bash -s -- --purge
```
`--purge` is destructive and unreversible — it deletes the encryption key, every secret, every sandbox, every volume.
## Where each thing lives
[Section titled “Where each thing lives”](#where-each-thing-lives)
The full layout is in the [Configuration reference](/docs/reference/config/#data-directory-layout). Short version:
* `<data_dir>/state.db` — SQLite, the source of truth
* `<data_dir>/age.key` — secret encryption key (back this up)
* `<data_dir>/runtime/` — the VMM/gateway/libkrun/kernel bundle (not user data)
* `<data_dir>/images/` — read-only base rootfs templates
* `<data_dir>/sandboxes/<id>/` — per-sandbox: CoW overlay, config drive, snapshot bits
* `<data_dir>/volumes/` — standalone volumes
* `<data_dir>/snapshots/` — named snapshots from `bhatti snapshot create`
* `/etc/bhatti/config.yaml` — daemon config (engine, runtime paths, listen, domain)
## Next steps
[Section titled “Next steps”](#next-steps)
* [Users & Auth](/docs/managing/users/) — API key rotation, per-user limits, deleting users
* [Custom domain](/docs/managing/custom-domain/) — TLS for `bhatti publish`, ACME, wildcard DNS
* [Concepts](/docs/concepts/) — mental model for sandboxes and thermal states
* [Architecture](/docs/under-the-hood/architecture/) — what each process does and how state flows
# Updating & Uninstalling
> Keep bhatti up to date or remove it cleanly.
## Updating
[Section titled “Updating”](#updating)
```bash
# CLI (any machine)
bhatti update
# Server (updates CLI, kernel, rootfs, lohar)
sudo bhatti update
# Server, but only the CLI binary
bhatti update --cli-only
# Server + install additional rootfs tiers
sudo bhatti update --tiers all
sudo bhatti update --tiers browser,docker
```
Only changed components are downloaded — if the kernel and rootfs haven’t changed since your last update, they’re skipped automatically.
Use `--cli-only` on a server when you want to test a CLI fix without touching kernel/rootfs/lohar. Use `--tiers` to pull additional rootfs images (`minimal`, `browser`, `docker`, `computer`) without re-running the full install.
Or re-run the install command:
```bash
curl -fsSL bhatti.sh/install | bash # CLI
curl -fsSL bhatti.sh/install | sudo bash # server
```
Note
`bhatti update` updating all server components (not just the CLI binary) requires v1.7.3 or later. On older versions, use the curl command above.
### Check for updates
[Section titled “Check for updates”](#check-for-updates)
```bash
bhatti version
```
Shows the installed version and checks GitHub for the latest release. Update notices also appear in `bhatti admin status`.
## Uninstalling
[Section titled “Uninstalling”](#uninstalling)
### Keep data (reinstallable)
[Section titled “Keep data (reinstallable)”](#keep-data-reinstallable)
```bash
curl -fsSL bhatti.sh/uninstall | sudo bash
```
Removes binaries and the systemd service. Keeps `/var/lib/bhatti` (images, volumes, snapshots, secrets, database) so you can reinstall later without losing state.
### Remove everything
[Section titled “Remove everything”](#remove-everything)
```bash
curl -fsSL bhatti.sh/uninstall | sudo bash -s -- --purge
```
Removes all data including volumes, snapshots, and the encryption key.
### CLI only
[Section titled “CLI only”](#cli-only)
```bash
sudo rm /usr/local/bin/bhatti
rm -rf ~/.bhatti
```
# Documentation
> Redirecting to Quickstart...
# Concepts
> Sandboxes, thermal states, and the two binaries — the mental model for bhatti.
## What a sandbox is
[Section titled “What a sandbox is”](#what-a-sandbox-is)
A sandbox is a [Firecracker](https://firecracker-microvm.github.io/) microVM. Not a container — a real virtual machine with its own Linux kernel, its own filesystem, and its own network stack. Process isolation is hardware-enforced by KVM, not namespace-based.
Sandboxes are created in seconds and destroyed instantly. When idle, they’re automatically paused — their entire memory state is snapshotted to disk — and resumed in under 3ms when needed.
## The two binaries
[Section titled “The two binaries”](#the-two-binaries)
**bhatti** runs on the host. It’s both the server daemon (`bhatti serve`) and the CLI client (`bhatti create`, `bhatti exec`, etc.). The server manages Firecracker VMs via its HTTP API over a Unix socket.
**lohar** runs inside the VM as PID 1. It’s the guest agent — a static binary with zero dependencies (no systemd, no libc, no initramfs). It handles command execution, file operations, PTY sessions, and communicates with the host over a vsock connection using a custom [wire protocol](/v1/docs/under-the-hood/wire-protocol/).
The names: *bhatti* (भट्टी) means furnace. *lohar* (लोहार) means blacksmith.
## Thermal states
[Section titled “Thermal states”](#thermal-states)
Every sandbox is in one of three thermal states:
| State | What’s happening | Resume latency | Memory cost |
| -------- | -------------------------------------- | -------------- | ----------- |
| **Hot** | VM is running, vCPUs active | — | Full |
| **Warm** | vCPUs paused, memory still allocated | \~400µs | Full |
| **Cold** | Memory snapshotted to disk, VM stopped | \~50ms | Zero |
The thermal manager transitions sandboxes automatically based on activity. From the API consumer’s perspective, every sandbox appears to be running — requests to cold sandboxes trigger a transparent wake before the operation executes.
You can opt out with `keep_hot: true` on sandbox creation or update, which prevents automatic pausing. Useful for sandboxes maintaining persistent external connections.
## Architecture at a glance
[Section titled “Architecture at a glance”](#architecture-at-a-glance)
```plaintext
CLI / API client
│
▼ HTTP
┌─────────────┐
│ bhatti serve │ ← host process: REST API, thermal manager, proxy
└──────┬──────┘
│ Firecracker API (Unix socket)
▼
┌─────────────┐
│ Firecracker │ ← VMM: boots the microVM
└──────┬──────┘
│ vsock
▼
┌─────────────┐
│ lohar │ ← PID 1 inside VM: exec, files, PTY sessions
└─────────────┘
```
For the full picture, see [Architecture Overview](/v1/docs/under-the-hood/architecture/).
# Adding a Tier
> How to create a new rootfs tier for bhatti.
Tiers are pre-built ext4 root filesystem images. Each tier builds on `minimal` and adds specific tooling. The server auto-discovers tiers at startup by globbing `rootfs-*-{arch}.ext4` in the images directory.
## Steps
[Section titled “Steps”](#steps)
### 1. Create the tier script
[Section titled “1. Create the tier script”](#1-create-the-tier-script)
Add `scripts/tiers/<name>.sh`. This runs inside a chroot during the build. It receives `$MOUNT`, `$ARCH`, `$DEB_ARCH`, `$AGENT`, and `$SCRIPT_DIR` as environment variables.
Most tiers source minimal first:
```bash
#!/bin/bash
set -euo pipefail
"$SCRIPT_DIR/tiers/minimal.sh"
chroot "$MOUNT" /bin/bash -c '
apt-get update -qq
apt-get install -y --no-install-recommends your-packages
apt-get clean
rm -rf /var/lib/apt/lists/*
'
```
### 2. Long-running daemons: ship them as systemd units
[Section titled “2. Long-running daemons: ship them as systemd units”](#2-long-running-daemons-ship-them-as-systemd-units)
If your tier needs a daemon running on boot — a database, a headless browser, a VNC gateway — ship it as a `.service` file and let lohar’s [`systemctl` shim](/v1/docs/under-the-hood/lohar-the-blacksmith/) manage it. Don’t write `init.sh` (the legacy pattern); don’t fork the daemon yourself; don’t try to manage PIDs, restarts, or cgroups.
The canonical short example is `headless-chrome.service` from the browser tier:
```ini
[Unit]
Description=Headless Chromium with Chrome DevTools Protocol
After=network.target
[Service]
Type=simple
# Defaults; user-supplied values in /run/bhatti/config-env override.
Environment=CHROME_REMOTE_PORT=9222
Environment=CHROME_FLAGS=
EnvironmentFile=-/run/bhatti/config-env
ExecStart=/usr/local/bin/headless_shell --no-sandbox --disable-gpu \
--disable-dev-shm-usage \
--remote-debugging-port=${CHROME_REMOTE_PORT} \
--remote-debugging-address=0.0.0.0 \
$CHROME_FLAGS
Restart=on-failure
RestartSec=2s
[Install]
WantedBy=multi-user.target
```
The pattern:
* **`Type=simple`** for daemons that stay in the foreground (the common case). **`Type=notify`** for daemons that speak `sd_notify(3)` and signal `READY=1` once their socket is up (`dockerd`, `postgresql`). **`Type=oneshot`** with `RemainAfterExit=yes` for first-boot setup helpers (password generation, env-file writers).
* **`ExecStart=`** points at the binary directly. No `/bin/sh -c` wrapper. The shim already wraps your command in `/bin/sh -c "exec <ExecStart>"` for variable expansion; nesting another shell is redundant at best.
* **`Restart=on-failure`** + **`RestartSec=2s`** is the right default for long-running daemons. The shim respects the standard `StartLimitBurst` / `StartLimitIntervalSec` so a crash loop gives up cleanly.
* **User-tunable knobs**: declare defaults via `Environment=` and add `EnvironmentFile=-/run/bhatti/config-env`. The leading `-` makes the file optional, so sandboxes with no `--env` flags still boot cleanly. Values passed at create time (`bhatti create --env CHROME_REMOTE_PORT=9333`) override the unit’s defaults via the config-env bridge that lohar materialises at boot ([`cmd/lohar/main.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go)).
Drop the unit into `/etc/systemd/system/<name>.service` and symlink it into `multi-user.target.wants/` so lohar’s `startEnabledServices` picks it up at boot:
```bash
cat > "$MOUNT/etc/systemd/system/headless-chrome.service" << UNIT
... (unit body above) ...
UNIT
ln -sf /etc/systemd/system/headless-chrome.service \
"$MOUNT/etc/systemd/system/multi-user.target.wants/headless-chrome.service"
```
That’s all. The shim handles cgroup placement (race-free, see [How services are spawned](/v1/docs/under-the-hood/lohar-the-blacksmith/#how-services-are-spawned)), pidfile creation, restart-on-failure, and `journalctl -u` log routing. Your unit just declares what to run.
#### Anti-patterns
[Section titled “Anti-patterns”](#anti-patterns)
Three mistakes will silently break your tier:
* **Don’t write `init.sh`.** The legacy pre-v1.11.3 pattern (one shell script in `/etc/bhatti/init.sh` that runs all your daemons in the background) is gone. Daemons started that way escape the shim’s lifecycle entirely — no `systemctl status`, no `journalctl -u`, no `Restart=`, no `cgroup.kill` on stop. Existing tiers were converted off `init.sh` in v1.11.7–v1.11.9.
* **Don’t wrap `ExecStart=` in `/bin/sh -c '...'` yourself.** The shim does it for you. An inner shell is redundant on a good day; on a bad day (a daemon that double-`fork`s during startup) it introduces a fork the placement helper has to chase. Use `Environment=` and `${VAR}` expansion in the unit file — systemd’s parser handles that natively.
* **Don’t manage cgroups, pidfiles, or restart loops yourself.** All three live in the shim once your unit is enabled. Writing your own `pkill` cleanup or pidfile parsing means you’re fighting the shim, not using it.
You don’t need to know about `lohar spawn` to author a unit. The helper sits upstream of `ExecStart=` and is invisible to the unit file — it’s how the shim gets the daemon into the right cgroup before any fork. If you find yourself reading [`cmd/lohar/spawn.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/spawn.go) you’ve probably gone too deep.
### 3. Add size default in `scripts/build-tier.sh`
[Section titled “3. Add size default in scripts/build-tier.sh”](#3-add-size-default-in-scriptsbuild-tiersh)
```bash
case "$TIER" in
minimal) SIZE_MB="${SIZE_MB:-512}" ;;
browser) SIZE_MB="${SIZE_MB:-2048}" ;;
new-tier) SIZE_MB="${SIZE_MB:-1024}" ;; # ← add this
*) echo "unknown tier: $TIER" >&2; exit 1 ;;
esac
```
### 4. Add to CI release matrix
[Section titled “4. Add to CI release matrix”](#4-add-to-ci-release-matrix)
In `.github/workflows/release.yml`:
```yaml
tier: [minimal, browser, docker, computer, new-tier]
```
### 5. Add to the install script
[Section titled “5. Add to the install script”](#5-add-to-the-install-script)
In `scripts/install.sh`, update the interactive tier prompt and the `ALL_KNOWN_TIERS` variable in `do_server_update()`.
### 6. Verify
[Section titled “6. Verify”](#6-verify)
The bats test suite (`scripts/install_test.bats`) validates that every tier in `scripts/tiers/` appears in all four places:
```bash
bats scripts/install_test.bats
```
If you miss a registration point, the tier consistency tests catch it.
## Checklist
[Section titled “Checklist”](#checklist)
```plaintext
[ ] scripts/tiers/<name>.sh
[ ] scripts/tiers/<name>.sh — .service unit(s) for long-running daemons,
symlinked into multi-user.target.wants/
[ ] scripts/build-tier.sh — SIZE_MB case
[ ] .github/workflows/release.yml — matrix.tier
[ ] scripts/install.sh — interactive menu
[ ] scripts/install.sh — ALL_KNOWN_TIERS
[ ] bats tests pass
```
# Kernel
> Custom kernel configuration and build process for Firecracker microVMs.
bhatti uses a custom-built Linux kernel optimized for Firecracker microVMs. The kernel is minimal — only the features needed for sandbox workloads are enabled.
## Building
[Section titled “Building”](#building)
```bash
# Clone the kernel source
git clone --depth 1 --branch v6.1 https://github.com/torvalds/linux.git
cd linux
# Apply the bhatti config
cp /path/to/bhatti/kernel/.config .config
# Build
make -j$(nproc) vmlinux
```
The output `vmlinux` is an uncompressed kernel binary — Firecracker loads it directly.
## Key config choices
[Section titled “Key config choices”](#key-config-choices)
The kernel config enables:
* **virtio** — virtio-blk (rootfs, config drive, volumes), virtio-net (networking), vsock (host↔guest)
* **ext4** — filesystem for rootfs and volumes
* **devtmpfs** — required for `/dev` device nodes
* **PTY** — for interactive shell sessions
* **process namespaces** — for process isolation within the VM
* **network** — TCP/IP stack, bridge support
The kernel config disables everything unnecessary for a sandbox environment: USB, sound, most filesystems, most network drivers, hardware-specific drivers.
## Architecture support
[Section titled “Architecture support”](#architecture-support)
Two kernel configs are maintained:
* `vmlinux-arm64` — for Raspberry Pi 5, Graviton, and other ARM64 hosts
* `vmlinux-amd64` — for x86\_64 bare metal and cloud instances
The correct kernel is selected automatically based on the host architecture.
# Testing
> Test philosophy, categories, and how to run ~11,000 lines of tests.
\~11,000 lines of tests across 25 test files. Zero mocks for VM tests — all Firecracker integration tests run on real microVMs.
## Philosophy
[Section titled “Philosophy”](#philosophy)
**Test at the real boundary.** The server tests use real Docker/Firecracker engines, not mocks. Mocks silently accept any input and hide broken behavior.
**Test the protocol without VMs.** Lohar has a test mode that listens on Unix sockets instead of vsock/TCP. The entire protocol handler test suite (40+ tests) runs on macOS in under 2 seconds without root or KVM.
**Test performance with percentiles.** Tests report p50/p95/p99, not averages. Assertions are on p99 to catch regressions.
## Running tests
[Section titled “Running tests”](#running-tests)
### Protocol and agent tests (macOS/Linux, no root)
[Section titled “Protocol and agent tests (macOS/Linux, no root)”](#protocol-and-agent-tests-macoslinux-no-root)
```bash
go test ./pkg/agent/proto/ # frame encoding, round-trips
go test ./cmd/lohar/ # agent handlers via Unix socket
go test ./pkg/agent/ # host-side client
```
### Server tests (requires Docker)
[Section titled “Server tests (requires Docker)”](#server-tests-requires-docker)
```bash
go test ./pkg/server/ # API, proxy, streaming, thermal
```
### Firecracker integration tests (Linux, KVM, root)
[Section titled “Firecracker integration tests (Linux, KVM, root)”](#firecracker-integration-tests-linux-kvm-root)
```bash
sudo go test ./pkg/engine/firecracker/ -count=1
```
### Performance tests
[Section titled “Performance tests”](#performance-tests)
```bash
sudo go test ./pkg/engine/firecracker/ -run TestPerf -count=1
```
## Test categories
[Section titled “Test categories”](#test-categories)
| Category | Location | Requires | Count |
| ----------- | ------------------------- | ----------- | ----- |
| Protocol | `pkg/agent/proto/` | Nothing | \~15 |
| Agent | `cmd/lohar/` | Nothing | \~40 |
| Client | `pkg/agent/` | Nothing | \~10 |
| Server | `pkg/server/` | Docker | \~30 |
| Engine | `pkg/engine/firecracker/` | KVM, root | \~25 |
| Performance | `pkg/engine/firecracker/` | KVM, root | \~8 |
| CLI | `cmd/bhatti/` | Firecracker | \~11 |
| Recovery | `cmd/bhatti/` | Nothing | \~8 |
### What each category covers
[Section titled “What each category covers”](#what-each-category-covers)
* **Protocol tests** — binary framing: round-trips, EOF handling, max frame size, concurrent writes
* **Agent tests** — exec (exit codes, env vars, kill, process groups), TTY sessions (detach/reattach, scrollback, resize), files (read/write/stat/ls, truncation, atomicity)
* **Server tests** — API endpoints, auth, streaming exec (NDJSON), proxy, thermal cycle, volume management
* **Engine tests** — full VM lifecycle, snapshot/restore, diff snapshots, networking (bridge, cross-VM), TAP cleanup
* **Performance tests** — exec latency, file I/O, concurrent operations, warm→exec, snapshot timings
* **CLI tests** — end-to-end: create → exec → file → shell → destroy
* **Recovery tests** — daemon restart scenarios: crashed VMs, missing snapshots, orphaned state
# Custom Domain
> Set up a domain so bhatti publish creates real URLs like alias.yourdomain.com.
By default, `bhatti publish` generates URLs using the server’s IP or hostname. With a custom domain, published sandboxes get proper URLs like `my-app.yourdomain.com` with TLS.
## What you need
[Section titled “What you need”](#what-you-need)
* A domain (e.g. `yourdomain.com`)
* DNS access to create A and wildcard records
* Either a wildcard TLS certificate or an email for Let’s Encrypt
## DNS records
[Section titled “DNS records”](#dns-records)
Point your API hostname and a wildcard for published sandboxes to your server:
```plaintext
A api.yourdomain.com → <your-server-ip>
A *.yourdomain.com → <your-server-ip>
```
The wildcard record is what makes `my-app.yourdomain.com` work — every published alias becomes a subdomain.
## Server configuration
[Section titled “Server configuration”](#server-configuration)
Add the `domain` block to `/etc/bhatti/config.yaml`:
### Option A: Wildcard certificate (recommended)
[Section titled “Option A: Wildcard certificate (recommended)”](#option-a-wildcard-certificate-recommended)
If you have a wildcard cert (from your DNS provider, Cloudflare Origin CA, or `certbot --manual` with DNS challenge):
```yaml
domain:
api_host: api.yourdomain.com
proxy_zone: yourdomain.com
tls_cert: /etc/bhatti/wildcard.pem
tls_key: /etc/bhatti/wildcard-key.pem
```
### Option B: Let’s Encrypt (automatic per-alias certs)
[Section titled “Option B: Let’s Encrypt (automatic per-alias certs)”](#option-b-lets-encrypt-automatic-per-alias-certs)
If you don’t have a wildcard cert, bhatti can issue individual certificates via Let’s Encrypt:
```yaml
domain:
api_host: api.yourdomain.com
proxy_zone: yourdomain.com
acme_email: you@example.com
```
Caution
Let’s Encrypt has a rate limit of 50 new certificates per week per registered domain. If you create and publish many sandboxes rapidly (e.g. per-PR previews in CI), you’ll hit this limit. Use a wildcard cert for production preview environments.
## What changes in domain mode
[Section titled “What changes in domain mode”](#what-changes-in-domain-mode)
Without `domain` config, bhatti listens on `:8080` for everything.
With `domain` config, bhatti starts three listeners:
| Port | Purpose |
| ---------------- | ---------------------------------------------------------------- |
| `:443` | API (`api.yourdomain.com`) + proxy (`*.yourdomain.com`) with TLS |
| `:80` | ACME challenges + HTTPS redirect |
| `127.0.0.1:8080` | Internal API (localhost only — health checks, admin) |
Routing is by `Host` header: requests to `api.yourdomain.com` go through normal API auth. Requests to `anything-else.yourdomain.com` go to the public proxy (no auth — this is the published URL).
## Restart
[Section titled “Restart”](#restart)
```bash
sudo systemctl restart bhatti
```
Check the logs for:
```plaintext
bhatti listening (domain mode)
api: https://api.yourdomain.com
proxy: https://*.yourdomain.com
```
## Verify
[Section titled “Verify”](#verify)
```bash
bhatti create --name test
bhatti publish test -p 8080 -a hello
# → https://hello.yourdomain.com
```
## Updating CLI config
[Section titled “Updating CLI config”](#updating-cli-config)
Remote CLI users should update their endpoint:
```bash
bhatti setup
# API endpoint: https://api.yourdomain.com
# API key: ****
```
## Getting a wildcard certificate
[Section titled “Getting a wildcard certificate”](#getting-a-wildcard-certificate)
### Cloudflare (if your DNS is on Cloudflare)
[Section titled “Cloudflare (if your DNS is on Cloudflare)”](#cloudflare-if-your-dns-is-on-cloudflare)
Origin CA certificates are free and last 15 years. Create one in the Cloudflare dashboard → SSL/TLS → Origin Server → Create Certificate. Select `*.yourdomain.com` and `yourdomain.com`. Download the PEM files.
### certbot with DNS challenge
[Section titled “certbot with DNS challenge”](#certbot-with-dns-challenge)
```bash
sudo certbot certonly --manual --preferred-challenges dns \
-d "*.yourdomain.com" -d "yourdomain.com"
```
This asks you to create a DNS TXT record to prove domain ownership. After validation, certs are saved to `/etc/letsencrypt/live/yourdomain.com/`. Renew every 90 days.
### acme.sh (lightweight alternative)
[Section titled “acme.sh (lightweight alternative)”](#acmesh-lightweight-alternative)
```bash
acme.sh --issue -d "*.yourdomain.com" -d "yourdomain.com" --dns dns_cf
```
Supports automatic DNS API integration with most providers. See [acme.sh DNS API](https://github.com/acmesh-official/acme.sh/wiki/dnsapi).
# Images & Tiers
> Built-in rootfs tiers, custom-image workflows, and how image scoping and sharing work.
Sandbox root filesystems are ext4 images. bhatti ships pre-built Ubuntu 24.04 *tier* images out of the box and supports three ways to add your own: pull from a public OCI registry, import a Docker image, or save a configured sandbox as an image.
The full CLI surface is in [Images reference](/v1/docs/reference/cli/images/). This page covers the tier landscape, the typical workflows, and how scoping/sharing work — none of which is in the per-command pages.
## Built-in tiers
[Section titled “Built-in tiers”](#built-in-tiers)
bhatti ships four built-in tiers. Each is a pre-built Ubuntu 24.04 rootfs that builds on top of `minimal`:
| Tier | Adds | Size |
| ----------------------------------------------- | -------------------------------------- | -------: |
| [`minimal`](/v1/docs/managing/tiers/) | (base) | \~200 MB |
| [`browser`](/v1/docs/managing/tiers/browser/) | Chromium, Playwright, Node 22 | \~600 MB |
| [`docker`](/v1/docs/managing/tiers/docker/) | Docker Engine, buildx, compose, binfmt | \~550 MB |
| [`computer`](/v1/docs/managing/tiers/computer/) | XFCE, KasmVNC, Chromium, full desktop | \~1.5 GB |
```bash
bhatti create --name scraper --image browser
bhatti create --name ci --image docker
```
The server install prompts for one tier on first run; install more later with `sudo bhatti update --tiers all` (or a comma-separated list). Tiers are auto-discovered from `<data_dir>/images/rootfs-<tier>-<arch>.ext4` — no hardcoded list.
The deep dive for each tier — how its long-running daemons are managed, what env knobs you can pass, sizing, troubleshooting — lives at [Tiers](/v1/docs/managing/tiers/). The rest of this page is about images you build *on top* of those tiers.
## Building custom images
[Section titled “Building custom images”](#building-custom-images)
Three workflows depending on where the source lives.
### Pull from a public registry
[Section titled “Pull from a public registry”](#pull-from-a-public-registry)
```bash
bhatti image pull python:3.12
bhatti image pull node:22-slim --name node-22
```
Async, server-side. The CLI shows progress and exits when the image is ready.
### Import from local Docker
[Section titled “Import from local Docker”](#import-from-local-docker)
```bash
docker pull ghcr.io/myorg/private:latest # any auth Docker can do
bhatti image import ghcr.io/myorg/private:latest
```
The CLI runs `docker save` locally and streams the tar to the server. This is the recommended path for private registries — Docker handles your existing auth, bhatti just receives bytes.
For tarballs without Docker:
```bash
docker save ubuntu:24.04 > /tmp/ubuntu.tar
bhatti image import --tar /tmp/ubuntu.tar --name ubuntu-24
```
### Save a configured sandbox
[Section titled “Save a configured sandbox”](#save-a-configured-sandbox)
```bash
bhatti create --name build --image minimal
bhatti exec build -- apt-get update && apt-get install -y nodejs pnpm
bhatti exec build -- pnpm install -g some-tool
bhatti image save build --name node-stack
```
The captured image now stamps out new sandboxes with everything pre-installed:
```bash
bhatti create --name worker-1 --image node-stack
bhatti create --name worker-2 --image node-stack
```
Only the **rootfs** is captured — persistent volumes are not part of the image. If your stack puts work in a volume, snapshot the volume separately ([`bhatti volume clone`](/v1/docs/reference/cli/volumes/clone/)) and re-attach when stamping.
## Disk usage
[Section titled “Disk usage”](#disk-usage)
A common question when you first look at `/var/lib/bhatti/`: *“how much disk does each sandbox actually cost?”* The answer depends heavily on which filesystem the data dir lives on, and the difference between what `ls -lh` shows and what `du -h` shows is the whole story.
A fresh sandbox dir looks like this:
```plaintext
/var/lib/bhatti/
images/
rootfs-computer-amd64.ext4 4.0G (logical) base image
rootfs-minimal-amd64.ext4 1.0G
vmlinux-amd64 43M
sandboxes/<id>/
rootfs.ext4 1.0G (logical) per-sandbox CoW
config.ext4 1.0M (logical) env, secrets, mounts
mem.snap 1.0G (logical, only when cold)
vm.snap ~50K (only when cold)
```
The `.ext4` files are sparse-allocated block devices. `ls -lh` shows the *logical* size (the maximum the file could be); `du -h` shows what’s actually on disk. The gap can be massive.
On **btrfs** with reflink + zstd:1 compression (the recommended setup, see [Filesystem](/v1/docs/self-hosting/#filesystem-recommended-btrfs)), `compsize` from `btrfs-progs` shows the breakdown explicitly:
```bash
$ sudo compsize /var/lib/bhatti/sandboxes/abc123/
Type Perc Disk Usage Uncompressed Referenced Size
TOTAL 18% 45M 250M 1.0G
```
“Referenced Size 1.0G” is what `ls -lh` shows: the sandbox sees a 1 GiB block device. “Disk Usage 45M” is what’s actually allocated on disk after reflink-sharing with the base image and zstd compression of the unique parts. The 95.5% gap is what makes running many sandboxes from one base affordable.
### The per-sandbox cost
[Section titled “The per-sandbox cost”](#the-per-sandbox-cost)
The marginal cost of adding a sandbox depends on the filesystem:
| Filesystem | Base image | Per sandbox (logical) | Per sandbox (physical) | 10 sandboxes total |
| ---------------------------- | ---------- | --------------------- | ----------------------- | ------------------ |
| **ext4** | 1.0 GiB | 1.0 GiB | \~232 MiB (sparse copy) | \~3.3 GiB |
| **btrfs** (reflink + zstd:1) | 1.0 GiB | 1.0 GiB | \~5–20 MiB | \~1.1 GiB |
Numbers from `agni-01` (1 GiB base computer-tier rootfs). On ext4 each sandbox is a sparse copy of the base — the *used* extents of the source, not the full logical size, but still hundreds of MiB per sandbox. On btrfs the per-sandbox cost is the bytes the sandbox actually writes (config tweaks, log lines, package installs since boot), typically single-digit MiB until you do something heavy.
For `mem.snap` files — written when a sandbox goes cold, equal to the configured memory — the difference is even larger: guest RAM is dominated by zero pages and highly-compressible kernel/userspace, so zstd:1 on btrfs hits about 21×. A stopped 1 GiB sandbox occupies \~48 MiB on disk on btrfs vs. \~1 GiB on ext4.
On ext4 the answer to “is there a way to streamline storage?” is **yes — switch to btrfs.** No bhatti config knob needed; reflink takes effect automatically as soon as `/var/lib/bhatti` is btrfs. See [Self-hosting → Filesystem](/v1/docs/self-hosting/#filesystem-recommended-btrfs) for the recipe and [Storage → What changes on ext4](/v1/docs/under-the-hood/storage/#what-changes-on-ext4) for the full cost-by-filesystem picture.
## Scoping and sharing
[Section titled “Scoping and sharing”](#scoping-and-sharing)
Images you create are private to your user — other users can’t see them in `image list`, can’t reference them by name, can’t read the underlying file.
To share an image with a specific user (or list of users):
```bash
sudo bhatti image share my-image --user alice --user bob
```
This command operates directly on the local SQLite database, so it requires running on the server with DB access (`sudo`). It is not an HTTP API call. There’s no “share with everyone” mode — sharing is always with named users. Inspect current shares with `--list`; revoke with [`bhatti image unshare`](/v1/docs/reference/cli/images/unshare/).
System tier images (`minimal`, `browser`, …) are visible to everyone on the server automatically; they live under `user_id=""` in the database.
## See also
[Section titled “See also”](#see-also)
* [Tiers](/v1/docs/managing/tiers/) — the four built-in starting points, with per-tier deep dives
* [Images reference](/v1/docs/reference/cli/images/) — every command
* [Adding a tier](/v1/docs/contributing/adding-a-tier/) — build a new system tier
# Secrets
> How encryption, scoping, and env-var injection work for sandbox secrets.
Secrets are values you’d rather not put on a command line. bhatti stores them encrypted, scoped per user, and injects them as environment variables when a sandbox boots.
The CLI surface lives in [Secrets reference](/v1/docs/reference/cli/secrets/). This page covers the *model* — what’s encrypted, what isn’t, and the rules for which env var wins when multiple sources set the same name.
## Encryption at rest
[Section titled “Encryption at rest”](#encryption-at-rest)
On first start the daemon generates an [age](https://age-encryption.org/) keypair at `<data_dir>/age.key`. Every secret you set is encrypted under this key. Plaintext is never written to disk on the server — not in the database, not in temp files.
The age key file is the only thing that decrypts your secrets. Back it up alongside the database; lose either and the secrets become unrecoverable.
## Scoping
[Section titled “Scoping”](#scoping)
Secrets are per-user. Other users can’t list, read, or reference them by name. There’s no shared / global secret store.
`secret list` returns names + timestamps but never the encrypted blob — once a value is set, you can replace it with `secret set NAME new-value` but you can’t read it back.
## How injection works
[Section titled “How injection works”](#how-injection-works)
When a sandbox is created with `--secret API_KEY`:
1. The server fetches the encrypted blob from the secrets table for that user.
2. It decrypts under the age key.
3. The plaintext is written into the sandbox’s config drive (`config.ext4`) as part of the env-var bundle.
4. lohar reads the config drive at boot and exports `API_KEY=<plaintext>` to every command run inside the sandbox.
The config drive is unmounted after boot so the value isn’t readable from the running rootfs without lohar’s help.
## What happens on wake
[Section titled “What happens on wake”](#what-happens-on-wake)
Nothing secret-related. Decryption happens exactly once, at sandbox *create*, when the daemon reads the ciphertext from SQLite, decrypts it with the age key on disk, and writes the plaintext into the config drive (`config.ext4`). From that moment on, the secret lives as bytes in guest RAM and is captured into `mem.snap` on snapshot.
When a [cold sandbox wakes](/v1/docs/under-the-hood/thermal-states/#cold--hot):
* The server does **not** re-read from SQLite.
* It does **not** load the age key.
* It does **not** rewrite the config drive.
* It loads `mem.snap` into a fresh Firecracker process and resumes. The env vars are already in the guest’s process memory.
The age key (`<data_dir>/age.key`) is only loaded on an explicit encrypt or decrypt call — `bhatti secret set` or `bhatti create --secret`. It is **not** kept in long-lived daemon memory.
This matters for latency: a cold wake doesn’t pay any secret-reconstruction cost. The 360 ms p50 number on the homepage is dominated by the `mem.snap` disk read, not by anything secret-handling — see [Storage → Cold wake and the page cache](/v1/docs/under-the-hood/storage/#cold-wake-and-the-page-cache).
### Threat model
[Section titled “Threat model”](#threat-model)
* **Stolen `bhatti.db`**: safe (ciphertext only, no key).
* **Cross-tenant access**: safe (per-user scoping + auth).
* **Stolen `bhatti.db` + `age.key` together**: not safe; both files live in `<data_dir>` and an attacker with filesystem access has everything.
* **Host root compromise**: not safe.
* **`mem.snap` exfiltration**: not safe; secrets are env vars in guest RAM and `mem.snap` is unencrypted on disk.
The host root is the trust boundary, same as Vault-on-a-box. If you need to weaken that — wrap `age.key` in KMS/HSM. We do not currently support that; it’s an open design question.
## Env var priority inside a sandbox
[Section titled “Env var priority inside a sandbox”](#env-var-priority-inside-a-sandbox)
When a command runs, env vars resolve in this order — later wins:
```plaintext
1. defaults (PATH, HOME, TERM, LANG)
↓
2. secrets (--secret NAME at create time)
↓
3. sandbox env (--env K=V at create time)
```
There is **no** per-`bhatti exec` env override. The `POST /sandboxes/:id/exec` endpoint doesn’t accept an `env` field — anything passed there is silently ignored. To inject env vars at exec time, use the WebSocket exec path (which does take an `env` map on the initial JSON command spec). For most use cases, set them at create time and accept that they apply to every command.
## Caveat: `--template` ignores `--secret`
[Section titled “Caveat: --template ignores --secret”](#caveat---template-ignores---secret)
Creating a sandbox from a template silently drops request-side `--secret` and `--file` flags. Only the secrets pre-configured on the template are honoured. If you need request-time secrets, create directly without `--template`.
## See also
[Section titled “See also”](#see-also)
* [Secrets reference](/v1/docs/reference/cli/secrets/) — `set`, `list`, `delete`
* [`bhatti create --secret`](/v1/docs/reference/cli/sandbox/create/) — inject at sandbox boot
# Tiers
> Built-in rootfs tiers, the per-tier operational story, and which one to pick.
A *tier* is a pre-built Ubuntu 24.04 rootfs that ships as a starting point for sandboxes. bhatti has four. Each builds on `minimal` and adds tooling for a specific workload.
```bash
bhatti create --name dev --image docker
bhatti create --name scraper --image browser
bhatti create --name desk --image computer --cpus 2 --memory 4096
```
Pick the one closest to what you’re doing — every tier is a real Ubuntu, so you can always `apt-get install` more on top.
## The four tiers
[Section titled “The four tiers”](#the-four-tiers)
| Tier | Adds on top of `minimal` | Approx size | Typical use |
| ------------------------- | -------------------------------------- | ----------: | ---------------------------------------- |
| [`minimal`](#minimal) | (nothing — bare base) | \~200 MB | Custom builds, anything CLI-first |
| [`browser`](./browser/) | Chromium, Playwright, Node 22 | \~600 MB | Scraping, CDP automation, headless tests |
| [`docker`](./docker/) | Docker Engine, buildx, compose, binfmt | \~550 MB | Docker-in-VM, OCI builds, **multi-arch** |
| [`computer`](./computer/) | XFCE, KasmVNC, Chromium, full desktop | \~1.5 GB | Visual / agent-driven desktop sessions |
The server install prompts for one tier on first run. Install more later:
```bash
sudo bhatti update --tiers all
# or pick: --tiers docker,browser
```
The server auto-discovers any `rootfs-<tier>-<arch>.ext4` it finds in its images directory, so installing a tier is just a download — no config changes, no daemon restart logic to think about.
## Units shipped by each tier
[Section titled “Units shipped by each tier”](#units-shipped-by-each-tier)
What’s actually running inside each tier, post-boot. Every unit listed here is managed by lohar’s [`systemctl` shim](/v1/docs/under-the-hood/lohar-the-blacksmith/) — no real systemd anywhere — and the same three commands (`systemctl status`, `journalctl -u`, `systemctl restart`) work the same way regardless of tier.
| Tier | Unit | `Type=` | What it does |
| ---------- | ---------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `minimal` | (none shipped) | — | Provides the shim infrastructure. Anything you `apt install` (openssh-server, postgresql, nginx) becomes shim-managed on first boot. |
| `browser` | `headless-chrome.service` | `simple` | `headless_shell` with `--remote-debugging-port=9222`. Long-running, no PIDFile, restarted on crash. |
| `docker` | `docker.service` | `notify` | `dockerd` from upstream `docker-ce`. Drop-in binds the socket directly (no `fd://` socket activation) and chmods it for the `lohar` user. |
| `computer` | `kasmvnc-firstboot.service` | `oneshot` | Generates a per-sandbox VNC password on first boot. `RemainAfterExit=yes` so dependent units’ `After=` clauses are satisfied. |
| `computer` | `kasmvnc.service` | `simple` | `Xkasmvnc` — X server + RFB→WebSocket gateway, the endpoint on `:6080`. After `kasmvnc-firstboot.service`. |
| `computer` | `xfce-session.service` | `simple` | XFCE desktop session. `Requires=kasmvnc.service` so killing kasmvnc cascades. |
| `computer` | `bhatti-display-env.service` | `oneshot` | Writes `DISPLAY=:99` into the env file `bhatti exec` reads, so `screenshot` and `xdotool` Just Work without per-call env. |
If a tier’s daemon crashes, the shim’s `Restart=on-failure` policy brings it back; you see the crash and the restart in `journalctl -u <unit>`. If you edit a config file, `systemctl restart <unit>` picks it up. Every unit in this table is `Restart=on-failure` (long-running daemons) or `oneshot` (firstboot helpers). The cgroup-placement mechanism that makes `systemctl stop` reliable for forking daemons like `Xkasmvnc` is documented at [How services are spawned](/v1/docs/under-the-hood/lohar-the-blacksmith/#how-services-are-spawned).
## The common operator story
[Section titled “The common operator story”](#the-common-operator-story)
Every long-running process in every built-in tier is managed by lohar’s **`systemctl` shim** ([lohar internals](/v1/docs/under-the-hood/lohar-the-blacksmith/)). The same three commands — `systemctl status`, `journalctl -u`, `systemctl restart` — work the same way regardless of which tier you started from:
```bash
bhatti exec dev -- systemctl status docker
bhatti exec dev -- journalctl -u docker -n 50
bhatti exec dev -- systemctl restart docker
```
If a managed daemon crashes, the shim’s `Restart=on-failure` policy brings it back; you see the crash in `journalctl`. If you edit a config file, `systemctl restart <unit>` picks it up. The shape is the same on every tier — that’s the point of having one shim instead of one bespoke init script per tier.
A few practical points the shim deliberately doesn’t try to replicate from real systemd, because they cost more than they’re worth in a microVM sandbox:
* **No socket activation.** Daemons that ship with `ExecStart=… -H fd://` need a drop-in to bind the socket directly. The docker tier does this for `docker.service`; you only need to know if you write your own units.
* **No D-Bus.** Daemons that default to the `systemd` cgroup driver (notably dockerd) need a config override to use `cgroupfs`. Again — the docker tier handles this; flagged only because the failure mode is opaque (`dial unix /run/systemd/private: …`) if you build a tier from scratch.
* **No `loginctl`, no `journald` binary format.** Single-user sandbox, plain-text per-unit logs in `/var/log/bhatti/`. `journalctl -u <unit>` reads those files; `-f` follows.
The deeper rationale is in [Decisions & learnings](/v1/docs/under-the-hood/decisions/) under “Why no real systemd.”
## Sizing and resources
[Section titled “Sizing and resources”](#sizing-and-resources)
| Tier | Minimum sensible | Comfortable |
| ---------- | ---------------- | ----------- |
| `minimal` | 1 vCPU / 256 MB | 1 / 512 |
| `browser` | 1 / 1024 | 2 / 2048 |
| `docker` | 2 / 2048 | 4 / 4096 |
| `computer` | 2 / 2048 | 4 / 4096 |
KasmVNC’s encoder thread count is sized to `nproc - 1`, so `--cpus 2` is the practical floor for the computer tier (any less and the encoder, X server, XFCE, and Chromium all share one core). For the docker tier, anything memory-intensive (large `buildx` builds, multi-container compose stacks) wants 4+ GB.
## Minimal
[Section titled “Minimal”](#minimal)
The base. Bare Ubuntu 24.04 with curl, sudo, ca-certificates, fuse3, and the `systemd-resolved` apt pin so package installs don’t accidentally pull in a real systemd. lohar is PID 1.
Useful when:
* You’re going to install everything yourself anyway.
* You want the smallest possible rootfs for snapshot/restore latency.
* You’re building a custom image with [`bhatti image save`](/v1/docs/reference/cli/images/save/) and want to start from clean.
The `lohar` user (uid 1000) has passwordless sudo. `bhatti exec` runs as this user. The shim’s `systemctl` / `journalctl` symlinks are wired in at this layer, so any service-aware package you install (openssh-server, postgresql, redis-server, nginx) Just Works under the shim.
```bash
bhatti create --name dev --image minimal
bhatti exec dev -- sudo apt-get update
bhatti exec dev -- sudo apt-get install -y postgresql redis-server
bhatti exec dev -- systemctl status postgresql redis-server
```
## Adding a tier
[Section titled “Adding a tier”](#adding-a-tier)
If your team has a stack that’s repeated often, you can ship a tier of your own. See [Adding a tier](/v1/docs/contributing/adding-a-tier/) for the build script convention, CI matrix entry, and install-flow integration.
## See also
[Section titled “See also”](#see-also)
* [Images & custom builds](/v1/docs/managing/images/) — pull from OCI, import from Docker, save a configured sandbox
* [Adding a tier](/v1/docs/contributing/adding-a-tier/) — build a new system tier from scratch
* [Lohar: the agent inside every VM](/v1/docs/under-the-hood/lohar-the-blacksmith/) — how the `systemctl` shim works
* [Decisions & learnings](/v1/docs/under-the-hood/decisions/) — why no real systemd, what’s intentionally out of scope
# Browser tier
> Headless Chromium with the Chrome DevTools Protocol exposed on :9222.
The `browser` tier ships Chromium (via Playwright’s `headless_shell` build), Playwright itself, and Node 22. Chromium starts at boot with CDP exposed on `:9222` ready to drive — no display server, no X, no VNC. The right tier for scraping, programmatic CDP control, headless tests, and PDF / screenshot rendering.
```bash
bhatti create --name scraper --image browser --cpus 2 --memory 2048
bhatti exec scraper -- curl -s http://localhost:9222/json/version | jq .Browser
# "Chrome/<version>"
```
## What’s in it
[Section titled “What’s in it”](#whats-in-it)
| Component | Source |
| ---------------------------------- | ------------------------------------------------------------------- |
| Chromium (`headless_shell` binary) | Playwright’s pinned download (`~/.cache/ms-playwright/chromium-*/`) |
| Playwright | `npm install -g playwright` |
| Node 22 | NodeSource apt repo |
| Chromium runtime deps | `npx playwright install-deps chromium` |
Playwright’s `headless_shell` is used rather than full Chrome — it’s the dedicated headless build with reliable CDP. The full Chrome binary has known CDP-attach issues in headless mode that show up as silent stream hangs.
## How Chromium is started
[Section titled “How Chromium is started”](#how-chromium-is-started)
Managed by lohar’s `systemctl` shim as `headless-chrome.service` (since v1.11.9). The unit ships with these defaults:
```ini
[Service]
Type=simple
Environment=CHROME_REMOTE_PORT=9222
Environment=CHROME_FLAGS=
EnvironmentFile=-/run/bhatti/config-env
ExecStart=/root/.cache/ms-playwright/chromium_headless_shell-<rev>/chrome-linux/headless_shell \
--no-sandbox \
--disable-gpu \
--disable-dev-shm-usage \
--remote-debugging-port=${CHROME_REMOTE_PORT} \
--remote-debugging-address=0.0.0.0 \
$CHROME_FLAGS
Restart=on-failure
RestartSec=2s
```
The path to `headless_shell` is resolved at build time by the tier script (Playwright stamps a version-suffixed directory) and baked into the unit. Operator UX is the same as every other tier:
```bash
bhatti exec scraper -- systemctl status headless-chrome
bhatti exec scraper -- journalctl -u headless-chrome -n 50
bhatti exec scraper -- systemctl restart headless-chrome
```
If Chromium crashes, the shim restarts it after 2 s. The crash + restart show up in `journalctl -u headless-chrome`.
## Driving it from outside the sandbox
[Section titled “Driving it from outside the sandbox”](#driving-it-from-outside-the-sandbox)
CDP is HTTP + WebSocket. Publish the port and connect from anywhere:
```bash
bhatti publish scraper -p 9222
# → https://scraper-abc123.bhatti.sh
```
From your host, point any CDP client at the published URL. Playwright on the host:
```js
const browser = await chromium.connectOverCDP('https://scraper-abc123.bhatti.sh');
```
Puppeteer:
```js
const browser = await puppeteer.connect({
browserWSEndpoint: 'wss://scraper-abc123.bhatti.sh/devtools/browser/…'
});
```
You can fetch the WebSocket endpoint from `/json/version`:
```bash
curl -s https://scraper-abc123.bhatti.sh/json/version | jq -r .webSocketDebuggerUrl
```
## Driving it from inside the sandbox
[Section titled “Driving it from inside the sandbox”](#driving-it-from-inside-the-sandbox)
`bhatti exec` runs as uid 1000 with Node and Playwright already in `$PATH`. The fastest agent loop is to write your script as a file and exec it:
```bash
bhatti file write scraper /workspace/scrape.js < ./scrape.js
bhatti exec scraper -- node /workspace/scrape.js
```
The Playwright globals (`chromium`, `firefox`, `webkit`) all work, but only Chromium has a running browser to attach to via CDP. To launch fresh Chromium instances from your script (rather than the boot-time one), use `chromium.launch({ executablePath: '/root/.cache/ms-playwright/chromium-*/chrome-linux/headless_shell' })` — the boot-time instance is a convenience, not a requirement.
## Tunables
[Section titled “Tunables”](#tunables)
Set at create time via `bhatti create --env K=V,K=V`. The values flow into the unit through lohar’s [config-env bridge](/v1/docs/under-the-hood/lohar-the-blacksmith/#config-env-bridge):
| Variable | Default | Effect |
| -------------------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `CHROME_REMOTE_PORT` | `9222` | CDP port to listen on |
| `CHROME_FLAGS` | `""` | Extra space-separated flags appended to ExecStart (`--user-agent=…`, `--proxy-server=…`, `--lang=…`, etc.) |
```bash
bhatti create --name scraper --image browser \
--env "CHROME_FLAGS=--user-agent=Mozilla/5.0 (compatible; bhatti-bot/1.0)"
```
To edit flags on a running sandbox, drop a unit override and restart:
```bash
bhatti exec scraper -- sudo mkdir -p /etc/systemd/system/headless-chrome.service.d
bhatti exec scraper -- sudo tee /etc/systemd/system/headless-chrome.service.d/custom.conf <<'EOF'
[Service]
Environment=CHROME_FLAGS=--proxy-server=http://my-proxy:3128
EOF
bhatti exec scraper -- sudo systemctl restart headless-chrome
```
## Sizing
[Section titled “Sizing”](#sizing)
| Workload | `--cpus` | `--memory` |
| ------------------------------ | -------: | ---------: |
| Single-page scrapes, CDP drive | 1 | 1024 |
| Concurrent Playwright tests | 2 | 2048 |
| PDF rendering at scale | 2–4 | 2048–4096 |
Chromium is memory-hungry — 512 MB will OOM on any non-trivial page. The 1 GB default minimum is a real floor.
## Troubleshooting
[Section titled “Troubleshooting”](#troubleshooting)
| Symptom | Cause | Fix |
| ----------------------------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `curl /json/version` returns connection refused | Chromium failed to start at boot | `systemctl status headless-chrome`; `journalctl -u headless-chrome` for the crash reason |
| Chromium crashed mid-session | crash recovered via `Restart=on-failure`, but flapping if it crashes repeatedly | `systemctl status headless-chrome` shows `(activating)` if currently restarting, `(failed)` if start-limit hit; `journalctl -u headless-chrome` for the underlying error |
| Pages OOM with `Aw, Snap!` | Memory too low | bump `--memory` to 2048+ |
| `chrome-sandbox` permission errors | `--no-sandbox` got dropped from a custom drop-in | check `/etc/systemd/system/headless-chrome.service.d/` — the upstream unit always passes `--no-sandbox` |
| CHROME\_FLAGS not picked up | `--env` syntax: must be comma-separated, not repeated `--env` flags | `bhatti create --env "K1=V1,K2=V2"` (single string, comma-separated) |
## See also
[Section titled “See also”](#see-also)
* [Tiers overview](/v1/docs/managing/tiers/)
* [Docker tier](./docker/) — when you’d combine Docker + headless Chromium
* [Computer tier](./computer/) — when you need a *visible* browser (KasmVNC + Chromium UI)
* [`bhatti publish`](/v1/docs/reference/cli/networking/publish/) — exposing :9222 publicly
# Computer tier
> A full XFCE desktop in a microVM, served over KasmVNC on a single web port.
The `computer` tier is a full graphical Linux desktop: KasmVNC + XFCE + Chromium, served over one HTTP/WebSocket port (`6080`) as a browser-based web client. The right tier for visual agent-driven work (a model that sees a real desktop), demos, and any scenario where “open a browser and look at it” is the interface.
```bash
bhatti create --name desk --image computer --cpus 2 --memory 4096 --disk-size 8192
bhatti publish desk -p 6080
bhatti exec desk -- vnc-creds # username + password
# Open the URL printed by `publish` in your browser, log in.
```
## Why `--cpus 2` is the practical floor
[Section titled “Why --cpus 2 is the practical floor”](#why---cpus-2-is-the-practical-floor)
KasmVNC’s encoder thread count is sized to the guest’s vCPUs (`nproc - 1`, leaving one core for the desktop itself). On `--cpus 1` the encoder, the X server, XFCE, and Chromium all share one core — expect <10 fps and laggy input. Two cores is usable; four is comfortable.
## What’s in it
[Section titled “What’s in it”](#whats-in-it)
| Component | Role |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **KasmVNC** (`Xkasmvnc`) | Combined X server + RFB → WebSocket gateway. Replaces the older Xvfb + x11vnc + noVNC + websockify stack with one binary, one port. |
| **XFCE** (`startxfce4`) | Lightweight desktop environment. Window manager, panel, file manager, app launcher. |
| **Chromium** | The browser. `chromium-browser <url>` wrapper applies sane flags (`--no-sandbox` because we’re in a VM, `--test-type` to suppress nags). |
| `xdotool`, `scrot`, `xdpyinfo` | Mouse/keyboard input, screenshots, display introspection. Used by the agent-helper scripts below. |
## Credentials
[Section titled “Credentials”](#credentials)
A random 16-character password is generated **on first boot of each sandbox** (not at image-bake time):
* Hashed into `/root/.kasmpasswd` for KasmVNC.
* Stored cleartext in `/root/.vnc/cleartext` (root-only, mode `0600`) for the `vnc-creds` helper.
* Each sandbox gets its own; the published rootfs image carries no shared secret.
* Snapshot/resume preserves the password (the file already exists; firstboot generation is skipped).
Retrieve them anytime:
```bash
bhatti exec desk -- vnc-creds # human-readable
bhatti exec desk -- vnc-creds --json # for scripts/agents
```
Caution
[`bhatti image save`](/v1/docs/reference/cli/images/save/) on a running computer-tier sandbox **bakes the current password into the saved image**. Treat saved-from-running images like any other secret-bearing artifact. If you share such an image, anyone with it has VNC access to every sandbox stamped from it. The clean alternative is to delete `/root/.kasmpasswd` and `/root/.vnc/cleartext` before saving, so firstboot regen kicks in on every new sandbox.
## Tunables
[Section titled “Tunables”](#tunables)
Pass at create time with `--env` (comma-separated):
| Variable | Default | What it controls |
| ---------------- | ----------- | ------------------------------------------------------------------------- |
| `DISPLAY_WIDTH` | `1280` | X server geometry width |
| `DISPLAY_HEIGHT` | `720` | X server geometry height |
| `DISPLAY_DEPTH` | `24` | Colour depth (16 / 24 / 32) |
| `KASM_FRAMERATE` | `60` | Max frames/sec the encoder will emit (matches KasmVNC’s upstream default) |
| `KASM_THREADS` | `nproc - 1` | Encoder thread count; default leaves one vCPU for the desktop |
```bash
bhatti create --name desk --image computer --cpus 4 --memory 4096 \
--env "DISPLAY_WIDTH=1920,DISPLAY_HEIGHT=1080,KASM_FRAMERATE=30"
```
`KASM_FRAMERATE=30` is worth setting on low-bandwidth links — encoder load drops roughly linearly with framerate, and 30 fps is more than enough for desktop work.
## Beyond the env knobs
[Section titled “Beyond the env knobs”](#beyond-the-env-knobs)
KasmVNC has dozens of options bhatti deliberately doesn’t surface (dynamic quality bounds, video-mode thresholds, scaling algorithms, DLP/clipboard policy, etc.). For those, edit `/etc/kasmvnc/kasmvnc.yaml` inside the sandbox and reconnect. The upstream docs are authoritative:
* [Video rendering options](https://github.com/kasmtech/KasmVNC/wiki/Video-Rendering-Options)
* [Stats / control API](https://github.com/kasmtech/KasmVNC/wiki/API)
* [Browser-side tuning](https://github.com/kasmtech/KasmVNC/wiki/Browser-Support)
## Agent helpers
[Section titled “Agent helpers”](#agent-helpers)
Run any of these via `bhatti exec`. `DISPLAY=:99` is pre-set (lohar writes it to `/run/bhatti/env` so exec inherits it), so X-talking helpers just work.
| Helper | Purpose |
| -------------------------------- | ------------------------------------------------ |
| `vnc-creds [--json]` | Print the username and password for this sandbox |
| `screenshot [--base64]` | Capture the current display as PNG |
| `screen-size` | Print the current X resolution |
| `active-window` / `list-windows` | Inspect window state |
| `xdotool …` | Drive mouse/keyboard input |
| `chromium-browser <url>` | Launch Chromium with sane flags |
The typical agent loop:
```bash
bhatti exec desk -- chromium-browser https://example.com &
sleep 2
bhatti exec desk -- screenshot --base64 > frame.b64
bhatti exec desk -- xdotool key Return
bhatti exec desk -- xdotool type "hello world"
```
## How the desktop is started
[Section titled “How the desktop is started”](#how-the-desktop-is-started)
Managed by lohar’s `systemctl` shim as four units (since v1.11.9), replacing the monolithic `init.sh`:
| Unit | Type | Purpose |
| ---------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kasmvnc-firstboot.service` | `oneshot` | Per-sandbox password generation + dynamic `KASM_THREADS` default. `ConditionPathExists=!/root/.kasmpasswd` makes it idempotent on snapshot/resume. |
| `kasmvnc.service` | `simple` | `Xkasmvnc :99` — combined X server + RFB-over-WebSocket gateway. Listens on port 6080. `After=kasmvnc-firstboot.service`. `Restart=on-failure`. |
| `xfce-session.service` | `simple` | `startxfce4` desktop session. `After=` + `Requires=kasmvnc.service` so killing the X server cascades. `Restart=on-failure`. |
| `bhatti-display-env.service` | `oneshot` | Writes `DISPLAY=:99` to `/run/bhatti/env` so `bhatti exec` inherits it. |
Activation order at boot: `kasmvnc-firstboot` and `bhatti-display-env` start in the first wave (no `After=`), then `kasmvnc.service`, then `xfce-session.service`.
Operator UX:
```bash
bhatti exec desk -- systemctl status kasmvnc xfce-session
bhatti exec desk -- journalctl -u kasmvnc -n 50
bhatti exec desk -- systemctl restart kasmvnc # cascades: xfce restarts too
bhatti exec desk -- systemctl restart xfce-session # just the desktop, X server stays
```
### What’s deliberately not running
[Section titled “What’s deliberately not running”](#whats-deliberately-not-running)
The pre-v1.11.9 `init.sh` started a system `dbus-daemon`, an orphan session bus via `dbus-launch`, and `pulseaudio`. None of those are started in the systemd-unit model. The reasoning:
* **`dbus-daemon --system`** keeps long-lived inotify watches on `/etc/dbus-1/`, an epoll on its listening socket, and per-connection timers. Firecracker snapshot/restore doesn’t preserve all kernel-side poller state cleanly on ARM64 — the same reason lohar runs as PID 1 instead of real systemd ([Decisions & learnings](/v1/docs/under-the-hood/decisions/)).
* **`dbus-launch`** in the original init.sh ran a session bus tied to the boot shell’s scope, leaked its address into the `startxfce4` env, and orphaned when init.sh exited. A bug, not a feature. Modern XFCE (4.16+) launches its own per-session bus on demand when it actually needs one.
* **`pulseaudio`** wasn’t connected to any sink and no documented agent flow uses audio.
The `dbus` and `dbus-x11` packages stay installed (libdbus links from XFCE / Chromium), they’re just not auto-started as a system bus. Practical fallout: a few `xfsettingsd` warnings on first XFCE start, Thunar’s D-Bus activation paths are skipped, Chromium’s notifications and password-manager features are unavailable. The desktop still loads.
If you need a system dbus for an app that depends on it, the narrowest fix is to scope it to xfce-session’s lifecycle rather than running at multi-user.target — so the snapshot-risk window is only “while xfce is up”:
```ini
# /etc/systemd/system/dbus-system.service (drop in via the sandbox)
[Unit]
Description=System bus for xfce
Before=xfce-session.service
[Service]
Type=forking
PIDFile=/run/dbus/pid
ExecStartPre=/bin/mkdir -p /run/dbus
ExecStart=/usr/bin/dbus-daemon --system --fork
```
Then `systemctl enable --now dbus-system.service`. We don’t ship this by default.
## Sizing
[Section titled “Sizing”](#sizing)
| Workload | `--cpus` | `--memory` |
| --------------------------------------------------- | -------: | ---------: |
| Light interactive use, agent loops with screenshots | 2 | 2048 |
| Browser + agent + occasional video | 4 | 4096 |
| Multiple Chromium tabs + multimedia | 4 | 4096–6144 |
| Heavy multimedia / many open apps | 6+ | 8192+ |
`--disk-size 8192` is the recommended floor — Chromium’s cache and any web content you accumulate during a session can easily hit 4–5 GB.
## Troubleshooting
[Section titled “Troubleshooting”](#troubleshooting)
| Symptom | Cause | Fix |
| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| KasmVNC web client loads, then shows a red `No password configured for VNC Auth` banner | `-SecurityTypes=None` flag missing from `Xkasmvnc` invocation; image is pre-fix | `sudo bhatti update --tiers computer` on the server |
| HTTP Basic auth challenge appears but the password is wrong | Sandbox was saved-from-image with stale creds | `bhatti exec desk -- sudo rm /root/.kasmpasswd && sudo /etc/bhatti/init.sh` to regen |
| Black screen, no XFCE panel | `startxfce4` failed (usually D-Bus). Check `journalctl -u bhatti` (host side) or `~/.xsession-errors` (guest) | Restart desktop: `bhatti exec desk -- /etc/bhatti/init.sh` |
| Mouse input lags by seconds | Encoder is CPU-starved | Bump `--cpus`; lower `KASM_FRAMERATE`; check `top` inside the sandbox |
| `xdotool` does nothing | `DISPLAY` not set in the exec environment | \`bhatti exec desk — env |
## See also
[Section titled “See also”](#see-also)
* [Tiers overview](/v1/docs/managing/tiers/)
* [`bhatti publish`](/v1/docs/reference/cli/networking/publish/) — exposing port 6080 with a public URL
* [`bhatti share`](/v1/docs/reference/cli/networking/share/) — single-use authenticated session URLs (useful for VNC handoff)
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — `--keep-hot` for active sessions, to prevent the VM from pausing under your feet
# Docker tier
> Run Docker Engine inside a microVM. Multi-arch buildx, compose, and the operator UX.
The `docker` tier ships Docker Engine, the buildx and compose plugins, containerd, and the kernel-side bits for cross-arch builds (`binfmt_misc` mounted at PID 1, qemu-user emulation available on demand). It’s the right starting point for anything that wants Docker-in-VM — CI runners, image builds, compose stacks, sandboxed `docker run` experiments.
```bash
bhatti create --name ci --image docker --cpus 4 --memory 4096
bhatti exec ci -- docker run hello-world
```
That’s the entire setup. There is no `docker setup`, no daemon to start by hand, no socket permission dance — the tier wires all of it at build time.
## What’s in it
[Section titled “What’s in it”](#whats-in-it)
| Component | Version | Source |
| --------------------------------------- | -------------- | -------------------------------------- |
| Docker Engine (`dockerd`, `docker` CLI) | 29.x | `docker-ce` from Docker’s apt repo |
| `containerd` | latest stable | `containerd.io` from Docker’s apt repo |
| `docker buildx` plugin | 0.34.x | `docker-buildx-plugin` |
| `docker compose` plugin | latest | `docker-compose-plugin` |
| `iptables-legacy` | from Ubuntu | configured as the default |
| `binfmt_misc` filesystem | kernel feature | mounted by lohar at PID 1 |
The custom bhatti kernel enables every flag dockerd needs at runtime — bridge networking, veth, overlayfs, netfilter conntrack and security tables, and `BINFMT_MISC` for cross-arch emulation. See [Kernel](/v1/docs/contributing/kernel/) for the full list and rationale.
## How dockerd is managed
[Section titled “How dockerd is managed”](#how-dockerd-is-managed)
The tier doesn’t ship its own `docker.service`. It reuses the one that comes with the `docker-ce` package and customises two things via a drop-in at `/etc/systemd/system/docker.service.d/bhatti.conf`:
```ini
[Service]
# Replace upstream `-H fd://` (which requires systemd socket activation
# lohar's shim deliberately doesn't implement) with a direct socket bind.
ExecStart=
ExecStart=/usr/bin/dockerd -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock
# Pick up `bhatti create --env DOCKER_*=...` overrides.
EnvironmentFile=-/run/bhatti/config-env
# Widen the socket so `bhatti exec` (uid 1000, no supplementary groups)
# can reach it. Safe inside an isolated single-user microVM.
ExecStartPost=/bin/chmod 666 /var/run/docker.sock
Restart=on-failure
RestartSec=2s
```
And `/etc/docker/daemon.json`:
```json
{ "exec-opts": ["native.cgroupdriver=cgroupfs"] }
```
Both overrides exist because the shim is **not** real systemd. Two specific gaps, both deliberate per [Decisions & learnings](/v1/docs/under-the-hood/decisions/):
> **Why `-H fd://` is replaced.** Real `docker.service` listens via a paired `docker.socket` unit; systemd binds the socket and hands the fd to dockerd. The shim doesn’t implement socket activation (“a lot of code for marginal benefit — services start fast in our VMs anyway”). The drop-in tells dockerd to bind the socket itself.
> **Why `cgroupfs` instead of `systemd`.** Docker 29 defaults to `native.cgroupdriver=systemd`, which talks to systemd over D-Bus during every container create. The microVM has no D-Bus (libpam-systemd is pinned out of the rootfs to keep snapshot/restore safe). Without the override, every `docker run` dies in `runc` with `dial unix /run/systemd/private: connect: no such file`.
You don’t need to think about any of this in normal use. The drop-in is in place from boot one.
## The operator UX
[Section titled “The operator UX”](#the-operator-ux)
```bash
bhatti exec dev -- systemctl status docker
# ● docker.service - Docker Application Container Engine
# Active: active (running, PID 310)
# …
bhatti exec dev -- journalctl -u docker -n 50
bhatti exec dev -- journalctl -u docker -f # follow
bhatti exec dev -- systemctl restart docker
bhatti exec dev -- systemctl is-active docker # exits 0 if active
bhatti exec dev -- systemctl is-failed docker # exits 0 if failed
```
If dockerd crashes, the shim restarts it (`Restart=on-failure`, 2 s back-off). The failure shows up in `journalctl -u docker`. There is no separate logs file you have to remember; everything daemon-level is in the journal.
`docker logs <container>` (per-container application logs) is unchanged from upstream Docker — those flow through containerd, not the shim.
## Environment knobs
[Section titled “Environment knobs”](#environment-knobs)
dockerd inherits anything you pass to `bhatti create --env`. The bridge happens through `/run/bhatti/config-env`, which lohar writes from the config drive at boot and the unit reads via `EnvironmentFile=`.
```bash
bhatti create --name reg --image docker \
--env "DOCKER_REGISTRY_MIRROR=https://mirror.example.com,DOCKER_OPTS=--debug"
```
`--env` takes a **single comma-separated list** (`K=V,K=V`), not a repeated flag.
Recognised env vars today:
| Variable | Effect |
| -------------------------------- | ---------------------------------------------------------------------- |
| Anything starting with `DOCKER_` | Visible in `dockerd`’s environment; useful for env-driven docker hooks |
The drop-in deliberately doesn’t translate env vars into dockerd flags. If you need to change daemon flags themselves, write a second drop-in or edit `/etc/docker/daemon.json` inside the sandbox and `systemctl restart docker`.
## Socket permissions
[Section titled “Socket permissions”](#socket-permissions)
`/var/run/docker.sock` is `0666` after boot — anyone in the sandbox can talk to dockerd. This is the world-writable workaround mentioned in the drop-in. The reasoning:
* `bhatti exec` runs as uid 1000 (the `lohar` user) without supplementary group membership, so the standard `docker` group ACL doesn’t apply.
* The VM is the security boundary, not the docker group. Inside, it’s a single-user sandbox.
* Hardening this further would require lohar to preserve supplementary groups across exec, which is a separate change that hasn’t shipped yet.
If you re-mount the rootfs read-write and tighten the socket yourself, expect to also write a unit that wraps `bhatti exec` with `sg docker -c '…'`.
## Multi-arch builds with buildx
[Section titled “Multi-arch builds with buildx”](#multi-arch-builds-with-buildx)
Same-arch builds are native and need no setup:
```bash
bhatti exec dev -- docker buildx build -t me/app:latest .
```
Cross-arch builds need one extra command — `tonistiigi/binfmt --install all` — to populate the kernel’s `binfmt_misc` handler table with qemu-user interpreters:
```bash
bhatti exec dev -- docker run --privileged --rm tonistiigi/binfmt --install all
bhatti exec dev -- docker buildx create --use --name xb --driver docker-container
bhatti exec dev -- docker buildx build --platform linux/amd64,linux/arm64 \
-t me/app:multi --push .
```
### What that one extra command actually does
[Section titled “What that one extra command actually does”](#what-that-one-extra-command-actually-does)
Three layers have to line up for cross-arch builds to work:
| Layer | Provided by | State on a fresh docker sandbox |
| -------------------------------------------------- | ----------------------------- | ------------------------------- |
| `CONFIG_BINFMT_MISC=y` in the kernel | bhatti kernel | ✅ on |
| `/proc/sys/fs/binfmt_misc` filesystem mounted | lohar at PID 1 | ✅ mounted |
| Per-arch handler registrations + qemu interpreters | `tonistiigi/binfmt` container | ❌ empty until you run it |
The kernel knows it could dispatch foreign-arch ELFs to a userspace interpreter, but it has no interpreters and no mapping yet. The `tonistiigi/binfmt` image carries pre-built static qemu binaries (`qemu-aarch64`, `qemu-arm`, `qemu-riscv64`, …) and uses the kernel’s **fix-binary (`F`) flag** when registering, which makes the kernel open the interpreter and keep an fd to it. After registration the container exits, the qemu binaries are gone, the kernel’s open fds keep working.
You run the command once per sandbox. Registrations survive snapshot/restore (they live in kernel state, not on disk).
### Performance: when emulation is fine, when it isn’t
[Section titled “Performance: when emulation is fine, when it isn’t”](#performance-when-emulation-is-fine-when-it-isnt)
qemu-user emulation is correct but slow — typically 5–20× slower than native for CPU-bound work. Reasonable rules of thumb:
| Workload | Cross-arch emulation | Native-per-arch |
| --------------------------------------------------------- | ------------------------- | --------------- |
| Single-image dev builds, occasional cross-arch validation | 👍 fine | overkill |
| CI pushing release images on every merge | 😬 painful (multi-minute) | 👍 right answer |
| Heavily CPU-bound layers (compilers, native deps, ML) | ❌ avoid | 👍 right answer |
| `apt-get install`-heavy layers (mostly disk + network) | 👌 acceptable | 👍 faster |
For real CI, run one bhatti sandbox per native arch and stitch them with buildx:
```bash
# Driver machine — orchestrates two remote builders.
docker buildx create --name xb --platform linux/amd64 ssh://user@amd64-sandbox-host
docker buildx create --append --name xb --platform linux/arm64 ssh://user@arm64-sandbox-host
docker buildx use xb
docker buildx build --platform linux/amd64,linux/arm64 --push -t me/app:x .
```
(Server-side requires either an SSH driver or a TCP-listening dockerd that you publish via `bhatti publish`.) Each platform builds natively on a matching host; buildx writes the multi-arch manifest. bhatti’s [warm-wake](/v1/docs/under-the-hood/thermal-states/) (\~4 ms) makes this cheap to leave idle between runs.
## Compose
[Section titled “Compose”](#compose)
`docker-compose-plugin` is installed, so `docker compose up` works as you’d expect. There’s no extra wiring.
```bash
bhatti create --name stack --image docker --cpus 4 --memory 4096
bhatti file write stack /workspace/compose.yaml < ./compose.yaml
bhatti exec stack -- docker compose -f /workspace/compose.yaml up -d
bhatti publish stack -p 8080 # if compose exposes :8080 on the host
```
For long-running compose stacks, consider `--keep-hot` on `bhatti create` so the [thermal manager](/v1/docs/under-the-hood/thermal-states/) doesn’t pause the VM during idle periods (TCP connections from the public preview URL survive pause, but new connections during pause cost a wake).
## Networking
[Section titled “Networking”](#networking)
The bhatti kernel ships `iptables-legacy` (not nft). Docker’s bridge driver expects either, and the tier sets the legacy alternatives at build time. If you install a package that switches the alternatives back to nft, dockerd will fail to set up bridge networking. To recover:
```bash
bhatti exec dev -- sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
bhatti exec dev -- sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
bhatti exec dev -- sudo systemctl restart docker
```
Inside the sandbox, dockerd creates its own `docker0` bridge. Container-to-container networking, port publishing inside the VM (`docker run -p 8080:80 ...`), and DNS for container names all work normally. The bhatti-level publish (`bhatti publish dev -p 8080`) targets the sandbox’s host-side port; if a container binds to `0.0.0.0:8080` inside, it’s reachable.
## Troubleshooting
[Section titled “Troubleshooting”](#troubleshooting)
| Symptom | Cause | Fix |
| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `docker run` hangs / `dial unix /run/systemd/private` in logs | daemon.json missing or wrong cgroup driver | `cat /etc/docker/daemon.json` should say `cgroupfs`; if not, write it and `systemctl restart docker` |
| `docker buildx: unknown command` | `docker-buildx-plugin` not installed (very old image) | `sudo apt-get install -y docker-buildx-plugin` or `sudo bhatti update --tiers docker` on the server |
| `tonistiigi/binfmt --install all` reports success but cross-arch builds fail | `/proc/sys/fs/binfmt_misc` not mounted (very old lohar) | Verify with \`mount |
| `permission denied` on `/var/run/docker.sock` from `bhatti exec` | socket mode regressed after `systemctl restart docker` and ExecStartPost didn’t fire | `systemctl status docker` to confirm active; if active but socket is `0660`, the drop-in is missing — check `/etc/systemd/system/docker.service.d/bhatti.conf` |
| `docker run …` fails with iptables errors | alternatives switched to nft | re-set to legacy (see [Networking](#networking)) |
| Cross-arch container runs but exits with `exec format error` | qemu handler not registered for that arch | re-run `tonistiigi/binfmt --install all`; check `ls /proc/sys/fs/binfmt_misc/` for `qemu-<arch>` |
| `docker.service` repeatedly entering “activating” | `dockerd` not sending `READY=1` (Type=notify) — usually a containerd issue | `systemctl status containerd`; check `journalctl -u containerd` |
## Resource sizing
[Section titled “Resource sizing”](#resource-sizing)
| Workload | `--cpus` | `--memory` |
| ------------------------------------------------------------- | -------: | ---------: |
| `docker run` of small images (`hello-world`, alpine tools) | 2 | 2048 |
| Light compose stacks (3–5 small services) | 2–4 | 2048–4096 |
| `buildx build` of a typical app image | 4 | 4096 |
| Cross-arch `buildx build` with qemu emulation | 4 | 4096–8192 |
| Heavy CI builds (large dependency graphs, native compilation) | 8 | 8192+ |
Memory matters more than CPU for most builds — overlayfs page cache, build context, and intermediate layers all live in the guest. dockerd itself runs comfortably in 256 MB; anything above that is your workload.
## See also
[Section titled “See also”](#see-also)
* [Tiers overview](/v1/docs/managing/tiers/) — what each tier is for
* [Adding a tier](/v1/docs/contributing/adding-a-tier/) — build your own
* [Kernel](/v1/docs/contributing/kernel/) — flags enabled for Docker, why
* [Lohar: systemctl shim](/v1/docs/under-the-hood/lohar-the-blacksmith/) — how unit files are managed without real systemd
* [Decisions & learnings](/v1/docs/under-the-hood/decisions/) — why no socket activation, why no D-Bus
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — `--keep-hot` and what idle behaviour to expect for long-running stacks
# Users & Auth
> Multi-tenant model — per-user API keys, resource caps, network isolation, sandbox scoping.
bhatti is multi-tenant by default. Every user gets:
* An API key (`bht_<64-hex>`).
* A `/24` bridge network of their own; sandboxes from different users cannot reach each other on layer 2.
* Per-user caps on sandbox count, per-sandbox CPU, and per-sandbox memory.
* Their own secrets, volumes, snapshots, and images.
Users are created server-side, with `sudo bhatti user create`. The CLI commands all live in [Server & admin](/v1/docs/reference/cli/admin/) — this page covers the model, not the surface.
You won’t usually run `user create` for yourself. The install script (`curl -fsSL bhatti.sh/install | sudo bash`) creates an `admin` user during setup and writes its API key to your `~/.bhatti/config.yaml`. The first manual `user create` you run is for adding a teammate — see [Self-hosting → adding teammates](/v1/docs/self-hosting/#adding-teammates).
## How keys work
[Section titled “How keys work”](#how-keys-work)
The server stores only the SHA-256 hash of each API key. The plaintext is shown once at creation and once on rotation; if the user loses it, [`rotate-key`](/v1/docs/reference/cli/admin/user-rotate-key/) is the only path back. There’s no recovery.
Auth is `Authorization: Bearer <token>` on every API request except `/health` and `/_shell/<id>`. WebSocket connections use the same header. There’s no `?token=…` query-param auth — keys never end up in URLs (and therefore not in proxy logs).
The web shell is the one place a token appears in a URL — embedded in the **fragment** (`#token=…`), which never leaves the browser. The page extracts it client-side and uses it as a Bearer header on the WebSocket.
## Resource caps
[Section titled “Resource caps”](#resource-caps)
Every user has three per-user limits:
| Limit | Meaning |
| --------------------------- | ------------------------------------------------------ |
| `max_sandboxes` | Concurrent (non-destroyed) sandboxes the user can own. |
| `max_cpus_per_sandbox` | Cap on `--cpus` at create time. |
| `max_memory_mb_per_sandbox` | Cap on `--memory` at create time. |
Lowering a cap doesn’t affect existing sandboxes. New ones are rejected with `403` if they exceed the cap, or `429` if the user is at their `max_sandboxes`.
Storage is not currently capped per-user — disk usage grows freely until the host fills up. Track per-user volume size with `bhatti volume list` and the relevant `--data-dir`.
## Sandbox scoping
[Section titled “Sandbox scoping”](#sandbox-scoping)
The API enforces strict per-user isolation:
* `GET /sandboxes` returns only sandboxes the authenticated user created.
* Looking up someone else’s sandbox by ID returns `404` (intentional — the existence isn’t leaked).
* The user can’t exec, file-read, or destroy a sandbox they don’t own.
This is consistent across every endpoint that takes a sandbox ID.
## Network isolation
[Section titled “Network isolation”](#network-isolation)
Each user’s sandboxes share a private bridge (`bhatti-<subnet-index>`) on a `/24` subnet. Cross-user traffic is blocked at the iptables layer. A user’s own sandboxes can talk to each other (e.g. an agent sandbox can curl a worker sandbox at its private IP).
The subnet index is assigned on `user create` and shown in `bhatti user list`. You don’t need to think about it; it just shows up in `bhatti inspect` output.
## See also
[Section titled “See also”](#see-also)
* [`bhatti user create`](/v1/docs/reference/cli/admin/user-create/), [`list`](/v1/docs/reference/cli/admin/user-list/), [`rotate-key`](/v1/docs/reference/cli/admin/user-rotate-key/), [`delete`](/v1/docs/reference/cli/admin/user-delete/)
* [Networking](/v1/docs/under-the-hood/networking/) — bridge layout, iptables rules, IP allocation
# Volumes
> Persistent ext4 volumes — attach semantics, mount restrictions, and the detached-state requirement.
Persistent volumes are ext4 filesystems that survive sandbox destruction. Create one, attach it to a sandbox, destroy the sandbox — the data is still there.
The CLI surface is in [Volumes reference](/v1/docs/reference/cli/volumes/). This page covers the rules — when you can attach, what mount paths are allowed, and which operations require the volume to be unattached.
## Attaching at create time
[Section titled “Attaching at create time”](#attaching-at-create-time)
Volumes are mounted by lohar during sandbox boot. Specify them with `--volume` (repeatable):
```bash
bhatti volume create --name workspace --size 5120
bhatti create --name dev --volume workspace:/workspace
```
The format is `<name>:<mount>[:ro]`:
```bash
bhatti create --name reader --volume workspace:/data:ro
```
`:ro` mounts read-only. Multiple sandboxes can mount the same volume read-only at the same time — useful for shared base data. Read-write attachment is exclusive: only one sandbox can have it as read-write at any moment.
You can attach more than one volume per sandbox by repeating `--volume`.
## Mount path restrictions
[Section titled “Mount path restrictions”](#mount-path-restrictions)
Mount paths must be **absolute**. The guest agent rejects anything else.
You also can’t shadow system paths. Specifically, the following are blocked:
```plaintext
/proc /sys /dev /etc /usr /bin /sbin /lib /lib64 /run /var/run
```
Use a fresh subtree like `/workspace`, `/data`, `/srv/<app>` — anywhere outside the system hierarchy.
## Detached state
[Section titled “Detached state”](#detached-state)
Four operations require the volume to be **detached** (no active sandbox attachments):
* [`bhatti volume delete`](/v1/docs/reference/cli/volumes/delete/)
* [`bhatti volume resize`](/v1/docs/reference/cli/volumes/resize/)
* [`bhatti volume clone`](/v1/docs/reference/cli/volumes/clone/)
* [`bhatti volume restore`](/v1/docs/reference/cli/volumes/restore/)
The server returns `409 Conflict` if any sandbox has the volume mounted. To detach, either destroy the sandbox or [`bhatti stop`](/v1/docs/reference/cli/sandbox/stop/) it (stopping detaches volumes; starting reattaches).
[`bhatti volume backup`](/v1/docs/reference/cli/volumes/backup/) is the exception — it works while the volume is attached. The trade-off is consistency: writes in flight at backup time may be captured mid-update. For a guaranteed-clean backup, stop the sandbox first.
## Backups
[Section titled “Backups”](#backups)
Volume backups go to S3-compatible storage (AWS, Backblaze B2, R2, MinIO, etc.). They require a `backup` block in the server config; without it, backup-related commands return `501`.
```bash
bhatti volume backup workspace
bhatti volume backup-list workspace
bhatti volume restore workspace --backup-id bk_a1b2c3d4
```
Schedule regular backups with retention via the `backup.schedule` config — see [Configuration](/v1/docs/reference/config/#backup).
## See also
[Section titled “See also”](#see-also)
* [Volumes reference](/v1/docs/reference/cli/volumes/) — every command
* [`bhatti create --volume`](/v1/docs/reference/cli/sandbox/create/) — attach syntax
* [Configuration: backup](/v1/docs/reference/config/#backup) — S3 setup
# Quickstart
> Install bhatti on your own Linux box, create a sandbox, and run something inside it. About five minutes including the install download.
bhatti v1 (Firecracker) is frozen
Active development moved to **v2 (krucible)**, a self-owned VMM that also runs on macOS — see the [current docs](/docs/quickstart/). These v1 pages are preserved for existing Firecracker users. **`bhatti.sh/install` now installs v2**, so the commands below are pinned to the frozen v1 release.
This is the path I recommend for everyone except remote-CLI users sharing someone else’s bhatti server. You’ll install the daemon on a Linux box you own, and from that same box you’ll create your first sandbox and run something inside it.
If you don’t have a Linux box with KVM in front of you, the [Self-hosting](/v1/docs/self-hosting/) page covers what you need (a Raspberry Pi 5, a Hetzner AX, or any cloud VM with nested virtualization is enough).
## Install
[Section titled “Install”](#install)
```bash
curl -fsSL https://raw.githubusercontent.com/sahil-shubham/bhatti/firecracker/scripts/install.sh | sudo BHATTI_VERSION=v1.11.12 bash
```
That’s the whole install (pinned to the frozen v1.11.12 release). The script:
1. Downloads `bhatti`, `lohar`, Firecracker, the kernel, and the minimal Ubuntu 24.04 rootfs (\~200 MB total).
2. Sets up the systemd service and starts the daemon.
3. Creates an `admin` user and writes its API key + endpoint to your user’s `~/.bhatti/config.yaml`. **You don’t need to run `bhatti setup` after.** The CLI on this box is already wired up.
The transcript ends with the admin API key printed once. Save it somewhere — you’ll want it later for adding teammates or driving the server from a different machine.
## Create a sandbox
[Section titled “Create a sandbox”](#create-a-sandbox)
```bash
bhatti create --name dev
```
```text
sandbox/dev created (1 vCPU, 1024 MB)
IP: 10.0.1.42
Shell: bhatti shell dev
```
Defaults are 1 vCPU and 1024 MB. Override with `--cpus` and `--memory`. See [`bhatti create`](/v1/docs/reference/cli/sandbox/create/) for every flag.
## Run a command
[Section titled “Run a command”](#run-a-command)
```bash
bhatti exec dev -- echo hello
# → hello
```
Anything after `--` runs verbatim inside the sandbox. The `--` is optional when there’s no ambiguity:
```bash
bhatti exec dev uname -a
```
## Open a shell
[Section titled “Open a shell”](#open-a-shell)
```bash
bhatti shell dev
```
Interactive PTY, full keyboard. Press `Ctrl+\` to detach — the shell keeps running. Reconnect with `bhatti shell dev` and the scrollback is replayed.
## Publish a port
[Section titled “Publish a port”](#publish-a-port)
If you want to reach a service running inside the sandbox from the internet:
```bash
bhatti exec dev -- bash -c 'python3 -m http.server 3000 &'
bhatti publish dev -p 3000 -a my-app
# → https://my-app.bhatti.sh
```
That URL is public. The sandbox can be cold and the URL still works — the first request wakes it (\~42 ms p50 on Hetzner). See [Preview URLs](/v1/docs/sandboxes/preview-urls/) for aliases, custom domains, and auth.
## Clean up
[Section titled “Clean up”](#clean-up)
```bash
bhatti destroy dev
```
Or `bhatti destroy dev -y` if you don’t want the confirmation prompt.
## What just happened
[Section titled “What just happened”](#what-just-happened)
1. **`bhatti create`** asked the daemon to boot a Firecracker microVM — a Linux VM with its own kernel, filesystem, and network interface.
2. **`bhatti exec`** sent a command over the wire protocol to lohar, the guest agent running as PID 1 inside the VM.
3. **`bhatti shell`** opened a WebSocket and attached to a PTY session.
4. **`bhatti destroy`** stopped the VM and cleaned up the rootfs, TAP device, and IP.
The sandbox was a full Linux environment, not a container. When idle, it would have paused itself automatically and resumed on the next request in milliseconds.
## Next steps
[Section titled “Next steps”](#next-steps)
* [Concepts](/v1/docs/concepts/) — sandboxes, thermal states, the two binaries
* [Self-hosting](/v1/docs/self-hosting/) — adding teammates, custom domains, backups, the rest of the operator path
* [`bhatti exec`](/v1/docs/reference/cli/exec/exec/) — streaming, timeouts, detach
* [`bhatti create`](/v1/docs/reference/cli/sandbox/create/) — every flag
***
### Driving a remote bhatti from your laptop
[Section titled “Driving a remote bhatti from your laptop”](#driving-a-remote-bhatti-from-your-laptop)
If someone else runs the bhatti daemon and you only need the CLI on your local machine — or you want to install bhatti on a server and drive it from your laptop — install the CLI without sudo and point it at the server:
```bash
# Install the v1 CLI binary
curl -fsSL https://raw.githubusercontent.com/sahil-shubham/bhatti/firecracker/scripts/install.sh | BHATTI_VERSION=v1.11.12 bash
# Configure the endpoint and key
bhatti setup --url https://your-server:8080 --token bht_abc...
# or interactively:
bhatti setup
```
`bhatti setup` accepts `--url` and `--token` for non-interactive configuration (agents, CI, provisioning scripts). Without flags, it prompts. See [`bhatti setup`](/v1/docs/reference/cli/admin/setup/) for the full reference.
The server operator gets your API key by running `bhatti user create --name <you>` once, on the server, and sending you the key. See [Self-hosting → adding teammates](/v1/docs/self-hosting/#adding-teammates).
# API Reference
> Every HTTP endpoint with a runnable curl example, the response shape, and notable status codes.
The bhatti server exposes a REST API over HTTP. Every authenticated endpoint requires a Bearer token; unauthenticated endpoints are limited to `/health` and `/_shell/<id>` (the embedded web-shell page).
This page is the canonical hand-curated reference. Each endpoint shows a runnable `curl` and the response shape. Anything not in the curl example uses the documented default.
## Base URL
[Section titled “Base URL”](#base-url)
| Mode | API base | Public proxy |
| ------------------------------------------------------ | --------------------------- | --------------------------------------------- |
| Default (`listen: :8080`) | `http://localhost:8080` | path-based at `/sandboxes/<id>/proxy/<port>/` |
| Path-based public proxy (`public_proxy_listen: :8443`) | `http://localhost:8080` | `http://<host>:8443/<alias>/` |
| Domain mode (`domain.api_host`, `domain.proxy_zone`) | `https://api.<your-domain>` | `https://<alias>.<your-domain>` |
## Authentication
[Section titled “Authentication”](#authentication)
All endpoints except `/health` and `/_shell/...` require a Bearer token in the `Authorization` header:
```http
Authorization: Bearer bht_abc123def456...
```
Tokens come from [`bhatti user create`](/v1/docs/reference/cli/admin/user-create/). The CLI loads them from `~/.bhatti/config.yaml`.
**WebSocket endpoints use the same Bearer header.** There is no `?token=…` query-param auth — leaving the token out of URLs eliminates accidental logging in proxy access logs.
The web shell is the exception: the URL contains `#token=…` in the **fragment** (never sent to the server), and the embedded page extracts the token client-side and uses it as a Bearer header on the WebSocket connection.
## Response shape
[Section titled “Response shape”](#response-shape)
JSON for every endpoint. Errors use `{"error": "...", "request_id": "..."}`:
```json
{"error": "sandbox not found", "request_id": "req_a1b2c3d4"}
```
`request_id` is included on every error response; quote it when reporting bugs.
## Status codes
[Section titled “Status codes”](#status-codes)
| Code | Meaning |
| ----- | ----------------------------------------------------------------------------------------------- |
| `200` | OK. |
| `201` | Created. Used for resource-creation endpoints. |
| `202` | Accepted. Used for async tasks (image pull) — body contains a `task_id`. |
| `204` | No content. Used by `unpublish`. |
| `400` | Validation error. Body has `{"error": "..."}`. |
| `401` | Missing or invalid Bearer token. |
| `403` | Forbidden — typically a per-sandbox cap (CPU, memory) being exceeded. |
| `404` | Resource not found. |
| `409` | Conflict — name already exists, or volume is attached when an op needs it detached. |
| `413` | Request too large (file write > 100 MB). |
| `429` | Rate-limited or quota exceeded (`max-sandboxes`, `max-cpus`, etc). |
| `500` | Internal server error. |
| `501` | Endpoint exists but the server isn’t configured for it (e.g. backups without an `s3_*` config). |
| `502` | Bad gateway from the reverse proxy — the sandbox process isn’t listening on the requested port. |
## Response headers
[Section titled “Response headers”](#response-headers)
The server stamps every response with:
* `X-Bhatti-Version` — server version. CLI clients use this to detect server upgrades.
* `X-Bhatti-Min-CLI` — the minimum CLI version the server requires. Clients older than this should upgrade.
* `X-Bhatti-Existing: true` — set on `POST /sandboxes` when the request was idempotent (a sandbox with that name already existed).
## Async tasks
[Section titled “Async tasks”](#async-tasks)
Long-running operations like `POST /images/pull` return immediately with `202 Accepted` and a body like:
```json
{"task_id": "tsk_abc123", "status": "running"}
```
Poll `GET /tasks/:id` until `status` is `completed` or `failed`. The `progress` field, when present, contains a human-readable progress string the CLI can display.
***
# System
[Section titled “System”](#system)
## Health check
[Section titled “Health check”](#health-check)
```http
GET /health
```
No auth required. Lightweight check.
```bash
curl http://localhost:8080/health
```
```json
{"status": "ok", "uptime": "2h15m30s"}
```
***
# Sandboxes
[Section titled “Sandboxes”](#sandboxes)
## List sandboxes
[Section titled “List sandboxes”](#list-sandboxes)
```http
GET /sandboxes
```
Returns sandboxes owned by the authenticated user, enriched with thermal state and any published URLs. Listing does **not** wake cold sandboxes.
```bash
curl http://localhost:8080/sandboxes \
-H "Authorization: Bearer $TOKEN"
```
```json
[
{
"id": "a1b2c3d4",
"name": "dev",
"status": "running",
"thermal": "hot",
"ip": "192.168.137.2",
"cpus": 2,
"memory_mb": 1024,
"image": "minimal",
"created_at": "2026-04-29T15:21:48Z",
"urls": ["https://dev-k3m9x2.bhatti.sh"]
}
]
```
## Create a sandbox
[Section titled “Create a sandbox”](#create-a-sandbox)
```http
POST /sandboxes
```
```bash
curl -X POST http://localhost:8080/sandboxes \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "dev",
"cpus": 2,
"memory_mb": 1024,
"env": {"NODE_ENV": "production"},
"init": "cd /workspace && npm install",
"image": "minimal",
"persistent_volumes": [
{"name": "workspace", "mount": "/workspace", "auto_create": false}
],
"secrets": ["OPENAI_KEY"],
"files": [
{"guest_path": "/etc/app/config.json", "content": "<base64>", "mode": "0644"}
]
}'
```
**Request fields:**
| Field | Type | Default | Notes |
| -------------------- | ------ | ---------- | ------------------------------------------------------------------------------------- |
| `name` | string | auto | Pattern: `[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}`. |
| `cpus` | float | `1` | Fractional values allowed. Capped at user’s `max_cpus_per_sandbox`. |
| `memory_mb` | int | `1024` | Capped at user’s `max_memory_mb_per_sandbox`. |
| `disk_size_mb` | int | image size | Resize the rootfs at create time. |
| `env` | object | — | Env vars baked into the config drive. |
| `init` | string | — | Boot-time script. Runs as a session named `init`. |
| `keep_hot` | bool | `false` | Disables the thermal manager for this sandbox. |
| `hugepages` | bool | `false` | 2 MB hugepages. Faster boot, no diff snapshots. |
| `image` | string | `minimal` | Image name from `GET /images`. |
| `template_id` | string | — | Create from a template. |
| `persistent_volumes` | array | — | `[{name, mount, auto_create, read_only, size_mb}]`. |
| `secrets` | array | — | Names of stored secrets to inject as env vars. **Ignored when `template_id` is set.** |
| `files` | array | — | `[{guest_path, content (base64), mode}]`. **Ignored when `template_id` is set.** |
| `new_volumes` | array | — | **Legacy** equivalent of `persistent_volumes`. The CLI no longer sends these. |
| `volumes` | array | — | **Legacy** equivalent. |
**Responses:**
* `201 Created` with the sandbox object.
* `200 OK` + `X-Bhatti-Existing: true` if a non-destroyed sandbox with that name already exists. Idempotent — safe to retry from scripts.
* `400` for invalid name / size / etc.
* `403` if `cpus` or `memory_mb` exceed the user’s per-sandbox cap.
* `429` if the user is at their `max-sandboxes` cap.
## Get a sandbox
[Section titled “Get a sandbox”](#get-a-sandbox)
```http
GET /sandboxes/:id
```
```bash
curl http://localhost:8080/sandboxes/dev \
-H "Authorization: Bearer $TOKEN"
```
Returns the full sandbox record. Both name and ID work as `:id`.
## Update a sandbox
[Section titled “Update a sandbox”](#update-a-sandbox)
```http
PATCH /sandboxes/:id
```
Currently only `keep_hot` is mutable. All other fields (cpus, memory, image) are immutable after creation — destroy and recreate.
```bash
curl -X PATCH http://localhost:8080/sandboxes/dev \
-H "Authorization: Bearer $TOKEN" \
-d '{"keep_hot": true}'
```
Setting `keep_hot: true` on a stopped sandbox immediately wakes it.
## Destroy a sandbox
[Section titled “Destroy a sandbox”](#destroy-a-sandbox)
```http
DELETE /sandboxes/:id
```
```bash
curl -X DELETE http://localhost:8080/sandboxes/dev \
-H "Authorization: Bearer $TOKEN"
```
```json
{"status": "destroyed"}
```
Persistent volumes attached to the sandbox are detached (not deleted). Published URLs are cleaned up.
## Stop a sandbox
[Section titled “Stop a sandbox”](#stop-a-sandbox)
```http
POST /sandboxes/:id/stop
```
Snapshot to disk and free memory. Resume with `POST /sandboxes/:id/start`. The first stop creates a full snapshot; subsequent stops create diff snapshots (dirty pages only).
```bash
curl -X POST http://localhost:8080/sandboxes/dev/stop \
-H "Authorization: Bearer $TOKEN"
```
## Start a sandbox
[Section titled “Start a sandbox”](#start-a-sandbox)
```http
POST /sandboxes/:id/start
```
Restore from snapshot.
```bash
curl -X POST http://localhost:8080/sandboxes/dev/start \
-H "Authorization: Bearer $TOKEN" \
-d '{"force": true}'
```
`force: true` retries even if the sandbox is in `unknown` state from a previously failed restore.
***
# Execution & shells
[Section titled “Execution & shells”](#execution--shells)
## Run a command
[Section titled “Run a command”](#run-a-command)
```http
POST /sandboxes/:id/exec
```
```bash
curl -X POST http://localhost:8080/sandboxes/dev/exec \
-H "Authorization: Bearer $TOKEN" \
-d '{"cmd": ["echo", "hello"]}'
```
```json
{"exit_code": 0, "stdout": "hello\n", "stderr": ""}
```
**Request fields:**
| Field | Type | Default | Notes |
| ------------- | ------ | ---------- | -------------------------------------------------------------------------- |
| `cmd` | array | (required) | Command and arguments. |
| `timeout_sec` | int | `300` | Max `86400` (24h). |
| `detach` | bool | `false` | Fire-and-forget. Returns `{pid, output_file, detached: true}` immediately. |
| `output_file` | string | auto | When `detach: true`, the guest path to redirect output to. |
This endpoint does **not** accept an `env` field. Env vars are baked into the sandbox’s config drive at creation time (`--env`, `--secret`). To pass per-call env vars, use the WebSocket exec path below.
### Streaming output
[Section titled “Streaming output”](#streaming-output)
Set `Accept: application/x-ndjson`:
```bash
curl -N -X POST http://localhost:8080/sandboxes/dev/exec \
-H "Authorization: Bearer $TOKEN" \
-H "Accept: application/x-ndjson" \
-d '{"cmd":["npm","install"]}'
```
```json
{"type":"stdout","data":"Installing dependencies...\n"}
{"type":"stderr","data":"npm warn deprecated...\n"}
{"type":"exit","exit_code":0}
```
Each line is flushed immediately. `type` is `stdout`, `stderr`, `exit`, or `error`.
Cold sandboxes wake automatically before executing.
## Interactive shell (WebSocket)
[Section titled “Interactive shell (WebSocket)”](#interactive-shell-websocket)
```http
GET /sandboxes/:id/ws
```
Upgrade to WebSocket. Authentication via `Authorization: Bearer ...` header on the upgrade request.
| Query param | Description |
| -------------- | ------------------------------------------------------ |
| `session=<id>` | Reattach to an existing session by ID. |
| `new=true` | Force a new session even if one is currently attached. |
**Wire protocol:**
* **Server → client (binary):** raw terminal output.
* **Server → client (text):** initial `{"type":"session","session_id":"s1"}` message; thereafter only on session events.
* **Client → server (binary):** keystrokes (forwarded to the PTY).
* **Client → server (text JSON):** `{"type":"resize","rows":N,"cols":N}` or `{"cmd":[...], "env":{}, "max_idle_sec":N}` (first message of a non-attached session — this is where per-call env injection lives).
Detach by closing the connection. The session keeps running.
## List sessions
[Section titled “List sessions”](#list-sessions)
```http
GET /sandboxes/:id/sessions
```
```bash
curl http://localhost:8080/sandboxes/dev/sessions \
-H "Authorization: Bearer $TOKEN"
```
```json
[
{"session_id": "init", "argv": "npm install", "running": true, "attached": false, "created_at": 1761792000},
{"session_id": "s1", "argv": "/bin/zsh -li", "running": true, "attached": true, "created_at": 1761792100}
]
```
## List listening ports (one sandbox)
[Section titled “List listening ports (one sandbox)”](#list-listening-ports-one-sandbox)
```http
GET /sandboxes/:id/ports
```
```bash
curl http://localhost:8080/sandboxes/dev/ports \
-H "Authorization: Bearer $TOKEN"
```
```json
[
{"sandbox_id": "a1b2c3d4", "container_port": 3000, "proxy_url": "/sandboxes/a1b2c3d4/proxy/3000/"}
]
```
The `proxy_url` is path-relative; prepend the API base URL for a full reachable URL.
## List listening ports (all sandboxes)
[Section titled “List listening ports (all sandboxes)”](#list-listening-ports-all-sandboxes)
```http
GET /ports
```
Same row shape as above; covers every sandbox the authenticated user owns.
***
# Files
[Section titled “Files”](#files)
All file operations target the same path with query parameters.
## Read a file
[Section titled “Read a file”](#read-a-file)
```http
GET /sandboxes/:id/files?path=/path
```
Returns raw file content with `Content-Type: application/octet-stream`. The `X-File-Size` header always carries the total size (so clients can detect truncation).
```bash
curl "http://localhost:8080/sandboxes/dev/files?path=/workspace/app.js" \
-H "Authorization: Bearer $TOKEN" \
-o app.js
```
**Server-side truncation:**
```bash
curl "http://localhost:8080/sandboxes/dev/files?path=/var/log/agent.log&offset=1&limit=2000&max_bytes=51200" \
-H "Authorization: Bearer $TOKEN"
```
| Param | Type | Description |
| ----------- | ------ | ------------------------------------ |
| `path` | string | Absolute path inside the sandbox. |
| `offset` | int | 1-indexed line number to start from. |
| `limit` | int | Max lines to return. |
| `max_bytes` | int | Max bytes to return. |
Whichever limit hits first stops the read.
## List a directory
[Section titled “List a directory”](#list-a-directory)
```http
GET /sandboxes/:id/files?path=/dir&ls=true
```
```bash
curl "http://localhost:8080/sandboxes/dev/files?path=/workspace&ls=true" \
-H "Authorization: Bearer $TOKEN"
```
```json
[
{"name": "app.js", "size": 1234, "mode": "0644", "is_dir": false, "mtime": 1761792345},
{"name": "node_modules", "size": 4096, "mode": "0755", "is_dir": true, "mtime": 1761792500}
]
```
Capped at 10 000 entries; a sentinel row indicates truncation if hit.
## Stat a file
[Section titled “Stat a file”](#stat-a-file)
```http
HEAD /sandboxes/:id/files?path=/path
```
```bash
curl -I "http://localhost:8080/sandboxes/dev/files?path=/workspace/app.js" \
-H "Authorization: Bearer $TOKEN"
```
Response headers:
| Header | Description |
| -------------- | ------------------------ |
| `X-File-Size` | Size in bytes. |
| `X-File-Mode` | Octal mode, e.g. `0644`. |
| `X-File-IsDir` | `true` or `false`. |
## Write a file
[Section titled “Write a file”](#write-a-file)
```http
PUT /sandboxes/:id/files?path=/path[&mode=0644]
```
Body is the raw content. `Content-Length` is required (chunked transfer-encoding is rejected).
```bash
curl -X PUT "http://localhost:8080/sandboxes/dev/files?path=/workspace/app.js&mode=0644" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Length: 25" \
--data-binary 'console.log("hello world")'
```
```json
{"status": "ok"}
```
Writes are atomic (temp file + fsync + rename). 100 MB hard cap.
***
# Networking
[Section titled “Networking”](#networking)
## Reverse proxy
[Section titled “Reverse proxy”](#reverse-proxy)
```http
ANY /sandboxes/:id/proxy/:port/*path
```
Tunnels HTTP and WebSocket through the engine into the sandbox. The path after `/proxy/<port>/` is forwarded verbatim to `localhost:<port>` inside the VM. Bearer auth required.
Useful for development. For public, auth-free URLs, use [`publish`](#publish-a-port).
```bash
curl http://localhost:8080/sandboxes/dev/proxy/3000/api/users \
-H "Authorization: Bearer $TOKEN"
```
`502` is returned if no process is listening on the port.
## Publish a port
[Section titled “Publish a port”](#publish-a-port)
```http
POST /sandboxes/:id/publish
```
```bash
curl -X POST http://localhost:8080/sandboxes/dev/publish \
-H "Authorization: Bearer $TOKEN" \
-d '{"port": 3000, "alias": "my-app"}'
```
```json
{
"id": "pub_a1b2c3d4",
"sandbox_id": "a1b2c3d4",
"port": 3000,
"alias": "my-app",
"url": "https://my-app.bhatti.sh",
"created_at": "2026-04-30T17:00:00Z"
}
```
If `alias` is omitted, an alias is auto-generated as `<sandbox-name>-<6-hex>`. `409` is returned on alias collision.
The `url` value depends on the server config:
| Server config | URL format |
| --------------------------------- | -------------------------------------------------------------------------------------------- |
| `domain.proxy_zone: bhatti.sh` | `https://my-app.bhatti.sh` |
| `public_proxy_listen: :8443` only | `http://<host>:8443/my-app/` |
| Neither | `(no public proxy configured) alias: my-app` — the CLI surfaces a hint to use the proxy URL. |
## List published ports
[Section titled “List published ports”](#list-published-ports)
```http
GET /sandboxes/:id/publish
```
Returns an array of publish rules with their generated URLs.
## Unpublish a port
[Section titled “Unpublish a port”](#unpublish-a-port)
```http
DELETE /sandboxes/:id/publish/:port
```
```bash
curl -X DELETE http://localhost:8080/sandboxes/dev/publish/3000 \
-H "Authorization: Bearer $TOKEN"
```
`204 No Content`.
## Mint a shell token
[Section titled “Mint a shell token”](#mint-a-shell-token)
```http
POST /sandboxes/:id/shell-token
```
Generates a new web-shell token bound to this sandbox. Each call invalidates the previous token — only one is active at a time.
```bash
curl -X POST http://localhost:8080/sandboxes/dev/shell-token \
-H "Authorization: Bearer $TOKEN"
```
```json
{
"url": "https://api.bhatti.sh/_shell/a1b2c3d4#token=k3m9x2qr...",
"token": "k3m9x2qr..."
}
```
The token is in the URL **fragment** (`#token=...`); fragments are not sent to the server. The `_shell` page extracts the token client-side and uses it as a Bearer header on the WebSocket.
## Revoke a shell token
[Section titled “Revoke a shell token”](#revoke-a-shell-token)
```http
DELETE /sandboxes/:id/shell-token
```
The previously-active token (if any) is invalidated immediately.
***
# Templates
[Section titled “Templates”](#templates)
Templates are reusable sandbox blueprints. The CLI doesn’t expose them yet; reach for the API directly.
```http
GET /templates
POST /templates
GET /templates/:id
DELETE /templates/:id
```
```bash
curl -X POST http://localhost:8080/templates \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "node-dev",
"cpus": 2,
"memory_mb": 1024,
"image": "minimal",
"env": {"NODE_ENV": "development"},
"user_data": "cd /workspace && npm install",
"secrets": ["OPENAI_KEY"]
}'
```
Use `template_id` in `POST /sandboxes` to create a sandbox from a template. Template fields are defaults; per-request fields override them. **Note:** the template path silently drops request-side `secrets` and `files` arrays — set them on the template instead, or don’t use a template.
***
# Secrets
[Section titled “Secrets”](#secrets)
Encrypted at rest with [age](https://age-encryption.org/), scoped per user.
## List secrets
[Section titled “List secrets”](#list-secrets)
```http
GET /secrets
```
Returns an array of `{name, created_at, updated_at}`. Values are never returned.
## Create or update a secret
[Section titled “Create or update a secret”](#create-or-update-a-secret)
```http
POST /secrets
```
```bash
curl -X POST http://localhost:8080/secrets \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "OPENAI_KEY", "value": "sk-..."}'
```
Re-posting with an existing name updates the value silently.
## Delete a secret
[Section titled “Delete a secret”](#delete-a-secret)
```http
DELETE /secrets/:name
```
```bash
curl -X DELETE http://localhost:8080/secrets/OPENAI_KEY \
-H "Authorization: Bearer $TOKEN"
```
***
# Volumes
[Section titled “Volumes”](#volumes)
Persistent ext4 volumes. Several operations require the volume to be detached (no active sandbox attachments) and return `409` otherwise: `delete`, `resize`, `clone` (snapshot), `restore`.
## List volumes
[Section titled “List volumes”](#list-volumes)
```http
GET /volumes
```
Returns an array of volume records including current attachments.
## Create a volume
[Section titled “Create a volume”](#create-a-volume)
```http
POST /volumes
```
```bash
curl -X POST http://localhost:8080/volumes \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "workspace", "size_mb": 5120}'
```
`429` if the user is over their storage quota.
## Get a volume
[Section titled “Get a volume”](#get-a-volume)
```http
GET /volumes/:name
```
## Delete a volume
[Section titled “Delete a volume”](#delete-a-volume)
```http
DELETE /volumes/:name
```
`409` if attached.
## Resize a volume
[Section titled “Resize a volume”](#resize-a-volume)
```http
POST /volumes/:name/resize
```
```bash
curl -X POST http://localhost:8080/volumes/workspace/resize \
-H "Authorization: Bearer $TOKEN" \
-d '{"size_mb": 10240}'
```
`new_size > current_size` only — shrinking is not supported. `409` if attached.
## Clone a volume
[Section titled “Clone a volume”](#clone-a-volume)
```http
POST /volumes/:name/snapshot
```
Independent point-in-time copy. Source must be detached.
```bash
curl -X POST http://localhost:8080/volumes/workspace/snapshot \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "workspace-pre-upgrade"}'
```
***
# Backups
[Section titled “Backups”](#backups)
Volume backups to S3-compatible storage. Requires a `backup` block in the server config; otherwise every endpoint returns `501`.
## List backups for a volume
[Section titled “List backups for a volume”](#list-backups-for-a-volume)
```http
GET /volumes/:name/backups
```
## Trigger a backup
[Section titled “Trigger a backup”](#trigger-a-backup)
```http
POST /volumes/:name/backups
```
Compresses (zstd) and uploads. Does not require the volume to be detached, but writes in flight at backup time may produce an inconsistent snapshot — stop the sandbox first if consistency matters.
## Restore from a backup
[Section titled “Restore from a backup”](#restore-from-a-backup)
```http
POST /volumes/:name/backups/restore
```
```bash
curl -X POST http://localhost:8080/volumes/workspace/backups/restore \
-H "Authorization: Bearer $TOKEN" \
-d '{"backup_id": "bk_a1b2c3d4"}'
```
`409` if attached.
## Delete a backup
[Section titled “Delete a backup”](#delete-a-backup)
```http
DELETE /volumes/:name/backups/:backup_id
```
***
# Images
[Section titled “Images”](#images)
## List images
[Section titled “List images”](#list-images)
```http
GET /images
```
Returns user-owned, system tier, and shared images.
## Get an image
[Section titled “Get an image”](#get-an-image)
```http
GET /images/:name
```
## Delete an image
[Section titled “Delete an image”](#delete-an-image)
```http
DELETE /images/:name
```
System tier images can’t be deleted via this endpoint.
## Pull an OCI image
[Section titled “Pull an OCI image”](#pull-an-oci-image)
```http
POST /images/pull
```
Async — returns `202` with a task ID. Poll `/tasks/<id>` until it completes.
```bash
curl -X POST http://localhost:8080/images/pull \
-H "Authorization: Bearer $TOKEN" \
-d '{"ref": "python:3.12", "name": "python-3.12"}'
```
```json
{"task_id": "tsk_abc123", "status": "running"}
```
Optional `auth: "user:token"` for simple HTTP basic auth. For real private registries with rotating credentials, prefer `image import` after a local `docker pull`.
If the same image was already pulled with the same content digest, returns `200` immediately with the existing record.
## Import an image
[Section titled “Import an image”](#import-an-image)
```http
POST /images/import?name=<name>
```
Body is a `docker save`-style tarball streamed as `Content-Type: application/x-tar`. The CLI uses this for `bhatti image import`.
```bash
docker save python:3.12 | curl -X POST \
"http://localhost:8080/images/import?name=python-3.12" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/x-tar" \
--data-binary @-
```
```json
{"name": "python-3.12", "size_mb": 284}
```
## Save a sandbox as an image
[Section titled “Save a sandbox as an image”](#save-a-sandbox-as-an-image)
```http
POST /sandboxes/:id/save-image
```
```bash
curl -X POST http://localhost:8080/sandboxes/dev/save-image \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "my-stack"}'
```
The sandbox keeps running. Only the rootfs is captured; persistent volumes are not part of the image.
***
# Snapshots
[Section titled “Snapshots”](#snapshots)
Named VM snapshots — memory + CPU + disk + processes.
## List snapshots
[Section titled “List snapshots”](#list-snapshots)
```http
GET /snapshots
```
## Get a snapshot
[Section titled “Get a snapshot”](#get-a-snapshot)
```http
GET /snapshots/:name
```
## Delete a snapshot
[Section titled “Delete a snapshot”](#delete-a-snapshot)
```http
DELETE /snapshots/:name
```
## Create a snapshot
[Section titled “Create a snapshot”](#create-a-snapshot)
```http
POST /sandboxes/:id/checkpoint
```
```bash
curl -X POST http://localhost:8080/sandboxes/dev/checkpoint \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "pre-experiment"}'
```
The source sandbox keeps running.
## Resume from a snapshot
[Section titled “Resume from a snapshot”](#resume-from-a-snapshot)
```http
POST /snapshots/:name/resume
```
Creates a **new** sandbox from the snapshot.
```bash
curl -X POST http://localhost:8080/snapshots/pre-experiment/resume \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "dev-restored"}'
```
`name` is optional — auto-generated from the source sandbox name if omitted. Persistent volumes are not re-attached.
***
# Tasks
[Section titled “Tasks”](#tasks)
Async operations get a task ID; poll its status here.
## Get task status
[Section titled “Get task status”](#get-task-status)
```http
GET /tasks/:id
```
```json
{
"id": "tsk_abc123",
"status": "running",
"progress": "downloading layers (45 MB / 142 MB)",
"error": "",
"result": ""
}
```
`status` is `running`, `completed`, or `failed`.
## Cancel a task
[Section titled “Cancel a task”](#cancel-a-task)
```http
DELETE /tasks/:id
```
Best-effort cancel. Some operations are uncancellable past a certain point (e.g. a finalizing image conversion).
***
# Image sharing
[Section titled “Image sharing”](#image-sharing)
There’s no HTTP API for image sharing. It’s a local SQLite operation done with [`bhatti image share --user <name>`](/v1/docs/reference/cli/images/share/) on the server itself, using `--data-dir` to point at the data directory. This is an intentional restriction — image visibility is controlled at the database level, not via the API.
# CLI Reference
> Every bhatti command, global flags, environment variables, output formats, and exit codes.
`bhatti` is a single binary. `bhatti serve` runs the daemon; everything else is a CLI client that talks to the daemon’s HTTP API. All sandbox commands accept the sandbox name or ID interchangeably.
```bash
bhatti setup # configure endpoint + API key
bhatti create --name dev # create a sandbox
bhatti exec dev -- echo hello # run a command
bhatti shell dev # interactive shell (Ctrl+\ to detach)
bhatti destroy dev # clean up
```
## Operations
[Section titled “Operations”](#operations)
| Command | Syntax | Description |
| ----------------------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------- |
| **Core** | | |
| [`create`](/v1/docs/reference/cli/sandbox/create/) | `bhatti create [flags]` | Create a new sandbox VM |
| [`destroy`](/v1/docs/reference/cli/sandbox/destroy/) | `bhatti destroy <sandbox> [-y]` | Destroy a sandbox (alias: `rm`) |
| [`edit`](/v1/docs/reference/cli/sandbox/edit/) | `bhatti edit <sandbox> [--keep-hot \| --allow-cold]` | Update mutable settings |
| [`exec`](/v1/docs/reference/cli/exec/exec/) | `bhatti exec <sandbox> [--] <command...>` | Run a command in a sandbox |
| [`inspect`](/v1/docs/reference/cli/sandbox/inspect/) | `bhatti inspect <sandbox>` | Show sandbox details (alias: `info`) |
| [`list`](/v1/docs/reference/cli/sandbox/list/) | `bhatti list [-o wide]` | List sandboxes (alias: `ls`) |
| [`ports`](/v1/docs/reference/cli/exec/ports/) | `bhatti ports <sandbox>` | List listening ports inside a sandbox |
| [`ps`](/v1/docs/reference/cli/exec/ps/) | `bhatti ps <sandbox>` | List active sessions in a sandbox |
| [`share`](/v1/docs/reference/cli/networking/share/) | `bhatti share <sandbox> [--revoke]` | Generate a web shell URL |
| [`shell`](/v1/docs/reference/cli/exec/shell/) | `bhatti shell <sandbox> [--new]` | Open an interactive shell (alias: `sh`) |
| [`start`](/v1/docs/reference/cli/sandbox/start/) | `bhatti start <sandbox> [--force]` | Resume a stopped sandbox |
| [`stop`](/v1/docs/reference/cli/sandbox/stop/) | `bhatti stop <sandbox>` | Snapshot and stop a sandbox |
| **Files** | | |
| [`file read`](/v1/docs/reference/cli/files/read/) | `bhatti file read <sandbox> <path>` | Read a file from a sandbox |
| [`file write`](/v1/docs/reference/cli/files/write/) | `bhatti file write <sandbox> <path>` | Write a file (reads from stdin) |
| [`file ls`](/v1/docs/reference/cli/files/ls/) | `bhatti file ls <sandbox> <path>` | List directory contents |
| **Networking** | | |
| [`publish`](/v1/docs/reference/cli/networking/publish/) | `bhatti publish <sandbox> -p <port> [-a <alias>] [--shell]` | Publish a port with a public URL |
| [`unpublish`](/v1/docs/reference/cli/networking/unpublish/) | `bhatti unpublish <sandbox> -p <port>` | Remove a published port |
| **Images** | | |
| [`image list`](/v1/docs/reference/cli/images/list/) | `bhatti image list` | List available rootfs images |
| [`image pull`](/v1/docs/reference/cli/images/pull/) | `bhatti image pull <ref> [--name N] [--auth U:T]` | Pull from a public OCI registry |
| [`image import`](/v1/docs/reference/cli/images/import/) | `bhatti image import <docker-ref> \| --tar <path> [--name N]` | Import a Docker image or tarball |
| [`image save`](/v1/docs/reference/cli/images/save/) | `bhatti image save <sandbox> --name <image>` | Save a sandbox’s filesystem as an image |
| [`image delete`](/v1/docs/reference/cli/images/delete/) | `bhatti image delete <name> [-y]` | Delete an image |
| [`image share`](/v1/docs/reference/cli/images/share/) | `bhatti image share <name> --user <u>...` | Share an image with specific users |
| [`image unshare`](/v1/docs/reference/cli/images/unshare/) | `bhatti image unshare <name> --user <u>...` | Revoke image access from users |
| **Volumes** | | |
| [`volume create`](/v1/docs/reference/cli/volumes/create/) | `bhatti volume create --name <v> --size <MB>` | Create a persistent volume |
| [`volume list`](/v1/docs/reference/cli/volumes/list/) | `bhatti volume list` | List volumes |
| [`volume delete`](/v1/docs/reference/cli/volumes/delete/) | `bhatti volume delete <name> [-y]` | Delete a volume |
| [`volume resize`](/v1/docs/reference/cli/volumes/resize/) | `bhatti volume resize <name> --size <MB>` | Resize a volume (grow only) |
| [`volume clone`](/v1/docs/reference/cli/volumes/clone/) | `bhatti volume clone <src> --name <new>` | Clone a volume (point-in-time copy) |
| [`volume backup`](/v1/docs/reference/cli/volumes/backup/) | `bhatti volume backup <name>` | Back up a volume to S3 |
| [`volume backup-list`](/v1/docs/reference/cli/volumes/backup-list/) | `bhatti volume backup-list <name>` | List backups for a volume |
| [`volume restore`](/v1/docs/reference/cli/volumes/restore/) | `bhatti volume restore <name> --backup-id <id>` | Restore a volume from a backup |
| [`volume backup-delete`](/v1/docs/reference/cli/volumes/backup-delete/) | `bhatti volume backup-delete <name> <id> [-y]` | Delete a volume backup |
| **Secrets** | | |
| [`secret set`](/v1/docs/reference/cli/secrets/set/) | `bhatti secret set <name> <value>` | Create or update a secret |
| [`secret list`](/v1/docs/reference/cli/secrets/list/) | `bhatti secret list` | List secret names |
| [`secret delete`](/v1/docs/reference/cli/secrets/delete/) | `bhatti secret delete <name> [-y]` | Delete a secret |
| **Snapshots** | | |
| [`snapshot create`](/v1/docs/reference/cli/snapshots/create/) | `bhatti snapshot create <sandbox> --name <snap>` | Checkpoint a running sandbox |
| [`snapshot list`](/v1/docs/reference/cli/snapshots/list/) | `bhatti snapshot list` | List snapshots |
| [`snapshot resume`](/v1/docs/reference/cli/snapshots/resume/) | `bhatti snapshot resume <snap> [--name <new>]` | Resume from a snapshot into a new sandbox |
| [`snapshot delete`](/v1/docs/reference/cli/snapshots/delete/) | `bhatti snapshot delete <name> [-y]` | Delete a snapshot |
| **Server & admin** | | |
| [`serve`](/v1/docs/reference/cli/admin/serve/) | `bhatti serve` | Start the bhatti daemon |
| [`setup`](/v1/docs/reference/cli/admin/setup/) | `bhatti setup` | Configure CLI endpoint and API key |
| [`version`](/v1/docs/reference/cli/admin/version/) | `bhatti version` | Print version and check for updates |
| [`update`](/v1/docs/reference/cli/admin/update/) | `bhatti update [--cli-only] [--tiers <list>]` | Update bhatti |
| [`completion`](/v1/docs/reference/cli/admin/completion/) | `bhatti completion <bash\|zsh\|fish>` | Generate shell completion script |
| [`user create`](/v1/docs/reference/cli/admin/user-create/) | `bhatti user create --name <u> [--max-...]` | Create a user (server-only) |
| [`user list`](/v1/docs/reference/cli/admin/user-list/) | `bhatti user list` | List users (server-only) |
| [`user rotate-key`](/v1/docs/reference/cli/admin/user-rotate-key/) | `bhatti user rotate-key <name>` | Rotate a user’s API key (server-only) |
| [`user delete`](/v1/docs/reference/cli/admin/user-delete/) | `bhatti user delete <name> [-y]` | Delete a user (server-only) |
| [`admin status`](/v1/docs/reference/cli/admin/admin-status/) | `bhatti admin status` | One-shot system overview (server-only) |
| [`admin events`](/v1/docs/reference/cli/admin/admin-events/) | `bhatti admin events [--type T] [--since 24h]` | Query the event log (server-only) |
| [`admin metrics`](/v1/docs/reference/cli/admin/admin-metrics/) | `bhatti admin metrics [--since 1h]` | Query metrics snapshots (server-only) |
## Global flags
[Section titled “Global flags”](#global-flags)
These flags are accepted by every command.
| Flag | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--url <url>` | API endpoint (overrides config and `BHATTI_URL`). |
| `--token <key>` | API key (overrides config and `BHATTI_TOKEN`). |
| `--data-dir <path>` | Path to the server’s data directory. Required for `bhatti user *` and `bhatti admin *` since those operate directly on the local SQLite database. |
| `--json` | Print machine-readable JSON instead of the default human format. Universally supported. |
| `--timing` | Print a per-request timing breakdown (DNS, connect, TLS, server, transfer, total) to stderr after the command runs. |
| `-h`, `--help` | Show help for any command. |
## Environment variables
[Section titled “Environment variables”](#environment-variables)
| Variable | Used by | Description |
| --------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `BHATTI_URL` | CLI | API endpoint. Falls back when `--url` is not set and the config file has no `api_url`. |
| `BHATTI_TOKEN` | CLI | API key. Same fallback semantics as `BHATTI_URL`. |
| `BHATTI_CONFIG` | CLI + server | Override path to the config file. When set, layered loading is bypassed and only this file is read. |
| `BHATTI_LOG_LEVEL` | server (`bhatti serve`) | `debug`, `info` (default), `warn`, `error`. |
| `BHATTI_FORCE_STREAM` | CLI (`bhatti exec`) | Set to `1` to force NDJSON streaming output even when stdout is not a TTY. Useful in CI logs where you want real-time progress. |
## Configuration precedence
[Section titled “Configuration precedence”](#configuration-precedence)
Per-call flag → config file → environment variable → built-in default.
```plaintext
--url <x> (highest)
api_url: <x> (in config file)
BHATTI_URL=<x> (env var)
http://localhost:8080 (default, lowest)
```
The config file is the primary source — `bhatti setup` writes it and most users never touch the rest. Environment variables are the fallback for CI pipelines and scripts.
See [Configuration](/v1/docs/reference/config/) for the file format and layered loading rules (`/etc/bhatti/config.yaml` for system settings, `~/.bhatti/config.yaml` for client credentials).
## Output formats
[Section titled “Output formats”](#output-formats)
**Human (default).** Compact, TTY-friendly. `bhatti list` prints columns; `bhatti inspect` prints kubectl-`describe`-style details with live disk usage on running VMs.
**`--json`.** Universally supported. Drops all human formatting; emits the raw API response (for queries) or a small status object (for mutations).
**`bhatti list -o wide`.** Adds CPUs, memory, disk, and image columns to `list`. Equivalent to `kubectl get -o wide`.
**Streaming exec output.** `bhatti exec` auto-detects how to format its output:
* Stdout is a TTY → NDJSON streaming, line-by-line, flushed immediately.
* Stdout is a pipe or `--json` is set → buffered JSON with `{exit_code, stdout, stderr}`.
* Override with `BHATTI_FORCE_STREAM=1` to force streaming when piping.
## Exit codes
[Section titled “Exit codes”](#exit-codes)
| Code | Meaning |
| ----- | -------------------------------------------------------------------------------------------------------------------------------- |
| `0` | Success. |
| `1` | CLI or API error (see stderr for details and recovery hints). |
| `<n>` | `bhatti exec` only — the child process’s exit code is forwarded verbatim, so a script’s `$?` reflects the command’s true status. |
Destructive commands (`destroy`, `volume delete`, `image delete`, `snapshot delete`, `volume backup-delete`, `secret delete`, `user delete`) require confirmation. Pass `-y` / `--yes` to skip the prompt; without `-y` in a non-interactive shell, the command exits `1`.
## Shell completion
[Section titled “Shell completion”](#shell-completion)
```bash
bhatti completion bash > /etc/bash_completion.d/bhatti
bhatti completion zsh > "${fpath[1]}/_bhatti"
bhatti completion fish > ~/.config/fish/completions/bhatti.fish
```
Sandbox-name completion uses a local cache (`/tmp/bhatti-completions-<uid>`) updated by `create`, `destroy`, and `list`. It never hits the network — instant, works offline.
# Server & admin
> Daemon, configuration, version, updates, user management, observability.
Commands that run on the bhatti server itself, plus a couple that work from any client (`setup`, `version`, `completion`, `update`).
## Categories
[Section titled “Categories”](#categories)
| | |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Daemon** | [`serve`](/v1/docs/reference/cli/admin/serve/) |
| **Client config** | [`setup`](/v1/docs/reference/cli/admin/setup/), [`version`](/v1/docs/reference/cli/admin/version/), [`completion`](/v1/docs/reference/cli/admin/completion/) |
| **Updates** | [`update`](/v1/docs/reference/cli/admin/update/) |
| **User management** | [`user create`](/v1/docs/reference/cli/admin/user-create/), [`user list`](/v1/docs/reference/cli/admin/user-list/), [`user rotate-key`](/v1/docs/reference/cli/admin/user-rotate-key/), [`user delete`](/v1/docs/reference/cli/admin/user-delete/) |
| **Observability** | [`admin status`](/v1/docs/reference/cli/admin/admin-status/), [`admin events`](/v1/docs/reference/cli/admin/admin-events/), [`admin metrics`](/v1/docs/reference/cli/admin/admin-metrics/) |
## Server-only commands
[Section titled “Server-only commands”](#server-only-commands)
`user *` and `admin *` operate **directly on the server’s SQLite database** (`/var/lib/bhatti/state.db`). They aren’t HTTP API calls — they need filesystem access to the database. Run them on the server, typically with `sudo`.
`serve` runs the daemon. Also server-only by definition (requires KVM and root).
The remaining commands (`setup`, `version`, `update`, `completion`) work from any client.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
# Bootstrap a new user (server)
sudo bhatti user create --name alice --max-sandboxes 10
# Check overall health (server)
sudo bhatti admin status
# Find recent thermal pauses (server)
sudo bhatti admin events --type thermal --since 24h
# Update the CLI on a developer machine
bhatti update
# Update everything on the server (CLI, kernel, rootfs, lohar)
sudo bhatti update
```
# bhatti admin events
> Query the event log. Filter by type, user, sandbox, time range.
# bhatti admin events server only
Query the structured event log.
## Synopsis
```
sudo bhatti admin events
[--type <prefix>] [--user <name|id>] [--sandbox <name|id>]
[--since <range>] [--limit <n>] [--count]
```
## Description
Returns rows from the audit/observability event log stored in the local database. Each event captures a moment in the system: sandbox lifecycle changes, thermal transitions, snapshot operations, image pulls, auth failures, proxy errors, daemon start/shutdown, and more.
Default output is one row per event with timestamp, type, user, sandbox, and a per-event-type summary line. JSON output includes the full meta object.
`--type` is a **prefix match**, so `--type thermal` matches `thermal.pause`, `thermal.wake`, `thermal.snapshot`, etc.
`--since` accepts:
* Relative durations: `30s`, `15m`, `24h`, `7d`.
* Absolute dates: `2026-04-01` (UTC midnight), `2026-04-01T12:00:00`, full RFC3339.
`--user` and `--sandbox` accept either the name or the internal ID. Names are easier; IDs are stable across renames.
`--count` returns a number-only response — useful for alerting/scripting (`if [[ $(bhatti admin events --type auth.failed --since 5m --count) -gt 10 ]]; then ...`).
## Examples
```bash
# Last 50 events of any kind
sudo bhatti admin events
```
```bash
# All thermal events in the last day
sudo bhatti admin events --type thermal --since 24h
```
```bash
# Auth failures in the last hour, count only
sudo bhatti admin events --type auth.failed --since 1h --count
```
```text
3
```
```bash
# Everything Alice did this week
sudo bhatti admin events --user alice --since 7d
```
```bash
# Drill into one sandbox's history
sudo bhatti admin events --sandbox dev --since 24h
```
```bash
# JSON for piping into jq
sudo bhatti admin events --type sandbox.destroyed --since 7d --json \
| jq '[.[] | .meta.lifetime_s] | add / length' # avg sandbox lifetime
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--type <prefix>` | — | Filter by event type. Prefix match (`thermal` matches `thermal.*`). | | `--user <name\|id>` | — | Filter by user. Name or user ID. | | `--sandbox <name\|id>` | — | Filter by sandbox. Name or ID. | | `--since <range>` | — | Earliest event to include. Relative (`24h`, `7d`) or absolute (`2026-04-01`). | | `--limit <int>` | `50` | Maximum number of events to return. | | `--count` | `false` | Return only the count of matching events, not the rows. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti admin status`](/v1/docs/reference/cli/admin/admin-status/) — shows the most recent \~10 events inline
* [`bhatti admin metrics`](/v1/docs/reference/cli/admin/admin-metrics/) — counter-based view
# bhatti admin metrics
> Time-bucketed metrics — request rates, errors, thermal counts, host load.
# bhatti admin metrics server only
Query metrics snapshots over time.
## Synopsis
```
sudo bhatti admin metrics [--since <range>]
```
## Description
The server takes a metrics snapshot roughly every minute and persists it to the database (in `metrics_snapshots`). This command reads them back and prints a compact table — request rates, error counts, proxy traffic, cold wakes, sandbox state distribution, host load, available memory.
The output is **bucketed automatically** based on the time range:
\| Range | Bucket size | | ----- | ----------- | | < 2 hours | 1 minute | | 2–24 hours | 15 minutes | | > 24 hours | 1 hour |
Each row aggregates the snapshots in its bucket (sums for counters like `req/min`, last-value for gauges like `LOAD`).
`--since` accepts the same formats as [`admin events --since`](/v1/docs/reference/cli/admin/admin-events/) — relative durations or absolute dates. The default is **1 hour**.
## Examples
```bash
# Last hour at 1-minute resolution (default)
sudo bhatti admin metrics
```
```text
TIME REQ/min ERR PROXY/min WAKES HOT WARM COLD LOAD MEM_AVAIL
14:21 47 0 8 0 4 6 7 0.4 19421 MB
14:20 52 1 9 1 4 5 8 0.5 19388 MB
14:19 44 0 7 0 4 4 9 0.5 19412 MB
...
```
```bash
# Last 12 hours, auto-bucketed to 15-minute intervals
sudo bhatti admin metrics --since 12h
```
```bash
# Last week, auto-bucketed to 1-hour intervals
sudo bhatti admin metrics --since 7d
```
```bash
# Machine-readable, raw 1-min snapshots for the last 30 minutes
sudo bhatti admin metrics --since 30m --json | jq '.[] | .api_requests'
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--since <range>` | `1h` | Earliest snapshot to include. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti admin status`](/v1/docs/reference/cli/admin/admin-status/) — current totals
* [`bhatti admin events`](/v1/docs/reference/cli/admin/admin-events/) — what happened, when
# bhatti admin status
> One-shot system overview — sandboxes, users, host load, recent events.
# bhatti admin status server only
One-shot system overview.
## Synopsis
```
sudo bhatti admin status
```
## Description
Reads the latest metrics snapshot and event log from the local SQLite database and prints a compact dashboard:
* **Version + update notice** — current server version, latest GitHub release if newer.
* **Host stats** — load average, memory in use vs. total.
* **Sandbox totals** — broken down by thermal state (hot/warm/cold).
* **User totals** — total users, currently-active users.
* **API + proxy counters** — total requests, errors, auth failures, cold-wake count.
* **Per-user breakdown** — sandbox count vs. limit, current running/stopped split.
* **Recent events** — last \~10 events from the audit log with a one-line summary each.
Reads database state directly; doesn't touch the engine. Safe to run while the daemon is busy.
## Examples
```bash
sudo bhatti admin status
```
```text
Bhatti v1.10.0
Host: 0.42 load, 12.4 / 32.0 GB memory
Sandboxes 17 total (4 hot, 6 warm, 7 cold)
Users 8 total, 5 active
API 184.2K requests, 423 errors, 12 auth failures
Proxy 47.1K requests, 89 errors, 156 cold wakes
USER SANDBOXES STATUS
alice 3/10 2 running, 1 stopped
bob 0/5 -
worker-bot 5/5 5 running
RECENT EVENTS
14:22:18 thermal.pause alice scraper idle=180s
14:21:45 sandbox.created bob test-1 cpus=2 mem=1024 image=minimal
14:20:11 proxy.error - staging alias=staging status=502
14:18:33 thermal.wake alice api from=cold 47ms
...
```
```bash
# Machine-readable
sudo bhatti admin status --json | jq .gauges.sandboxes_hot
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti admin events`](/v1/docs/reference/cli/admin/admin-events/) — drill into the event log
* [`bhatti admin metrics`](/v1/docs/reference/cli/admin/admin-metrics/) — time-series view
# bhatti completion
> Generate shell completion scripts for bash, zsh, or fish.
Generate a shell completion script.
## Synopsis
```
bhatti completion <bash | zsh | fish>
```
## Description
Prints the completion script for the requested shell to stdout. Save it to the right place for your shell, restart, and tab-completion is enabled.
**Sandbox-name completion** uses a local cache (`/tmp/bhatti-completions-<uid>`), populated by `create`, `destroy`, and `list`. It never hits the network, so completions are instant and work offline. The cache may be slightly stale right after another machine creates a sandbox; running `bhatti list` refreshes it.
## Examples
### bash
```bash
# System-wide
bhatti completion bash | sudo tee /etc/bash_completion.d/bhatti > /dev/null
# Or per-user (no sudo)
echo 'source <(bhatti completion bash)' >> ~/.bashrc
```
### zsh
```bash
bhatti completion zsh > "${fpath[1]}/_bhatti"
# Or simpler
echo 'source <(bhatti completion zsh)' >> ~/.zshrc
```
### fish
```bash
bhatti completion fish > ~/.config/fish/completions/bhatti.fish
```
## Options
This command takes no flags besides the positional shell argument.
## See also
* [`bhatti setup`](/v1/docs/reference/cli/admin/setup/) — `setup` detects your shell and prints the right snippet automatically.
# bhatti serve
> Start the bhatti daemon. Requires root and KVM.
# bhatti serve server only
Start the bhatti daemon.
## Synopsis
```
sudo bhatti serve
```
## Description
Starts the HTTP API listener, the thermal manager, the public proxy (if domain mode or `public_proxy_listen` is configured), the port scanner, and the metrics + event recorders. Recovers any previously running sandboxes from the SQLite database on startup and re-attaches their TAP devices.
Requires:
* Linux with KVM (`/dev/kvm`)
* Root, because Firecracker needs it
* Config at `/etc/bhatti/config.yaml` (or `$BHATTI_CONFIG`)
* The Firecracker binary, kernel image, and at least one rootfs image present at the paths in the config
Logs are emitted as structured JSON to stdout. Set `BHATTI_LOG_LEVEL=debug` for verbose logging. Most operators run this under systemd; the install script sets up the unit file at `/etc/systemd/system/bhatti.service`.
## Examples
```bash
# Foreground (development)
sudo bhatti serve
```
```bash
# Under systemd (production)
sudo systemctl start bhatti
sudo journalctl -u bhatti -f
```
```bash
# Verbose logs
BHATTI_LOG_LEVEL=debug sudo bhatti serve
```
## Options
This command takes no flags. Configuration is via `/etc/bhatti/config.yaml`. See [Configuration](/v1/docs/reference/config/).
## Lifecycle
* **Startup:** load config → open store → initialise engine → recover VMs → start servers (API + optional public proxy + optional internal loopback).
* **Shutdown** (`SIGTERM` / `SIGINT`): drain HTTP, snapshot all running VMs to disk, close store, exit. Snapshots are restored on the next start.
## See also
* [Configuration](/v1/docs/reference/config/)
* [Self-hosting](/v1/docs/self-hosting/)
# bhatti setup
> Configure the CLI's API endpoint and key. Saves to ~/.bhatti/config.yaml. Interactive by default; --url and --token make it scriptable.
Configure the CLI's API endpoint and key.
## Synopsis
```
bhatti setup [--url URL] [--token KEY]
```
## Description
Writes the API endpoint and key to `~/.bhatti/config.yaml` and tests the connection by listing sandboxes. The auth test hits `GET /sandboxes` (an authenticated endpoint), so a bad key fails immediately instead of appearing to succeed.
With no flags, runs interactively — prompts for endpoint and key, masks the key as you type. With `--url` and `--token`, runs non-interactively. Use the latter form for agents, CI scripts, and provisioning tools.
You usually don't need to run this on the server itself: the install script (`curl -fsSL bhatti.sh/install | sudo bash`) creates the admin user and writes the config to your `$SUDO_USER`'s `~/.bhatti/config.yaml` automatically. Run `setup` on a different machine that you want to drive the server from, or to switch your local CLI to a different bhatti server.
## Examples
### Interactive (the default)
```bash
bhatti setup
```
```text
API endpoint [http://localhost:8080]: https://api.bhatti.sh
API key: <input is hidden>
Saved to /home/alice/.bhatti/config.yaml
Testing connection... ✓ authenticated (3 sandboxes)
Enable completions:
echo 'source <(bhatti completion zsh)' >> ~/.zshrc
```
### Non-interactive (agents, CI, provisioners)
```bash
bhatti setup --url https://api.bhatti.sh --token bht_abc123
```
```text
Saved to /home/alice/.bhatti/config.yaml
Testing connection... ✓ authenticated (3 sandboxes)
```
The completions hint is suppressed — there's no human reading the output anyway. If the connection test fails, the command exits non-zero so your provisioner can detect the failure.
## Options
``string required
API endpoint URL. Sets the prompt's default. With `--token` set too, the prompt is skipped entirely and `setup` writes immediately.
``string required
API key. With `--url` set too, the prompt is skipped and `setup` writes immediately.
The auth test always runs. If you need to write the config without contacting a server (pre-baking an image, for example), edit `~/.bhatti/config.yaml` directly — it's just two YAML keys.
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token` (the global versions of these — they override config but don't write to it; `setup`'s flags are the persisting form), `--json`, `--timing`, `--data-dir`.
## Configuration file
After `setup`, your config looks like:
```yaml
api_url: https://api.bhatti.sh
auth_token: bht_abc123def456...
```
At runtime, this can be overridden with `--url`, `--token`, `BHATTI_URL`, or `BHATTI_TOKEN`. See [Configuration precedence](/v1/docs/reference/cli/#configuration-precedence).
## See also
* [`bhatti version`](/v1/docs/reference/cli/admin/version/) — check connection without changing config
* [`bhatti user create`](/v1/docs/reference/cli/admin/user-create/) — create a user and get an API key (server-side, run by the operator)
* [`bhatti completion`](/v1/docs/reference/cli/admin/completion/) — shell completions
* [Configuration](/v1/docs/reference/config/) — full file format
* [Self-hosting → adding teammates](/v1/docs/self-hosting/#adding-teammates) — the full flow for setting someone up against a remote bhatti
# bhatti update
> Update bhatti. Auto-detects CLI vs server.
Update bhatti to the latest release.
## Synopsis
```
bhatti update # auto-detect: CLI on a client, full update on a server
sudo bhatti update # full server update (CLI, kernel, lohar, rootfs)
sudo bhatti update --tiers all # also pull additional rootfs tiers
bhatti update --cli-only # update just the CLI binary, even on a server
```
## Description
Downloads `https://bhatti.sh/install` and runs it. The installer auto-detects whether this is a server (config exists at `/etc/bhatti/config.yaml`) or a CLI-only host, and updates accordingly.
**On a CLI-only machine:** updates the `bhatti` binary at `/usr/local/bin/bhatti`. Doesn't need root unless `/usr/local/bin` isn't writable by the user.
**On a server:** updates everything — CLI, Firecracker, lohar, kernel, and any installed rootfs tier images. Only changed components are re-downloaded; unchanged tiers and kernels are skipped.
**`--cli-only`** forces the binary-only update path even on a server. Useful when you want to test a CLI fix without touching kernel/rootfs.
**`--tiers`** takes a comma-separated list (or `all`) of additional rootfs tiers to install (`browser`, `docker`, `computer`). The default install only ships `minimal`; this is how you add more without re-running the full install command.
Server updates require root (Firecracker and `/var/lib/bhatti` are root-owned). The command refuses up front if it detects a server install but the user isn't root, rather than partly-completing then failing inside the install script.
## Examples
```bash
# CLI host
bhatti update
```
```bash
# Server, default behaviour (full update)
sudo bhatti update
```
```bash
# Server, also pull all tiers
sudo bhatti update --tiers all
```
```bash
# Server, only update the CLI binary
bhatti update --cli-only
```
```bash
# Server, add a single tier
sudo bhatti update --tiers browser
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--cli-only` | `false` | Update only the CLI binary, even on a server. | | `--tiers <list>` | — | Comma-separated rootfs tiers to install, or `all`. Server-only — has no effect on CLI hosts. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti version`](/v1/docs/reference/cli/admin/version/) — check current version + latest release
* [Self-hosting](/v1/docs/self-hosting/) — initial install
* [Updating](/v1/docs/updating/) — upgrade and uninstall guide
# bhatti user create
> Create a user with API key and per-user resource limits.
# bhatti user create server only
Create a user with an API key and per-user resource limits.
## Synopsis
```
sudo bhatti user create --name <name>
[--max-sandboxes <n>] [--max-cpus <n>] [--max-memory <MB>]
```
## Description
Allocates a new user record in the local SQLite database. The install script already runs this once for `admin` during setup, so the first time you'd call `user create` yourself is when you want to add a teammate or a CI account.
Each user gets:
* A user ID (`usr_<8-hex>`).
* An **API key** (`bht_<64-hex>`) — printed once, then only its SHA-256 hash is stored. **Save the key immediately**; it can't be recovered. Use [`user rotate-key`](/v1/docs/reference/cli/admin/user-rotate-key/) if it's lost.
* A **subnet index** — an integer assigned monotonically. The server uses it to derive a per-user `/24` bridge network for sandbox isolation. You don't need to think about it; it just shows up in `inspect` output.
* Per-sandbox resource caps (`--max-cpus`, `--max-memory`) and a per-user sandbox count cap (`--max-sandboxes`).
Records the `user.created` event in the audit log.
This is a **server-only** command — it operates directly on `/var/lib/bhatti/state.db`. Runs without the daemon needing to be up, but typically as `sudo` because the data dir is root-owned.
## Examples
```bash
sudo bhatti user create --name alice
```
```text
User created:
ID: usr_a1b2c3d4
Name: alice
Subnet: 2
API key: bht_abc123def456...
This key will not be shown again. Save it now.
For remote CLI use:
bhatti setup --url https://your-server:8080 --token bht_abc123def456...
```
```bash
# Custom limits
sudo bhatti user create --name beefy \
--max-sandboxes 20 \
--max-cpus 8 \
--max-memory 16384
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | (required) | Username. Must be unique. | | `--max-sandboxes <int>` | `5` | Maximum concurrent sandboxes. | | `--max-cpus <int>` | `4` | Maximum vCPUs per sandbox (capped at create time). | | `--max-memory <int>` | `4096` | Maximum memory MB per sandbox (capped at create time). |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
`--data-dir` is honoured here when you need to point at a non-default data directory.
## See also
* [`bhatti user list`](/v1/docs/reference/cli/admin/user-list/)
* [`bhatti user rotate-key`](/v1/docs/reference/cli/admin/user-rotate-key/) — replace a lost or compromised key
* [Users & auth](/v1/docs/managing/users/) — broader narrative on multi-tenant setup
# bhatti user delete
> Delete a user. Fails if the user has active sandboxes.
# bhatti user delete server only
Delete a user.
## Synopsis
```
sudo bhatti user delete <name> [-y]
```
## Description
Removes the user record. The user's API key stops working immediately.
**Fails if the user has active (non-destroyed) sandboxes.** Destroy them first:
```bash
sudo bhatti user list # find them via the user's sandbox count
# (server-side — to list a specific user's sandboxes you need the SQL or the user's API key)
```
Volumes, secrets, snapshots, and images owned by the user remain on disk after the user is deleted (orphaned). A future `--purge` mode could clean these up; for now, manual cleanup is required if you want to reclaim the space.
By default, prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
Records a `user.deleted` event in the audit log.
## Examples
```bash
sudo bhatti user delete alice
```
```text
Delete user "alice"? [y/N]: y
deleted user "alice"
```
```bash
# Active sandboxes still around
sudo bhatti user delete alice
```
```text
Error: user "alice" has 3 active sandbox(es)
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti user list`](/v1/docs/reference/cli/admin/user-list/)
* [`bhatti user rotate-key`](/v1/docs/reference/cli/admin/user-rotate-key/) — usually a better answer than delete
# bhatti user list
> List users with their resource limits and current sandbox counts.
# bhatti user list server only
List all users on the server.
## Synopsis
```
sudo bhatti user list
```
## Description
Lists all users with ID, name, current sandbox count vs. limit, per-sandbox CPU and memory caps, and assigned subnet index. Reads from the local SQLite database.
## Examples
```bash
sudo bhatti user list
```
```text
ID NAME SANDBOXES CPUS MEM SUBNET
usr_a1b2c3d4 alice 3/10 4 4096 2
usr_e5f6g7h8 bob 0/5 4 4096 3
usr_i9j0k1l2 worker-bot 5/5 2 2048 4
```
The `SANDBOXES` column is `current/limit` — `5/5` means at-cap.
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti user create`](/v1/docs/reference/cli/admin/user-create/)
* [`bhatti admin status`](/v1/docs/reference/cli/admin/admin-status/) — also shows per-user sandbox breakdown
# bhatti user rotate-key
> Rotate a user's API key. Old key is immediately invalidated.
# bhatti user rotate-key server only
Rotate a user's API key.
## Synopsis
```
sudo bhatti user rotate-key <name>
```
## Description
Generates a new API key, stores its SHA-256 hash, and prints the plaintext **once**. The previous key is invalidated server-side immediately — any in-flight requests using the old key fail with `401`.
Use this when:
* A key may have leaked.
* A user has lost their key (the original is unrecoverable).
* You have a key-rotation policy.
The user's existing sandboxes, secrets, volumes, snapshots, and images are all preserved — only the auth token changes.
Records a `user.key_rotated` event in the audit log.
## Examples
```bash
sudo bhatti user rotate-key alice
```
```text
API key rotated for "alice"
New key: bht_xyz789...
The old key is immediately invalidated.
This key will not be shown again. Save it now.
```
After rotation, give the user the new key. They run `bhatti setup` to update their config.
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti user create`](/v1/docs/reference/cli/admin/user-create/)
* [`bhatti setup`](/v1/docs/reference/cli/admin/setup/) — what the user runs to install the new key
# bhatti version
> Print CLI version, server version, and check for updates.
Print CLI and server versions; check for updates.
## Synopsis
```
bhatti version
```
## Description
Prints:
* **CLI version** — embedded at build time via `-ldflags`. Shows `dev` for unreleased builds.
* **API endpoint** — currently configured from flags, env, or config.
* **Server version** — read from the `X-Bhatti-Version` response header on a probe to `GET /sandboxes`. Skipped if unreachable or unauthenticated.
* **Update notice** — if the CLI's GitHub release check (cached for 1 hour at `~/.bhatti/.latest-version`) finds a newer release, prints it.
The server-version probe also drives the `X-Bhatti-Min-CLI` warning that appears on every command — if your CLI is older than the server's minimum, every command emits a one-line stderr warning.
## Examples
```bash
bhatti version
```
```text
bhatti v1.10.0
api: https://api.bhatti.sh
server: v1.10.0
```
```bash
# When an update is available
bhatti version
```
```text
bhatti v1.9.0
api: https://api.bhatti.sh
server: v1.10.0
Update available: v1.9.0 → v1.10.0 (bhatti update)
```
```bash
# Machine-readable
bhatti version --json
```
```json
{
"version": "v1.10.0",
"api": "https://api.bhatti.sh",
"server_version": "v1.10.0",
"latest_version": "v1.10.0"
}
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti update`](/v1/docs/reference/cli/admin/update/) — apply available updates
* [`bhatti admin status`](/v1/docs/reference/cli/admin/admin-status/) — also prints version info on the server side
# Execution & shells
> Run commands, open shells, list active sessions, and discover listening ports inside sandboxes.
Commands that drive execution inside a running sandbox: one-shot commands, interactive shells, and visibility into what’s currently running and listening.
| Command | Description |
| ---------------------------------------------------- | ------------------------------------------------------------------------------- |
| [`bhatti exec`](/v1/docs/reference/cli/exec/exec/) | Run a command. Streaming or buffered output, optional timeout, optional detach. |
| [`bhatti shell`](/v1/docs/reference/cli/exec/shell/) | Open an interactive shell with detach/reattach semantics (alias: `sh`). |
| [`bhatti ps`](/v1/docs/reference/cli/exec/ps/) | List active sessions in a sandbox (init scripts, shells, detached execs). |
| [`bhatti ports`](/v1/docs/reference/cli/exec/ports/) | List listening ports detected inside a sandbox. |
All commands wake cold sandboxes transparently — you don’t need to start the sandbox first.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
bhatti exec dev -- npm install # one-shot command
bhatti exec dev -- bash -c 'echo $API_KEY' # shell expansion via /bin/bash
bhatti exec dev --timeout 60 -- long-running.sh # cap runtime at 60s
bhatti exec dev --detach -- ./worker.sh # run in background, return PID
bhatti shell dev # interactive (Ctrl+\ to detach)
bhatti shell dev --new # don't reattach to a running session
bhatti ps dev # what's running inside?
bhatti ports dev # what's listening?
```
## See also
[Section titled “See also”](#see-also)
* [Sandbox commands](/v1/docs/reference/cli/sandbox/) — create/list/destroy
* [Files](/v1/docs/reference/cli/files/) — read and write files inside sandboxes
* [API: `POST /sandboxes/:id/exec`](/v1/docs/reference/api/#run-a-command) — the underlying HTTP endpoint, streaming and detach modes
# bhatti exec
> Execute a command inside a sandbox. Streaming or buffered output, optional timeout, optional detach.
Execute a command inside a sandbox. The exit code of the child process is forwarded to the CLI's exit code. Sleeping sandboxes wake automatically.
## Synopsis
```
bhatti exec <sandbox> [--] <command...> [flags]
```
## Description
The `--` separator is optional — `bhatti exec dev echo hello` works just like `bhatti exec dev -- echo hello`. Use `--` only when the command contains flags that cobra would otherwise try to parse (e.g. `bhatti exec dev -- ls -la /tmp`).
Commands run as the unprivileged user `lohar` (uid 1000), not root. Use `sudo` inside the sandbox for root operations. Stdout and stderr are kept on separate streams; when output is captured, the result distinguishes them.
**Output mode auto-detects:**
* **Stdout is a TTY** → streaming NDJSON, line-by-line, flushed as the process emits it. You see `npm install`'s progress bar in real time.
* **Stdout is a pipe or `--json` is set** → buffered. The CLI prints a JSON object `{exit_code, stdout, stderr}` once the command finishes.
* **`BHATTI_FORCE_STREAM=1`** → forces streaming even when piping. Useful for CI logs that want progressive output.
**Cold sandboxes wake transparently.** If the sandbox is `cold` or `warm`, the server resumes it before executing — typically \~50ms for cold, sub-millisecond for warm. The wake is invisible to the caller; the command just takes a bit longer.
**Failures get recovery hints.** Common errors (sandbox not found, sandbox not running, name conflict, quota reached) are followed by a one-line suggestion in stderr. For example:
```text
404 Not Found: sandbox not found
Check sandbox name:
bhatti ls
```
## Examples
```bash
# Most basic
bhatti exec dev -- echo hello
```
```bash
# -- is optional
bhatti exec dev echo hello
```
```bash
# Pipe output to other tools — CLI switches to buffered mode automatically
bhatti exec dev -- cat /workspace/data.json | jq .name
```
```bash
# Shell expansion needs a shell — env vars in the *guest*, not in your terminal
bhatti exec dev -- bash -c 'echo $API_KEY'
```
```bash
# Cap a long-running command at 60s
bhatti exec dev --timeout 60 -- ./benchmark.sh
```
```bash
# Run in the background, return immediately with the guest PID
bhatti exec dev --detach -- ./worker.sh
```
```text
pid: 4218
output: /var/log/lohar/sessions/sess-7b3c.log
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--timeout <int>` | `300` | Exec timeout in seconds. The server kills the process group on timeout and returns exit code `137`. Hard max: `86400` (24 hours). | | `--detach` | `false` | Run the command in the background and return immediately with the guest PID and the path to the captured output file. The session continues until it finishes or is killed. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## Exit codes
\| Code | Meaning | | ---- | ------- | | `<n>` | The child process's exit code is forwarded verbatim. | | `137` | The child was killed (timeout exceeded, or session manually killed). | | `1` | CLI / API error before the command ran (sandbox not found, network, auth). |
## See also
* [`bhatti shell`](/v1/docs/reference/cli/exec/shell/) — interactive shell with reattach
* [`bhatti ps`](/v1/docs/reference/cli/exec/ps/) — see detached / running sessions
* [API: `POST /sandboxes/:id/exec`](/v1/docs/reference/api/#run-a-command)
# bhatti ports
> List listening TCP ports detected inside a sandbox, with proxy URLs for each.
List the listening TCP ports detected inside a sandbox.
## Synopsis
```
bhatti ports <sandbox>
```
## Description
The host periodically scans each running sandbox for listening ports and caches them. `bhatti ports` reads that cache and prints one row per port. Each row also shows the path-based proxy URL — a developer convenience for hitting the port from the host without a domain or `bhatti publish`.
The proxy URL goes through the API server at `<api>/sandboxes/<id>/proxy/<port>/...`. It requires the same Bearer auth as any other API call — useful for poking at services during development. For public, auth-free URLs, use [`bhatti publish`](/v1/docs/reference/cli/networking/publish/).
Cold sandboxes are not scanned (they have no listening processes by definition). To list ports for a sleeping sandbox, wake it first or just send a request that triggers a wake.
## Examples
```bash
bhatti ports dev
```
```text
PORT URL
3000 http://localhost:8080/sandboxes/dev/proxy/3000/
6379 http://localhost:8080/sandboxes/dev/proxy/6379/
```
```bash
# Quick test against a service inside the sandbox
bhatti ports dev --json | jq -r '.[] | .proxy_url' | head -1 | xargs curl -H "Authorization: Bearer $BHATTI_TOKEN"
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti publish`](/v1/docs/reference/cli/networking/publish/) — public auth-free URL for a port
* [API: `GET /sandboxes/:id/ports`](/v1/docs/reference/api/#list-listening-ports)
* [API: `ANY /sandboxes/:id/proxy/:port/*`](/v1/docs/reference/api/#reverse-proxy)
# bhatti ps
> List active sessions inside a sandbox — init scripts, shells, detached execs.
List active sessions inside a sandbox.
## Synopsis
```
bhatti ps <sandbox>
```
## Description
A *session* is a process the guest agent is tracking — typically because it has a PTY, was started detached, or was launched as the sandbox's `--init` script. Active sessions persist across client disconnections and show up here regardless of whether anyone is attached.
The `init` script (from `bhatti create --init "..."`) appears as a session named `init`. You can attach to it with [`bhatti shell`](/v1/docs/reference/cli/exec/shell/) (passing its session ID via the API) to watch progress.
Sessions without an attached client and without recent I/O are eventually garbage-collected by an idle timer. Active sessions (producing output or accepting input) are never killed by the timer.
## Examples
```bash
bhatti ps dev
```
```text
ID COMMAND RUNNING ATTACHED
init npm install && npm run dev true false
s1 /bin/zsh -li true true
s2 ./long-running-worker.sh true false
```
```bash
bhatti ps dev --json | jq '.[] | select(.attached == false) | .session_id'
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti shell`](/v1/docs/reference/cli/exec/shell/) — open or reattach to an interactive session
* [`bhatti exec --detach`](/v1/docs/reference/cli/exec/exec/) — start a session in the background
* [API: `GET /sandboxes/:id/sessions`](/v1/docs/reference/api/#list-sessions)
# bhatti shell
> Open an interactive shell inside a sandbox. Detach with Ctrl+\, reattach later, scrollback preserved.
Open an interactive terminal inside a sandbox. Aliased as `bhatti sh`.
## Synopsis
```
bhatti shell <sandbox> [--new]
bhatti sh <sandbox>
```
## Description
Opens a WebSocket connection to the sandbox's guest agent and attaches a PTY session. The default shell is `/bin/zsh`. Terminal resize is forwarded to the guest.
**Detach with `Ctrl+\`.** The shell keeps running; output flows into a 64 KB scrollback ring buffer in the guest. Reconnect with `bhatti shell <sandbox>` and you'll see the recent scrollback followed by live I/O. Detach does **not** send `SIGHUP` — child processes survive.
**Single attached client per session.** If you reattach from a second machine, the first client is disconnected. Use `--new` to force a brand-new session instead of reattaching.
**Clean exit vs. lost connection.** When the shell exits cleanly (Ctrl+D, `exit`), the CLI returns silently. Any other disconnection (network drop, server restart) shows:
```text
connection lost (session s3f9 still running)
reconnect: bhatti shell dev
```
The session keeps running in the guest. Reattach with the same command to pick it back up.
## Examples
```bash
bhatti shell dev
```
```bash
# Force a new session, even if one is already attached
bhatti shell dev --new
```
```bash
# Detach scenario:
bhatti shell dev # attach, run things, hit Ctrl+\
# ...session keeps running in the guest...
bhatti shell dev # reattach later, scrollback replayed
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--new` | `false` | Don't reattach to an existing session. Start a fresh one. The previous session, if any, is left running. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti exec`](/v1/docs/reference/cli/exec/exec/) — non-interactive command execution
* [`bhatti ps`](/v1/docs/reference/cli/exec/ps/) — list active sessions including this shell
* [`bhatti share`](/v1/docs/reference/cli/networking/share/) — browser-based shell URL
* [API: `GET /sandboxes/:id/ws`](/v1/docs/reference/api/#interactive-shell-websocket)
# File operations
> Read, write, and list files inside sandboxes.
Move bytes in and out of a sandbox without opening a shell. Useful for scripted setup, fetching results, and inspecting state.
| Command | Description |
| ---------------------------------------------------------- | ----------------------------------------------- |
| [`bhatti file read`](/v1/docs/reference/cli/files/read/) | Read a file from a sandbox to stdout. |
| [`bhatti file write`](/v1/docs/reference/cli/files/write/) | Write a file from stdin into a sandbox. Atomic. |
| [`bhatti file ls`](/v1/docs/reference/cli/files/ls/) | List directory contents inside a sandbox. |
The CLI exposes the simplest forms — read whole files, write whole files. The underlying API supports partial reads (`offset`, `limit`, `max_bytes`), permission control (`mode`), and HEAD-style stat requests; reach for it directly when you need those. See the [API reference](/v1/docs/reference/api/#files) for the full surface.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
bhatti file read dev /workspace/app.js
echo 'config' | bhatti file write dev /etc/config
bhatti file ls dev /workspace/
bhatti file read dev /workspace/data.json | jq .
```
## Limits
[Section titled “Limits”](#limits)
* Writes are capped at **100 MB** per request.
* Directory listings are capped at **10 000 entries**; a sentinel entry indicates truncation.
* Writes are atomic — temp file, fsync, rename. Concurrent readers never see a partial file.
# bhatti file ls
> List directory contents inside a sandbox.
List directory contents inside a sandbox.
## Synopsis
```
bhatti file ls <sandbox> <path>
```
## Description
Lists files and subdirectories at the given absolute path. The default human format prints one row per entry: a directory flag, mode, size in bytes, and name.
Listings are capped at **10 000 entries**. If the directory has more entries than that, a sentinel row indicates truncation. For listings with this many entries you almost certainly want to be running `find` inside the sandbox via [`bhatti exec`](/v1/docs/reference/cli/exec/exec/) and processing the results there instead.
## Examples
```bash
bhatti file ls dev /workspace
```
```text
drwxr-xr-x 4096 node_modules
-rw-r--r-- 1234 app.js
-rw-r--r-- 256 package.json
-rw-r--r-- 512 README.md
```
```bash
# JSON for scripting
bhatti file ls dev /workspace --json | jq '.[] | select(.is_dir == false) | .name'
```
## Output fields (JSON)
```json
[
{"name": "app.js", "size": 1234, "mode": "0644", "is_dir": false, "mtime": 1761792345},
{"name": "node_modules", "size": 4096, "mode": "0755", "is_dir": true, "mtime": 1761792500}
]
```
`mtime` is a Unix timestamp in seconds.
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti file read`](/v1/docs/reference/cli/files/read/) / [`bhatti file write`](/v1/docs/reference/cli/files/write/)
* [API: `GET /sandboxes/:id/files?ls=true`](/v1/docs/reference/api/#list-directory)
# bhatti file read
> Read a file from a sandbox to stdout.
Read a file from a sandbox and write its bytes to stdout.
## Synopsis
```
bhatti file read <sandbox> <path>
```
## Description
Streams the file's contents to stdout, byte-for-byte. The path must be absolute. Cold sandboxes wake transparently before the read.
The CLI always reads the whole file. The API supports partial reads (`offset`, `limit`, `max_bytes` query params) for tailing logs without transferring the entire file — call it directly when you need that. Every response includes an `X-File-Size` header so clients can detect truncation. See [API: files](/v1/docs/reference/api/#files).
## Examples
```bash
bhatti file read dev /workspace/app.js
```
```bash
# Pipe to other tools
bhatti file read dev /workspace/data.json | jq .name
```
```bash
# Save to a local file
bhatti file read dev /var/log/agent.log > agent-$(date +%F).log
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti file write`](/v1/docs/reference/cli/files/write/) — opposite direction
* [`bhatti file ls`](/v1/docs/reference/cli/files/ls/) — list a directory
* [API: `GET /sandboxes/:id/files`](/v1/docs/reference/api/#read-a-file)
# bhatti file write
> Write a file into a sandbox from stdin. Atomic, capped at 100 MB.
Write a file into a sandbox. Reads bytes from stdin and writes them to the absolute `<path>` inside the sandbox.
## Synopsis
```
bhatti file write <sandbox> <path>
```
## Description
Writes are atomic — the guest agent writes to a temp file, fsyncs it, then renames it over the target. Concurrent readers never see partial content.
Writes are capped at **100 MB** per request. For larger files, split into chunks or attach a [persistent volume](/v1/docs/reference/cli/volumes/) instead.
The CLI doesn't expose `--mode` for setting the file permissions (defaults to `0644`). The underlying [API endpoint](/v1/docs/reference/api/#write-a-file) accepts a `mode=0755` query parameter — use it directly when you need to set a specific mode (e.g. `0700` for keys, `0755` for executables).
## Examples
```bash
echo 'console.log("hello")' | bhatti file write dev /workspace/app.js
```
```bash
# From a local file
bhatti file write dev /workspace/config.json < ./config.json
```
```bash
# Generate and write inline
date | bhatti file write dev /tmp/last-deploy.txt
```
```bash
# Set executable bit via the API directly
local_token="$BHATTI_TOKEN"
curl -X PUT "http://localhost:8080/sandboxes/dev/files?path=/usr/local/bin/run.sh&mode=0755" \
-H "Authorization: Bearer $local_token" \
--data-binary @./run.sh
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti file read`](/v1/docs/reference/cli/files/read/) — opposite direction
* [`bhatti create --file local:guest`](/v1/docs/reference/cli/sandbox/create/) — inject a file at boot, no separate write call
* [API: `PUT /sandboxes/:id/files`](/v1/docs/reference/api/#write-a-file)
# Images
> Manage rootfs images. Pull from public OCI registries, import from Docker, save sandboxes as reusable images.
Images are ext4 filesystem snapshots used as sandbox root filesystems. bhatti ships pre-built tier images (`minimal`, `browser`, `docker`, `computer`); you can also pull from public OCI registries, import from Docker, or save a running sandbox’s filesystem as an image.
| Command | Description |
| ---------------------------------------------------------------- | ---------------------------------------------------------- |
| [`bhatti image list`](/v1/docs/reference/cli/images/list/) | List available images. |
| [`bhatti image pull`](/v1/docs/reference/cli/images/pull/) | Pull from a public OCI registry. Async; returns a task ID. |
| [`bhatti image import`](/v1/docs/reference/cli/images/import/) | Import a local Docker image, or a tarball with `--tar`. |
| [`bhatti image save`](/v1/docs/reference/cli/images/save/) | Save a running sandbox’s rootfs as a reusable image. |
| [`bhatti image delete`](/v1/docs/reference/cli/images/delete/) | Delete an image. |
| [`bhatti image share`](/v1/docs/reference/cli/images/share/) | Share an image with specific users (server-only). |
| [`bhatti image unshare`](/v1/docs/reference/cli/images/unshare/) | Revoke image access from users (server-only). |
## Built-in tiers
[Section titled “Built-in tiers”](#built-in-tiers)
| Tier | What’s in it | Size |
| ---------- | --------------------------------- | -------- |
| `minimal` | Bare Ubuntu 24.04 + curl + fuse3 | \~200 MB |
| `browser` | + Chromium, Playwright, Node 22 | \~600 MB |
| `docker` | + Docker Engine | \~550 MB |
| `computer` | + XFCE desktop, KasmVNC, Chromium | \~1.5 GB |
```bash
bhatti create --name scraper --image browser
bhatti create --name ci --image docker
```
The server auto-discovers tiers from `/var/lib/bhatti/images/`. Install more with `sudo bhatti update --tiers all`.
## Image scoping
[Section titled “Image scoping”](#image-scoping)
Each image is owned by the user who created it (or by the system, for built-in tiers). Other users can’t see or use your custom images by default.
To share with a specific user: [`sudo bhatti image share <image> --user alice`](/v1/docs/reference/cli/images/share/). The command operates directly on the local SQLite database, so it requires running on the server with DB access — it isn’t an API call you can make remotely.
Use `--list` to inspect current shares; pass `--user alice --user bob` to share with multiple users in one call. [`unshare`](/v1/docs/reference/cli/images/unshare/) revokes access by user.
# bhatti image delete
> Delete an image. Cannot be undone.
Delete an image.
## Synopsis
```
bhatti image delete <name> [-y]
```
## Description
Removes the image record and the underlying ext4 file. **Cannot be undone.**
Sandboxes that were created from this image keep running — they have their own copy-on-write rootfs derived from the image at creation time. They don't reference the original after boot.
By default, `delete` prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
You cannot delete a system tier image (`minimal`, `browser`, etc.) — those are owned by `user_id=""` and only the operator can remove them by removing the file from `/var/lib/bhatti/images/`.
## Examples
```bash
bhatti image delete python-3.12
```
```text
Delete image "python-3.12"? [y/N]: y
deleted
```
```bash
# Non-interactive
bhatti image delete python-3.12 -y
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti image list`](/v1/docs/reference/cli/images/list/)
* [API: `DELETE /images/:name`](/v1/docs/reference/api/#delete-an-image)
# bhatti image import
> Import a local Docker image (via docker save) or a raw tarball.
Import an image from a local Docker daemon, or directly from a tarball without Docker.
## Synopsis
```
bhatti image import <docker-ref> [--name <string>]
bhatti image import --tar <path> --name <string>
```
## Description
Two modes:
**Docker mode** (default). The CLI runs `docker save <ref>` on your machine and streams the resulting tarball to the bhatti server, which converts it to an ext4 rootfs. Requires Docker installed locally. The image must already be in your local Docker daemon — pull it first with `docker pull` if needed.
**Tarball mode** (`--tar`). Skip Docker entirely. The CLI reads the file at `<path>` and streams it to the server. The tarball must be in the format produced by `docker save` (an OCI layout or v1 image manifest). `--name` is required in this mode since there's no ref to derive the name from.
If you don't pass `--name` in Docker mode, a name is derived from the ref the same way as `image pull`: `python:3.12` → `python-3.12`.
For private registries, this is the recommended path — `docker pull` handles authentication on your machine using your existing Docker credentials, then bhatti just receives the bytes:
```bash
docker pull ghcr.io/org/private:latest
bhatti image import ghcr.io/org/private:latest
```
## Examples
```bash
# Import from local Docker (name derived from ref)
bhatti image import python:3.12
```
```bash
# Private image: pull with Docker first, then import
docker pull ghcr.io/org/private:latest
bhatti image import ghcr.io/org/private:latest
```
```bash
# Locally-built image
docker build -t my-env .
bhatti image import my-env
```
```bash
# Custom name override
bhatti image import python:3.12 --name py312
```
```bash
# Raw tarball — no Docker needed
docker save ubuntu:24.04 -o /tmp/ubuntu.tar
bhatti image import --tar /tmp/ubuntu.tar --name ubuntu-24
```
## Failure modes
* **Docker not found**: install Docker, or use `--tar` with a tarball.
* **Image not in local Docker**: `docker pull <ref>` first, or use `--tar` with a saved tarball.
* **Server rejects the upload**: image is too large or malformed; check stderr for the server message.
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | derived from ref (Docker mode); required (`--tar` mode) | Image name to register on the server. | | `--tar <path>` | — | Skip Docker; stream the given tarball directly. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti image pull`](/v1/docs/reference/cli/images/pull/) — server-side pull from public registries (no Docker needed)
* [`bhatti image save`](/v1/docs/reference/cli/images/save/) — save a sandbox's filesystem as an image
* [API: `POST /images/import`](/v1/docs/reference/api/#import-an-image)
# bhatti image list
> List all rootfs images available to the authenticated user.
List all rootfs images available to the authenticated user.
## Synopsis
```
bhatti image list
```
## Description
Lists images in three scopes:
* **System tiers** (`minimal`, `browser`, `docker`, `computer`) — visible to everyone.
* **Your own images** — created via `image pull`, `image import`, or `image save`.
* **Images shared with you** by another user.
## Examples
```bash
bhatti image list
```
```text
ID NAME SOURCE SIZE SCOPE
img_abc123 minimal built-in 186MB admin
img_def456 browser built-in 612MB admin
img_ghi789 python-3.12 oci:python:3.12 284MB user
img_jkl012 my-stack snapshot:dev 512MB user
```
```bash
# JSON for scripting
bhatti image list --json | jq '.[] | select(.user_id != "") | .name'
```
## Output fields (JSON)
```json
[
{"id": "img_abc123", "name": "minimal", "source": "built-in", "size_mb": 186, "user_id": ""},
{"id": "img_jkl012", "name": "my-stack", "source": "snapshot:dev", "size_mb": 512, "user_id": "usr_..."}
]
```
`user_id` is empty for system images; non-empty values denote ownership.
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti image pull`](/v1/docs/reference/cli/images/pull/) / [`import`](/v1/docs/reference/cli/images/import/) / [`save`](/v1/docs/reference/cli/images/save/)
* [API: `GET /images`](/v1/docs/reference/api/#list-images)
# bhatti image pull
> Pull a public OCI/Docker image and convert it to a bhatti rootfs.
Pull a public image from any OCI-compatible registry. The server fetches the image, flattens its layers, and writes them to an ext4 file usable as a sandbox rootfs.
## Synopsis
```
bhatti image pull <ref> [--name <string>] [--auth <user:token>]
```
## Description
The pull runs asynchronously on the server. The CLI returns a task ID, then polls `/tasks/<id>` every 2 seconds and prints progress until the task completes or fails. **Press `Ctrl+C` to detach** — the pull continues on the server. Check progress later with `bhatti image list` (the image appears once it's ready).
If you don't pass `--name`, a name is derived from the ref:
* `python:3.12` → `python-3.12`
* `node:22-slim` → `node-22-slim`
* `ghcr.io/myorg/sandbox-base:latest` → `sandbox-base-latest`
`--auth` enables a single-shot HTTP basic auth header (`user:token`). For real private registries with rotating credentials, prefer the [`image import`](/v1/docs/reference/cli/images/import/) flow — `docker pull` handles auth on your machine, then bhatti just receives the tarball.
If the same image was previously pulled with the same content digest, the server returns immediately without re-downloading.
## Examples
```bash
bhatti image pull python:3.12
```
```text
pulling python:3.12 as "python-3.12" (task: tsk_abc123)
resolving manifest...
downloading layers (45 MB / 142 MB)...
flattening to ext4...
done
```
```bash
# Custom name
bhatti image pull node:22-slim --name node-22
# Detach: press Ctrl+C after seeing the task ID
bhatti image pull ubuntu:24.04
^C
Interrupted. Pull continues on server.
Check status: bhatti image list
```
```bash
# Use the pulled image
bhatti create --name py --image python-3.12
```
## Authentication
For private registries, `--auth` works for simple `user:token` schemes. If the pull fails with `unauthorized` / `403`, the CLI prints a hint suggesting you use `image import` instead:
```text
This image may require authentication.
Pull it with Docker locally, then import:
docker pull ghcr.io/org/private:latest
bhatti image import ghcr.io/org/private:latest
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | derived from ref | Image name to register on the server. | | `--auth <user:token>` | — | HTTP basic auth credentials for the registry. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti image import`](/v1/docs/reference/cli/images/import/) — local Docker / tarball import (handles private registries via local auth)
* [`bhatti create --image`](/v1/docs/reference/cli/sandbox/create/) — use the pulled image
* [API: `POST /images/pull`](/v1/docs/reference/api/#pull-an-image)
* [API: `GET /tasks/:id`](/v1/docs/reference/api/#poll-an-async-task)
# bhatti image save
> Save a running sandbox's rootfs as a reusable image.
Snapshot a running sandbox's filesystem and save it as a reusable image.
## Synopsis
```
bhatti image save <sandbox> --name <image-name>
```
## Description
Captures the current state of the sandbox's rootfs (every file written since boot) and writes it to a new ext4 image owned by the user. The sandbox keeps running; the save is read-only from its perspective.
Only the rootfs is captured — attached persistent volumes are **not** included. If your sandbox stores work in a volume, save the volume separately with [`bhatti volume clone`](/v1/docs/reference/cli/volumes/clone/) and recreate the attachment when stamping out new sandboxes from the saved image.
The saved image is private to the user. To make it available to other users, use [`bhatti image share`](/v1/docs/reference/cli/images/share/) (server-only).
## Examples
```bash
# Build a custom environment, then save it
bhatti create --name build --image minimal
bhatti exec build -- apt-get update && apt-get install -y nodejs
bhatti exec build -- npm install -g pnpm
bhatti image save build --name node-app
# saved "node-app" (512MB)
```
```bash
# Stamp out workers from the saved image
bhatti create --name worker-1 --image node-app
bhatti create --name worker-2 --image node-app
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | (required) | Name for the saved image. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume clone`](/v1/docs/reference/cli/volumes/clone/) — for snapshotting volume data alongside an image
* [`bhatti snapshot create`](/v1/docs/reference/cli/snapshots/create/) — full VM snapshot (memory + disk + processes), not just filesystem
* [API: `POST /sandboxes/:id/save-image`](/v1/docs/reference/api/#save-a-sandbox-as-an-image)
# bhatti image share
> Share an image with specific users. Operates directly on the local SQLite database; requires DB access (server-only).
# bhatti image share server only
Share an image with one or more specific users.
## Synopsis
```
bhatti image share <image-name> --user <name> [--user <name>]...
bhatti image share <image-name> --list
```
## Description
Adds an entry to the image-share table for each `<user>` so that user can reference the image with `--image <name>` from their sandbox creation calls. Sharing is per-user; there's no "public" or "everyone" mode.
The command operates directly on the server's SQLite database — it isn't an HTTP API call. You must run it on the server, with access to `/var/lib/bhatti/state.db` (typically as `sudo`).
`--list` prints the users an image is currently shared with. `--unshare` is its own command, [`bhatti image unshare`](/v1/docs/reference/cli/images/unshare/).
## Examples
```bash
# Share with two users
sudo bhatti image share spc-golden --user kowshik --user sumo
```
```text
shared "spc-golden" with: kowshik, sumo
```
```bash
# Inspect current shares
sudo bhatti image share spc-golden --list
```
```text
spc-golden shared with: kowshik, sumo
```
```bash
# Image not yet shared
sudo bhatti image share spc-golden --list
```
```text
spc-golden: not shared
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--user <name>` | — | User to share with. Repeatable. Required unless `--list`. | | `--list` | `false` | Print current shares for the image instead of adding new ones. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## Failure modes
* **Image not found**: bhatti can't see your image because of name typos or because it's owned by a different user. Check with `bhatti image list`.
* **User not found**: the `<name>` doesn't match any user. Check with `bhatti user list`.
* **No `--user`**: the command refuses to do anything without an explicit user list (or `--list`). There's no global "share with all" mode.
## See also
* [`bhatti image unshare`](/v1/docs/reference/cli/images/unshare/) — revoke access
* [`bhatti user list`](/v1/docs/reference/cli/admin/user-list/) — find usernames
# bhatti image unshare
> Revoke image access from specific users. Server-only.
# bhatti image unshare server only
Revoke an image's share entries for one or more users.
## Synopsis
```
bhatti image unshare <image-name> --user <name> [--user <name>]...
```
## Description
Removes share-table entries for the listed users. After unsharing, those users will no longer see the image in [`bhatti image list`](/v1/docs/reference/cli/images/list/) and can't reference it by name in `bhatti create --image`.
Sandboxes the affected users already created from the image keep running — they have their own copy-on-write rootfs and don't reference the original.
Like [`image share`](/v1/docs/reference/cli/images/share/), this operates directly on the local SQLite database. Requires DB access on the server (typically `sudo`).
## Examples
```bash
sudo bhatti image unshare spc-golden --user sumo
```
```text
unshared "spc-golden" from: sumo
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--user <name>` | (required) | User to revoke. Repeatable. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti image share`](/v1/docs/reference/cli/images/share/) — add a share
* [`bhatti image share --list`](/v1/docs/reference/cli/images/share/) — see who has access
# Networking & sharing
> Publish ports with public URLs, share browser-based shells.
Expose sandbox services to the world (or a teammate) without an API key on the receiver’s side.
| Command | Description |
| ------------------------------------------------------------------ | --------------------------------------------------------------------------------- |
| [`bhatti publish`](/v1/docs/reference/cli/networking/publish/) | Publish a sandbox port at a public URL. Cold sandboxes wake on the first request. |
| [`bhatti unpublish`](/v1/docs/reference/cli/networking/unpublish/) | Remove a published port. |
| [`bhatti share`](/v1/docs/reference/cli/networking/share/) | Generate a one-shot URL that opens an interactive terminal in the browser. |
Both `publish` and `share` produce URLs that don’t require an API key. Treat them like any other secret — anyone with the URL has access.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
# Publish on a friendly subdomain
bhatti publish dev -p 3000 -a my-app
# → https://my-app.bhatti.sh
# Auto-generated alias (random suffix prevents URL guessing)
bhatti publish dev -p 3000
# → https://dev-k3m9x2.bhatti.sh
# Publish + give a teammate a browser shell in one call
bhatti publish dev -p 3000 -a demo --shell
# Share a sandbox shell over the web
bhatti share dev
# → https://api.bhatti.sh/_shell/<id>#token=...
# Revoke any active share link
bhatti share dev --revoke
# Tear down a published port
bhatti unpublish dev -p 3000
```
## Domain mode vs. path-based
[Section titled “Domain mode vs. path-based”](#domain-mode-vs-path-based)
Public URLs at `https://<alias>.<your-zone>` only work when the server is configured with a [custom domain](/v1/docs/managing/custom-domain/). Without one, `bhatti publish` falls back to a path-based proxy:
```text
http://<api-host>/sandboxes/<id>/proxy/<port>/
```
That URL still works, but it requires Bearer auth. Useful for development; not suitable for sharing publicly. See [`bhatti publish`](/v1/docs/reference/cli/networking/publish/) for the exact fallback behaviour.
# bhatti publish
> Publish a sandbox port with a public URL. Cold sandboxes wake on first request.
Publish a sandbox port at a public URL.
## Synopsis
```
bhatti publish <sandbox> -p <port> [-a <alias>] [--shell]
```
## Description
Creates a routing rule on the bhatti server: incoming requests to `https://<alias>.<proxy-zone>` get reverse-proxied to `localhost:<port>` inside the sandbox.
**Aliases.** With `-a my-app`, the URL is `https://my-app.<proxy-zone>` exactly. Without `-a`, an alias is generated from the sandbox name with a random suffix (`dev-k3m9x2`) to prevent URL guessing — useful for ephemeral previews.
**Cold sandboxes wake automatically.** The first request after a sandbox went cold takes \~50ms longer than usual; subsequent requests are full speed.
**No auth.** Published URLs are public. Anyone with the URL can reach the port. There's no per-URL access control — share carefully.
**HTTP and WebSocket.** Both work through the proxy. WebSocket idle timeout is 10 minutes; HTTP body cap is 50 MB per request.
**`--shell` combo.** Pass `--shell` to also generate a [web shell token](/v1/docs/reference/cli/networking/share/) in the same call. Useful for "send a teammate a preview URL and a debug terminal in one click".
**Without a custom domain configured.** If the server isn't running in [domain mode](/v1/docs/managing/custom-domain/), there's no `<alias>.<zone>` URL. The CLI prints a fallback hint:
```text
Publishing requires a custom domain.
Setup: https://bhatti.sh/docs/managing/custom-domain
In the meantime, use the proxy URL directly:
http://<api-host>/sandboxes/<id>/proxy/<port>/
```
The proxy URL works but requires Bearer auth, so it's developer-only.
## Examples
```bash
# Friendly subdomain
bhatti publish dev -p 3000 -a my-app
# Published: https://my-app.bhatti.sh
```
```bash
# Auto-generated alias
bhatti publish dev -p 3000
# Published: https://dev-k3m9x2.bhatti.sh
```
```bash
# Preview URL + browser shell, single call
bhatti publish dev -p 3000 -a demo --shell
# Published: https://demo.bhatti.sh
# Shell: https://api.bhatti.sh/_shell/<id>#token=...
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-p`, `--port <int>` | (required) | Port inside the sandbox to publish. | | `-a`, `--alias <string>` | auto-generated | Custom subdomain alias. Without this, an alias derived from the sandbox name with a random suffix is used. | | `--shell` | `false` | Also generate a [web shell URL](/v1/docs/reference/cli/networking/share/) in the same call. The shell URL is printed alongside the published URL. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti unpublish`](/v1/docs/reference/cli/networking/unpublish/) — remove a published port
* [`bhatti share`](/v1/docs/reference/cli/networking/share/) — web shell URL on its own
* [Custom domain](/v1/docs/managing/custom-domain/) — required for `<alias>.<zone>` URLs
* [API: `POST /sandboxes/:id/publish`](/v1/docs/reference/api/#publish-a-port)
# bhatti share
> Generate a one-shot URL that opens an interactive terminal in the browser.
Generate a URL that opens an interactive terminal in the browser. No CLI install or API key needed on the receiver's side.
## Synopsis
```
bhatti share <sandbox> [--revoke]
```
## Description
Mints a fresh token bound to the sandbox and embeds it in the URL fragment (`#token=...`). The fragment is **not** sent to the server in HTTP request lines, so the token doesn't appear in access logs or referer headers — but anyone with the URL has shell access until the token is revoked.
**Each call invalidates the previous token.** Calling `bhatti share dev` again generates a new URL; the old one stops working. Only one share link is active at a time per sandbox.
**Scope.** A share token grants access *only* to the specific sandbox it was minted for. It is not interchangeable with the user's API key.
**Behaviour in the browser.** The page opens an xterm.js terminal connected via WebSocket; it's a full PTY session, identical to [`bhatti shell`](/v1/docs/reference/cli/exec/shell/). Closing the tab detaches; the session keeps running and scrollback is preserved. Reopening the URL reattaches.
**Revoke** with `--revoke` — the token is deleted server-side and the URL stops working.
## Examples
```bash
bhatti share dev
```
```text
Shell: https://api.bhatti.sh/_shell/a1b2c3d4#token=k3m9x2qr...
```
```bash
# JSON for scripting
bhatti share dev --json
```
```json
{
"url": "https://api.bhatti.sh/_shell/a1b2c3d4#token=k3m9x2qr...",
"token": "k3m9x2qr..."
}
```
```bash
# Revoke
bhatti share dev --revoke
```
```text
Shell access revoked.
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--revoke` | `false` | Delete the active token for this sandbox. The shared URL stops working immediately. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti publish --shell`](/v1/docs/reference/cli/networking/publish/) — preview URL + share URL in one call
* [`bhatti shell`](/v1/docs/reference/cli/exec/shell/) — terminal-based interactive shell
* [`bhatti shell`](/v1/docs/reference/cli/exec/shell/) — the CLI counterpart
* [API: `POST /sandboxes/:id/shell-token`](/v1/docs/reference/api/#mint-a-shell-token)
# bhatti unpublish
> Remove a published port. The URL stops working immediately.
Remove a published port. The URL stops working immediately.
## Synopsis
```
bhatti unpublish <sandbox> -p <port>
```
## Description
Deletes the routing rule. New requests to the previously published URL get a 404 from the proxy. In-flight WebSocket connections continue until the client closes them.
Publish rules are also automatically cleaned up when a sandbox is destroyed — `unpublish` is only needed when you want to keep the sandbox but stop exposing the port.
## Examples
```bash
bhatti unpublish dev -p 3000
```
```text
Unpublished port 3000
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-p`, `--port <int>` | (required) | Port to unpublish. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti publish`](/v1/docs/reference/cli/networking/publish/)
* [API: `DELETE /sandboxes/:id/publish/:port`](/v1/docs/reference/api/#unpublish-a-port)
# Sandbox commands
> Lifecycle commands for sandbox VMs — create, list, inspect, edit, stop, start, destroy.
Sandboxes are isolated Linux VMs. These commands cover the full lifecycle: create them, look at them, change mutable settings, pause and resume them on disk, and tear them down.
| Command | Description |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| [`bhatti create`](/v1/docs/reference/cli/sandbox/create/) | Create a new sandbox VM. Optional flags cover CPU, memory, env vars, init scripts, volumes, secrets, and file injection. |
| [`bhatti list`](/v1/docs/reference/cli/sandbox/list/) | List your sandboxes (alias: `ls`). `-o wide` adds resource columns. |
| [`bhatti inspect`](/v1/docs/reference/cli/sandbox/inspect/) | Show full details for one sandbox (alias: `info`) — kubectl-`describe` style. |
| [`bhatti edit`](/v1/docs/reference/cli/sandbox/edit/) | Toggle mutable settings. Currently only `--keep-hot` / `--allow-cold`. |
| [`bhatti stop`](/v1/docs/reference/cli/sandbox/stop/) | Snapshot to disk and free memory. |
| [`bhatti start`](/v1/docs/reference/cli/sandbox/start/) | Resume from snapshot. Continues exactly where it left off. |
| [`bhatti destroy`](/v1/docs/reference/cli/sandbox/destroy/) | Permanently destroy a sandbox (alias: `rm`). Volumes are detached, not deleted. |
All commands accept either the sandbox name or its ID. Names are easier; IDs are stable across renames.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
bhatti create --name dev --cpus 2 --memory 1024
bhatti ls -o wide
bhatti inspect dev
bhatti exec dev -- npm install
bhatti stop dev # save state, free memory
bhatti start dev # resume from where it stopped
bhatti destroy dev -y
```
In normal operation you don’t need `stop` and `start` — idle sandboxes are paused and resumed automatically. See [Thermal states](/v1/docs/under-the-hood/thermal-states/) for the engineering. Use them to control thermal state explicitly when you know a sandbox will be idle for a while.
## See also
[Section titled “See also”](#see-also)
* [Execution & shells](/v1/docs/reference/cli/exec/) — `exec`, `shell`, `ps`, `ports`
* [Files](/v1/docs/reference/cli/files/) — read/write files inside a sandbox
* [Networking & sharing](/v1/docs/reference/cli/networking/) — `publish`, `share`
* [API: `POST /sandboxes`](/v1/docs/reference/api/#create-a-sandbox)
# bhatti create
> Create a new sandbox VM with custom resources, environment, init script, volumes, and secrets.
Create a new sandbox VM. Each sandbox is an isolated Linux environment with its own kernel, filesystem, and network interface — created in seconds, paused for free, resumed in microseconds.
## Synopsis
```
bhatti create [flags]
```
## Description
Boots a Firecracker microVM and registers it under the authenticated user. Returns once the VM is reachable on its private IP and the guest agent is responding.
The created sandbox is in the `running` (hot) thermal state. Without `--keep-hot`, it pauses to *warm* after 30 seconds of inactivity and freezes to *cold* (memory snapshotted to disk, RAM freed) after 30 minutes. Any incoming request transparently wakes it.
`--volume` and `--secret` reference resources that must already exist (create them with [`bhatti volume create`](/v1/docs/reference/cli/volumes/create/) and [`bhatti secret set`](/v1/docs/reference/cli/secrets/set/)). `--file` reads a local file at submit time and writes its bytes to the guest path during boot — convenient for injecting config files without a volume.
When used with `--template`, request-side `--secret` and `--file` are merged into the template's defaults: secrets from both lists are resolved (request-side names that don't exist in your secret store return `400`); `--file` adds to the template (templates have no files of their own). For environment variables, `--secret` wins over `--env` for the same name.
`--cpus` accepts fractional values (e.g. `0.5`, `1.5`); the host scheduler runs vCPUs as time-shared threads, so fractional allocation maps to "you're entitled to \~N vCPU-seconds per second."
`--hugepages` uses 2MB hugepages for the VM's memory. Boot is faster but disables diff snapshots — every snapshot is a full memory dump. Use it for short-lived sandboxes that don't snapshot, not for long-lived agents.
The `--memory` server default is 1024 MB. The `--cpus` default is 1. Both are capped by the user's per-sandbox limits set via [`bhatti user create`](/v1/docs/reference/cli/admin/user-create/).
## Examples
```bash
# Smallest possible sandbox
bhatti create --name dev
```
```bash
# Custom resources, env vars, and an init script
bhatti create --name api \
--cpus 2 --memory 2048 \
--env NODE_ENV=production,LOG_LEVEL=info \
--init "cd /workspace && npm install"
```
```bash
# Mount a persistent volume; create the volume first
bhatti volume create --name workspace --size 5120
bhatti create --name dev --volume workspace:/workspace
```
```bash
# Reference a stored secret as a guest env var
bhatti secret set OPENAI_KEY sk-...
bhatti create --name agent --secret OPENAI_KEY
```
```bash
# Inject a local config file at boot
bhatti create --name worker --file ./config.json:/etc/worker/config.json
```
```bash
# Use a non-default rootfs image (browser tier with Chromium + Playwright)
bhatti create --name scraper --image browser
```
```bash
# Autonomous agent — never paused, even when idle
bhatti create --name agent --init "hermes gateway" --keep-hot
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | auto-generated | Sandbox name. Must match `[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}`. | | `--cpus <float>` | `1` | Number of vCPUs. Fractional values allowed (e.g. `0.5`). Capped by per-sandbox limit. | | `--memory <int>` | `0` (server uses `1024`) | Memory in MB. `0` means use the server default. Capped by per-sandbox limit. | | `--disk-size <int>` | `0` (use image size) | Rootfs disk size in MB. `0` means inherit from the base image. | | `--env <K=V,K=V>` | — | Environment variables, comma-separated, applied to every command. | | `--init <string>` | — | Init script that runs at boot as an attachable session named `init`. Survives the boot path; visible in [`bhatti ps`](/v1/docs/reference/cli/exec/ps/). | | `--keep-hot` | `false` | Disable thermal transitions. Use for sandboxes that maintain external connections (websockets, polling agents). | | `--hugepages` | `false` | Use 2MB hugepages. Faster boot, no diff snapshots — full snapshot every time. | | `--template <name>` | — | Create from a template. Template fields are defaults; flags here override them. | | `--image <name>` | — | Rootfs image. Built-in: `minimal`, `browser`, `docker`, `computer`. Or a name from [`bhatti image list`](/v1/docs/reference/cli/images/list/). | | `--volume <spec>` | — | Attach a persistent volume. Format: `name:mount[:ro]`. Repeatable. | | `--secret <name>` | — | Reference a stored secret by name. Decrypted and exposed as an env var. Repeatable. | | `--file <local:guest>` | — | Read a local file and write it to `guest` inside the sandbox at boot. Repeatable. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## Output
Default output is a verbose summary:
```
sandbox/dev created (1 vCPU, 1024 MB)
IP: 192.168.137.42
Shell: bhatti shell dev
```
With `--json`, the full sandbox object is printed.
If a sandbox with the requested name already exists *and is identical*, the server returns the existing record and the CLI prints:
```
sandbox/dev unchanged (already exists)
```
This makes `create` safe to re-run from scripts.
## Exit codes
\| Code | Meaning | | ---- | ------- | | `0` | Sandbox created, or already existed with the same configuration. | | `1` | Validation error, quota exceeded (e.g. `max-sandboxes` reached), name conflict, or boot failure. |
## See also
* [`bhatti list`](/v1/docs/reference/cli/sandbox/list/) — find your sandboxes
* [`bhatti exec`](/v1/docs/reference/cli/exec/exec/) — run a command in the new sandbox
* [`bhatti shell`](/v1/docs/reference/cli/exec/shell/) — open an interactive shell
* [`bhatti destroy`](/v1/docs/reference/cli/sandbox/destroy/) — clean up
* [API: `POST /sandboxes`](/v1/docs/reference/api/#create-a-sandbox)
# bhatti destroy
> Permanently destroy a sandbox and all its data.
Permanently destroy a sandbox. Aliased as `bhatti rm`.
## Synopsis
```
bhatti destroy <sandbox> [-y]
bhatti rm <sandbox> [-y]
```
## Description
Kills the Firecracker process, removes the rootfs and snapshot files, releases the IP, deletes the TAP device, and removes the sandbox record from the database. **This cannot be undone.**
Persistent volumes attached to the sandbox are detached but **not** deleted — their data survives. Use [`bhatti volume delete`](/v1/docs/reference/cli/volumes/delete/) to remove a volume. Published URLs are automatically cleaned up.
By default, destroy prompts for confirmation. In a non-interactive shell without `-y`, the command refuses to proceed and exits `1`. Pass `-y` / `--yes` to skip the prompt.
## Examples
```bash
bhatti destroy dev
```
```text
Destroy sandbox "dev"? [y/N]: y
sandbox/dev destroyed
```
```bash
# Non-interactive (CI / scripts)
bhatti destroy dev -y
```
```bash
# Tear down everything you own
bhatti list --json | jq -r '.[].name' | xargs -I{} bhatti destroy {} -y
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip the interactive confirmation. Required for non-interactive use. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## Exit codes
\| Code | Meaning | | ---- | ------- | | `0` | Destroyed. | | `1` | Sandbox not found, or non-interactive shell without `-y`. |
## See also
* [`bhatti stop`](/v1/docs/reference/cli/sandbox/stop/) — pause without destroying (data preserved)
* [`bhatti volume delete`](/v1/docs/reference/cli/volumes/delete/) — detached volumes don't go with the sandbox
* [API: `DELETE /sandboxes/:id`](/v1/docs/reference/api/#destroy-a-sandbox)
# bhatti edit
> Update mutable settings on a sandbox. Currently keep-hot only.
Update mutable settings on a running or stopped sandbox.
## Synopsis
```
bhatti edit <sandbox> --keep-hot
bhatti edit <sandbox> --allow-cold
```
## Description
Currently the only mutable setting is the thermal-management toggle. Resource fields like `cpus` and `memory_mb` are immutable after creation — destroy and re-create instead.
Setting `--keep-hot` on a stopped sandbox immediately wakes it, since "kept hot" implies running. Setting `--allow-cold` doesn't pause the sandbox; it just re-enables the thermal manager's automatic transitions.
`--keep-hot` and `--allow-cold` are mutually exclusive.
## Examples
```bash
# Pin an autonomous agent so it's never paused
bhatti edit my-agent --keep-hot
```
```bash
# Re-enable automatic thermal transitions
bhatti edit my-agent --allow-cold
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--keep-hot` | `false` | Disable thermal transitions. Wakes the sandbox if it was stopped. | | `--allow-cold` | `false` | Re-enable thermal transitions. Does not actively pause the sandbox. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti create --keep-hot`](/v1/docs/reference/cli/sandbox/create/) — set this at creation
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — how hot/warm/cold transitions work
* [API: `PATCH /sandboxes/:id`](/v1/docs/reference/api/#update-a-sandbox)
# bhatti inspect
> Show full details for a single sandbox. Aliased as bhatti info.
Show full details for one sandbox in `kubectl describe`-style output. Aliased as `bhatti info`.
## Synopsis
```
bhatti inspect <sandbox>
bhatti info <sandbox>
```
## Description
Prints status, thermal state, image, creation/stop timestamps, resource allocation, network info, and `keep_hot`. For sandboxes in the `running` state, disk usage is queried live by running `df -m /` inside the guest — the row shows total / used / free in MB.
Stopped sandboxes show their static disk size; the live `df` query is skipped to avoid waking them.
Use `--json` to get the full sandbox record as JSON for scripting.
## Examples
```bash
bhatti inspect dev
```
```text
Name: dev
ID: a1b2c3d4
Status: running
Thermal: hot
Image: minimal
Created: 2026-04-29T15:21:48Z
Resources:
CPUs: 2
Memory: 1024 MB
Disk: 8192 MB (1247 MB used, 6945 MB free)
Network:
IP: 192.168.137.2
```
```bash
bhatti info agent --json | jq .keep_hot
```
```text
true
```
```bash
# Stopped sandbox — no live df, static disk size shown
bhatti inspect scraper
```
```text
Name: scraper
ID: e5f6g7h8
Status: stopped
Thermal: cold
Image: browser
Created: 2026-04-22T10:14:03Z
Stopped: 2026-04-29T08:55:12Z
Resources:
CPUs: 1
Memory: 2048 MB
Disk: — (stopped)
Network:
IP: 192.168.137.8
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti list`](/v1/docs/reference/cli/sandbox/list/) — list all sandboxes
* [`bhatti edit`](/v1/docs/reference/cli/sandbox/edit/) — change mutable settings
* [API: `GET /sandboxes/:id`](/v1/docs/reference/api/#get-a-sandbox)
# bhatti list
> List sandboxes you own. Adds a wide format and JSON output.
List sandboxes owned by the authenticated user. Aliased as `bhatti ls`.
## Synopsis
```
bhatti list [-o wide]
bhatti ls [-o wide]
```
## Description
Returns one row per sandbox with status, thermal state, IP, and the first published URL (if any). The default columns are tuned for fitting in a normal terminal; `-o wide` adds CPUs, memory, disk, and the rootfs image.
Listing does not wake cold sandboxes — the server reads from its database, not from the engines. The `THERMAL` column shows the current state (`hot`, `warm`, `cold`).
As a side effect, `list` refreshes the local sandbox-name completion cache (`/tmp/bhatti-completions-<uid>`). Tab-completing sandbox names will pick up new sandboxes after the first `list` call following a `create` from another machine.
## Examples
```bash
bhatti list
```
```text
NAME STATUS THERMAL IP
dev running hot 192.168.137.2
agent running cold 192.168.137.5
scraper stopped - 192.168.137.8
```
```bash
bhatti ls -o wide
```
```text
NAME STATUS THERMAL IP CPUS MEMORY DISK IMAGE
dev running hot 192.168.137.2 1 1024 0 minimal
agent running cold 192.168.137.5 2 4096 8192 browser
scraper stopped - 192.168.137.8 1 1024 0 minimal
```
```bash
# Machine-readable output
bhatti ls --json | jq '.[] | select(.status == "running") | .name'
```
```bash
# When at least one sandbox has a published URL, an extra URL column appears
bhatti list
```
```text
NAME STATUS THERMAL IP URL
api running hot 192.168.137.3 https://api.bhatti.sh
dev running hot 192.168.137.2
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-o`, `--output <fmt>` | (default columns) | Output format. Currently only `wide` is supported. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti inspect`](/v1/docs/reference/cli/sandbox/inspect/) — full details for a single sandbox
* [`bhatti ps`](/v1/docs/reference/cli/exec/ps/) — sessions inside a sandbox
* [API: `GET /sandboxes`](/v1/docs/reference/api/#list-sandboxes)
# bhatti start
> Resume a stopped sandbox from its snapshot.
Resume a stopped sandbox from its on-disk snapshot. The VM continues exactly where it left off — same processes, same memory, same network connections.
## Synopsis
```
bhatti start <sandbox> [--force]
```
## Description
Restores the VM from `mem.snap` + `vm.snap` and re-attaches it to its TAP device. Typical resume latency is \~50ms for a cold sandbox (snapshot read from disk) and sub-millisecond for a warm one (memory still allocated).
If a previous restore failed (for example because the underlying rootfs file was corrupt or a TAP device couldn't be allocated), the sandbox enters the `unknown` state and `bhatti start` refuses to retry by default. Pass `--force` to retry — useful after fixing the underlying issue.
You normally don't need to call `start` explicitly. Any API request — `exec`, `file read`, an HTTP request to a published URL — wakes a cold sandbox transparently before executing.
## Examples
```bash
bhatti start dev
```
```text
sandbox/dev started
```
```bash
# Retry after a previously failed restore
bhatti start dev --force
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--force` | `false` | Retry start even if the sandbox is in an `unknown` state from a previously failed restore. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti stop`](/v1/docs/reference/cli/sandbox/stop/) — snapshot and stop
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — automatic resume
* [API: `POST /sandboxes/:id/start`](/v1/docs/reference/api/#start-a-sandbox)
# bhatti stop
> Snapshot a sandbox to disk and free its memory.
Snapshot a sandbox to disk and free its memory. The VM keeps its identity, IP, and on-disk state; resume with [`bhatti start`](/v1/docs/reference/cli/sandbox/start/).
## Synopsis
```
bhatti stop <sandbox>
```
## Description
Pauses the VM, dumps memory + CPU state to disk, then frees the host RAM. The first `stop` on a sandbox is a *full* snapshot (\~3-4s for a 1GB VM); subsequent stops are *diff* snapshots that only persist dirty pages (\~20-50ms typical).
Stopped sandboxes use zero CPU and zero memory on the host. Their disk footprint includes the rootfs, attached volumes, and the snapshot files (`mem.snap`, `vm.snap`).
In normal operation you don't need this — idle sandboxes pause and resume automatically per the thermal manager. Use `bhatti stop` to control thermal state explicitly when you know a sandbox will be idle for a while and want to free memory immediately.
## Examples
```bash
bhatti stop dev
```
```text
sandbox/dev stopped
```
```bash
# Resume later
bhatti start dev
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti start`](/v1/docs/reference/cli/sandbox/start/) — resume from snapshot
* [`bhatti snapshot create`](/v1/docs/reference/cli/snapshots/create/) — named snapshot you can resume into a *new* sandbox
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — automatic pause/resume
* [API: `POST /sandboxes/:id/stop`](/v1/docs/reference/api/#stop-a-sandbox)
# Secrets
> Encrypted secrets injected into sandboxes as environment variables.
Secrets are key-value pairs encrypted at rest with [age](https://age-encryption.org/) and scoped to your user. Reference them by name when creating a sandbox; they’re decrypted server-side and injected as environment variables at boot.
| Command | Description |
| ---------------------------------------------------------------- | ---------------------------------------------- |
| [`bhatti secret set`](/v1/docs/reference/cli/secrets/set/) | Create or update a secret. |
| [`bhatti secret list`](/v1/docs/reference/cli/secrets/list/) | List secret names (values are never returned). |
| [`bhatti secret delete`](/v1/docs/reference/cli/secrets/delete/) | Delete a secret. |
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
# Store
bhatti secret set OPENAI_KEY sk-...
bhatti secret set DATABASE_URL "postgres://user:pass@db.internal/app"
# Reference at create
bhatti create --name agent --secret OPENAI_KEY --secret DATABASE_URL
# List names (no values)
bhatti secret list
# Update (set with the same name overwrites)
bhatti secret set OPENAI_KEY sk-...new-value
# Delete (no confirmation prompt — be careful)
bhatti secret delete OPENAI_KEY
```
## How secrets work
[Section titled “How secrets work”](#how-secrets-work)
1. **Encryption at rest.** Values are encrypted with an age key the server stores at `/var/lib/bhatti/age.key`. The key is generated on first server start. Plaintext is never written to disk.
2. **Scoped per user.** Each user has their own secrets. Other users can’t see them, list them, or reference them by name.
3. **Names only in `list`.** `bhatti secret list` returns metadata (name, created/updated timestamps). The encrypted value is never serialized to JSON output.
4. **Inject by name.** `bhatti create --secret API_KEY` decrypts `API_KEY` and adds it to the sandbox’s environment as `API_KEY=<plaintext>`.
5. **Update is idempotent.** Re-running `bhatti secret set NAME value` overwrites the previous value with no error.
## Env var priority inside a sandbox
[Section titled “Env var priority inside a sandbox”](#env-var-priority-inside-a-sandbox)
When a command runs inside a sandbox, env vars are resolved in this order (later wins):
```plaintext
defaults (PATH, TERM, HOME, LANG)
↓ overridden by
secrets (--secret on create)
↓ overridden by
sandbox env (--env on create)
```
There is **no** per-`bhatti exec` env override. The exec endpoint doesn’t accept an `env` field; the env passed in the request is silently ignored. To inject env vars, do it at create time.
## Caveat: `--template` ignores `--secret`
[Section titled “Caveat: --template ignores --secret”](#caveat---template-ignores---secret)
When creating from a template, the server uses only the template’s pre-configured secret list. Request-side `--secret` flags are silently dropped. See [`bhatti create`](/v1/docs/reference/cli/sandbox/create/) for the workaround.
# bhatti secret delete
> Delete a secret. Cannot be undone.
Delete a secret.
## Synopsis
```
bhatti secret delete <name> [-y]
```
## Description
Removes the secret from the encrypted store. **Cannot be undone.**
By default, prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
Sandboxes that already have the secret resolved into their environment keep running with the value they got at boot. Future creates that reference the deleted secret will fail with `secret "<name>" not found`.
## Examples
```bash
bhatti secret delete OLD_KEY
```
```text
Delete secret "OLD_KEY"? [y/N]: y
deleted
```
```bash
# Non-interactive
bhatti secret delete OLD_KEY -y
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti secret list`](/v1/docs/reference/cli/secrets/list/) — confirm the right name first
* [API: `DELETE /secrets/:name`](/v1/docs/reference/api/#delete-a-secret)
# bhatti secret list
> List secret names. Values are never returned.
List your secrets by name. Values are never returned by the API.
## Synopsis
```
bhatti secret list
```
## Description
Returns metadata for each secret you own — name, created-at, updated-at — but never the encrypted value or the decrypted plaintext. Once a secret is set, the value can be replaced (with `bhatti secret set NAME new-value`) but not read back.
This is intentional. The encryption-at-rest model assumes the value is read by the server only when injecting into a sandbox; exposing it via the API would defeat the point.
## Examples
```bash
bhatti secret list
```
```text
OPENAI_KEY
DATABASE_URL
GITHUB_TOKEN
```
```bash
# JSON for scripting
bhatti secret list --json
```
```json
[
{"name": "OPENAI_KEY", "created_at": "2026-04-15T10:00:00Z", "updated_at": "2026-04-29T14:22:00Z"},
{"name": "DATABASE_URL", "created_at": "2026-04-22T09:15:00Z", "updated_at": "2026-04-22T09:15:00Z"}
]
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [API: `GET /secrets`](/v1/docs/reference/api/#list-secrets)
# bhatti secret set
> Create or update an encrypted secret.
Create or update a secret.
## Synopsis
```
bhatti secret set <name> <value>
```
## Description
Encrypts the value with the server's age key and stores it under the given name, scoped to the authenticated user. Re-running with the same name overwrites the previous value with no error.
The CLI takes the value as a positional argument. **There's no stdin variant** — long or multiline values pass through your shell history. For sensitive values, use a shell that doesn't record (`HISTCONTROL=ignorespace` + leading space) or pipe through a temp variable that you `unset` afterwards.
## Examples
```bash
bhatti secret set OPENAI_KEY sk-abc123def456
```
```text
ok
```
```bash
# Update is the same call
bhatti secret set OPENAI_KEY sk-new-value
```
```bash
# Multiline / connection string
bhatti secret set DATABASE_URL "postgres://user:pass@db.internal:5432/app?sslmode=require"
```
```bash
# Avoid shell history (bash with HISTCONTROL=ignorespace, note leading space)
bhatti secret set OPENAI_KEY sk-abc123def456
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti create --secret`](/v1/docs/reference/cli/sandbox/create/) — inject the secret into a sandbox at boot
* [API: `POST /secrets`](/v1/docs/reference/api/#create-or-update-a-secret)
# Snapshots
> Named VM snapshots that capture memory, CPU, and disk. Resume into a new sandbox at any time.
A snapshot captures the entire VM state at a point in time — memory, CPU registers, disk, running processes, open file descriptors, network connections. Resume restores all of it, exactly. The new sandbox picks up where the original left off.
| Command | Description |
| -------------------------------------------------------------------- | ------------------------------------------ |
| [`bhatti snapshot create`](/v1/docs/reference/cli/snapshots/create/) | Checkpoint a running sandbox. |
| [`bhatti snapshot list`](/v1/docs/reference/cli/snapshots/list/) | List your snapshots. |
| [`bhatti snapshot resume`](/v1/docs/reference/cli/snapshots/resume/) | Resume from a snapshot into a new sandbox. |
| [`bhatti snapshot delete`](/v1/docs/reference/cli/snapshots/delete/) | Delete a snapshot. |
## How is this different from `bhatti stop`?
[Section titled “How is this different from bhatti stop?”](#how-is-this-different-from-bhatti-stop)
`bhatti stop` snapshots the *current* sandbox to disk and freezes it; `bhatti start` resumes that same sandbox. The sandbox doesn’t go away — it’s just paused.
`bhatti snapshot create` saves a **named** copy of the VM state. The original sandbox keeps running. `bhatti snapshot resume` later creates a **new** sandbox from that state, with a new name and new IP. You can resume from the same snapshot multiple times to stamp out copies.
| | `stop`/`start` | snapshot |
| -------------------------- | ----------------------------- | -------------------------------- |
| Original sandbox affected? | Yes — paused | No — keeps running |
| Result of resume | Same sandbox awake | A *new* sandbox |
| Multiple uses? | Once (it pairs with the stop) | Many — fork as often as you want |
| Captures volumes? | No (volumes are detached) | No (rootfs + memory only) |
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
# Checkpoint before a risky experiment
bhatti snapshot create dev --name dev-pre-experiment
# Fork: same VM state, multiple copies
bhatti snapshot resume dev-pre-experiment --name copy-1
bhatti snapshot resume dev-pre-experiment --name copy-2
# Clean up
bhatti snapshot delete dev-pre-experiment -y
```
## What’s *not* in a snapshot
[Section titled “What’s not in a snapshot”](#whats-not-in-a-snapshot)
* **Persistent volumes.** Snapshots only capture the rootfs and memory; volumes are detached when you resume into a new sandbox. Re-attach them on the resumed sandbox if you need them.
* **External state.** TCP connections to *other* hosts may be in an awkward state on resume — kernel state is preserved, but the remote peer doesn’t know the sandbox stopped and resumed.
# bhatti snapshot create
> Checkpoint a running sandbox to a named snapshot.
Checkpoint a running sandbox to a named snapshot.
## Synopsis
```
bhatti snapshot create <sandbox> --name <snapshot-name>
```
## Description
Captures memory, CPU registers, disk, and running processes from the running sandbox and saves them under `<snapshot-name>`. The source sandbox is briefly paused during the capture (typically a few seconds for a full snapshot, \~50ms for a diff snapshot if a previous one exists), then resumes.
The snapshot is stored on the server in the user's snapshot directory (`/var/lib/bhatti/snapshots/<user>/`). Resume into a new sandbox with [`bhatti snapshot resume`](/v1/docs/reference/cli/snapshots/resume/) at any time.
`--name` is required. Names must match `[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}` — same rules as sandbox names.
## Examples
```bash
bhatti snapshot create dev --name pre-experiment
```
```text
checkpoint "pre-experiment" created (512MB)
```
```bash
# Re-running with the same snapshot name overwrites the previous one
bhatti snapshot create dev --name daily
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | (required) | Snapshot name. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti snapshot resume`](/v1/docs/reference/cli/snapshots/resume/) — create a sandbox from this snapshot
* [`bhatti stop`](/v1/docs/reference/cli/sandbox/stop/) — pause the original instead of forking
* [API: `POST /sandboxes/:id/checkpoint`](/v1/docs/reference/api/#create-a-snapshot)
# bhatti snapshot delete
> Delete a snapshot. Cannot be undone.
Delete a named snapshot.
## Synopsis
```
bhatti snapshot delete <snapshot-name> [-y]
```
## Description
Removes the snapshot record and the on-disk snapshot files (`mem.snap`, `vm.snap`, copied rootfs). **Cannot be undone.**
By default, prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
Sandboxes that were previously resumed from this snapshot are unaffected — they have their own copy of the state and don't reference the original.
## Examples
```bash
bhatti snapshot delete pre-experiment
```
```text
Delete snapshot "pre-experiment"? [y/N]: y
deleted
```
```bash
# Non-interactive
bhatti snapshot delete pre-experiment -y
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [API: `DELETE /snapshots/:name`](/v1/docs/reference/api/#delete-a-snapshot)
# bhatti snapshot list
> List named VM snapshots.
List your named VM snapshots.
## Synopsis
```
bhatti snapshot list
```
## Description
Lists snapshots owned by the authenticated user. Each row includes the snapshot ID, name, source sandbox name, and on-disk size in MB.
## Examples
```bash
bhatti snapshot list
```
```text
ID NAME SOURCE SIZE
snap_a1b2c3d4 pre-experiment dev 512MB
snap_e5f6g7h8 daily worker 1024MB
snap_i9j0k1l2 ready base 384MB
```
```bash
# JSON for scripting
bhatti snapshot list --json | jq '.[] | select(.source_sandbox == "base")'
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [API: `GET /snapshots`](/v1/docs/reference/api/#list-snapshots)
# bhatti snapshot resume
> Resume a snapshot into a new sandbox. The original is unaffected.
Resume a named snapshot into a new sandbox.
## Synopsis
```
bhatti snapshot resume <snapshot-name> [--name <new-sandbox-name>]
```
## Description
Creates a brand-new sandbox by restoring the snapshot's memory, CPU, and disk state. Running processes pick up exactly where they left off; open file descriptors are still open; in-memory data structures are intact.
The new sandbox gets a new ID and a new IP. **Persistent volumes are not re-attached** — re-attach them with the same `--volume` semantics if needed.
`--name` is optional. Without it, a name is generated from the snapshot's source sandbox plus a random suffix.
You can resume the same snapshot multiple times — each call produces a fresh sandbox with the same starting state. Useful for fork-and-experiment workflows.
## Examples
```bash
# Resume into a sandbox with a specific name
bhatti snapshot resume pre-experiment --name dev-restored
```
```text
abc123 dev-restored 192.168.137.42
```
```bash
# Auto-generated name (source sandbox + suffix)
bhatti snapshot resume pre-experiment
```
```bash
# Stamp out parallel copies for an A/B test
bhatti snapshot resume base --name worker-a
bhatti snapshot resume base --name worker-b
bhatti snapshot resume base --name worker-c
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | auto-generated | Name for the new sandbox. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti snapshot create`](/v1/docs/reference/cli/snapshots/create/) — make the snapshot
* [`bhatti snapshot list`](/v1/docs/reference/cli/snapshots/list/) — find snapshot names
* [API: `POST /snapshots/:name/resume`](/v1/docs/reference/api/#resume-a-snapshot)
# Volumes
> Persistent ext4 volumes that survive sandbox destruction. Resize, clone, back up, restore.
Persistent volumes are ext4 filesystems that survive sandbox destruction. Create one, attach it at sandbox creation with `--volume name:/mount`, destroy the sandbox — the data stays.
| Command | Description |
| ------------------------------------------------------------------------------ | ------------------------------------------------------------ |
| [`bhatti volume create`](/v1/docs/reference/cli/volumes/create/) | Create a persistent volume. |
| [`bhatti volume list`](/v1/docs/reference/cli/volumes/list/) | List your volumes and current attachments. |
| [`bhatti volume delete`](/v1/docs/reference/cli/volumes/delete/) | Delete a volume. Requires it to be detached. |
| [`bhatti volume resize`](/v1/docs/reference/cli/volumes/resize/) | Grow a volume. Cannot shrink. |
| [`bhatti volume clone`](/v1/docs/reference/cli/volumes/clone/) | Point-in-time copy. Source must be detached. |
| [`bhatti volume backup`](/v1/docs/reference/cli/volumes/backup/) | Back up to S3-compatible storage. Requires server S3 config. |
| [`bhatti volume backup-list`](/v1/docs/reference/cli/volumes/backup-list/) | List backups for a volume. |
| [`bhatti volume restore`](/v1/docs/reference/cli/volumes/restore/) | Restore from a backup. Volume must be detached. |
| [`bhatti volume backup-delete`](/v1/docs/reference/cli/volumes/backup-delete/) | Delete a backup. |
## Detached state
[Section titled “Detached state”](#detached-state)
Four operations require the volume to be **detached** from any sandbox before they’ll run: `delete`, `resize`, `clone`, and `restore` all return a `409 Conflict` if the volume is currently attached.
Detach by either destroying the sandbox using the volume, or stopping it with [`bhatti stop`](/v1/docs/reference/cli/sandbox/stop/) (which also detaches volumes).
`backup` does **not** check for attached state — it can run while the sandbox is mounted, but be aware the snapshot may be inconsistent if writes are in flight at backup time. Stop the sandbox first if you need a guaranteed-consistent backup.
## Quick patterns
[Section titled “Quick patterns”](#quick-patterns)
```bash
# Create + attach in one workflow
bhatti volume create --name workspace --size 5120
bhatti create --name dev --volume workspace:/workspace
# Grow on the fly
bhatti volume resize workspace --size 10240
# Snapshot before a risky operation
bhatti volume clone workspace --name workspace-pre-upgrade
# Off-site backup (requires `backup` config block on server)
bhatti volume backup workspace
bhatti volume backup-list workspace
```
## Backup configuration
[Section titled “Backup configuration”](#backup-configuration)
`backup`, `restore`, `backup-list`, and `backup-delete` require an S3-compatible storage backend configured on the server. Without it, they return `501 Not Implemented` with the message `backup not configured — add backup section to config.yaml`. See [Configuration: backup](/v1/docs/reference/config/#backup).
# bhatti volume backup
> Back up a volume to S3-compatible storage. Requires server backup config.
Back up a persistent volume to S3-compatible storage.
## Synopsis
```
bhatti volume backup <volume-name>
```
## Description
Compresses (zstd) the volume's ext4 file and uploads it to the S3-compatible bucket configured on the server. Returns a backup record with an ID, size, and S3 key.
The volume **does not** need to be detached for this to run. However, if writes are in flight at backup time, the snapshot may be inconsistent. For a guaranteed-consistent backup, stop the sandbox first:
```bash
bhatti stop dev # detaches volume, flushes guest writes
bhatti volume backup workspace
bhatti start dev # resume
```
Requires a `backup` block in the server's `config.yaml`. Without it, the API returns `501` with `backup not configured — add backup section to config.yaml`. See [Configuration: backup](/v1/docs/reference/config/#backup).
Backups are stored under the per-user S3 prefix configured by the server. Restore with [`bhatti volume restore`](/v1/docs/reference/cli/volumes/restore/).
## Examples
```bash
bhatti volume backup workspace
```
```text
backup bk_a1b2c3d4 created (workspace, 5234567890 bytes)
```
```bash
# JSON for scripting (capture the ID for later restore)
backup_id=$(bhatti volume backup workspace --json | jq -r .id)
echo "backed up: $backup_id"
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume backup-list`](/v1/docs/reference/cli/volumes/backup-list/) — see existing backups
* [`bhatti volume restore`](/v1/docs/reference/cli/volumes/restore/) — recover from a backup
* [`bhatti volume clone`](/v1/docs/reference/cli/volumes/clone/) — local point-in-time copy (no S3, no compression)
* [API: `POST /volumes/:name/backups`](/v1/docs/reference/api/#backup-a-volume)
# bhatti volume backup-delete
> Delete a backup from S3 and the local backup list.
Delete a volume backup.
## Synopsis
```
bhatti volume backup-delete <volume-name> <backup-id> [-y]
```
## Description
Removes the backup record from the server and deletes the corresponding object in S3. Cannot be undone.
By default, prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
## Examples
```bash
bhatti volume backup-delete workspace bk_a1b2c3d4
```
```text
Delete backup bk_a1b2c3d4? [y/N]: y
deleted
```
```bash
# Non-interactive
bhatti volume backup-delete workspace bk_a1b2c3d4 -y
```
```bash
# Trim to last 7 backups
bhatti volume backup-list workspace --json \
| jq -r 'sort_by(.created_at) | reverse | .[7:][] | .id' \
| xargs -I{} bhatti volume backup-delete workspace {} -y
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume backup-list`](/v1/docs/reference/cli/volumes/backup-list/)
* [API: `DELETE /volumes/:name/backups/:id`](/v1/docs/reference/api/#delete-a-backup)
# bhatti volume backup-list
> List backups for a volume.
List backups for a volume.
## Synopsis
```
bhatti volume backup-list <volume-name>
```
## Description
Lists backups recorded server-side for the given volume. The backups themselves live in S3; this command shows the metadata (ID, size, creation time) so you can pick one to restore.
If there are no backups, prints `no backups`.
## Examples
```bash
bhatti volume backup-list workspace
```
```text
ID SIZE CREATED
bk_a1b2c3d4 4998 MB 2026-04-30T03:00:14Z
bk_e5f6g7h8 5021 MB 2026-04-29T03:00:11Z
bk_i9j0k1l2 4892 MB 2026-04-28T03:00:09Z
```
```bash
# Latest backup ID
bhatti volume backup-list workspace --json | jq -r 'sort_by(.created_at) | last | .id'
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume backup`](/v1/docs/reference/cli/volumes/backup/)
* [API: `GET /volumes/:name/backups`](/v1/docs/reference/api/#list-volume-backups)
# bhatti volume clone
> Make a point-in-time copy of a volume. Source must be detached.
Make a point-in-time copy of a persistent volume.
## Synopsis
```
bhatti volume clone <source-name> --name <new-name>
```
## Description
Server-side copy of the volume's ext4 file. The clone is independent — writes to the source after the clone do not affect the copy, and vice versa.
The source volume must be **detached** when you clone it. If any sandbox has it mounted, the server returns `409 Conflict`. Detach by stopping or destroying the sandbox using it.
Useful for:
* Snapshotting before a risky upgrade or migration.
* Forking a workspace for an experiment.
* Pre-warming a base volume before stamping out workers.
## Examples
```bash
# Snapshot before upgrade
bhatti stop dev # detaches workspace
bhatti volume clone workspace --name workspace-pre-upgrade
bhatti start dev
```
```text
Cloned workspace → workspace-pre-upgrade
```
```bash
# Fork for an experiment
bhatti volume clone workspace --name workspace-experiment
bhatti create --name experiment --volume workspace-experiment:/workspace
```
```bash
# Source still attached — fails
bhatti volume clone workspace --name copy
```
```text
409 Conflict: source volume is attached — detach before snapshotting
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | (required) | Name for the cloned volume. Must match the standard volume-name pattern. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume backup`](/v1/docs/reference/cli/volumes/backup/) — off-site copy via S3 (works while attached, but may be inconsistent)
* [`bhatti stop`](/v1/docs/reference/cli/sandbox/stop/) — detach by stopping the sandbox using it
* [API: `POST /volumes/:name/snapshot`](/v1/docs/reference/api/#clone-a-volume)
# bhatti volume create
> Create a persistent ext4 volume. Survives sandbox destruction.
Create a persistent ext4 volume.
## Synopsis
```
bhatti volume create --name <name> --size <MB>
```
## Description
Allocates an ext4 filesystem image on the server, owned by the calling user. The volume is unattached at creation; mount it at sandbox creation time with [`--volume name:/mount`](/v1/docs/reference/cli/sandbox/create/).
The volume survives sandbox destruction. Multiple sandboxes can attach the same volume sequentially, but only one sandbox can have it attached at a time (read-only mounts via `name:/mount:ro` are the only way to share live).
## Examples
```bash
bhatti volume create --name workspace --size 5120
```
```text
vol_a1b2c3d4 workspace 5120MB ready
```
```bash
# Attach at sandbox create
bhatti create --name dev --volume workspace:/workspace
```
```bash
# Attach read-only — multiple sandboxes can mount this simultaneously
bhatti create --name reader --volume workspace:/data:ro
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--name <string>` | (required) | Volume name. Must match `[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}`. | | `--size <int>` | (required) | Initial size in MB. Grow later with [`bhatti volume resize`](/v1/docs/reference/cli/volumes/resize/). |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti create --volume`](/v1/docs/reference/cli/sandbox/create/) — attach at boot
* [`bhatti volume resize`](/v1/docs/reference/cli/volumes/resize/) — grow the volume
* [API: `POST /volumes`](/v1/docs/reference/api/#create-a-volume)
# bhatti volume delete
> Delete a persistent volume. Cannot be undone. Volume must be detached.
Delete a persistent volume. **Cannot be undone.**
## Synopsis
```
bhatti volume delete <name> [-y]
```
## Description
Removes the volume record from the database and deletes the underlying ext4 file.
The volume must be **detached** before deleting — if any sandbox has it mounted, the server returns `409 Conflict`. Stop or destroy the sandbox first, then retry.
By default, prompts for confirmation. In a non-interactive shell without `-y`, the command refuses and exits `1`.
## Examples
```bash
bhatti volume delete workspace
```
```text
Delete volume "workspace"? [y/N]: y
deleted
```
```bash
# Non-interactive
bhatti volume delete workspace -y
```
```bash
# Volume still attached
bhatti volume delete workspace -y
```
```text
409 Conflict: volume "workspace" attached to sandbox "dev"
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `-y`, `--yes` | `false` | Skip confirmation. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume list`](/v1/docs/reference/cli/volumes/list/) — find current attachments
* [API: `DELETE /volumes/:name`](/v1/docs/reference/api/#delete-a-volume)
# bhatti volume list
> List persistent volumes you own.
List your persistent volumes.
## Synopsis
```
bhatti volume list
```
## Description
Lists volumes owned by the authenticated user. Each row includes ID, name, size, and current status.
The default output is compact. Use `--json` to get the full record including current attachments (which sandbox has it mounted, where, and read-only flag).
## Examples
```bash
bhatti volume list
```
```text
ID NAME SIZE STATUS
vol_a1b2c3d4 workspace 5120MB ready
vol_e5f6g7h8 data 20480MB ready
```
```bash
# Find which sandbox has a volume mounted
bhatti volume list --json | jq '.[] | {name, attachments}'
```
```json
{
"name": "workspace",
"attachments": [
{"sandbox_id": "abc123", "mount": "/workspace", "read_only": false}
]
}
```
## Options
This command takes only global flags. See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [API: `GET /volumes`](/v1/docs/reference/api/#list-volumes)
# bhatti volume resize
> Grow a persistent volume. Cannot shrink.
Grow a persistent volume.
## Synopsis
```
bhatti volume resize <name> --size <new-MB>
```
## Description
Resizes the underlying ext4 image file to the new size. The new size must be **larger** than the current size — shrinking is not supported.
The volume must be **detached** before resizing. The server returns `409 Conflict` if any sandbox has it mounted. After the resize, attach it again and the new capacity is visible to the guest filesystem on next mount.
## Examples
```bash
# Was 5120 MB; double it
bhatti volume resize workspace --size 10240
```
```text
resized to 10240MB
```
```bash
# Trying to shrink fails
bhatti volume resize workspace --size 1024
```
```text
400 Bad Request: new size (1024MB) must be larger than current size (5120MB)
```
```bash
# Volume still attached
bhatti volume resize workspace --size 10240
```
```text
409 Conflict: volume is attached — detach before resizing
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--size <int>` | (required) | New size in MB. Must be larger than the current size. |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [API: `POST /volumes/:name/resize`](/v1/docs/reference/api/#resize-a-volume)
# bhatti volume restore
> Restore a volume from a backup. Volume must be detached.
Restore a volume from a backup.
## Synopsis
```
bhatti volume restore <volume-name> --backup-id <id>
```
## Description
Downloads the backup from S3, decompresses it, and overwrites the volume's ext4 file with the backup's contents. **Existing data on the volume is replaced** — there's no merge.
The volume must be **detached** before restoring. The server returns `409 Conflict` if any sandbox has it mounted.
The backup ID comes from [`bhatti volume backup-list`](/v1/docs/reference/cli/volumes/backup-list/). The restore is in-place: the volume keeps its name and any pending attachments resolve once it's done.
## Examples
```bash
# Find a backup, restore from it
bhatti stop dev
bhatti volume backup-list workspace
bhatti volume restore workspace --backup-id bk_a1b2c3d4
bhatti start dev
```
```text
volume restored from backup bk_a1b2c3d4
```
```bash
# Volume still attached
bhatti volume restore workspace --backup-id bk_a1b2c3d4
```
```text
409 Conflict: volume is attached to a sandbox — detach before restoring
```
## Options
\| Flag | Default | Description | | ---- | ------- | ----------- | | `--backup-id <string>` | (required) | The backup to restore from. Get one from [`backup-list`](/v1/docs/reference/cli/volumes/backup-list/). |
See [Global flags](/v1/docs/reference/cli/#global-flags) for `--url`, `--token`, `--json`, `--timing`, `--data-dir`.
## See also
* [`bhatti volume backup-list`](/v1/docs/reference/cli/volumes/backup-list/) — find a backup to restore from
* [`bhatti volume clone`](/v1/docs/reference/cli/volumes/clone/) — make a clone before restoring, just in case
* [API: `POST /volumes/:name/backups/restore`](/v1/docs/reference/api/#restore-a-volume)
# Configuration
> Server and CLI config files, layered loading, environment variables, and the data directory layout.
bhatti has a single YAML config file. The same file format is used by the daemon and the CLI client; the daemon ignores the client-only fields, the client ignores the daemon-only ones.
## Layered loading
[Section titled “Layered loading”](#layered-loading)
The config is loaded from multiple paths and merged. Higher rows override lower rows:
| Source | Loaded? | Used for |
| ------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------- |
| `$BHATTI_CONFIG` (path) | If set, **only** this is loaded — no merging. | Override everything; useful for tests. |
| `/etc/bhatti/config.yaml` | First match wins. | Server settings — engine paths, listen address, data dir. |
| `~/.bhatti/config.yaml` | Always — fills in only `api_url` and `auth_token` if still empty. | CLI client credentials. |
This means a developer machine that’s also running a server can have the server config in `/etc/bhatti/config.yaml` (with no client credentials) and the CLI’s API key in `~/.bhatti/config.yaml` (without disturbing the server config). Both files are read, neither overwrites the other.
The deprecated location `/var/lib/bhatti/config.yaml` is still honoured as a fallback; the daemon prints a stderr warning when it loads from there. Migrate to `/etc/bhatti/config.yaml`.
## Environment variables
[Section titled “Environment variables”](#environment-variables)
| Variable | Used by | Description |
| --------------------- | ------- | --------------------------------------------------------------------- |
| `BHATTI_CONFIG` | both | Override the config file path. When set, layered loading is bypassed. |
| `BHATTI_LOG_LEVEL` | server | `debug`, `info` (default), `warn`, `error`. |
| `BHATTI_URL` | CLI | API endpoint. Falls back when `--url` and `api_url:` aren’t set. |
| `BHATTI_TOKEN` | CLI | API key. Same fallback semantics. |
| `BHATTI_FORCE_STREAM` | CLI | Force NDJSON streaming output even when stdout isn’t a TTY. |
The CLI’s value precedence is: `--flag` > config file > env var > built-in default.
## Server config
[Section titled “Server config”](#server-config)
A typical server config (the one written by `curl -fsSL bhatti.sh/install | sudo bash`) looks like:
```yaml
engine: firecracker
listen: :8080
data_dir: /var/lib/bhatti
firecracker_bin: /usr/local/bin/firecracker
firecracker_jailer: /usr/local/bin/jailer
jail_uid: 10000
jail_gid: 10000
firecracker_kernel: /var/lib/bhatti/images/vmlinux-arm64
firecracker_rootfs: /var/lib/bhatti/images/rootfs-minimal-arm64.ext4
```
### Top-level fields
[Section titled “Top-level fields”](#top-level-fields)
| Field | Default | Description |
| --------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `engine` | `firecracker` | Engine backend. Only `firecracker` is implemented. |
| `listen` | `:8080` | Address for the HTTP API. |
| `data_dir` | `~/.bhatti` (CLI) or whatever the install script writes (typically `/var/lib/bhatti` on a server) | Root directory for state — DB, images, sandboxes, snapshots, jails. |
| `firecracker_bin` | — | Absolute path to the `firecracker` binary. |
| `firecracker_kernel` | — | Path to the `vmlinux` kernel image. |
| `firecracker_rootfs` | — | Path to the default rootfs image (used when `bhatti create` is called without `--image`). |
| `firecracker_jailer` | — | Path to the `jailer` binary. **Empty** = bare mode (no jailer; less isolation). When set, `jail_uid` / `jail_gid` apply. |
| `jail_uid` | `0` | UID Firecracker runs as inside the jail. Production: a non-root UID like `10000`. |
| `jail_gid` | `0` | GID. Same shape. |
| `public_proxy_listen` | — | When set (e.g. `:8443`), the daemon also exposes a path-based public proxy at this address. URLs are `http://<host>:8443/<alias>/`. Skip this and use [domain mode](#domain-mode) for production. |
| `api_url` | — | CLI-only field. Put it in `~/.bhatti/config.yaml`, not `/etc/bhatti/config.yaml`. |
| `auth_token` | — | CLI-only. Same. |
| `domain` | — | Optional. Enables [domain mode](#domain-mode) — host-based routing + TLS. |
| `backup` | — | Optional. Enables [volume backups](#backup) to S3-compatible storage. |
### Domain mode
[Section titled “Domain mode”](#domain-mode)
For host-based routing with TLS — `https://api.<your-domain>` for the API, `https://<alias>.<your-domain>` for published sandboxes.
```yaml
domain:
api_host: api.bhatti.sh
proxy_zone: bhatti.sh
tls_cert: /etc/bhatti/wildcard.pem
tls_key: /etc/bhatti/wildcard-key.pem
```
| Field | Description |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `api_host` | Hostname for the API. Requests to this host go through normal Bearer auth. |
| `proxy_zone` | Zone for published sandboxes. `<alias>.<proxy_zone>` is the URL `bhatti publish` generates. |
| `tls_cert`, `tls_key` | Paths to a wildcard cert covering `*.<proxy_zone>` and `<api_host>`. **Recommended.** |
| `acme_email` | Fallback: per-alias Let’s Encrypt certificates. **Rate-limited** to 50 new aliases per registered domain per week — fine for stable subdomains, fast to hit if you’re stamping out preview environments. |
When domain mode is on, the daemon listens on `:443` (TLS, both API and proxy by Host header), `:80` (ACME challenges + HTTPS redirect), and `127.0.0.1:8080` (internal API for health checks).
You must set either `tls_cert`+`tls_key` or `acme_email`. A wildcard cert is the right answer for any setup that creates more than a handful of aliases per week. See [Custom domain](/v1/docs/managing/custom-domain/) for the full setup walk-through.
### Backup
[Section titled “Backup”](#backup)
For volume backups to S3-compatible storage. Enables [`bhatti volume backup`](/v1/docs/reference/cli/volumes/backup/), `restore`, `backup-list`, and `backup-delete`. Without this block, those endpoints return `501`.
```yaml
backup:
s3_endpoint: https://s3.eu-central-003.backblazeb2.com
s3_region: eu-central-003
s3_bucket: bhatti-backups
s3_access_key: ...
s3_secret_key: ...
schedule:
- volume: workspace
cron: "0 3 * * *"
retention: 7
```
| Field | Description |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `s3_endpoint` | S3-compatible endpoint URL. Backblaze B2, AWS S3, MinIO, R2, etc. |
| `s3_region` | Region. Required by AWS-compatible APIs even when irrelevant. |
| `s3_bucket` | Bucket name. Must already exist. |
| `s3_access_key`, `s3_secret_key` | Credentials with read/write/delete on the bucket. |
| `schedule` | Optional. Array of automatic backup schedules. Each entry has `volume`, `cron` (5-field cron expression), and `retention` (keep last N backups for that volume). |
Schedules run inside the daemon — no external cron required. Retention is enforced after each scheduled backup.
## CLI client config
[Section titled “CLI client config”](#cli-client-config)
Lives at `~/.bhatti/config.yaml`. `bhatti setup` writes it for you.
```yaml
api_url: https://api.bhatti.sh
auth_token: bht_abc123def456...
```
| Field | Description |
| ------------ | --------------------------------------------- |
| `api_url` | Bhatti API endpoint. |
| `auth_token` | The user’s API key from `bhatti user create`. |
## Data directory layout
[Section titled “Data directory layout”](#data-directory-layout)
```plaintext
data_dir/
├── state.db SQLite database (WAL mode)
├── age.key Secret-encryption key (auto-generated on first secret set)
├── id_ed25519, id_ed25519.pub SSH keypair (auto-generated on first start; for guest agent identity)
├── .latest-version Cache for `bhatti version`'s GitHub-release check (~/.bhatti only)
├── images/
│ ├── vmlinux-<arch> Kernel
│ ├── rootfs-minimal-<arch>.ext4
│ ├── rootfs-browser-<arch>.ext4
│ └── ... (other tier images)
├── sandboxes/<id>/
│ ├── rootfs.ext4 CoW copy of the base image
│ ├── config.ext4 Config drive (env, secrets, files; ~1 MB)
│ ├── vol-<name>.ext4 Hard-linked attached volumes (jailer mode only)
│ ├── firecracker.sock FC API socket
│ ├── mem.snap Memory snapshot (when stopped)
│ └── vm.snap VM state snapshot (when stopped)
├── volumes/<user_id>/
│ └── <name>.ext4 Standalone volume images
├── snapshots/<user_id>/
│ └── <name>/ Named-snapshot bundle (rootfs copy + mem.snap + vm.snap)
├── jails/firecracker/<id>/ Jailer chroots, when `firecracker_jailer` is set
└── certs/ ACME certificate cache (when `domain.acme_email` is set, no wildcard cert)
```
`<arch>` is `arm64` on aarch64 hosts, `amd64` on x86\_64.
## Validating a config
[Section titled “Validating a config”](#validating-a-config)
The daemon prints which file it loaded on startup:
```plaintext
config loaded path=/etc/bhatti/config.yaml
```
If the path is empty, no config file was found and built-in defaults are used. The CLI client doesn’t print this; check with `bhatti version` (which uses the loaded `api_url`).
# Self-Hosting
> Install the bhatti server on your hardware, add teammates, set a custom domain, back up the data directory.
bhatti v1 (Firecracker) is frozen
Active development moved to **v2 (krucible)** — see the [current docs](/docs/self-hosting/). **`bhatti.sh/install` now installs v2.** To install the frozen Firecracker v1, use the pinned command:
```bash
curl -fsSL https://raw.githubusercontent.com/sahil-shubham/bhatti/firecracker/scripts/install.sh | sudo BHATTI_VERSION=v1.11.12 bash
```
The `bhatti.sh/install` snippets below are shown as originally written; prepend the pinned URL + `BHATTI_VERSION=v1.11.12` to install v1.
The [Quickstart](/v1/docs/quickstart/) covers the install in three lines and gets you to your first sandbox. This page is the longer version — what’s actually happening on your server, how to add teammates, and the operational details I’ve picked up running bhatti on real hardware.
## Requirements
[Section titled “Requirements”](#requirements)
* Linux with KVM (`/dev/kvm` must exist and be readable)
* Root access (Firecracker requires it; the daemon runs as root by default and Firecracker drops to an unprivileged UID per VM via the jailer — see [Architecture → jailer mode](/v1/docs/under-the-hood/architecture/#3-jailer-mode-is-the-production-path))
* 1 GB+ RAM, NVMe recommended for snapshot performance
I run on:
* Two Raspberry Pi 5s with NVMe HATs (home, integration tests)
* A Hetzner AX102 (Ryzen 9, NVMe — my main box)
Both are linked from the homepage benchmark. Anything in that ballpark or stronger works.
## Filesystem (recommended: btrfs)
[Section titled “Filesystem (recommended: btrfs)”](#filesystem-recommended-btrfs)
bhatti’s `bhatti create`, snapshot create, and snapshot resume paths all lean on `cp --reflink=auto --sparse=always` for block-device copies ([`pkg/engine/firecracker/fc.go::copyBlock`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/fc.go#L236)). On **btrfs** or **xfs** this is a metadata-only CoW clone — instant, near-zero disk. On **ext4** it falls back to a full sparse copy. Same correctness; very different time and disk cost. **The performance numbers on the homepage are measured on btrfs and don’t apply on ext4.** See [Storage](/v1/docs/under-the-hood/storage/) for the cost-by-filesystem breakdown.
The minimum viable setup is a btrfs loopback file. Do this **before** running the install script, so `/var/lib/bhatti` is already btrfs when the daemon first writes to it:
```bash
# Pre-install setup. 500 GiB is sized for tens to low-hundreds of
# sandboxes; adjust to your disk. fallocate is instant on btrfs/xfs
# hosts (just reserves space, no zero-write).
sudo fallocate -l 500G /var/lib/bhatti-btrfs.img
sudo mkfs.btrfs -f /var/lib/bhatti-btrfs.img
sudo mkdir -p /var/lib/bhatti
sudo mount -o loop,noatime,compress=zstd:1 \
/var/lib/bhatti-btrfs.img /var/lib/bhatti
echo '/var/lib/bhatti-btrfs.img /var/lib/bhatti btrfs loop,noatime,compress=zstd:1 0 0' \
| sudo tee -a /etc/fstab
```
Loopback because it works on any host — you don’t need a spare partition or a repartition. Native btrfs on `/var/lib/bhatti`’s underlying device is preferred on multi-disk hosts; the install works the same way.
If bhatti is already running on ext4 and you want to switch, you’ll need to stop the daemon and rsync the data dir across. The archived [migration recipe](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/MIGRATION-v0.5.14-btrfs.md) in the monorepo walks through that path; expect \~10–15 minutes of downtime.
xfs also supports reflink and is a fine alternative if btrfs stability is a concern. xfs lacks transparent compression — you’ll get reflink savings but not the additional \~2× from zstd on `mem.snap` files.
If you must run on ext4, the system works correctly — only performance and disk usage are worse. See [Storage → What changes on ext4](/v1/docs/under-the-hood/storage/#what-changes-on-ext4) for the concrete cost.
## Install
[Section titled “Install”](#install)
```bash
curl -fsSL bhatti.sh/install | sudo bash
```
The script will:
1. **Detect your architecture** (`arm64` or `amd64`) and the OS.
2. **Prompt for a rootfs tier** (or pass `--tier <name>` to skip the prompt; see below).
3. **Download the components** (`bhatti`, `lohar`, Firecracker + jailer, kernel, rootfs). Components that haven’t changed since a previous install are skipped automatically.
4. **Install a systemd unit** at `/etc/systemd/system/bhatti.service` and start the daemon.
5. **Create an `admin` user** with high resource caps, save its API key to `/root/.bhatti/config.yaml`, and **also save it to your user’s `~/.bhatti/config.yaml`** if you ran with `sudo` (so the CLI on the same box works without an explicit `bhatti setup`).
6. **Print the admin API key once** — save it. Anyone with this key can do anything on this server.
### Rootfs tiers
[Section titled “Rootfs tiers”](#rootfs-tiers)
| Tier | What’s in it | Size |
| ---------- | --------------------------------------- | -------- |
| `minimal` | Bare Ubuntu 24.04 | \~200 MB |
| `browser` | + Chromium, Playwright, Node 22 | \~600 MB |
| `docker` | + Docker Engine | \~550 MB |
| `computer` | + Full desktop: XFCE, KasmVNC, Chromium | \~1.5 GB |
You pick one tier as the *default* for `bhatti create` (used when you don’t pass `--image`). Other tiers are still installable later and can be selected per sandbox with `--image <tier>`.
### Non-interactive install
[Section titled “Non-interactive install”](#non-interactive-install)
For Ansible / packer / CI, skip the tier prompt with flags. The `bash -s -- ...` syntax passes flags through `curl | bash`:
```bash
# Specific tier as the default
curl -fsSL bhatti.sh/install | sudo bash -s -- --tier browser
# Default minimal, plus install every other tier
curl -fsSL bhatti.sh/install | sudo bash -s -- --tiers all
# Default docker, plus also install browser
curl -fsSL bhatti.sh/install | sudo bash -s -- --tier docker --tiers browser
```
`--tier` (singular) sets the *default* tier — what `bhatti create` uses when no `--image` is passed. `--tiers` (plural, comma-separated list or `all`) installs additional tiers alongside the default.
## Verifying the install
[Section titled “Verifying the install”](#verifying-the-install)
From the same box where you ran the install, no setup needed:
```bash
bhatti create --name test
bhatti exec test -- uname -a
bhatti destroy test
```
If the create succeeds, the daemon is healthy, the engine has KVM access, the agent is reachable, and the CLI’s config was written correctly.
## Adding teammates
[Section titled “Adding teammates”](#adding-teammates)
The remote-CLI flow that used to be Quickstart lives here.
On the server:
```bash
sudo bhatti user create --name alice --max-sandboxes 5 --max-cpus 4 --max-memory 4096
# → API key: bht_... (shown once, save it)
```
Send Alice the key over a secure channel (1Password, Signal, encrypted email — not Slack, not GitHub Issues). On her machine:
```bash
# CLI only, no sudo
curl -fsSL bhatti.sh/install | bash
# Wire it up. Either interactive or via flags.
bhatti setup --url https://your-server:8080 --token bht_...
# or:
bhatti setup
```
Now Alice can `bhatti create`, `bhatti exec`, etc. — but only against sandboxes she creates. Each user is isolated at the API layer (scoped queries) and at L2 (per-user bridge — see [Networking](/v1/docs/under-the-hood/networking/)).
For driving bhatti from agents, CI, or provisioning scripts, use the non-interactive form: `bhatti setup --url ... --token ...`. The auth test always runs and the command exits non-zero on failure, so your provisioner picks up bad credentials immediately.
## Custom domain (optional but recommended)
[Section titled “Custom domain (optional but recommended)”](#custom-domain-optional-but-recommended)
By default `bhatti publish` generates URLs at `<alias>.<your-server-ip>.nip.io` or similar — fine for testing, ugly for sharing. To get URLs like `my-app.yourdomain.com` with TLS, see [Custom domain](/v1/docs/managing/custom-domain/).
## Backups
[Section titled “Backups”](#backups)
What to back up:
* `/var/lib/bhatti/state.db` — every sandbox/user/secret/template/volume row. The daemon writes WAL, so a hot snapshot of just `state.db` may be missing recent commits; back up `state.db` + `state.db-wal` + `state.db-shm` together, or run `sqlite3 .backup` for a clean copy.
* `/var/lib/bhatti/age.key` — **the encryption key for every secret.** Lose it and every encrypted secret on the server is unrecoverable. Treat it the same way you’d treat a TLS private key.
* `/var/lib/bhatti/volumes/` — standalone volumes (the ones created with `bhatti volume create`). These are real ext4 images.
* Per-sandbox files under `/var/lib/bhatti/sandboxes/` are usually not worth backing up — sandboxes are reproducible from images and init scripts. The exception is sandboxes whose state you actually care about; for those, take a named snapshot (`bhatti snapshot create <name> --label keep`) which writes a self-contained copy under `/var/lib/bhatti/snapshots/`.
For S3-compatible volume backups, see [Volumes → backups](/v1/docs/managing/volumes/#backups).
## Updating
[Section titled “Updating”](#updating)
```bash
sudo bhatti update # bhatti + lohar + kernel + jailer
sudo bhatti update --tiers all # also pull additional rootfs tiers
```
Or just re-run the install:
```bash
curl -fsSL bhatti.sh/install | sudo bash
```
The install script is idempotent — it skips components that haven’t changed and updates the rest.
## Uninstalling
[Section titled “Uninstalling”](#uninstalling)
```bash
# Remove binaries + service, keep /var/lib/bhatti so you can reinstall
curl -fsSL bhatti.sh/uninstall | sudo bash
# Remove everything, including all sandbox state, volumes, and age.key
curl -fsSL bhatti.sh/uninstall | sudo bash -s -- --purge
```
`--purge` is destructive and unreversible — it deletes the encryption key, every secret, every sandbox, every volume.
## Where each thing lives
[Section titled “Where each thing lives”](#where-each-thing-lives)
The full layout is on the [Architecture page](/v1/docs/under-the-hood/architecture/#where-state-lives). Short version:
* `/var/lib/bhatti/state.db` — SQLite, the source of truth
* `/var/lib/bhatti/age.key` — secret encryption key (back this up)
* `/var/lib/bhatti/images/` — read-only base rootfs templates + kernel
* `/var/lib/bhatti/sandboxes/<id>/` — per-sandbox: rootfs, config drive, snapshots
* `/var/lib/bhatti/volumes/` — standalone volumes
* `/var/lib/bhatti/snapshots/` — named snapshots from `bhatti snapshot create`
* `/var/lib/bhatti/jails/` — jailer chroots (one per running sandbox)
* `/etc/bhatti/config.yaml` — daemon config (engine, listen, domain)
## Next steps
[Section titled “Next steps”](#next-steps)
* [Users & Auth](/v1/docs/managing/users/) — API key rotation, per-user limits, deleting users
* [Custom domain](/v1/docs/managing/custom-domain/) — TLS for `bhatti publish`, ACME, wildcard DNS
* [Concepts](/v1/docs/concepts/) — mental model for sandboxes and thermal states
* [Architecture](/v1/docs/under-the-hood/architecture/) — what each process does and how state flows
# Architecture overview
> A 10-minute mental model of bhatti — what runs where, what state lives where, and the four structural decisions that shape everything else.
This page is the map. It exists so that when you go into [Lohar](/v1/docs/under-the-hood/lohar-the-blacksmith/), [Thermal states](/v1/docs/under-the-hood/thermal-states/), [Networking](/v1/docs/under-the-hood/networking/), or [Decisions](/v1/docs/under-the-hood/decisions/), you already have a sense of where each fits.
If you're using bhatti, you don't need this page. If you're evaluating it for production or thinking about contributing, this is the right starting point.
## The shape
Bhatti is **two binaries** — `bhatti` on the host, `lohar` inside each microVM — talking over **TCP** with a small binary protocol.
`bhatti` is a single Go binary that does many jobs depending on how you invoke it:
* `bhatti serve` — the daemon (HTTP API, Firecracker engine, thermal manager, reverse proxy, all in one process)
* `bhatti create`, `bhatti exec`, `bhatti shell`, ... — the CLI client, talking to the daemon's HTTP API
* `bhatti user create`, `bhatti admin status` — admin commands that read the daemon's SQLite database directly (root only)
`lohar` runs as PID 1 inside every Firecracker microVM. It handles exec, file operations, PTY sessions, and pretends to be `systemctl` and `journalctl` for in-guest callers. The full story is in [Lohar](/v1/docs/under-the-hood/lohar-the-blacksmith/).
```
┌─ Host ──────────────────────────────────────────────────────────────────────┐
│ │
│ ┌─ bhatti daemon (bhatti serve, single Go process) ────────────────────┐ │
│ │ │ │
│ │ REST / WS API Engine Store (SQLite, WAL mode) │ │
│ │ :8080 Firecracker sandboxes, users, secrets, │ │
│ │ reverse proxy create / destroy templates, volumes, events, │ │
│ │ thermal manager stop / start publish_rules, fc_state │ │
│ │ exec, shell, files │ │
│ │ snapshot, restore │ │
│ │ │ │
│ └─────────────────────────────────┬─────────────────────────────────────┘ │
│ │ │
│ HTTP over Unix socket (Firecracker API) │
│ TCP over TAP (agent protocol, port 1024) │
│ │ │
│ ▼ │
│ ┌─ Firecracker microVM × N ─────────────────────────────────────────────┐ │
│ │ │ │
│ │ vmlinux rootfs.ext4 config.ext4 vol-*.ext4 │ │
│ │ │ │
│ │ ┌─ lohar (PID 1) ──────────────────────────────────────────────┐ │ │
│ │ │ │ │ │
│ │ │ TCP :1024 ── control plane (exec, files, sessions) │ │ │
│ │ │ TCP :1025 ── forward plane (port forwarding) │ │ │
│ │ │ │ │ │
│ │ │ sessions scrollback atomic file ops │ │ │
│ │ │ systemctl shim journalctl shim syslog receiver │ │ │
│ │ │ │ │ │
│ │ └──────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ tapXXXXXXXX ── brbhatti-N (per-user) ── iptables NAT │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
The daemon's HTTP API is the only public surface. Everything below it — the FC API, the agent protocol, the bridge — is internal. A client outside the host talks only to `:8080` (or `:443` in domain mode).
## Four decisions that shape the rest
These aren't all the choices in the codebase, but they're the ones you can't understand the rest of the docs without.
### 1. The daemon is a single Go process
REST/WS API, Firecracker engine, thermal manager, reverse proxy, SQLite store, rate limiter — all in one process. No microservices, no message queue, no separate workers. State lives in SQLite (see [Where state lives](#where-state-lives)).
This is a deliberately small box. One binary you can `scp`. One log to read. One process to restart. Whatever growth bhatti does, it won't be by adding more processes — it'll be by getting better at the work the single process already does.
The cost is real: one bhatti daemon manages the sandboxes on one machine. There's no clustering inside bhatti and no plans for it. If you need to scale across hosts, the way I'd do it (and the way I'm planning to, eventually) is to put a thin proxy in front of several bhatti daemons that records which daemon a sandbox was created on and routes subsequent requests for that sandbox — including wake-on-request — to the same daemon. Each bhatti stays single-process; the proxy is the only piece that knows about the fleet. That's still outside bhatti itself.
### 2. We talk to Firecracker directly, no SDK
Firecracker exposes an HTTP API over a Unix domain socket — about eight endpoints we actually use (`/machine-config`, `/drives`, `/network-interfaces`, `/vsock`, `/balloon`, `/boot-source`, `/actions`, `/snapshot/{create,load}`).
The official [firecracker-go-sdk](https://github.com/firecracker-microvm/firecracker-go-sdk) wraps those eight endpoints in \~15,000 lines of generated client code, plus a `containerd`-derived vsock helper, plus the `go-openapi` runtime, plus `cni` for networking, plus `grpc`. It's designed for the AWS Lambda use case and pulls in roughly that much machinery.
Our adapter is `pkg/engine/firecracker/fc.go` — a few HTTP helpers (`fcPut`, `fcPatch`, `fcGet`) over an `http.Transport` whose `DialContext` returns a Unix socket connection. About 20 lines.
What I lose: type-safe JSON bodies. A typo in `"vcpu_count"` shows up at runtime, not compile time.
What I get back, in order of how much it actually matters:
* **The hard parts of the FC integration aren't in the API surface itself.** Snapshot/restore sequencing, the [vm.snap path problem](#jailer-mode-and-the-snapshot-path-problem), TAP lifecycle, thermal state — none of that gets easier with the SDK. It's all in code we have to write either way.
* **When something goes wrong, the failing call is in code I wrote.** Not in 200 lines of generated middleware.
* **The integration tests catch typos within seconds.** They run on real Firecracker against real microVMs, on two Raspberry Pi 5s with NVMe HATs that I keep at home and run every PR through. Fast feedback on a real kernel is worth more than compile-time checks on a JSON shape.
### 3. Jailer mode is the production path
Firecracker's [Jailer](https://github.com/firecracker-microvm/firecracker/blob/main/docs/jailer.md) runs each FC process in a chroot, drops it to an unprivileged UID, puts it in its own PID namespace, and applies a seccomp filter. The chroot lives at `/var/lib/bhatti/jails/firecracker/<id>/` and contains a hard-linked copy of every file FC needs (kernel, rootfs, config drive, volumes, snapshot artifacts).
Without the jailer, FC runs as root and sees the whole host filesystem. A theoretical VM escape lands the attacker as root on the host with full visibility. With the jailer, a successful escape lands them as an unprivileged UID inside a near-empty chroot.
For a multi-tenant deployment, the jailer is not optional — it's the entire point of running real VMs instead of containers. The setting is `firecracker_jailer` in the server config; the install script enables it by default. Bhatti only runs on Linux with KVM ([`cmd/bhatti/engine_other.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/bhatti/engine_other.go) refuses to start anywhere else), so there's no "non-jailed macOS" path to worry about — if `bhatti serve` runs at all, you're on Linux and you should turn the jailer on.
#### Jailer mode and the snapshot path problem
Bhatti hit v1.0.0 specifically because of an incompatibility between bare and jailed snapshots.
The reason: Firecracker's `vm.snap` records the *paths* of every file the VMM had open at snapshot time. In bare mode that's an absolute host path like `/var/lib/bhatti/sandboxes/abc/rootfs.ext4`. In jailed mode it's a chroot-relative path like `/rootfs.ext4`. On resume, the VMM reopens those exact paths.
Switching modes invalidates every existing snapshot:
* Bare → jailed: `/var/lib/bhatti/...` doesn't exist inside the new chroot.
* Jailed → bare: `/rootfs.ext4` doesn't exist as an absolute host path.
There's no fix-up step that rewrites paths in `vm.snap` — it's a binary blob of FC's internal device-model state, not something bhatti can edit. So the migration plan was: destroy every snapshot, upgrade, recreate. That's a major version bump in semver terms. We did it once, called it v1.0, and the codebase has been jailer-by-default since.
Bare mode is still in the code as a fallback (it's just FC without the jailer wrapper) and is what some integration tests use. For any deployment a real user touches, jailer mode is what you want.
### 4. Per-VM mutex with capture-and-release
Concurrency is the part that bites every "single Go process" design eventually. Bhatti's pattern is small but load-bearing.
Each `VM` struct has its own `stateMu sync.Mutex` ([`pkg/engine/firecracker/engine.go:75-86`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/engine.go#L75-L86)). The engine-level `sync.RWMutex` only protects the map of VMs — it doesn't protect any individual VM's state.
The pattern every operation follows:
```go
vm.stateMu.Lock() // 1. take the lock
if vm.Thermal != "hot" { // 2. check invariants
vm.stateMu.Unlock()
return errNotHot
}
ag := vm.Agent // 3. capture references we need
vm.stateMu.Unlock() // 4. release the lock immediately
return ag.Exec(ctx, cmd) // 5. do the slow work outside the lock
```
The key move is steps 3-4: copy the references, drop the lock, then make the network call. The lock protects access to `vm.Agent` itself, not any operation that uses it.
The reason this matters: a `Shell` or `Tunnel` call lasts as long as the user's session — minutes, hours. If those calls held `stateMu` for their duration, the thermal manager couldn't pause *any* VM (because every cycle iterates the VM map and tries to take each VM's lock to read its thermal state). One slow shell would freeze idle pause / cold snapshot for everything else.
The safety argument: `vm.Agent` is only replaced during `Start()` (snapshot restore), and `Start()` takes the same `stateMu` to do it. So if a Shell goroutine grabs `vm.Agent`, releases the lock, and then `Start()` runs to completion and replaces `vm.Agent` with a new client — the Shell goroutine still has a valid reference to the *old* Agent's TCP connection. That connection is good until the Firecracker process is killed, which happens only on `Stop()` or `Destroy()`. By that point, both `Stop()` and `Destroy()` will have closed the connection, the Shell will see a read error, and clean up.
So: long-running operations don't block the system, and they don't explode when state is updated underneath them — they just see the underlying connection close and exit cleanly. The pattern is in every method on the Firecracker engine that touches an agent.
## Where state lives
State is in two places: SQLite, on disk; Firecracker memory and disk images, on disk per-sandbox.
**SQLite.** One database file at `/var/lib/bhatti/state.db` on a default server install. Tables:
* `sandboxes` — id, name, owner, status, IP, MAC, engine ID
* `fc_state` — per-VM Firecracker config (CPU, memory, drive paths, snapshot paths, agent token), serialized as JSON; this is what recovery reads
* `users` — name, hashed API key (SHA-256), per-user limits, subnet index
* `secrets` — name, encrypted value (age), scoped per user
* `templates`, `volumes`, `publish_rules`
* `events` — audit log
* `task_progress` — async tasks (image pull)
WAL mode is on so background workers (the thermal manager, port scanner, metrics snapshotter) don't block HTTP request handlers and vice versa.
**Per-sandbox.** Under `/var/lib/bhatti/sandboxes/<id>/`:
```
rootfs.ext4 CoW copy of the base rootfs
config.ext4 1MB ext4 with hostname, env, files, volume specs
vol-<name>.ext4 attached volumes (per-sandbox; shared volumes are
elsewhere — see below)
firecracker.sock FC API Unix socket (only while the VM is running)
mem.snap memory snapshot (only when cold)
vm.snap VM state snapshot (only when cold)
```
The CoW copy uses `cp --reflink=auto`. On btrfs or XFS this is instant and near-zero disk; on ext4 it falls back to a full copy. If you're benchmarking `bhatti create`, run on btrfs. See [Storage](/v1/docs/under-the-hood/storage/) for the cost-by-filesystem breakdown, compression ratios, and the page-cache story for cold wake.
**Other disk locations under `/var/lib/bhatti/`:**
* `images/` — read-only base rootfs templates (`rootfs-minimal-arm64.ext4` etc.) and the kernel
* `volumes/<name>.ext4` — standalone volumes (created by `bhatti volume create`, attachable to any sandbox)
* `snapshots/<user>/<name>/` — named snapshots from `bhatti snapshot create` (memory + disk artifacts, fully self-contained)
* `jails/firecracker/<id>/` — jailer chroots (when jailer is on)
* `age.key` — the X25519 private key used to encrypt user secrets. It's generated *the first time anyone calls `bhatti secret set`*, not on daemon startup ([`pkg/secrets/age.go:13-37`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/secrets/age.go#L13-L37)). **Back this up.** If you lose it, every encrypted secret on the server is unrecoverable.
The 1 MB config drive is sized that way because ext4's minimum filesystem size is 1 MB — anything smaller fails to format. The contents (\~few hundred bytes of JSON describing hostname, env vars, volume mounts) would fit in less, but the filesystem can't.
## Recovery on startup
When the bhatti daemon starts — fresh boot, after a crash, after a `systemctl restart` — every Firecracker process from the previous run is dead. The daemon's job at startup is to look at every sandbox row in SQLite and decide, for each one, whether it can still be brought back.
There are only two things that need to be true to bring a sandbox back:
1. **The row is still in `sandboxes`** — we know the sandbox exists and we have its config in `fc_state`.
2. **The snapshot files exist on disk** (`mem.snap` and `vm.snap`) — we have somewhere to restore the VM from.
If both are true, the sandbox is marked `stopped` in memory and is ready to wake on the next API call. If the row exists but the snapshot doesn't, the sandbox is marked `unknown` — a clear error on the next operation, and a hint to the operator to dig in.
The normal case is the boring one: the daemon was shut down cleanly (`systemctl stop bhatti`), `SnapshotAll` ran, every running sandbox got a snapshot, and on next startup every sandbox comes back as `stopped`. No data loss, no manual recovery.
The abnormal cases are where it gets interesting: a power cut between snapshot and disk flush, a btrfs subvolume that disappeared, a half-finished `bhatti destroy` that left a row but cleaned up the files. In each case the sandbox lands in `unknown`, the operator sees a `recovery: snapshot missing` warning in the log, and they can decide whether to recreate it. Volume data is on separate ext4 images and survives the loss of the memory snapshot — so the recovery path even from `unknown` is detach volumes, destroy, recreate, reattach.
After the sandbox loop, the engine cleans up orphans: TAP devices, user bridges, stale snapshot temp directories, `.ext4` files with no row in the volumes table. The order matters: sandboxes load first so that cleanup knows which TAPs and bridges *should* still exist.
## What's not on this page
This was the map. The depth is in the topic pages:
* [Lohar — the agent inside every VM](/v1/docs/under-the-hood/lohar-the-blacksmith/) — PID 1, the systemctl shim, sessions, file ops
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — hot / warm / cold, the balloon trick, snapshot mechanics
* [Networking](/v1/docs/under-the-hood/networking/) — per-user bridges, the six iptables rules, the ARP trick, kernel `ip=`
* [Wire protocol](/v1/docs/under-the-hood/wire-protocol/) — frame format, all frame types, atomic writes
* [Firecracker engine internals](/v1/docs/under-the-hood/engine/) — the actual API call sequence during create, jailer specifics, rate limiters
* [Decisions & learnings](/v1/docs/under-the-hood/decisions/) — the bugs we paid for, in narrative form
# Decisions & learnings
> Real bugs, real bad afternoons, and what they taught us. The kind of engineering log most projects don't ship — but the one I'd want to read if I were evaluating bhatti.
Every system has a paper trail of bad afternoons. This page is bhatti's.
The decisions on the other Under-the-Hood pages are summarised at the top of each — this page tells the *story* behind a few of them in more depth than fits on a how-it-works page. If you're evaluating bhatti for production, this is the page I'd read.
The order is roughly chronological with respect to when each problem hit me, not by importance.
## The 1-second SYN retransmit
`bhatti create` on a Pi 5 with btrfs takes about 1.2 seconds. About **1 second of that** is a single TCP SYN retransmission timeout. I spent a long time before I understood why.
The investigation lives in [`docs/archive/INVESTIGATION-create-performance.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/INVESTIGATION-create-performance.md). Phase-instrumented every step of the create pipeline. Got this:
```
Phase Time What
───────────────── ───── ────
rootfs_copy 14ms btrfs reflink clone
lohar_inject 93ms loop mount + cp + umount
network 34ms IP alloc + TAP + ARP flush
config_drive 14ms mke2fs -d
fc_start (jailer) 21ms chroot + FC process + socket
fc_api + InstanceStart 31ms HTTP PUTs + boot API call
─────
Host work total: 207ms
TCP SYN wait: 1005ms ← here
AUTH + exec: 18ms Agent responds
─────
WaitReady total: 1023ms
```
The host-side work is 207 ms. The remaining \~1000 ms is the TCP SYN-retransmit timer.
The `rootfs_copy` row is on btrfs with reflink — 14 ms is the metadata-only `BTRFS_IOC_CLONE` ioctl. On ext4 the same `cp --reflink=auto` falls back to a real sparse copy and the row jumps to \~320 ms; the host-side total goes from 207 ms to \~500 ms. See [Storage](/v1/docs/under-the-hood/storage/) for the filesystem story that sits underneath every "how fast is `bhatti create`?" conversation.
What's happening: after `InstanceStart`, the host immediately starts polling TCP :1024 with `exec true` to detect when the agent is up. The agent isn't up yet — the kernel is still booting. The host sends a SYN. No response. Linux's default initial SYN RTO is 1 second ([RFC 6298](https://www.rfc-editor.org/rfc/rfc6298), `net.ipv4.tcp_syn_retries`, `net.ipv4.tcp_synack_retries`). After 1 second, retransmit. By then, the agent is up; the second SYN succeeds.
Two parts of the fix that *do* exist in code:
* **Pre-populated permanent ARP entry** ([`pkg/engine/firecracker/create.go:386-395`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/create.go#L386-L395)). Without this, the *first* probe is wasted on ARP resolution (another 1-second penalty). With it, the SYN goes out immediately with the right MAC. See [The ARP trick](/v1/docs/under-the-hood/networking/#the-arp-trick).
* **Stale ARP flush before allocation** ([`create.go:130-135`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/create.go#L130-L135)). Without this, IP reuse within `gc_stale_time` (60 s) sends SYNs to the dead VM's MAC, also losing seconds.
What's *not* done yet, that we know would help:
* Delay the first WaitReady probe by \~200 ms after `InstanceStart`. Skip the doomed first SYN.
* Tune the initial SYN RTO via `ip route` on the bridge.
* Use a faster probe (vsock on cold boot — except vsock has its own problems; see below).
I haven't shipped these because they're optimization for an already- acceptable wall-clock. But the existence of a 1-second timer waiting for a guest kernel to boot is a real, weird, unlovely thing — and you will see it in your benchmarks.
## The rory incident
April 2026. A user's persistent sandbox named `rory` came back from a restore with a corrupted virtio ring buffer. The agent was unreachable. The VM was wedged. We had to destroy it and lost the working state of the sandbox.
Three failures compounded into one bad outcome ([`docs/archive/PLAN-reliability.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-reliability.md), [`PLAN-snapshot-reliability-fixes.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-snapshot-reliability-fixes.md)):
* **Diff snapshot corruption.** `mem.snap` was a Diff — only dirty pages — and the dirty page bitmap was incomplete. Firecracker tracks dirty pages via KVM's bitmap, but certain virtio device writes happen in Firecracker's userspace device-model code and don't go through KVM. Those writes don't trigger the bitmap. Restoring a Diff snapshot then loads pages from a stale base and any in-flight virtio ring buffer state gets clobbered.
* **No snapshot verification** — we'd written a corrupt mem file and marked the snapshot as good without checking.
* **Volume attachment metadata wasn't persisted on snapshot resume** — `vm.Volumes` was empty when a sandbox came back from a named snapshot, so even when we tried to recover, the volumes were unreachable.
The fixes:
* `track_dirty_pages: false` everywhere ([`pkg/engine/firecracker/create.go:307-318`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/create.go#L307-L318)). Every snapshot is now a Full snapshot. \~500 ms slower per snapshot on NVMe; correct in every case. See [Why all snapshots are Full](/v1/docs/under-the-hood/thermal-states/#why-all-snapshots-are-full).
* `verifySnapshotArtifacts` after every `/snapshot/create`. Wrong size, bad path, corrupted JSON — error out before marking the snapshot good.
* `vm.Volumes` populated on snapshot resume ([`PLAN-snapshot-reliability-fixes.md` Bug N1](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-snapshot-reliability-fixes.md)), with reciprocal `volume_attachments` rows in SQLite.
The audit also found 17 other bugs — race conditions, missing defer-close, type assertions that would panic on certain SQLite inputs. The `PLAN-reliability.md` covers all of them. v0.7 was the release that fixed the entire batch.
The lesson I keep coming back to: **the cost of correctness is much smaller than the cost of one lost user sandbox.** Diff snapshots saved \~500 ms per snapshot. Losing rory cost a user a working state, me a week of debugging, and a release-line of compounding bug fixes. Not even close.
## The systemd snapshot/restore problem
I had lohar running as a systemd-managed service for a while. Fresh boot worked. Snapshot/restore broke in a way that took a week to understand.
The setup: a regular Ubuntu rootfs with systemd as PID 1, lohar running as `lohar.service` (`Type=simple`, `Restart=always`). On a fresh boot, `systemctl is-system-running` returned `running`, lohar accepted connections, `bhatti exec` worked.
Then `bhatti stop` (snapshot to disk) and `bhatti start` (restore).
The restore looked successful: FC's API returned ok, the guest kernel resumed, lohar's TCP listener was still bound to port 1024 according to `ss -tlnp` from inside the VM. The first `bhatti exec` after restore would succeed. Every subsequent exec would hang forever.
What I observed:
* The kernel-level TCP state was fine. Three-way handshake completed, packets visible in `tcpdump`.
* The connection sat in the kernel's accept queue.
* The Go runtime — sitting on top of `epoll_wait` — was not waking up. The goroutine blocked on `Accept()` never got scheduled.
I don't have a clean explanation for *why* this happens specifically when lohar is a child of systemd, and not when it's PID 1 itself. My guess: something about how systemd manages file descriptors, control groups, or its own epoll sets across PID 1 leaves the resumed Go process holding a poller in a state Go's runtime doesn't recover from. That's speculation, not diagnosis.
What I know empirically:
* Reproduced on Firecracker 1.14.0 and 1.15.1.
* Reproduced with `Restart=no` and `Restart=always`.
* Lohar as PID 1 (no systemd) — never reproduced, in CI or production.
So lohar stayed PID 1, and we built a [systemctl shim](/v1/docs/under-the-hood/lohar-the-blacksmith/#the-systemctl-shim) to satisfy the package machinery. **If you've traced this further or think the diagnosis is wrong**, please [open an issue](https://github.com/sahil-shubham/bhatti/issues). The shim works, but real systemd would be less code to maintain.
## Why TCP, not vsock, after restore
Firecracker's vsock implementation is supposed to be the canonical host↔guest communication channel. We started there. It works perfectly during normal operation. After snapshot/restore, it breaks.
What I see post-restore: vsock connections complete the host-side handshake (the `CONNECT <port>` reply succeeds) but never reach the guest's vsock listener. From the host, the connection looks "connected." From the guest, no `accept()` ever fires. Reads block. Writes are accepted by the kernel but the guest never sees them.
Tested with kernel 5.10, 6.1; Firecracker 1.6.0 through 1.14.0. The break is consistent and lives in the Firecracker/vsock layer, not in our code. As a public corroboration point, SlicerVM dropped their suspend/restore feature in v0.1.108 — apparently for the same upstream cause ([archive notes on Firecracker vsock behavior post-restore](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/slicer-learnings.md)).
The fix: use TCP over the TAP network instead. Virtio-net survives snapshot/restore cleanly. Lohar listens on both vsock and TCP at ports 1024/1025, and the host always uses TCP after restore ([`pkg/engine/firecracker/lifecycle.go:419`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go#L419)). TCP adds \~0.1 ms of latency over vsock; negligible.
In practice we use TCP for cold boot too, because the simplification of "agent client always uses TCP, regardless of state" is worth more than the trivial vsock-faster-on-cold-boot saving. The vsock listener in lohar is still set up but the host never dials it.
## The Cloudflare Tunnel disconnect
A user reported their `bhatti shell` dropping silently when they ran a long-running command through `api.bhatti.sh` ([`docs/archive/PLAN-shell-sessions.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-shell-sessions.md)). Screenshot: no error, no "detached" message, no bash prompt — just a silent return to the shell on their Mac.
Root cause:
* They ran `bhatti shell rory`, started `hermes gateway` (a daemon that prints a startup banner then waits for events).
* Cloudflare Tunnel has a WebSocket idle timeout around 100 seconds. No traffic in either direction = connection killed.
* The CLI's `conn.ReadMessage()` returned an error. The error path was silent — `defer term.Restore()` ran, terminal mode reset, but nothing got printed.
* `bhatti shell rory` again created a *new* session. The old session was still alive inside the VM with `hermes gateway` running and scrollback accumulating, but there was no way to get back to it.
Three problems compounded:
1. The WebSocket layer didn't have ping/pong keepalives, so idle connections were getting killed by intermediaries.
2. The CLI didn't print anything on disconnect, so users didn't know what happened.
3. There was no session reattach. `bhatti shell` always created a new session.
The fix was a long PR ([`docs/archive/PLAN-shell-sessions.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-shell-sessions.md)) with a 12-bug inventory. The notable ones:
* **WebSocket ping/pong keepalives** every 30 seconds, both client and server.
* **Concurrent WebSocket write race** — the CLI's main loop and its resize-handling goroutine both wrote to the connection without coordination. Replaced with a single writer goroutine fed by a channel.
* **Session reattach** — `bhatti shell <name>` now attaches to the most recent live session (if any) rather than always creating new.
* **Disconnect message** — CLI prints `[disconnected: <reason>; reconnect with bhatti shell <name>]` on the way out.
* **Scrollback ring buffer thread safety** — was being accessed concurrently without a mutex; reads could get torn.
The takeaway: **production network conditions kill connections that look idle to intermediaries.** If you're building over WebSocket, ping/pong is not optional. And the user-visible behavior on disconnect is part of the API — silently exiting is an outage from the user's point of view.
## The public proxy rate-limiter recalibration
The first version of the public proxy used a per-alias token bucket with burst 100 and refill 200/min as the primary rate limiter. Logic: "each published URL gets its own quota."
Then a real Vite app with hot module reload generated 90+ requests on a single page load, exhausted the bucket on the first reload, and served 18 of 91 requests as 429s — a blank page on reload.
Documented in [`docs/archive/PLAN-public-proxy-ratelimit.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-public-proxy-ratelimit.md). Two things were wrong:
1. **The aggregation level was wrong.** Standard reverse-proxy practice (nginx `limit_req_zone`, Cloudflare, AWS WAF) is per-source-IP rate limiting. Per-destination is a *secondary* aggregate against distributed attacks, not the primary mechanism. Browsers HMRing are one source IP, not one URL.
2. **The numbers were too low.** 200 requests/min ≈ 3.3 req/s. A modern bundler-driven page exceeds that during a single load.
The fix: per-source-IP primary limit (much higher burst, much higher refill), per-alias as a secondary cap (very high — only catches runaway loops), global as a third (host-level safety net).
This is the kind of decision that's hard to get right in advance, because the number depends on the workload. We had reasonable defaults for an API server. The defaults were wrong for a frontend. The calibration came from one user with one Vite app.
## Pure-Go SQLite
The choice was `mattn/go-sqlite3` (CGo binding to the C SQLite library) vs `modernc.org/sqlite` (pure-Go translation of SQLite's C code).
The CGo version is faster and more compact. The pure-Go version is \~10% slower and the binary is \~3 MB larger. For metadata CRUD (sandboxes, users, secrets, events), the speed difference is irrelevant.
What matters: cross-compilation. Bhatti is built on a Mac and deployed to a Pi. With CGo, this requires a cross-compiler toolchain (`aarch64-linux-gnu-gcc`), careful library management, and different build commands per platform. It also means the binary is dynamically linked against `libc` and isn't fully portable across distros.
With pure-Go SQLite, `GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build` produces a static binary that runs on any Linux. Same binary works on Pi, Hetzner, Graviton, an Alpine container, an Ubuntu host, anything.
The 3 MB and 10% are bought, gladly, for the build simplicity.
## Per-user bridges (replacing a single shared bridge)
The first version of bhatti had one Linux bridge — `brbhatti0` — and all VMs across all users shared it. Subnet `192.168.137.0/24`. One masquerade rule. Simple to operate. *Insecure* once you actually have multiple tenants, because at L2, alice's VM could ARP-scan the bridge and reach bob's VM directly. Iptables FORWARD rules between guests were the only barrier, and a misconfigured rule meant cross-tenant traffic.
Per-user bridges ([`pkg/engine/firecracker/network.go:23-52`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/network.go#L23-L52)) push the isolation down to the kernel's bridge forwarding. alice's VMs are on `brbhatti-1`, bob's on `brbhatti-2`. The kernel doesn't forward frames between them. Even if alice's VM is compromised at the kernel level, it can only see alice's other VMs.
The migration: `cleanupOldBridge()` runs once on engine startup, removes `brbhatti0` and the legacy 192.168.137 NAT rule, and existing sandboxes get reattached to per-user bridges on their next start. We made the migration silent because the upgrade path needed to "just work" for existing single-user deployments.
The lesson: **start with the security model you'll need**, not the simplest one that works for now. A bridge-per-tenant is one or two extra lines of bridge management; retrofitting tenant isolation is a release-cycle of careful migration code.
## Server-side file truncation
Coding agents read files. They almost always truncate — typically to the first 2000 lines or 50 KB. We discovered this by watching what clients actually did with our `FILE_READ_RESP` data: they'd ask for a 100 MB log file and immediately throw away 99.95% of it on the client side. We were paying full bandwidth for almost no information.
The fix is small. `FILE_READ_REQ` accepts `offset`, `limit`, `max_bytes`. Lohar reads line-by-line with `bufio.Scanner` and stops at whichever limit hits first. The response includes the *full* file size so the consumer knows whether content was truncated. See [wire protocol — file read](/v1/docs/under-the-hood/wire-protocol/#file-read-protocol).
Performance: a truncated read on a 10 K-line file is about 4.5× faster at p50 than a full read of the same file. For larger files, proportionally bigger.
Why not `head -n` via exec? Fork-exec, pipe setup, shell argument parsing — all of that is overhead the file protocol path skips. A 1 KB file read takes about 472 µs in the benchmark; the equivalent exec is more like 1 ms. For a coding agent that does hundreds of file reads per task, this adds up.
The general lesson: **the client knows what it wants; tell the server.** Even when bandwidth is "free" inside a single host, the allocations, copies, and discards still cost.
## Sessions everywhere
Most sandbox systems split exec and shell into different concepts. Bhatti unifies them: every TTY exec is a session. There's no separate "shell" code path. A shell is just a TTY exec of `/bin/zsh` or `/bin/bash`.
Three reasons this fell out of real product needs:
1. **Init scripts need to be attachable.** When you create a sandbox with `--init "npm install && npm run build"`, that command runs as a session named `init`. You can attach to it from the host to watch progress. If exec weren't a session, the init output would be invisible from the host.
2. **Shells need to survive proxy disconnects** (the Cloudflare Tunnel story above).
3. **Snapshot/restore needs to preserve interactive shells.** A shell running a long-running command when the VM is snapshotted should be the same shell after restore. Sessions are the data structure that survives that round-trip.
Trade-off: every TTY session allocates a 64 KB scrollback buffer. With 100 concurrent sessions per VM, that's 6.4 MB. We cap at 20 sessions per VM ([`cmd/lohar/handler.go:50`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/handler.go#L50)) which keeps it bounded.
## When kasmvnc.service couldn't restart
The computer tier shipped in v1.11.9 — XFCE + KasmVNC + Chromium, all running as systemd units managed by the shim. First boot worked end-to-end (desktop renders, password generation runs once, screenshot works). Restarting `kasmvnc.service` did not. The unit transitioned to `failed` and the journal showed:
```
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
Fatal server error:
Cannot establish any listening sockets - Make sure an X server isn't
already running
```
`Xkasmvnc` from the previous run was still alive, holding the abstract Unix socket `@/tmp/.X11-unix/X99`. No file-system cleanup frees an abstract socket — only killing the holder does.
`systemctl stop kasmvnc` *should* have killed it. The shim does `cgroup.kill` on the unit's cgroup, which the kernel translates into SIGKILL for every PID in `cgroup.procs`. Why didn't `Xkasmvnc` go down?
```
$ sudo cat /proc/$(pgrep -x Xkasmvnc)/cgroup
0::/
$ sudo cat /sys/fs/cgroup/system.slice/kasmvnc.service/cgroup.procs
(empty)
```
`Xkasmvnc` was in the **root** cgroup, not the unit's. `cgroup.kill` on the unit's cgroup had nothing to kill.
The shim's old `startDaemon` called `cmd.Start()` and *then* wrote the daemon's PID into the unit's `cgroup.procs`. The window between those two operations is microseconds, but `Xkasmvnc` uses `daemon(3)` — fork, parent exits, child detaches — during startup. The child fork happened during the window, before the placement write, so the child inherited whatever cgroup the parent was in (lohar's, i.e. root). Once the parent exited, the placement write moved only the dead parent's PID. The detached `Xkasmvnc` was forever in the wrong cgroup.
The same shape applies to any double-forking daemon: classic Apache, `dbus-daemon` in default mode, `nginx` without `daemon off`. v1.11.9 shipped the contract bug — every future tier author would have to audit whether their daemon double-forks during startup, and we'd have no way to enforce it without reading every package's source.
The fix is to remove the race entirely by placing the cgroup before any fork can happen. Real systemd does this with `clone3(CLONE_INTO_CGROUP)` — a single syscall that forks and places atomically. Go's runtime doesn't expose `clone3`; reproducing it means reimplementing `syscall.ForkExec`'s post-fork setup (close-on-exec, signal-handler reset, credentials, working dir) under `syscall.ForkLock`. That's real surgery on code where bugs are silent and only surface under load.
The alternative is a thin one-purpose helper. The supervisor invokes `lohar spawn` instead of the daemon directly; `spawn` writes its own PID into `<cgroup>/cgroup.procs` and then `execve`s into the daemon. `execve` preserves PID **and** cgroup membership, so the daemon — same PID throughout — is already in the unit's cgroup when it forks anything later. About 60 lines of plain Go ([`cmd/lohar/spawn.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/spawn.go)) gets the same correctness guarantee as `clone3` with code anyone can read in one sitting. The dispatch is on `argv[1]` (`/proc/self/exe spawn ...`) rather than the busybox-style symlink we use for `systemctl` and `journalctl`, because `lohar spawn` is a private supervisor primitive — not something a user should be wrapping their own commands with. `clone3` stays open as a future migration if the extra `execve` ever shows up in profiling.
Shipped in v1.11.10. Every existing unit file works unchanged. The operator-visible difference: `systemctl stop`/`restart` now works for every kind of daemon, including the ones that fork during startup. See [How services are spawned](/v1/docs/under-the-hood/lohar-the-blacksmith/#how-services-are-spawned) for the mechanism.
## What I'd do differently
Things I'd revisit if I were starting bhatti from scratch today:
* **Use a real init for the systemctl shim from the start.** I'm reasonably happy with the shim (it's small, it covers the surface Debian's tools actually call), but I went through several rounds of "what about this directive too?" before it stabilized. Looking at it again, I'd start from a small supervisor like [s6-rc](https://www.skarnet.org/software/s6-rc/) and add a thin systemctl-compatible wrapper. The shim would still exist, but the service-supervisor code would be battle-tested rather than home- rolled.
* **Snapshot verification from day one.** I shipped Diff snapshots without verification, and the rory incident is what it cost. The verification logic is \~50 lines. Not having it from the start was a real mistake.
* **Per-source-IP rate limiting on the public proxy from day one.** Per-destination was the wrong default; "match what nginx does" is the boring-and-correct answer.
* **TLS for the agent protocol.** Right now the agent protocol uses a 16-byte hex token over plaintext TCP. The TCP runs on a private bridge that VMs from other users can't reach (per-user bridges enforce this), and the host always initiates so there's no certificate-pinning equivalent on the guest side. Still, TLS would give us defense in depth. Haven't done it yet because the threat model doesn't demand it on a single-host deployment, but it should be on the table for any real multi-host setup.
## Where to go next
If you want to read more like this, the full archive is in [`docs/archive/`](https://github.com/sahil-shubham/bhatti/tree/main/docs/archive) in the repo. Plans, post-mortems, write-ups on the Linux / Firecracker / cloud-init behaviors that drove our design choices (with cross-references to publicly documented incidents at other orchestrators in the space — SlicerVM, fly.io's Sprites — where their reports matched ours), and the migrations that got us to v1.7.
* [Architecture](/v1/docs/under-the-hood/architecture/) — how all the pieces fit together
* [Lohar](/v1/docs/under-the-hood/lohar-the-blacksmith/) — the agent inside every VM
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — pause / snapshot / restore in detail
* [Networking](/v1/docs/under-the-hood/networking/) — bridges, the ARP trick, the kernel `ip=` solve
* [Wire protocol](/v1/docs/under-the-hood/wire-protocol/) — frames, ports, atomic writes
# The Firecracker engine: HTTP API, jailer, rate limits
> How bhatti talks to Firecracker — the actual API calls during boot, the jailer setup, rate limiter configuration, and the implementation choices behind it.
This page is the lower-level companion to [Architecture](/v1/docs/under-the-hood/architecture/). It walks through what the Firecracker engine actually does to boot a VM, what HTTP calls it sends, what the jailer setup looks like, and how the rate limiters work.
If you're using bhatti, you don't need this page. If you're hacking on the engine, extending it, or debugging a weird FC-level failure, this is the reference.
## Design decisions on this page
1. **No Firecracker SDK.** We talk directly to FC's Unix-socket HTTP API with \~20 lines of helpers. The reasoning lives on the [Architecture page](/v1/docs/under-the-hood/architecture/#2-we-talk-to-firecracker-directly-no-sdk); this page covers what the API surface actually looks like in code.
2. **Jailer mode is opt-in via config.** Without jailer, FC runs as root. With jailer, each VM's FC process drops to an unprivileged UID inside a chroot. Single-user dev: skip the jailer. Multi-tenant: turn it on. See [Jailer mode](#jailer-mode).
3. **Rate limiters are off by default.** Firecracker's per-drive and per-network rate limiters add overhead and aren't useful on a single-user box. We configure them only when you set them. See [Rate limiting](#rate-limiting).
4. **We never use FC's diff snapshots.** `track_dirty_pages: false` in machine-config means every snapshot is Full. The full reasoning is in [Thermal states](/v1/docs/under-the-hood/thermal-states/#why-all-snapshots-are-full).
## The HTTP adapter
Bhatti's entire FC-API layer is a Unix-socket-aware `http.Client` and three helpers. From [`pkg/engine/firecracker/fc.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/fc.go):
```go
func fcAPIClient(socketPath string) *http.Client {
return &http.Client{Transport: &http.Transport{
DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
var d net.Dialer
return d.DialContext(ctx, "unix", socketPath)
},
}}
}
func fcPut(ctx context.Context, c *http.Client, path, body string) error {
req, _ := http.NewRequestWithContext(ctx, "PUT",
"http://localhost"+path, strings.NewReader(body))
req.Header.Set("Content-Type", "application/json")
resp, err := c.Do(req)
// ... check status, read body on error, done
}
```
Plus `fcPatch` and `fcGet` along the same lines. Every Firecracker API call goes through one of these three. JSON bodies are built with `fmt.Sprintf` from the call site (yes, really — see the boot sequence below for what that looks like in practice).
## Boot sequence: what FC sees
A `bhatti create` walks through a fixed sequence of API calls. The order matters — FC enforces it (you can't set drives before machine-config is set, you can't `InstanceStart` before drives are registered).
In `pkg/engine/firecracker/create.go`, each phase emits a `phase()` marker so the create timing is visible if you crank logging up. The sequence:
```
1. Write bootArgs to /boot-source
PUT /boot-source
{"kernel_image_path":".../vmlinux-arm64",
"boot_args":"reboot=k panic=1 pci=off 8250.nr_uarts=0
init=/usr/local/bin/lohar quiet loglevel=0
ip=10.0.1.2::10.0.1.1:255.255.255.0::eth0:off:1.1.1.1:8.8.8.8:"}
2. Configure rootfs drive (read-write)
PUT /drives/rootfs
{"drive_id":"rootfs","path_on_host":".../rootfs.ext4",
"is_root_device":true,"is_read_only":false}
3. Configure machine
PUT /machine-config
{"vcpu_count":2,"mem_size_mib":1024,
"track_dirty_pages":false,"huge_pages":"None"}
4. Configure entropy device (so guests don't block on getrandom())
PUT /entropy
{"rate_limiter":{"bandwidth":{"size":1024,"one_time_burst":8192,"refill_time":100}}}
5. Configure balloon device
PUT /balloon
{"amount_mib":0,"deflate_on_oom":true,"stats_polling_interval_s":5}
6. Configure config drive (read-only)
PUT /drives/config
{"drive_id":"config","path_on_host":".../config.ext4",
"is_root_device":false,"is_read_only":true}
7. Configure each volume drive (vdc, vdd, …)
PUT /drives/vol-<name>
{"drive_id":"vol-<name>","path_on_host":".../vol-<name>.ext4", ...}
8. Configure network interface
PUT /network-interfaces/eth0
{"iface_id":"eth0","host_dev_name":"tapXXXXXXXX","guest_mac":"02:..."}
9. Configure vsock (still configured even though we don't use it post-boot)
PUT /vsock
{"vsock_id":"vsock0","guest_cid":3,"uds_path":".../vsock.sock"}
10. Start the instance
PUT /actions
{"action_type":"InstanceStart"}
```
Steps 4 (entropy) and 5 (balloon) are easy to miss but matter. Without the entropy device, the guest's `getrandom()` blocks at boot waiting for entropy that the kernel never gathers (microVMs don't have a physical entropy source). Without the balloon, the [balloon trick](/v1/docs/under-the-hood/thermal-states/#the-balloon-trick) during hot→warm doesn't work.
After `InstanceStart`, the host pre-populates the [ARP cache](/v1/docs/under-the-hood/networking/#the-arp-trick) with the guest's MAC and starts polling TCP :1024 with `exec true`. When the agent answers, the create returns to the user.
## Jailer mode
[Firecracker's Jailer](https://github.com/firecracker-microvm/firecracker/blob/main/docs/jailer.md) is the recommended way to run FC in production. It chroots each VM's FC process into a private directory, drops to an unprivileged UID/GID, and applies a seccomp filter. Bhatti's jailer integration is in [`pkg/engine/firecracker/jail.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/jail.go).
When `firecracker_jailer` and `jail_uid`/`jail_gid` are set in `config.yaml`, the engine:
1. Creates the chroot at `<data_dir>/jails/firecracker/<sandbox_id>/`.
2. Resolves all file references — kernel, rootfs, config drive, volumes — to chroot-relative paths. The actual files don't need to move; jailer hardlinks them into the chroot.
3. Spawns FC as `jailer ... --exec-file /path/to/firecracker --uid <uid> --gid <gid> --id <sandbox_id> --chroot-base-dir <data_dir>/jails ...`.
4. Talks to FC's API socket through the chroot path.
For snapshot operations under jailer, FC writes the snapshot artifacts inside the chroot, then bhatti moves them out atomically after FC exits — see [`pkg/engine/firecracker/lifecycle.go:84-94`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go#L84-L94).
Without jailer, FC runs as root with full host visibility. Single-user dev: this is fine. Multi-tenant: turn it on. The jailer hardens the isolation between FC processes — a kernel exploit that breaks out of KVM still has to get past seccomp + UID + chroot before reaching anything that matters.
## Rate limiting
Firecracker supports per-drive and per-network bandwidth and IOPS rate limiters ([FC rate limiter docs](https://github.com/firecracker-microvm/firecracker/blob/main/docs/design.md#io-storage-networking-and-rate-limiting)). Bhatti's wrapper is in [`pkg/engine/firecracker/engine.go:21-34`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/engine.go#L21-L34):
```go
type RateLimitConfig struct {
NetBandwidthBytes int64 // bytes/s per direction
NetBurstBytes int64 // one-time burst bytes
DiskBandwidthBytes int64 // bytes/s
DiskIOPS int64 // ops/s
}
```
Defaults are zero, which means **disabled**. We don't apply rate limiters unless you configure them. The reasoning: FC's rate limiter is a token-bucket per drive/iface, and on a single-user box it adds syscall overhead without any real benefit.
When configured, the limiters apply at the drive and network-interface level via FC's API. For drives this is in [`create.go:298-302`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/create.go#L298-L302):
```go
if bw := e.cfg.RateLimits.diskBandwidth(); bw > 0 {
iops := e.cfg.RateLimits.diskIOPS()
rootfsDrive += fmt.Sprintf(
`,"rate_limiter":{"bandwidth":{"size":%d,"refill_time":1000},
"ops":{"size":%d,"refill_time":1000}}`,
bw, iops)
}
```
`refill_time` is in milliseconds — `1000` means the bucket refills once per second. So a `DiskBandwidthBytes: 100_000_000` config gives each VM 100 MB/s sustained disk throughput.
For a multi-tenant deployment, useful values:
```yaml
firecracker:
rate_limits:
net_bandwidth: 12500000 # 100 Mbps per direction
net_burst: 10000000 # 10 MB burst
disk_bandwidth: 104857600 # 100 MB/s
disk_iops: 10000 # 10K ops/s
```
These give each VM a hard ceiling so one noisy neighbor can't saturate the host's disk or upstream link.
## Listening port discovery
When a request hits a sandbox proxy URL, the daemon needs to know which ports the guest has open. Rather than maintain a host-side registry that gets stale, bhatti just asks the guest:
```go
// pkg/engine/firecracker/exec.go:198
result, _ := ag.Exec(ctx, []string{"ss", "-tln", "--no-header"}, nil, "")
return parseSSOutput(result.Stdout), nil
```
This is on-demand: we don't poll. The proxy queries listening ports when a request arrives, and the cost is one `exec` (a few milliseconds) per query. Guest-side `ss` is fast and accurate — it queries the kernel directly rather than walking `/proc/net/tcp`.
The trade-off: this only works on hot VMs. The `ListeningPorts` call errors out for warm or cold sandboxes. The public proxy handles this by `ensureHot`-ing the sandbox first — if you publish a port and a request hits the cold URL, the proxy wakes the VM, queries ports, checks that the requested port is in the list, then forwards.
## Recovery: what gets persisted
FC's state machine doesn't survive process restart on its own — you get a fresh FC process every time. Bhatti persists the per-VM configuration as JSON in the `fc_state` table so the next daemon startup can rebuild and resume.
The state captured per VM ([`pkg/engine/firecracker/helpers.go:120-180`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/helpers.go#L120-L180)):
* `vcpu_count`, `mem_size_mib`
* `tap_device`, `guest_ip`, `guest_mac`
* `rootfs_path`, `snap_mem_path`, `snap_vm_path`
* `volumes` — list of attached volume IDs and mount points
* `agent_token` — for reattaching to the agent after restore
* `fc_path_origin` — the original sandbox ID whose paths got baked into the snapshot (matters when you create a sandbox from a named snapshot — see below)
* `keep_hot`
On daemon startup, [`recoverVMs`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/bhatti/main.go) reads each row, calls `engine.RestoreVM(...)`, and the engine rebuilds the in-memory `VM` struct from the persisted state. If the snapshot files exist and the FC process is dead, the VM is marked `stopped` — ready to resume on the next API call.
### `fc_path_origin` and snapshot-derived sandboxes
When you create a sandbox from a named snapshot (`bhatti snapshot resume <snap> --name new-vm`), FC's snapshot file contains absolute paths to the rootfs and config files of the *original* sandbox. Those paths don't exist anymore (the original may have been destroyed) and even if they did, we'd be operating on the wrong files.
The fix in [`lifecycle.go:354-380`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go#L354-L380) is to symlink the original paths to the new paths just before `/snapshot/load`, then remove the symlinks once FC has the file descriptors open. Hacky. Effective. Documented in the commit history because it tripped up several rounds of debugging.
In jailer mode, this isn't an issue because all paths are chroot-relative — FC only sees its own chroot. The symlink dance is a bare-mode concession.
## Snapshot artifact layout
A cold sandbox has, on disk:
```
<data_dir>/sandboxes/<id>/
├── rootfs.ext4 CoW copy, may have been written to
├── config.ext4 the 1MB config drive
├── vol-<name>.ext4 attached volumes
├── mem.snap full memory snapshot (size of VM RAM)
└── vm.snap VM state snapshot (small, KB-range)
```
`mem.snap` is the size of the VM's allocated RAM. If you have a 4 GB VM that's been cold for an hour, `mem.snap` is 4 GB on disk. `vm.snap` is small — it holds CPU registers, vCPU state, and FC's device-model state.
For named snapshots (created via `bhatti snapshot create`), the artifacts go to `<data_dir>/snapshots/<user>/<name>/` and include copies of the rootfs and volumes alongside `mem.snap` and `vm.snap`, so the snapshot is fully self-contained — you can resume it into a fresh sandbox even after the original is destroyed.
## Where to go next
* [Architecture](/v1/docs/under-the-hood/architecture/) — the bird's-eye view that this page sits underneath
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — what the engine does for `Stop`/`Start`/`Pause`/`Resume`
* [Networking](/v1/docs/under-the-hood/networking/) — the bridge, TAP, and IP plumbing that the create flow above depends on
* [Decisions & learnings](/v1/docs/under-the-hood/decisions/) — the no-SDK choice, the rory incident, and other engine-level lessons
# Lohar: the agent inside every VM
> PID 1, the systemctl shim, sessions, PTY allocation, file operations — what runs inside each sandbox and the design decisions behind it.
PID 1 is the first userspace process the Linux kernel starts. It's responsible for mounting filesystems, reaping zombie processes, and ends up the parent of every other process in the system. If PID 1 exits, the kernel panics. On a normal Linux box, PID 1 is systemd.
In each bhatti microVM, PID 1 is a small Go program called `lohar` (लोहार — Hindi for *blacksmith*; it works inside the *bhatti*, the furnace). One binary, statically linked. It does the PID 1 init duties, listens for the agent protocol over TCP, manages exec sessions and their scrollback buffers, handles file operations, and contains a small shim that pretends to be `systemctl` and `journalctl` for in-guest callers.
## Design decisions on this page
Four decisions worth understanding before reading the rest:
1. **lohar runs as PID 1, not as a systemd-managed service.** Snapshot/restore broke when lohar ran as a child of systemd, in ways I haven't fully diagnosed. See [Why not real systemd](#why-not-real-systemd).
2. **systemctl is a shim, and real systemd is pinned out of apt.** Package post-install scripts assume systemctl exists, but installing real systemd alongside the shim breaks the rootfs. The shim covers the surface Debian's tools actually call; the apt pin keeps real systemd from sneaking in. See [The systemctl shim](#the-systemctl-shim).
3. **Every TTY exec is a session.** Sessions survive client disconnects and snapshot/restore. Without them, a closed laptop, a flaky network, or a Cloudflare Tunnel idle timeout kills your shell. See [Sessions](#sessions).
4. **File writes are atomic, with `fsync` before `rename`.** The host might snapshot the VM the next millisecond. Without `fsync`, the snapshot can contain a renamed file with zero bytes. See [File operations](#file-operations).
The list isn't exhaustive — smaller decisions (the busybox-style multi-call binary, server-side file truncation, the env-var precedence chain) are explained in their sections.
## One binary, three names
The first lines of [`cmd/lohar/main.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go#L30-L36):
```go
switch filepath.Base(os.Args[0]) {
case "systemctl":
runSystemctl(os.Args[1:])
return
case "journalctl":
runJournalctl(os.Args[1:])
return
}
// otherwise: PID 1 init duties
```
This is the [busybox pattern](https://en.wikipedia.org/wiki/BusyBox#Single_binary): one binary, multiple identities, dispatched on `argv[0]`. In the rootfs, `/usr/bin/systemctl` and `/usr/bin/journalctl` are symlinks pointing at `/usr/local/bin/lohar` ([`scripts/tiers/minimal.sh:62-63`](https://github.com/sahil-shubham/bhatti/blob/main/scripts/tiers/minimal.sh#L62-L63)). The kernel and shell don't care that the destination is the same binary; they invoke whatever name they want, and lohar picks the matching code path based on what name it was called by.
## Why not real systemd
I had lohar running as a systemd-managed service for a while (`Type=simple`, `Restart=always`, with the rootfs unchanged from a regular Ubuntu install). On a fresh boot it worked: `systemctl is-system-running` returned `running`, lohar accepted connections, `bhatti exec` worked.
Then `bhatti stop` (snapshot to disk) and `bhatti start` (restore from disk).
The restore itself looked successful: Firecracker's API returned ok, the guest kernel resumed, lohar's TCP listener was still bound to port 1024 according to `ss -tlnp` from inside the VM. The first `bhatti exec` after restore would succeed. Every subsequent exec would hang forever.
What I observed:
* The kernel-level TCP state was fine. Three-way handshake completed, packets visible in `tcpdump`.
* The connection sat in the kernel's accept queue.
* The Go runtime — sitting on top of `epoll_wait` — was not waking up. The goroutine blocked on `Accept()` never got scheduled.
I don't have a clean explanation for *why* this happens specifically when lohar is a child of systemd, and not when it's PID 1 itself. My current guess: something about how systemd manages file descriptors, control groups, or its own epoll sets across PID 1 leaves the resumed Go process holding a poller in a state Go's runtime doesn't recover from. That's speculation, not diagnosis.
What I know empirically:
* Reproduced on Firecracker 1.14.0 and 1.15.1.
* Reproduced with `Restart=no` and `Restart=always`.
* Lohar as PID 1 (no systemd) — never reproduced, in CI or production.
:::note\[If you know more, tell me.] This is the design decision I'd most like to revisit. If you've seen the same symptom and traced it further, or if you think my diagnosis is wrong, please [open an issue](https://github.com/sahil-shubham/bhatti/issues). The systemctl shim works, but real systemd would be less code to maintain. :::
So lohar stayed PID 1.
## The systemctl shim
The downside of "lohar is PID 1, no systemd" is that package post-install scripts assume systemctl exists. Most of Debian's package machinery — `deb-systemd-helper`, `deb-systemd-invoke`, `invoke-rc.d`, `policy-rc.d`, `/sbin/runlevel` — calls into a chain that ultimately runs `systemctl start <unit>`. Without it, a fresh `apt install openssh-server` looks like this:
```
Setting up openssh-server (1:9.6p1-3ubuntu13.5) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
ssh.service is a disabled or a static unit, not starting it.
```
Binary on disk, configuration in place, service never started — and the package's "if you can't start, fail" check trips. The fix is to make each tool in that chain happy.
The shim ([`cmd/lohar/systemctl.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/systemctl.go)) is about 1600 lines. It covers the surface that Debian's tools actually call (`start`, `stop`, `restart`, `status`, `enable`, `disable`, `is-active`, `is-enabled`, `kill`, `daemon-reload`, `show -p`) plus the flags they pass (`--no-reload`, `--quiet`, `--system`, `--no-block`, `--root=`, `--state=`, `--type=`).
It reads real `.service` files from the standard paths (`/etc/systemd/system`, `/usr/lib/systemd/system`, `/lib/systemd/system`) and parses the directives that show up in real-world unit files:
* `ExecStart`, `ExecStartPre`, `ExecReload`, `ExecStop`
* `Type=simple`, `Type=forking`
* `User=`, `Group=` — drop privileges before exec
* `Environment=`, `EnvironmentFile=` — merged into the child's environment
* `Restart=`, `RestartSec=` — auto-restart with backoff
* `WantedBy=multi-user.target` — picked up by `enable` and our boot
The walkthrough below describes what I observe when running `apt install postgresql` inside a sandbox. I haven't verified each step against the maintainer scripts of every Debian package, but it matches the chain documented in [deb-systemd-helper(1)](https://manpages.debian.org/testing/init-system-helpers/deb-systemd-helper.1.en.html) and [invoke-rc.d(8)](https://manpages.debian.org/testing/init-system-helpers/invoke-rc.d.8.en.html). Corrections welcome.
1. apt unpacks the deb.
2. The maintainer postinst script calls `deb-systemd-helper enable postgresql.service` (this is what `update-rc.d` calls now on systemd hosts).
3. `deb-systemd-helper` checks whether systemd looks present by stat'ing `/run/systemd/system/`. Lohar creates this empty directory at boot ([`cmd/lohar/main.go:97`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go#L97)) so the helper proceeds. Without it, the helper falls back to legacy sysvinit handling and our shim never gets called.
4. The helper creates `/etc/systemd/system/multi-user.target.wants/postgresql.service` as a symlink. That's how a service is "enabled" — there's no central database, just symlinks in `wants/` directories.
5. The postinst calls `deb-systemd-invoke start postgresql.service`, which delegates to `invoke-rc.d`.
6. `invoke-rc.d` runs `/usr/sbin/policy-rc.d` to check whether the action is allowed. Our shim ([`scripts/tiers/minimal.sh:73-78`](https://github.com/sahil-shubham/bhatti/blob/main/scripts/tiers/minimal.sh#L73-L78)) exits 0 unconditionally — actions are permitted.
7. `invoke-rc.d` runs `/sbin/runlevel` to determine the current runlevel. Our shim ([`scripts/tiers/minimal.sh:81-89`](https://github.com/sahil-shubham/bhatti/blob/main/scripts/tiers/minimal.sh#L81-L89)) prints `N 5` (multi-user with networking).
8. `invoke-rc.d` decides everything looks fine and runs `systemctl start postgresql.service`. Our shim reads the unit file, resolves `User=postgres`, sets up logging, drops privileges, and `exec`s the binary.
9. Postgres comes up. `systemctl status postgresql` queries the running process by reading the pidfile under `/run/bhatti/services/`. `journalctl -u postgresql` reads `/var/log/bhatti/postgresql.log`.
If postgres crashes, the shim's restart logic ([`cmd/lohar/systemctl.go:745-823`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/systemctl.go#L745-L823)) restarts it according to `Restart=` and `RestartSec=`. At boot, lohar reads `multi-user.target.wants/` and starts each enabled service ([`startEnabledServices` at `systemctl.go:1318`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/systemctl.go#L1318)).
### How services are spawned
When the shim starts a `Type=simple` or `Type=notify` daemon, it doesn't `exec` the binary directly. It goes through a one-purpose helper — `lohar spawn` — so the daemon and any descendants land in the unit's cgroup before they ever run a line of code.
```
┌── lohar (PID 1, supervisor) ────────────────┐
│ CreateCgroup(/sys/fs/cgroup/.../X.service) │
│ cmd := exec.Command("/proc/self/exe", │
│ "spawn", "--cgroup", <path>, "--", │
│ "/bin/sh", "-c", "exec "+ExecStart) │
│ cmd.Start() ──────────────────────┐ │
└────────────────────────────────────────────┼────────┘
│
┌── new process, lohar binary ──────────────▼───┐
│ 1. write own PID to <cgroup>/cgroup.procs │
│ 2. syscall.Exec("/bin/sh", ...) │
│ PID preserved. Becomes /bin/sh which │
│ execs into the daemon. │
└────────────────────────────────────────────────────┘
```
Two invariants do all the work:
1. **The cgroup write is observed by the kernel before any descendant of the writer is born.** `lohar spawn` writes its own PID into `<cgroup>/cgroup.procs` at the top of its `main`, before any `fork(2)` or `execve(2)`. It then `execve`s into `/bin/sh`, which `execve`s into the daemon. PID is preserved through all three transitions; no `fork` happens in between.
2. **`execve` does not change cgroup membership.** A process's cgroup is a property of the kernel task, not the loaded image. So the daemon — same PID throughout — is still in the unit's cgroup. Any subsequent `fork()` the daemon does (X-server detach, dbus pre-fork workers, classic Apache, anything that double-`fork`s during startup) inherits the unit's cgroup. Stop-time `cgroup.kill` catches all of them.
Real systemd does this in one syscall via `clone3(CLONE_INTO_CGROUP)`. Reproducing that in Go means reimplementing `syscall.ForkExec`'s post-fork setup (close-on-exec, signal handlers, credentials) under `syscall.ForkLock` — real surgery on code where bugs are silent and expensive. The helper approach is \~60 LoC of plain Go ([`cmd/lohar/spawn.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/spawn.go)) and gets the same correctness guarantee. `clone3` stays open as a future migration if the extra `execve` ever shows up in profiling — current measurements put it well under a millisecond against services-started times in the tens to hundreds of milliseconds.
For unit-file authors: **nothing changes.** `ExecStart=` works exactly the same way. The shim wraps your command in `/bin/sh -c "exec <ExecStart>"` regardless — systemd's parser does the same thing. The spawn helper is upstream of all of that; you don't need to know about it.
This is the mechanism that makes `KillMode=control-group` (systemd's default) reliable for daemons that fork during startup. Before this landed in v1.11.10 the placement happened *after* `cmd.Start()` returned, and any daemon that `daemon(3)`d during startup could have its child escape into lohar's root cgroup. `systemctl stop kasmvnc` would then leave an orphan `Xkasmvnc` holding the abstract `@/tmp/.X11-unix/X99` socket, breaking the next start with `Cannot establish any listening sockets`. Every shipped tier benefits from the fix transparently — no unit-file changes required.
### What the shim doesn't do
* **Targets** beyond `multi-user.target` and a few well-known names (`sysinit`, `network`, `default`). Anything checking these gets `active`. Anything more specific is a no-op.
* **Socket activation.** `.socket` units resolve to their associated `.service` and the service is started directly. Fine for sshd, breaks for things that genuinely need lazy socket activation.
* **Timers** (`.timer` units). If you need scheduled work, install cron.
* **D-Bus.** You can install `dbus` and the shim will start it as a normal service, but the host doesn't speak DBus to the guest.
## Pinning systemd out
The shim only works if real systemd never gets installed. The trap: many useful packages (openssh-server, sudo, postgresql, nginx) declare `Recommends: libpam-systemd`. By default, apt installs Recommends. And `libpam-systemd` Depends on `systemd`. So one innocent `apt install openssh-server` would silently:
* install real systemd
* install `/bin/systemctl` over our `/usr/bin/systemctl` symlink (PATH resolution prefers `/bin` for some shells)
* install systemd-resolved, which would overwrite `/etc/resolv.conf` on next service start
* leave the rootfs in a half-broken state where snapshot/restore fails (the original problem)
The fix is in [`scripts/tiers/minimal.sh:34-41`](https://github.com/sahil-shubham/bhatti/blob/main/scripts/tiers/minimal.sh#L34-L41):
```bash
printf "Package: systemd systemd-sysv systemd-resolved
Pin: release *
Pin-Priority: -1
" > /etc/apt/preferences.d/no-systemd-daemon
```
`Pin-Priority: -1` tells apt the package is never installable from any source. Server packages only Recommend `libpam-systemd` (never Depend), so apt's resolver just drops it from the install set and proceeds.
If you build a custom rootfs based on a tier other than `minimal`, this pin is inherited because every tier sources `minimal.sh` first.
### Privileged systemctl from a non-root caller
`bhatti exec dev -- systemctl status sshd` runs as the `bhatti` user inside the VM. That user can read `.service` files but can't kill a process owned by root. The shim handles this with a small in-guest IPC:
* The user-side `systemctl` invocation marshals its request as a [`SystemctlRequest`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/messages.go#L67) and sends it over a Unix socket at `/run/bhatti/systemctl.sock`.
* PID 1 lohar accepts on that socket, uses [`SO_PEERCRED`](https://man7.org/linux/man-pages/man7/socket.7.html) to learn the caller's UID *from the kernel*, runs the operation as root, and sends back the formatted output.
* The user-side `systemctl` prints what came back and exits with the matching code.
`SO_PEERCRED` is the trust boundary, not anything in the request payload — a non-root caller could otherwise forge a UID claim and get a privileged operation done.
## Boot sequence
The kernel command line ends with `init=/usr/local/bin/lohar`. After the kernel finishes its own boot, lohar is the first userspace process.
Lohar writes phase markers to `/run/bhatti/boot-timing.txt` as it goes, so you can `cat` it inside any sandbox and see exactly where time went. A run might look like this:
```
+0ms start
+2ms mounts_done
+3ms lo_up
+8ms config_applied
+12ms network_done
+18ms tcp_listen ← agent reachable here
+19ms services_started
+21ms init_session_started
```
That's lohar's own clock — the time between PID 1 starting and the agent listening. The wall-clock time from `bhatti create` to the daemon flipping the sandbox to `running` is dominated by the kernel boot and varies with hardware and rootfs. To measure it on your machine, [`bench/run.sh`](https://github.com/sahil-shubham/bhatti/blob/main/bench/run.sh) runs the full suite.
What each phase does ([`cmd/lohar/main.go:48`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go#L48)):
1. **Mount filesystems** — `/proc`, `/sys`, `/dev`, `/dev/pts`, `/tmp`, `/run`, `/dev/shm`. The kernel hands you a bare ext4 with none of these.
2. **Mount cgroup v2** with `+cpu +memory +io +pids` enabled. cgroup v2 is needed for in-guest tools that do their own resource accounting — Docker is the most common, but the same applies to anything using cgroup-style limits (containerd, podman, slurmd).
3. **Bring up `lo`** with raw `ioctl(SIOCSIFFLAGS)`. `iproute2` isn't wired up yet — we're seven milliseconds into PID 1. `eth0` is already up because the kernel processed the `ip=` boot parameter before init ran; see [Networking](/v1/docs/under-the-hood/networking/) for that part of the story.
4. **Read the per-sandbox config** from a small ext4 image attached as `/dev/vdb`. Apply hostname, DNS, env vars, files, volume mounts. Then unmount and remove the mount point — secrets shouldn't sit on disk after they've been read.
5. **Install signal handlers, start the zombie reaper.** PID 1 is responsible for reaping orphaned child processes; [`go reapZombies()`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go#L128) does it.
6. **Start the syslog receiver.** `/dev/log` becomes a Unix datagram socket; lohar parses RFC-3164 messages and routes them to per-service log files under `/var/log/bhatti/`. Anything in the guest using `syslog(3)` (cron, sshd, postgres) writes to a readable log without any extra configuration.
7. **Listen on `/run/bhatti/systemctl.sock`** for the privileged systemctl IPC described above.
8. **Listen on TCP :1024 (control) and :1025 (forward).** The host has been polling :1024 since Firecracker started. Now `exec true` succeeds, and the daemon flips the sandbox's status to `running`.
9. **Start enabled services** from `/etc/systemd/system/multi-user.target.wants/`.
10. **Run `/etc/bhatti/init.sh`** if it exists (30 s timeout, output to stderr). Useful when you want to bake boot logic into a custom rootfs without forcing every sandbox to specify `--init`.
11. **Run the `--init` script** from sandbox creation as a TTY session named `init`. Attach with `bhatti shell dev --session init` to watch your provisioning run.
12. **Block forever.** PID 1 is not allowed to exit.
## Sessions
Every TTY exec in bhatti creates a *session* — a persistent handle to the running command, with an ID, a 64 KB scrollback ring buffer, and a PTY master file descriptor. Sessions survive client disconnects: the process keeps running, output keeps flowing into the scrollback. A client reconnecting to the same session ID gets the scrollback replayed, then live I/O.
Three reasons this is the default rather than a feature:
1. **Init scripts need to be attachable.** When you create a sandbox with `--init "npm install && npm run build"`, that command runs as a session named `init`. You can `bhatti shell dev --session init` to attach mid-run and see what's happening. If exec weren't a session, the init output would be invisible from the host.
2. **Shells need to survive proxy disconnects.** A real example from `PLAN-shell-sessions.md`: a user ran `bhatti shell rory` through `api.bhatti.sh` (Cloudflare Tunnel) and started `hermes gateway`, a daemon that prints a startup banner then waits for events. Cloudflare Tunnel kills idle WebSocket connections after about 100 seconds. The shell dropped, the daemon kept running inside the VM, and the user had no way back. With sessions, reconnecting replays the scrollback and resumes the live PTY.
3. **Snapshot/restore needs to preserve interactive shells.** A shell that's running a long-running command when the VM is snapshotted should be the same shell after restore. Sessions are the data structure that survives that round-trip.
The session registry lives in [`cmd/lohar/session.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/session.go). Two caps that matter, both per-VM ([`cmd/lohar/handler.go:49-51`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/handler.go#L49-L51)):
* 50 concurrent connections
* 20 active sessions
For a normal sandbox, you'll never hit either. Services started by `startEnabledServices` are tracked separately and don't count against the session cap.
### Scrollback ring buffer
Each session owns a fixed-size 64 KB ring buffer ([`cmd/lohar/session.go:50`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/session.go#L50)). Writes wrap when full, overwriting the oldest data. The buffer's `Bytes()` method returns the contents in order — oldest first — so a reattaching client sees the same byte sequence the original PTY emitted, just truncated to the most recent 64 KB.
Why a fixed size? An interactive shell can produce arbitrary output (think `find /` or a chatty test runner). If the buffer grew without bound, a forgotten session would eventually OOM the VM. 64 KB is enough to replay the last few hundred lines of typical output, which is what the "what was happening when I disconnected" use case actually needs.
### PTY allocation
The PTY pair for each TTY session is allocated with raw syscalls ([`cmd/lohar/tty.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/tty.go)), no CGo:
1. Open `/dev/ptmx` (the PTY multiplexor).
2. `TIOCGPTN` ioctl — get the slave PTY number assigned to this master.
3. `TIOCSPTLCK` ioctl with `0` — unlock the slave (kernel won't let you open it until you do this).
4. Open `/dev/pts/<N>` as the slave file.
The child process is started with `Setsid: true` and `Setctty: true`, which makes it a new session leader and assigns the slave PTY as its controlling terminal — that's what makes Ctrl+C send SIGINT and the shell prompt look like a terminal. Window size is applied via `TIOCSWINSZ` ioctl on the master whenever the host sends a `RESIZE` frame.
## Piped exec (the simpler path)
`bhatti exec dev -- npm install` is *not* a session by default. It's a piped exec — stdout and stderr stream back as `STDOUT` / `STDERR` frames, the child runs in its own process group, and the connection lives until the child exits. Sessions add overhead a one-shot command doesn't need.
The lifecycle ([`cmd/lohar/exec.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/exec.go), [`piped_session.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/piped_session.go)):
1. `exec.Command` with `Setpgid: true`. Child gets its own process group, distinct from lohar.
2. Three pipes (stdin, stdout, stderr) connected to the child.
3. Two reader goroutines fan stdout and stderr into `STDOUT` / `STDERR` frames. Both push to a single writer goroutine through a channel, which serializes frame writes — concurrent writes would interleave at byte boundaries and corrupt frames on the wire.
4. Wait for both readers to drain *before* `cmd.Wait()`. Otherwise the `EXIT` frame can race ahead of the last output frame and the host would see the exit code before the last line of output.
5. `syscall.Sync()` — flush any files the command wrote. The host might snapshot the VM the next millisecond; dirty pages in the guest's page cache aren't part of the FC snapshot.
6. Send the `EXIT` frame with the integer exit code, close.
If the host kills the connection (you `Ctrl+C`'d the CLI), lohar sends `SIGKILL` to the negative PID — kills the entire process group. `npm install` spawns Node, which spawns dozens of children; without process-group kill, they orphan and keep running.
There's a third mode (`session: true` in the ExecRequest, no TTY) for piped sessions with scrollback — fire-and-forget a long-running build, get the output streamed when you next attach. The CLI uses this internally for `--detach`.
### Environment variable precedence
Every exec inherits a merged environment, applied in this order (later wins):
```
1. defaults PATH, TERM, HOME, LANG
2. config drive env secrets and env vars set at sandbox creation
3. per-request env --env on the exec command, or env in API body
```
So `bhatti exec dev --env API_KEY=test -- env` will see `API_KEY=test` even if a secret with the same name was set on the sandbox. This is deliberate: the per-request env wins so you can override secrets in CI or locally without rotating them.
### Config-env bridge
Layer 2 above — the config-drive env from `bhatti create --env K=V,K=V` — reaches `bhatti exec` natively because exec merges from `configEnv` in memory. To reach *units* started by the systemctl shim, lohar additionally serialises `configEnv` to `/run/bhatti/config-env` as canonical `KEY=VALUE` lines, after the config drive is applied and before `startEnabledServices` runs. Unit files opt in with:
```ini
[Service]
EnvironmentFile=-/run/bhatti/config-env
```
The leading `-` makes the file optional, so units in `minimal`-derived rootfs images without any `--env` flags still boot cleanly.
This is how every tier's tunables work in v1.11.8+:
* `docker` tier: dockerd inherits `DOCKER_*` env vars at process start.
* `browser` tier: `CHROME_REMOTE_PORT` and `CHROME_FLAGS` expand inside `headless-chrome.service`'s ExecStart.
* `computer` tier: `DISPLAY_WIDTH`, `KASM_FRAMERATE`, etc. flow into `kasmvnc.service`'s ExecStart and override the unit's own `Environment=` defaults (`EnvironmentFile=` always wins over `Environment=` in systemd's precedence rules; the shim matches).
User-supplied values **always win** over tier defaults because the config-env file is written before any service starts; the unit's `Environment=` defaults are loaded first, then `EnvironmentFile=` overrides matching keys. (One subtlety: `kasmvnc-firstboot.service` appends a computed `KASM_THREADS=$((nproc-1))` to `config-env` only if the key isn't already present, preserving user precedence.)
## File operations
The agent owns file ops because `cat | exec` is slow, awkward, and loses metadata. The frame types ([`pkg/agent/proto/constants.go:42-49`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/constants.go#L42-L49)):
\| Op | Frame | Returns | |-------|------------------|---------| | Read | `FILE_READ_REQ` | response with size + mode, then `STDOUT` frames, then `EXIT` | | Write | `FILE_WRITE_REQ` | take `STDIN` frames, write atomically, return OK | | Stat | `FILE_STAT_REQ` | `FileInfo` | | Ls | `FILE_LS_REQ` | `[]FileInfo` |
Two details worth their own paragraph.
**Atomic writes.** The write path: open `path.bhatti-tmp`, stream the content into it, `fsync`, `rename` over the target. POSIX guarantees `rename` is atomic on the same filesystem — a process inside the VM that's `tail`-ing the file will see either the old content or the new content, never a half-written state. The `fsync` before rename is the load-bearing part: without it, the rename is metadata-durable but the data is still in the guest's page cache, not on the virtio-blk device. If the host snapshots the VM between the rename and the kernel's periodic flush, the snapshot contains a file that exists with zero bytes.
**Server-side truncation.** Coding agents always truncate file reads — typically 2000 lines or 50 KB. If we did that on the client side, we'd ship 100 MB of log file across the wire just to throw 99.95% of it away. Instead, `FILE_READ_REQ` accepts `offset`, `limit`, `max_bytes` parameters, and lohar applies them as it reads line by line. The response always includes the *full* file size so the consumer can detect truncation.
## What lohar isn't
A few things the docs sometimes imply lohar does, that it doesn't:
* **It's not your shell.** Your shell is whatever the rootfs has (usually `/bin/bash`); lohar exec's it.
* **It's not a service supervisor for arbitrary daemons.** It runs unit files registered through `enable` and its own session children. Daemons that fork-and-detach the old sysvinit way work, but their state isn't fully tracked.
* **It doesn't reach out to the network on its own.** No telemetry, no phone-home, no update checks. The only outbound TCP from the guest is whatever you run in it.
## Where to go next
* [Architecture overview](/v1/docs/under-the-hood/architecture/) — how lohar fits into the broader system
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — what happens to this lohar instance when it goes idle
* [Networking](/v1/docs/under-the-hood/networking/) — the per-sandbox network setup and how `/dev/vdb` gets there
* [Wire protocol](/v1/docs/under-the-hood/wire-protocol/) — frame format, frame types, atomic writes
* [Decisions & learnings](/v1/docs/under-the-hood/decisions/) — the systemd story above, the diff-snapshot incident, and other lessons paid for in real time
# Networking: bridges, TAPs, and the ARP trick
> Per-user bridges in 10.0.x.0/24, the six iptables rules that protect the host, the kernel ip= parameter that brings the network up before init, and a permanent ARP entry that saves a second of boot time.
Every VM gets its own network interface, its own IP, and internet access. Users are isolated at L2: VMs belonging to different users live on different bridges and can't see each other's traffic. The host itself is unreachable from inside any VM unless the host opened the connection first. None of this requires per-VM iptables management; it's six rules total, regardless of how many users or VMs you have.
This page is about the host side of networking. The guest side (`eth0`, DNS, kernel `ip=`) is short and lives at the end.
## Design decisions on this page
1. **Per-user bridges, not a single shared one.** Earlier versions of bhatti used a single `brbhatti0` and all VMs lived on the same L2 segment. Tenants could ARP-scan and reach each other. The current design gives each user their own `brbhatti-N` and `/24`. See [Per-user bridges](#per-user-bridges).
2. **Six global iptables rules cover every VM.** Cross-bridge isolation, NAT, return traffic, and a `DROP` rule that prevents guest-initiated connections to the host on `INPUT`. No per-VM rules, regardless of how many VMs exist. See [The six iptables rules](#the-six-iptables-rules).
3. **Kernel `ip=` configures `eth0` before init runs.** Solves the chicken-and-egg of "the agent needs the network before the host can reach the agent." See [Kernel `ip=`](#kernel-ip).
4. **Pre-populated permanent ARP entry on create.** Skips the first ARP retransmit (Linux default `retrans_time_ms = 1000`), saving a full second of boot time. See [The ARP trick](#the-arp-trick).
5. **TAP devices survive `Stop()`, are destroyed only on `Destroy()`.** The Firecracker snapshot contains virtio-net state that references the TAP device. Destroying it would break resume. See [TAP lifecycle](#tap-lifecycle).
## Per-user bridges
When the second user creates their first sandbox, bhatti creates them a fresh Linux bridge and a `/24` subnet. The third user gets another. Each user is alone on their own L2 segment.
The mapping ([`pkg/engine/firecracker/network.go:23-52`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/network.go#L23-L52)):
\| User index | Bridge | Subnet | Gateway | |---|---|---|---| | 1 | `brbhatti-1` | `10.0.1.0/24` | `10.0.1.1` | | 2 | `brbhatti-2` | `10.0.2.0/24` | `10.0.2.1` | | ... | ... | ... | ... | | 254 | `brbhatti-254` | `10.0.254.0/24` | `10.0.254.1` | | 255 | `brbhatti-255` | `10.1.0.0/24` | `10.1.0.1` | | 65 024 | `brbhatti-65024` | `10.255.254.0/24` | ... |
The `subnetFromIndex` function in network.go does the math: 1-based index, `10.<hi>.<lo>.0/24`, skipping `10.0.0.0/24` to avoid the `hi=0,lo=0` ambiguity. Each user can have up to 253 concurrent VMs (`.2` through `.254` — `.0` is the network, `.1` is the gateway, `.255` is broadcast). The total addressable users on a single host is 65,024. If you ever need more than that on one box, you've outgrown bhatti and probably need a real cluster.
A bridge is created on demand when the first sandbox for a user is created ([`ensureUserBridge`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/network.go#L135-L143)). When a user destroys their last sandbox, the bridge is removed ([`removeUserNetworkIfEmpty` in `lifecycle.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go)).
### Why per-user, not single-shared
Earlier versions of bhatti used a single bridge `brbhatti0` on `192.168.137.0/24`. All VMs across all users shared it. This was simpler to operate but broke the multi-tenancy model — at L2, alice's VMs could ARP-scan the bridge and reach bob's VMs directly. Application-level isolation only.
Per-user bridges solve this at the network layer: alice's VMs are on `brbhatti-1`, bob's on `brbhatti-2`, and the kernel's bridge forwarding doesn't relay frames between them. Even if alice's VM is compromised at the kernel level, it can only see alice's other VMs.
If you're upgrading from an old install, the engine startup runs [`cleanupOldBridge()`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/network.go#L204-L238) once. It removes `brbhatti0`, the legacy 192.168.137 NAT rule, and the per-bridge FORWARD rules from the old design. Existing sandboxes get reattached to per-user bridges on their next start.
## TAP devices
Each VM gets its own TAP device on its user's bridge. The naming pattern is `tap` plus the first 8 characters of the sandbox ID ([`pkg/engine/firecracker/network.go:240-253`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/network.go#L240-L253)):
```go
tapName = "tap" + sandboxID[:8]
ip tuntap add <tap> mode tap
ip link set <tap> master <bridge>
ip link set <tap> up
```
Firecracker binds the VM's virtio-net device to this TAP. From the guest's perspective, it's a regular ethernet interface (eth0); from the host's perspective, it's a TAP attached to a bridge.
### TAP lifecycle
TAPs are created in `Create()` and destroyed in `Destroy()`. They are **not** destroyed on `Stop()` (snapshot to disk). The Firecracker snapshot contains virtio-net state that references the TAP by name. If we delete and recreate it, the resumed guest's network stack ends up referencing a TAP that doesn't exist anymore, breaking outbound traffic.
So TAPs survive across stop/start cycles. Across a daemon restart, orphan TAPs (created by a previous run that crashed before tearing them down) are cleaned up at engine startup ([`network.go:286-310`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/network.go#L286-L310)) — list all `tun`-type devices, delete any `tap*` not in the known set.
### MAC addresses
Each VM gets a random locally-administered unicast MAC ([`helpers.go:268-274`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/helpers.go#L268-L274)):
```go
b := make([]byte, 6)
rand.Read(b)
b[0] = (b[0] & 0xfe) | 0x02 // locally administered, unicast
```
Two bit ops on the first byte:
* `& 0xfe` — clears the lowest bit, which would mean "multicast". MACs starting with an odd byte are multicast addresses; we don't want that.
* `| 0x02` — sets the second bit, which marks the address as locally administered. The IEEE guarantees that no real hardware vendor will assign a MAC with this bit set, so we can't collide with real hardware on the same LAN.
The remaining 46 random bits make collisions astronomically unlikely even within bhatti. Not registered with anyone, but it doesn't need to be — these MACs only need to be unique within the bridge.
## The six iptables rules
That's the entire isolation model on the host side. Inserted at the top of each chain so they take priority over any UFW or k8s rules ([`network.go:147-200`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/network.go#L147-L200)):
```
# 1. Cross-bridge isolation: alice's VMs cannot reach bob's VMs.
iptables -I FORWARD 1 -s 10.0.0.0/8 -d 10.0.0.0/8 -j DROP
# 2. VM → internet
iptables -I FORWARD 1 -s 10.0.0.0/8 ! -d 10.0.0.0/8 -j ACCEPT
# 3. Return traffic from internet → VM
iptables -I FORWARD 1 -d 10.0.0.0/8 -m state --state RELATED,ESTABLISHED -j ACCEPT
# 4. Allow agent TCP responses to reach the host (host initiated, so
# the SYN-ACK enters INPUT with src 10.0.0.0/8)
iptables -I INPUT 1 -s 10.0.0.0/8 -m state --state RELATED,ESTABLISHED -j ACCEPT
# 5. Block VM-initiated connections to the host (API, SSH, everything).
# Only NEW connections are dropped — established ones (from rule 4) survive.
iptables -I INPUT 1 -s 10.0.0.0/8 -m state --state NEW -j DROP
# 6. NAT for outbound
iptables -t nat -I POSTROUTING 1 -s 10.0.0.0/8 -o <default> -j MASQUERADE
```
Rule 1 enforces tenant isolation. Rule 5 enforces guest-host isolation. The combination: a VM can reach the internet, can reach its own user's other VMs, and cannot reach anything else, including the host's bhatti API or SSH.
This is a feature you don't configure. It's true the moment the engine starts. The same six rules cover one user, ten users, ten thousand users.
The default outbound interface is auto-detected from `ip route show default`. If you want to bind to a specific interface on a multi-homed host, set it in your config.
### Rule order matters
Rule 4 must come before rule 5. Both match `INPUT` traffic from `10.0.0.0/8`; rule 4 matches established connections (the SYN-ACK from a VM responding to a host-initiated SYN), rule 5 matches new connections (a VM trying to reach the host). If they were in the wrong order, rule 5 would kill all agent connections. The code inserts rules in *reverse* order with `-I … 1` so the final chain order is correct.
## The ARP trick
If you read the create code carefully you'll see two `ip neigh` commands you might not expect. Both save a second of boot time; together they explain why bhatti's first SYN to the agent doesn't sit in retransmit purgatory.
### Pre-populating the cache
[`pkg/engine/firecracker/create.go:386-395`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/create.go#L386-L395):
```go
// Pre-populate ARP with the guest's MAC so the first TCP SYN is sent
// immediately without waiting for ARP resolution.
exec.Command("ip", "neigh", "replace", guestIP, "lladdr", mac,
"dev", userNet.BridgeName, "nud", "permanent").Run()
```
When the host first tries to dial the agent at `10.0.1.2:1024`, the host kernel doesn't know the MAC address that goes with that IP. So it sends an ARP request: "who has 10.0.1.2?" The guest can't answer yet — the kernel's still booting, the agent hasn't started, eth0 is up but no one is listening. Linux's ARP retransmit timer (`retrans_time_ms = 1000ms`) adds a full second before the next probe. So our `WaitReady` polling spends a second of every poll waiting on ARP alone.
We already know the MAC, because we generated it in `generateMAC()`. So we pre-populate the host's ARP cache with `nud permanent` — a permanent neighbor entry that won't be flushed by the kernel's normal aging. The first SYN goes out the moment the agent is listening; no ARP round-trip needed.
On `Destroy()` we delete the entry ([`lifecycle.go:455-460`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go#L455-L460)). Without this cleanup, entries accumulate on the bridge until the bridge itself is destroyed.
### Flushing stale entries on create
[`pkg/engine/firecracker/create.go:130-135`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/create.go#L130-L135):
```go
// Flush stale ARP entry for this IP. When a sandbox is destroyed and
// a new one reuses the same IP, the host ARP cache still maps the IP
// to the old sandbox's MAC (STALE, gc_stale_time=60s). The new VM
// gets a fresh MAC, so the host sends TCP SYNs to the old MAC —
// which no longer exists on any TAP. WaitReady times out at 30s.
exec.Command("ip", "neigh", "del", guestIP, "dev", userNet.BridgeName).Run()
```
When a sandbox is destroyed and a new one reuses the same IP within 60 seconds (`gc_stale_time` default), the host's ARP cache may still map that IP to the old sandbox's MAC. The new VM gets a fresh MAC, so the host sends TCP SYNs to the old MAC — which doesn't exist on any TAP anymore. `WaitReady` times out at 30 s and the create looks broken.
The flush is one line and it makes the failure go away. This is the kind of bug that takes a day to find and a one-line fix to solve. There are several of these in bhatti. They're not documented anywhere because they're invisible when they're working.
## Kernel `ip=`
The guest's `eth0` is configured by the kernel command line, before init runs. The boot args bhatti emits look like this:
```
... ip=10.0.1.2::10.0.1.1:255.255.255.0::eth0:off:1.1.1.1:8.8.8.8: init=/usr/local/bin/lohar ...
```
The `ip=` parameter is documented in [Linux kernel parameters](https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html) under `ip=`. Format:
```
ip=<client>:<server>:<gateway>:<netmask>:<hostname>:<device>:<autoconf>:<dns0>:<dns1>:<ntp>
```
Most fields we leave empty (the empty `:server:` and the empty `:hostname:` slots). What's there:
* `<client>` — `10.0.1.2`, the guest's IP.
* `<gateway>` — `10.0.1.1`, the bridge IP on the host.
* `<netmask>` — `255.255.255.0` for a /24.
* `<device>` — `eth0`, the virtio-net interface name.
* `<autoconf>` — `off`, no DHCP.
* `<dns0>`, `<dns1>` — DNS servers, defaulted to Cloudflare and Google.
By the time lohar starts as PID 1, eth0 is up, has its IP, knows its gateway, has a default route. The host can dial `10.0.1.2:1024` immediately — no DHCP server on the bridge, no agent-side configuration step.
This solves a chicken-and-egg problem: if we configured the network from inside the agent, the host couldn't reach the agent to tell it what IP to use until the agent had configured the network. Kernel `ip=` is a stock Linux feature that side-steps it entirely.
DNS gets a second pass: the kernel `ip=` parameter sets `/etc/resolv.conf` with the static values from boot args, but lohar may overwrite this from the per-sandbox config drive ([`cmd/lohar/main.go:107-110`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go#L107-L110)) if the user passed custom DNS at create time.
## Reaching services inside sandboxes
Two paths in. They both go through the daemon — there's no direct path from a remote client to a sandbox's IP, because that IP is on an internal bridge.
### Authenticated proxy
```
GET /sandboxes/:id/proxy/:port/<path>
Authorization: Bearer <token>
```
The daemon authenticates, looks up the sandbox, calls `engine.Tunnel(id, port)` to open a TCP tunnel to `localhost:<port>` inside the VM, and proxies HTTP/WebSocket traffic. Cold sandboxes wake on the first request. Useful for development — you don't need to publish a port to use one.
### Public proxy (published URLs)
```
ANY https://<alias>.bhatti.sh
```
When you `bhatti publish dev -p 3000 -a my-app`, the daemon creates a publish rule mapping `my-app` → `(sandbox=dev, port=3000)`. Public requests hit `:443`, get routed by Host header, looked up against the rule cache ([`pkg/server/public_proxy.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/public_proxy.go)), tunneled into the VM.
The public proxy has a few extra moves over the authenticated one:
* **In-memory route cache** (LRU, 10 K entries) so a hot URL doesn't query SQLite per request.
* **`singleflight.Group` for resume coalescing** — if 50 concurrent requests hit a cold sandbox, only one wake actually happens; the others wait on it. Without this, you'd get 50 simultaneous snapshot loads.
* **Per-alias and global rate limiting** — protects against abuse.
* **5-minute per-request deadline, 50 MB body limit.**
Both proxies use `httputil.ReverseProxy` with a custom transport that treats the engine's tunnel as an `http.RoundTripper`. WebSocket connections are hijacked and relayed bidirectionally with a 10-minute idle timeout.
## After a snapshot/restore: the conntrack pause
When a VM resumes from cold, the host's iptables conntrack table still has stale entries for the guest's IP. Host-initiated connections work fine (rule 4 + new conntrack entries replace the stale ones). But *guest-initiated* outbound connections — say, the guest does `ping 8.8.8.8` — can hit the stale conntrack and the SYN gets stuck in retransmit backoff for tens of seconds.
In normal operation this doesn't matter, because the host always initiates connections to the agent. But it is a thing you'll notice if you're benchmarking outbound throughput right after a resume. An ARP flush helps:
```bash
ip neigh flush dev brbhatti-1
```
We don't run this automatically because it's only relevant for a brief window post-resume and most workloads don't hit it.
## Recovery after crashes
If the daemon crashes mid-create or mid-destroy, you can be left with orphan resources: a TAP that's not attached to any VM, an ARP entry for an IP that's been freed, a bridge that should have been removed when its last user's last sandbox was destroyed.
Recovery happens at engine startup ([`network.go:286-345`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/network.go#L286-L345)):
* `cleanupOrphanedTapDevices(knownTaps)` — list all tap devices, delete any not in the known-VMs set. The known-VMs set is empty before sandbox recovery loads, so on a fresh start every tap looks orphan and gets cleaned. This is intentional: better to start fresh than to leak.
* `cleanupAllUserBridges()` — same logic for bridges. Recovery recreates them as it loads sandboxes.
* `cleanupOldBridge()` — runs once per startup as a no-op if the legacy `brbhatti0` is already gone.
## Where to go next
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — how the network stays consistent across snapshot/resume
* [Lohar](/v1/docs/under-the-hood/lohar-the-blacksmith/) — how the agent inside the VM uses TCP to listen for host commands
* [Custom domain](/v1/docs/managing/custom-domain/) — TLS for the public proxy, wildcard certs, ACME flow
* [Decisions & learnings](/v1/docs/under-the-hood/decisions/) — the per-user bridge migration, the ARP trick, and other small bugs that took whole afternoons
# Storage
> How bhatti stores sandbox state on disk — reflinks, compression, page cache, and why btrfs is the recommended filesystem.
bhatti's storage layout is one of the most load-bearing design decisions in the system and the least visible. Every `bhatti create`, every snapshot, every resume from cold leans on assumptions about the underlying filesystem. On btrfs those assumptions are correct and the numbers we publish on the homepage hold. On ext4 the system still *works* — but the costs go up, sometimes by an order of magnitude.
This page covers what's actually on disk per sandbox, the kernel primitives that make the costs small, and what changes on a filesystem that doesn't support reflinks.
## What's on disk per sandbox
Four files under `/var/lib/bhatti/sandboxes/<id>/`:
```
rootfs.ext4 CoW copy of the base rootfs (sparse-allocated)
config.ext4 1 MiB ext4 with hostname, env, secrets, volume specs
mem.snap guest RAM dump (equal to configured memory; only when cold)
vm.snap KB-range VM device state (only when cold)
```
Plus shared resources elsewhere:
* `/var/lib/bhatti/images/rootfs-<tier>-<arch>.ext4` — read-only base rootfs templates. Every sandbox's `rootfs.ext4` is reflinked from one of these.
* `/var/lib/bhatti/volumes/<name>.ext4` — standalone volumes (`bhatti volume create`), attached to one sandbox at a time.
The `.ext4` files are virtio-blk-backed block devices from the guest's perspective. Their on-disk size is *sparse* — `ls -lh` reports the logical size, `du -h` reports what's actually allocated. The difference matters; see [Disk usage](/v1/docs/managing/images/#disk-usage) on the images page for the walkthrough.
## How rootfs cloning works
`bhatti create` calls [`copyBlock`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/fc.go#L236):
```go
exec.Command("cp", "--reflink=auto", "--sparse=always", src, dst).Run()
```
`--reflink=auto` means "use the kernel's `FICLONE`/`BTRFS_IOC_CLONE` ioctl if the filesystem supports it; otherwise fall back to a normal copy." Same source code, very different runtime cost:
\| Filesystem | Mechanism | Time | Disk used | |---|---|---|---| | **btrfs** | `BTRFS_IOC_CLONE` (metadata-only) | \~14 ms | one extent table entry; shares blocks with source | | **xfs** (with reflink=1) | `FICLONE` (metadata-only) | comparable | shares blocks; refcount btree | | **ext4** | sparse `read`/`write` loop | \~320 ms | physical copy of every used extent |
The 14 ms / 320 ms numbers come from [`INVESTIGATION-create-performance.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/INVESTIGATION-create-performance.md) on `agni-01` (NVMe RAID-1, 1 GiB base rootfs).
On btrfs the new sandbox `rootfs.ext4` shares extents with the base image until the sandbox writes a block, at which point CoW splits that block on first write. The read-only majority of every tier (`/usr`, `/lib`, package payloads) stays shared in the page cache across every sandbox built from the same image — one set of pages in RAM serves all of them.
The same `copyBlock` runs for snapshot create ([`snapshot.go:280`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/snapshot.go#L280)) and for volume backups ([`pkg/server/volume_handlers.go:271`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/volume_handlers.go#L271)). On btrfs every "expensive" copy in the system is metadata-only.
## zstd on `mem.snap`
The recommended btrfs mount option is `loop,noatime,compress=zstd:1` — transparent zstd at level 1 on every file in the data dir. `mem.snap` benefits dramatically because guest RAM is dominated by zero pages, deduplicated read-only kernel text, and other highly-compressible data.
Production measurements from `agni-01`:
\| Artefact | Apparent | Physical | Ratio | |---|---|---|---| | One 1024 MB `mem.snap` | 1.0 GiB | **48 MiB** | 21× | | One stopped sandbox dir | 2.1 GiB | **176 MiB** | 12× | | All sandboxes, 63 sandboxes / 6 users | 571 GiB referenced | 95 GiB physical | **6×** combined |
The 6× combined number factors in both reflink dedup (2.6×) and zstd (2.3×) across the whole data dir. You can inspect the breakdown yourself with `compsize` from `btrfs-progs`:
```bash
sudo compsize /var/lib/bhatti
```
It splits the report by file type (mostly ext4 block devices in our case) and shows what each compression algorithm achieved.
Compression level `:1` is deliberate. Higher levels (zstd:3, zstd:5) get more compression at the cost of write throughput, which matters for snapshot create. Level 1 is roughly the same write speed as no compression while still hitting the 2× ratio on `mem.snap` — guest RAM is so compressible that a basic dictionary catches most of it.
## Cold wake and the page cache
When a sandbox is in the [Cold state](/v1/docs/under-the-hood/thermal-states/), its `mem.snap` is the only thing standing between "dead VM" and "fully restored guest." How much time the restore takes depends heavily on whether `mem.snap` is still in the host page cache.
Measurements from `agni-01` (10 samples each, raw curl to `/sandboxes/.../exec`, 1024 MB sandbox, btrfs + zstd:1):
\| Scenario | Median exec | Notes | |---|---|---| | **Hot page cache** (`sync` between runs) | **57–78 ms** | `mem.snap` still resident; only the FC process needs to spin up | | **Cold page cache** (`sync && drop_caches`) | **186–342 ms** | Full disk read of `mem.snap` working set | | **Production-realistic** (concurrent traffic on the proxy) | **\~400 ms** | Matches the homepage 360 ms p50 / 430 ms p99 | | **`keep_hot=true`** (no thermal management) | 13–14 ms | Sandbox never goes cold |
The full investigation lives at [`INVESTIGATION-cold-wake-cache.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/INVESTIGATION-cold-wake-cache.md) in the monorepo.
Bhatti doesn't currently bias eviction of `mem.snap` files in the page cache. Right after a snapshot is written, the pages are hot. Twenty-five minutes later, under memory pressure from a busy host, the kernel may have evicted them. The 360 ms p50 we publish accepts that as a fact of operating: in steady-state on a real host with real workloads, a cold wake reads `mem.snap` from disk.
`posix_fadvise(POSIX_FADV_WILLNEED)` immediately before `/snapshot/load` is a possible future optimization — it hints to the kernel to start prefetching while FC's own startup runs in parallel. Bounded by demand and not yet measured to clear an LRU-eviction bar that matters; tracked for future work.
## What changes on ext4
bhatti works correctly on ext4. The system doesn't refuse to start, no sandbox fails to boot, no snapshot is unsafe. What changes is performance and disk usage:
* **`bhatti create` is \~10× slower.** The rootfs copy is a real `read`/`write` loop (320 ms vs. 14 ms on btrfs). For 1 GiB base images this is hundreds of milliseconds of wall-clock per create; for the 4 GiB computer-tier base it's seconds.
* **Per-sandbox disk usage is linear.** Where btrfs's per-sandbox delta is the bytes the sandbox actually writes (typically single- digit MiB until you install something heavy), on ext4 each sandbox carries a sparse copy of its rootfs.ext4 — the *used* extents of the source, around 232 MiB for the computer tier. 10 sandboxes ≈ 2.3 GiB vs. \~1.1 GiB on btrfs.
* **No zstd compression.** `mem.snap` files are full-size on disk (the configured memory size). One stopped 1 GiB sandbox is 1 GiB on disk; on btrfs it's 48 MiB.
* **Snapshot resume copies the block devices for real.** Slower resume, more disk write per restore cycle.
* **The performance numbers on the homepage don't apply.** Those are measured on btrfs and don't reflect the ext4 cost cliff.
If you must run on ext4, the system still works correctly. If you have a choice — and most self-hosters do, since loopback btrfs works on any host — choose btrfs. See [Filesystem (recommended: btrfs)](/v1/docs/self-hosting/#filesystem-recommended-btrfs) for the setup recipe.
## xfs and others
xfs with `reflink=1` also supports reflink. You'll get the create- time and disk-usage benefits but not the additional \~2× from zstd (xfs doesn't compress transparently). xfs's refcount btree degrades mildly for very high clone counts (thousands of sandboxes from one base image); btrfs handles arbitrary fanout.
Other filesystems (zfs on Linux with `--reflink=auto` via OpenZFS 2.2+, bcachefs) may also reflink — the `cp` falls back gracefully if the ioctl isn't supported. We haven't measured these in production.
## See also
* [Self-hosting → Filesystem (recommended: btrfs)](/v1/docs/self-hosting/#filesystem-recommended-btrfs) — the setup recipe
* [Images → Disk usage](/v1/docs/managing/images/#disk-usage) — what you see in `/var/lib/bhatti/images/` and `/var/lib/bhatti/sandboxes/<id>/`
* [Thermal states](/v1/docs/under-the-hood/thermal-states/) — the state machine that decides when `mem.snap` gets written and read
* [Architecture → Where state lives](/v1/docs/under-the-hood/architecture/#where-state-lives) — full inventory of `/var/lib/bhatti/`
# Thermal states: hot, warm, cold
> How sandboxes pause themselves, why all snapshots are Full, the balloon trick, the conntrack trap, the circuit breaker, and what actually happens during restore.
A sandbox you aren't using doesn't keep its CPUs running, doesn't keep its memory allocated, and doesn't show up as a process on the host. It costs you nothing while it's idle. The next request wakes it.
That's the headline. The rest of this page is how it works, and what happens when it doesn't.
If you haven't read it yet, the [Firecracker engine internals](/v1/docs/under-the-hood/engine/) page is the right pre-read — this page leans heavily on three concepts from it: vCPU pause/resume, snapshot create/load, and the virtio-balloon device. There's a one-paragraph recap below.
## Background, in three paragraphs
**vCPUs.** Firecracker exposes a `PATCH /vm` API call that pauses or resumes the guest's vCPUs. "Paused" means the kernel scheduler stops running guest instructions; the FC process stays alive, the memory stays allocated, the TAP device stays attached. Resuming is another `PATCH /vm`. The whole round-trip takes single-digit milliseconds.
**Snapshots.** FC's `PUT /snapshot/create` writes the VM's full memory and its device-model state to two files: `mem.snap` (the size of the VM's RAM) and `vm.snap` (small, KB-range). `PUT /snapshot/load` does the inverse — reads the files back into a fresh FC process and resumes from there. After a successful load + resume, every guest process picks up exactly where it was when the snapshot was taken, right down to in-flight TCP connections.
**The balloon.** Firecracker exposes a virtio-balloon device. The host can tell the guest "give me back N megabytes of your RAM" via `PATCH /balloon`. The guest's balloon driver allocates that many pages from its page allocator and reports them back; the host marks those physical pages free. With `deflate_on_oom` set, the balloon automatically deflates when the guest needs the memory back. Bhatti uses this on hot→warm to take \~50% of an idle VM's RAM back without snapshotting.
## Design decisions on this page
1. **Three thermal states, not two.** Hot/warm/cold gives us a cheap mid-point (vCPUs paused but memory hot) so the common case of "you come back five minutes later" doesn't pay the cost of a snapshot restore. See [The three states](#the-three-states).
2. **All snapshots are `Full`, not `Diff`.** A real production incident (the rory incident, April 2026) traced corruption to KVM's dirty-page tracking missing host-side virtio writes during Diff snapshots. The fix was to disable `track_dirty_pages` and always snapshot every page. Slower; correct. See [Why all snapshots are Full](#why-all-snapshots-are-full).
3. **Balloon inflation on hot→warm.** When a VM goes warm, the host inflates the guest's virtio-balloon to take back \~50% of its RAM. The host gets the memory back immediately; on resume, `deflate_on_oom` lets the guest reclaim it. Undocumented elsewhere on the site; meaningful for self-hosting density. See [The balloon trick](#the-balloon-trick).
4. **Cold check uses host-side timing, not an agent query.** vCPUs are paused while warm — the agent can't respond. Querying it would either time out or wake the VM. So we use the timestamp recorded when we transitioned to warm. See [Why the cold check doesn't talk to the agent](#why-the-cold-check-doesnt-talk-to-the-agent).
5. **Circuit breaker on a stuck VM.** If the agent fails ten Activity queries in a row (\~100 s of silence), the thermal manager force-pauses the VM rather than leaving it hot and unresponsive. See [The circuit breaker](#the-circuit-breaker).
6. **Three retries before marking unknown.** Snapshot writes can fail on a transient I/O hiccup. We retry. Three failures in a row gets the sandbox marked `unknown` and surfaces an event. See [Snapshot retries](#snapshot-retries).
## The three states
```
idle 30 s idle 30 min
Hot ────────────────────► Warm ──────────────────────────► Cold
▲ │ │
│ │ │
│ warm wake ~2 ms │ cold wake (see below) │
└───────────────────────┴─────────────────────────────┘
any API request
```
\| State | Firecracker process | vCPUs | Host RAM | Resume to hot | |---|---|---|---|---| | **Hot** | alive | running | full | — | | **Warm** | alive | paused | \~50% (balloon inflated) | \~2 ms p50 | | **Cold** | dead | — | 0 (snapshot on disk) | see below |
Warm resume is the homepage benchmark number, measured on Hetzner AX102 (Ryzen 9, NVMe). It's a one-API-call resume from a paused VM — the FC process is alive, the guest pages are still in host RAM, no disk read is needed.
Cold resume has its own breakdown because it spans more than one thing: spawn a fresh FC process, read `mem.snap` from disk, restore the VM, wait for the agent to respond. How much of that disk read hits the page cache vs. SSD depends on how recently the sandbox went cold and what else is happening on the host. See [What "cold wake" actually costs](#what-cold-wake-actually-costs) below.
Slower hardware (a Raspberry Pi 5 with its capped NVMe) gets proportionally slower numbers across the board; the shape of the state machine is the same.
Transitions happen automatically. Every API call to a sandbox runs through [`EnsureHot`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go#L215-L244) first. If the sandbox is warm it resumes in milliseconds. If cold, it loads the snapshot, then your operation runs. From an API client's perspective, every sandbox is always "running" — the resume is just the first slice of latency on the first call after it went cold.
### What "cold wake" actually costs
Cold wake numbers in earlier versions of this page said "\~42 ms p50." That's the *orchestration-only* number — `bhatti start` returning, with `mem.snap` still hot in page cache from the snapshot we just wrote. The homepage benchmark, which measures end-to-end from an HTTP request to a response through the public proxy, says **360 ms p50 / 430 ms p99**. The two numbers measure different things; both are real.
Measurements on `agni-01` (10 samples each, raw curl to `/sandboxes/.../exec`, 1024 MB sandbox, btrfs + zstd:1):
\| Scenario | Median exec | Notes | |---|---|---| | **Hot page cache** (`sync` between runs) | **57–78 ms** | `mem.snap` still resident; only the FC process needs to spin up | | **Cold page cache** (`sync && drop_caches`) | **186–342 ms** | Full disk read of `mem.snap` working set | | **Production-realistic** (concurrent traffic on the proxy) | **\~400 ms** | Matches the homepage 360 ms p50 / 430 ms p99 | | **`keep_hot=true`** (no thermal management) | 13–14 ms | Sandbox never goes cold |
The gap between the two-digit and three-digit numbers is the disk read of `mem.snap`. With page cache hot, the kernel has the working set already; FC starts in tens of ms and resume is fast. With page cache cold, the read happens for real. Twenty-five minutes of memory pressure between snapshot write and resume is enough to evict most of the relevant pages on a busy host.
Full methodology, raw data, and what we don't yet know are at [`INVESTIGATION-cold-wake-cache.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/INVESTIGATION-cold-wake-cache.md). The page cache framing belongs as much to the storage layer as to the thermal layer — see [Storage → Cold wake and the page cache](/v1/docs/under-the-hood/storage/#cold-wake-and-the-page-cache) for the wider context (compression ratios, ext4 fallback, what's not tested).
You can opt out per sandbox with `bhatti create --keep-hot` on creation, or `bhatti edit <name> --keep-hot` later. A keep-hot sandbox skips thermal management entirely ([`pkg/server/server.go:489-494`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go#L489-L494)) and holds its full RAM. This is for sandboxes that maintain external connections — a Slack WebSocket, a Discord gateway, a long-running training job that you don't want randomly suspended.
## Hot → Warm
After 30 seconds with no API activity *and* no attached TTY sessions, the thermal manager pauses the VM:
1. `PATCH /vm {"state":"Paused"}` to Firecracker's API. The vCPU pause is single-digit milliseconds.
2. Inflate the virtio-balloon to \~50% of guest memory ([`pkg/server/server.go:626-632`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go#L626-L632)). The host gets that RAM back immediately — the balloon driver in the guest hands free pages back to the host.
3. Record the pause time as the sandbox's last-activity timestamp. The warm-to-cold timer starts from now, not from your last `exec`.
The Firecracker process is still alive, the TAP device is still attached, the IP is still allocated. The vCPUs are simply frozen. Resuming is another `PATCH /vm` call.
The "no attached TTY sessions" check matters. If someone is in `bhatti shell` right now, we don't pause — that's a person staring at a terminal. The attached-session count comes from the agent's `Activity` response ([`cmd/lohar/handler.go:35`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/handler.go#L35)), which is the only authoritative source.
## Warm → Cold
After 30 minutes warm, the manager takes a Full memory snapshot to disk and kills the Firecracker process ([`pkg/engine/firecracker/lifecycle.go:21-118`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go#L21-L118)):
1. `PATCH /vm {"state":"Paused"}` (skipped if already paused).
2. The host runs `sync` *inside the guest* via the agent. Pausing vCPUs doesn't flush dirty pages from the guest's page cache to the virtio-blk device. We learned this the hard way; without it, the snapshot can contain a filesystem state where files exist but their contents aren't on disk yet.
3. `PUT /snapshot/create {"snapshot_type":"Full","snapshot_path":...,"mem_file_path":...}`. The mem file is the size of the VM's RAM. For a 512 MB VM, that's 512 MB on disk and \~620 ms p50 on Hetzner.
4. Sanity-check the snapshot artifacts (right size, valid). On failure, we error out — a corrupt snapshot is worse than no snapshot.
5. Kill the Firecracker process (SIGTERM, then SIGKILL after 3 s).
6. Update the database — sandbox is now `stopped`, thermal is `cold`.
The TAP device is **not** destroyed. The snapshot contains virtio-net state that references the TAP device by name; if we delete and recreate it, the resumed guest's network stack ends up referencing a TAP that no longer exists. So TAPs survive across stop/start cycles ([`pkg/engine/firecracker/lifecycle.go:108-114`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go#L108-L114)). They're only torn down on `Destroy()`.
### Why all snapshots are Full
Firecracker supports [Diff snapshots](https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md): only write pages modified since the last snapshot. For an idle VM this is 10–50 MB instead of 512 MB, taking \~52 ms instead of \~600 ms. We had this enabled.
Then the rory incident. April 2026. A user's persistent sandbox came back after a restore with a corrupted virtio ring buffer. The agent was unreachable. The VM was wedged. We had to destroy it and lost the working state of the sandbox.
Three failures compounded:
* **Diff snapshot corruption.** `mem.snap` was a Diff — only dirty pages — and the dirty page bitmap was incomplete. KVM tracks dirty pages at the level of guest writes, but certain virtio device writes happen in Firecracker's userspace device-model code and don't go through KVM. Those writes don't trigger KVM's dirty page bitmap. Restoring a Diff snapshot then loads pages from a stale base, and any in-flight virtio ring buffer state gets clobbered.
* **No snapshot verification** — we'd written a corrupt mem file and marked the snapshot as good.
* **Silent thermal skip** on a different sandbox during the same daemon shutdown that took rory cold, because of an unrelated bug. So we lost a sandbox AND another one didn't get snapshotted.
The full audit and fix plan is in [`docs/archive/PLAN-reliability.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-reliability.md). The relevant fix here is in [`pkg/engine/firecracker/create.go:307-318`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/create.go#L307-L318):
```go
// track_dirty_pages is disabled — all snapshots are Full. This eliminates
// Diff snapshot corruption (the rory incident) at the cost of ~500ms extra
// per snapshot. With NVMe + btrfs this is negligible.
fcPut(ctx, client, "/machine-config", fmt.Sprintf(
`{"vcpu_count":%d,"mem_size_mib":%d,"track_dirty_pages":false,"huge_pages":%q}`,
vcpuCount, memMB, hugePages))
```
`track_dirty_pages: false` means every snapshot writes every page. The snapshot type field in [`pkg/engine/firecracker/lifecycle.go:77`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go#L77) is hardcoded `Full`.
It's slower. A 512 MB Full snapshot is \~620 ms p50 on a Hetzner AX102 (Ryzen 9, NVMe). On any modern host disk this is fine, and the trade is correctness for speed — which I'll take any day after losing rory.
We'll re-enable Diff if and when Firecracker's dirty-page tracking covers all write paths. Until then, every snapshot is a complete, self-consistent image.
### Snapshot retries
Disks fail in transient ways: a brief I/O stall, a btrfs balance running in the background, a momentary contention spike. A snapshot that fails once usually succeeds on the second try. The thermal manager retries up to three times per cycle before marking the sandbox `unknown` ([`pkg/server/server.go:514-541`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go#L514-L541)):
```
attempt 1 fails → log warning, increment counter, try again next cycle
attempt 2 fails → log warning, increment counter, try again next cycle
attempt 3 fails → log error, mark sandbox unknown, record event
```
Each failure also records a `thermal.snapshot_failed` event with the attempt number and error, so an operator can see why (`bhatti admin events --type thermal.snapshot_failed`).
### Why the cold check doesn't talk to the agent
Look at [`runThermalCycle`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go#L496) and you'll see the warm→cold transition computed without an agent query. That's deliberate.
Agents can't respond when vCPUs are paused. A query would either time out — silently skipping the cold check — or trigger a vCPU resume to service the request, which defeats the entire reason we paused. So the cold check uses the timestamp the manager set when it transitioned the sandbox to warm.
It's a small detail. It took an afternoon to figure out why my warm sandboxes never went cold.
## Cold → Hot
Any API call to a cold sandbox triggers `EnsureHot`, which calls `Start` ([`pkg/engine/firecracker/lifecycle.go:251-431`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go#L251-L431)):
1. Recreate the TAP device if it's gone (e.g. after a daemon crash that triggered the orphan cleanup). The TAP comes back attached to the same user bridge it was on before.
2. Spawn a fresh Firecracker process. Its API socket name is the original plus `.resume` (so we don't collide with the dead one and so the path doesn't grow on repeated cycles — Unix socket paths have a length limit, and we hit it once).
3. `PUT /snapshot/load`. Crucially with `network_overrides` so FC remaps the TAP device name to whatever it's actually called now:
```json
{
"snapshot_path": "/vm.snap",
"mem_backend": {"backend_path": "/mem.snap", "backend_type": "File"},
"resume_vm": true,
"network_overrides": [{"iface_id": "eth0", "host_dev_name": "tap1234abcd"}]
}
```
4. Build a new TCP `AgentClient` over the guest's IP. Vsock state doesn't survive snapshot restore — see [decisions](/v1/docs/under-the-hood/decisions/) — but virtio-net does, so post-restore agents are always reached over TCP.
5. `WaitReady` — poll `exec true` for up to 30 seconds. If we get a response, the sandbox is hot. If we don't, the sandbox is marked `restoreFailed`.
A successful cold→hot resume is 57–78 ms with a hot page cache and 186–342 ms with it cold (`agni-01`, btrfs + zstd:1, 1024 MB sandbox). The host-side homepage benchmark sees 360 ms p50 / 430 ms p99 end-to-end through the public proxy. Most of the cold-cache cost is the `mem.snap` read; the FC API call itself returns in tens of milliseconds. The kernel and the guest's processes don't restart — they pick up exactly where they were. See [the cold-wake breakdown above](#what-cold-wake-actually-costs) and [Storage → Cold wake and the page cache](/v1/docs/under-the-hood/storage/#cold-wake-and-the-page-cache) for the full picture.
### When restore fails
If `PUT /snapshot/load` returns an error, or if the agent doesn't respond within 30 seconds, the VM is marked corrupt ([`pkg/engine/firecracker/lifecycle.go:264-275`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/lifecycle.go#L264-L275)):
```
Error: sandbox "dev" snapshot is corrupt: <reason> —
use 'bhatti start --force' to retry or destroy and recreate
(volume data is safe).
```
`bhatti start --force <name>` clears the circuit-breaker flag and retries once. If the underlying issue (disk full, broken btrfs subvolume) is fixed, this gets you back to running. If not, it fails again and you keep the flag.
Volume data is on separate ext4 images attached as virtio-blk drives; they survive even when the memory snapshot doesn't. So the fix is usually: detach volumes, destroy the sandbox, recreate, reattach.
## The balloon trick
This is the part of thermal management most other docs skip. When a VM goes warm, the host doesn't just freeze the vCPUs — it tells the guest's virtio-balloon driver to claim back \~50% of the VM's memory ([`pkg/server/server.go:626-632`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go#L626-L632)):
```go
te.BalloonSet(bCtx, sb.EngineID, memMiB/2)
```
What happens behind that one line:
* The virtio-balloon driver inside the guest allocates pages from the guest's page allocator and tells the host "I don't need these."
* The host marks those pages free and reclaims the physical memory.
* A 1 GB VM that was holding 1 GB of host RAM now holds \~500 MB.
On resume, the guest's `deflate_on_oom` setting ([`pkg/engine/firecracker/create.go:333`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/firecracker/create.go#L333)) lets it reclaim balloon pages automatically when something allocates. You don't configure this; the guest just gets its memory back as it needs it.
For a self-hosted box with twenty sandboxes, this is the difference between "I can run twenty VMs" and "I can run forty." It happens automatically; you don't configure it.
The flip side: if a guest's working set is genuinely 512 MB and you inflate to 256 MB, you'll force pages to the guest's swap (if any) or trigger a guest OOM. The 50% default is conservative for typical workloads; if you have a memory-hungry VM that you keep\_hot anyway, this isn't your problem.
## The host-side activity cache
Every API call records `time.Now()` in `s.lastActivity` ([`pkg/server/server.go:115-122`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go#L115-L122)). The thermal cycle checks this map before opening a TCP connection to the agent. If the cached timestamp is within the warm timeout, the agent query is skipped.
Why bother? Because with 50 active sandboxes the cycle runs every 10 seconds, and naively that's 50 TCP connections every 10 seconds — most of them returning "still active." The cache eliminates the connections for sandboxes that have had recent API traffic. Only genuinely-idle sandboxes get queried.
The activity cache is a heuristic, not the source of truth. The agent's own `lastActivity` (an `int64` updated by every exec and stdin) is authoritative — but querying it costs a connection. So we use the cache as a fast-path negative ("definitely active, skip") and fall through to the agent for everything else.
## The circuit breaker
If the agent fails to respond ten times in a row to Activity queries — about 100 seconds of silence — the manager force-pauses the VM ([`pkg/server/server.go:580-602`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go#L580-L602)):
```go
const maxThermalFailures = 10
```
Pausing is a Firecracker API call; it doesn't need the agent. After force-pause we also inflate the balloon, so even a wedged VM can't hog memory indefinitely.
This catches the "agent is alive but stuck" case. A user sees their sandbox go to `warm` even though they never asked for it; their next request wakes it normally. If the agent is genuinely dead, the next restore will fail and they'll get the corrupt-snapshot error described above.
The threshold of 10 is arbitrary. Real values:
* 10-second tick interval × 10 failures = \~100 seconds of unresponsiveness
* A normally-loaded agent answers Activity in single-digit milliseconds
* A real network or scheduling hiccup might lose 1–2 queries; not 10
If you want to monitor this, watch for `thermal.force_pause` events.
## After a snapshot/restore: the conntrack pause
When a VM resumes from cold, the host's iptables conntrack table still has stale entries for the guest's IP. Host-initiated connections work fine (rule 4 of the iptables setup matches and conntrack replaces the stale entries). But *guest-initiated* outbound connections — say, the guest does `ping 8.8.8.8` — can hit a stale conntrack entry and the SYN gets stuck in retransmit backoff for tens of seconds.
In normal operation this doesn't matter, because the host always initiates connections to the agent. But it is a thing you'll notice if you're benchmarking outbound throughput right after a resume. An ARP flush helps:
```bash
ip neigh flush dev brbhatti-1
```
We don't run this automatically because it's only relevant for a brief window post-resume and most workloads don't hit it.
## What you'll see in practice
A short field guide.
**`bhatti list` shows `cold` next to your sandbox.** That's normal. It's saving you memory. Run any command on it and it'll wake.
**First request after a long pause is slow.** That's a cold wake. Subsequent requests are normal speed.
**Sandbox transitioned to warm even though I'm using it.** The check requires no attached TTY *and* >30 s since the last API call. Background processes inside the VM don't count as activity from the manager's perspective. If you have a build running and want it to keep\_hot, set the flag.
**`bhatti list` shows `unknown`.** Three snapshot retries failed. Look at the bhatti server logs and `bhatti admin events --type thermal.snapshot_failed`. You can usually `bhatti start --force` once the underlying issue (disk full, broken btrfs subvolume) is fixed.
**Resume fails with "snapshot is corrupt".** Volume data is safe; destroy and recreate. If this happens twice in a row on the same VM, file an issue — we'd like to see it.
## Where to go next
* [How a sandbox boots](/v1/docs/under-the-hood/architecture/) — the cold start path in detail
* [Networking](/v1/docs/under-the-hood/networking/) — TAP devices, per-user bridges, why TCP survives snapshot/restore
* [Lohar](/v1/docs/under-the-hood/lohar-the-blacksmith/) — the agent that resumes inside the VM
* [Decisions & learnings](/v1/docs/under-the-hood/decisions/) — the rory incident in full, plus other paid-for knowledge
# The wire protocol
> Binary framing between host and guest — frame format, frame types, atomic writes, kill semantics, and the choices that shaped it.
All communication between the bhatti host process and a guest agent happens over a binary framing protocol. The same protocol runs over TCP (production), Unix sockets (tests), and — historically — vsock (left over from before we fully moved to TCP). The protocol is engine-independent: the entire agent test suite runs on macOS over `net.Pipe()` without any VM or root.
## Design decisions on this page
1. **Custom binary framing, not gRPC or HTTP.** The framing layer is \~130 lines of Go. It runs inside a microVM where every dependency is dead weight. gRPC would add protobuf codegen, a runtime, and complexity disproportionate to a protocol with eight frame types. See [Why not HTTP or gRPC](#why-not-http-or-grpc).
2. **`WriteFrame` does one `Write()` call.** The entire frame is assembled into a single buffer before going to the wire. Otherwise, two goroutines writing concurrent stdout/stderr chunks would interleave at byte boundaries and corrupt frames. See [Atomic frame writes](#atomic-frame-writes).
3. **AUTH must be the first frame.** If a token is configured, lohar refuses any other frame on a fresh connection until it sees an AUTH frame within 5 seconds. See [Auth](#auth).
4. **Unknown frame types are skipped, not errored.** A new frame type can be added without breaking older clients. See [Forward compatibility](#forward-compatibility).
5. **One connection per operation.** Most agent calls open a TCP connection, do their thing, close. TTY sessions and forwards are the exceptions. See [Connection model](#connection-model).
## Frame format
```
┌────────────────┬───────────┬──────────────────────┐
│ Length (4B BE) │ Type (1B) │ Payload (N bytes) │
└────────────────┴───────────┴──────────────────────┘
```
**Length** is a 4-byte big-endian unsigned integer. It equals `1 + len(Payload)` — the type byte plus the payload. It does *not* include the 4-byte length prefix itself.
**Type** is a single byte identifying the frame kind.
**Payload** is variable-length, up to `MaxFrameSize - 1` bytes.
**Maximum frame size** is 1 MB ([`pkg/agent/proto/constants.go:69`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/constants.go#L69)). Both `WriteFrame` and `ReadFrame` enforce this — oversized frames are rejected, not truncated.
The encoding is straightforward ([`pkg/agent/proto/frame.go:10-28`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/frame.go#L10-L28)):
```go
buf := make([]byte, 4+frameLen)
binary.BigEndian.PutUint32(buf[0:4], uint32(frameLen))
buf[4] = msgType
copy(buf[5:], payload)
_, err := w.Write(buf)
```
## Atomic frame writes
The single-buffer approach above isn't an optimization — it's a correctness requirement. The agent's piped exec has stdout and stderr goroutines writing to the same connection at the same time. If `WriteFrame` did three `Write()` calls (length, type, payload), two goroutines could interleave at any byte boundary. The kernel would deliver something like:
```
[len=1024][type=STDOUT] ← goroutine A starts
[len=512][type=STDERR] ← goroutine B writes its full frame
[1024 bytes of stdout] ← goroutine A finishes, but the order on the wire is now broken
```
The receiver tries to parse `len=1024 type=STDOUT` plus the next 1024 bytes — but the next bytes are goroutine B's length prefix, not goroutine A's stdout. Frames are corrupt for the rest of the connection.
By assembling the whole frame into one buffer and calling `Write()` once, the kernel atomically delivers the whole frame to the socket buffer (for sizes under the kernel's pipe-buffer limit, which is much larger than 1 MB on every platform we run on). Concurrent goroutines get their frames serialized end-to-end, never interleaved.
This is why the [piped exec](/v1/docs/under-the-hood/lohar-the-blacksmith/#piped-exec-the-simpler-path) goroutines push frames through a single channel to a single writer goroutine — defense in depth on top of `WriteFrame`'s atomicity. Both levels matter.
## Frame types
All frame type constants are in [`pkg/agent/proto/constants.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/constants.go).
### I/O streams
\| Type | Byte | Direction | Payload | |---|---|---|---| | `STDIN` | `0x01` | host → guest | raw bytes for child's stdin | | `STDOUT` | `0x02` | guest → host | child's stdout bytes | | `STDERR` | `0x03` | guest → host | child's stderr bytes |
### Control
\| Type | Byte | Direction | Payload | |---|---|---|---| | `RESIZE` | `0x04` | host → guest | `[u16 rows BE][u16 cols BE]` (4 bytes exactly) | | `EXIT` | `0x05` | guest → host | `[i32 exit_code BE]` (4 bytes exactly) | | `ERROR` | `0x06` | either | UTF-8 error message (variable length) | | `KILL` | `0x07` | host → guest | empty payload |
### Exec
\| Type | Byte | Direction | Payload | |---|---|---|---| | `EXEC_REQ` | `0x10` | host → guest | JSON-encoded `ExecRequest` |
The `ExecRequest` shape is in [`pkg/agent/proto/messages.go:5-19`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/messages.go#L5-L19): `argv`, `env`, `tty`, `rows`, `cols`, `cwd`, `session_id`, `max_idle_sec`, `if_detached`, `detach`, `output_file`, `session`. Most are optional pointers — nil means "use default."
### Auth
\| Type | Byte | Direction | Payload | |---|---|---|---| | `AUTH` | `0x11` | host → guest | raw token bytes |
If a token is configured (see [Auth](#auth) below), this must be the first frame on every connection.
### Port forwarding
\| Type | Byte | Direction | Payload | |---|---|---|---| | `FWD_REQ` | `0x20` | host → guest | JSON `{"port": 8080}` | | `FWD_RESP` | `0x21` | guest → host | JSON `{"status": "ok"}` or `{"status":"error","message":"..."}` |
### Sessions
\| Type | Byte | Direction | Payload | |---|---|---|---| | `EXEC_LIST_REQ` | `0x30` | host → guest | empty | | `EXEC_LIST_RESP` | `0x31` | guest → host | JSON `[]SessionInfo` | | `EXEC_KILL` | `0x32` | host → guest | JSON `{"session_id": "..."}` | | `SESSION_INFO` | `0x33` | guest → host | JSON `SessionInfo` (sent on create or attach, before STDOUT) |
### Activity
\| Type | Byte | Direction | Payload | |---|---|---|---| | `ACTIVITY_REQ` | `0x40` | host → guest | empty | | `ACTIVITY_RESP` | `0x41` | guest → host | JSON `ActivityInfo` (last activity timestamp, attached count) |
### File operations
\| Type | Byte | Direction | Payload | |---|---|---|---| | `FILE_READ_REQ` | `0x50` | host → guest | JSON `{"path":"...","offset":1,"limit":2000,"max_bytes":51200}` | | `FILE_READ_RESP` | `0x51` | guest → host | JSON `{"size":1234,"mode":"0644"}` | | `FILE_WRITE_REQ` | `0x52` | host → guest | JSON `{"path":"...","mode":"0644","size":1234}` | | `FILE_WRITE_RESP` | `0x53` | guest → host | JSON `{"status":"ok"}` | | `FILE_STAT_REQ` | `0x54` | host → guest | JSON `{"path":"..."}` | | `FILE_STAT_RESP` | `0x55` | guest → host | JSON `FileInfo` | | `FILE_LS_REQ` | `0x56` | host → guest | JSON `{"path":"..."}` | | `FILE_LS_RESP` | `0x57` | guest → host | JSON `[]FileInfo` |
### Systemctl IPC (in-guest, not host↔guest)
\| Type | Byte | Direction | Payload | |---|---|---|---| | `SYSTEMCTL_REQ` | `0x60` | client → lohar (Unix socket) | JSON `SystemctlRequest` | | `SYSTEMCTL_RESP` | `0x61` | lohar → client | JSON `SystemctlResponse` |
These are spoken over `/run/bhatti/systemctl.sock` *inside* the guest, not over the host↔guest TCP connection. They exist so the systemctl shim's user-facing invocation can ask PID 1 lohar to perform privileged operations — see [Lohar's systemctl IPC](/v1/docs/under-the-hood/lohar-the-blacksmith/#privileged-systemctl-from-a-non-root-caller).
The trust boundary is `SO_PEERCRED` on the socket, **not** anything in the request payload. A non-root caller could otherwise forge a UID claim and get a privileged operation done.
## Connection model
Two TCP ports inside each VM, two purposes:
* **Port 1024** (control) — exec, sessions, files, activity queries
* **Port 1025** (forward) — port forwarding / TCP tunneling
`MaxFrameSize` is 1 MB. Both ports also exist as vsock listeners for historical reasons, but the host always uses TCP — see [the lohar story](/v1/docs/under-the-hood/lohar-the-blacksmith/#why-not-real-systemd) for why.
### Control connection lifecycle
One connection per operation. The host dials port 1024, optionally sends an `AUTH` frame, sends exactly one request frame, reads responses until the operation completes, then the connection closes.
```
Host Lohar
│ │
├──TCP connect :1024 ─────────────────►│
├──AUTH frame (if token configured) ──►│
├──EXEC_REQ frame ───────────────────►│
│ ├──fork/exec child
│◄──STDOUT frame──────────────────────┤
│◄──STDOUT frame──────────────────────┤
│◄──STDERR frame──────────────────────┤
│◄──EXIT frame────────────────────────┤
└──connection closed──────────────────┘
```
Exception: TTY sessions keep the connection open for bidirectional I/O. The host sends `STDIN` and `RESIZE` frames; the guest sends `STDOUT` frames and eventually an `EXIT` frame. If the host disconnects, the session detaches (process keeps running, scrollback buffer captures output — see [Sessions](/v1/docs/under-the-hood/lohar-the-blacksmith/#sessions)).
### Forward connection lifecycle
One connection per tunnel. After the `FWD_REQ`/`FWD_RESP` handshake, the framing protocol is **abandoned** — the connection becomes a raw bidirectional TCP relay.
```
Host Lohar Target (localhost:8080)
│ │ │
├──TCP connect :1025 ─────────────────►│ │
├──AUTH frame ────────────────────────►│ │
├──FWD_REQ {"port": 8080} ───────────►│ │
│ ├──TCP connect :8080 ─────►│
│◄──FWD_RESP {"status": "ok"} ────────┤ │
│ │ │
│═══ raw bytes (no framing) ══════════►│═════════════════════════►│
│◄═════════════════════════════════════│◄═════════════════════════│
```
This is what `bhatti exec` doesn't use, but the reverse proxy does. When you hit a published URL, the daemon picks up an HTTP request, opens a forward connection to the agent, sends `FWD_REQ` for the target port, and then pipes the request body through. The agent relays it to localhost:port inside the VM, and the response comes back the same way.
Forward connections don't have framing because they're proxying arbitrary TCP — HTTP, WebSocket, SSH, whatever. Adding framing would mean parsing HTTP on the agent side, which we don't want to do.
## Auth
If a token is configured (via the [config drive](/v1/docs/under-the-hood/lohar-the-blacksmith/#boot-sequence) at boot), the first frame on every connection must be `AUTH` with the token as payload. Lohar validates it within a 5-second deadline. Invalid or missing auth gets an `ERROR` frame and the connection is closed.
The token is generated per-sandbox during `Create()` — 16 random bytes, hex-encoded — and injected into the VM via the config drive. It's then stored in the host's `AgentClient` and not used for anything else. If the daemon restarts and re-reads the agent token from `fc_state`, the token survives — VMs across daemon restarts use the same token they were created with.
The token comparison uses [`subtle.ConstantTimeCompare`](https://pkg.go.dev/crypto/subtle#ConstantTimeCompare) ([`cmd/lohar/handler.go:69`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/handler.go#L69)) so a network observer can't time the comparison to figure out a prefix of the token.
## File read protocol
File reads support server-side truncation to avoid transferring large files when the consumer only needs the first N lines. This matters because coding agents always truncate file reads — typically 2000 lines or 50 KB — and shipping a 100 MB log over the wire just to discard 99.95% of it is a waste.
```
host guest
FILE_READ_REQ
{"path":"/app.log","offset":1,"limit":2000,"max_bytes":51200}
│
▼
FILE_READ_RESP
{"size":10485760,"mode":"0644"} ← total file size
│
▼
STDOUT (line data)
STDOUT (line data)
... ← stops when limit or max_bytes hit
│
▼
EXIT code=0
```
* `offset` — 1-indexed line number to start from (0 or absent = beginning).
* `limit` — maximum lines to return (0 = unlimited).
* `max_bytes` — maximum bytes to return (0 = unlimited).
Whichever limit hits first stops the read. Without any truncation parameters the full file is streamed. The `FILE_READ_RESP` always contains the *total* file size so the consumer knows whether content was truncated.
Directories and non-regular files are rejected with an `ERROR` frame. File reads are cancellable via context — closing the connection gives lohar a broken pipe, stopping the transfer immediately.
## File write protocol
Writes are atomic. Lohar writes to a temp file, fsyncs, then renames over the target. Concurrent readers see either the old content or the new content, never a half-written state.
```
host guest
FILE_WRITE_REQ
{"path":"/workspace/app.js","mode":"0644","size":1234}
│
STDIN (content bytes) │
STDIN (content bytes) │
... │ ← until size bytes sent
│
▼
FILE_WRITE_RESP
{"status":"ok"}
```
Negative sizes are rejected (prevents silent data loss from a missing `Content-Length` on the HTTP side that would otherwise pass through as a default zero). If the connection drops mid-write, the temp file is cleaned up.
Why fsync before rename? See [the explanation in lohar.md](/v1/docs/under-the-hood/lohar-the-blacksmith/#file-operations) — short version: the host might snapshot the VM the next millisecond, and dirty pages in the guest's page cache aren't part of the FC snapshot. Without fsync, the rename is metadata-durable but the data isn't on the virtio-blk device.
## Kill semantics
\| Context | Signal | Why | |---|---|---| | **Piped exec** (non-TTY) | `SIGKILL` to process group | Agents need instant, reliable abort. Child processes (npm → node) must die immediately. | | **TTY session disconnect** | None (detach) | Process keeps running, session detaches. Scrollback captures output. | | **TTY session KILL frame** | `SIGTERM` to process group | Allows graceful shutdown. If the process handles SIGTERM and survives, the session remains reattachable. | | **EXEC\_KILL API** | `SIGKILL` to process group | Explicit force-kill by session ID. | | **Idle timer expiry** | `SIGKILL` to process group | Session is abandoned, no observer. |
All kill operations target the *process group* (negative PID), not just the session leader. This requires `Setpgid: true` on the `SysProcAttr` so child processes are in the same group. Without this, `npm install` would survive killing the shell that launched it.
## Forward compatibility
`ReadFrame` in the client skips unknown frame types rather than erroring. This allows the protocol to be extended without breaking existing clients — a new frame type added to lohar won't crash an older bhatti host.
The flip side: a new client sending a frame an older agent doesn't understand will get its connection closed (the agent ignores the frame, but no response comes back). So forward compatibility is asymmetric — host upgrades cleanly, but the agent has to be at least as new as the most-recent frame type the host wants to use.
In practice the only frame types added since v1.0 are `SYSTEMCTL_REQ`/`RESP` (in-guest, not host↔guest) and the file ops expanding from read/write to include stat/ls. Both are guarded by capability checks before use.
## Why not HTTP or gRPC
The framing layer is \~130 lines of Go ([`pkg/agent/proto/frame.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/frame.go)) and handles concurrent stdout/stderr multiplexing, binary file transfers, and terminal I/O with zero dependencies.
gRPC would mean:
* protobuf code generation in the build pipeline
* a runtime dependency inside the VM (heavy for an agent that's supposed to be small and static)
* complexity disproportionate to a protocol with eight frame types
* streaming RPCs would handle the stdout/stderr case but with more moving parts than the current channel-serialized writer
HTTP would mean:
* header parsing on every exec
* chunked encoding negotiation
* content-type negotiation
* a connection that's already authenticated and never leaves the host paying the cost of HTTP semantics it doesn't need
The framing protocol is what we'd write if we sat down to design a host↔guest protocol from scratch. Length, type, payload. Atomic write of the whole thing. That's it.
## Where to go next
* [Lohar](/v1/docs/under-the-hood/lohar-the-blacksmith/) — the agent that speaks this protocol on the guest side
* [Architecture](/v1/docs/under-the-hood/architecture/) — the bigger picture of host ↔ guest communication
* [Decisions & learnings](/v1/docs/under-the-hood/decisions/) — including why post-restore vsock didn't work and we ended up TCP-everywhere
# Updating & Uninstalling
> Keep bhatti up to date or remove it cleanly.
## Updating
[Section titled “Updating”](#updating)
```bash
# CLI (any machine)
bhatti update
# Server (updates CLI, kernel, rootfs, lohar)
sudo bhatti update
# Server, but only the CLI binary
bhatti update --cli-only
# Server + install additional rootfs tiers
sudo bhatti update --tiers all
sudo bhatti update --tiers browser,docker
```
Only changed components are downloaded — if the kernel and rootfs haven’t changed since your last update, they’re skipped automatically.
Use `--cli-only` on a server when you want to test a CLI fix without touching kernel/rootfs/lohar. Use `--tiers` to pull additional rootfs images (`minimal`, `browser`, `docker`, `computer`) without re-running the full install.
Or re-run the install command:
```bash
curl -fsSL bhatti.sh/install | bash # CLI
curl -fsSL bhatti.sh/install | sudo bash # server
```
Note
`bhatti update` updating all server components (not just the CLI binary) requires v1.7.3 or later. On older versions, use the curl command above.
### Check for updates
[Section titled “Check for updates”](#check-for-updates)
```bash
bhatti version
```
Shows the installed version and checks GitHub for the latest release. Update notices also appear in `bhatti admin status`.
## Uninstalling
[Section titled “Uninstalling”](#uninstalling)
### Keep data (reinstallable)
[Section titled “Keep data (reinstallable)”](#keep-data-reinstallable)
```bash
curl -fsSL bhatti.sh/uninstall | sudo bash
```
Removes binaries and the systemd service. Keeps `/var/lib/bhatti` (images, volumes, snapshots, secrets, database) so you can reinstall later without losing state.
### Remove everything
[Section titled “Remove everything”](#remove-everything)
```bash
curl -fsSL bhatti.sh/uninstall | sudo bash -s -- --purge
```
Removes all data including volumes, snapshots, and the encryption key.
### CLI only
[Section titled “CLI only”](#cli-only)
```bash
sudo rm /usr/local/bin/bhatti
rm -rf ~/.bhatti
```
# Architecture overview
> A 10-minute mental model of bhatti — what runs where, what state lives where, and the four structural decisions that shape everything else.
This page is the map. It exists so that when you go into [Lohar](/docs/under-the-hood/lohar-the-blacksmith/), [Thermal states](/docs/under-the-hood/thermal-states/), [Networking](/docs/under-the-hood/networking/), or [Decisions](/docs/under-the-hood/decisions/), you already have a sense of where each fits.
If you're using bhatti, you don't need this page. If you're evaluating it for production or thinking about contributing, this is the right starting point.
## The shape
Bhatti is a **host daemon plus three helper processes per running sandbox**. The daemon (`bhatti serve`) is **pure Go and never links the VMM**. Per sandbox it spawns `bhatti-vmm` (the microVM itself) and, per owner, `bhatti-netd` (the network gateway); inside each VM, `lohar` runs as PID 1. The daemon drives `bhatti-vmm` out-of-band over a **control socket** and talks to `lohar` over **vsock** with a small binary protocol.
`bhatti` is a single Go binary that does many jobs depending on how you invoke it:
* `bhatti serve` — the daemon (HTTP API, krucible engine, thermal manager, public proxy, all in one process)
* `bhatti create`, `bhatti exec`, `bhatti shell`, ... — the CLI client, talking to the daemon's HTTP API
* `bhatti user create`, `bhatti admin status` — admin commands that read the daemon's SQLite database directly (root only)
Two more binaries ship in the release bundle and are launched by the daemon, never by you:
* `bhatti-vmm` — the cgo helper that *is* the VM. It links [krucible](/docs/under-the-hood/engine/) (bhatti's fork of [libkrun](https://github.com/containers/libkrun)) and calls `krun_start_enter`, at which point the process **becomes** the guest and blocks for the life of the VM. The daemon controls it from outside over a Unix-socket control channel.
* `bhatti-netd` — a per-owner userspace [gVisor](https://gvisor.dev/) network gateway. It polices egress, isolates the guest from the host, and makes same-owner sandboxes reachable — with no host TAP devices, bridges, or iptables rules. **On by default.**
`lohar` runs as PID 1 inside every microVM. It handles exec, file operations, PTY sessions, and pretends to be `systemctl` and `journalctl` for in-guest callers. The full story is in [Lohar](/docs/under-the-hood/lohar-the-blacksmith/).
```
┌─ Host ──────────────────────────────────────────────────────────────────────┐
│ │
│ ┌─ bhatti daemon (bhatti serve, single pure-Go process) ───────────────┐ │
│ │ │ │
│ │ REST / WS API Engine (krucible) Store (SQLite, WAL mode) │ │
│ │ :8080 create / destroy sandboxes, users, secrets, │ │
│ │ public proxy stop / start templates, volumes, events │ │
│ │ thermal manager pause / resume │ │
│ │ snapshot / fork │ │
│ │ exec, shell, files │ │
│ │ │ │
│ └───────────┬───────────────────────────────────┬───────────────────────┘ │
│ │ │ │
│ control socket (UDS) unix socket (per owner) │
│ PAUSE/RESUME/STATUS/SNAPSHOT virtio-net │
│ │ │ │
│ ▼ ▼ │
│ ┌─ bhatti-vmm × N (krucible/libkrun VMM) ──┐ ┌─ bhatti-netd × owners ─┐ │
│ │ │ │ │ │
│ │ KVM (Linux) / HVF (macOS) │◄──┤ gVisor userspace │ │
│ │ qcow2 CoW root (/dev/vda) │ │ netstack gateway │ │
│ │ config drive (/dev/vdb) volumes … │ │ policed egress │ │
│ │ │ │ sibling reachability │ │
│ │ ┌─ lohar (PID 1) ─────────────────┐ │ └────────────────────────┘ │
│ │ │ vsock ── control plane │ │ │
│ │ │ (exec, files, PTY) │ │ │
│ │ │ vsock ── forward plane │ │ │
│ │ │ sessions scrollback │ │ │
│ │ │ systemctl shim journald │ │ │
│ │ └─────────────────────────────────┘ │ │
│ └───────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
The daemon's HTTP API is the only public surface. Everything below it — the control socket, the vsock agent protocol, the netd gateway — is internal. A client outside the host talks only to `:8080` (or `:443` in domain mode).
## Four decisions that shape the rest
These aren't all the choices in the codebase, but they're the ones you can't understand the rest of the docs without.
### 1. The daemon is a single Go process
REST/WS API, krucible engine, thermal manager, public proxy, SQLite store, rate limiter — all in one process. No microservices, no message queue, no separate workers. The daemon *coordinates* the per-sandbox `bhatti-vmm` and per-owner `bhatti-netd` helpers, but it owns all the state and all the decisions. State lives in SQLite (see [Where state lives](#where-state-lives)).
This is a deliberately small box. One binary you can `scp`. One log to read. One process to restart. Whatever growth bhatti does, it won't be by adding more processes to the control plane — it'll be by getting better at the work the single process already does.
The cost is real: one bhatti daemon manages the sandboxes on one machine. There's no clustering inside bhatti and no plans for it. If you need to scale across hosts, the way I'd do it (and the way I'm planning to, eventually) is to put a thin proxy in front of several bhatti daemons that records which daemon a sandbox was created on and routes subsequent requests for that sandbox — including wake-on-request — to the same daemon. Each bhatti stays single-process; the proxy is the only piece that knows about the fleet. That's still outside bhatti itself.
### 2. We own the VMM, and drive it out of band
The daemon never links libkrun. Linking it wouldn't work anyway: libkrun's entry call, `krun_start_enter`, *becomes* the VM and blocks for its whole life — a daemon that called it would stop being a daemon. So the VMM lives in a separate process, `bhatti-vmm`, and the daemon drives it from outside.
That out-of-band channel is a tiny Unix-socket **control socket**. The daemon writes a verb, the VMM acts, and answers:
```
PAUSE -> OK paused | ERR <reason> (freeze the vCPUs)
RESUME -> OK running | ERR <reason> (thaw them)
STATUS -> OK <state> (running|pausing|paused|resuming)
SNAPSHOT <dir> -> OK (write RAM + device + vCPU state)
```
Everything the thermal manager and snapshot code needs is built from these primitives. A warm pause is `PAUSE`; a warm resume is `RESUME`. A **checkpoint** (a memory snapshot) is `PAUSE`, `SNAPSHOT <dir>`, `RESUME` — captured *without stopping* the sandbox. A **restore** is a fresh `bhatti-vmm` launched with the bundle directory, so libkrun reloads `memory.img` instead of cold-booting. A **fork** (`create --from`) is a checkpoint into a throwaway bundle immediately restored into a new identity. See [the engine page](/docs/under-the-hood/engine/) for the full sequence.
What I get from owning the VMM: pause, resume, snapshot, restore, and fork exist *at all*. They're the whole point of bhatti's thermal model, and they are permanently off upstream libkrun's roadmap — so bhatti maintains the fork ([krucible](/docs/under-the-hood/engine/)) rather than wrap a stock VMM. The cost is that I carry a hypervisor fork; the [engine page](/docs/under-the-hood/engine/) is the argument for why that's worth it.
### 3. No jailer — the VMM runs as an ordinary host process
There is no jailer in v2. `bhatti-vmm` runs as a normal host process under the daemon's user. Isolation comes from three places that don't require a chroot wrapper:
* **The hardware boundary.** Each sandbox is a real VM behind **KVM** (Linux) or **HVF** (Apple Silicon). The guest kernel is not the host kernel; a process in the guest is not a process on the host.
* **The network gateway.** `bhatti-netd` is a userspace gVisor netstack, so the guest never touches host network devices — no TAP, no bridge, no iptables to get wrong. Egress is policed and same-owner siblings are routed; the host is not on the guest's network.
* **The storage boundary.** The VMM only opens the files for its own sandbox — a qcow2 CoW root overlay, a config drive, and any attached volumes — not the wider filesystem.
Hardening the host-process boundary further — dropping to an unprivileged UID, a seccomp filter, a private mount namespace for hostile multi-tenant on Linux — is planned as a separate **Track J** capability. It is future work, not the current path: today the VMM is an ordinary process, and that is the honest description of what runs. On macOS the shipped `bhatti-vmm` is Developer-ID signed and notarized with the `com.apple.security.hypervisor` entitlement, so HVF works without root at all.
### 4. Per-VM mutex with capture-and-release
Concurrency is the part that bites every "single Go process" design eventually. Bhatti's pattern is small but load-bearing.
Each `VM` struct has two mutexes ([`pkg/engine/krucible/engine.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/engine.go)): `launchMu` serializes lifecycle *transitions* (launch / start / stop / pause / resume / destroy) so a burst of concurrent wake-on-request calls can't double-spawn the helper; `mu` guards field reads and writes. The lock order is always `launchMu` before `mu`, never the reverse — read-only `Status`/`List` take only `mu` and must not block behind an in-flight transition. The engine-level `sync.RWMutex` only protects the *map* of VMs.
The pattern every operation follows:
```go
vm.mu.Lock() // 1. take the lock
if vm.Status != "running" { // 2. check invariants
vm.mu.Unlock()
return errNotRunning
}
ag := vm.Agent // 3. capture references we need
vm.mu.Unlock() // 4. release the lock immediately
return ag.Exec(ctx, cmd) // 5. do the slow work outside the lock
```
The key move is steps 3-4: copy the references, drop the lock, then make the network call. The lock protects access to `vm.Agent` itself, not any operation that uses it.
The reason this matters: a `Shell` or `Tunnel` call lasts as long as the user's session — minutes, hours. If those calls held the lock for their duration, the thermal manager couldn't pause *any* VM (because every cycle iterates the VM map and tries to read each VM's thermal state). One slow shell would freeze idle pause / cold snapshot for everything else.
The safety argument: `vm.Agent` is only replaced during a launch (cold restore), and that path takes `launchMu` and then `mu` to do it. So if a Shell goroutine grabs `vm.Agent`, releases the lock, and then a restore runs to completion and replaces `vm.Agent` with a new client — the Shell goroutine still has a valid reference to the *old* agent's vsock connection. That connection is good until the `bhatti-vmm` process is killed, which happens only on `Stop()` or `Destroy()`. By that point the connection is closed, the Shell sees a read error, and cleans up.
So: long-running operations don't block the system, and they don't explode when state is updated underneath them — they just see the underlying connection close and exit cleanly. The pattern is in every method on the krucible engine that touches an agent.
## Where state lives
State is in two places: SQLite, on disk; and per-sandbox VMM state (disk overlays, config drive, cold-snapshot bundle) on disk under the data directory.
**SQLite.** One database file at `/var/lib/bhatti/state.db` on a default server install. Tables:
* `sandboxes` — id, name, owner, status, engine ID
* `users` — name, hashed API key (SHA-256), per-user limits
* `secrets` — name, encrypted value (age), scoped per user
* `templates`, `volumes`, `publish_rules`
* `events` — audit log
* `task_progress` — async tasks (image pull)
WAL mode is on so background workers (the thermal manager, port scanner, metrics snapshotter) don't block HTTP request handlers and vice versa.
**Per-sandbox.** Under `/var/lib/bhatti/sandboxes/<id>/`:
```
<CoW root overlay> qcow2 copy-on-write overlay over the base image
config drive ~1 MB ext4 (/dev/vdb): hostname, env, secrets,
files, per-sandbox token
state.json recovery metadata — helper PID, socket paths,
bundle ref; this is what recovery reads
(when cold) the cold-snapshot bundle: memory image + VM
state + a copy of the root overlay + any
attached volumes + manifest.json
```
The root overlay is a **qcow2 copy-on-write image** stacked over a shared read-only base. CoW lives at the image-format layer, so it's filesystem-independent — ext4, XFS, btrfs, APFS all give instant create and base-sharing with \~0.5% overhead. There's **no btrfs or reflink requirement**; that was v1. See [Storage](/docs/under-the-hood/storage/) for the overhead breakdown and the page-cache story for cold wake.
**Other disk locations under `/var/lib/bhatti/`:**
* `runtime/` — the relocatable v2 runtime bundle, re-laid on every install/update: `bin/` (`bhatti-vmm`, `bhatti-netd`), `lib/` (libkrun + symlinks), and `kernel/` (the lean external kernel)
* `images/` — read-only base rootfs templates (`rootfs-minimal-arm64.ext4` etc.), CoW-overlaid per sandbox
* `volumes/<user_id>/` — standalone persistent volumes (created by `bhatti volume create`, attachable to any sandbox as `/dev/vdc+`)
* `snapshots/<user_id>/<name>/` — named snapshots from `bhatti snapshot create` (memory + disk artifacts, fully self-contained)
* `age.key` — the X25519 private key used to encrypt user secrets. It's generated *the first time anyone calls `bhatti secret set`*, not on daemon startup ([`pkg/secrets/age.go:13-37`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/secrets/age.go#L13-L37)). **Back this up.** If you lose it, every encrypted secret on the server is unrecoverable.
There are no `jails/` and no `firecracker.sock` under a sandbox dir — those were v1. The config drive is 1 MB because ext4's minimum filesystem size is 1 MB; the contents (a few hundred bytes of JSON describing hostname, env vars, volume mounts, plus the token) would fit in less, but the filesystem can't.
## Recovery on startup
When the bhatti daemon starts — fresh boot, after a crash, after a `systemctl restart` — the helper processes from the previous run may or may not still be alive. The daemon's job at startup is to look at every sandbox and decide, for each one, what state it's really in.
Recovery is driven by the **per-sandbox `state.json`**, not the database. Each sandbox's file records everything needed to re-adopt or re-launch it: the helper PID, the socket paths, the base spec, the bundle dir, and (for the net backend) the owner's `bhatti-netd` key and subnet index. `bhatti-vmm` and `bhatti-netd` are spawned *detached* from the daemon's process group, so they survive a daemon restart or crash.
On startup the engine globs every `sandboxes/*/state.json` and, for each one:
1. **Adopts a live helper.** If the recorded PID is still alive *and* its agent answers a probe over vsock, the sandbox is adopted as-is — running, no restart, sessions intact.
2. **Marks a dead-but-restorable one cold.** If the helper is gone but a cold-snapshot bundle exists on disk, the sandbox is marked `stopped`/`cold` — the next request cold-restores it from the bundle (a sub-second operation).
3. **Marks the rest stopped.** A dead helper with no bundle (crashed before its first snapshot, RAM gone but the qcow2 root persists) needs an explicit `Start`, which cold-boots it fresh.
Then the engine re-adopts each owner's shared `bhatti-netd` gateway from its persisted record rather than respawning onto a socket it still holds, and writes the reconciled status back to `state.json`.
There is **no system-wide device cleanup** step, because there are no host TAP devices, bridges, or iptables rules to leave behind — the network lives entirely inside `bhatti-netd`. That whole class of "stale plumbing after a crash" problem doesn't exist in v2.
## What's not on this page
This was the map. The depth is in the topic pages:
* [Lohar — the agent inside every VM](/docs/under-the-hood/lohar-the-blacksmith/) — PID 1, the systemctl shim, sessions, file ops
* [Thermal states](/docs/under-the-hood/thermal-states/) — hot / warm / cold, pause/resume, snapshot mechanics
* [Networking](/docs/under-the-hood/networking/) — the `bhatti-netd` gVisor gateway, policed egress, sibling reachability, vsock forward
* [Wire protocol](/docs/under-the-hood/wire-protocol/) — frame format, all frame types, atomic writes
* [The krucible engine](/docs/under-the-hood/engine/) — the control socket, the `krun_*` boot sequence, block-root qcow2, the lean kernel, and why bhatti owns the VMM
* [Decisions & learnings](/docs/under-the-hood/decisions/) — the bugs we paid for, in narrative form
# Decisions & learnings
> Real bugs, real bad afternoons, and what they taught us. The kind of engineering log most projects don't ship — but the one I'd want to read if I were evaluating bhatti.
Every system has a paper trail of bad afternoons. This page is bhatti's.
The decisions on the other Under-the-Hood pages are summarised at the top of each — this page tells the *story* behind a few of them in more depth than fits on a how-it-works page. If you're evaluating bhatti for production, this is the page I'd read.
The order is roughly chronological with respect to when each problem hit me, not by importance.
## The 1-second SYN retransmit (and why it's gone)
On the early Firecracker-based builds, `bhatti create` had a mysterious floor: about **1 second** of every create was a single TCP SYN retransmission timeout. I spent a long time before I understood why, and the answer only makes sense once you know it was a TAP-networked, TCP-agent design.
The mechanism: after the VM started, the host immediately polled the guest's agent port with `exec true` to detect readiness. The guest kernel was still booting, so the SYN got no answer. Linux's default initial SYN RTO is 1 second ([RFC 6298](https://www.rfc-editor.org/rfc/rfc6298), `net.ipv4.tcp_syn_retries`); after that timer expired the host retransmitted, and by then the agent was up. A whole second, burned waiting for a guest kernel to boot, visible in every benchmark. Two hacks kept it from being worse — a pre-populated permanent ARP entry so the first probe didn't also pay an ARP-resolution second, and a stale-ARP flush before IP reuse.
krucible deletes the entire phenomenon rather than tuning it:
* **The transport is vsock, not TCP-over-TAP.** The host reaches the guest agent through libkrun's vsock bridge (it dials a host Unix socket bridged to the guest's vsock port). There is no SYN, no three-way handshake to a not-yet-listening port, and no ARP resolution to get wrong — so no TAP device, no bridge, no permanent-ARP trick, and no stale-entry flush.
* **The lean external kernel boots to agent-ready in \~312 ms** on Apple Silicon (HVF, measured — roughly 2× faster than libkrun's bundled kernel), so the window the old design spent retransmitting through barely exists.
The lesson that survived the rewrite: **a mysterious constant floor in a latency number is almost always a timer, not your code.** The 1-second SYN RTO was invisible in host-side instrumentation because the host work genuinely finished in \~200 ms — the extra second belonged to a kernel timer neither side controlled. Once you can *name* the timer, you can design it away.
## The rory incident
April 2026. A user's persistent sandbox named `rory` came back from a restore with a corrupted virtio ring buffer. The agent was unreachable. The VM was wedged. We had to destroy it and lost the working state of the sandbox.
Three failures compounded into one bad outcome ([`docs/archive/PLAN-reliability.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-reliability.md), [`PLAN-snapshot-reliability-fixes.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-snapshot-reliability-fixes.md)):
* **Diff snapshot corruption.** The memory image was a *diff* — only the pages the dirty-page bitmap said had changed — and that bitmap was incomplete. KVM tracks the pages the guest CPU touches, but a userspace VMM's device model writes guest memory *outside* KVM: a virtio device servicing the guest writes straight into ring buffers and DMA regions, and those writes never set a bit in KVM's bitmap. Restoring a diff snapshot then loads those pages from a stale base, and any in-flight virtio ring-buffer state gets clobbered.
* **No snapshot verification** — we'd written a corrupt mem file and marked the snapshot as good without checking.
* **Volume attachment metadata wasn't persisted on snapshot resume** — `vm.Volumes` was empty when a sandbox came back from a named snapshot, so even when we tried to recover, the volumes were unreachable.
The fixes:
* **No diff snapshots, ever.** krucible only writes *full* snapshots — there is no dirty-page / incremental path left to get wrong. A full snapshot is a few hundred ms slower to write but correct in every case. See [Why all snapshots are Full](/docs/under-the-hood/thermal-states/#why-all-snapshots-are-full).
* **Verify the artifacts after every checkpoint.** Wrong size, bad path, corrupted state — error out before marking the snapshot good, rather than discovering it on restore.
* **The device set travels with the snapshot.** A memory snapshot (or a fork) reproduces every attached volume, not just RAM and the root overlay — so a restored VM's disk view is always consistent with the RAM it came back with.
The audit also found 17 other bugs — race conditions, missing defer-close, type assertions that would panic on certain SQLite inputs. The `PLAN-reliability.md` covers all of them. v0.7 was the release that fixed the entire batch.
The lesson I keep coming back to: **the cost of correctness is much smaller than the cost of one lost user sandbox.** Diff snapshots saved \~500 ms per snapshot. Losing rory cost a user a working state, me a week of debugging, and a release-line of compounding bug fixes. Not even close.
## The systemd snapshot/restore problem
I had lohar running as a systemd-managed service for a while. Fresh boot worked. Snapshot/restore broke in a way that took a week to understand.
The setup: a regular Ubuntu rootfs with systemd as PID 1, lohar running as `lohar.service` (`Type=simple`, `Restart=always`). On a fresh boot, `systemctl is-system-running` returned `running`, lohar accepted connections, `bhatti exec` worked.
Then `bhatti stop` (snapshot to disk) and `bhatti start` (restore).
The restore looked successful: the restore returned ok, the guest kernel resumed, lohar's listener was still bound to port 1024 according to `ss -tlnp` from inside the VM. The first `bhatti exec` after restore would succeed. Every subsequent exec would hang forever.
What I observed:
* The kernel-level TCP state was fine. Three-way handshake completed, packets visible in `tcpdump`.
* The connection sat in the kernel's accept queue.
* The Go runtime — sitting on top of `epoll_wait` — was not waking up. The goroutine blocked on `Accept()` never got scheduled.
I don't have a clean explanation for *why* this happens specifically when lohar is a child of systemd, and not when it's PID 1 itself. My guess: something about how systemd manages file descriptors, control groups, or its own epoll sets across PID 1 leaves the resumed Go process holding a poller in a state Go's runtime doesn't recover from. That's speculation, not diagnosis.
What I know empirically:
* Reproduced across multiple VMM and guest-kernel versions.
* Reproduced with `Restart=no` and `Restart=always`.
* Lohar as PID 1 (no systemd) — never reproduced, in CI or production.
So lohar stayed PID 1, and we built a [systemctl shim](/docs/under-the-hood/lohar-the-blacksmith/#the-systemctl-shim) to satisfy the package machinery. That decision carried straight into v2: krucible boots lohar as PID 1 from `/init.krun` (block-root boot), so the systemd-child hazard never had a chance to reappear. **If you've traced this further or think the diagnosis is wrong**, please [open an issue](https://github.com/sahil-shubham/bhatti/issues). The shim works, but real systemd would be less code to maintain.
## Making vsock survive snapshot/restore
vsock is the natural host↔guest channel for an in-process VMM: libkrun exposes each guest vsock port as a host Unix socket, and the daemon dials it directly. It works perfectly during normal operation. The hazard is snapshot/restore — and it isn't in the transport, it's in the guest's Go runtime.
What happens with a naive listener: after a cold restore, the guest's Go runtime can come back with its netpoller (`epoll`) registrations in a state it never recovers from. A goroutine blocked in `Accept()` on a runtime-registered vsock socket simply never gets scheduled again — the host connects, the kernel completes the accept, and the guest never wakes to it. (This is the same failure shape as the systemd story above: a resumed Go poller that wedges.)
The fix lives in lohar, not the transport. lohar's vsock listener is a plain blocking socket with a one-second `SO_RCVTIMEO`, so `accept4` returns `EAGAIN` on a *kernel* timer rather than parking on the Go netpoller. When an accept that should have returned in milliseconds instead takes seconds, lohar reads that as "we were just restored," tears the listener down, and recreates it fresh ([`cmd/lohar/net.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/net.go)). A kernel-level timer keeps firing even when the Go runtime is stalled, so the guest always gets a chance to notice the resume and rebuild a working listener.
The lesson: **the thing that breaks across snapshot/restore is rarely the device — it's the userspace runtime state layered on top of it.** An earlier design reached for a different transport to dodge the problem; the durable fix was to stop trusting the runtime's poller across a resume and lean on a timer the kernel owns. As a public corroboration point that suspend/restore is genuinely hard here, SlicerVM dropped their suspend/restore feature in v0.1.108 ([archive notes](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/slicer-learnings.md)).
## The Cloudflare Tunnel disconnect
A user reported their `bhatti shell` dropping silently when they ran a long-running command through `api.bhatti.sh` ([`docs/archive/PLAN-shell-sessions.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-shell-sessions.md)). Screenshot: no error, no "detached" message, no bash prompt — just a silent return to the shell on their Mac.
Root cause:
* They ran `bhatti shell rory`, started `hermes gateway` (a daemon that prints a startup banner then waits for events).
* Cloudflare Tunnel has a WebSocket idle timeout around 100 seconds. No traffic in either direction = connection killed.
* The CLI's `conn.ReadMessage()` returned an error. The error path was silent — `defer term.Restore()` ran, terminal mode reset, but nothing got printed.
* `bhatti shell rory` again created a *new* session. The old session was still alive inside the VM with `hermes gateway` running and scrollback accumulating, but there was no way to get back to it.
Three problems compounded:
1. The WebSocket layer didn't have ping/pong keepalives, so idle connections were getting killed by intermediaries.
2. The CLI didn't print anything on disconnect, so users didn't know what happened.
3. There was no session reattach. `bhatti shell` always created a new session.
The fix was a long PR ([`docs/archive/PLAN-shell-sessions.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-shell-sessions.md)) with a 12-bug inventory. The notable ones:
* **WebSocket ping/pong keepalives** every 30 seconds, both client and server.
* **Concurrent WebSocket write race** — the CLI's main loop and its resize-handling goroutine both wrote to the connection without coordination. Replaced with a single writer goroutine fed by a channel.
* **Session reattach** — `bhatti shell <name>` now attaches to the most recent live session (if any) rather than always creating new.
* **Disconnect message** — CLI prints `[disconnected: <reason>; reconnect with bhatti shell <name>]` on the way out.
* **Scrollback ring buffer thread safety** — was being accessed concurrently without a mutex; reads could get torn.
The takeaway: **production network conditions kill connections that look idle to intermediaries.** If you're building over WebSocket, ping/pong is not optional. And the user-visible behavior on disconnect is part of the API — silently exiting is an outage from the user's point of view.
## The public proxy rate-limiter recalibration
The first version of the public proxy used a per-alias token bucket with burst 100 and refill 200/min as the primary rate limiter. Logic: "each published URL gets its own quota."
Then a real Vite app with hot module reload generated 90+ requests on a single page load, exhausted the bucket on the first reload, and served 18 of 91 requests as 429s — a blank page on reload.
Documented in [`docs/archive/PLAN-public-proxy-ratelimit.md`](https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-public-proxy-ratelimit.md). Two things were wrong:
1. **The aggregation level was wrong.** Standard reverse-proxy practice (nginx `limit_req_zone`, Cloudflare, AWS WAF) is per-source-IP rate limiting. Per-destination is a *secondary* aggregate against distributed attacks, not the primary mechanism. Browsers HMRing are one source IP, not one URL.
2. **The numbers were too low.** 200 requests/min ≈ 3.3 req/s. A modern bundler-driven page exceeds that during a single load.
The fix: per-source-IP primary limit (much higher burst, much higher refill), per-alias as a secondary cap (very high — only catches runaway loops), global as a third (host-level safety net).
This is the kind of decision that's hard to get right in advance, because the number depends on the workload. We had reasonable defaults for an API server. The defaults were wrong for a frontend. The calibration came from one user with one Vite app.
## Pure-Go SQLite
The choice was `mattn/go-sqlite3` (CGo binding to the C SQLite library) vs `modernc.org/sqlite` (pure-Go translation of SQLite's C code).
The CGo version is faster and more compact. The pure-Go version is \~10% slower and the binary is \~3 MB larger. For metadata CRUD (sandboxes, users, secrets, events), the speed difference is irrelevant.
What matters: cross-compilation. Bhatti is built on a Mac and deployed to a Pi. With CGo, this requires a cross-compiler toolchain (`aarch64-linux-gnu-gcc`), careful library management, and different build commands per platform. It also means the binary is dynamically linked against `libc` and isn't fully portable across distros.
With pure-Go SQLite, `GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build` produces a static binary that runs on any Linux. Same binary works on Pi, Hetzner, Graviton, an Alpine container, an Ubuntu host, anything.
The 3 MB and 10% are bought, gladly, for the build simplicity.
## Per-owner network isolation (no shared bridge)
The first version of bhatti had one Linux bridge — `brbhatti0` — and all VMs across all users shared it on a single subnet. Simple to operate. *Insecure* once you actually have multiple tenants: at L2, alice's VM could ARP-scan the bridge and reach bob's VM directly, and iptables FORWARD rules were the only barrier. A misconfigured rule meant cross-tenant traffic.
v1 fixed the L2 leak with a bridge per user. v2 removes the L2 fabric entirely. Under krucible there are **no host TAP devices, no bridges, and no iptables rules**. Each guest's `eth0` is a virtio-net link over a Unix socket to a **per-owner `bhatti-netd` gateway** — a userspace [gVisor](https://gvisor.dev/) netstack that is the guest's router, DNS, and egress policer. One netd per owner: alice's VMs connect to alice's gateway, bob's to bob's, and there is no shared switch for either to scan. Same-owner siblings reach each other *through* their gateway (an L3-routed proxy), not over a shared L2 segment. See [Networking](/docs/under-the-hood/networking/).
The lesson, unchanged from v1: **start with the security model you'll need**, not the simplest one that works for now. v1 learned it the hard way — a single shared bridge, then a scramble to per-user bridges; v2 bakes per-owner isolation into the transport itself, so there is no shared fabric left to misconfigure.
## Server-side file truncation
Coding agents read files. They almost always truncate — typically to the first 2000 lines or 50 KB. We discovered this by watching what clients actually did with our `FILE_READ_RESP` data: they'd ask for a 100 MB log file and immediately throw away 99.95% of it on the client side. We were paying full bandwidth for almost no information.
The fix is small. `FILE_READ_REQ` accepts `offset`, `limit`, `max_bytes`. Lohar reads line-by-line with `bufio.Scanner` and stops at whichever limit hits first. The response includes the *full* file size so the consumer knows whether content was truncated. See [wire protocol — file read](/docs/under-the-hood/wire-protocol/#file-read-protocol).
Performance: a truncated read on a 10 K-line file is about 4.5× faster at p50 than a full read of the same file. For larger files, proportionally bigger.
Why not `head -n` via exec? Fork-exec, pipe setup, shell argument parsing — all of that is overhead the file protocol path skips. A 1 KB file read takes about 472 µs in the benchmark; the equivalent exec is more like 1 ms. For a coding agent that does hundreds of file reads per task, this adds up.
The general lesson: **the client knows what it wants; tell the server.** Even when bandwidth is "free" inside a single host, the allocations, copies, and discards still cost.
## Sessions everywhere
Most sandbox systems split exec and shell into different concepts. Bhatti unifies them: every TTY exec is a session. There's no separate "shell" code path. A shell is just a TTY exec of `/bin/zsh` or `/bin/bash`.
Three reasons this fell out of real product needs:
1. **Init scripts need to be attachable.** When you create a sandbox with `--init "npm install && npm run build"`, that command runs as a session named `init`. You can attach to it from the host to watch progress. If exec weren't a session, the init output would be invisible from the host.
2. **Shells need to survive proxy disconnects** (the Cloudflare Tunnel story above).
3. **Snapshot/restore needs to preserve interactive shells.** A shell running a long-running command when the VM is snapshotted should be the same shell after restore. Sessions are the data structure that survives that round-trip.
Trade-off: every TTY session allocates a 64 KB scrollback buffer. With 100 concurrent sessions per VM, that's 6.4 MB. We cap at 20 sessions per VM ([`cmd/lohar/handler.go:50`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/handler.go#L50)) which keeps it bounded.
## When kasmvnc.service couldn't restart
The computer tier shipped in v1.11.9 — XFCE + KasmVNC + Chromium, all running as systemd units managed by the shim. First boot worked end-to-end (desktop renders, password generation runs once, screenshot works). Restarting `kasmvnc.service` did not. The unit transitioned to `failed` and the journal showed:
```
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
Fatal server error:
Cannot establish any listening sockets - Make sure an X server isn't
already running
```
`Xkasmvnc` from the previous run was still alive, holding the abstract Unix socket `@/tmp/.X11-unix/X99`. No file-system cleanup frees an abstract socket — only killing the holder does.
`systemctl stop kasmvnc` *should* have killed it. The shim does `cgroup.kill` on the unit's cgroup, which the kernel translates into SIGKILL for every PID in `cgroup.procs`. Why didn't `Xkasmvnc` go down?
```
$ sudo cat /proc/$(pgrep -x Xkasmvnc)/cgroup
0::/
$ sudo cat /sys/fs/cgroup/system.slice/kasmvnc.service/cgroup.procs
(empty)
```
`Xkasmvnc` was in the **root** cgroup, not the unit's. `cgroup.kill` on the unit's cgroup had nothing to kill.
The shim's old `startDaemon` called `cmd.Start()` and *then* wrote the daemon's PID into the unit's `cgroup.procs`. The window between those two operations is microseconds, but `Xkasmvnc` uses `daemon(3)` — fork, parent exits, child detaches — during startup. The child fork happened during the window, before the placement write, so the child inherited whatever cgroup the parent was in (lohar's, i.e. root). Once the parent exited, the placement write moved only the dead parent's PID. The detached `Xkasmvnc` was forever in the wrong cgroup.
The same shape applies to any double-forking daemon: classic Apache, `dbus-daemon` in default mode, `nginx` without `daemon off`. v1.11.9 shipped the contract bug — every future tier author would have to audit whether their daemon double-forks during startup, and we'd have no way to enforce it without reading every package's source.
The fix is to remove the race entirely by placing the cgroup before any fork can happen. Real systemd does this with `clone3(CLONE_INTO_CGROUP)` — a single syscall that forks and places atomically. Go's runtime doesn't expose `clone3`; reproducing it means reimplementing `syscall.ForkExec`'s post-fork setup (close-on-exec, signal-handler reset, credentials, working dir) under `syscall.ForkLock`. That's real surgery on code where bugs are silent and only surface under load.
The alternative is a thin one-purpose helper. The supervisor invokes `lohar spawn` instead of the daemon directly; `spawn` writes its own PID into `<cgroup>/cgroup.procs` and then `execve`s into the daemon. `execve` preserves PID **and** cgroup membership, so the daemon — same PID throughout — is already in the unit's cgroup when it forks anything later. About 60 lines of plain Go ([`cmd/lohar/spawn.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/spawn.go)) gets the same correctness guarantee as `clone3` with code anyone can read in one sitting. The dispatch is on `argv[1]` (`/proc/self/exe spawn ...`) rather than the busybox-style symlink we use for `systemctl` and `journalctl`, because `lohar spawn` is a private supervisor primitive — not something a user should be wrapping their own commands with. `clone3` stays open as a future migration if the extra `execve` ever shows up in profiling.
Shipped in v1.11.10. Every existing unit file works unchanged. The operator-visible difference: `systemctl stop`/`restart` now works for every kind of daemon, including the ones that fork during startup. See [How services are spawned](/docs/under-the-hood/lohar-the-blacksmith/#how-services-are-spawned) for the mechanism.
## Owning the VMM (krucible)
bhatti's whole product is thermal states: sandboxes that pause, snapshot to disk, restore in a fraction of a second, and fork into instant copies. Those are VMM-level capabilities — pause/resume, checkpoint/restore, and memory-clone fork — and they are not on the upstream roadmap of the in-process VMM library we build on ([libkrun](https://github.com/containers/libkrun)). You can't bolt them on from the outside; they touch how the VMM saves and restores vCPU state, memory, and device state.
So we forked libkrun into **krucible** and added exactly those pieces: a control socket carrying `PAUSE`/`RESUME`/`STATUS`/`CHECKPOINT`/`RESTORE`/`FORK`, full-VM snapshot and restore, and a block-root boot path that lets lohar run as PID 1 from `/init.krun`. The bhatti daemon stays pure Go and never links libkrun — it spawns one `bhatti-vmm` helper per sandbox (the helper *becomes* the VM, since libkrun's entry call blocks for the VM's life) and drives it out-of-band over that control socket.
The lesson: **if a capability is the product, own the layer it lives in.** Waiting for upstream to grow pause/resume/fork would have meant not shipping the product. A fork is a maintenance cost — we rebase onto upstream libkrun on our own cadence — but it's the only way to control the thing the product is made of.
## Why krucible over Firecracker
v1 ran on Firecracker. krucible replaces it for four reasons, each of which was awkward or impossible on the old engine:
* **One engine on the laptop and the server.** krucible runs on **KVM** on Linux and **HVF (Hypervisor.framework)** on Apple Silicon, so the same sandbox model boots on a developer's Mac and on a Linux host. Firecracker is Linux/KVM only.
* **Live snapshot, pause/resume, and fork as first-class verbs**, driven over the control socket — the thermal-state machinery the product is built on, owned end to end rather than pieced together around an external process.
* **Host-owned networking.** The guest link is a virtio-net device to a per-owner `bhatti-netd` gateway; there are no TAP devices, bridges, or iptables rules on the host, and no per-user subnet to allocate. The network policy lives in a userspace gVisor netstack we control.
* **A zero-secret-on-disk posture.** Per-sandbox env, secrets, and the agent token ride a small config drive that lohar mounts read-only, applies, then unmounts and removes — the secrets don't linger on a guest filesystem after boot.
Firecracker is a genuinely excellent VMM; the switch isn't a knock on it. It's that bhatti's requirements — cross-platform, snapshot/fork-native, host-owned networking — line up with a VMM we can extend, not one we drive from the outside.
## qcow2 copy-on-write (dropping the btrfs dependency)
v1's fast `create` leaned on the *host* filesystem: a btrfs (or xfs-reflink) volume so a new sandbox's rootfs was a metadata-only reflink clone of a base image. It was fast, but it made the filesystem a deployment prerequisite — and on plain ext4 the same clone fell back to a full copy and `create` got noticeably slower.
v2 moves copy-on-write up into the **disk image format**: each sandbox gets a thin **qcow2 overlay** over a shared read-only base, and the CoW is the image format's job, not the filesystem's. Instant create and base-sharing now work identically on **ext4, xfs, btrfs, and APFS**, with \~0.5% overhead on agent workloads. There is nothing to pre-provision.
The lesson: **push a requirement down a layer until it stops being the operator's problem.** "You must run bhatti on btrfs" is a real barrier; "bhatti stores qcow2 files on whatever filesystem you already have" isn't a barrier at all.
## No jailer yet (Track J)
On Firecracker, the VMM process ran under a *jailer* — a chroot, a dropped-privilege uid, cgroups, and a seccomp filter around the VM process — so a hypothetical VM escape landed in an empty jail rather than on the host. krucible does not have an equivalent yet: the daemon spawns `bhatti-vmm` directly (on Linux it runs as root because it needs `/dev/kvm`; on macOS HVF needs no root, but the helper is unsandboxed).
I'd rather be honest about this than paper over it. Jailing the helper for genuinely *hostile* multi-tenant use on Linux is planned work — internally "Track J" — not something that ships today. For single-user and dev use (the common case, and the whole story on a Mac laptop) it isn't load bearing; for hostile multi-tenant on shared Linux hardware, it's a gap you should know about.
## What I'd do differently
Things I'd revisit if I were starting bhatti from scratch today:
* **Use a real init for the systemctl shim from the start.** I'm reasonably happy with the shim (it's small, it covers the surface Debian's tools actually call), but I went through several rounds of "what about this directive too?" before it stabilized. Looking at it again, I'd start from a small supervisor like [s6-rc](https://www.skarnet.org/software/s6-rc/) and add a thin systemctl-compatible wrapper. The shim would still exist, but the service-supervisor code would be battle-tested rather than home- rolled.
* **Snapshot verification from day one.** I shipped snapshots without verifying the artifacts, and the rory incident is what it cost. The verification logic is \~50 lines. Not having it from the start was a real mistake — and it's why v2 only ever writes full snapshots and checks them before marking them good.
* **Per-source-IP rate limiting on the public proxy from day one.** Per-destination was the wrong default; "match what nginx does" is the boring-and-correct answer.
* **TLS for the agent protocol.** The agent protocol authenticates with a 16-byte hex token, framed over vsock — the host dials the guest's socket through libkrun's bridge, so there's no network path another guest could reach it on, and the host always initiates (no certificate-pinning equivalent on the guest side). Still, TLS would give us defense in depth. Haven't done it yet because the threat model doesn't demand it on a single-host deployment, but it should be on the table for any real multi-host setup.
## Where to go next
If you want to read more like this, the full archive is in [`docs/archive/`](https://github.com/sahil-shubham/bhatti/tree/main/docs/archive) in the repo. Plans, post-mortems, write-ups on the Linux / KVM / cloud-init behaviors that drove our design choices (with cross-references to publicly documented incidents at other orchestrators in the space — SlicerVM, fly.io's Sprites — where their reports matched ours), and the migrations that got us to v1.7.
* [Architecture](/docs/under-the-hood/architecture/) — how all the pieces fit together
* [Lohar](/docs/under-the-hood/lohar-the-blacksmith/) — the agent inside every VM
* [Thermal states](/docs/under-the-hood/thermal-states/) — pause / snapshot / restore in detail
* [Networking](/docs/under-the-hood/networking/) — the per-owner bhatti-netd gateway, egress policy, and host↔guest forwarding
* [Wire protocol](/docs/under-the-hood/wire-protocol/) — frames, ports, atomic writes
# The krucible engine: control socket, block-root boot, snapshots
> How bhatti drives its VMM — the out-of-band control socket, the krun_* boot sequence, the qcow2 block-root substrate, the lean kernel, and why bhatti owns the fork.
This page is the lower-level companion to [Architecture](/docs/under-the-hood/architecture/). It walks through what the krucible engine actually does to boot a VM, how the daemon drives the VMM from outside, what the storage substrate looks like, and why bhatti maintains its own hypervisor fork.
If you're using bhatti, you don't need this page. If you're hacking on the engine, extending it, or debugging a weird VMM-level failure, this is the reference.
**krucible** is bhatti's fork of [libkrun](https://github.com/containers/libkrun) — an in-process VMM library. bhatti extended it with pause/resume, snapshot/restore, a control socket, block-root boot, and a lean external kernel. It runs on **KVM** (Linux, `/dev/kvm`) and **HVF** (macOS, Apple Silicon).
## Design decisions on this page
1. **The daemon never links the VMM.** libkrun's entry call *becomes* the VM and blocks forever, so bhatti puts it in a separate process (`bhatti-vmm`) and drives it out-of-band over a **control socket**. See [The control socket](#the-control-socket).
2. **We own the fork.** Pause, resume, snapshot, restore, and fork are permanently off upstream libkrun's roadmap — and they're the whole point of bhatti's thermal model — so bhatti maintains krucible rather than wrap a stock VMM. See [Why bhatti owns the VMM](#why-bhatti-owns-the-vmm).
3. **Block-root, qcow2 copy-on-write.** The guest boots from a qcow2 CoW overlay over a shared base image, so create is instant and filesystem-independent. See [The block-root substrate](#the-block-root-substrate).
4. **All snapshots are Full.** There are no diff or dirty-page snapshots in krucible — a checkpoint always writes the whole memory image. The full reasoning is in [Thermal states](/docs/under-the-hood/thermal-states/#why-all-snapshots-are-full).
## The control socket
The daemon and the VMM are two processes. `bhatti-vmm` links libkrun and calls `krun_start_enter`, which *becomes* the guest and only returns on a boot error — so the daemon can't call into it. Instead the daemon drives the running VM from outside over a tiny Unix-socket **control socket** (`krun_set_control_socket` in the fork).
The protocol is line-oriented text. From [`pkg/engine/krucible/control.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/control.go):
```
PAUSE -> OK paused | ERR <reason>
RESUME -> OK running | ERR <reason>
STATUS -> OK <state> (running|pausing|paused|resuming)
SNAPSHOT <dir> -> OK (write RAM + device + vCPU state to <dir>)
```
Four verbs. Everything the thermal manager and the snapshot code needs is composed from them:
* **Warm pause / resume** ([Thermal states](/docs/under-the-hood/thermal-states/#cold--hot)) is a bare `PAUSE` / `RESUME` — the vCPUs freeze, memory stays resident, resume is \~4 ms.
* **Checkpoint** (a memory snapshot) is `PAUSE`, `SNAPSHOT <dir>`, `RESUME` — captured *without stopping* the sandbox. libkrun writes `manifest.json`, `checkpoint.bin`, and `memory.img` into `<dir>`.
* **Restore** is not a control-socket verb at all: it's a fresh `bhatti-vmm` launched with `krun_set_snapshot(<dir>)`, so libkrun reloads `memory.img` instead of cold-booting.
* **Fork** (`create --from`) is a checkpoint into a throwaway bundle, immediately restored into a new sandbox identity, then the bundle is discarded. Instant, and the source is undisturbed.
The lock discipline around these transitions — `launchMu` before `mu`, capture-and-release for long-running agent calls — is on the [Architecture page](/docs/under-the-hood/architecture/#4-per-vm-mutex-with-capture-and-release).
## Boot sequence: what the VMM does
A `bhatti create` spawns `bhatti-vmm` with a `VMSpec`, and the helper walks a fixed sequence of `krun_*` calls before it becomes the VM. The order matters — disks must be added so they enumerate in the right order, devices must precede the port bridges. From [`cmd/vmm/main.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/vmm/main.go):
```
1. krun_create_ctx() new VM context
2. krun_set_vm_config(vcpus, mem_mib) CPU + RAM
3. krun_set_kernel(lean kernel, cmdline) load our own kernel;
cmdline carries root=/dev/vda init=/usr/local/bin/lohar
(this makes krun_start_enter skip libkrunfw entirely)
4. krun_set_root_disk2(root, QCOW2|RAW) root overlay -> /dev/vda
5. krun_add_disk("config", drive, rw) config drive -> /dev/vdb
6. krun_add_virtiofs3(...) (optional) live --mount host-dir binds
7. krun_add_disk2(vol, fmt, ro) × N volumes -> /dev/vdc+
8. krun_add_virtio_console_default(...) console -> the daemon's log
9. krun_add_vsock(features) plain vsock (net backend on)
or TSI-inet hijack (net off)
10. krun_add_net_unixstream(uds, mac, ...) virtio-net -> bhatti-netd
(only with the net backend)
11. krun_add_vsock_port2(1024, ctrl-uds) agent control plane bridge
krun_add_vsock_port2(1025, fwd-uds) agent forward plane bridge
12. krun_set_control_socket(ctrl-uds) the PAUSE/RESUME/etc. socket
13. krun_set_snapshot(bundle) (optional) cold-restore instead of boot
14. krun_start_enter() THIS PROCESS BECOMES THE VM
```
A few of these are easy to miss but load-bearing:
* **Step 3 is the whole "lean kernel" story** (below). Setting an external kernel makes libkrun skip its bundled `libkrunfw`; the cmdline supplies `init=`, so `lohar` boots as PID 1 directly. On the bundled-kernel path instead, `krun_disable_implicit_init` is called so libkrun doesn't inject its own `/init.krun` ahead of lohar.
* **Step 9's vsock is dual-purpose.** With the network gateway on (the default), the guest's real networking goes over `eth0` via `bhatti-netd`, so the vsock device is added *plain* and used only for the agent port bridges (step 11). With the legacy TSI backend, the same vsock carries `KRUN_TSI_HIJACK_INET` so the guest borrows the host netstack — see [Networking](/docs/under-the-hood/networking/).
* **Step 11 is how the daemon reaches lohar.** The host dials the bridge UDS and libkrun forwards it to the guest port where lohar listens: `1024` for the control plane (exec, files, sessions), `1025` for the forward plane (port forwarding). This is plain vsock, not TCP over a host network device.
After `krun_start_enter`, the daemon polls the agent control bridge until lohar answers, then the create returns to the user. There is no ARP priming and no host-side IP plumbing — the guest reaches the network through `bhatti-netd`, and the daemon reaches the guest through vsock.
## The block-root substrate
krucible boots from a **block device**, not a shared host directory. Each sandbox's root is a **qcow2 copy-on-write overlay** stacked over a single read-only base image (`krun_set_root_disk2` with the qcow2 format; the overlay is provisioned by `krun_create_disk_overlay`, imago-backed, in the fork). The guest sees an ordinary ext4 at `/dev/vda`; libkrun translates qcow2 host-side.
Because CoW lives at the image-format layer, it's **filesystem-independent** — ext4, XFS, btrfs, APFS all give instant create and base-sharing, with roughly 0.5% overhead on agent workloads. There is **no btrfs or reflink requirement**; that was a v1 constraint. Setting `KRUCIBLE_ROOT_RAW=1` opts a sandbox out of qcow2 onto a raw image (losing the instant-clone property on non-CoW filesystems).
The rest of the device set stacks on top in a fixed order:
* `/dev/vda` — the qcow2 root overlay
* `/dev/vdb` — the \~1 MB config drive (a raw ext4 lohar mounts read-only to read hostname, env, secrets, attached-volume specs, and the per-sandbox token)
* `/dev/vdc+` — attached block volumes, in attach order
A memory checkpoint or fork restores the **whole device set**, not just RAM: `Checkpoint` freezes each attached volume as an independent copy consistent with the paused VM and records it in `manifest.json`; restore clones each back and re-attaches it, so the restored RAM's view of its disks stays valid. See [Storage](/docs/under-the-hood/storage/) for the format details.
## The lean external kernel
krucible boots its **own kernel** instead of the one bundled inside `libkrunfw`. `krun_set_kernel` (step 3 above) hands libkrun an external `Image` (arm64) or `vmlinux` (x86\_64) plus a full boot cmdline, and libkrun skips `libkrunfw` entirely.
The payoff is speed. It's the same 6.12.x kernel `libkrunfw` ships, but built with a lean config (roughly 1,000 options versus \~1,400) — just what a microVM guest needs. The result is about **2× faster cold-start**: boot→agent measured at \~312 ms versus \~610 ms on HVF, validated cross-arch on HVF and on KVM (arm64 + x86\_64). The kernel is *ours* — pinned, reproducible, built from the config under `scripts/lean-kernel/` via [`scripts/build-lean-kernel.sh`](https://github.com/sahil-shubham/bhatti/blob/main/scripts/build-lean-kernel.sh) (see [Contributing → the kernel](/docs/contributing/kernel/) for the build).
The lean kernel is opt-in-by-presence: point `krucible_kernel_image` at it (the install lays it under `runtime/kernel/`), or leave it unset and the daemon falls back to the bundled `libkrunfw` kernel. The external kernel is what enables the block-root path — it's the one that reads `root=/dev/vda init=/usr/local/bin/lohar` from the cmdline.
## Why bhatti owns the VMM
bhatti forks libkrun rather than depending on it, and that's a deliberate, load-bearing choice.
The entire thermal model — warm pause/resume, cold snapshot/restore, and fork — depends on the VMM being able to freeze a running guest, serialize its RAM + device + vCPU state to disk, and reconstruct it later (in the same process, or a fresh one, or a *new identity*). Stock libkrun does none of that, and those capabilities are **permanently off its upstream roadmap** — it's designed to launch a workload and run it, not to checkpoint and clone live VMs. There is no config flag that turns them on.
So the capabilities live in the fork:
* **`krun_set_control_socket`** — the PAUSE/RESUME/STATUS/SNAPSHOT channel the daemon drives from outside.
* **Snapshot / restore** — `SNAPSHOT <dir>` and `krun_set_snapshot(<dir>)`, including the GIC save/restore on arm64 KVM that makes cold-restore work there.
* **Warm-clock freeze** — a paused guest's `CLOCK_MONOTONIC` must not jump forward by the pause duration on resume; the fork rewinds the guest-visible virtual counter.
* **Block-root + qcow2** — `krun_set_root_disk2` and `krun_create_disk_overlay`, the CoW substrate above.
* **The external-kernel path** — `krun_set_kernel` skipping `libkrunfw`.
The cost is real: bhatti carries a hypervisor fork ([`libkrucible`](https://github.com/sahil-shubham/bhatti), vendored as a git submodule; the gitlink SHA is the version-of-record), rebased onto upstream libkrun on its own cadence. The upside is that pause, resume, snapshot, restore, and fork *exist at all* — and they're not bolted on, they're first-class in the VMM. For an agent-sandbox platform whose whole economics rest on sub-second cold wake, that trade is the point.
## Listening port discovery
When a request hits a sandbox proxy URL, the daemon needs to know which ports the guest has open. Rather than maintain a host-side registry that gets stale, bhatti just asks the guest over the agent:
```go
// pkg/engine/krucible/agent.go
result, _ := ag.Exec(ctx, []string{"ss", "-tln", "--no-header"}, nil, "")
return parseSSOutput(result.Stdout), nil
```
This is on-demand: we don't poll. The proxy queries listening ports when a request arrives, and the cost is one `exec` (a few milliseconds) per query. Guest-side `ss` is fast and accurate — it queries the kernel directly rather than walking `/proc/net/tcp`.
The trade-off: this only works on hot VMs. The `ListeningPorts` call errors out for warm or cold sandboxes. The public proxy handles this by `ensureHot`-ing the sandbox first — if you publish a port and a request hits the cold URL, the proxy wakes the VM (a sub-second cold-restore), queries ports, checks that the requested port is in the list, then forwards.
## Recovery: what gets persisted
The VMM doesn't survive a daemon restart on its own — a crashed or restarted daemon inherits helper processes that may or may not still be alive. krucible persists everything needed to re-adopt or re-launch each sandbox to a **per-sandbox `state.json`**, and the helpers are spawned *detached* from the daemon's process group so they outlive it.
The record captured per VM ([`pkg/engine/krucible/recovery.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/recovery.go)):
* `id`, `name`, `user_id`, `status`, `thermal`
* `sandbox_dir`, `sock_dir`, `bundle_dir` (the cold-snapshot bundle), `log_path`
* `base_spec` — the full spec to re-launch or cold-restore with
* `helper_pid` — the `bhatti-vmm` PID, for adopt-by-PID
* `netd_key`, `subnet_idx` — the owner's shared `bhatti-netd` gateway, so recovery re-adopts the running gateway instead of respawning onto a socket it still holds
On daemon startup the engine globs every `sandboxes/*/state.json` and, for each, decides:
* **Adopt** if the recorded PID is still alive *and* its agent answers a vsock probe — the sandbox is running, no restart.
* **Cold** if the helper is dead but a cold-snapshot bundle exists — marked `stopped`, ready to cold-restore from the bundle on the next request.
* **Stopped** otherwise — a dead helper with no bundle (RAM gone, but the qcow2 root persists) needs an explicit `Start` to cold-boot fresh.
There's no system-wide device cleanup afterward, because there are no host TAP devices, bridges, or iptables rules to leave behind — the network lives entirely inside `bhatti-netd`.
## Cold-snapshot bundle layout
A cold sandbox — or a named snapshot from `bhatti snapshot create` — has a self-contained bundle on disk. A memory snapshot writes:
```
<bundle-dir>/
├── manifest.json arch/feature gate, config-drive + token, volumes, mounts
├── checkpoint.bin VM + device + vCPU state (small)
├── memory.img full guest RAM image (size of the VM's allocated RAM)
├── <root overlay> the qcow2 root at snapshot time
└── vol0.img, vol1.img frozen copies of attached volumes, if any
```
`memory.img` is the size of the VM's allocated RAM. A 4 GB VM that's been cold for an hour has a 4 GB `memory.img` on disk. `checkpoint.bin` is small — CPU registers, vCPU state, and device-model state.
`manifest.json` is what makes a snapshot portable and safe. It records the host architecture and feature set, so restore **refuses a cross-arch bundle cleanly** rather than crashing; and it lists the config drive, the per-sandbox token to re-enforce from RAM, and every attached volume, so a restore into a *fresh sandbox* — even after the original is destroyed — reconstructs the full device set. A `filesystem`-type snapshot is disk-only (no `memory.img`); it restores with a cold boot rather than a memory reload.
## Where to go next
* [Architecture](/docs/under-the-hood/architecture/) — the bird's-eye view that this page sits underneath
* [Thermal states](/docs/under-the-hood/thermal-states/) — what the engine does for `Stop`/`Start`/`Pause`/`Resume`, and why all snapshots are Full
* [Networking](/docs/under-the-hood/networking/) — the `bhatti-netd` gVisor gateway, the vsock forward primitive, and TSI
* [Storage](/docs/under-the-hood/storage/) — the qcow2 CoW substrate, volumes, and the cost-by-filesystem story
* [Decisions & learnings](/docs/under-the-hood/decisions/) — the own-the-fork choice and other engine-level lessons
# Lohar: the agent inside every VM
> PID 1, the systemctl shim, sessions, PTY allocation, file operations — what runs inside each sandbox and the design decisions behind it.
PID 1 is the first userspace process the Linux kernel starts. It's responsible for mounting filesystems, reaping zombie processes, and ends up the parent of every other process in the system. If PID 1 exits, the kernel panics. On a normal Linux box, PID 1 is systemd.
In each bhatti microVM, PID 1 is a small Go program called `lohar` (लोहार — Hindi for *blacksmith*; it works inside the *bhatti*, the furnace). One binary, statically linked. It does the PID 1 init duties, listens for the agent protocol over vsock, manages exec sessions and their scrollback buffers, handles file operations, and contains a small shim that pretends to be `systemctl` and `journalctl` for in-guest callers.
## Design decisions on this page
Four decisions worth understanding before reading the rest:
1. **lohar runs as PID 1, not as a systemd-managed service.** Snapshot/restore broke when lohar ran as a child of systemd, in ways I haven't fully diagnosed. See [Why not real systemd](#why-not-real-systemd).
2. **systemctl is a shim, and real systemd is pinned out of apt.** Package post-install scripts assume systemctl exists, but installing real systemd alongside the shim breaks the rootfs. The shim covers the surface Debian's tools actually call; the apt pin keeps real systemd from sneaking in. See [The systemctl shim](#the-systemctl-shim).
3. **Every TTY exec is a session.** Sessions survive client disconnects and snapshot/restore. Without them, a closed laptop, a flaky network, or a Cloudflare Tunnel idle timeout kills your shell. See [Sessions](#sessions).
4. **File writes are atomic, with `fsync` before `rename`.** The host might snapshot the VM the next millisecond. Without `fsync`, the snapshot can contain a renamed file with zero bytes. See [File operations](#file-operations).
The list isn't exhaustive — smaller decisions (the busybox-style multi-call binary, server-side file truncation, the env-var precedence chain) are explained in their sections.
## One binary, three names
The first lines of [`cmd/lohar/main.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go#L30-L36):
```go
switch filepath.Base(os.Args[0]) {
case "systemctl":
runSystemctl(os.Args[1:])
return
case "journalctl":
runJournalctl(os.Args[1:])
return
}
// otherwise: PID 1 init duties
```
This is the [busybox pattern](https://en.wikipedia.org/wiki/BusyBox#Single_binary): one binary, multiple identities, dispatched on `argv[0]`. In the rootfs, `/usr/bin/systemctl` and `/usr/bin/journalctl` are symlinks pointing at `/usr/local/bin/lohar` ([`scripts/tiers/minimal.sh:62-63`](https://github.com/sahil-shubham/bhatti/blob/main/scripts/tiers/minimal.sh#L62-L63)). The kernel and shell don't care that the destination is the same binary; they invoke whatever name they want, and lohar picks the matching code path based on what name it was called by.
## Why not real systemd
I had lohar running as a systemd-managed service for a while (`Type=simple`, `Restart=always`, with the rootfs unchanged from a regular Ubuntu install). On a fresh boot it worked: `systemctl is-system-running` returned `running`, lohar accepted connections, `bhatti exec` worked.
Then `bhatti stop` (snapshot to disk) and `bhatti start` (restore from disk).
The restore itself looked successful: the VMM's restore returned ok, the guest kernel resumed, lohar's listener was still bound to port 1024 according to `ss -tlnp` from inside the VM. The first `bhatti exec` after restore would succeed. Every subsequent exec would hang forever.
What I observed:
* The kernel-level TCP state was fine. Three-way handshake completed, packets visible in `tcpdump`.
* The connection sat in the kernel's accept queue.
* The Go runtime — sitting on top of `epoll_wait` — was not waking up. The goroutine blocked on `Accept()` never got scheduled.
I don't have a clean explanation for *why* this happens specifically when lohar is a child of systemd, and not when it's PID 1 itself. My current guess: something about how systemd manages file descriptors, control groups, or its own epoll sets across PID 1 leaves the resumed Go process holding a poller in a state Go's runtime doesn't recover from. That's speculation, not diagnosis.
What I know empirically:
* Reproduced across multiple VMM and guest-kernel versions.
* Reproduced with `Restart=no` and `Restart=always`.
* Lohar as PID 1 (no systemd) — never reproduced, in CI or production.
:::note\[If you know more, tell me.] This is the design decision I'd most like to revisit. If you've seen the same symptom and traced it further, or if you think my diagnosis is wrong, please [open an issue](https://github.com/sahil-shubham/bhatti/issues). The systemctl shim works, but real systemd would be less code to maintain. :::
So lohar stayed PID 1 — and still is under krucible's block-root boot, where the rootfs's own `/init.krun` (lohar) is the first userspace process the kernel starts.
## The systemctl shim
The downside of "lohar is PID 1, no systemd" is that package post-install scripts assume systemctl exists. Most of Debian's package machinery — `deb-systemd-helper`, `deb-systemd-invoke`, `invoke-rc.d`, `policy-rc.d`, `/sbin/runlevel` — calls into a chain that ultimately runs `systemctl start <unit>`. Without it, a fresh `apt install openssh-server` looks like this:
```
Setting up openssh-server (1:9.6p1-3ubuntu13.5) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
ssh.service is a disabled or a static unit, not starting it.
```
Binary on disk, configuration in place, service never started — and the package's "if you can't start, fail" check trips. The fix is to make each tool in that chain happy.
The shim ([`cmd/lohar/systemctl.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/systemctl.go)) is about 1600 lines. It covers the surface that Debian's tools actually call (`start`, `stop`, `restart`, `status`, `enable`, `disable`, `is-active`, `is-enabled`, `kill`, `daemon-reload`, `show -p`) plus the flags they pass (`--no-reload`, `--quiet`, `--system`, `--no-block`, `--root=`, `--state=`, `--type=`).
It reads real `.service` files from the standard paths (`/etc/systemd/system`, `/usr/lib/systemd/system`, `/lib/systemd/system`) and parses the directives that show up in real-world unit files:
* `ExecStart`, `ExecStartPre`, `ExecReload`, `ExecStop`
* `Type=simple`, `Type=forking`
* `User=`, `Group=` — drop privileges before exec
* `Environment=`, `EnvironmentFile=` — merged into the child's environment
* `Restart=`, `RestartSec=` — auto-restart with backoff
* `WantedBy=multi-user.target` — picked up by `enable` and our boot
The walkthrough below describes what I observe when running `apt install postgresql` inside a sandbox. I haven't verified each step against the maintainer scripts of every Debian package, but it matches the chain documented in [deb-systemd-helper(1)](https://manpages.debian.org/testing/init-system-helpers/deb-systemd-helper.1.en.html) and [invoke-rc.d(8)](https://manpages.debian.org/testing/init-system-helpers/invoke-rc.d.8.en.html). Corrections welcome.
1. apt unpacks the deb.
2. The maintainer postinst script calls `deb-systemd-helper enable postgresql.service` (this is what `update-rc.d` calls now on systemd hosts).
3. `deb-systemd-helper` checks whether systemd looks present by stat'ing `/run/systemd/system/`. Lohar creates this empty directory at boot ([`cmd/lohar/main.go:97`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go#L97)) so the helper proceeds. Without it, the helper falls back to legacy sysvinit handling and our shim never gets called.
4. The helper creates `/etc/systemd/system/multi-user.target.wants/postgresql.service` as a symlink. That's how a service is "enabled" — there's no central database, just symlinks in `wants/` directories.
5. The postinst calls `deb-systemd-invoke start postgresql.service`, which delegates to `invoke-rc.d`.
6. `invoke-rc.d` runs `/usr/sbin/policy-rc.d` to check whether the action is allowed. Our shim ([`scripts/tiers/minimal.sh:73-78`](https://github.com/sahil-shubham/bhatti/blob/main/scripts/tiers/minimal.sh#L73-L78)) exits 0 unconditionally — actions are permitted.
7. `invoke-rc.d` runs `/sbin/runlevel` to determine the current runlevel. Our shim ([`scripts/tiers/minimal.sh:81-89`](https://github.com/sahil-shubham/bhatti/blob/main/scripts/tiers/minimal.sh#L81-L89)) prints `N 5` (multi-user with networking).
8. `invoke-rc.d` decides everything looks fine and runs `systemctl start postgresql.service`. Our shim reads the unit file, resolves `User=postgres`, sets up logging, drops privileges, and `exec`s the binary.
9. Postgres comes up. `systemctl status postgresql` queries the running process by reading the pidfile under `/run/bhatti/services/`. `journalctl -u postgresql` reads `/var/log/bhatti/postgresql.log`.
If postgres crashes, the shim's restart logic ([`cmd/lohar/systemctl.go:745-823`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/systemctl.go#L745-L823)) restarts it according to `Restart=` and `RestartSec=`. At boot, lohar reads `multi-user.target.wants/` and starts each enabled service ([`startEnabledServices` at `systemctl.go:1318`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/systemctl.go#L1318)).
### How services are spawned
When the shim starts a `Type=simple` or `Type=notify` daemon, it doesn't `exec` the binary directly. It goes through a one-purpose helper — `lohar spawn` — so the daemon and any descendants land in the unit's cgroup before they ever run a line of code.
```
┌── lohar (PID 1, supervisor) ────────────────┐
│ CreateCgroup(/sys/fs/cgroup/.../X.service) │
│ cmd := exec.Command("/proc/self/exe", │
│ "spawn", "--cgroup", <path>, "--", │
│ "/bin/sh", "-c", "exec "+ExecStart) │
│ cmd.Start() ──────────────────────┐ │
└────────────────────────────────────────────┼────────┘
│
┌── new process, lohar binary ──────────────▼───┐
│ 1. write own PID to <cgroup>/cgroup.procs │
│ 2. syscall.Exec("/bin/sh", ...) │
│ PID preserved. Becomes /bin/sh which │
│ execs into the daemon. │
└────────────────────────────────────────────────────┘
```
Two invariants do all the work:
1. **The cgroup write is observed by the kernel before any descendant of the writer is born.** `lohar spawn` writes its own PID into `<cgroup>/cgroup.procs` at the top of its `main`, before any `fork(2)` or `execve(2)`. It then `execve`s into `/bin/sh`, which `execve`s into the daemon. PID is preserved through all three transitions; no `fork` happens in between.
2. **`execve` does not change cgroup membership.** A process's cgroup is a property of the kernel task, not the loaded image. So the daemon — same PID throughout — is still in the unit's cgroup. Any subsequent `fork()` the daemon does (X-server detach, dbus pre-fork workers, classic Apache, anything that double-`fork`s during startup) inherits the unit's cgroup. Stop-time `cgroup.kill` catches all of them.
Real systemd does this in one syscall via `clone3(CLONE_INTO_CGROUP)`. Reproducing that in Go means reimplementing `syscall.ForkExec`'s post-fork setup (close-on-exec, signal handlers, credentials) under `syscall.ForkLock` — real surgery on code where bugs are silent and expensive. The helper approach is \~60 LoC of plain Go ([`cmd/lohar/spawn.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/spawn.go)) and gets the same correctness guarantee. `clone3` stays open as a future migration if the extra `execve` ever shows up in profiling — current measurements put it well under a millisecond against services-started times in the tens to hundreds of milliseconds.
For unit-file authors: **nothing changes.** `ExecStart=` works exactly the same way. The shim wraps your command in `/bin/sh -c "exec <ExecStart>"` regardless — systemd's parser does the same thing. The spawn helper is upstream of all of that; you don't need to know about it.
This is the mechanism that makes `KillMode=control-group` (systemd's default) reliable for daemons that fork during startup. Before this landed in v1.11.10 the placement happened *after* `cmd.Start()` returned, and any daemon that `daemon(3)`d during startup could have its child escape into lohar's root cgroup. `systemctl stop kasmvnc` would then leave an orphan `Xkasmvnc` holding the abstract `@/tmp/.X11-unix/X99` socket, breaking the next start with `Cannot establish any listening sockets`. Every shipped tier benefits from the fix transparently — no unit-file changes required.
### What the shim doesn't do
* **Targets** beyond `multi-user.target` and a few well-known names (`sysinit`, `network`, `default`). Anything checking these gets `active`. Anything more specific is a no-op.
* **Socket activation.** `.socket` units resolve to their associated `.service` and the service is started directly. Fine for sshd, breaks for things that genuinely need lazy socket activation.
* **Timers** (`.timer` units). If you need scheduled work, install cron.
* **D-Bus.** You can install `dbus` and the shim will start it as a normal service, but the host doesn't speak DBus to the guest.
## Pinning systemd out
The shim only works if real systemd never gets installed. The trap: many useful packages (openssh-server, sudo, postgresql, nginx) declare `Recommends: libpam-systemd`. By default, apt installs Recommends. And `libpam-systemd` Depends on `systemd`. So one innocent `apt install openssh-server` would silently:
* install real systemd
* install `/bin/systemctl` over our `/usr/bin/systemctl` symlink (PATH resolution prefers `/bin` for some shells)
* install systemd-resolved, which would overwrite `/etc/resolv.conf` on next service start
* leave the rootfs in a half-broken state where snapshot/restore fails (the original problem)
The fix is in [`scripts/tiers/minimal.sh:34-41`](https://github.com/sahil-shubham/bhatti/blob/main/scripts/tiers/minimal.sh#L34-L41):
```bash
printf "Package: systemd systemd-sysv systemd-resolved
Pin: release *
Pin-Priority: -1
" > /etc/apt/preferences.d/no-systemd-daemon
```
`Pin-Priority: -1` tells apt the package is never installable from any source. Server packages only Recommend `libpam-systemd` (never Depend), so apt's resolver just drops it from the install set and proceeds.
If you build a custom rootfs based on a tier other than `minimal`, this pin is inherited because every tier sources `minimal.sh` first.
### Privileged systemctl from a non-root caller
`bhatti exec dev -- systemctl status sshd` runs as the `bhatti` user inside the VM. That user can read `.service` files but can't kill a process owned by root. The shim handles this with a small in-guest IPC:
* The user-side `systemctl` invocation marshals its request as a [`SystemctlRequest`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/messages.go#L67) and sends it over a Unix socket at `/run/bhatti/systemctl.sock`.
* PID 1 lohar accepts on that socket, uses [`SO_PEERCRED`](https://man7.org/linux/man-pages/man7/socket.7.html) to learn the caller's UID *from the kernel*, runs the operation as root, and sends back the formatted output.
* The user-side `systemctl` prints what came back and exits with the matching code.
`SO_PEERCRED` is the trust boundary, not anything in the request payload — a non-root caller could otherwise forge a UID claim and get a privileged operation done.
## Boot sequence
Under krucible's block-root boot, lohar is placed at `/init.krun` in the root image and runs as PID 1. With the lean external kernel the kernel command line carries `root=/dev/vda … init=/init.krun`; with libkrun's bundled kernel the implicit init is disabled so the rootfs's own `/init.krun` boots instead. Either way, after the kernel finishes its own boot, lohar is the first userspace process.
Lohar writes phase markers to `/run/bhatti/boot-timing.txt` as it goes, so you can `cat` it inside any sandbox and see exactly where time went. A run might look like this:
```
+0ms start
+2ms mounts_done
+3ms lo_up
+8ms config_applied
+12ms network_done
+18ms tcp_listen ← agent reachable here
+19ms services_started
+21ms init_session_started
```
That's lohar's own clock — the time between PID 1 starting and the agent listening. The wall-clock time from `bhatti create` to the daemon flipping the sandbox to `running` is dominated by the kernel boot and varies with hardware and rootfs. To measure it on your machine, [`bench/run.sh`](https://github.com/sahil-shubham/bhatti/blob/main/bench/run.sh) runs the full suite.
What each phase does ([`cmd/lohar/main.go:48`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go#L48)):
1. **Mount filesystems** — `/proc`, `/sys`, `/dev`, `/dev/pts`, `/tmp`, `/run`, `/dev/shm`. The kernel hands you a bare ext4 with none of these.
2. **Mount cgroup v2** with `+cpu +memory +io +pids` enabled. cgroup v2 is needed for in-guest tools that do their own resource accounting — Docker is the most common, but the same applies to anything using cgroup-style limits (containerd, podman, slurmd).
3. **Bring up `lo`** with raw `ioctl(SIOCSIFFLAGS)`. `iproute2` isn't wired up yet — we're a few milliseconds into PID 1. The guest's `eth0` is a virtio-net device wired to the per-owner `bhatti-netd` gateway, and lohar brings it up from the config drive in the next step (raw rtnetlink, no `ip` binary, no kernel IP autoconfig). See [Networking](/docs/under-the-hood/networking/) for that story.
4. **Read the per-sandbox config** from a small ext4 image attached as `/dev/vdb`. Apply hostname, the `eth0` address/gateway/DNS, env vars, files, and volume mounts. Then unmount and remove the mount point — secrets shouldn't sit on disk after they've been read.
5. **Install signal handlers, start the zombie reaper.** PID 1 is responsible for reaping orphaned child processes; [`go reapZombies()`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go#L128) does it.
6. **Start the syslog receiver.** `/dev/log` becomes a Unix datagram socket; lohar parses RFC-3164 messages and routes them to per-service log files under `/var/log/bhatti/`. Anything in the guest using `syslog(3)` (cron, sshd, postgres) writes to a readable log without any extra configuration.
7. **Listen on `/run/bhatti/systemctl.sock`** for the privileged systemctl IPC described above.
8. **Listen on vsock ports 1024 (control) and 1025 (forward).** The host has been polling the control port since the VM started (libkrun bridges each guest vsock port to a host Unix socket the daemon dials). Now `exec true` succeeds, and the daemon flips the sandbox's status to `running`.
9. **Start enabled services** from `/etc/systemd/system/multi-user.target.wants/`.
10. **Run `/etc/bhatti/init.sh`** if it exists (30 s timeout, output to stderr). Useful when you want to bake boot logic into a custom rootfs without forcing every sandbox to specify `--init`.
11. **Run the `--init` script** from sandbox creation as a TTY session named `init`. Attach with `bhatti shell dev --session init` to watch your provisioning run.
12. **Block forever.** PID 1 is not allowed to exit.
## Sessions
Every TTY exec in bhatti creates a *session* — a persistent handle to the running command, with an ID, a 64 KB scrollback ring buffer, and a PTY master file descriptor. Sessions survive client disconnects: the process keeps running, output keeps flowing into the scrollback. A client reconnecting to the same session ID gets the scrollback replayed, then live I/O.
Three reasons this is the default rather than a feature:
1. **Init scripts need to be attachable.** When you create a sandbox with `--init "npm install && npm run build"`, that command runs as a session named `init`. You can `bhatti shell dev --session init` to attach mid-run and see what's happening. If exec weren't a session, the init output would be invisible from the host.
2. **Shells need to survive proxy disconnects.** A real example from `PLAN-shell-sessions.md`: a user ran `bhatti shell rory` through `api.bhatti.sh` (Cloudflare Tunnel) and started `hermes gateway`, a daemon that prints a startup banner then waits for events. Cloudflare Tunnel kills idle WebSocket connections after about 100 seconds. The shell dropped, the daemon kept running inside the VM, and the user had no way back. With sessions, reconnecting replays the scrollback and resumes the live PTY.
3. **Snapshot/restore needs to preserve interactive shells.** A shell that's running a long-running command when the VM is snapshotted should be the same shell after restore. Sessions are the data structure that survives that round-trip.
The session registry lives in [`cmd/lohar/session.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/session.go). Two caps that matter, both per-VM ([`cmd/lohar/handler.go:49-51`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/handler.go#L49-L51)):
* 50 concurrent connections
* 20 active sessions
For a normal sandbox, you'll never hit either. Services started by `startEnabledServices` are tracked separately and don't count against the session cap.
### Scrollback ring buffer
Each session owns a fixed-size 64 KB ring buffer ([`cmd/lohar/session.go:50`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/session.go#L50)). Writes wrap when full, overwriting the oldest data. The buffer's `Bytes()` method returns the contents in order — oldest first — so a reattaching client sees the same byte sequence the original PTY emitted, just truncated to the most recent 64 KB.
Why a fixed size? An interactive shell can produce arbitrary output (think `find /` or a chatty test runner). If the buffer grew without bound, a forgotten session would eventually OOM the VM. 64 KB is enough to replay the last few hundred lines of typical output, which is what the "what was happening when I disconnected" use case actually needs.
### PTY allocation
The PTY pair for each TTY session is allocated with raw syscalls ([`cmd/lohar/tty.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/tty.go)), no CGo:
1. Open `/dev/ptmx` (the PTY multiplexor).
2. `TIOCGPTN` ioctl — get the slave PTY number assigned to this master.
3. `TIOCSPTLCK` ioctl with `0` — unlock the slave (kernel won't let you open it until you do this).
4. Open `/dev/pts/<N>` as the slave file.
The child process is started with `Setsid: true` and `Setctty: true`, which makes it a new session leader and assigns the slave PTY as its controlling terminal — that's what makes Ctrl+C send SIGINT and the shell prompt look like a terminal. Window size is applied via `TIOCSWINSZ` ioctl on the master whenever the host sends a `RESIZE` frame.
## Piped exec (the simpler path)
`bhatti exec dev -- npm install` is *not* a session by default. It's a piped exec — stdout and stderr stream back as `STDOUT` / `STDERR` frames, the child runs in its own process group, and the connection lives until the child exits. Sessions add overhead a one-shot command doesn't need.
The lifecycle ([`cmd/lohar/exec.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/exec.go), [`piped_session.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/piped_session.go)):
1. `exec.Command` with `Setpgid: true`. Child gets its own process group, distinct from lohar.
2. Three pipes (stdin, stdout, stderr) connected to the child.
3. Two reader goroutines fan stdout and stderr into `STDOUT` / `STDERR` frames. Both push to a single writer goroutine through a channel, which serializes frame writes — concurrent writes would interleave at byte boundaries and corrupt frames on the wire.
4. Wait for both readers to drain *before* `cmd.Wait()`. Otherwise the `EXIT` frame can race ahead of the last output frame and the host would see the exit code before the last line of output.
5. `syscall.Sync()` — flush any files the command wrote. The host might snapshot the VM the next millisecond; dirty pages in the guest's page cache aren't on the virtio-blk device yet, so a snapshot taken before the flush would miss them.
6. Send the `EXIT` frame with the integer exit code, close.
If the host kills the connection (you `Ctrl+C`'d the CLI), lohar sends `SIGKILL` to the negative PID — kills the entire process group. `npm install` spawns Node, which spawns dozens of children; without process-group kill, they orphan and keep running.
There's a third mode (`session: true` in the ExecRequest, no TTY) for piped sessions with scrollback — fire-and-forget a long-running build, get the output streamed when you next attach. The CLI uses this internally for `--detach`.
### Environment variable precedence
Every exec inherits a merged environment, applied in this order (later wins):
```
1. defaults PATH, TERM, HOME, LANG
2. config drive env secrets and env vars set at sandbox creation
3. per-request env --env on the exec command, or env in API body
```
So `bhatti exec dev --env API_KEY=test -- env` will see `API_KEY=test` even if a secret with the same name was set on the sandbox. This is deliberate: the per-request env wins so you can override secrets in CI or locally without rotating them.
### Config-env bridge
Layer 2 above — the config-drive env from `bhatti create --env K=V,K=V` — reaches `bhatti exec` natively because exec merges from `configEnv` in memory. To reach *units* started by the systemctl shim, lohar additionally serialises `configEnv` to `/run/bhatti/config-env` as canonical `KEY=VALUE` lines, after the config drive is applied and before `startEnabledServices` runs. Unit files opt in with:
```ini
[Service]
EnvironmentFile=-/run/bhatti/config-env
```
The leading `-` makes the file optional, so units in `minimal`-derived rootfs images without any `--env` flags still boot cleanly.
This is how every tier's tunables work in v1.11.8+:
* `docker` tier: dockerd inherits `DOCKER_*` env vars at process start.
* `browser` tier: `CHROME_REMOTE_PORT` and `CHROME_FLAGS` expand inside `headless-chrome.service`'s ExecStart.
* `computer` tier: `DISPLAY_WIDTH`, `KASM_FRAMERATE`, etc. flow into `kasmvnc.service`'s ExecStart and override the unit's own `Environment=` defaults (`EnvironmentFile=` always wins over `Environment=` in systemd's precedence rules; the shim matches).
User-supplied values **always win** over tier defaults because the config-env file is written before any service starts; the unit's `Environment=` defaults are loaded first, then `EnvironmentFile=` overrides matching keys. (One subtlety: `kasmvnc-firstboot.service` appends a computed `KASM_THREADS=$((nproc-1))` to `config-env` only if the key isn't already present, preserving user precedence.)
## File operations
The agent owns file ops because `cat | exec` is slow, awkward, and loses metadata. The frame types ([`pkg/agent/proto/constants.go:42-49`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/constants.go#L42-L49)):
\| Op | Frame | Returns | |-------|------------------|---------| | Read | `FILE_READ_REQ` | response with size + mode, then `STDOUT` frames, then `EXIT` | | Write | `FILE_WRITE_REQ` | take `STDIN` frames, write atomically, return OK | | Stat | `FILE_STAT_REQ` | `FileInfo` | | Ls | `FILE_LS_REQ` | `[]FileInfo` |
Two details worth their own paragraph.
**Atomic writes.** The write path: open `path.bhatti-tmp`, stream the content into it, `fsync`, `rename` over the target. POSIX guarantees `rename` is atomic on the same filesystem — a process inside the VM that's `tail`-ing the file will see either the old content or the new content, never a half-written state. The `fsync` before rename is the load-bearing part: without it, the rename is metadata-durable but the data is still in the guest's page cache, not on the virtio-blk device. If the host snapshots the VM between the rename and the kernel's periodic flush, the snapshot contains a file that exists with zero bytes.
**Server-side truncation.** Coding agents always truncate file reads — typically 2000 lines or 50 KB. If we did that on the client side, we'd ship 100 MB of log file across the wire just to throw 99.95% of it away. Instead, `FILE_READ_REQ` accepts `offset`, `limit`, `max_bytes` parameters, and lohar applies them as it reads line by line. The response always includes the *full* file size so the consumer can detect truncation.
## What lohar isn't
A few things the docs sometimes imply lohar does, that it doesn't:
* **It's not your shell.** Your shell is whatever the rootfs has (usually `/bin/bash`); lohar exec's it.
* **It's not a service supervisor for arbitrary daemons.** It runs unit files registered through `enable` and its own session children. Daemons that fork-and-detach the old sysvinit way work, but their state isn't fully tracked.
* **It doesn't reach out to the network on its own.** No telemetry, no phone-home, no update checks. The only outbound TCP from the guest is whatever you run in it.
## Where to go next
* [Architecture overview](/docs/under-the-hood/architecture/) — how lohar fits into the broader system
* [Thermal states](/docs/under-the-hood/thermal-states/) — what happens to this lohar instance when it goes idle
* [Networking](/docs/under-the-hood/networking/) — the per-sandbox network setup and how `/dev/vdb` gets there
* [Wire protocol](/docs/under-the-hood/wire-protocol/) — frame format, frame types, atomic writes
* [Decisions & learnings](/docs/under-the-hood/decisions/) — the systemd story above, the diff-snapshot incident, and other lessons paid for in real time
# Networking: bhatti-netd, gVisor, and the vsock tunnel
> A per-owner userspace gVisor gateway that polices egress and isolates the guest from the host — no host TAP, bridge, or iptables — how a guest's packets get in and out, and the ways to reach a service inside a sandbox from outside.
Every VM gets internet access, isolation from the host, and reachability to its owner's other sandboxes — and none of it touches the host's own network stack. There are no TAP devices, no bridges, no iptables rules. Each owner's sandboxes hang off a single userspace process, **`bhatti-netd`**, that *is* their network: a gVisor netstack running as an ordinary host process, on by default.
This page is about how that gateway works, how a guest's packets get in and out, and the paths for reaching a service inside a sandbox from outside.
## Design decisions on this page
1. **A userspace gateway per owner, not host bridges.** Each owner's VMs share one `bhatti-netd`; different owners get different gateways and never share a network. See [A gateway per owner](#a-gateway-per-owner).
2. **The guest link is virtio-net over a unix socket, not a TAP.** libkrun connects, netd listens; the guest sees a normal `eth0`. See [The guest link](#the-guest-link).
3. **Egress is policed and the host is unreachable by construction.** Every guest flow is terminated at the gateway and re-originated through a vetting dialer. See [Egress policy and host isolation](#egress-policy-and-host-isolation).
4. **Same-owner siblings reach each other through the gateway, never directly.** An L3-routed proxy inside netd carries the sibling leg. See [Sibling reachability](#sibling-reachability).
5. **Host↔guest ports ride the vsock tunnel.** `bhatti forward` and the proxies never expose the guest on a host-routable address. See [Reaching services inside sandboxes](#reaching-services-inside-sandboxes).
6. **The legacy TSI backend is off by default.** It shares the host netstack — simpler, but the guest can reach host loopback. See [The legacy TSI backend](#the-legacy-tsi-backend).
## A gateway per owner
When a user creates their first sandbox, bhatti spawns them a `bhatti-netd` — one gVisor userspace netstack that serves all of that owner's sandboxes. The second user gets their own. Each owner is alone on their own network; there's no shared segment to ARP-scan.
The addressing ([`pkg/engine/krucible/engine.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/engine.go)): each owner's gateway lives on `100.64.<idx>.0/24` — RFC-6598 carrier-grade-NAT space, used here for per-owner vnets. The gateway is `.1`; guests are `.2`, `.3`, … in creation order.
\| Owner index | Gateway | Guest 1 | Guest 2 | |---|---|---|---| | 0 | `100.64.0.1` | `100.64.0.2` | `100.64.0.3` | | 1 | `100.64.1.1` | `100.64.1.2` | `100.64.1.3` | | ... | ... | ... | ... |
Grouping is by owner (the sandbox's `UserID`): same owner ⇒ same netd ⇒ siblings reach each other. A sandbox with no owner gets its own isolated netd keyed by sandbox id, so the single-sandbox case is just N = 1.
netd is spawned **detached**, so it survives a daemon restart, and its state (socket path, next-guest index, pid) is persisted in a `netd.json` next to its socket. On restart the daemon **re-adopts** the running gateway by pid instead of respawning onto a socket it still holds; it's reference-counted, so the owner's last `destroy` tears it down ([`recovery.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/recovery.go)).
The Firecracker-era per-user `subnet` index still exists in the store as `subnetIdx`, and it picks the `100.64.<idx>` octet — but there are no per-user bridges behind it anymore. It's a vnet number, not a Linux bridge.
## The guest link
There are no TAP devices. The guest's NIC is a **virtio-net** device wired to a **unix stream socket** (`krun_add_net_unixstream`). netd *listens* on that socket; libkrun's virtio-net backend *connects* to it as the VM boots ([`cmd/bhatti-netd/main.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/bhatti-netd/main.go)). Each accepted connection becomes a switch port on the gateway; the frames on it are the raw ethernet the guest emits, carried over the socket by `pkg/gateway.FrameConn`.
From the guest's side it's an ordinary `eth0`. lohar brings it up from the per-sandbox config drive via **rtnetlink** directly — no `ip` binary, no DHCP, no kernel IP autoconfig (our lean kernels build without `CONFIG_IP_PNP`) ([`cmd/lohar/netlink.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/netlink.go)). It assigns the guest its `100.64.<idx>.<n>/24`, adds an on-link route to the gateway `.1`, and a default route via `.1`. So the guest sends *everything* — internet and siblings alike — to the gateway.
Each guest gets a deterministic locally-administered MAC (`52:54:00:00:00:<n>`); the gateway's is `52:54:00:00:00:01`. netd learns each guest's MAC → port as frames arrive, so it can demux stack-originated frames back to the right guest and flood only genuine broadcasts ([`cmd/bhatti-netd/netstack.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/bhatti-netd/netstack.go)).
### The offloaded-checksum gotcha
Guests offload their TX checksums (partial / pseudo-header only), and libkrun strips the `virtio_net_hdr` flag that would tell us so — so the checksums that reach netd on the wire aren't final. If gVisor verified them it would drop every guest packet and egress would silently time out.
We trust frames arriving over the local vsock link, so the gateway advertises `CapabilityRXChecksumOffload`: gVisor marks received packets checksum-validated and skips verification. We deliberately do *not* advertise TX offload — the stack computes real checksums on the frames it sends *to* guests (including a re-originated sibling leg), so there are no manual fixups. There's a regression test that fires a SYN with a deliberately-wrong checksum and asserts it still reaches the forwarder ([`delivery_test.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/bhatti-netd/delivery_test.go)). This is the kind of thing that's invisible when it works and takes a day to find when it doesn't.
## Egress policy and host isolation
netd doesn't route packets; it **terminates** every guest TCP flow at a gVisor TCP forwarder and **re-originates** it ([`cmd/bhatti-netd/forward.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/bhatti-netd/forward.go)). For an internet destination, the new leg is dialed through a vetting dialer — the egress guard (`pkg/gateway`) — which classifies the destination:
* **Public internet** — allowed. This is the default posture (`PosturePublic`).
* **Private ranges** (RFC-1918, ULA, CGNAT) — denied unless an explicit allow-CIDR opts them back in.
* **Loopback, link-local, the host itself, multicast, and cloud metadata** (`169.254.169.254`) — never allowable, including the IPv4-mapped and NAT64-embedded spellings of those addresses that a naive check would miss.
A denied or unreachable destination gets a clean RST back to the guest rather than a hang (the forwarder dials first, then completes). And because the guest has its *own* loopback inside the VM and its only route out is the gateway, there is simply no path from the guest to the host's bhatti API, SSH, or anything else on the host — the isolation the old iptables `INPUT` rules used to buy is now a property of the topology. The integration suite asserts both halves: a guest dial to an RFC-1918 address is refused, and public egress works end to end ([`net_test.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/net_test.go)).
This is a feature you don't configure. It's true the moment the gateway starts, for one sandbox or ten thousand. Because every flow — egress and sibling alike — passes through this one policed, observable chokepoint, it's also the natural home for per-sandbox egress policy and L7 [secret substitution](/docs/managing/secrets/).
## Sibling reachability
Same-owner sandboxes can reach each other — an agent sandbox can `curl` a worker at `100.64.<idx>.3:<port>`, or by name. But they never talk directly. When the forwarder sees a destination inside the owner's own subnet, it dials that sibling *through the gVisor stack itself*, which routes the new leg out to the target guest's link ([`forward.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/bhatti-netd/forward.go)).
So the sibling leg passes through the same chokepoint as egress; guests can't reach each other behind netd's back, checksums are native on the re-originated leg, and it's all observable. A sandbox belonging to a *different* owner is on a different netd entirely and has no route to it — cross-owner traffic never shares a network (the sibling test asserts both the reachable and the isolated direction).
Names: lohar points `/etc/resolv.conf` at the gateway's in-cluster resolver when it's available — that resolver answers sibling sandbox names (`<sandbox>`, `<sandbox>.sb`) and forwards everything else upstream, so it's the only nameserver the guest needs. If it isn't reachable, lohar falls back to public resolvers (Cloudflare, then Google) directly ([`cmd/lohar/main.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/main.go)). We never list the internal and public resolvers together: a stale sibling name would otherwise fall through to public DNS and resolve to the wrong thing.
## Reaching services inside sandboxes
Nothing outside reaches a guest by its address — `100.64.<idx>.<n>` only exists inside its owner's gateway. Every path in goes through the host, over the engine's **vsock tunnel** to `localhost:<port>` inside the VM. There are three.
### `bhatti forward` (host↔guest)
```
bhatti forward <sandbox> <guest-port> [host-port]
```
The daemon binds `127.0.0.1:<host-port>` on the host and bridges each accepted TCP connection to `localhost:<guest-port>` inside the VM over the vsock `Tunnel` primitive ([`pkg/forward`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/forward/forward.go)). It's raw bytes, so any TCP service works — HTTP, Postgres, Redis — unlike the HTTP-aware public proxy. The forward is torn down when the command exits (Ctrl-C) or the sandbox is destroyed. This same primitive is the building block for the server-brokered inter-sandbox mesh: a sandbox can get a stable host endpoint that other sandboxes reach via the host.
### Authenticated proxy
```
GET /sandboxes/:id/proxy/:port/<path>
Authorization: Bearer <token>
```
The daemon authenticates, looks up the sandbox, opens a tunnel to `localhost:<port>` inside the VM, and proxies HTTP/WebSocket traffic. Cold sandboxes wake on the first request. Useful for development — you don't need to publish a port to use one.
### Public proxy (published URLs)
```
ANY https://<alias>.bhatti.sh
```
When you `bhatti publish dev -p 3000 -a my-app`, the daemon creates a publish rule mapping `my-app` → `(sandbox=dev, port=3000)`. Public requests hit `:443`, get routed by Host header, looked up against the rule cache ([`pkg/server/public_proxy.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/public_proxy.go)), and tunneled into the VM.
The public proxy has a few extra moves over the authenticated one:
* **In-memory route cache** (LRU, 10 K entries) so a hot URL doesn't query SQLite per request.
* **`singleflight.Group` for resume coalescing** — wake-then-serve: if 50 concurrent requests hit a cold sandbox, only one wake actually happens; the others wait on it. Without this you'd get 50 simultaneous snapshot loads.
* **Per-alias and global rate limiting** — protects against abuse.
* **5-minute per-request deadline, 50 MB body limit.**
Both proxies use `httputil.ReverseProxy` with a custom transport that treats the engine's tunnel as an `http.RoundTripper`. WebSocket connections are hijacked and relayed bidirectionally with a 10-minute idle timeout.
## Snapshot, restore, and recovery
Because netd is a separate, detached process and the guest link is a unix socket, snapshot/restore doesn't have to preserve any host-side network device — there are no TAPs or bridges to leak or rebuild. When a stopped sandbox is restored, its virtio-net backend simply reconnects to its owner's `bhatti-netd`, which is still listening (netd outlives individual VMs).
If the daemon itself restarts, it re-adopts each still-running gateway by pid from that gateway's `netd.json` record rather than respawning onto the socket it holds, and reference-counts it so the owner's last `destroy` still shuts it down ([`recovery.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/recovery.go)). There's no system-wide network cleanup step on startup — there's nothing on the host to clean up.
## The legacy TSI backend
netd is on by default (`krucible_net_backend: true`). Setting it to `false` selects libkrun's built-in **TSI** (transparent socket impersonation) backend instead: the guest has no separate netstack, and its sockets are impersonated directly on the *host's* network stack.
It's simpler and needs no gateway process, but it means the guest shares the host's netstack — it can reach the host's loopback — so it gets neither the host isolation nor the policed egress that netd provides. It's there for debugging and legacy configs; leave the gateway on. See [Configuration → `krucible_net_backend`](/docs/reference/config/#server-config).
## Where to go next
* [Thermal states](/docs/under-the-hood/thermal-states/) — how the network stays consistent across snapshot/resume
* [Lohar](/docs/under-the-hood/lohar-the-blacksmith/) — the guest agent, which talks to the host over vsock (a separate channel from this data-plane network)
* [Custom domain](/docs/managing/custom-domain/) — TLS for the public proxy, wildcard certs, ACME flow
* [Configuration](/docs/reference/config/#server-config) — the `krucible_net_backend` and runtime-path fields
# Storage
> How bhatti stores sandbox state on disk — qcow2 copy-on-write overlays, base-image sharing, and why v2 needs no special filesystem.
bhatti's storage layout is one of the most load-bearing design decisions in the system and the least visible. Every `bhatti create`, every snapshot, every resume from cold leans on it. In v1 those assumptions lived in the *filesystem* — reflinks on btrfs, a real copy on ext4 — and the cost of a create swung by an order of magnitude depending on which one you had.
v2 moves copy-on-write down a layer, into the disk-image format itself. Each sandbox's root is a **qcow2 overlay** over a shared read-only base image. Because the CoW happens inside qcow2, not in the filesystem, it works the same on ext4, xfs, btrfs, and APFS — no btrfs requirement, no reflink, no cost cliff. This page covers what's actually on disk per sandbox, how the overlay stays cheap, and the one escape hatch (`KRUCIBLE_ROOT_RAW`) for when you don't want it.
## What's on disk per sandbox
Under `/var/lib/bhatti/sandboxes/<id>/`:
```
root.qcow2 qcow2 CoW overlay over the shared base image (the sandbox's disk deltas)
config.ext4 ~1 MiB ext4 with hostname, env, secrets, file drops, per-sandbox token
vmspec.json the VM spec the helper booted from
state.json recovery metadata (helper PID, socket paths, bundle ref)
vmm.log the bhatti-vmm helper's log
bundle/ the cold snapshot bundle (only while the sandbox is cold)
```
The cold `bundle/` directory holds a self-contained snapshot:
```
memory.img guest RAM image (equal to configured memory)
+ VM/device + vCPU state
rootfs.qcow2 point-in-time copy of the root overlay
config.ext4 the config drive
<volume images> any attached persistent volumes
manifest.json bundle metadata (vCPUs, memory, arch/feature gate, proto_ver)
```
Plus shared resources elsewhere:
* `/var/lib/bhatti/images/rootfs-<tier>-<arch>.ext4` — read-only base rootfs templates. Every sandbox's `root.qcow2` is a thin overlay *over* one of these; the base is never modified.
* `/var/lib/bhatti/volumes/<user_id>/` — standalone volumes (`bhatti volume create`), attached to one sandbox at a time.
From the guest's perspective these are all virtio-blk block devices. The root overlay is `/dev/vda`, the config drive is `/dev/vdb`, and attached volumes enumerate from `/dev/vdc` upward. On the host, `root.qcow2`'s on-disk size is just the *delta* the sandbox has written — `ls -lh` shows the logical device size, `du -h` shows the real allocation. The gap is the whole point; see [Disk usage](/docs/managing/images/#disk-usage) on the images page for the walkthrough.
## How the CoW root overlay works
`bhatti create` provisions the overlay through the VMM helper — the daemon is pure Go and never links libkrun, so it shells to `bhatti-vmm create-overlay`, which creates a qcow2 CoW node via `krun_create_disk_overlay` (libkrun's `imago` disk library — the same code that opens the image at boot) ([`pkg/engine/krucible/engine.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/engine.go), [`cmd/vmm/main.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/vmm/main.go)):
```go
// Default: a qcow2 CoW root — instant + host-FS-independent (no
// reflink/btrfs). Raw is the opt-out (KRUCIBLE_ROOT_RAW=1).
rootImg = filepath.Join(sandboxDir, "root.qcow2")
e.createRootOverlayQcow2(rootImg, base)
baseSpec.RootDiskFormat = "qcow2"
```
A fresh overlay is **metadata-only** — a tiny qcow2 v3 header that records the backing image, never a copy of it. Creation is instant regardless of base size, and the read-only majority of every tier (`/usr`, `/lib`, package payloads) stays shared: one base file on disk backs every sandbox built from it, and the OS page cache holds one copy of those pages in RAM across all of them. The sandbox only allocates blocks when it *writes* — a config tweak, a log line, a package install. Typical overhead of the qcow2 indirection on agent/dev workloads is **\~0.5%**.
The key difference from v1: this is all filesystem-independent.
\| Filesystem | Create-from-base | Base sharing | Notes | |---|---|---|---| | **ext4** | instant (metadata overlay) | yes | no reflink needed | | **xfs** | instant | yes | — | | **btrfs** | instant | yes | optional transparent compression on top | | **APFS** (macOS) | instant | yes | Apple Silicon dev boxes |
There's no "what changes on ext4" penalty section any more, because nothing changes. The overlay is a qcow2 file; qcow2 does the CoW; the underlying filesystem just stores bytes.
Inside the guest the overlay is transparent. libkrun (imago) presents `root.qcow2` as a plain raw ext4 at `/dev/vda` — the kernel command line gets `root=/dev/vda` and the guest never knows it's talking to a CoW overlay ([`cmd/vmm/main.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/vmm/main.go)).
## Volumes and the config drive
Standalone volumes (`bhatti volume create`) and the per-sandbox config drive are also block devices, attached to the VM as additional virtio-blk disks ([`pkg/engine/krucible/engine.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/engine.go)):
* **`/dev/vdb`** — the config drive, a \~1 MiB ext4 image carrying the sandbox's hostname, environment, decrypted secrets, injected files, and its per-sandbox auth token. Read-only from the guest's side.
* **`/dev/vdc` and up** — attached persistent volumes, in attach order. Each carries its guest mount path and filesystem in the config drive's volume spec. A volume image may itself be raw or a qcow2 CoW node; the VMM opens each with the right format.
Because volumes are separate block images, their data is independent of the root overlay: it survives even when a memory snapshot doesn't, and a snapshot/fork of a sandbox restores the *whole device set* (RAM
* root overlay + attached volumes) as one consistent bundle.
## Snapshots, forks, and saved images
Everything expensive in v1's storage layer was a `cp --reflink`. In v2 it's a qcow2 operation, and the three write paths all reuse the same overlay machinery:
* **`bhatti stop` / cold tier.** The thermal manager pauses the VM at a quiesced boundary and writes the full `bundle/` (memory image + VM state + a copy of `root.qcow2` + config drive + volumes + `manifest.json`), then frees RAM. `bhatti start` restores it. See [Thermal states → Warm → Cold](/docs/under-the-hood/thermal-states/#warm--cold).
* **Fork (`bhatti create --from`).** An instant CoW clone: the new sandbox's root is a fresh qcow2 overlay over the parent's image, and its memory is cloned from the parent's snapshot. No copy of the base.
* **`bhatti image save`.** Captures the running root as a bootable qcow2 image without stopping the sandbox — a quiesce (in-guest `sync`, pause, host flush, copy the overlay), yielding a qcow2 CoW node that itself still backs the shared base ([`pkg/engine/krucible/snapshot.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/snapshot.go)). Stamping a new sandbox from it is another instant overlay.
All snapshots are **Full** — v2 has no diff/incremental snapshot mode. The reasoning lives on the thermal page, [Why all snapshots are Full](/docs/under-the-hood/thermal-states/#why-all-snapshots-are-full). The bundle is portable across machines of the same architecture (`manifest.json` records an arch/feature gate; a cross-arch restore is refused).
## Cold wake and the page cache
When a sandbox is in the [Cold state](/docs/under-the-hood/thermal-states/), its `bundle/memory.img` is the only thing standing between "dead VM" and "fully restored guest." Restore re-launches a fresh `bhatti-vmm` helper that reads the memory image back into a new VM and resumes from the snapshot point. How fast that is depends heavily on whether `memory.img` is still in the host page cache.
Right after a snapshot is written, its pages are hot in the cache, so the restore read is nearly free. Thirty minutes later, under memory pressure from a busy host, the kernel may have evicted them — and the restore does a real disk read of the memory working set. This is filesystem-agnostic: it's the OS page cache doing its job, not anything qcow2 or btrfs specific.
In practice a cold restore is **sub-second** — \~380 ms measured end-to-end on Apple Silicon (M-series, NVMe) for a small sandbox. Slower storage (a Raspberry Pi 5's capped NVMe) is proportionally slower; the shape is the same. A **warm** resume, where the helper is still alive and the pages are still resident, is \~4 ms — no disk read at all. See [Thermal states → What "cold wake" actually costs](/docs/under-the-hood/thermal-states/#what-cold-wake-actually-costs) for the full breakdown.
Bhatti doesn't currently bias eviction of `memory.img` files in the page cache, or prefetch them ahead of a restore. Both are possible future optimizations, bounded by demand and not yet measured to clear a bar that matters; tracked for later.
## No special filesystem needed
The headline for self-hosters: put `/var/lib/bhatti` on your normal root filesystem and you're done. ext4, xfs, btrfs, APFS — all give instant create-from-base and shared-base dedup, because CoW is in the qcow2 layer, not the filesystem.
btrfs and xfs still offer *transparent compression* on the base images and bundles if you mount them that way, and it can shrink the data dir meaningfully. But that's an optimization you opt into, not a prerequisite the way btrfs+reflink was in v1. If you're coming from a v1 install on a btrfs loopback, you don't need to carry it forward — v2 is a fresh install.
If you genuinely want the raw path — a plain ext4 root image instead of a qcow2 overlay, for native disk performance or to loop-mount the image on the host — set `KRUCIBLE_ROOT_RAW=1`. The daemon then clones the base into a raw `root.img` per sandbox instead of overlaying it ([`pkg/engine/krucible/engine.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/engine.go)). You trade instant, dedup'd creates for a full base copy per sandbox — the qcow2 default is the right choice for almost everyone.
## See also
* [Self-hosting → Storage, no special filesystem needed](/docs/self-hosting/#storage--no-special-filesystem-needed) — the operator's short version
* [Images → Disk usage](/docs/managing/images/#disk-usage) — what you see in `/var/lib/bhatti/images/` and `/var/lib/bhatti/sandboxes/<id>/`
* [Thermal states](/docs/under-the-hood/thermal-states/) — the state machine that decides when the memory image gets written and read
* [Architecture → Where state lives](/docs/under-the-hood/architecture/#where-state-lives) — full inventory of `/var/lib/bhatti/`
# Thermal states: hot, warm, cold
> How sandboxes pause themselves, why all snapshots are Full, krucible's lazy-commit memory model, the circuit breaker, and what actually happens during restore.
A sandbox you aren't using doesn't keep its CPUs running, doesn't keep its memory resident, and eventually doesn't show up as a process on the host at all. It costs you nothing while it's idle. The next request wakes it.
That's the headline. The rest of this page is how it works, and what happens when it doesn't.
If you haven't read it yet, the [engine internals](/docs/under-the-hood/engine/) page is the right pre-read — this page leans on three concepts from it: vCPU pause/resume over the control socket, the memory checkpoint/restore bundle, and krucible's lazy-commit memory model. There's a one-paragraph recap of each below.
## Background, in three paragraphs
**vCPUs.** Each sandbox is a `bhatti-vmm` helper process — a cgo wrapper around krucible (libkrun) that *is* the VM. The daemon drives it out-of-band over a **control socket**: a one-line `PAUSE` freezes the guest's vCPUs, `RESUME` unfreezes them, `STATUS` reports state ([`pkg/engine/krucible/control.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/control.go)). "Paused" means the hypervisor stops running guest instructions; the helper process stays alive and the guest's memory stays mapped. The round-trip is single-digit milliseconds.
**Checkpoints.** A control-socket `SNAPSHOT <dir>` writes the VM's full memory image, its device/vCPU state, a copy of the qcow2 root overlay, the config drive, and any attached volumes into a self-contained bundle, plus a `manifest.json`. Restore is the inverse: re-launch a fresh helper pointed at the bundle, which loads memory and device state back and resumes from the checkpoint point. After a successful restore every guest process picks up exactly where it was, right down to in-flight TCP connections.
**The memory model.** krucible maps guest RAM `MAP_PRIVATE | MAP_ANONYMOUS` with lazy commit, so a VM's host memory footprint only ever counts the pages the guest has *actually touched* — not its configured ceiling. This is why a paused (warm) VM's resident memory naturally shrinks to its working set with no extra work, and why v2 needs none of v1's balloon-reclaim machinery ([`pkg/engine/krucible/thermal.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/thermal.go)).
## Design decisions on this page
1. **Three thermal states, not two.** Hot/warm/cold gives us a cheap mid-point (vCPUs paused but memory still resident) so the common case of "you come back five minutes later" doesn't pay the cost of a checkpoint restore. See [The three states](#the-three-states).
2. **All snapshots are `Full`.** v2 has no diff/incremental snapshot mode — every checkpoint writes every touched page. This is a deliberate correctness choice paid for by a real production incident. See [Why all snapshots are Full](#why-all-snapshots-are-full).
3. **No balloon needed on hot→warm.** Because krucible commits guest RAM lazily, a paused VM already only holds its touched pages resident — there's no separate step to "take memory back." See [The memory model: lazy commit](#the-memory-model-lazy-commit).
4. **Cold check uses host-side timing, not an agent query.** vCPUs are paused while warm — the agent can't respond. Querying it would either time out or wake the VM. So we use the timestamp recorded when we transitioned to warm. See [Why the cold check doesn't talk to the agent](#why-the-cold-check-doesnt-talk-to-the-agent).
5. **Circuit breaker on a stuck VM.** If the agent fails ten Activity queries in a row (\~100 s of silence), the thermal manager force-pauses the VM rather than leaving it hot and unresponsive. See [The circuit breaker](#the-circuit-breaker).
6. **Three retries before marking unknown.** Snapshot writes can fail on a transient I/O hiccup. We retry. Three failures in a row gets the sandbox marked `unknown` and surfaces an event. See [Snapshot retries](#snapshot-retries).
## The three states
```
idle 30 s idle 30 min
Hot ────────────────────► Warm ──────────────────────────► Cold
▲ │ │
│ │ │
│ warm wake ~4 ms │ cold wake (see below) │
└───────────────────────┴─────────────────────────────┘
any API request
```
\| State | `bhatti-vmm` process | vCPUs | Host RAM | Resume to hot | |---|---|---|---|---| | **Hot** | alive | running | working set | — | | **Warm** | alive | paused | working set (still resident) | \~4 ms | | **Cold** | dead | — | 0 (checkpoint on disk) | sub-second (see below) |
Warm resume is a single `RESUME` over the control socket — the helper is alive, the guest pages are still mapped in host RAM, no disk read is needed. It's \~4 ms.
Cold resume has its own breakdown because it spans more than one thing: spawn a fresh `bhatti-vmm` helper, read the memory image from disk, restore the VM, wait for the agent to respond. How much of that disk read hits the page cache vs. storage depends on how recently the sandbox went cold and what else is happening on the host. See [What "cold wake" actually costs](#what-cold-wake-actually-costs) below.
Slower hardware (a Raspberry Pi 5 with its capped NVMe) gets proportionally slower numbers across the board; the shape of the state machine is the same.
Transitions happen automatically. Every API call to a sandbox runs through the engine's wake path ([`Wake`/`EnsureHot`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/thermal.go)) first. If the sandbox is warm it resumes in milliseconds. If cold, it restores the checkpoint, then your operation runs. From an API client's perspective, every sandbox is always "running" — the resume is just the first slice of latency on the first call after it went cold.
### What "cold wake" actually costs
A cold restore re-launches the helper and reads the memory image back into a fresh VM. In practice this is **sub-second** — \~380 ms measured end-to-end on Apple Silicon (M-series, NVMe) for a small sandbox.
The dominant variable is the host page cache. Right after a checkpoint is written, the memory image is hot in the cache and the restore read is nearly free. Thirty minutes of memory pressure between checkpoint and resume is enough to evict most of those pages on a busy host, and then the read happens for real off disk. This is filesystem-agnostic — it's the OS page cache, not anything qcow2- or btrfs-specific. The wider context (base-image sharing, what's not yet optimized) is on the storage page: [Storage → Cold wake and the page cache](/docs/under-the-hood/storage/#cold-wake-and-the-page-cache).
You can opt out per sandbox with `bhatti create --keep-hot` on creation, or `bhatti edit <name> --keep-hot` later. A keep-hot sandbox skips thermal management entirely ([`pkg/server/sandbox_handlers.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/sandbox_handlers.go)) and holds its full working set resident. This is for sandboxes that maintain external connections — a Slack WebSocket, a Discord gateway, a long-running training job that you don't want randomly suspended.
## Hot → Warm
After 30 seconds with no API activity *and* no attached TTY sessions, the thermal manager pauses the VM ([`pkg/engine/krucible/thermal.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/thermal.go)):
1. `PAUSE` over the control socket. The vCPU pause is single-digit milliseconds.
2. Record the pause time as the sandbox's last-activity timestamp. The warm-to-cold timer starts from now, not from your last `exec`.
There's no balloon step. Because krucible commits guest RAM lazily, the paused VM's resident footprint is already just the pages the guest has touched — the host doesn't need to ask for memory back. The `bhatti-vmm` helper is still alive; the guest's network link (its virtio-net socket to the owner's `bhatti-netd` gateway) stays connected. The vCPUs are simply frozen. Resuming is another `RESUME` call.
The "no attached TTY sessions" check matters. If someone is in `bhatti shell` right now, we don't pause — that's a person staring at a terminal. The attached-session count comes from the agent's `Activity` response ([`cmd/lohar/handler.go`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/handler.go)), which is the only authoritative source.
## Warm → Cold
After 30 minutes warm, the manager takes a Full checkpoint to disk and kills the `bhatti-vmm` helper ([`pkg/engine/krucible/engine.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/engine.go)):
1. `PAUSE` over the control socket (a no-op if already paused).
2. `SNAPSHOT <bundle-dir>` — the helper streams the whole guest RAM, the device/vCPU state, a copy of the qcow2 root overlay, the config drive, and any attached volumes into the bundle, plus a `manifest.json`. The guest's page cache travels *inside* the memory image, so in-flight file writes are preserved across the round-trip without a separate flush. This is a generous-deadline operation (streaming a whole RAM image to disk).
3. If the snapshot write fails — say the disk is out of space for the memory image — the guest is still paused, so we `RESUME` it rather than leave it frozen (a frozen guest hangs the next `exec`), then surface the error. The sandbox stays usable.
4. On success, kill the helper. Its RAM is returned to the host, and the sandbox's on-disk footprint drops to the bundle.
5. Update the database — sandbox is now `stopped`, thermal is `cold`.
There are no host TAP devices or bridges to tear down — networking is handled by the per-owner `bhatti-netd` gateway, which is shared across the owner's sandboxes and outlives any single VM. See [Networking](/docs/under-the-hood/networking/).
### Why all snapshots are Full
A diff/incremental snapshot writes only the pages modified since the last one — for an idle VM that's a fraction of the memory and a fraction of the time. It's tempting. bhatti does **not** do it.
The reason is the rory incident (April 2026). A user's persistent sandbox came back after a restore with corrupted device state. The agent was unreachable, the VM was wedged, and we had to destroy it and lost the working state of the sandbox. The root cause was a diff snapshot whose dirty-page set was *incomplete*: the hypervisor's dirty-page tracking missed writes that happened in the VMM's own userspace device-model code, so restoring the diff loaded some pages from a stale base and clobbered in-flight device state. A diff snapshot is only ever as trustworthy as its dirty-page bitmap, and that bitmap had a blind spot.
The fix was to stop chasing diffs entirely. v2 has **no diff snapshot mode** — every checkpoint is a complete, self-consistent image of every touched page ([`pkg/engine/krucible/engine.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/engine.go), [`pkg/engine/krucible/snapshot.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/snapshot.go)). The bundle is also *verified* before the sandbox is marked stopped — a corrupt checkpoint is worse than no checkpoint.
It's slower to write than a diff would be. On any modern host disk that's fine, and the trade is correctness for speed — which I'll take any day after losing rory. The full audit is in the [decisions](/docs/under-the-hood/decisions/) page.
### Snapshot retries
Disks fail in transient ways: a brief I/O stall, a momentary contention spike. A snapshot that fails once usually succeeds on the second try. The thermal manager retries up to three times per cycle before marking the sandbox `unknown` ([`pkg/server/server.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go)):
```
attempt 1 fails → log warning, increment counter, try again next cycle
attempt 2 fails → log warning, increment counter, try again next cycle
attempt 3 fails → log error, mark sandbox unknown, record event
```
Each failure also records a `thermal.snapshot_failed` event with the attempt number and error, so an operator can see why (`bhatti admin events --type thermal.snapshot_failed`). The VM stays warm (alive, vCPUs paused) between retries — a failed snapshot never kills a live sandbox.
### Why the cold check doesn't talk to the agent
Look at [`runThermalCycle`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go) and you'll see the warm→cold transition computed without an agent query. That's deliberate.
Agents can't respond when vCPUs are paused. A query would either time out — silently skipping the cold check — or trigger a vCPU resume to service the request, which defeats the entire reason we paused. So the cold check uses the timestamp the manager set when it transitioned the sandbox to warm.
It's a small detail. It took an afternoon to figure out why my warm sandboxes never went cold.
## Cold → Hot
Any API call to a cold sandbox triggers the wake path, which calls `Start` to restore from the checkpoint bundle ([`pkg/engine/krucible/engine.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/engine.go)):
1. Validate the bundle. If it's present and well-formed, restore from it; if it's missing or corrupt (a crashed or never-snapshotted sandbox whose RAM is gone but whose root overlay persists), fall back to a fresh cold boot. Recovery relies on this fallback for restart-safety.
2. Ensure the owner's `bhatti-netd` gateway is listening on its net socket, spawning it if this is the owner's first live sandbox. Sibling sandboxes reuse the same gateway.
3. Remove any stale control/forward sockets from the dead incarnation, then spawn a fresh `bhatti-vmm` helper pointed at the bundle. The helper loads the memory image and device/vCPU state back and resumes the guest from the checkpoint point.
4. `WaitReady` — poll the guest agent over its vsock-backed control socket for up to 30 seconds. If lohar answers, the sandbox is hot. If it doesn't, the restore is treated as failed (see below).
The kernel and the guest's processes don't restart — they pick up exactly where they were, including open connections. A successful cold restore is sub-second (\~380 ms on Apple Silicon); most of the cost on a cold cache is the memory-image read. See [the cold-wake breakdown above](#what-cold-wake-actually-costs) and [Storage → Cold wake and the page cache](/docs/under-the-hood/storage/#cold-wake-and-the-page-cache) for the full picture.
### When restore fails
If the helper can't load the bundle, or the agent doesn't respond within 30 seconds, the restore is aborted: the half-spawned helper is killed and the sandbox surfaces an error rather than sitting in a half-restored state ([`pkg/engine/krucible/engine.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/engine.go)).
```
Error: sandbox "dev" failed to restore: <reason> —
use 'bhatti start --force' to retry or destroy and recreate
(volume data is safe).
```
`bhatti start --force <name>` retries the restore. If the underlying issue (disk full, a truncated bundle) is fixed, this gets you back to running. If not, it fails again.
Volume data lives on separate block images attached as virtio-blk drives; they survive even when the memory checkpoint doesn't. So the fix is usually: detach volumes, destroy the sandbox, recreate, reattach.
## The memory model: lazy commit
This is the part of thermal management most other systems bolt on a device for, and krucible gets for free. In v1 (Firecracker), a paused VM still held all of its configured RAM as host memory, so pausing alone reclaimed nothing — you had to inflate a virtio-balloon to hand pages back to the host, and deflate it on resume.
krucible doesn't have that problem. It maps the guest's RAM `MAP_PRIVATE | MAP_ANONYMOUS`, which the kernel commits *lazily* — a guest page only consumes host physical memory once the guest actually writes to it. A 1 GB VM whose working set is 200 MB only ever costs the host \~200 MB, hot or warm. Pausing the vCPUs freezes that working set in place; nothing needs to be "taken back."
Concretely, the balloon interface is a no-op on krucible — it's kept on the engine interface only for compatibility, and calling it does nothing ([`pkg/engine/krucible/thermal.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/engine/krucible/thermal.go)):
```go
// BalloonSet is a no-op on krucible: libkrun maps guest RAM
// MAP_PRIVATE|MAP_ANONYMOUS (lazy commit), so a paused VM's host RSS
// already only counts touched pages.
func (e *Engine) BalloonSet(ctx context.Context, id string, amountMiB int64) error {
return nil
}
```
For a self-hosted box with twenty sandboxes, this is the difference between "I can run twenty VMs" and "I can run many more" — density falls out of the memory model rather than a reclaim trick, and there's nothing to configure or tune.
## The host-side activity cache
Every API call records `time.Now()` for the sandbox in the server's activity cache ([`pkg/server/server.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go)). The thermal cycle checks this before querying the agent at all. If the cached timestamp is within the warm timeout, the agent query is skipped.
Why bother? Because with 50 active sandboxes the cycle runs every \~10 seconds, and naively that's 50 agent round-trips every cycle — most of them returning "still active." The cache eliminates the queries for sandboxes that have had recent API traffic. Only genuinely-idle sandboxes get queried over vsock.
The activity cache is a heuristic, not the source of truth. The agent's own `lastActivity` (updated by every exec and stdin) is authoritative — but querying it costs a round-trip. So we use the cache as a fast-path negative ("definitely active, skip") and fall through to the agent for everything else.
## The circuit breaker
If the agent fails to respond ten times in a row to Activity queries — about 100 seconds of silence — the manager force-pauses the VM ([`pkg/server/server.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/server/server.go)):
```go
const maxThermalFailures = 10
```
Pausing is a control-socket `PAUSE`; it doesn't need the agent. This catches the "agent is alive but stuck" case and stops a wedged VM from burning a vCPU indefinitely. A user sees their sandbox go to `warm` even though they never asked for it; their next request wakes it normally. If the agent is genuinely dead, the next restore will fail and they'll get the restore-failed error described above. keep\_hot sandboxes are exempt.
The threshold of 10 is a heuristic:
* \~10-second tick interval × 10 failures = \~100 seconds of unresponsiveness
* A normally-loaded agent answers Activity in single-digit milliseconds
* A real network or scheduling hiccup might lose 1–2 queries; not 10
If you want to monitor this, watch for `thermal.force_pause` events.
## What you'll see in practice
A short field guide.
**`bhatti list` shows `cold` next to your sandbox.** That's normal. It's saving you memory. Run any command on it and it'll wake.
**First request after a long pause is slow.** That's a cold wake. Subsequent requests are normal speed.
**Sandbox transitioned to warm even though I'm using it.** The check requires no attached TTY *and* >30 s since the last API call. Background processes inside the VM don't count as activity from the manager's perspective. If you have a build running and want it to stay hot, set `--keep-hot`.
**`bhatti list` shows `unknown`.** Three snapshot retries failed. Look at the bhatti server logs and `bhatti admin events --type thermal.snapshot_failed`. You can usually `bhatti start --force` once the underlying issue (disk full, truncated bundle) is fixed.
**Resume fails with "failed to restore".** Volume data is safe; destroy and recreate. If this happens twice in a row on the same VM, file an issue — we'd like to see it.
## Where to go next
* [How a sandbox boots](/docs/under-the-hood/architecture/) — the cold start path in detail
* [Networking](/docs/under-the-hood/networking/) — the per-owner `bhatti-netd` gVisor gateway, and why open connections survive a checkpoint/restore
* [Storage](/docs/under-the-hood/storage/) — the qcow2 overlays and the checkpoint bundle on disk
* [Lohar](/docs/under-the-hood/lohar-the-blacksmith/) — the agent that resumes inside the VM
* [Decisions & learnings](/docs/under-the-hood/decisions/) — the rory incident in full, plus other paid-for knowledge
# The wire protocol
> Binary framing between host and guest — frame format, frame types, atomic writes, kill semantics, and the choices that shaped it.
All communication between the bhatti host process and a guest agent happens over a binary framing protocol. In production it runs over vsock — libkrun bridges each guest vsock port to a host Unix socket, and the daemon dials that socket directly. The protocol itself is engine- and transport-independent: the entire agent test suite runs on macOS over `net.Pipe()` (or plain Unix sockets) without any VM or root.
## Design decisions on this page
1. **Custom binary framing, not gRPC or HTTP.** The framing layer is \~130 lines of Go. It runs inside a microVM where every dependency is dead weight. gRPC would add protobuf codegen, a runtime, and complexity disproportionate to a protocol with eight frame types. See [Why not HTTP or gRPC](#why-not-http-or-grpc).
2. **`WriteFrame` does one `Write()` call.** The entire frame is assembled into a single buffer before going to the wire. Otherwise, two goroutines writing concurrent stdout/stderr chunks would interleave at byte boundaries and corrupt frames. See [Atomic frame writes](#atomic-frame-writes).
3. **AUTH must be the first frame.** If a token is configured, lohar refuses any other frame on a fresh connection until it sees an AUTH frame within 5 seconds. See [Auth](#auth).
4. **Unknown frame types are skipped, not errored.** A new frame type can be added without breaking older clients. See [Forward compatibility](#forward-compatibility).
5. **One connection per operation.** Most agent calls open a connection, do their thing, close. TTY sessions and forwards are the exceptions. See [Connection model](#connection-model).
## Frame format
```
┌────────────────┬───────────┬──────────────────────┐
│ Length (4B BE) │ Type (1B) │ Payload (N bytes) │
└────────────────┴───────────┴──────────────────────┘
```
**Length** is a 4-byte big-endian unsigned integer. It equals `1 + len(Payload)` — the type byte plus the payload. It does *not* include the 4-byte length prefix itself.
**Type** is a single byte identifying the frame kind.
**Payload** is variable-length, up to `MaxFrameSize - 1` bytes.
**Maximum frame size** is 1 MB ([`pkg/agent/proto/constants.go:69`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/constants.go#L69)). Both `WriteFrame` and `ReadFrame` enforce this — oversized frames are rejected, not truncated.
The encoding is straightforward ([`pkg/agent/proto/frame.go:10-28`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/frame.go#L10-L28)):
```go
buf := make([]byte, 4+frameLen)
binary.BigEndian.PutUint32(buf[0:4], uint32(frameLen))
buf[4] = msgType
copy(buf[5:], payload)
_, err := w.Write(buf)
```
## Atomic frame writes
The single-buffer approach above isn't an optimization — it's a correctness requirement. The agent's piped exec has stdout and stderr goroutines writing to the same connection at the same time. If `WriteFrame` did three `Write()` calls (length, type, payload), two goroutines could interleave at any byte boundary. The kernel would deliver something like:
```
[len=1024][type=STDOUT] ← goroutine A starts
[len=512][type=STDERR] ← goroutine B writes its full frame
[1024 bytes of stdout] ← goroutine A finishes, but the order on the wire is now broken
```
The receiver tries to parse `len=1024 type=STDOUT` plus the next 1024 bytes — but the next bytes are goroutine B's length prefix, not goroutine A's stdout. Frames are corrupt for the rest of the connection.
By assembling the whole frame into one buffer and calling `Write()` once, the kernel atomically delivers the whole frame to the socket buffer (for sizes under the kernel's pipe-buffer limit, which is much larger than 1 MB on every platform we run on). Concurrent goroutines get their frames serialized end-to-end, never interleaved.
This is why the [piped exec](/docs/under-the-hood/lohar-the-blacksmith/#piped-exec-the-simpler-path) goroutines push frames through a single channel to a single writer goroutine — defense in depth on top of `WriteFrame`'s atomicity. Both levels matter.
## Frame types
All frame type constants are in [`pkg/agent/proto/constants.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/constants.go).
### I/O streams
\| Type | Byte | Direction | Payload | |---|---|---|---| | `STDIN` | `0x01` | host → guest | raw bytes for child's stdin | | `STDOUT` | `0x02` | guest → host | child's stdout bytes | | `STDERR` | `0x03` | guest → host | child's stderr bytes |
### Control
\| Type | Byte | Direction | Payload | |---|---|---|---| | `RESIZE` | `0x04` | host → guest | `[u16 rows BE][u16 cols BE]` (4 bytes exactly) | | `EXIT` | `0x05` | guest → host | `[i32 exit_code BE]` (4 bytes exactly) | | `ERROR` | `0x06` | either | UTF-8 error message (variable length) | | `KILL` | `0x07` | host → guest | empty payload |
### Exec
\| Type | Byte | Direction | Payload | |---|---|---|---| | `EXEC_REQ` | `0x10` | host → guest | JSON-encoded `ExecRequest` |
The `ExecRequest` shape is in [`pkg/agent/proto/messages.go:5-19`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/messages.go#L5-L19): `argv`, `env`, `tty`, `rows`, `cols`, `cwd`, `session_id`, `max_idle_sec`, `if_detached`, `detach`, `output_file`, `session`. Most are optional pointers — nil means "use default."
### Auth
\| Type | Byte | Direction | Payload | |---|---|---|---| | `AUTH` | `0x11` | host → guest | raw token bytes |
If a token is configured (see [Auth](#auth) below), this must be the first frame on every connection.
### Port forwarding
\| Type | Byte | Direction | Payload | |---|---|---|---| | `FWD_REQ` | `0x20` | host → guest | JSON `{"port": 8080}` | | `FWD_RESP` | `0x21` | guest → host | JSON `{"status": "ok"}` or `{"status":"error","message":"..."}` |
### Sessions
\| Type | Byte | Direction | Payload | |---|---|---|---| | `EXEC_LIST_REQ` | `0x30` | host → guest | empty | | `EXEC_LIST_RESP` | `0x31` | guest → host | JSON `[]SessionInfo` | | `EXEC_KILL` | `0x32` | host → guest | JSON `{"session_id": "..."}` | | `SESSION_INFO` | `0x33` | guest → host | JSON `SessionInfo` (sent on create or attach, before STDOUT) |
### Activity
\| Type | Byte | Direction | Payload | |---|---|---|---| | `ACTIVITY_REQ` | `0x40` | host → guest | empty | | `ACTIVITY_RESP` | `0x41` | guest → host | JSON `ActivityInfo` (last activity timestamp, attached count) |
### File operations
\| Type | Byte | Direction | Payload | |---|---|---|---| | `FILE_READ_REQ` | `0x50` | host → guest | JSON `{"path":"...","offset":1,"limit":2000,"max_bytes":51200}` | | `FILE_READ_RESP` | `0x51` | guest → host | JSON `{"size":1234,"mode":"0644"}` | | `FILE_WRITE_REQ` | `0x52` | host → guest | JSON `{"path":"...","mode":"0644","size":1234}` | | `FILE_WRITE_RESP` | `0x53` | guest → host | JSON `{"status":"ok"}` | | `FILE_STAT_REQ` | `0x54` | host → guest | JSON `{"path":"..."}` | | `FILE_STAT_RESP` | `0x55` | guest → host | JSON `FileInfo` | | `FILE_LS_REQ` | `0x56` | host → guest | JSON `{"path":"..."}` | | `FILE_LS_RESP` | `0x57` | guest → host | JSON `[]FileInfo` |
### Systemctl IPC (in-guest, not host↔guest)
\| Type | Byte | Direction | Payload | |---|---|---|---| | `SYSTEMCTL_REQ` | `0x60` | client → lohar (Unix socket) | JSON `SystemctlRequest` | | `SYSTEMCTL_RESP` | `0x61` | lohar → client | JSON `SystemctlResponse` |
These are spoken over `/run/bhatti/systemctl.sock` *inside* the guest, not over the host↔guest vsock connection. They exist so the systemctl shim's user-facing invocation can ask PID 1 lohar to perform privileged operations — see [Lohar's systemctl IPC](/docs/under-the-hood/lohar-the-blacksmith/#privileged-systemctl-from-a-non-root-caller).
The trust boundary is `SO_PEERCRED` on the socket, **not** anything in the request payload. A non-root caller could otherwise forge a UID claim and get a privileged operation done.
## Connection model
Two vsock ports inside each VM, two purposes:
* **Port 1024** (control) — exec, sessions, files, activity queries
* **Port 1025** (forward) — port forwarding / TCP tunneling
`MaxFrameSize` is 1 MB. lohar listens on these vsock ports; libkrun bridges each to a host Unix socket, and the daemon dials that socket directly — no CONNECT handshake, no TCP, no TAP device.
### Control connection lifecycle
One connection per operation. The host dials port 1024, optionally sends an `AUTH` frame, sends exactly one request frame, reads responses until the operation completes, then the connection closes.
```
Host Lohar
│ │
├──vsock connect :1024 ──────────────►│
├──AUTH frame (if token configured) ──►│
├──EXEC_REQ frame ───────────────────►│
│ ├──fork/exec child
│◄──STDOUT frame──────────────────────┤
│◄──STDOUT frame──────────────────────┤
│◄──STDERR frame──────────────────────┤
│◄──EXIT frame────────────────────────┤
└──connection closed──────────────────┘
```
Exception: TTY sessions keep the connection open for bidirectional I/O. The host sends `STDIN` and `RESIZE` frames; the guest sends `STDOUT` frames and eventually an `EXIT` frame. If the host disconnects, the session detaches (process keeps running, scrollback buffer captures output — see [Sessions](/docs/under-the-hood/lohar-the-blacksmith/#sessions)).
### Forward connection lifecycle
One connection per tunnel. After the `FWD_REQ`/`FWD_RESP` handshake, the framing protocol is **abandoned** — the connection becomes a raw bidirectional byte relay.
```
Host Lohar Target (localhost:8080)
│ │ │
├──vsock connect :1025 ──────────────►│ │
├──AUTH frame ────────────────────────►│ │
├──FWD_REQ {"port": 8080} ───────────►│ │
│ ├──TCP connect :8080 ─────►│
│◄──FWD_RESP {"status": "ok"} ────────┤ │
│ │ │
│═══ raw bytes (no framing) ══════════►│═════════════════════════►│
│◄═════════════════════════════════════│◄═════════════════════════│
```
This is what `bhatti exec` doesn't use, but the reverse proxy does. When you hit a published URL, the daemon picks up an HTTP request, opens a forward connection to the agent, sends `FWD_REQ` for the target port, and then pipes the request body through. The agent relays it to localhost:port inside the VM, and the response comes back the same way.
Forward connections don't have framing because they're proxying arbitrary TCP — HTTP, WebSocket, SSH, whatever. Adding framing would mean parsing HTTP on the agent side, which we don't want to do.
## Auth
If a token is configured (via the [config drive](/docs/under-the-hood/lohar-the-blacksmith/#boot-sequence) at boot), the first frame on every connection must be `AUTH` with the token as payload. Lohar validates it within a 5-second deadline. Invalid or missing auth gets an `ERROR` frame and the connection is closed.
The token is generated per-sandbox during `Create()` — 16 random bytes, hex-encoded — and injected into the VM via the config drive. It's then stored in the host's `AgentClient` and not used for anything else. If the daemon restarts and re-reads the agent token from the sandbox's persisted state, the token survives — VMs across daemon restarts use the same token they were created with.
The token comparison uses [`subtle.ConstantTimeCompare`](https://pkg.go.dev/crypto/subtle#ConstantTimeCompare) ([`cmd/lohar/handler.go:69`](https://github.com/sahil-shubham/bhatti/blob/main/cmd/lohar/handler.go#L69)) so a network observer can't time the comparison to figure out a prefix of the token.
## File read protocol
File reads support server-side truncation to avoid transferring large files when the consumer only needs the first N lines. This matters because coding agents always truncate file reads — typically 2000 lines or 50 KB — and shipping a 100 MB log over the wire just to discard 99.95% of it is a waste.
```
host guest
FILE_READ_REQ
{"path":"/app.log","offset":1,"limit":2000,"max_bytes":51200}
│
▼
FILE_READ_RESP
{"size":10485760,"mode":"0644"} ← total file size
│
▼
STDOUT (line data)
STDOUT (line data)
... ← stops when limit or max_bytes hit
│
▼
EXIT code=0
```
* `offset` — 1-indexed line number to start from (0 or absent = beginning).
* `limit` — maximum lines to return (0 = unlimited).
* `max_bytes` — maximum bytes to return (0 = unlimited).
Whichever limit hits first stops the read. Without any truncation parameters the full file is streamed. The `FILE_READ_RESP` always contains the *total* file size so the consumer knows whether content was truncated.
Directories and non-regular files are rejected with an `ERROR` frame. File reads are cancellable via context — closing the connection gives lohar a broken pipe, stopping the transfer immediately.
## File write protocol
Writes are atomic. Lohar writes to a temp file, fsyncs, then renames over the target. Concurrent readers see either the old content or the new content, never a half-written state.
```
host guest
FILE_WRITE_REQ
{"path":"/workspace/app.js","mode":"0644","size":1234}
│
STDIN (content bytes) │
STDIN (content bytes) │
... │ ← until size bytes sent
│
▼
FILE_WRITE_RESP
{"status":"ok"}
```
Negative sizes are rejected (prevents silent data loss from a missing `Content-Length` on the HTTP side that would otherwise pass through as a default zero). If the connection drops mid-write, the temp file is cleaned up.
Why fsync before rename? See [the explanation in lohar.md](/docs/under-the-hood/lohar-the-blacksmith/#file-operations) — short version: the host might snapshot the VM the next millisecond, and dirty pages in the guest's page cache aren't captured by a snapshot until they're flushed. Without fsync, the rename is metadata-durable but the data isn't on the virtio-blk device.
## Kill semantics
\| Context | Signal | Why | |---|---|---| | **Piped exec** (non-TTY) | `SIGKILL` to process group | Agents need instant, reliable abort. Child processes (npm → node) must die immediately. | | **TTY session disconnect** | None (detach) | Process keeps running, session detaches. Scrollback captures output. | | **TTY session KILL frame** | `SIGTERM` to process group | Allows graceful shutdown. If the process handles SIGTERM and survives, the session remains reattachable. | | **EXEC\_KILL API** | `SIGKILL` to process group | Explicit force-kill by session ID. | | **Idle timer expiry** | `SIGKILL` to process group | Session is abandoned, no observer. |
All kill operations target the *process group* (negative PID), not just the session leader. This requires `Setpgid: true` on the `SysProcAttr` so child processes are in the same group. Without this, `npm install` would survive killing the shell that launched it.
## Forward compatibility
`ReadFrame` in the client skips unknown frame types rather than erroring. This allows the protocol to be extended without breaking existing clients — a new frame type added to lohar won't crash an older bhatti host.
The flip side: a new client sending a frame an older agent doesn't understand will get its connection closed (the agent ignores the frame, but no response comes back). So forward compatibility is asymmetric — host upgrades cleanly, but the agent has to be at least as new as the most-recent frame type the host wants to use.
In practice the only frame types added since v1.0 are `SYSTEMCTL_REQ`/`RESP` (in-guest, not host↔guest) and the file ops expanding from read/write to include stat/ls. Both are guarded by capability checks before use.
## Why not HTTP or gRPC
The framing layer is \~130 lines of Go ([`pkg/agent/proto/frame.go`](https://github.com/sahil-shubham/bhatti/blob/main/pkg/agent/proto/frame.go)) and handles concurrent stdout/stderr multiplexing, binary file transfers, and terminal I/O with zero dependencies.
gRPC would mean:
* protobuf code generation in the build pipeline
* a runtime dependency inside the VM (heavy for an agent that's supposed to be small and static)
* complexity disproportionate to a protocol with eight frame types
* streaming RPCs would handle the stdout/stderr case but with more moving parts than the current channel-serialized writer
HTTP would mean:
* header parsing on every exec
* chunked encoding negotiation
* content-type negotiation
* a connection that's already authenticated and never leaves the host paying the cost of HTTP semantics it doesn't need
The framing protocol is what we'd write if we sat down to design a host↔guest protocol from scratch. Length, type, payload. Atomic write of the whole thing. That's it.
## Where to go next
* [Lohar](/docs/under-the-hood/lohar-the-blacksmith/) — the agent that speaks this protocol on the guest side
* [Architecture](/docs/under-the-hood/architecture/) — the bigger picture of host ↔ guest communication
* [Decisions & learnings](/docs/under-the-hood/decisions/) — the snapshot and networking bugs that shaped these choices]]></plaintext></data_dir></id></arch></tier></data_dir></your-server-ip></your-server-ip></x.y.z></arch></version></arch></version></name></name></name></execstart></name></arch></name></user_id></user_id></cow></id></arch></tier></arch></ver></arch></ver></api_host></proxy_zone></proxy_zone></alias></your-domain></alias></your-domain></data_dir></data_dir></data_dir></name></name></id></host></sandbox-name></port></port></id></base64></your-domain></alias></your-domain></alias></host></port></id></id></you></span>