Skip to content
[bhatti]

Networking: bhatti-netd, gVisor, and the vsock tunnel

Every VM gets internet access, isolation from the host, and reachability to its owner's other sandboxes — and none of it touches the host's own network stack. There are no TAP devices, no bridges, no iptables rules. Each owner's sandboxes hang off a single userspace process, bhatti-netd, that is their network: a gVisor netstack running as an ordinary host process, on by default.

This page is about how that gateway works, how a guest's packets get in and out, and the paths for reaching a service inside a sandbox from outside.

Design decisions on this page

  1. A userspace gateway per owner, not host bridges. Each owner's VMs share one bhatti-netd; different owners get different gateways and never share a network. See A gateway per owner.
  2. The guest link is virtio-net over a unix socket, not a TAP. libkrun connects, netd listens; the guest sees a normal eth0. See The guest link.
  3. Egress is policed and the host is unreachable by construction. Every guest flow is terminated at the gateway and re-originated through a vetting dialer. See Egress policy and host isolation.
  4. Same-owner siblings reach each other through the gateway, never directly. An L3-routed proxy inside netd carries the sibling leg. See Sibling reachability.
  5. Host↔guest ports ride the vsock tunnel. bhatti forward and the proxies never expose the guest on a host-routable address. See Reaching services inside sandboxes.
  6. The legacy TSI backend is off by default. It shares the host netstack — simpler, but the guest can reach host loopback. See The legacy TSI backend.

A gateway per owner

When a user creates their first sandbox, bhatti spawns them a bhatti-netd — one gVisor userspace netstack that serves all of that owner's sandboxes. The second user gets their own. Each owner is alone on their own network; there's no shared segment to ARP-scan.

The addressing (pkg/engine/krucible/engine.go): each owner's gateway lives on 100.64.<idx>.0/24 — RFC-6598 carrier-grade-NAT space, used here for per-owner vnets. The gateway is .1; guests are .2, .3, … in creation order.

| Owner index | Gateway | Guest 1 | Guest 2 | |---|---|---|---| | 0 | 100.64.0.1 | 100.64.0.2 | 100.64.0.3 | | 1 | 100.64.1.1 | 100.64.1.2 | 100.64.1.3 | | ... | ... | ... | ... |

Grouping is by owner (the sandbox's UserID): same owner ⇒ same netd ⇒ siblings reach each other. A sandbox with no owner gets its own isolated netd keyed by sandbox id, so the single-sandbox case is just N = 1.

netd is spawned detached, so it survives a daemon restart, and its state (socket path, next-guest index, pid) is persisted in a netd.json next to its socket. On restart the daemon re-adopts the running gateway by pid instead of respawning onto a socket it still holds; it's reference-counted, so the owner's last destroy tears it down (recovery.go).

The Firecracker-era per-user subnet index still exists in the store as subnetIdx, and it picks the 100.64.<idx> octet — but there are no per-user bridges behind it anymore. It's a vnet number, not a Linux bridge.

There are no TAP devices. The guest's NIC is a virtio-net device wired to a unix stream socket (krun_add_net_unixstream). netd listens on that socket; libkrun's virtio-net backend connects to it as the VM boots (cmd/bhatti-netd/main.go). Each accepted connection becomes a switch port on the gateway; the frames on it are the raw ethernet the guest emits, carried over the socket by pkg/gateway.FrameConn.

From the guest's side it's an ordinary eth0. lohar brings it up from the per-sandbox config drive via rtnetlink directly — no ip binary, no DHCP, no kernel IP autoconfig (our lean kernels build without CONFIG_IP_PNP) (cmd/lohar/netlink.go). It assigns the guest its 100.64.<idx>.<n>/24, adds an on-link route to the gateway .1, and a default route via .1. So the guest sends everything — internet and siblings alike — to the gateway.

Each guest gets a deterministic locally-administered MAC (52:54:00:00:00:<n>); the gateway's is 52:54:00:00:00:01. netd learns each guest's MAC → port as frames arrive, so it can demux stack-originated frames back to the right guest and flood only genuine broadcasts (cmd/bhatti-netd/netstack.go).

The offloaded-checksum gotcha

Guests offload their TX checksums (partial / pseudo-header only), and libkrun strips the virtio_net_hdr flag that would tell us so — so the checksums that reach netd on the wire aren't final. If gVisor verified them it would drop every guest packet and egress would silently time out.

We trust frames arriving over the local vsock link, so the gateway advertises CapabilityRXChecksumOffload: gVisor marks received packets checksum-validated and skips verification. We deliberately do not advertise TX offload — the stack computes real checksums on the frames it sends to guests (including a re-originated sibling leg), so there are no manual fixups. There's a regression test that fires a SYN with a deliberately-wrong checksum and asserts it still reaches the forwarder (delivery_test.go). This is the kind of thing that's invisible when it works and takes a day to find when it doesn't.

Egress policy and host isolation

netd doesn't route packets; it terminates every guest TCP flow at a gVisor TCP forwarder and re-originates it (cmd/bhatti-netd/forward.go). For an internet destination, the new leg is dialed through a vetting dialer — the egress guard (pkg/gateway) — which classifies the destination:

  • Public internet — allowed. This is the default posture (PosturePublic).
  • Private ranges (RFC-1918, ULA, CGNAT) — denied unless an explicit allow-CIDR opts them back in.
  • Loopback, link-local, the host itself, multicast, and cloud metadata (169.254.169.254) — never allowable, including the IPv4-mapped and NAT64-embedded spellings of those addresses that a naive check would miss.

A denied or unreachable destination gets a clean RST back to the guest rather than a hang (the forwarder dials first, then completes). And because the guest has its own loopback inside the VM and its only route out is the gateway, there is simply no path from the guest to the host's bhatti API, SSH, or anything else on the host — the isolation the old iptables INPUT rules used to buy is now a property of the topology. The integration suite asserts both halves: a guest dial to an RFC-1918 address is refused, and public egress works end to end (net_test.go).

This is a feature you don't configure. It's true the moment the gateway starts, for one sandbox or ten thousand. Because every flow — egress and sibling alike — passes through this one policed, observable chokepoint, it's also the natural home for per-sandbox egress policy and L7 secret substitution.

Sibling reachability

Same-owner sandboxes can reach each other — an agent sandbox can curl a worker at 100.64.<idx>.3:<port>, or by name. But they never talk directly. When the forwarder sees a destination inside the owner's own subnet, it dials that sibling through the gVisor stack itself, which routes the new leg out to the target guest's link (forward.go).

So the sibling leg passes through the same chokepoint as egress; guests can't reach each other behind netd's back, checksums are native on the re-originated leg, and it's all observable. A sandbox belonging to a different owner is on a different netd entirely and has no route to it — cross-owner traffic never shares a network (the sibling test asserts both the reachable and the isolated direction).

Names: lohar points /etc/resolv.conf at the gateway's in-cluster resolver when it's available — that resolver answers sibling sandbox names (<sandbox>, <sandbox>.sb) and forwards everything else upstream, so it's the only nameserver the guest needs. If it isn't reachable, lohar falls back to public resolvers (Cloudflare, then Google) directly (cmd/lohar/main.go). We never list the internal and public resolvers together: a stale sibling name would otherwise fall through to public DNS and resolve to the wrong thing.

Reaching services inside sandboxes

Nothing outside reaches a guest by its address — 100.64.<idx>.<n> only exists inside its owner's gateway. Every path in goes through the host, over the engine's vsock tunnel to localhost:<port> inside the VM. There are three.

bhatti forward (host↔guest)

bhatti forward <sandbox> <guest-port> [host-port]

The daemon binds 127.0.0.1:<host-port> on the host and bridges each accepted TCP connection to localhost:<guest-port> inside the VM over the vsock Tunnel primitive (pkg/forward). It's raw bytes, so any TCP service works — HTTP, Postgres, Redis — unlike the HTTP-aware public proxy. The forward is torn down when the command exits (Ctrl-C) or the sandbox is destroyed. This same primitive is the building block for the server-brokered inter-sandbox mesh: a sandbox can get a stable host endpoint that other sandboxes reach via the host.

Authenticated proxy

GET /sandboxes/:id/proxy/:port/<path>
Authorization: Bearer <token>

The daemon authenticates, looks up the sandbox, opens a tunnel to localhost:<port> inside the VM, and proxies HTTP/WebSocket traffic. Cold sandboxes wake on the first request. Useful for development — you don't need to publish a port to use one.

Public proxy (published URLs)

ANY https://<alias>.bhatti.sh

When you bhatti publish dev -p 3000 -a my-app, the daemon creates a publish rule mapping my-app → (sandbox=dev, port=3000). Public requests hit :443, get routed by Host header, looked up against the rule cache (pkg/server/public_proxy.go), and tunneled into the VM.

The public proxy has a few extra moves over the authenticated one:

  • In-memory route cache (LRU, 10 K entries) so a hot URL doesn't query SQLite per request.
  • singleflight.Group for resume coalescing — wake-then-serve: if 50 concurrent requests hit a cold sandbox, only one wake actually happens; the others wait on it. Without this you'd get 50 simultaneous snapshot loads.
  • Per-alias and global rate limiting — protects against abuse.
  • 5-minute per-request deadline, 50 MB body limit.

Both proxies use httputil.ReverseProxy with a custom transport that treats the engine's tunnel as an http.RoundTripper. WebSocket connections are hijacked and relayed bidirectionally with a 10-minute idle timeout.

Snapshot, restore, and recovery

Because netd is a separate, detached process and the guest link is a unix socket, snapshot/restore doesn't have to preserve any host-side network device — there are no TAPs or bridges to leak or rebuild. When a stopped sandbox is restored, its virtio-net backend simply reconnects to its owner's bhatti-netd, which is still listening (netd outlives individual VMs).

If the daemon itself restarts, it re-adopts each still-running gateway by pid from that gateway's netd.json record rather than respawning onto the socket it holds, and reference-counts it so the owner's last destroy still shuts it down (recovery.go). There's no system-wide network cleanup step on startup — there's nothing on the host to clean up.

The legacy TSI backend

netd is on by default (krucible_net_backend: true). Setting it to false selects libkrun's built-in TSI (transparent socket impersonation) backend instead: the guest has no separate netstack, and its sockets are impersonated directly on the host's network stack.

It's simpler and needs no gateway process, but it means the guest shares the host's netstack — it can reach the host's loopback — so it gets neither the host isolation nor the policed egress that netd provides. It's there for debugging and legacy configs; leave the gateway on. See Configuration → krucible_net_backend.

Where to go next

  • Thermal states — how the network stays consistent across snapshot/resume
  • Lohar — the guest agent, which talks to the host over vsock (a separate channel from this data-plane network)
  • Custom domain — TLS for the public proxy, wildcard certs, ACME flow
  • Configuration — the krucible_net_backend and runtime-path fields