Skip to content

Networking: bhatti-netd, gVisor, and the vsock tunnel

A newly created sandbox has no network device. Shell, exec, files, and published ports still work: they use the guest's separate vsock control/forwarding path. If the workload needs a network, choose one explicitly with create --net, --egress deny plus allow rules, or --allow-siblings. The server's default_egress is none unless configured otherwise. See Network & egress for recipes.

Design decisions on this page

  1. No NIC unless needed. none means there is no guest network link and no gateway for that sandbox. See A gateway per owner.
  2. A userspace gateway, not host bridges. Networked VMs of one owner share bhatti-netd, a gVisor netstack; different owners never share a gateway. See The guest link.
  3. Each sandbox has its own policy. deny allows only explicit rules; public allows public egress. Siblings require their own opt-in. See Egress policy and host isolation.
  4. Host↔guest ports use vsock. Publishing or proxying a port doesn't give a guest internet access. See Reaching services inside sandboxes.
  5. Credentials are substituted at the gateway. Host-scoped grants put placeholders in the VM, not the real secret. See Secret grants.

A gateway per owner

When an owner first uses networking, bhatti starts one bhatti-netd for that owner's networked sandboxes. A sandbox without a NIC does not need netd. The gateways are separate processes and networks; owners do not share an L2 segment. Same-owner siblings also cannot connect by default. The sending sandbox must be created with --allow-siblings (and the receiving sandbox needs a NIC). A sandbox with no owner gets an isolated gateway keyed to its sandbox ID.

Networked guests use RFC-6598 addresses 100.64.<idx>.0/24: the gateway is .1, and guests are assigned .2, .3, … . This is an address inside netd, not a host-routable IP. The per-owner index surviving from v1 is a vnet number, not a Linux bridge. Gateway lifecycle/recovery is in pkg/engine/krucible/recovery.go: the detached process may be re-adopted after a daemon restart and is torn down after its last networked sandbox is gone.

A networked VM sees eth0, a virtio-net device linked over a Unix stream socket to netd's gVisor netstack. There is no host TAP, bridge, or iptables fabric. The VM's agent, lohar, configures the guest address, gateway and routes with rtnetlink; it does not require DHCP. With none, the VM has no virtio-net link or eth0 at all. See cmd/lohar/netlink.go and cmd/bhatti-netd/netstack.go.

libkrun strips the virtio header indicating a partially offloaded guest TX checksum. netd advertises RX checksum offload so gVisor accepts these frames, but computes real checksums for traffic sent to guests. More importantly for isolation, the link is authenticated and netd drops frames whose source IP, MAC, or ARP sender is not the one assigned to that VM. The daemon registers a sandbox's policy before it launches its NIC; missing or malformed policies fail closed rather than retaining a prior grant.

Egress policy and host isolation

netd terminates guest TCP and re-originates it via a policy-checked dialer; UDP is checked too. A connection that fails policy is not forwarded. Each sandbox's policy has three possible default postures:

| Posture | NIC | What can leave the sandbox | | --- | --- | --- | | none | No | Nothing over guest networking (fresh-install default). | | deny | Yes | Only destinations named by --allow-host or --allow-cidr; siblings still need --allow-siblings. | | public | Yes | Public internet; private ranges and siblings still denied unless separately allowed. --net is shorthand for this posture. |

With --allow-host/--allow-cidr/--allow-siblings but no explicit posture, the server resolves the policy to deny. --allow-host api.example.com matches an exact public host; --allow-host '*.example.com' matches subdomains, not the bare domain. DNS-aware host rules validate the actual resolved address, so a public name resolving to a private IP is rejected. --allow-cidr can opt into otherwise private ranges (RFC-1918/ULA/CGNAT), not hard-denied targets. Host addresses are denied except for the exact address/port pairs of configured daemon TCP service listeners when the remaining policy permits them. A permitted public API/proxy listener can be reached under public egress; a private daemon-service address needs an explicit CIDR allowance. Other host ports, the gateway, other owners' subnets, loopback, link-local (including cloud metadata), and multicast remain blocked. API authentication still applies. See pkg/gateway/guard.go.

A fork made with create --from inherits its source's network policy; it cannot override --net, --egress, or allow rules during creation. Inspect the effective policy with bhatti inspect NAME. A guest without networking is not an obstacle to inbound HTTP: incoming traffic goes over vsock instead of the guest NIC.

Sibling reachability

Two networked sandboxes of the same owner share netd's address space, but the sender's --allow-siblings opt-in is required for TCP and UDP. netd re-originates the sibling leg inside its gVisor stack, applies policy rather than bridging the two NICs, and never routes to another owner's netd. Use the receiver's numeric network IP from bhatti inspect NAME: v2.5.10 has no automatic bare-name or <sandbox>.sb resolution. DNS queries are policy-checked and forwarded to an external resolver, and --allow-siblings does not authorize DNS names. To talk to one another, create each sender with --allow-siblings, and create each receiver with a NIC (for example --egress deny --allow-siblings).

Secret grants

A plain create --secret NAME places the real value in guest environment variables. For an agent that should call one HTTPS host without reading the real key, use create --secret-grant NAME@api.example.com or secret grant NAME --sandbox dev --host api.example.com after creation. The guest gets a placeholder. On a permitted outbound connection to port 443 with matching TLS SNI, netd intercepts HTTP/1.1 using a per-sandbox CA installed in the guest trust store, verifies the upstream host's TLS certificate, and substitutes the value in request headers only. A placeholder in the URL, Host header, or body is refused. A grant never opens egress by itself: under deny, the granted host must already be allowed by --allow-host. No NIC means no grant. Revocation blocks new connections; existing connections may retain the resolved value until they close. See Credentials for agents and cmd/bhatti-netd/intercept.go.

Reaching services inside sandboxes

Guest 100.64.* addresses are not directly reachable from outside their gateway. Host-to-guest TCP uses a separate vsock tunnel to the guest's localhost and does not require guest networking:

  • bhatti forward dev 3000 binds 127.0.0.1:3000 on the CLI machine by default and forwards TCP through authenticated API WebSocket tunnels to guest port 3000. Use LOCAL:GUEST for different ports, 0:GUEST for an ephemeral local port, or --address to change the bind address. The CLI owns the listeners and stays running, including with --json.
  • The authenticated API proxy at /sandboxes/:id/proxy/:port/ routes HTTP/WebSocket via vsock and needs a bearer token. On a socket-only install use curl --unix-socket <data_dir>/api.sock or the CLI rather than assuming an HTTP listener.
  • bhatti publish dev -p 3000 adds a public HTTP/WebSocket route. HTTPS alias URLs need custom-domain/TLS setup; without domain mode, public_proxy_listen can instead serve unauthenticated path-based HTTP URLs for development/testing. With neither public proxy configured, the CLI prints an authenticated API proxy fallback, not a public URL. Publishing and guest egress are separate.

Snapshot, recovery, and the old network fabric

On restore, a networked VM reconnects its virtio-net link to its owner's gateway and retains its inherited policy. No TAP device, bridge, or host iptables rule needs rebuilding. A no-NIC sandbox has no network link to restore. netd is detached and re-adopted after daemon restarts; only owners with networked sandboxes need one.

History: v1 used host TAP devices and per-user Linux bridges to avoid a shared cross-user L2 network. v2 dropped those host devices in favor of userspace netd; there is no alternative guest-network backend to configure.

Where to go next