Running vmon serve

vmon serve is the long-running server process. This page covers what each role needs, which ports must be reachable, what the server keeps on disk, and how to run it on a production host. Every flag is listed in vmon serve options.

Roles and topologies

A vmon serve process runs one or both roles, selected with --role (default node,driver):

Both topologies use the same binary and flags:

• Monolithic: one vmon serve --role node,driver process; its single-voter raft KV holds all state. No gossip seeds are needed.
• Distributed: any number of --role driver and --role node processes (or mixed ones) that share one cluster secret, replicate all state in the drivers’ raft KV, and find each other through gossip seeds (--join). See Clusters.

The node role runs on Linux with KVM (x86_64 and aarch64 guests) and on macOS 15+ on Apple Silicon with HVF (aarch64 guests). The macOS binary must be ad-hoc signed with hvf.entitlements. The driver role runs on both. Windows builds contain only vmon vmm. See Installation.

Listeners

Peers dial the advertised IPs (--advertise) on the two internal listener ports, not the bind addresses. When --advertise is unset, a listener bound to a specific IP advertises that IP, and the default 0.0.0.0 listeners advertise every interface IP except loopback, link-local, and container or VM bridges. Each peer prefers an address on its own LAN and falls back to the others (details in Cluster and gossip).

Cluster HTTP requires Authorization: Bearer <cluster secret>. Gossip packets carry an HMAC tag keyed from the same secret, and unauthenticated packets are dropped. Even so, keep both internal listeners on a private network, because the node API controls every VM.

GET /health on the public listener needs no authentication. It returns 204 while the driver can serve linearizable reads of the cluster KV (a raft leader is reachable and the local replica has caught up), so load balancers can use it as a health check.

Data and run directories

--data-dir defaults to /var/lib/vmon for root on Linux and to ~/.vmon otherwise. It holds:

node-id                     random 64-bit member ID, written on first start
cluster-secret              generated secret (0600) when --cluster-secret is unset
raft/                       drivers: raft log, vote, snapshots (fsync'd) of the KV holding all driver state
assets/                     nodes: pinned fallback kernel, when downloaded
egress/                     Linux nodes: node-local MITM CA for egress header rules
files/<templateID>/<hash>.tar   nodes: build file-context archives
cache/oci/                  nodes: pulled image layers
cache/steps/, cache/context/    nodes: build step cache, extracted COPY contexts
builds/<buildID>/           nodes: template artifacts (rootfs, snapshot, logs)
sandboxes/<sandboxID>/      nodes: disk overlay, console and metrics logs, live snapshot dir
paused/<sandboxID>/         nodes: paused sandbox state
volumes/<volumeID>/         nodes: volume contents (unless --volumes-dir)

Keep the data directory across restarts; a driver’s raft/ is its replica of every team, key, template, and sandbox record. Its node-id is the process’s identity in gossip, raft, and GET /nodes, and a restarted node adopts its running sandboxes only from the same directory. Node free disk is measured on the data directory. For the storage layout of disks, pauses, and volumes, see Storage, object store, and volumes.

--run-dir holds one short subdirectory per VM with its api.sock and agent.sock. It defaults to $XDG_RUNTIME_DIR/vmon, or to /tmp/vmon-<uid>. The path must stay short because of the 104-byte macOS sun_path limit.

Guest kernel and agent

At startup the node resolves the guest kernel and the guest agent:

• Kernel: --kernel, else vmon-kernel-<arch> next to the vmon binary (release assets), else target/kernel/ of a dev tree (just kernel). As a last resort the node downloads a pinned cloud-hypervisor kernel into <data-dir>/assets/. That kernel lacks netfilter, IPv6, bridge, and NFS, which breaks Docker-in-sandbox and templates that rely on those features, so vmon logs a warning when it falls back to it.
• Agent: --agent, else vmon-agent-<arch> next to the binary, else target/test-assets/ of a dev tree (just agent-musl). It must be a static ELF for the guest architecture. If no agent is found, the node does not start.
• GPU driver bundle: --gpu-driver, else vmon-nvidia-<arch>/ next to the binary, else target/kernel/ of a dev tree (just gpu-driver). Optional: without it, GPUs are listed but not assignable. See GPUs.

Keep the release assets next to vmon, or pass both flags explicitly. See Guest kernel.

Cluster secret

The cluster secret authenticates cluster HTTP and gossip. It is also the root of every derived key: JWTs, envd and traffic access tokens, signed upload URLs, and the AES key for API keys, env vars, secrets, and webhook secrets stored in the cluster KV.

• When --cluster-secret (VMON_CLUSTER_SECRET) is unset, vmon serve uses <data-dir>/cluster-secret, and generates the file on first start (64 hex characters, mode 0600) with a warning.
• Every process in a cluster must use the same secret. Generate it once and distribute it, for example by copying the first process’s cluster-secret file into VMON_CLUSTER_SECRET for the others.
• vmon admin needs the same secret to encrypt the API keys it creates. It reads --cluster-secret, or else <data-dir>/cluster-secret. See Administration.
• Changing the secret invalidates stored ciphertexts, JWTs, and derived envd and traffic tokens; stored refresh tokens are not derived from it.

Tip

Pass secrets through VMON_CLUSTER_SECRET and VMON_ADMIN_TOKEN instead of flags. --help hides these values, but command-line arguments show up in process listings.

Logging

Logs go to stderr. --log (VMON_LOG) takes a tracing env-filter, by default info,openraft=warn, and an invalid filter aborts startup. --log-json (VMON_LOG_JSON=true) switches to one JSON object per line.

VMON_LOG='info,openraft=warn,vmon_driver=debug' vmon serve --log-json

Each sandbox’s guest console and VMM log are kept under sandboxes/<id>/ in the data directory, not in the server log.

Shutdown and restart

vmon serve installs its SIGINT and SIGTERM handlers before it starts the roles, so a signal during startup still shuts down cleanly. On either signal:

1. The listeners stop accepting. In-flight public connections get up to 10 s to drain.
2. The node role stops its background tasks and removes idle pre-created TAPs. Sandbox VMMs keep running: they live in their own process group, and the next start on the same data directory adopts them.
3. Raft shuts down, and the process leaves gossip (state Left), waiting up to 2 s for the announcement.

If a listener fails, for example because its port is taken, the process exits with the error without running the graceful node and cluster shutdown sequence.

A node restarted with SIGTERM can adopt its surviving VMMs when it comes back within the leader’s 60 s grace period for Left nodes. On Linux, a sandbox with lazily restored memory loses its node-local page server and fails its next remote page fault after restart. A node that dies without leaving (SIGKILL, crash, network loss) goes Dead, and its sandboxes are marked killed. See Failure handling.

Linux privileges

• KVM: VMM processes need read-write access to /dev/kvm, including the uid:gid selected by --vmm-user; they open it after dropping privileges.
• TAP networking (--net tap, the Linux default via auto) needs root or CAP_NET_ADMIN. The node creates TAP devices with ip and installs NAT and egress rules with iptables, iptables-restore, ip6tables, and ip6tables-restore. With --net tap and no such privilege, startup fails. With auto, the node starts but rejects creates. See Networking and egress.
• Hugepages: --hugepage-pool-mb writes /proc/sys/vm/nr_hugepages, which needs root.
• VMM privilege drop: when vmon serve runs as root, set --vmm-user <uid>:<gid>. VMMs then drop to that uid and gid and run under their Landlock and seccomp filters. The node makes each VM’s run directory, disk directory, and snapshot directory, as well as its TAP device, accessible to that user. Without --vmm-user, a root node starts VMMs with --no-sandbox, so they keep root and run without filters.

Capacity

A node advertises its capacity in gossip, and drivers place sandboxes from it:

When no node has room, creates fail with 503 sandbox_capacity_unavailable. When no node matches the request’s architecture, features, or pin, they fail with 503 sandbox_no_compatible_node. Per-team quotas are separate; see Authentication and teams. Lower --max-vcpus and --max-memory-mb to reserve host resources for other workloads.

Hugepages

On Linux, sandbox RAM restored from a snapshot (create, resume, fork) can be backed by 2 MiB hugetlb pages. This requires room in the host hugetlb pool, userfaultfd WP_ASYNC and PAGEMAP_SCAN support (6.7+), permission to create userfaultfd, readable pagemap state, and a 2 MiB-aligned memory layout. If these prerequisites are unavailable, restore falls back to 4 KiB pages. Pages are filled on first touch from the memory file, and KVM maps them with 2 MiB stage-2 entries.

--hugepage-pool-mb N sets vm.nr_hugepages to N / 2 pages when the node starts. This sets the pool size, not an increment. The kernel can grant fewer pages on a fragmented host, and vmon logs what it got. The default 0 leaves the host setting alone, so a pool configured by other means is used as is.

The latency figures were measured under nested KVM (CHANGELOG, ARCHITECTURE.md §8.2).

Size the pool for the total RAM of the restored sandboxes you expect to run at once. A VM that the pool cannot cover falls back to 4 KiB pages. Surplus pages (vm.nr_overcommit_hugepages) also work, but can stall a create for seconds in direct compaction on a fragmented host.

Example: monolithic Linux host

One root process with both roles, HTTPS ingress on a wildcard domain, internal listeners on loopback, and VMMs dropped to an unprivileged uid. Place vmon-kernel-<arch> and vmon-agent-<arch> next to vmon, and point *.sbx.example.com and api.sbx.example.com at the host.

export VMON_ADMIN_TOKEN='<random admin token>'

vmon serve \
  --role node,driver \
  --data-dir /var/lib/vmon \
  --domain sbx.example.com \
  --listen 127.0.0.1:3000 \
  --tls-listen 0.0.0.0:443 \
  --tls-cert /etc/vmon/tls/fullchain.pem \
  --tls-key /etc/vmon/tls/privkey.pem \
  --cluster-listen 127.0.0.1:7947 \
  --gossip-listen 127.0.0.1:7946 \
  --vmm-user 64000:64000 \
  --hugepage-pool-mb 32768 \
  --registry-mirror https://mirror.gcr.io \
  --log-json

On first start this generates /var/lib/vmon/cluster-secret. Run vmon admin on the same host while the server runs (it reads that file and reaches the driver at http://127.0.0.1:7947 by default) to create the first team. With --tls-listen, --public-url defaults to https://api.sbx.example.com. Clients need only E2B_API_KEY and E2B_DOMAIN=sbx.example.com; see Connecting SDKs and the CLI.