Introduction

Vibemon is one Rust binary, vmon. A driver role serves the E2B REST API, sandbox ingress, and scheduling; a node role hosts the microVMs and builds templates. Run both in one process on a laptop, or any number of each across hosts, joined by gossip, with drivers sharing a raft-replicated KV that holds all state; there is no external database. There is no vibemon SDK: the clients are the official e2b, e2b_code_interpreter / @e2b/code-interpreter, e2b_desktop / @e2b/desktop packages and the e2b CLI, unmodified.

# E2B_API_KEY=e2b_… E2B_DOMAIN=localhost
# E2B_API_URL=http://127.0.0.1:3000 E2B_SANDBOX_URL=http://127.0.0.1:3000
from e2b import Sandbox

sbx = Sandbox.create()                      # copy-on-write fork of the `base` template
print(sbx.commands.run("uname -a").stdout)  # runs inside the microVM
sbx.pause()                                 # memory + disk snapshot
sbx = Sandbox.connect(sbx.sandbox_id)       # resumed where it left off
sbx.kill()

Start here

Platforms

Details per platform: Installation and Networking and egress.

Source and design

The code lives at github.com/stencil-hq/vibemon. ARCHITECTURE.md is the normative design; these pages describe how to run and use the system.

Quickstart

Run a single-process vibemon (node and driver roles in one vmon serve) on one machine, build the default base template, and drive a sandbox with the official E2B SDKs. Works on macOS 15+ on Apple Silicon and on Linux with KVM.

Prerequisites

Linux hosts also need /dev/kvm, ip (iproute2), and iptables; see Installation.

No database is needed: the driver keeps all state in its raft KV under the data directory.

1. Build the guest kernel (recommended)

just kernel --arch aarch64          # macOS (in Docker), or an aarch64 Linux host
just kernel --arch x86_64           # x86_64 Linux host

Outputs land in target/kernel/, where a dev-tree vmon finds them. Without this step the node downloads the pinned cloud-hypervisor kernel and logs a warning: that kernel has no netfilter, IPv6, bridge, or NFS, so Docker and some units of the default templates fail. Templates capture the kernel they were built with, so build it before step 3. Details: Guest kernel.

2. Start the server

Leave the server running in its own terminal.

macOS: HVF needs no root, and networking is user-mode NAT.

just serve

just serve builds and ad-hoc signs vmon, builds the static guest agent into target/test-assets/, and runs:

vmon serve --agent target/test-assets/vmon-agent-<arch> --listen 127.0.0.1:3000 \
  --public-url http://127.0.0.1:3000 --domain localhost

--public-url matters with a loopback listener: signed template-file upload URLs point at it, and its default is derived from the cluster listener’s address, not from --listen.

Extra arguments are forwarded, e.g. add --registry-mirror https://mirror.gcr.io. State goes to ~/.vmon: the raft KV with every team, key, template, and sandbox record (raft/), and the generated cluster-secret that vmon admin reads.

Linux: the node creates TAP devices and NAT rules, which needs root or CAP_NET_ADMIN. just serve runs unprivileged, so the node starts but rejects sandbox creates and builds. Run the same command under sudo. Use an explicit cluster secret so the unprivileged vmon admin and just templates can use it (as root, the generated secret is written to /var/lib/vmon/cluster-secret with mode 0600):

just build agent-musl
export VMON_CLUSTER_SECRET="$(openssl rand -hex 32)"
sudo "$(just bin)" serve --cluster-secret "$VMON_CLUSTER_SECRET" \
  --agent "$PWD/target/test-assets/vmon-agent-$(uname -m)" \
  --listen 127.0.0.1:3000 --domain localhost --public-url http://127.0.0.1:3000

Export the same VMON_CLUSTER_SECRET in every shell that runs the commands below. Alternatives to running everything as root: Linux privileges.

Check that the driver is up. /health returns 204 once the raft KV serves reads:

curl -i http://127.0.0.1:3000/health

3. Build the base template

just templates base

templates/build.py creates (or reuses) the system team system through vmon admin, mints a key for it, builds base with the E2B Template builder, and publishes it. A system team’s public templates claim bare aliases, so base then resolves for every team. Run just templates without arguments to build all defaults (base, code-interpreter-v1, desktop, mcp-gateway, ubuntu-systemd).

Important

build.py uses E2B_API_KEY if it is set. Run it in a shell without your own team’s key, or the templates land in that team’s namespace and the bare alias stays unclaimed.

Tip

Docker Hub’s anonymous pull limit shows up as HTTP 429 TOOMANYREQUESTS. Restart the server with --registry-mirror https://mirror.gcr.io (or VMON_REGISTRY_MIRROR).

More: Templates.

4. Create a team and API key

"$(just bin)" admin create-team dev

vmon admin talks to the running server’s cluster listener (http://127.0.0.1:7947 by default).

The output has team, slug, and api key lines. Export the key and point the SDKs at the server:

export E2B_API_KEY=e2b_...                  # the `api key` line
export E2B_DOMAIN=localhost
export E2B_API_URL=http://127.0.0.1:3000
export E2B_SANDBOX_URL=http://127.0.0.1:3000

E2B_SANDBOX_URL sends sandbox (envd) traffic to the API listener, which routes it by the E2b-Sandbox-* headers. Connecting SDKs and the CLI covers the TLS setup that needs only E2B_DOMAIN.

5. Use a sandbox

Python:

# uv run --with e2b python quickstart.py
from e2b import Sandbox

sbx = Sandbox.create()                              # forks the `base` template snapshot
print(sbx.commands.run("uname -a").stdout)
sbx.files.write("/home/user/hello.txt", "hi")
sbx.pause()                                         # memory + disk snapshot
sbx = Sandbox.connect(sbx.sandbox_id)               # resumes it
print(sbx.files.read("/home/user/hello.txt"))
sbx.kill()

TypeScript:

// npm install e2b && npx tsx quickstart.ts
import { Sandbox } from 'e2b'

const sbx = await Sandbox.create()
console.log((await sbx.commands.run('uname -a')).stdout)
await sbx.files.write('/home/user/hello.txt', 'hi')
console.log(await sbx.files.read('/home/user/hello.txt'))
await sbx.kill()

Lifecycle details (timeouts, auto-pause, fork, snapshots): Sandboxes.

With release binaries (Linux)

Tagged releases ship Linux binaries only (x86_64 and aarch64, static musl); macOS uses the dev path above. Install vmon, vmon-kernel-<arch>, and vmon-agent-<arch> into one directory as described in Installation, here /opt/vmon. Then:

export VMON_CLUSTER_SECRET="$(openssl rand -hex 32)"
sudo /opt/vmon/vmon serve --cluster-secret "$VMON_CLUSTER_SECRET" \
  --listen 127.0.0.1:3000 --domain localhost --public-url http://127.0.0.1:3000

No --kernel or --agent is needed: the node picks up the files next to vmon. The template definitions are not in the release tarballs; build them from a checkout of the repository (no Rust toolchain needed):

VMON_BIN=/opt/vmon/vmon uv run templates/build.py base
/opt/vmon/vmon admin create-team dev

Export the printed key and the client variables as in step 4, then continue at step 5.

Installation

What to install on each host: the release assets, platform support, host requirements, driver state, and building from source. For a single-machine trial, the Quickstart is shorter.

Platform support

Egress features differ per platform; see Networking and egress.

Release assets

Tags matching v* run .github/workflows/release.yml, which attaches these files to the GitHub release (each with a .sha256 sidecar):

<target> is x86_64-unknown-linux-musl or aarch64-unknown-linux-musl; <arch> is x86_64 or aarch64.

Note

Releases up to 0.3.2 are the previous product (vmond) and do not have this layout. See CHANGELOG.md.

Install a release

Nodes look for vmon-kernel-<arch> and vmon-agent-<arch> in the directory that holds vmon, so put all three in one place. Both tarballs contain a vmon-agent-<arch>; either works.

TAG=vX.Y.Z                       # a release tag
ARCH=$(uname -m)                 # x86_64 or aarch64
BASE=https://github.com/stencil-hq/vibemon/releases/download/$TAG
for f in vmon-$ARCH-unknown-linux-musl.tar.gz vmon-assets-$ARCH.tar.gz; do
  curl -fLO "$BASE/$f" && curl -fLO "$BASE/$f.sha256"
done
sha256sum --check vmon-$ARCH-unknown-linux-musl.tar.gz.sha256 vmon-assets-$ARCH.tar.gz.sha256
tar -xzf vmon-$ARCH-unknown-linux-musl.tar.gz
tar -xzf vmon-assets-$ARCH.tar.gz
sudo install -d /opt/vmon
sudo install -m 0755 vmon-$ARCH-unknown-linux-musl/vmon vmon-$ARCH-unknown-linux-musl/vmon-agent-$ARCH /opt/vmon/
sudo install -m 0644 vmon-assets-$ARCH/vmon-kernel-$ARCH /opt/vmon/

Driver-only hosts need just vmon. --kernel (VMON_KERNEL) and --agent (VMON_AGENT) override the lookup; the full search order is in Running vmon serve.

Host requirements

Linux nodes

• /dev/kvm.
• Root or CAP_NET_ADMIN for TAP networking (the only Linux mode; --net user is rejected on Linux). Without it, the default --net auto starts the node but rejects sandbox creates; explicit --net tap fails at startup. See Linux privileges for running VMMs as an unprivileged uid with --vmm-user.
• ip (iproute2), iptables, and iptables-restore, which the node runs to set up TAPs and NAT rules.

macOS nodes

• macOS 15 or later on Apple Silicon.

• vmon must be ad-hoc signed with the hypervisor entitlement in hvf.entitlements (com.apple.security.hypervisor). just build and just release do this; by hand:

  codesign --sign - --entitlements hvf.entitlements --force target/release/vmon
  

• No root: HVF needs none, and guest networking is user-mode NAT inside the VMM.

Drivers

Drivers need no external database: they keep all durable state (teams, keys, templates, builds, sandboxes, events, webhooks, secrets, volumes) in the raft KV they replicate among themselves, in memory and snapshotted under <data-dir>/raft/ (history such as events, build logs, and webhook deliveries goes to an on-disk journal there instead of memory). Keep that directory on persistent storage. Every driver of a cluster uses the same cluster secret; see Running vmon serve.

Build from source

Requires rustup (the toolchain is pinned in rust-toolchain.toml) and just.

just release                     # target/release/vmon (+ HVF codesign on macOS)
just agent-musl                  # target/test-assets/vmon-agent-<arch>
just kernel --arch "$(uname -m | sed 's/arm64/aarch64/')"   # target/kernel/

A vmon built at target/release/vmon finds the agent in target/test-assets/ and the kernel in target/kernel/ relative to its executable path, not the working directory. To install it elsewhere, copy the agent and kernel next to it as vmon-agent-<arch> and vmon-kernel-<arch> (the kernel file is target/kernel/Image-aarch64 or target/kernel/bzImage-x86_64).

just local-release <target> builds the release tarballs for a Linux musl target into dist/ without uploading anything. It needs cargo-zigbuild and the target arch’s kernel in target/kernel/ (just kernel), and it runs the packaged vmon to smoke-check it, so a cross-arch target on Linux needs qemu-user or a VMON_RELEASE_RUNNER wrapper. On macOS, VMON_RELEASE_RUNNER must name an executable VM/container wrapper even when the CPU architecture matches. Toolchains, tests, and the other recipes: Building and testing.

Verify the install

vmon --version                   # prints `vmon <version>`
curl -i http://127.0.0.1:3000/health

GET /health on a driver’s --listen address returns 204 No Content when the driver can serve linearizable reads of the raft KV, and 500 or 503 otherwise.

Connecting SDKs and the CLI

Configure the official E2B SDKs (e2b, e2b_code_interpreter / @e2b/code-interpreter, e2b_desktop / @e2b/desktop) and the e2b CLI to talk to vibemon instead of E2B’s cloud. They need no code changes, only environment variables. This page also shows how to call vibemon’s API extensions, which the SDKs cannot send.

Environment variables

Leave E2B_DEBUG, E2B_HTTP_VERSION, and E2B_ACCESS_TOKEN unset; the e2e harness unsets all three. E2B_ACCESS_TOKEN is deprecated upstream: the CLI never reads it, and the driver answers the old /access-tokens routes with 410.

Routing modes

The driver serves the API and sandbox traffic on the same listener. A request goes to a sandbox when its Host is <port>-<sandboxId>.<anything>, or when it carries both E2b-Sandbox-Id and E2b-Sandbox-Port; requests without either target go to the API. Incomplete or malformed routing headers get 400. Clients pick one of two setups. The server side (domains, certificates, DNS) is in Domains, TLS, and ingress.

Header routing (no TLS, development)

Point everything at the plain-HTTP listener:

export E2B_API_KEY=e2b_...
export E2B_DOMAIN=localhost                    # the driver's --domain
export E2B_API_URL=http://127.0.0.1:3000
export E2B_SANDBOX_URL=http://127.0.0.1:3000
export E2B_VOLUME_API_URL=http://127.0.0.1:3000

SDK calls (commands, files, PTY, code execution, volumes) work. Domain-based hosts and URLs still require the sandbox subdomains; signed file URLs also need routing information when opened outside the SDK:

• get_host() / getHost() results, opened in a browser or HTTP client;
• signed file URLs (download_url(), upload_url()): Python honors E2B_SANDBOX_URL, but requests to that URL still need the sandbox routing headers;
• get_mcp_url() / getMcpUrl();
• the desktop stream URL (stream.get_url()).

To reach a guest port by hand in this mode, set the Host or the routing headers yourself:

curl -H 'E2b-Sandbox-Id: <sandboxId>' -H 'E2b-Sandbox-Port: 8080' http://127.0.0.1:3000/
curl -H 'Host: 8080-<sandboxId>.localhost' http://127.0.0.1:3000/

Subdomain routing (TLS, production)

With the driver on --tls-listen, a certificate for *.<domain> and api.<domain>, and DNS for both pointing at the driver, the SDKs need only the upstream variables:

export E2B_API_KEY=e2b_...
export E2B_DOMAIN=vmon.example.com

The API is https://api.vmon.example.com, and sandbox port 8080 is https://8080-<sandboxId>.vmon.example.com. This supports SDK calls and the URLs listed above (unset any plain-HTTP URL overrides). This is the topology of upstream E2B and of the e2e suites’ --ingress tls mode.

TLS with a private CA

• Node (JS SDKs, CLI): NODE_EXTRA_CA_CERTS=/path/ca.pem.
• Python on Linux: SSL_CERT_FILE pointing at a bundle of the system roots plus your CA.
• Python on macOS: the SDK’s transport verifies through the macOS keychain and ignores SSL_CERT_FILE. Trust the CA in the keychain, or keep header routing (E2B_API_URL / E2B_SANDBOX_URL) for the Python SDK while other clients use the TLS subdomains. The e2e harness does the latter.

To check that the TLS listener negotiates HTTP/2:

curl -v --cacert ca.pem https://api.vmon.example.com/health    # expect "ALPN: server accepted h2"

SDK notes

Default templates are resolved by bare alias (base, code-interpreter-v1, desktop, mcp-gateway). They exist only after an operator builds them with just templates; see Templates.

Code interpreter: Sandbox.create() uses code-interpreter-v1. Jupyter on guest port 49999 is reached through the same ingress, in either routing mode.

from e2b_code_interpreter import Sandbox

with Sandbox.create() as sbx:
    print(sbx.run_code("1 + 1").text)

Desktop: Sandbox.create() uses desktop. Viewing the stream from stream.get_url() needs subdomain routing.

from e2b_desktop import Sandbox

desktop = Sandbox.create()
desktop.stream.start()
print(desktop.stream.get_url())   # https://6080-<sandboxId>.<domain>/vnc.html?autoconnect=true&resize=scale

MCP gateway: Sandbox.create(mcp=…) starts sandboxes from the mcp-gateway template. get_mcp_url() returns https://50005-<sandboxId>.<domain>/mcp, so it needs subdomain routing; send get_mcp_token() as the bearer token. The template runs Docker in the guest and therefore needs the vmon guest kernel (Guest kernel). Supported catalog entries and limits: E2B compatibility, §4 and §5.

Egress header rules and SOCKS5 upstreams (network rules, egressProxy) work only on Linux nodes; creates that need them fail with 503 when no Linux node is available. See Networking and egress.

CLI login

The e2b CLI authenticates with E2B_API_KEY, or with the key that e2b auth login stores in ~/.e2b/config.json. Login needs a user, created by an operator:

vmon admin create-user dev@example.com --password 'change-me' --team dev

--team takes the team’s id or slug and makes it the user’s default team. Then:

npm install -g @e2b/cli
export E2B_DOMAIN=localhost
export E2B_API_URL=http://127.0.0.1:3000
export E2B_SANDBOX_URL=http://127.0.0.1:3000          # only without TLS subdomains
export E2B_DOCS_BASE=http://127.0.0.1:3000/docs       # only if https://$E2B_DOMAIN/docs is not the driver
e2b auth login
e2b sandbox create base

The CLI opens ${E2B_DOCS_BASE:-https://$E2B_DOMAIN/docs}/api/cli. The driver serves this login page on any Host that routes to the API rather than to a sandbox. After you sign in, it redirects back to the CLI’s localhost callback, and the CLI picks the user’s default team. Command-by-command status: E2B compatibility, §6.

Vibemon extensions over raw HTTP

The SDKs only send upstream fields. For vibemon extensions, call the REST API directly with X-API-Key, then attach the SDK to the sandbox by id. Full semantics: E2B compatibility, §5.

Bucket mounts (vmonS3Mounts)

curl -sX POST "$E2B_API_URL/v2/sandboxes" -H "X-API-Key: $E2B_API_KEY" \
  -H 'Content-Type: application/json' -d '{"templateID": "base",
  "vmonS3Mounts": [{"path": "/mnt/data", "uri": "s3://my-bucket/datasets",
    "mode": "ro", "credentialsSecret": "my-bucket-keys"}]}'

The response carries sandboxID. Attach to it:

from e2b import Sandbox

sbx = Sandbox.connect("<sandboxID>")
print(sbx.commands.run("ls /mnt/data").stdout)

mode is ro (default), overlay, or writeback. credentialsSecret names a team secret holding the credentials JSON; inline credentials work too. Operator settings (--s3-mounts-allow-private): Storage, object store, and volumes.

Sandbox self-keepalive

Every sandbox response (POST /v2/sandboxes, GET /sandboxes/{id}) includes envdAccessToken. Sent as X-Access-Token, it authorizes three routes, and only for that sandbox: GET /sandboxes/{id}, POST /sandboxes/{id}/timeout, and POST /sandboxes/{id}/refreshes. Give it to code inside the sandbox so the sandbox can extend its own lifetime without a team key:

curl -sX POST "$E2B_API_URL/sandboxes/$SANDBOX_ID/timeout" \
  -H "X-Access-Token: $ENVD_ACCESS_TOKEN" -H 'Content-Type: application/json' \
  -d '{"timeout": 300}'                                  # seconds from now -> 204
curl -sX POST "$E2B_API_URL/sandboxes/$SANDBOX_ID/refreshes" \
  -H "X-Access-Token: $ENVD_ACCESS_TOKEN" -H 'Content-Type: application/json' \
  -d '{"duration": 60}'                                  # extend by seconds -> 204

Wrong tokens, and killed or unknown sandboxes, get 401. The team’s maximum sandbox length still caps the expiry. From inside the sandbox, the API URL must be reachable under the sandbox’s egress policy; private ranges are denied unless the operator allows them with --egress-internal-allow (Networking and egress).

Architecture

How a vibemon deployment is put together: which processes run, where state lives, which ports they open, and what happens on the three requests that matter most. Read it before planning a topology. The normative design, with every internal detail, is ARCHITECTURE.md.

Components

The vmon binary provides the host components; vmon-agent is a separate guest binary.

A process may run both roles (--role node,driver, the default). One such process is a complete monolithic deployment. A distributed deployment is any mix of driver-only, node-only, and combined processes. They find each other through a gossip memberlist seeded with --join. Topologies and flags: Running vmon serve.

The guest boots its distro init: systemd on Debian, Ubuntu, Fedora, CentOS, Rocky, AlmaLinux and Arch, and OpenRC on Alpine. envd runs under that init as a service. Images without a matching ID or ID_LIKE distro profile use the agent as PID 1 instead. VMMs run in their own process group. The restarted node reconnects to VMMs that are still running.

Where state lives

A paused sandbox has no KV route. It exists as a paused sandbox record in the store plus the paused state on its node, or in the object store. Data layout and the object store: Storage, object store, and volumes. Cluster mechanics: Clusters.

Listeners

The cluster listener and gossip are authenticated with the shared cluster secret: a bearer token on cluster HTTP, and an HMAC tag on every gossip packet. Never expose them publicly. Guest ports are never reached over the guest network. Ingress always goes through GC4 tunnels, so no inbound NAT is needed. Details: Running vmon serve.

The driver routes each public request by its Host header or its E2b-Sandbox-* headers:

1. Host <port>-<sandboxID>.<domain> → ingress to that sandbox port.
2. Headers E2b-Sandbox-Id and E2b-Sandbox-Port → ingress.
3. Requests without a sandbox target → the E2B API (including /health and the CLI login pages). Incomplete or malformed routing headers get 400.

See Domains, TLS, and ingress.

 E2B SDK / e2b CLI
        │  HTTPS: api.<domain>, <port>-<id>.<domain>
        ▼
 ┌─────────────┐  raft (cluster HTTP)  ┌─────────────┐
 │   driver    │◄─────────────────────►│   driver    │  raft KV: all state
 └──────┬──────┘   gossip (all procs)  └─────────────┘
        │ cluster HTTP: node API, tunnels
        ▼
 ┌─────────────┐  spawn  ┌───────────┐  virtio-console (GC4)  ┌──────────────────────┐
 │    node     │────────►│ vmon vmm  │◄──────────────────────►│ guest: init + envd   │
 └─────────────┘         └───────────┘                        │ (vmon-agent, :49983) │
                                                              └──────────────────────┘

Flow: creating a sandbox

Sandbox.create() sends POST /v2/sandboxes.

1. The driver authenticates the API key and resolves the template reference (template ID, alias, or namespace/alias, with an optional :tag) to a ready build. It then checks the build’s vCPUs, memory, and disk against the team limits.
2. After selecting the initial node, it admits the sandbox with one KV transaction. The transaction checks the team’s concurrent-sandbox limit and writes the route sbx/<id> plus the admission index entry. Over the limit, the create fails with 429.
3. Placement picks a node from the gossip members. Candidates are alive, not draining, of the build’s architecture, with the required features. Nodes with free headroom and nodes that already hold the build’s artifacts score higher. The driver calls the node’s POST /node/v1/sandboxes. An unreachable or busy node is skipped and another is tried, up to 3 attempts; other node errors fail the create immediately.
4. If the node lacks the artifacts, it fetches them from a peer that holds the build, or from the object store. When a reusable memory snapshot and matching mount devices are available, it starts vmon vmm as a copy-on-write fork over a block overlay of the rootfs. Otherwise it cold-boots; node-local volume mounts always require a cold boot.
5. The node requests a guest kernel RNG reseed on memory restores (failures only warn), configures the network, and attaches the egress policy. It then calls envd POST /init through the tunnel with the env vars, the sandbox’s envd access token, and the default user and workdir.
6. The driver records the sandbox in the store and a sandbox.lifecycle.created event. It returns 201 with sandboxID, envdVersion, envdAccessToken, trafficAccessToken (when public traffic is off), and domain.

Every create runs to completion even if the client disconnects. A failure after admission rolls back the route and the admission slot. Lifecycle details: Sandboxes.

Flow: an SDK call reaching envd

sbx.commands.run("ls") is a Connect RPC to envd on guest port 49983.

1. The SDK sends it to https://49983-<id>.<domain>. When E2B_SANDBOX_URL is set, it sends it there with E2b-Sandbox-Id and E2b-Sandbox-Port headers. See Connecting SDKs and the CLI.
2. Any driver accepts it. Ingress reads the route sbx/<id> from its local KV replica to find the owning node. If the sandbox is paused and has auto-resume, ingress resumes it first.
3. The driver opens GET /node/v1/sandboxes/{id}/tunnel/49983 on the node’s cluster listener (HTTP upgrade). It then proxies the request over that tunnel, streaming bodies and passing WebSocket upgrades through.
4. The node opens a GC4 tcp_connect stream to the agent over the VM’s virtio-console port. GC4 is credit-flow-controlled, so streams do not block each other.
5. The agent connects to 127.0.0.1:49983 in the guest. envd checks the X-Access-Token header and runs the command.

The same path serves any guest port. For ports other than 49983, ingress also enforces the traffic access token when allowPublicTraffic is false. envd’s control paths (/init, /freeze, …) are not reachable through ingress. See Domains, TLS, and ingress.

Flow: building a template

Template.build(...) or e2b template create drives this sequence.

1. POST /v3/templates registers the template (or reuses the existing one for the alias) and creates a waiting build. It answers 202.
2. For every COPY step, the client asks GET /templates/{id}/files/{hash}. If the archive is missing, the client uploads it to the signed URL the driver returns. The driver streams each upload to the alive, non-draining node with the most free disk.
3. POST /v2/templates/{id}/builds/{buildID} validates the steps and picks a builder node. Builders are alive and not draining, of the parent template’s architecture, else of the cluster’s majority architecture. The most free capacity wins. The driver then calls POST /node/v1/builds.
4. The node pulls the OCI image and unpacks and provisions it inside a build VM. It runs the steps with the step cache, boots the result with its own init, runs the start and ready commands, and snapshots the VM.
5. Whichever driver holds the build’s lease polls the builder every 500 ms. It copies log lines into the store, and on success records which node holds the artifacts and assigns the tags. Clients poll GET /templates/{id}/builds/{buildID}/status.

Pipeline details: Templates.

Background work

The raft leader runs the cluster-wide loops:

Build supervision runs on every driver, one lease per build, so a dead driver’s builds are adopted by another driver.

Sandboxes

How a sandbox lives: create, timeouts, pause and resume, fork, snapshots, kill, and the metrics, logs, events, and webhooks it produces. Every operation is the upstream E2B API, so the official SDKs work as documented. This page states the exact vibemon semantics. Examples use the Python e2b SDK, with TypeScript where the call shape differs.

States

GET /sandboxes/{id} and Sandbox.list() report a sandbox as running or paused. Killed sandboxes disappear from both.

Transitions of one sandbox are serialized by a per-sandbox lock in the raft KV, except that an uncontended running-sandbox kill atomically removes the route without acquiring that lock. A request that finds another transition in progress waits for it, up to 2 minutes. A running sandbox has a route in the KV. A paused sandbox has only its sandbox record in the driver store and its paused state, on its node or in the object store. Lifecycle operations finish even if the client disconnects. See Architecture.

Create

from e2b import Sandbox

sbx = Sandbox.create("base", timeout=600, metadata={"job": "42"}, envs={"FOO": "bar"})

import { Sandbox } from 'e2b'

const sbx = await Sandbox.create('base', { timeoutMs: 600_000, metadata: { job: '42' } })

• A create normally is a copy-on-write fork of the template’s memory snapshot, captured with any configured start command already running. The rootfs is a block overlay of the template’s disk. See Templates.
• vCPUs, memory, and disk come from the template build, not the request. The create fails with 400 if the build exceeds the team’s max_vcpus, max_memory_mb, or max_disk_mb.
• GPUs come from metadata["vmon.gpus"], else the template’s VMON_GPUS environment variable, else none. GPU sandboxes always cold-boot, and their pauses and snapshots keep only the filesystem. See GPUs.
• Admission checks the team’s concurrent-sandbox limit atomically in the KV and answers 429 at the limit. Paused sandboxes do not count. See Authentication and teams.
• Creates with node-local volumeMounts cold-boot the template rootfs instead of forking the snapshot. Object-store volumes and vmonS3Mounts can reuse memory only when their device tags match the snapshot; otherwise they also cold-boot. See Storage, object store, and volumes.
• On memory restores, the node requests 64 bytes of fresh host entropy and a kernel CRNG reseed before envd is reinitialized. It also reseeds the source after a snapshot. Reseed failures are logged as warnings rather than failing the operation.
• The response carries envdAccessToken (required by envd), plus trafficAccessToken when network.allowPublicTraffic is false. See Authentication and teams.
• Sandbox IDs are i followed by 20 characters of [a-z0-9].

Idempotency

POST /v2/sandboxes accepts an Idempotency-Key header matching ^[a-z0-9]{1,255}$. The record lives 24 hours in the KV.

A failed create deletes the record, so the key can be retried. The SDKs do not send this header; use raw HTTP.

Timeouts

When the timeout expires, the raft leader’s reaper (every 1 s) pauses the sandbox if it has autoPause, otherwise kills it. If an auto-pause fails, the sandbox is killed.

A process in the guest can extend its own sandbox with its envd access token. See Sandbox self-keepalive.

Pause

sbx.pause()                      # memory + filesystem
# Alternatively: sbx.pause(mode="filesystem")  # filesystem only

POST /sandboxes/{id}/pause answers 204. It answers 409 when the sandbox is already paused (the SDK then returns False) or is in another transition.

• Memory pause (memory true, the default): the VMM snapshots the guest. When a fork base is available, RAM is written as a delta against it, so the RAM capture scales with pages written since the fork; otherwise it is a full snapshot. Resume restores guest processes and memory, but GC4 tunnel connections are reset.
• Filesystem pause (memory: false, SDK mode="filesystem"): the guest agent requests a guest page-cache flush, the VM stops, and only the disk is kept. Resume cold-boots from the disk: processes and in-memory state are lost.

The paused state stays on the node that ran the sandbox. With --object-store, it is also uploaded in the background so another node can resume it. See Pause artifacts and Portable paused sandboxes.

Resume and connect

sbx = Sandbox.connect(sandbox_id, timeout=300)             # resume if paused
# Alternatively: Sandbox.connect(sandbox_id, on_resume="reboot")  # cold boot if paused

const sbx = await Sandbox.connect(sandboxId, { timeoutMs: 300_000 })

• memory: false in the body (SDK on_resume="reboot") cold-boots from the paused disk instead of restoring memory. A sandbox paused filesystem-only always cold-boots.
• A resume goes back to the node holding the paused state. If that node is gone and no object store holds a copy, the resume fails with 503. With an object store, another node of the same architecture resumes it. A node with the same backend and CPU baseline restores memory. Any other same-arch node cold-boots from the disk.
• A resume starts a new execution. It emits sandbox.lifecycle.resumed, and the maximum lifetime restarts from it.
• Resume counts against the concurrent-sandbox limit (429 at the limit).

Auto-pause and auto-resume

The SDK’s lifecycle option sets the create fields autoPause, autoPauseMemory, and autoResume:

sbx = Sandbox.create("base", timeout=300,
                     lifecycle={"on_timeout": "pause", "auto_resume": True})

const sbx = await Sandbox.create('base', {
  timeoutMs: 300_000,
  lifecycle: { onTimeout: 'pause', autoResume: true },
})

An explicit connect resumes any paused sandbox, whether or not autoResume is set. A traffic-triggered resume uses the create timeout raised to at least 300 s, capped at the team’s maximum lifetime. GET /sandboxes/{id} reports the policy as lifecycle: {autoResume, onTimeout: "pause"|"kill"}. Ingress behavior: Auto-resume.

Fork

children = sbx.fork(count=3, timeout=600)
for child in children:
    if isinstance(child, Exception):
        print("fork failed:", child)

POST /sandboxes/{id}/fork with {timeout, count} answers 201 with one entry per child: either {sandbox} or {error}.

• The source must be running. It is snapshotted once, with memory, into a hidden build. It is briefly paused, then resumed with its ID and end time unchanged.
• All children start on the source’s node from that snapshot and inherit its files, env vars, metadata, lifecycle flags, network policy, and mounts. Processes are preserved only when the child can restore the memory snapshot; node-local volume mounts force a cold boot. Each child succeeds or fails on its own.
• count defaults to 1. It must be 1–100 and lower than the team’s concurrent-sandbox limit (400 otherwise). The SDK documents 1–20. timeout defaults to 15 s.
• The hidden template is retired once the children have started. Its build stays while any running or paused child uses it, then artifact GC removes it.

Snapshots

A snapshot turns a running sandbox into a template that new sandboxes start from.

info = sbx.create_snapshot("my-snapshot:v1")     # or no name; mode="filesystem" for disk only
clone = Sandbox.create(info.snapshot_id)
Sandbox.delete_snapshot(info.snapshot_id)

• POST /sandboxes/{id}/snapshots with {name?, memory?} answers 201 {snapshotID, names}. The source is paused, snapshotted, and resumed, and keeps running. The capture emits sandbox.lifecycle.checkpointed.
• name is [namespace/]alias[:tag]. The namespace must be the team’s slug (400 otherwise). An existing snapshot name owned by the team gets a new build under that tag. A name used by another kind of template answers 409.
• snapshotID is <slug>/<alias>:<tag> for named snapshots and <templateID>:default otherwise. Pass it, or the name, as the template of a create.
• memory: false (SDK mode="filesystem") captures only the disk. Sandboxes created from it cold-boot.
• GET /snapshots lists the team’s snapshots, filtered by sandboxID or name and paginated. Delete one with DELETE /templates/{id}.
• Snapshot builds live on the capturing node and are fetched by other nodes like any build. See Templates.

Kill

sbx.kill()                       # or Sandbox.kill(sandbox_id)

DELETE /sandboxes/{id} answers 204. It kills a running sandbox and deletes paused state, including the object-store copy. It answers 404 when neither exists, and the SDK then returns False. The reply comes once the VMM is killed and the route is gone. Disk, network, and process cleanup finish in the background. Logs and metrics stay readable for 7 days.

Metrics

Each node scrapes envd GET /metrics of every running sandbox every 5 s into the sandbox’s metrics.jsonl. Samples are kept for 7 days after the sandbox ends.

for m in sbx.get_metrics():                       # start/end default to the sandbox's lifetime
    print(m.timestamp, m.cpu_used_pct, m.mem_used)

• GET /sandboxes/{id}/metrics?start&end (Unix seconds) works for running, paused, and recently killed sandboxes. The samples are thinned to one per step: 5 s under 1 h, 30 s under 6 h, 60 s under 12 h, 120 s under 24 h, 300 s under 7 days, else 900 s.
• GET /sandboxes/metrics?sandbox_ids=a,b returns the latest sample (from the last 60 s) of up to 100 running sandboxes.
• e2b sandbox metrics <id> (-f to follow) prints the same samples.

Logs

Sandbox logs are the VM’s console output: kernel messages and the agent’s and envd’s JSON log lines. envd logs every process start and end as upstream does (logger=process, event_type=process_start|process_end). Logs stay readable for 7 days after a kill.

e2b sandbox logs <sandbox-id> --level info -f

fields values are strings, so numeric envd fields arrive as strings. See E2B compatibility.

Events

Every lifecycle change is recorded in the driver store (kept for 30 days) as an event of type sandbox.lifecycle.created, killed (with kill_reason: request, timeout, admin, or orphaned), updated (timeout or network change), paused, resumed, or checkpointed.

curl -s "$E2B_API_URL/events/sandboxes/$SANDBOX_ID?limit=50&orderAsc=true" -H "X-API-Key: $E2B_API_KEY"

GET /events/sandboxes (team) and GET /events/sandboxes/{id} accept offset, limit (1–100, default 10), orderAsc, and types (repeated or comma-separated).

Webhooks

Webhooks deliver the same events to your HTTP endpoint. Manage them with GET|POST /events/webhooks and GET|PATCH|DELETE /events/webhooks/{id}. Inspect deliveries with GET /events/webhooks/{id}/deliveries and GET /events/webhooks/{id}/stats.

curl -sX POST "$E2B_API_URL/events/webhooks" -H "X-API-Key: $E2B_API_KEY" \
  -H 'Content-Type: application/json' -d '{"name": "lifecycle",
  "url": "https://hooks.example.com/e2b", "signatureSecret": "s3cret",
  "events": ["sandbox.lifecycle.created", "sandbox.lifecycle.killed"]}'

• url must be an absolute http or https URL (≤ 2048 chars). events is a non-empty subset of the six types. signatureSecret is 1–512 characters, stored encrypted.
• The raft-leader driver delivers events in order. Each event gets up to 3 attempts, 500 ms then 1 s apart, with a 10 s timeout each. After the last attempt, delivery moves on to the next event. Every attempt is recorded.
• Each request is a JSON POST with the headers e2b-webhook-id, e2b-delivery-id, e2b-signature-version: v1, and e2b-signature. The body is snake_case: id, version, type, timestamp, event_category, event_label, event_data, sandbox_id, sandbox_execution_id, sandbox_template_id, sandbox_build_id, sandbox_team_id, events_ttl_days.
• e2b-signature is the upstream scheme: unpadded standard base64 of SHA-256(signatureSecret + body).

import base64, hashlib, hmac

def verify(secret: str, body: bytes, signature: str) -> bool:
    expected = base64.b64encode(hashlib.sha256(secret.encode() + body).digest()).decode().rstrip("=")
    return hmac.compare_digest(expected, signature)

Templates

A template is what every sandbox starts from: a root filesystem and, normally, a memory snapshot of a booted VM with any configured start command already running. This page covers how vibemon builds templates from OCI images, how you define them with the E2B SDK or CLI, the default templates, naming and publishing, and when artifacts are deleted.

What a template is

A sandbox create normally forks the snapshot copy-on-write, so the start command’s processes are already up, and boots over a block overlay of the rootfs. See Sandboxes. Other nodes fetch artifacts on first use from a node that holds them, transferring only allocated extents, or from the object store when builds are published there. See Clusters and Published templates.

A snapshot is reused only on a node whose architecture, hypervisor backend, and CPU baseline match the builder’s. Any other node of the same architecture regenerates the snapshot locally from the rootfs: it boots the rootfs and reruns the start and ready commands before the first sandbox starts.

Build pipeline

An uncached fromImage build runs on one node, in VMs, through these phases. fromTemplate instead clones the parent rootfs and skips unpacking and provisioning; cache hits also skip completed layers:

1. Pull. For fromImage, the node’s own OCI registry client (Docker Registry v2 / OCI distribution, no Docker or skopeo) resolves the reference. It runs the bearer-token flow, picks the linux/<arch> manifest for the node’s architecture, and downloads layers into the shared cache cache/oci/blobs/sha256/, verifying their digests. Tags are resolved with HEAD, which Docker Hub does not count against pull limits, and manifests are cached by digest. Interrupted layer downloads resume with HTTP Range. A registry 429 is retried for up to 6 attempts when Retry-After is absent or a numeric value of at most 60 s; other values fail immediately. Errors include the registry message and Retry-After when present. For fromTemplate, the node fetches the parent build’s artifacts instead.

2. Unpack. The node creates an empty sparse ext4 disk of the build’s disk size. It boots a VM with the agent as /init, sharing the layers read-only over virtio-fs. The agent applies the layers inside the guest, so ownership, setuid bits, xattrs, device nodes, and whiteouts are handled by the guest kernel, never the host. It installs the agent as /usr/bin/envd and sets the hostname to vibemon.

3. Provision. The rootfs boots with the agent as PID 1. The distro is detected from /etc/os-release and provisioned with upstream E2B’s script and package set:

   ID	Init	Package manager
   debian, ubuntu	systemd	apt
   fedora, centos, rocky, almalinux	systemd	dnf / microdnf / yum
   arch, archarm	systemd	pacman
   alpine	OpenRC (busybox init)	apk

   Other IDs fall back to a matching ID_LIKE entry, best effort and with a warning. Profiles install the init system, sudo, chrony, OpenSSH, curl, git, iptables/nftables, and related tools, and register the agent as the envd service. If neither ID nor ID_LIKE matches a profile, provisioning selects the agent as PID 1 and skips package setup.

   apt runs apt-get update before installing. If the install fails and the image’s apt sources use deb.debian.org or security.debian.org, provisioning switches those sources to archive.debian.org and runs apt-get update and the install once more, with a warning in the build log. Debian moves end-of-life releases to the archive, while the default hosts can keep serving indexes for packages they no longer carry. Debian 11 after its August 2026 end of life fails with 404 Not Found on bullseye-security packages, for example. The switched sources stay in the template, so its own apt-get steps use the archive too.

4. Steps. In the same VM, with networking and the guest clock set, the template’s steps run against the build’s file context mounted read-only. Unless the client is an E2B SDK older than 2.3.0 (template version 2.0, selected from the User-Agent), a first layer creates the user user with passwordless sudo and makes it the default user.

5. Finalize. A fresh VM boots the rootfs with the distro init, networking, and the build’s vCPUs and memory. Upstream’s configure.sh creates /.e2b, user user with sudo, and /code. envd is initialized, the start command runs in the background, and the ready command is polled. The node then syncs the filesystem and snapshots the VM.

A node runs at most --max-builds builds at once (default 2). A build that runs longer than 1 hour fails with build status polling timed out. Maximum build time is 1h0m0s. A build also fails if its builder node dies. The build’s disk is the team’s max_disk_mb.

Registry authentication and mirror

Credentials are not stored with the build spec. vmon serve --registry-mirror https://mirror.gcr.io (VMON_REGISTRY_MIRROR) is tried first for anonymous docker.io pulls, with Docker Hub as the fallback. Credentials are never sent to the mirror. See vmon serve options.

Steps and the step cache

The SDK lowers its builder calls to v2 steps. The driver validates them:

Other step types fail with 400 command type … is not implemented. Context archives are uploaded before the build starts. The client asks GET /templates/{id}/files/{hash}, then uploads to the signed URL it returns (valid 30 minutes) if the archive is not yet present. The driver’s --public-url must be reachable by the client for this.

Every layer has a cache key chained by SHA-256. The base key covers the image digest (or the parent build), disk size, provisioning script, and agent binary. Each step’s key covers the previous key plus the step’s type, args, and files hash. A rebuild resumes from the deepest cached layer. force on a step, or skip_cache (Python SDK), skipCache (JS SDK), or --no-cache (e2b template create) for the whole template, rebuilds that layer and every layer after it. The cache is per node (cache/steps/). It keeps at most 48 layers and at most --step-cache-mb of disk (default 10% of the data directory’s filesystem), evicting the least recently used layers first. Sizes count allocated blocks, so a sparse 20 GiB rootfs counts only its data; a layer larger than the whole budget is not kept.

Start and ready commands

• The start command runs in the background during finalize, and the snapshot captures it running.
• The ready command is polled every 2 s until it exits 0, for at most 10 minutes. Without a ready command, a template with a start command waits sleep 20.
• A fromTemplate build that sets neither command inherits the parent’s pair.

Building with the SDK

from e2b import Template, default_build_logger, wait_for_port

template = (
    Template()
    .from_image("python:3.13")
    .apt_install(["curl"])
    .copy("requirements.txt", "/home/user/")
    .run_cmd("pip install -r /home/user/requirements.txt")
    .set_workdir("/home/user")
    .set_start_cmd("python -m http.server 8000", wait_for_port(8000))
)

info = Template.build(template, "my-server:v1", cpu_count=2, memory_mb=1024,
                      on_build_logs=default_build_logger())
print(info.template_id, info.build_id)

import { Template, defaultBuildLogger, waitForPort } from 'e2b'

const template = Template()
  .fromImage('python:3.13')
  .setStartCmd('python -m http.server 8000', waitForPort(8000))

await Template.build(template, 'my-server:v1', { cpuCount: 2, memoryMB: 1024, onBuildLogs: defaultBuildLogger() })

Template.build calls POST /v3/templates (202, build waiting), uploads file contexts, calls POST /v2/templates/{id}/builds/{buildID} (202), and polls status. Resource rules:

Every build disk is a sparse ext4 image of the team’s max_disk_mb, so the free space after the steps is that size minus what the steps wrote, whatever minFreeDiskMb asks for. Upstream grows the filesystem to the requested free space instead; here it is already as large as the team allows.

The team’s concurrent-build limit answers 429. Starting a build cancels the template’s other running builds that target an overlapping tag.

The builder node must have the parent’s architecture for fromTemplate. Otherwise the driver uses the cluster’s majority architecture (ties prefer x86_64), and the node with the most free memory and vCPUs, minus running builds, wins. A template is therefore built for one architecture, and its sandboxes run only on nodes of that architecture. See Placement.

Building from a Dockerfile with the CLI

e2b template create my-image -d Dockerfile -c "/start.sh" --ready-cmd "test -f /tmp/ready" \
  --cpu-count 2 --memory-mb 2048

The CLI converts the Dockerfile into v2 steps (FROM, ENV, RUN, WORKDIR, COPY with file upload, USER) and builds it like the SDK. e2b template init and e2b template migrate generate SDK build scripts that build against vibemon too. The legacy e2b template build (v1 API) is not served. See E2B compatibility.

Build status and logs

The supervising driver copies the builder’s log into the driver store (the raft-replicated journal) every 500 ms, so logs outlive the builder; they are kept for 30 days. Template.get_build_status(...) and on_build_logs read these endpoints.

Names, aliases, and tags

• A name is [namespace/]alias[:tag]. Aliases and namespaces match ^[a-z0-9-_]+$, tags match ^[a-z0-9-_.]+$ and cannot be UUIDs, and names are lowercased. default means no tag.
• A build’s alias is registered in the team’s namespace, its slug. The template’s names are <slug>/<alias>. Passing a namespace other than your slug fails with 400.
• A build gets the tag from its name plus any tags, or default. Tags move to the build when it becomes ready. POST /templates/tags {target: "name:tag", tags} points more tags at an existing build. DELETE /templates/tags removes them, and GET /templates/{id}/tags lists them.
• A reference without a tag resolves the default tag, falling back to the latest ready build. An explicit tag that does not exist answers 404.
• GET /templates/aliases/{alias} answers 200 for your template, 403 for another team’s, and 404 otherwise.

Resolution order

An unqualified reference such as base is looked up in this order:

1. As a template ID, if it has the ID shape.
2. In the caller’s namespace (<slug>/base).
3. As a bare alias (base), held only by public templates of system teams.

A reference with an explicit namespace is looked up only there. Another team’s template resolves only if it is public. A team’s own base therefore shadows the default base.

Publishing and system teams

e2b template publish my-image -y      # PATCH /v2/templates/{id} {"public": true}
e2b template unpublish my-image -y

A public template is usable by every team as <slug>/<alias>, for sandboxes and as a fromTemplate parent. After unpublishing, other teams get 403.

Bare aliases are reserved for system teams (vmon admin create-team --system, set-system, unset-system). When a system team publishes a template, each of the template’s aliases also claims the free bare alias. Unpublishing, deleting the template, or unsetting the system flag releases them, and setting the flag claims them for the team’s already-public templates. An ordinary team’s publish only toggles public. Reserving bare aliases keeps base from being claimed by an ordinary team. See System teams and Administration.

Default templates

The SDKs default to bare aliases: Sandbox uses base, the code-interpreter SDK uses code-interpreter-v1, the desktop SDK uses desktop, and Sandbox.create(mcp=…) uses mcp-gateway. templates/ holds their definitions as E2B v2 build scripts:

The definitions pick architecture-specific packages at build time, so they build on aarch64 and x86_64 guests. JDK 11 is OpenJDK on x64 and Temurin on aarch64. Desktop uses Google Chrome on amd64 and Chromium as google-chrome-stable on arm64. mcp-gateway runs Docker in the guest and needs the vmon guest kernel. Provenance and licenses: templates/SOURCES.md.

Sandboxes take the template’s resources, so a team needs limits of at least 8 vCPUs and 8192 MiB to start desktop. Those are the default team limits.

Build and publish them against a running server:

just templates                       # all five, as the system team
just templates -j 2 base desktop     # selected aliases, 2 concurrent builds
uv run templates/build.py --skip-cache code-interpreter-v1

templates/build.py options:

It uses E2B_API_KEY if set; that key must belong to a system team. Otherwise it runs vmon admin ($VMON_BIN, else vmon on PATH, against a running driver’s cluster listener, VMON_CLUSTER_URL defaulting to http://127.0.0.1:7947) to find or create the system team, mark it system, and mint a key. E2B_API_URL defaults to http://127.0.0.1:3000. After each build it publishes the template with PATCH /v2/templates/{id}, which claims the bare alias.

Deletion and artifact GC

e2b template delete my-image -y      # DELETE /templates/{id}

Deleting soft-deletes the template: its aliases and tags are freed at once, its running builds are cancelled, and a second delete answers 404. Running and paused sandboxes keep working. The raft leader’s housekeeping (every 60 s) collects artifacts of finished builds of deleted templates that no running or paused sandbox uses. It deletes them on every holder node and, with --object-store, removes the published copy under builds/<buildID>/. Snapshot templates are deleted the same way, and hidden fork templates are collected once no child uses them.

Upgrades and the snapshot format

Memory snapshots carry a format version, currently 4. A VMM rejects snapshots of an older version with recapture required. When a release bumps the version, it is listed under Breaking Changes in CHANGELOG.md, and every template must be rebuilt: rerun just templates and your own builds.

Authentication and teams

Every credential vibemon accepts, where it is accepted, and how it is derived or stored, plus the team model (members, limits, blocking, system teams) that the credentials resolve to. Read this before you issue keys or expose the API.

Credentials

Team API errors normally use the E2B body {"code": <status>, "message": "…"}. The volume-content API uses string error codes, and OAuth token errors use error and error_description. Team routes without credentials get 401 authorization header is missing.

Team routes and precedence

Team-scoped routes resolve the caller to exactly one team. The driver checks the headers in this order and uses the first one present; a recognized but invalid credential fails the request instead of falling through to the next. Empty headers and an Authorization value without a non-empty Bearer token count as absent:

1. X-API-Key: the key’s team.
2. Authorization: Bearer <access token>: the team named by X-Team-ID (a team ID the user is a member of), or the user’s default team when X-Team-ID is absent. A team the user is not a member of is 401.
3. X-Admin-Token: requires X-Team-ID with a team ID (not a slug). 401 X-Team-ID header is required with X-Admin-Token without it.
4. X-Access-Token: only on the three sandbox routes listed above, only for the sandbox whose ID is in the path.

Some routes narrow this further. /api-keys refuses X-API-Key: listing, renaming, and deleting keys take a user token or the admin token (with X-Team-ID); creating a key through POST /api-keys takes a user token only. GET /teams takes a user token only.

Team API keys

Generated keys are e2b_ followed by 40 lowercase hex characters. Keys without the e2b_ prefix are rejected before any lookup with the upstream “API key is malformed” message.

The driver stores each key twice in its API key record (in the raft KV):

• key_hash: SHA-256 of the hex-decoded key bytes, used for lookups.
• key_enc: the key encrypted with AES-256-GCM under the data key derived from the cluster secret (see Key material). This is what lets GET /teams hand the CLI a usable key.

Only the first 2 and last 4 characters after e2b_ are kept in clear (mask in GET /api-keys). The plaintext is returned on creation and can also be returned by GET /teams; /api-keys listings return the mask.

Each driver caches the key → team lookup for 30 seconds. Deleting a key through the API evicts it from the cache of the driver that served the delete; other drivers may accept it until their cache entry expires. The same 30-second window applies to block and limit changes for callers that authenticate with an API key.

Users and CLI login

Users exist for the e2b CLI login flow and the /api-keys routes. Operators create them with vmon admin create-user; passwords are stored as Argon2id hashes. There is no self-signup.

e2b auth login opens GET /docs/api/cli, served on any Host that routes to the API rather than to a sandbox. The form posts email and password; on success the driver redirects to the CLI’s loopback callback (http://localhost, 127.0.0.1, or [::1] only) with:

• POST /auth/token (form, grant_type=refresh_token) returns a new access token and a new refresh token. Every refresh rotates the refresh token; presenting an already rotated token revokes all of the user’s refresh tokens and returns invalid_grant.
• POST /auth/revoke (form, token=<refresh token>) revokes that refresh token and always answers 200.
• GET /teams with the access token lists the user’s teams with isDefault and an apiKey per team: the newest key that user created in the team, or a new key named CLI (<email>) minted when no decryptable user-owned key is available. The CLI stores that key and uses it as X-API-Key.
• /access-tokens answers 410 with the upstream deprecation message; E2B_ACCESS_TOKEN is not supported.

Client configuration for the login is in Connecting SDKs and the CLI.

Admin token

The admin token is the value of --admin-token (VMON_ADMIN_TOKEN) on a driver, compared in constant time against X-Admin-Token. It is not stored in the cluster, and each driver checks its own flag, so set the same value on every driver. Without --admin-token the admin routes reject every request with 401.

The admin routes are served on the public listener next to the E2B API. Operator workflows and curl examples are in Administration.

envd access token

Every sandbox has an envd access token: hex(HMAC-SHA256(seed, sandboxID)) with seed = HKDF(cluster secret, "envd-token"). It is deterministic, so paused and resumed sandboxes keep it, and it is always set (the deprecated secure create flag is ignored). The driver delivers it to the guest through envd /init; the SDKs read it from envdAccessToken and send it as X-Access-Token.

Inside the guest, envd requires X-Access-Token on every route except GET /health, GET/POST /files, and POST /init. /files accepts either the header or an SDK-style URL signature (signature, optional signature_expiration) computed from the token. The ingress answers 404 for envd’s control paths (/init and the pause hooks) on port 49983, so /init is not reachable from outside.

Sandbox self-keepalive (vibemon extension). The same token also authenticates three driver routes, for the sandbox that owns it only:

The token is compared in constant time before the sandbox is looked up; wrong tokens, killed sandboxes, and unknown sandboxes all get 401. Team blocking and the team’s maximum sandbox length still apply. It grants nothing else. Upstream E2B has no equivalent, so the SDKs do not use it; call it with raw HTTP from inside the sandbox:

# API: the driver's API URL; SBX: the sandbox ID; TOKEN: its envdAccessToken
curl -sX POST "$API/sandboxes/$SBX/timeout" \
  -H "X-Access-Token: $TOKEN" -H 'Content-Type: application/json' \
  -d '{"timeout": 600}'

Traffic access token

Sandboxes created with allowPublicTraffic: false get a traffic access token, hex(HMAC-SHA256(seed, "sandbox-traffic-" + sandboxID)) with the same seed as the envd token, returned as trafficAccessToken. The ingress then requires e2b-traffic-access-token on every port except envd’s 49983. Routing and the related options are covered in Domains, TLS, and ingress.

Volume tokens

POST /volumes and GET /volumes/{volumeID} return a volume token: an HS256 JWT with audience vmon-volume, scoped to one volume of one team, valid 24 hours. The volume-content API (/volumecontent/{volumeID}/path|dir|file) takes it as Authorization: Bearer; a missing or expired token is 401, a token for another volume is 403. Volumes themselves are covered in Storage, object store, and volumes.

Key material

All derived secrets come from the shared cluster secret (--cluster-secret, VMON_CLUSTER_SECRET; generated into <data-dir>/cluster-secret when unset, see Running vmon serve) through HKDF-SHA256 with a fixed label:

The internal cluster listener and gossip are authenticated with the cluster secret as well (Clusters).

Warning

Changing the cluster secret makes every stored ciphertext undecryptable and changes every derived token: envd and traffic tokens of existing sandboxes, outstanding JWTs, and signed upload URLs. API key lookups use the unencrypted hash and keep working. vmon admin encrypts the keys it mints with the secret it is given, so it must use the drivers’ secret.

Teams

A team is the tenant every API key, template, sandbox, volume, secret, and webhook belongs to. Teams are created with vmon admin create-team; there is no API route that creates one.

Members and the default team

Users join teams through vmon admin create-user --team or add-member. A membership can be the user’s default team; making one default clears the flag on the user’s other memberships. When none is flagged, the oldest membership counts as the default. Bearer requests without X-Team-ID, and the CLI’s team pick after login, use the default team.

Limits

Limits are stored on the team row and enforced by the driver:

Set them at creation or later with vmon admin set-limits.

Blocking

On team-authenticated API routes, a blocked team keeps read access but every mutating request (any method other than GET, HEAD, OPTIONS, TRACE) gets 403 team is blocked, whichever credential authenticated it. Running sandboxes are not stopped, and existing volume-content tokens are not checked against the block flag. Block and unblock with vmon admin block / unblock.

System teams

The SDKs default to the bare aliases base, code-interpreter-v1, desktop, and mcp-gateway. In vibemon only public templates of a system team hold bare aliases: publishing such a template claims each of its names that no other template holds as a bare alias, and every team resolves an unqualified reference as a template ID first if it has the ID shape, then in its own namespace, then among bare aliases. Ordinary teams’ public templates are addressed as <slug>/<alias>. Mark a team with vmon admin create-team --system or set-system; unset-system releases its bare aliases. The build pipeline for the default templates is in Templates.

Running vmon serve

vmon serve is the long-running server process. This page covers what each role needs, which ports must be reachable, what the server keeps on disk, and how to run it on a production host. Every flag is listed in vmon serve options.

Roles and topologies

A vmon serve process runs one or both roles, selected with --role (default node,driver):

Both topologies use the same binary and flags:

• Monolithic: one vmon serve --role node,driver process; its single-voter raft KV holds all state. No gossip seeds are needed.
• Distributed: any number of --role driver and --role node processes (or mixed ones) that share one cluster secret, replicate all state in the drivers’ raft KV, and find each other through gossip seeds (--join). See Clusters.

The node role runs on Linux with KVM (x86_64 and aarch64 guests) and on macOS 15+ on Apple Silicon with HVF (aarch64 guests). The macOS binary must be ad-hoc signed with hvf.entitlements. The driver role runs on both. Windows builds contain only vmon vmm. See Installation.

Listeners

Peers dial the advertised IPs (--advertise) on the two internal listener ports, not the bind addresses. When --advertise is unset, a listener bound to a specific IP advertises that IP, and the default 0.0.0.0 listeners advertise every interface IP except loopback, link-local, and container or VM bridges. Each peer prefers an address on its own LAN and falls back to the others (details in Cluster and gossip).

Cluster HTTP requires Authorization: Bearer <cluster secret>. Gossip packets carry an HMAC tag keyed from the same secret, and unauthenticated packets are dropped. Even so, keep both internal listeners on a private network, because the node API controls every VM.

GET /health on the public listener needs no authentication. It returns 204 while the driver can serve linearizable reads of the cluster KV (a raft leader is reachable and the local replica has caught up), so load balancers can use it as a health check.

Data and run directories

--data-dir defaults to /var/lib/vmon for root on Linux and to ~/.vmon otherwise. It holds:

node-id                     random 64-bit member ID, written on first start
cluster-secret              generated secret (0600) when --cluster-secret is unset
raft/                       drivers: raft log, vote, snapshots (fsync'd) of the KV holding all driver state
assets/                     nodes: pinned fallback kernel, when downloaded
egress/                     Linux nodes: node-local MITM CA for egress header rules
files/<templateID>/<hash>.tar   nodes: build file-context archives
cache/oci/                  nodes: pulled image layers
cache/steps/, cache/context/    nodes: build step cache, extracted COPY contexts
builds/<buildID>/           nodes: template artifacts (rootfs, snapshot, logs)
sandboxes/<sandboxID>/      nodes: disk overlay, console and metrics logs, live snapshot dir
paused/<sandboxID>/         nodes: paused sandbox state
volumes/<volumeID>/         nodes: volume contents (unless --volumes-dir)

Keep the data directory across restarts; a driver’s raft/ is its replica of every team, key, template, and sandbox record. Its node-id is the process’s identity in gossip, raft, and GET /nodes, and a restarted node adopts its running sandboxes only from the same directory. Node free disk is measured on the data directory. For the storage layout of disks, pauses, and volumes, see Storage, object store, and volumes.

--run-dir holds one short subdirectory per VM with its api.sock and agent.sock. It defaults to $XDG_RUNTIME_DIR/vmon, or to /tmp/vmon-<uid>. The path must stay short because of the 104-byte macOS sun_path limit.

Guest kernel and agent

At startup the node resolves the guest kernel and the guest agent:

• Kernel: --kernel, else vmon-kernel-<arch> next to the vmon binary (release assets), else target/kernel/ of a dev tree (just kernel). As a last resort the node downloads a pinned cloud-hypervisor kernel into <data-dir>/assets/. That kernel lacks netfilter, IPv6, bridge, and NFS, which breaks Docker-in-sandbox and templates that rely on those features, so vmon logs a warning when it falls back to it.
• Agent: --agent, else vmon-agent-<arch> next to the binary, else target/test-assets/ of a dev tree (just agent-musl). It must be a static ELF for the guest architecture. If no agent is found, the node does not start.
• GPU driver bundle: --gpu-driver, else vmon-nvidia-<arch>/ next to the binary, else target/kernel/ of a dev tree (just gpu-driver). Optional: without it, GPUs are listed but not assignable. See GPUs.

Keep the release assets next to vmon, or pass both flags explicitly. See Guest kernel.

Cluster secret

The cluster secret authenticates cluster HTTP and gossip. It is also the root of every derived key: JWTs, envd and traffic access tokens, signed upload URLs, and the AES key for API keys, env vars, secrets, and webhook secrets stored in the cluster KV.

• When --cluster-secret (VMON_CLUSTER_SECRET) is unset, vmon serve uses <data-dir>/cluster-secret, and generates the file on first start (64 hex characters, mode 0600) with a warning.
• Every process in a cluster must use the same secret. Generate it once and distribute it, for example by copying the first process’s cluster-secret file into VMON_CLUSTER_SECRET for the others.
• vmon admin needs the same secret to encrypt the API keys it creates. It reads --cluster-secret, or else <data-dir>/cluster-secret. See Administration.
• Changing the secret invalidates stored ciphertexts, JWTs, and derived envd and traffic tokens; stored refresh tokens are not derived from it.

Tip

Pass secrets through VMON_CLUSTER_SECRET and VMON_ADMIN_TOKEN instead of flags. --help hides these values, but command-line arguments show up in process listings.

Logging

Logs go to stderr. --log (VMON_LOG) takes a tracing env-filter, by default info,openraft=warn, and an invalid filter aborts startup. --log-json (VMON_LOG_JSON=true) switches to one JSON object per line.

VMON_LOG='info,openraft=warn,vmon_driver=debug' vmon serve --log-json

Each sandbox’s guest console and VMM log are kept under sandboxes/<id>/ in the data directory, not in the server log.

Shutdown and restart

vmon serve installs its SIGINT and SIGTERM handlers before it starts the roles, so a signal during startup still shuts down cleanly. On either signal:

1. The listeners stop accepting. In-flight public connections get up to 10 s to drain.
2. The node role stops its background tasks and removes idle pre-created TAPs. Sandbox VMMs keep running: they live in their own process group, and the next start on the same data directory adopts them.
3. Raft shuts down, and the process leaves gossip (state Left), waiting up to 2 s for the announcement.

If a listener fails, for example because its port is taken, the process exits with the error without running the graceful node and cluster shutdown sequence.

A node restarted with SIGTERM can adopt its surviving VMMs when it comes back within the leader’s 60 s grace period for Left nodes. On Linux, a sandbox with lazily restored memory loses its node-local page server and fails its next remote page fault after restart. A node that dies without leaving (SIGKILL, crash, network loss) goes Dead, and its sandboxes are marked killed. See Failure handling.

Linux privileges

• KVM: VMM processes need read-write access to /dev/kvm, including the uid:gid selected by --vmm-user; they open it after dropping privileges.
• TAP networking (--net tap, the Linux default via auto) needs root or CAP_NET_ADMIN. The node creates TAP devices with ip and installs NAT and egress rules with iptables, iptables-restore, ip6tables, and ip6tables-restore. With --net tap and no such privilege, startup fails. With auto, the node starts but rejects creates. See Networking and egress.
• Hugepages: --hugepage-pool-mb writes /proc/sys/vm/nr_hugepages, which needs root.
• VMM privilege drop: when vmon serve runs as root, set --vmm-user <uid>:<gid>. VMMs then drop to that uid and gid and run under their Landlock and seccomp filters. The node makes each VM’s run directory, disk directory, and snapshot directory, as well as its TAP device, accessible to that user. Without --vmm-user, a root node starts VMMs with --no-sandbox, so they keep root and run without filters.

Capacity

A node advertises its capacity in gossip, and drivers place sandboxes from it:

When no node has room, creates fail with 503 sandbox_capacity_unavailable. When no node matches the request’s architecture, features, or pin, they fail with 503 sandbox_no_compatible_node. Per-team quotas are separate; see Authentication and teams. Lower --max-vcpus and --max-memory-mb to reserve host resources for other workloads.

Hugepages

On Linux, sandbox RAM restored from a snapshot (create, resume, fork) can be backed by 2 MiB hugetlb pages. This requires room in the host hugetlb pool, userfaultfd WP_ASYNC and PAGEMAP_SCAN support (6.7+), permission to create userfaultfd, readable pagemap state, and a 2 MiB-aligned memory layout. If these prerequisites are unavailable, restore falls back to 4 KiB pages. Pages are filled on first touch from the memory file, and KVM maps them with 2 MiB stage-2 entries.

--hugepage-pool-mb N sets vm.nr_hugepages to N / 2 pages when the node starts. This sets the pool size, not an increment. The kernel can grant fewer pages on a fragmented host, and vmon logs what it got. The default 0 leaves the host setting alone, so a pool configured by other means is used as is.

The latency figures were measured under nested KVM (CHANGELOG, ARCHITECTURE.md §8.2).

Size the pool for the total RAM of the restored sandboxes you expect to run at once. A VM that the pool cannot cover falls back to 4 KiB pages. Surplus pages (vm.nr_overcommit_hugepages) also work, but can stall a create for seconds in direct compaction on a fragmented host.

Example: monolithic Linux host

One root process with both roles, HTTPS ingress on a wildcard domain, internal listeners on loopback, and VMMs dropped to an unprivileged uid. Place vmon-kernel-<arch> and vmon-agent-<arch> next to vmon, and point *.sbx.example.com and api.sbx.example.com at the host.

export VMON_ADMIN_TOKEN='<random admin token>'

vmon serve \
  --role node,driver \
  --data-dir /var/lib/vmon \
  --domain sbx.example.com \
  --listen 127.0.0.1:3000 \
  --tls-listen 0.0.0.0:443 \
  --tls-cert /etc/vmon/tls/fullchain.pem \
  --tls-key /etc/vmon/tls/privkey.pem \
  --cluster-listen 127.0.0.1:7947 \
  --gossip-listen 127.0.0.1:7946 \
  --vmm-user 64000:64000 \
  --hugepage-pool-mb 32768 \
  --registry-mirror https://mirror.gcr.io \
  --log-json

On first start this generates /var/lib/vmon/cluster-secret. Run vmon admin on the same host while the server runs (it reads that file and reaches the driver at http://127.0.0.1:7947 by default) to create the first team. With --tls-listen, --public-url defaults to https://api.sbx.example.com. Clients need only E2B_API_KEY and E2B_DOMAIN=sbx.example.com; see Connecting SDKs and the CLI.

Domains, TLS, and ingress

Each driver serves the E2B API and proxies sandbox traffic on the same public listener. This page covers the sandbox domain, how requests are routed, TLS and DNS setup, and the per-sandbox ingress settings. You need it when you expose vibemon beyond localhost. Client-side configuration is in Connecting SDKs and the CLI.

Sandbox URLs

--domain (VMON_DOMAIN, default localhost) is the sandbox domain:

The router never requires the api. host: any request that does not address a sandbox goes to the API. Sandbox API responses carry domain with the --domain value, which the SDKs use to build sandbox hosts. Sandbox IDs never contain -, so the first - in the left-most label separates the port from the ID.

Guest ports are reached through a GC4 tunnel to the guest agent, never through the guest network. Services bound to the guest’s 127.0.0.1 are reachable, and no inbound NAT or firewall rule is needed on the node.

Routing

A driver dispatches every public request in this order:

1. Host: the left-most label of Host (HTTP/1.1) or :authority (HTTP/2) is <port>-<id>, where the port is a number up to 65535 and the ID consists of lowercase letters and digits. The request goes to ingress. Labels after the ID (8080-<id>-<anything>.…) are ignored, and so is the rest of the host name: any domain works. IP literals and hosts without a dot never match.
2. Headers: both E2b-Sandbox-Id and E2b-Sandbox-Port are present. The request goes to ingress, whatever its Host. A host match takes precedence over the headers.
3. Otherwise: the request goes to the E2B API router.

Header routing lets a client send sandbox traffic to a single address, such as http://127.0.0.1:3000, without wildcard DNS. This is what E2B_SANDBOX_URL does (see Connecting SDKs and the CLI). Malformed routing headers get a plain-text 400: missing header when only one of them is set, Invalid sandbox port, or Invalid sandbox ID.

Any driver serves any sandbox. It reads the sandbox’s route (sbx/<id>) from its local raft KV replica, finds the owning node, and opens a tunnel to the guest port over that node’s --cluster-listen. A load balancer can therefore spread public traffic across all drivers with no stickiness.

TLS

--tls-listen opens an HTTPS listener with the same router as --listen. It needs --tls-cert (PEM chain) and --tls-key (PEM key), and offers h2 and http/1.1 through ALPN. --listen stays open alongside it, serving plain HTTP/1.1 and h2c, and cannot be disabled. Bind it to loopback or firewall it if all clients should use HTTPS.

The certificate must cover *.<domain> (sandbox hosts) and api.<domain>. The verification setup also adds the bare <domain>. vibemon does not obtain or renew certificates: it reads the files once at startup, so restart the driver after renewing.

vmon serve --role driver \
  --domain sbx.example.com \
  --tls-listen 0.0.0.0:443 \
  --tls-cert /etc/vmon/tls/fullchain.pem \
  --tls-key /etc/vmon/tls/privkey.pem

With TLS on port 443 and DNS in place, the SDKs need only E2B_API_KEY and E2B_DOMAIN. They then address https://<port>-<id>.<domain> directly.

Note

The Python SDK volume content API (Volume.read_file, …) ignores E2B_API_URL. It defaults to https://api.<domain> outside debug mode, but accepts E2B_VOLUME_API_URL or a per-operation api_url override, so a plain-HTTP listener can also be used.

To terminate TLS on an external load balancer instead, forward both api.<domain> and *.<domain> to --listen with the original Host header, and set --public-url. WebSocket upgrades must pass through the load balancer.

Public URL

--public-url is the base URL clients use to reach this driver. The driver puts it in:

• signed template file upload URLs (GET /templates/{id}/files/{hash} returns a PUT …/upload?expires=…&signature=… URL, valid for 30 minutes and authenticated by its signature alone);
• tokenEndpoint and revokeEndpoint of e2b auth login.

The default is https://api.<domain> with --tls-listen, and http://<first advertised IP>:<listen port> otherwise (see --advertise in vmon serve). Set it explicitly when clients reach the driver through another host, a port other than 443, or a load balancer. Otherwise template builds that upload files fail from outside. In a cluster, each driver advertises its own public URL.

DNS

Point these names at the drivers, or at a load balancer in front of them:

api.sbx.example.com.   A     203.0.113.10
*.sbx.example.com.     A     203.0.113.10

For local testing without DNS, use --domain localhost with header routing, or the resolver approach in Distributed, KVM, and TLS verification.

Traffic access tokens

A sandbox created with network.allowPublicTraffic: false (Python: network={"allow_public_traffic": False}) accepts requests on its ports only with an e2b-traffic-access-token header. The create response returns the token as trafficAccessToken. It is derived from the cluster secret and the sandbox ID, so it stays the same across pause and resume and is accepted by every driver.

from e2b import Sandbox
import httpx

sbx = Sandbox.create(network={"allow_public_traffic": False})
sbx.commands.run("python3 -m http.server 8080", background=True)
r = httpx.get(f"https://{sbx.get_host(8080)}",
              headers={"e2b-traffic-access-token": sbx.traffic_access_token})

• Port 49983 (envd) is exempt. envd checks its own X-Access-Token.
• A missing or wrong token gets 403. The comparison is constant-time, and a private sandbox without a stored token refuses every request.
• The default is true: sandbox ports are public to anyone who knows the URL.

Host masking and HTTPS ports

Both are set at create time in the sandbox’s network config, and apply to every port except 49983:

Proxy behaviour

• Protocol: requests reach the guest as HTTP/1.1 over the tunnel, whatever the client spoke. Bodies stream in both directions without buffering. Hop-by-hop headers are removed. If the client disconnects, the guest connection closes.
• WebSockets and upgrades: an HTTP/1.1 Upgrade request is passed through. After the guest answers 101, bytes are spliced in both directions.
• Dial retry: when the guest port refuses the connection, the proxy makes up to 5 attempts with a 100–400 ms linear backoff between them before it answers 502.
• CORS: API and sandbox responses allow any origin. The driver answers API preflight requests itself (cached for 86400 s). Sandbox preflights go to the guest, and the driver answers them only when routing fails. Upgrade responses (101) get no CORS headers.
• Blocked envd paths: on port 49983, /init, /freeze, /unfreeze, /collapse, /fsfreeze, /fsthaw, and /upgrade return 404 for every method. These paths are reserved for the control plane. Paths are matched after percent-decoding and cleaning, so /%69nit and /a/../init are blocked too.

Errors are JSON ({"sandboxId", "message", "code"}), or a small HTML page when a browser navigates to the URL:

Auto-resume

Traffic to a paused sandbox resumes it when the sandbox was created with auto-resume (lifecycle={"on_timeout": "pause", "auto_resume": True} in Python). The rules:

• A sandbox that is pausing with auto-resume enabled is resumed once the pause completes.
• While another request is resuming the sandbox, ingress polls every 200 ms for up to 1 minute.
• A paused sandbox without auto-resume, or a killed one, gets 502 The sandbox was not found.
• The resumed sandbox gets the timeout remembered at create time, raised to at least 300 s and capped at the team’s maximum sandbox length.
• A pause ends every proxied stream to the sandbox (open command or log streams, long downloads). Ingress keeps such a response open, sending nothing, while the sandbox is pausing or paused, and reports the broken stream only once the sandbox runs again, is killed, or after 620 s. Without this, the SDKs react to the failed stream with an envd health check through the ingress, which would auto-resume the sandbox right after it paused.

Resume may move a sandbox to another node when its own node is gone and an object store is configured; see Storage, object store, and volumes. Lifecycle semantics are in Sandboxes.

Clusters

A distributed deployment runs separate driver and node processes that find each other through gossip. The drivers replicate all state, live and durable, in a raft KV; there is no external database. This page covers how to start such a cluster, how work is placed, and what happens when processes fail. For a single host, see Running vmon serve.

Shape

            clients (SDKs, CLI, browsers)
                       │  --listen / --tls-listen
       ┌───────────────┼───────────────┐
   driver d1       driver d2       driver d3     raft voters, E2B API, ingress
       │ ╲             │             ╱ │
       │   ╲  raft KV (all state)      │
       │                               │
   ──────── gossip :7946 (every process) ────────
   ──────── cluster HTTP :7947 (raft, KV, node API, tunnels) ────────
       │               │               │
    node n1         node n2   …     node nN       VMs, builds, artifacts

All processes need the same --cluster-secret. All drivers should share --domain, --admin-token, --shared-volumes, and --object-store so that every driver behaves the same.

Gossip

Every process joins gossip on --gossip-listen (default 0.0.0.0:7946, UDP and TCP). Pass the seeds with --join host:port,host:port. Seed host names are resolved through DNS, and the process’s own address is skipped, so you can pass the same seed list to every process. Listing all drivers is a good choice. If no seed answers at startup, the process logs a warning and retries every 2 s until one does.

Peers reach a process at its advertised IPs, on the --gossip-listen port for gossip and the --cluster-listen port for cluster HTTP. With the default wildcard listeners, a process advertises every IP of its interfaces (LAN, VPN such as Tailscale, public), leaving out loopback, link-local, and container or VM bridges. A listener bound to a specific IP advertises only that IP, and --advertise ip,ip,… sets the list explicitly, for example behind NAT (details in Advertise addresses).

Each process picks its own route to every peer from that peer’s list: a shared subnet (LAN) first, then other private addresses, then overlay networks, then public addresses. When that address stops answering, it falls back to another one, so a peer is lost only when none of its addresses works. Transfers between nodes (template artifacts, build file contexts) follow the fetching node’s route, so two nodes on the same LAN copy over the LAN even when the driver reaches them over a VPN. Every member also publishes its host name, which the dashboard shows next to the node ID.

Failure detection: each member probes one peer per second. An unanswered probe (500 ms timeout, then 3 indirect probes) marks the peer Suspect, and it becomes Dead after 5 × max(1, ⌈log10(n+1)⌉) seconds. A process that shuts down cleanly announces Left instead. Every 30 s, members exchange full state with a random peer (and with a random dead one), which heals partitions. Packets are authenticated with an HMAC keyed from the cluster secret, and unauthenticated ones are dropped.

Raft

Drivers are the raft voters. Raft state lives in <data-dir>/raft/: every driver holds the KV in memory and snapshots it there. History (sandbox events, webhook deliveries, build logs) is not kept in memory: every driver appends it to an on-disk journal in <data-dir>/raft/journal/, replicated through raft and kept for 30 days. Records of killed sandboxes stay in the KV for 7 days.

• Bootstrap: start the initial drivers with --bootstrap-expect N, where N is the number of initial drivers. A driver with uninitialized raft waits until gossip shows N alive drivers that have no cluster ID yet. The driver with the smallest node ID among them then initializes raft with exactly those N voters.
• Joining later: a driver that sees any peer with a cluster ID never bootstraps. The leader reconciles membership every 2 s, adding new alive drivers as learners and promoting them to voters once they have caught up.
• Removal: a voter that has been Dead or Left in gossip for longer than --raft-reap-after seconds (default 600) is removed. The voter set never drops below 1.
• Node-only processes run no raft. Their KV calls go to a live driver over cluster HTTP, and they retry the next driver when a connection fails. Followers forward writes and consistent reads to the leader.

Important

Drivers that use different openraft versions cannot replicate to each other. This release moved from openraft 0.9 to 0.10, which changed how snapshots are sent between drivers. To upgrade a cluster with several drivers from an earlier release, stop every driver, then start all of them on the new version. Their raft/ directories stay valid. You don’t need to restart node-only processes at the same time.

Warning

--bootstrap-expect defaults to 1. A lone driver with the default bootstraps as soon as it sees itself, so if several drivers start with the default before gossip connects them, more than one raft cluster can form. Give every initial driver the same --bootstrap-expect. It only matters while raft/ is empty.

Raft needs a majority of voters to elect a leader and commit writes, so three drivers tolerate the loss of one. KV calls wait up to about 8 s for a reachable leader and then fail. Ingress reads sandbox routes from the driver’s local replica. API reads of durable records are linearizable on every driver: a follower first fetches the leader’s applied revision and waits until its replica has applied it. Writes are optimistic KV transactions, guarded on the revisions of the records they read and rerun on a conflict.

Placement

The driver that handles a create picks the node (driver/src/scheduler.rs):

1. Filter: alive node members only, excluding nodes that are draining or unhealthy. The node’s architecture must match the build’s, and it must have the required features: egress-proxy for egress rules and egressProxy, egress-domains for domain allow entries when no proxy is needed, and object-store for object-store volumes and relocated resumes. A volume on node-local storage (no --shared-volumes, no object store) pins the sandbox to the node that holds the volume. Capacity: allocated vCPUs up to 4 × advertised CPUs, allocated memory up to 2 × advertised memory, and at least 512 MiB free disk.
2. Score: free vCPU and memory headroom. Add 1.0 when the node already holds the build’s artifacts, and another 0.5 when its snapshot is reusable there (same backend and CPU baseline, so the create is a copy-on-write fork). Subtract 0.25 per create already in flight on that node from this driver.
3. Pick: two random candidates from the top half; the one with the higher score wins. If the node is busy or unreachable, the driver tries the next candidate, up to 3 attempts in total.

When nothing fits, the create fails with 503: sandbox_no_compatible_node when no node matches, or sandbox_capacity_unavailable when matching nodes are full. Template builds go to alive nodes that are not draining, with the parent template’s architecture, else the majority architecture (ties go to x86_64). Their score is free memory (MiB) plus 1024 per free advertised vCPU, minus 2048 per running build; higher scores come first. Advertised capacity is set with --max-vcpus and --max-memory-mb; see Capacity.

Node status

Node status combines gossip liveness with an admin override stored in KV (node/<id>/status). GET /nodes and GET /nodes/{nodeID} return it. Both require X-Admin-Token, and the detail view also lists the node’s sandboxes and cached builds.

Set the override with POST /nodes/{nodeID} on any driver:

curl -X POST "https://api.$DOMAIN/nodes/$NODE_ID" \
  -H "X-Admin-Token: $VMON_ADMIN_TOKEN" -H 'Content-Type: application/json' \
  -d '{"status": "draining"}'        # 204; "ready" removes the override

Only ready, draining, and unhealthy can be set. Other values get 400, and an unknown node gets 404. While a node is draining and alive, the raft leader evacuates it every 5 s: it migrates the node’s running sandboxes that have more than a minute left (two at a time per node) to compatible ready nodes — same architecture, backend, and CPU baseline, enough capacity, the needed features, no volume pin to the draining node. The migration pauses the sandbox on the draining node, the target pulls the paused state directly from it over the cluster listener (only written disk extents and dirtied memory; template files the target already has are linked, not transferred), resumes it with the same ID, and the route moves; connections open during the move drop. A sandbox without a compatible node stays and is retried after a minute. Paused sandboxes held only on the draining node (no object store) are moved the same way and stay paused; with an object store, their background upload already lets any compatible node resume them. unhealthy stops placements without evacuating. For the operator workflow, see Administration.

Failure handling

The raft leader runs the cluster-wide loops: the reaper (every 1 s: timeouts and auto-pause, and sandboxes that a dead driver left mid-transition), the reconciler (every 10 s), housekeeping and artifact GC (every 60 s), and webhook delivery. Build supervision is spread across drivers through 20 s KV leases.

Distributed, KVM, and TLS verification describes the gated test that exercises these cases with 3 drivers and 2 nodes.

Artifacts and portability

Each node keeps template build artifacts (rootfs and memory snapshot) in <data-dir>/builds/, and drivers record which nodes hold each build (art/<buildID>/<nodeID>). When a sandbox lands on a node without the build, the node fetches it from a holder over cluster HTTP. The transfer is extent-aware, so sparse disks stay sparse, and it is verified by SHA-256. A same-architecture node whose backend or CPU baseline differs from the holder’s fetches only the rootfs and regenerates the template snapshot locally.

Without --object-store, all state is tied to nodes: builds survive only while some node holds them, and a paused sandbox resumes only on its own node (503 while that node is unavailable). With --object-store on every node and driver:

• finished builds are also published to the store, so sandboxes can start from a template after every node that held it is gone;
• paused sandboxes are uploaded in the background and resume on another compatible node when their own node is gone or draining (a memory restore with the same backend and CPU baseline, else a rootfs-only cold boot);
• volumes live in the store and mount on any node.

Details are in Storage, object store, and volumes.

Example: 3 drivers and N nodes

Hosts: drivers 10.0.0.1–10.0.0.3, nodes 10.0.1.1…10.0.1.N, and an S3 bucket. Generate the cluster secret once (for example openssl rand -hex 32) and set the same environment on every host:

export VMON_CLUSTER_SECRET='<64 hex chars>'
export VMON_JOIN='10.0.0.1:7946,10.0.0.2:7946,10.0.0.3:7946'
export VMON_OBJECT_STORE='s3://vmon-state/prod'    # credentials from the AWS environment or role

Each driver (here 10.0.0.1):

export VMON_ADMIN_TOKEN='<admin token, same on every driver>'

vmon serve --role driver \
  --bootstrap-expect 3 \
  --cluster-listen 10.0.0.1:7947 \
  --gossip-listen 10.0.0.1:7946 \
  --domain sbx.example.com \
  --tls-listen 0.0.0.0:443 \
  --tls-cert /etc/vmon/tls/fullchain.pem \
  --tls-key /etc/vmon/tls/privkey.pem

Each node (here 10.0.1.1, running as root on Linux/KVM):

vmon serve --role node \
  --cluster-listen 10.0.1.1:7947 \
  --gossip-listen 10.0.1.1:7946 \
  --vmm-user 64000:64000

Binding the internal listeners to the host’s private IP also makes it the only advertised address; with the default 0.0.0.0 listeners, the node advertises all its interface IPs instead. Put the drivers’ public listeners behind DNS or a load balancer for api.sbx.example.com and *.sbx.example.com (see Domains, TLS, and ingress). Then check that GET /nodes on any driver lists every node as ready:

curl -s https://api.sbx.example.com/nodes -H "X-Admin-Token: $VMON_ADMIN_TOKEN"

Run vmon admin against the drivers’ cluster listeners with the same VMON_CLUSTER_SECRET:

VMON_CLUSTER_URL=http://10.0.0.1:7947,http://10.0.0.2:7947,http://10.0.0.3:7947 vmon admin list-teams

Networking and egress

How sandboxes reach the network on each platform, and how vibemon enforces the E2B egress policy (allow_internet_access, network.allowOut/denyOut, rules, egressProxy). Read this before running nodes on Linux or before giving sandboxes a restricted network.

Flag reference: vmon serve options.

Inbound traffic never uses the guest network: the driver’s ingress reaches guest ports through tunnels over the agent channel, so services bound to localhost in the guest work and no inbound NAT exists on any platform. See Domains, TLS, and ingress.

Network modes

vmon serve --net (VMON_NET) selects how a node connects its VMs.

On Linux, --net auto without root or CAP_NET_ADMIN starts the node with a warning (no CAP_NET_ADMIN: this node cannot network sandboxes and will reject creates) and every create on it fails. --net tap without the capability refuses to start. Nodes advertise net-tap or net-user in their gossip features; an auto node without net-admin advertises neither.

Guests run no DHCP client: after every boot, fork, and resume the node pushes the address over the agent channel.

Linux: TAP and NAT

Each sandbox gets a TAP device vmon<N> with a /30 lease from --net-pool (VMON_NET_POOL, default 172.20.0.0/16). The pool prefix must be between /8 and /30 with no host bits set; a /16 holds 16384 leases. Leases are handed out round-robin, so a just-released block is reused as late as possible.

Setting up a TAP:

• creates the device (owned by the --vmm-user uid when VMMs drop privileges), assigns the host address, brings it up, and attempts to disable IPv6 on it (write errors are ignored);
• writes 1 to /proc/sys/net/ipv4/ip_forward;
• installs, in one iptables-restore --noflush transaction per table: nat POSTROUTING MASQUERADE for the lease, filter FORWARD accept rules for the guest, and mangle PREROUTING CONNMARK --set-mark <index+1> tagging every guest flow;
• installs the egress hooks with closed policy chains (below).

The node needs ip, iptables, iptables-restore, ip6tables, and ip6tables-restore on the host.

Ready TAP pool

After restart recovery, the node removes idle vmon<N> TAPs that no recovered sandbox holds, then pre-creates one ready TAP per boot slot (--boot-concurrency), at most 32, and refills the pool in the background. Creates and resumes take a ready TAP instead of running ip/iptables on the request path.

When a sandbox’s VM exits, its TAP’s policy chains are reset to closed, the guest’s conntrack entries are flushed (one ctnetlink delete filtered by the lease’s connection mark; conntrack-tools is not needed), and the TAP returns to the pool while the pool holds fewer than twice its target. Otherwise it is torn down: egress rules, NAT rules, and the device are removed and the mark’s conntrack entries flushed. Node shutdown removes the idle TAPs.

Flushing conntrack on reuse keeps NAT mappings and established UDP state of the previous sandbox from leaking to the next one on the same address.

Egress hooks

Every pooled TAP carries these rules from creation, tagged -m comment --comment vmon-egress:

The policy chains are closed (a single DROP) while no sandbox is attached, so a resumed sandbox cannot reach the network before its policy is in place. Attaching a sandbox is one add-only iptables-restore that inserts its policy above the DROP and runs while the guest configures its network; a live update replaces the chains atomically; detaching resets them to closed in the background.

Template build VMs get a fresh TAP without egress hooks: builds have unrestricted egress.

macOS: user-mode NAT

The VMM runs libslirp and NATs guest traffic through the host’s sockets; networking itself needs no extra privileges or entitlements (the VMM still needs its Hypervisor.framework entitlement). Every user-net sandbox VMM gets a compiled packet filter (vmon vmm --user-net-policy <file>, written to sandboxes/<id>/net-policy.json), replaced live through the VMM control method set_net_policy. Even with an empty policy the filter blocks the private ranges, which include the slirp gateway 10.0.2.2 (the host’s loopback).

The filter admits a new flow by the first matching rule:

1. DHCP to slirp, and DNS to 10.0.2.3:53 unless the policy denies 8.8.8.8.
2. TCP to port 80 or 443 of an address learned from a slirp-DNS answer for a name matching an allowOut domain, unless that address is in an always-denied range.
3. Always-denied ranges → drop, unless exempted by --egress-internal-allow.
4. allowOut CIDRs → pass.
5. denyOut CIDRs → drop.
6. Pass.

TCP is judged on the SYN; established flows keep passing after a policy update, so updates affect only new connections. UDP and ICMP are judged per packet. Learned addresses expire with their DNS TTL, clamped to 60 s–1 h, in a table capped at 4096 entries. Addresses resolved through any other resolver (DNS over HTTPS, 8.8.8.8, …) never unlock a domain.

Egress policy

The E2B API carries the policy; the driver validates and normalizes it and the node enforces it. It applies from the first instruction the sandbox runs after create, fork, or resume, and follows the sandbox through pause, resume, and resume on another node.

Matching

• IP / CIDR: IPv4 or IPv6; a bare address becomes /32 or /128. In denyOut, an unspecified address (0.0.0.0, ::) is only accepted as the literal 0.0.0.0/0 (the SDKs export it as ALL_TRAFFIC).
• Domains (allowOut only): example.com (exact), *.example.com (strict subdomains at any depth, not the apex), or * (every hostname). Matching is case-insensitive. A domain is recognized from the HTTP Host header on port 80 and the TLS SNI on port 443; on other ports only CIDR rules apply.
• allowOut with any domain requires denyOut to contain 0.0.0.0/0; otherwise the request fails with 400.
• When allowOut has domains, 8.8.8.8 is implicitly allowed so DNS works under deny-all.

Precedence for a TCP flow:

1. Allowed domain → allowed. The flow is re-dialed by hostname from the host, and a resolved address in an always-denied range is never dialed.
2. Destination in an always-denied range → denied.
3. allowOut CIDR → allowed.
4. denyOut CIDR → denied.
5. Otherwise allowed.

UDP and ICMP follow steps 2–5.

Always-denied ranges

By default, sandboxes cannot reach these destinations:

10.0.0.0/8      100.64.0.0/10   127.0.0.0/8     169.254.0.0/16
172.16.0.0/12   192.168.0.0/16  ::1/128         fc00::/7        fe80::/10

vmon serve --egress-internal-allow (VMON_EGRESS_INTERNAL_ALLOW, comma-separated CIDRs) exempts ranges on that node, for example an internal package mirror:

vmon serve --role node --egress-internal-allow 10.20.0.0/16,192.168.5.10/32

The exemption only lifts the always-denied block; the sandbox’s own denyOut still applies.

HTTPS header rules

rules injects or overrides request headers on HTTPS requests to matching hosts. Keys are exact names or a single leading wildcard (*.example.com); a bare * is rejected, the most specific key wins, and matching rule sets are not merged. Rules do not grant access: the destination must also be allowed.

On Linux, a TLS flow to port 443 whose SNI matches a rule is intercepted: the proxy presents a leaf certificate minted for that name by the node’s CA, opens verified TLS to the real origin, and proxies HTTP/1.1 (ALPN http/1.1) with the headers applied to every request. Upgrade requests (WebSocket) are spliced through after the 101.

The CA is created once per node in <data-dir>/egress/ (ca.pem, ca-key.pem). Every sandbox on a Linux node receives it in envd /init, which appends it to whichever of these guest trust bundles exist (else creates the first):

/etc/ssl/certs/ca-certificates.crt
/etc/pki/tls/certs/ca-bundle.crt
/etc/ssl/cert.pem
/etc/ssl/ca-bundle.pem

API limits: at most 10 rule domains, each 1–128 characters; one rule per domain; at most 20 headers per rule; header names ≤ 64 characters, values ≤ 2048.

SOCKS5 egress proxy

egressProxy tunnels every allowed outbound TCP flow through a SOCKS5 proxy you operate, after allow/deny filtering. Domain-allowed flows are sent to the proxy by hostname (remote DNS); others by address. UDP and ICMP are not proxied. tls.enabled wraps the hop to the proxy in TLS, verified against tls.caCert (PEM, ≤ 8 KiB) instead of the system roots when set; tls.serverName overrides the verified name.

Validation: address must be host:port; username and password ≤ 255 bytes; password requires username; serverName/caCert require enabled: true. GET /sandboxes/{id} returns the network config without the proxy password and CA.

Platform support

Placement only considers nodes with the features a policy needs: egress-proxy for rules or egressProxy, egress-domains for domain entries. When no such node exists, create fails with 503 and error_code sandbox_no_compatible_node. A live update that needs a feature the sandbox’s node lacks fails with 400.

SDK examples

Deny everything except two package hosts:

from e2b import ALL_TRAFFIC, Sandbox

sbx = Sandbox.create(
    network={
        "allow_out": ["pypi.org", "*.pythonhosted.org"],
        "deny_out": [ALL_TRAFFIC],
    },
)

import { ALL_TRAFFIC, Sandbox } from 'e2b'

const sbx = await Sandbox.create({
  network: {
    allowOut: ['pypi.org', '*.pythonhosted.org'],
    denyOut: [ALL_TRAFFIC],
  },
})

No internet access:

sbx = Sandbox.create(allow_internet_access=False)

const sbx = await Sandbox.create({ allowInternetAccess: false })

Inject a header and tunnel through a SOCKS5 proxy (Linux nodes only):

sbx = Sandbox.create(
    network={
        "rules": {
            "api.example.com": [
                {"transform": {"headers": {"Authorization": "Bearer <token>"}}},
            ],
        },
        "egress_proxy": {
            "address": "proxy.example.com:1080",
            "username": "proxy-user",
            "password": "proxy-password",
        },
    },
)

const sbx = await Sandbox.create({
  network: {
    rules: {
      'api.example.com': [
        { transform: { headers: { Authorization: 'Bearer <token>' } } },
      ],
    },
    egressProxy: {
      address: 'proxy.example.com:1080',
      username: 'proxy-user',
      password: 'proxy-password',
    },
  },
})

Live updates

PUT /sandboxes/{id}/network replaces the egress policy of a running sandbox (204; 409 unless running). It is a full replacement: omitted fields (allowOut, denyOut, rules, egressProxy) are cleared, while ingress settings (allowPublicTraffic, maskRequestHost, httpsPorts) are kept. The body takes allow_internet_access in snake case, as on create.

sbx.update_network({"allow_out": ["github.com"], "deny_out": [ALL_TRAFFIC]})

await sbx.updateNetwork({ allowOut: ['github.com'], denyOut: [ALL_TRAFFIC] })

On Linux the node swaps the TAP’s policy chains in one transaction and the proxy’s policy for the guest. On macOS it sends set_net_policy to the VMM; established TCP flows survive.

Storage, object store, and volumes

Where a node keeps templates, sandbox disks, and paused state; what the optional object store (--object-store) adds; and how E2B volumes and bring-your-own-bucket mounts (vmonS3Mounts) are stored. Read this when sizing node disks, enabling portable pauses, or choosing a volume backend.

Flag reference: vmon serve options, object store credentials.

Data directory

vmon serve --data-dir (VMON_HOME) defaults to /var/lib/vmon when running as root on Linux, else ~/.vmon. Per-VM Unix sockets live separately under --run-dir (VMON_RUN_DIR, default <runtime dir>/vmon or /tmp/vmon-<uid>), one short directory per VM, to stay under the 104-byte macOS socket path limit.

<data-dir>/
├── node-id
├── cluster-secret              generated when --cluster-secret is unset
├── raft/                       raft log and state (drivers)
│   └── journal/                history journal: sandbox events, build logs, webhook deliveries (30 days)
├── files/<templateID>/<hash>.tar   build file-context uploads (stored on nodes)
├── cache/
│   ├── oci/blobs/sha256/<hex>  pulled image layers, shared by builds
│   ├── steps/<layerKey>/       build step cache layers
│   └── context/<templateID>/<hash>/  extracted COPY contexts
├── builds/<buildID>/           template artifacts: rootfs, snapshot/, meta.json, build.log
├── sandboxes/<sandboxID>/      running sandbox: disk overlay, logs, metrics, metadata
├── paused/<sandboxID>/         paused sandbox: memory snapshot + disk overlay
├── volumes/<volumeID>/         node-local volumes (unless --volumes-dir)
├── assets/                     fallback guest kernel download
└── egress/                     egress interception CA (Linux)

Other nodes fetch a build’s artifacts from a node that holds it when they first create a sandbox from it; transfers send only data extents, so destinations stay sparse.

Sandbox disks

A sandbox disk is a block-level copy-on-write overlay over the template’s raw rootfs, independent of the host filesystem:

• rootfs.img is a sparse overlay: written 4 KiB clusters live at their own offset, everything else is read from the base. An allocation bitmap and footer sit past the disk end.
• rootfs.img.base is a hard link to the template’s rootfs.ext4. Bases are never written. The link count keeps a base alive after its template is deleted.
• An overlay of an overlay (resume, creates from snapshot templates) clones the source’s data and bitmap onto the same raw base, so chains never form. Clones reflink when the filesystem supports it (XFS, btrfs on Linux; fclonefileat on macOS), else copy only the written extents.
• Guest flushes skip the host fsync (the disk is private to its VMM). The node fdatasyncs a paused disk before marking the pause durable.

Local raw-base overlay operations do not copy the template rootfs, including on ext4. Remote resume and materializing a lazy disk can fetch and write the full disk. Hard links require builds/, sandboxes/, and paused/ to be on one filesystem; they all live under the data directory.

Sizing

A template built from an image gets its rootfs as a sparse ext4 image of the building team’s max_disk_mb limit (default 20480 MiB; set with vmon admin set-limits --max-disk-mb). Its sandboxes see that size; host usage grows with written data only: the template’s allocated blocks once, plus each sandbox’s written clusters. Delta paused memory snapshots add the pages written since the fork (below). Nodes report the free space of the data directory in their capacity (disk_free_mb).

Pause artifacts

A pause snapshots the VM and moves its state into paused/<id>/:

• Memory: snapshot/. When the sandbox has a local RAM base (for example, after a template fork or a non-lazy memory resume), the snapshot is a delta against that base (the memory file it was forked from, hard-linked as base/): only pages that differ from the base are stored, so the cost scales with memory written since the fork, not guest RAM. Lazily restored Linux memory has no local RAM base, so its next pause writes a full memory snapshot.
• Disk: the sandbox’s rootfs.img overlay and its base link.
• Commit: the node fdatasyncs the overlay, fsyncs the directory entries, then creates committed. Restart recovery keeps a paused sandbox only with that marker, so memory and disk are always a durable pair.

A resume forks from paused/<id>/snapshot with a new overlay cloned from the paused disk. pause with memory=false keeps only the disk; the resume cold-boots it.

Object store

vmon serve --object-store (VMON_OBJECT_STORE) points the cluster at an S3-compatible bucket. It enables portable paused sandboxes, template publishing, and object-store volumes. Configure the same store on every node and driver: nodes advertise the object-store feature, and drivers use the store for volumes, the volume-content API, and cleanup on kill.

The location cannot contain credentials, a query, or a fragment; s3:// locations cannot carry a port.

Without static keys the node uses the ECS task role or the EC2 instance role and refreshes the credentials before they expire.

VMON_S3_ENDPOINT=https://minio.internal:9000 \
VMON_S3_ACCESS_KEY=vmon VMON_S3_SECRET_KEY=… \
vmon serve --object-store s3://vmon/cluster-a

Object layout under the store prefix:

Published templates

After every build commit (build ready, peer fetch, snapshot template, fork template) the holder publishes the build to builds/<buildID>/ unless it is already there. A node that cannot fetch a build from a live peer fetches it from the store with the same verification, and the driver accepts creates for a build with no live holder when its published manifest exists. Templates therefore survive the loss of every node that held them. Deleting a template also deletes its published build.

Portable paused sandboxes

After a pause commits, the node uploads one generation in the background. The RAM base is uploaded with it unless it is the template’s own snapshot and that build is already published. Only after manifest.json is written does the sandbox’s KV record point at the generation. A failed upload leaves the local pause usable; restart recovery re-uploads a committed pause that has no record. Kill deletes the generation and its record.

Resume prefers the node that holds the pause while it is alive. Otherwise the driver picks an object-store node of the same architecture:

• same backend and CPU baseline: memory and disk are restored, processes keep running;
• otherwise: rootfs-only cold boot on a same-arch node; guest processes are lost, the disk is kept.

Sandboxes with volumes on node-local storage stay pinned to that node. Without an object store, pauses stay node-local and a sandbox whose node is gone cannot be resumed.

Lazy resume

--lazy-resume (VMON_LAZY_RESUME, default true) lets a node resume another node’s paused sandbox without downloading it first. The node fetches only metadata, the snapshot state, and the block index, pins the rootfs.blk ETag, and boots with a remote disk: the VMM fetches each 1 MiB frame by HTTP range on first touch into a local sparse cache, and guest writes go to a private overlay.

The next pause materializes the full disk while the VM is paused, then deletes the lazy files and the consumed generation. Lazy preparation or boot failures fall back to the full download; later remote I/O failures do not. --lazy-resume false downloads the required paused state eagerly before boot; cold boots omit saved memory.

Warning

Lazy resume limits: the pre-signed block URL expires after 7 days, after which blocks touched for the first time fail with I/O errors; a node restart orphans a Linux page server, so the adopted VM dies on its next remote page fault; block fetches stall the VM’s device worker while they run.

Volumes

The E2B Volumes API (/volumes, volume-content API under /volumecontent/{id}/…) works on every deployment. Where a volume lives is fixed when it is created:

--shared-volumes (VMON_SHARED_VOLUMES) declares that --volumes-dir (VMON_VOLUMES_DIR) is shared storage visible to every node at the same path; set --volumes-dir on every node and --shared-volumes on every node and driver. Volumes created before an object store was configured keep their node directory.

Volume mount paths must be absolute and clean, not /, and unique per sandbox. Node-local volumes in one sandbox must all live on the same node. A create with node-local volumes cold-boots the template rootfs instead of forking its memory snapshot, because template snapshots carry no shared-directory devices.

Deleting a volume removes its directory or its whole object-store prefix. Deleting a node-local volume whose owning node is gone drops only the record.

Bring-your-own-bucket mounts

vmonS3Mounts is a vibemon extension on POST /sandboxes and POST /v2/sandboxes that mounts a user’s bucket into the sandbox. The request shape is in E2B compatibility; this section covers the node side.

S3 mounts and object-store volumes are VMM remote virtio-fs devices backed by one host proxy per sandbox, listening on <run-dir>/<id>-s3/fs<n>.sock. The proxy talks to S3; credentials never enter the guest.

Operator-relevant behavior:

• Limits: at most 8 S3-backed mounts per sandbox, counting object-store volumes. Paths must not overlap each other or volumeMounts.
• Caching: each mount keeps a 32 MiB LRU of 1 MiB chunks keyed by ETag; attributes are cached ≤ 1 s (writeback) or 5 s (ro, overlay).
• Consistency: last writer wins per object; no cross-node locking. Directory renames are copy + delete, at most 10 000 keys (EXDEV above). Symlinks and hard links are EPERM.
• Lifecycle: pause flushes all pending uploads before the snapshot, so a resume on another node sees them. Kill and VMM exit flush for up to 5 minutes, then drop staging. A create with S3 mounts forks the template snapshot only when the snapshot was captured with the same mount devices, else cold-boots.
• Credentials: inline credentials are stored only encrypted in the cluster KV (and dropped from the record when the sandbox is killed); credentialsSecret names a team secret that is re-read on every resume and fork, so a rotated secret applies at the next resume. Credentials are never logged or returned. The node keeps the mount list with credentials at mode 0600 in the sandbox directory so a restarted node can re-serve adopted VMMs; staged writes not yet uploaded by the previous process are lost.
• Anonymous access: mounts without credentials send unsigned requests, never the node’s identity.
• Private endpoints: a node refuses endpoints that resolve to loopback, private, link-local, CGNAT (100.64.0.0/10), IPv6 unique-local, unspecified, or multicast addresses, so users cannot reach the node’s own network. vmon serve --s3-mounts-allow-private (VMON_S3_MOUNTS_ALLOW_PRIVATE) lifts this, for example for a MinIO on the node’s network.
• Egress: mount traffic is made by the host and is not subject to the sandbox’s egress policy.

Guest kernel

Every sandbox and template build boots one guest kernel chosen by the node. This page covers the vmon kernel (the default), what it adds over the plain cloud-hypervisor kernel, how to build it, and how a node picks its kernel. Read it when installing nodes from source or when a template needs Docker, nft, IPv6, or NFS in the guest.

Why the vmon kernel

The vmon kernel is the cloud-hypervisor kernel source and ch_defconfig plus the features E2B guests rely on. The plain cloud-hypervisor kernel, which a node downloads only as a last resort, has no netfilter, no IPv6, no bridge, and no NFS support, so guest features such as Docker, nft, ip6tables, and NFS cannot work fully under it. The mcp-gateway template runs Docker in the guest and needs the vmon kernel.

Release assets ship it: vmon-assets-<arch>.tar.gz contains vmon-kernel-<arch> (and its resolved config vmon-kernel-<arch>.config) next to vmon-agent-<arch>; the raw images are also attached to each GitHub release. See Installation.

What it adds

Everything is built in: guests never need /lib/modules or modprobe. Module support is enabled (unsigned, no usermode modprobe) only so GPU sandboxes can load the NVIDIA modules of the GPU driver bundle; the build turns every option that would become a module into a built-in one.

Removed from ch_defconfig: the ktimer PPS test client and PPS_DEBUG (a wakeup and two dmesg lines every second in every sandbox), and the RAID6 boot benchmark pulled in by Btrfs (~1.1 s of a ~2 s boot under HVF).

The boot path is unchanged from the cloud-hypervisor kernel: virtio-mmio and virtio-pci; virtio blk/net/console/fs/rng/vsock/balloon; 8250 and PL011 consoles; GICv3 + PSCI on arm64; kvm-clock/PVH on x86_64. Source: cloud-hypervisor/linux tag ch-release-v6.16.9-20260508.

Building

just kernel                      # both arches → target/kernel/
just kernel --arch aarch64       # one arch
kernel/build.sh --config-only    # resolve and verify the .config only

just kernel passes its arguments to kernel/build.sh:

Linux (aarch64 or x86_64): build.sh runs natively. The host arch builds natively; the other arch is cross-compiled with gcc-<arch>-linux-gnu. On Debian/Ubuntu install the toolchain plus the cross compiler for the other arch (gcc-aarch64-linux-gnu on x86_64 hosts, gcc-x86-64-linux-gnu on aarch64 hosts):

sudo apt-get install build-essential flex bison bc libelf-dev libssl-dev \
  dwarves python3 cpio xz-utils curl

macOS: just kernel runs build.sh in one Ubuntu 24.04 container built from kernel/Dockerfile, capped at 4 CPUs and niced, with the source and build trees in the Docker volume vmon-kernel-cache (the kernel tree needs a case-sensitive filesystem). Docker must be running. Outputs still land in the checkout’s target/kernel/.

Every line in the config fragments is verified after olddefconfig: an option that was dropped (unmet dependency, renamed symbol) fails the build instead of silently producing a weaker kernel. Builds are reproducible: with the same toolchain (the Ubuntu 24.04 toolchain used by the macOS container and the release job), images are bit-identical regardless of --jobs. A cross build embeds a different compiler name in /proc/version and the config, so compare like with like.

Outputs

NVIDIA GPU driver bundle

Sandboxes that get a whole NVIDIA GPU (VFIO passthrough) load the NVIDIA driver from a bundle built for the exact guest kernel they boot. Build it together with the kernel:

just gpu-driver --arch x86_64    # = kernel/build.sh --nvidia --arch x86_64

--nvidia (also accepted by just kernel) downloads the pinned NVIDIA driver installer (615.71.09, sha256-verified, cached under --cache), builds the NVIDIA open kernel modules (nvidia.ko, nvidia-uvm.ko; Turing and newer GPUs, including GeForce RTX 50xx) against the kernel’s build tree, and writes target/kernel/vmon-nvidia-<arch>/:

On Linux the build needs zstd in addition to the kernel toolchain; on macOS it runs in the kernel build container. The x86_64 bundle is about 610 MiB.

The bundle matches one kernel build: modules built for another kernel release are refused in the guest. Rebuild the bundle whenever you rebuild the kernel, and install both together: put vmon-nvidia-<arch>/ next to the vmon binary, beside vmon-kernel-<arch>.

In a GPU sandbox the node shares the bundle read-only as virtio-fs tag vmon_gpu. Before envd starts, the guest agent mounts it at /run/vmon/gpu, loads the modules, creates /dev/nvidiactl, /dev/nvidia<N>, /dev/nvidia-uvm and /dev/nvidia-uvm-tools, registers lib/ with the dynamic linker (/etc/ld.so.conf.d/vmon-nvidia.conf + ldconfig), and links the tools into /usr/bin, so nvidia-smi and CUDA programs work like in a container started by the NVIDIA Container Toolkit. GPU templates therefore need a glibc image with the CUDA runtime they use (for example nvidia/cuda:13.0.3-runtime-ubuntu24.04) and no driver libraries of their own. Failures are logged to the sandbox console (NVIDIA GPU driver setup failed) and do not stop the sandbox; sandboxes without a GPU skip all of this.

Warning

A GPU whose VM exits with the driver still loaded cannot be initialized by the next VM: VFIO’s function-level reset does not stop the GPU’s GSP firmware (NVRM: ... unexpected WPR2 already up). The node therefore asks the guest agent to unload the driver (GC4 gpu_release) before tearing a GPU VM down. Do not reset the GPU from the host with secondary bus resets (reset_subordinate, bridge control); on GeForce RTX 50xx this can drop the GPU off the bus until the host is power-cycled.

Important

Load vfio-pci with disable_idle_d3=1 (options vfio-pci disable_idle_d3=1 in /etc/modprobe.d/, or vfio_pci.disable_idle_d3=1 on the host kernel command line) on hosts that pass whole GPUs through. Otherwise vfio-pci puts the idle GPU into D3hot between VMs; on a GeForce RTX 5070 the wake-up and resets when the next VM opened it ended in a fatal PCIe error that the platform firmware answered with a host reset. With the GPU kept in D0, back-to-back VMs that release the driver work, and the driver’s GPU initialization in the guest takes about 2 s instead of about 7.5 s.

How a node picks its kernel

At startup the node resolves the guest kernel in this order and logs the result:

1. --kernel <path> (VMON_KERNEL). The file must exist.
2. vmon-kernel-<arch> next to the vmon binary, where the release assets tarball puts it.
3. target/kernel/Image-aarch64 or target/kernel/bzImage-x86_64 in the nearest ancestor of the binary’s directory: a dev tree after just kernel.
4. The pinned cloud-hypervisor kernel, downloaded once into <data-dir>/assets/ and verified by SHA-256.

The fallback logs a warning:

no vmon guest kernel (vmon-kernel-<arch> next to vmon, or target/kernel/ from `just kernel`); falling back to the pinned cloud-hypervisor kernel, which lacks netfilter (nftables/iptables, Docker), IPv6, bridge and NFS

Warning

Template snapshots capture the running kernel, so templates built under one kernel keep booting it. Rebuild templates after switching kernels.

See Running vmon serve for how the node finds the guest agent binary.

The GPU driver bundle is not part of this lookup; see NVIDIA GPU driver bundle.

GPUs

Sandboxes can get whole NVIDIA GPUs (or NVIDIA vGPU virtual functions) passed through with VFIO, so CUDA programs run in them unchanged. This page covers preparing a host, installing the guest driver bundle, requesting GPUs from the E2B SDKs, what pause and snapshots do to a GPU sandbox, and where the dashboard shows GPUs.

GPU passthrough needs a Linux x86_64 node with KVM. macOS (HVF) and Windows nodes never publish GPUs.

How it fits together

• Every node lists the NVIDIA display-class PCI functions of its host in its gossip capacity: whole GPUs, and SR-IOV virtual functions that have a vGPU profile (a physical function with VFs enabled is not listed). Binding a GPU to vfio-pci is the operator’s opt-in. A GPU that is not ready for passthrough is still listed, with a problem that says why, and sandboxes never get it.
• A sandbox asks for a GPU count (see Requesting GPUs). The scheduler only places it on a node with that many free, assignable GPUs. The node reserves them atomically and starts the VM with them.
• In the guest, the agent mounts the node’s GPU driver bundle and loads the NVIDIA driver before envd starts. nvidia-smi and the CUDA driver library then work as in a container started by the NVIDIA Container Toolkit. The template supplies the CUDA runtime.
• GPU sandboxes always cold-boot. They cannot snapshot guest memory, so pauses and snapshots keep only the filesystem (see Pause, snapshots, and resume).

Preparing a host

The example host has a GeForce RTX 5070 at 0000:52:00.0 (10de:2f04) and the GPU’s HDMI audio function at 0000:52:00.1 (10de:2f80), on an Intel CPU running Fedora. Change the IDs, addresses, and distribution commands for your hardware.

1. Enable VT-d (Intel) or AMD-Vi (AMD) in the firmware setup.

2. Find the vendor:device IDs of the GPU and of every other function in its slot:

   lspci -nn -d 10de:
   # 52:00.0 VGA compatible controller [0300]: NVIDIA Corporation GB205 [GeForce RTX 5070] [10de:2f04]
   # 52:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:2f80]
   

3. Turn on the IOMMU and have vfio-pci claim those IDs at boot. On Intel CPUs:

   sudo grubby --update-kernel=ALL \
     --args="intel_iommu=on iommu=pt vfio-pci.ids=10de:2f04,10de:2f80 vfio-pci.disable_idle_d3=1"
   

   vfio-pci.disable_idle_d3=1 keeps the GPU powered (D0) between VMs. Without it, vfio-pci puts the idle GPU into D3hot after a sandbox ends, and on an RTX 5070 the next VM’s open then failed with a config-read timeout that the platform escalated to a host reset. Nodes report a whole GPU as unassignable until the parameter is set.

   AMD CPUs enable the IOMMU by default; add iommu=pt vfio-pci.ids=… vfio-pci.disable_idle_d3=1. On Debian and Ubuntu, put the same arguments in GRUB_CMDLINE_LINUX in /etc/default/grub and run sudo update-grub.

4. Keep the host’s NVIDIA and nouveau drivers off the GPU, and make the sound driver wait for vfio-pci so it cannot take the audio function first. /etc/modprobe.d/vmon-gpu.conf:

   blacklist nouveau
   blacklist nvidia
   blacklist nvidia_drm
   blacklist nvidia_modeset
   blacklist nvidia_uvm
   softdep snd_hda_intel pre: vfio-pci
   

5. Load the VFIO modules at boot. /etc/modules-load.d/vmon-gpu.conf:

   vfio-pci
   iommufd
   

   Also stop host services that keep loading the NVIDIA driver, if installed: sudo systemctl disable --now nvidia-persistenced nvidia-powerd.

6. Regenerate the initramfs so it includes the new module options (sudo dracut --force on Fedora, sudo update-initramfs -u on Debian and Ubuntu), then reboot.

7. Check the result:

   lspci -nnk -s 52:00          # both functions: "Kernel driver in use: vfio-pci"
   ls /sys/bus/pci/devices/0000:52:00.0/iommu_group/devices
   # 0000:52:00.0  0000:52:00.1   (every function of the group must be on vfio-pci)
   ls /dev/iommu /dev/vfio/devices/
   

Every function in the GPU’s IOMMU group must be bound to vfio-pci. If the group also holds unrelated devices (common when the slot sits behind a chipset bridge without ACS), move the card to another slot.

Do not reset a GPU from the host with secondary bus resets (reset_subordinate, setpci bridge control, or PCI remove and rescan). On GeForce RTX 50xx this can drop the GPU off the bus until the host is power-cycled.

Installing the guest driver bundle

The guest needs an NVIDIA driver built for the exact guest kernel it boots. Build the bundle together with the kernel:

just gpu-driver --arch x86_64    # = kernel/build.sh --nvidia --arch x86_64

This writes target/kernel/vmon-nvidia-x86_64/, containing NVIDIA driver 615.71.09 built for the current guest kernel. See NVIDIA GPU driver bundle for its contents. Install it next to the vmon binary, beside vmon-kernel-x86_64, as vmon-nvidia-x86_64/, or point the node at it with vmon serve --gpu-driver <DIR> (VMON_GPU_DRIVER). In a source checkout, the node also finds target/kernel/vmon-nvidia-<arch>/. Rebuild and reinstall the bundle whenever you rebuild the kernel. A bundle built for another kernel makes every GPU on the node report a problem.

GPU problems

A node publishes each GPU it finds with an empty problem when the GPU is assignable. Otherwise the problem says what to fix:

• the GPU is not bound to vfio-pci (for example, it is bound to nvidia);
• the IOMMU is off, or the GPU has no IOMMU group;
• another member of its IOMMU group is not on vfio-pci;
• the VFIO device node (/dev/vfio/devices/vfioN) or /dev/iommu is missing;
• no guest driver bundle was found, or the bundle was built for another kernel;
• the host does not support passthrough (not Linux/KVM/x86_64).

Unclean VM exits. On current NVIDIA GPUs a VFIO function-level reset does not reset the GPU’s firmware. A GPU whose VM exited while the driver was still loaded therefore cannot be used by the next VM. Before the node kills a GPU sandbox or pauses it, it asks the guest agent to stop GPU users and unload the driver, with a 30 s budget. If that fails, or the VM exits on its own (guest poweroff, VMM crash, a failed boot), the node quarantines the GPU with the problem needs host reboot: VM exited without releasing the GPU. Quarantine persists across node restarts and clears automatically after a host reboot. A reboot is the only fix; do not try a bus reset.

Requesting GPUs

A sandbox gets this many GPUs, decided at create:

1. metadata["vmon.gpus"], if set. This works with stock E2B SDKs:

   from e2b import Sandbox
   
   sbx = Sandbox.create("cuda", metadata={"vmon.gpus": "2"})
   

   const sbx = await Sandbox.create('cuda', { metadata: { 'vmon.gpus': '2' } })
   

2. Otherwise, the template’s environment variable VMON_GPUS, set when the template is built:

   template = Template().from_image("nvidia/cuda:13.0.3-runtime-ubuntu24.04").set_envs({"VMON_GPUS": "1"})
   

   A Dockerfile ENV VMON_GPUS=1 does the same.

3. Otherwise, 0.

Each value must be a decimal from 0 to 8. Anything else fails the create with 400, and the message names vmon.gpus or VMON_GPUS. Metadata always wins, so {"vmon.gpus": "0"} starts a GPU template without a GPU, which helps when preparing files.

When no node can take the request, the create fails with 503:

Give GPU templates at least 2 GiB of memory; the driver and a CUDA context need it on top of the workload.

The cuda template

just templates cuda (or uv run templates/build.py cuda) builds the public cuda template. It is nvidia/cuda:13.0.3-runtime-ubuntu24.04 (the CUDA 13.0 runtime; the bundle’s driver runs any CUDA 12.x or 13.x runtime) plus python3, pip, and venv, with VMON_GPUS=1, 2 vCPUs, and 4 GiB of memory. Its vmon-gpu-check command lists the GPUs and round-trips a buffer through each one’s memory:

sbx = Sandbox.create("cuda")
print(sbx.commands.run("vmon-gpu-check").stdout)
print(sbx.commands.run("nvidia-smi").stdout)

Templates for GPUs need a glibc image with the CUDA runtime they use, and must not ship NVIDIA driver libraries (libcuda, libnvidia-*) of their own. Those would shadow the bundle’s driver.

Pause, snapshots, and resume

A GPU’s state cannot be captured in a VM snapshot, so GPU sandboxes never save memory:

• Pause (sbx.pause(), with or without memory) and autoPause on timeout stop the sandbox and keep only its filesystem. The node unloads the driver first and frees the GPUs.
• Resume (Sandbox.connect, the deprecated resume, or autoResume traffic) boots the paused disk again and reserves GPUs again. Guest processes and memory are gone; the template’s start and ready commands run again. If the node holding the paused sandbox has no free GPUs, the resume fails with 503 sandbox_capacity_unavailable. With an object store, a paused GPU sandbox whose node is gone resumes on another node with free GPUs.
• Snapshots (sbx.create_snapshot()) capture only the filesystem. Sandboxes created from such a snapshot cold-boot. To be sure they get GPUs, pass metadata["vmon.gpus"] when you create them.
• Forks capture the filesystem once, then cold-boot each child on the same node with the same GPU count. Each child needs free GPUs there.
• autoPauseMemory and autoResume are validated as the E2B API defines them and stored as requested; only the pause itself drops the memory.
• Migration: GPU sandboxes are not migrated. POST /admin/sandboxes/{id}/migrate refuses them with 400, and draining a node leaves them running there until they end.

Dashboard

• Nodes lists a “GPUs free” column (free / total, plus how many are unavailable) when any node has GPUs.
• A node’s page has a GPUs card with each GPU’s model, PCI address, and state: free, in use by a sandbox (linked), or its problem.
• A sandbox’s page shows its GPU count.

The data comes from GET /admin/cluster (each node member’s node.gpus: [{address, model, sandbox, problem}]) and GET /admin/sandboxes (gpus per sandbox). See Administration.

Administration

Day-to-day operator tasks: creating teams, keys, and CLI users, setting limits, blocking teams, publishing default templates, watching the fleet in the admin dashboard, and using the admin HTTP API for keys, sandboxes, builds, and nodes. Every vmon admin flag is listed in vmon admin commands; the credentials themselves are explained in Authentication and teams.

Three tools

Running vmon admin

vmon admin needs the cluster listener of a running driver and the cluster secret:

export VMON_CLUSTER_URL=http://10.0.0.1:7947,http://10.0.0.2:7947   # drivers' --cluster-listen
export VMON_CLUSTER_SECRET=…        # the drivers' --cluster-secret
vmon admin list-teams

• Cluster URL (--cluster-url, VMON_CLUSTER_URL; default http://127.0.0.1:7947, the local driver): one or more driver cluster-listener URLs, comma-separated; requests go to the first that answers. The drivers keep all durable state in their raft KV, so vmon admin works only once a driver has started.
• Cluster secret (--cluster-secret, VMON_CLUSTER_SECRET): API keys are stored encrypted with AES-256-GCM under a key derived from the cluster secret (next to a SHA-256 hash used for lookups). vmon admin derives that key itself, so it must use the drivers’ secret. When the flag is unset it reads <data-dir>/cluster-secret (--data-dir, VMON_HOME; default /var/lib/vmon as root on Linux, else ~/.vmon), which is where vmon serve stores a generated secret. Running on a server host with the server’s data dir therefore needs no secret flag. vmon admin never generates a secret; without one it fails with no cluster secret: pass --cluster-secret (VMON_CLUSTER_SECRET) or --data-dir of a server (<path> missing).

Global options go before the subcommand. Commands that take a team accept its ID or its slug.

First team and key

vmon admin create-team acme

team    acme (6f0c3c1e-…)
slug    acme
api key e2b_…

The printed key (named Default API key) is shown once. Hand it to the team as E2B_API_KEY; client setup is in Connecting SDKs and the CLI. Pass --max-* flags to create-team to start with non-default limits.

More keys and rotation

Mint another key at any time; only the key is printed:

vmon admin create-api-key acme --name ci

There is no in-place rotation. Rotate by minting a new key, switching clients to it, then deleting the old one through the API (see Keys). Drivers cache key lookups for 30 seconds, so a deleted key can keep working on other drivers for up to that long.

CLI login users

e2b auth login signs in with an email and password that you create:

VMON_ADMIN_PASSWORD='…' vmon admin create-user dev@example.com --team acme

--team makes the user a member with that team as default. The password must be at least 8 UTF-8 bytes; reading it from VMON_ADMIN_PASSWORD instead of --password keeps it out of the process list. After login, the CLI’s GET /teams returns one API key per team for the user, minting a key named CLI (<email>) the first time.

Add an existing user to more teams:

vmon admin add-member dev@example.com other-team             # ordinary membership
vmon admin add-member dev@example.com other-team --default   # and make it the default

add-member on an existing membership rewrites its default flag: running it without --default clears that membership’s default flag.

Note

vmon admin has no commands to delete teams or users, remove members, or change passwords.

Limits

vmon admin set-limits acme --max-concurrent-sandboxes 20 --max-length-hours 4

Only the flags you pass change; the others keep their current values. The command prints the team’s updated line. What each limit enforces is in Authentication and teams. Drivers pick up new limits immediately for user-token and admin callers and within 30 seconds for API-key callers.

Blocking a team

vmon admin block acme
vmon admin unblock acme

A blocked team can still read, but every mutating team-scoped request is 403 team is blocked. Running sandboxes keep running; to stop them, use POST /admin/teams/{teamID}/sandboxes/kill. As with limits, API-key callers see the change within 30 seconds.

System team for the default templates

The SDKs’ default aliases (base, code-interpreter-v1, desktop, mcp-gateway) resolve for every team only when a system team publishes them. just templates (templates/build.py) builds and publishes them. When E2B_API_KEY is unset or empty, it runs vmon admin to find the team with slug --team (default system), marks it as a system team if needed or creates it with create-team --system, and mints a key; a supplied key is used without checking whether its team is a system team. vmon admin reads VMON_CLUSTER_URL and VMON_CLUSTER_SECRET / VMON_HOME from the environment as usual, and needs a running driver. Doing it by hand:

vmon admin create-team system --system      # or: vmon admin set-system <team>

set-system also claims free bare aliases for the team’s already public templates; unset-system releases all of them. Build and publishing details are in Templates.

Admin dashboard

Every driver serves a web console at /dashboard/ on its public listener (--listen, and --tls-listen when set), next to the API; / redirects there. Start the driver with --admin-token (VMON_ADMIN_TOKEN), open http://127.0.0.1:3000/ (or https://api.<domain>/), and sign in with that token. The token stays in the browser’s session storage (local storage with “Remember on this device”) and is sent only as X-Admin-Token to the same origin.

The terminal and the file browser reach the sandbox through the driver’s ingress with the sandbox’s envd access token, exactly like an SDK, so they work on any node of the cluster. Charts come from the serving driver’s in-memory fleet sampler (one sample every 5 seconds, last hour), so they start empty after a driver restart and can differ slightly between drivers. Node create counters are per driver as well.

Warning

The admin token grants every team’s sandboxes, including shells and files. Serve the dashboard over TLS (--tls-listen) whenever the driver is reachable from other hosts.

Admin HTTP API

Admin routes are served by every driver on its public listener and require X-Admin-Token equal to that driver’s --admin-token (VMON_ADMIN_TOKEN). Without --admin-token they answer 401. Team IDs in paths and in X-Team-ID are UUIDs (vmon admin list-teams, first column), not slugs.

API=https://api.example.com          # or http://127.0.0.1:3000
AUTH="X-Admin-Token: $VMON_ADMIN_TOKEN"
TEAM=6f0c3c1e-…

Keys

curl -s -H "$AUTH" -H "X-Team-ID: $TEAM" "$API/api-keys"
curl -s -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"name": "ci-2"}' "$API/admin/teams/$TEAM/api-keys"
curl -s -X DELETE -H "$AUTH" "$API/admin/teams/$TEAM/api-keys/<keyID>"

X-Admin-Token with X-Team-ID also authenticates the team routes of the E2B API (all except POST /api-keys and GET /teams, which need a user token), so an operator can act as a team, for example list its sandboxes or templates, without one of its keys.

Sandboxes and builds

curl -s -H "$AUTH" "$API/admin/sandboxes/running-counts"
curl -s -X POST -H "$AUTH" "$API/admin/teams/$TEAM/sandboxes/kill"
curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"nodeID": "<nodeID>"}' "$API/admin/sandboxes/<sandboxID>/migrate"

The five /clusters/{clusterID}/rigs… operations (upstream’s autoscaled node pools) check the admin token and then answer 501 Not implemented by this deployment, the spec’s response for deployments without rigs.

Dashboard data

The dashboard reads three vmon-specific routes, usable from scripts as well:

curl -s -H "$AUTH" "$API/admin/sandboxes?state=running&limit=50"

Nodes

Node IDs are 16 lowercase hex characters. The reported status is derived in this order:

1. Gossip liveness: a dead member is unhealthy, one that left is shutting_down, a suspect one is connecting.
2. The operator status set with POST /nodes/{nodeID}, stored in the raft KV (node/<id>/status).
3. draining when the node itself announces that it is draining.

connecting, standby, and shutting_down cannot be set (400).

Draining a node

1. Find the node: curl -s -H "$AUTH" "$API/nodes".

2. Mark it draining:

   curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json' \
     -d '{"status": "draining"}' "$API/nodes/<nodeID>"
   

   The scheduler places new sandboxes and builds only on ready nodes, and the driver starts evacuating the node: every few seconds it migrates the node’s running sandboxes with more than a minute left, two at a time, plus its paused sandboxes when the node keeps paused state only locally (without an object store). A migrated sandbox keeps its ID, processes, memory, and disk; the move pauses it, transfers its written disk and dirtied memory to the target, and resumes it there, so the blackout grows with that amount. Connections open during the move drop; clients reconnect to the same sandbox on the new node.

   Each sandbox needs a compatible ready node: same architecture, hypervisor backend, and CPU baseline, enough capacity, the features its network settings need, and no volume on node-local storage pinning it to the draining node. Sandboxes without one stay where they are and are retried every minute (the driver logs why). Sandboxes with less than a minute left expire in place. One sandbox can be moved by hand with POST /admin/sandboxes/{sandboxID}/migrate.

3. Watch sandboxCount in GET /nodes/<nodeID> fall as sandboxes move or end.

4. Return it to service with {"status": "ready"}, which deletes the stored override.

unhealthy has the same placement effect but does not evacuate. What happens to paused sandboxes, failover, and node shutdown is covered in Clusters.

vmon serve options

Every flag of vmon serve, grouped by topic. Every configuration flag can also be set through the listed VMON_* environment variable. When both are given, the flag wins. Behaviour that vmon serve --help leaves out comes from src/serve.rs and the node and driver crates. For how to run the server, see Running vmon serve.

vmon serve [OPTIONS]

Boolean switches (--log-json, --s3-mounts-allow-private, --shared-volumes) take no value on the command line. Enable them through the environment with true, for example VMON_LOG_JSON=true. --lazy-resume is different: it always takes a value, true or false.

Note

--help does not print the values of VMON_CLUSTER_SECRET and VMON_ADMIN_TOKEN.

Roles and logging

Default directories:

• --data-dir is /var/lib/vmon when vmon runs as root (effective uid 0) on Linux. Otherwise it is $HOME/.vmon, falling back to /var/lib/vmon when HOME is unset. vmon admin uses the same default when it looks for cluster-secret.
• --run-dir is $XDG_RUNTIME_DIR/vmon when XDG_RUNTIME_DIR is set and not empty. Otherwise it is /tmp/vmon-<uid>. Socket paths must stay under the 104-byte macOS sun_path limit, so keep this path short.

The files kept in both directories are listed in Data and run directories.

Cluster and gossip

Cluster secret. When --cluster-secret is unset, vmon serve reads <data-dir>/cluster-secret. If that file does not exist, it generates 32 random bytes and writes them hex-encoded (64 characters) to the file with mode 0600. It then logs a warning to share the secret with every process of the cluster. The secret authenticates cluster HTTP (bearer token) and gossip (HMAC-SHA256 keyed by HKDF(secret, "vmon-gossip")). The keys for JWTs, envd and traffic tokens, signed upload URLs, and encryption of stored data (API keys, env vars, secrets, webhook secrets) are all derived from it as well. Changing the secret invalidates the stored ciphertexts. For how to share it, see Cluster secret.

Advertise addresses. Without --advertise, the advertised IPs follow the listeners:

• If --cluster-listen or --gossip-listen binds a specific IP, vmon advertises only that IP. When both bind specific IPs, they must be the same IP, or startup fails and asks for --advertise.
• If both bind a wildcard (0.0.0.0, the default), vmon advertises every IP of every up interface, except loopback, link-local, and container or VM bridges (docker*, veth*, br-*, vibemon’s own vmon* TAPs, and similar). The source IP of the default route comes first. 0.0.0.0 advertises IPv4 addresses only; [::] advertises IPv6 as well. A host with no other address advertises 127.0.0.1. The interface list is re-read every 30 s, and a changed list is re-advertised.

At most 16 addresses are advertised. Every process publishes all its advertised IPs and its host name in gossip. Each peer chooses the address it dials: an IP in one of its own subnets (LAN) first, then other private addresses, then overlay networks (Tailscale 100.64.0.0/10, fd7a:115c:a1e0::/48), then public addresses. A new member’s addresses are all probed when it joins. The route is the best-ranked address that answered the last probe: when it stops answering, the peer switches to another advertised address, and it moves back as soon as the better address answers again. A node therefore stays reachable while any one of its addresses works. Startup logs the advertised IPs (cluster layer started, field advertised). The default plain-HTTP --public-url uses the first advertised IP. Behind NAT, list the translated addresses in --advertise, because vmon cannot see them.

Node identity. On first start each process writes a random 64-bit ID to <data-dir>/node-id and keeps it from then on. This ID is the gossip member ID, the raft node ID, and the id shown by GET /nodes. A restarted node therefore keeps its identity only if it reuses the same data directory.

Public listener and TLS

These flags apply to the driver role. Node-only processes do not open a public listener.

Public URL default. With --tls-listen set, the default is https://api.<domain>, with no port. Without it, the default is http://<first advertised IP>:<listen port> (see Advertise addresses). The driver uses this URL in signed template file upload URLs (PUT /templates/{id}/files/{hash}/upload?expires=…&signature=…) and in the tokenEndpoint and revokeEndpoint it hands to e2b auth login. It is also published in gossip. If clients reach the TLS listener on a port other than 443, or through a proxy, set it explicitly.

Routing rules, certificates, and DNS are covered in Domains, TLS, and ingress.

Database and admin

Admin workflows are covered in Administration.

Guest assets

These flags apply to the node role. Both assets are resolved once, at node startup.

Kernel lookup when --kernel is unset:

1. vmon-kernel-<arch> in the directory of the vmon binary (release assets).
2. target/kernel/Image-aarch64 or target/kernel/bzImage-x86_64 under the nearest ancestor of the binary’s directory (a dev tree after just kernel).
3. A pinned cloud-hypervisor kernel, downloaded into <data-dir>/assets/ and verified by SHA-256. vmon logs a warning when it falls back to this kernel, because the kernel lacks netfilter (nftables/iptables, Docker), IPv6, bridge, and NFS.

Agent lookup when --agent is unset: vmon-agent-<arch> next to the vmon binary, then ../../target/test-assets/vmon-agent-<arch> relative to it (just agent-musl). If neither exists, the node fails to start. See Guest kernel.

GPU driver bundle lookup when --gpu-driver is unset: vmon-nvidia-<arch>/ next to the vmon binary, then target/kernel/vmon-nvidia-<arch>/ under the nearest ancestor of the binary’s directory (just gpu-driver). Without a bundle the node still starts; its GPUs report no guest driver bundle and are not assignable. See GPUs.

Networking and egress

These flags apply to the node role.

VMM privilege drop. The node launches each VMM as vmon vmm:

• Root on Linux, --vmm-user set: the VMM gets --sandbox-uid/--sandbox-gid. It drops to that uid and gid and applies its Landlock and seccomp filters. TAP devices are created owned by that uid so the unprivileged VMM can open them.
• Root on Linux, --vmm-user unset: VMMs are started with --no-sandbox. They keep root and run without the Landlock and seccomp filters.
• Not root: VMMs run as the invoking user, with the default filters on Linux. macOS has no host process sandbox.

Policy semantics per platform are covered in Networking and egress.

Templates and registries

See Templates.

Object store and volumes

Object store credentials

With --object-store set, these environment variables configure the S3 client. They are not flags.

Role credentials are refreshed before they expire. Data layout and portability are covered in Storage, object store, and volumes.

Capacity and memory

These flags apply to the node role.

The hugepage write sets the absolute size of the host pool; it does not add to the current size. It needs root. The kernel can grant fewer pages than requested when contiguous memory is short. vmon logs the result either way. On other platforms a non-zero value logs a warning and is otherwise ignored. For the trade-offs, see Hugepages.

vmon admin commands

Complete reference for vmon admin, the operator tool for teams, users, API keys, limits, and system teams. It reads and writes the cluster’s store through the cluster listener of a running driver, so at least one driver must be up. Task-oriented walkthroughs are in Administration.

Usage: vmon admin [OPTIONS] <COMMAND>

Every invocation connects to the first reachable --cluster-url; a write reaches the raft leader through it, and reads are linearizable (they see every write acknowledged before them). Failures print vmon: error: <message> to stderr and exit with status 1; invalid arguments are reported by the argument parser with status 2. vmon admin help [COMMAND] prints the help of any subcommand.

Global options

Global options go between admin and the subcommand. Every subcommand also accepts -h, --help; --system and --default are off unless supplied.

When --cluster-secret is unset and <data-dir>/cluster-secret does not exist, every command fails with no cluster secret: pass --cluster-secret (VMON_CLUSTER_SECRET) or --data-dir of a server (<path> missing). The secret file is read with surrounding whitespace trimmed. Other file-read failures report the read error. vmon admin never creates the file.

Team arguments

Every <TEAM> argument and --team option accepts the team’s ID (UUID) or its slug. An unknown team fails with team <TEAM> not found.

Team lines

list-teams, set-limits, block, unblock, set-system, and unset-system print teams as one tab-separated line each:

<id>	<slug>	<name>	sandboxes=<n> builds=<n> vcpus=<n> memory_mb=<n> disk_mb=<n> hours=<n>[	BLOCKED][	SYSTEM]

The fourth column holds the limits separated by spaces: max_concurrent_sandboxes, max_concurrent_builds, max_vcpus, max_memory_mb, max_disk_mb, max_length_hours. BLOCKED and SYSTEM are extra tab-separated columns, present only when set.

Limit options

create-team and set-limits take the same limit options:

Values are unsigned integers. An omitted option means “keep the base value”: the default above for create-team, the team’s current value for set-limits.

create-team

Create a team and print its first API key.

Usage: vmon admin create-team [OPTIONS] <NAME>

The slug is derived from the name: lowercase ASCII letters and digits, every other run of characters becomes one -, no leading or trailing -, and team when nothing remains. On a collision -2, -3, … is appended.

The command also mints the team’s first API key, named Default API key, and prints:

team    <name> (<id>)
slug    <slug>
api key e2b_<40 hex>

The plaintext key is only shown here.

list-teams

List teams.

Usage: vmon admin list-teams

Prints one team line per team, oldest first.

create-api-key

Mint another API key for a team (id or slug).

Usage: vmon admin create-api-key [OPTIONS] <TEAM>

Prints only the new key (e2b_ + 40 lowercase hex) on one line. The key is stored as a SHA-256 hash plus an AES-256-GCM ciphertext under the cluster secret’s data key, with no creating user.

create-user

Create a CLI login user, optionally in a team (id or slug) as its default team.

Usage: vmon admin create-user [OPTIONS] --password <PASSWORD> <EMAIL>

Fails when a user with the email already exists or the team is unknown (the team is checked before the user is created). Prints:

user <email> (<id>)

add-member

Add an existing user to a team.

Usage: vmon admin add-member [OPTIONS] <EMAIL> <TEAM>

If the user is already a member, the membership’s default flag is set to the value of --default, so running without --default clears it. Prints nothing on success.

set-limits

Change a team’s limits (omitted flags keep their current values).

Usage: vmon admin set-limits [OPTIONS] <TEAM>

All six limit columns are written in one update. Each value comes from its option when given and from the team’s current limit otherwise, so omitted options leave that limit unchanged and running with no options rewrites the current values. Prints the updated team line.

block

Block a team: its credentials may still read, every change is refused with 403 team is blocked.

Usage: vmon admin block <TEAM>

Sets the team’s blocked flag. All of its credentials keep authenticating, but every team-scoped request with a method other than GET, HEAD, OPTIONS, or TRACE is rejected with 403 team is blocked; running sandboxes are not stopped. Drivers cache API-key lookups for 30 seconds, so API-key callers see the change within that window. Prints the updated team line.

unblock

Unblock a team.

Usage: vmon admin unblock <TEAM>

Clears the blocked flag. Prints the updated team line.

set-system

Make a team a system team (its public templates claim free bare aliases).

Usage: vmon admin set-system <TEAM>

Sets the system flag and, in the same transaction, claims a bare alias for each name of the team’s public, non-deleted templates whose bare alias no other template holds. Prints the updated team line. See System teams.

unset-system

Make a system team an ordinary team (releases its bare aliases).

Usage: vmon admin unset-system <TEAM>

Clears the system flag and, in the same transaction, deletes every bare alias held by the team’s templates. The team’s namespaced aliases (<slug>/<alias>) are unaffected. Prints the updated team line.

E2B compatibility matrix

Target: the official E2B SDKs (e2b, @e2b/code-interpreter / e2b_code_interpreter, @e2b/desktop / e2b_desktop) and the e2b CLI work unmodified against vibemon when pointed at it with E2B_DOMAIN / E2B_API_URL / E2B_SANDBOX_URL / E2B_API_KEY.

Sources: spec/e2b/ (vendored upstream specs; provenance in spec/e2b/SOURCES.md). Design: ARCHITECTURE.md.

Legend: Core is what the stripped core (vmm + vmon-agent) provided before the rework. Implementation is what vibemon adds, and where. Proof for each area is in §4a (official SDK suites), §6 (CLI), Distributed, Linux/KVM, and TLS verification and e2e/README.md.

1. What the core VM gives us

2. Control-plane REST API (spec/e2b/infra/spec/openapi.yml)

Every operation is routed. “SDK” = called by the JS/Python SDKs, “CLI” = called by the e2b CLI, “Admin” = operator-only (X-Admin-Token).

Volume-content API (spec/e2b/sdk/spec/openapi-volumecontent.yml): GET|PATCH|DELETE /volumecontent/{id}/path, GET|POST /volumecontent/{id}/dir, GET|PUT /volumecontent/{id}/file — volume JWT auth; object-store volumes are served by the driver directly against S3 (same objects the mounts see), node-local ones are proxied to the owning node. Point the SDK’s E2B_VOLUME_API_URL at the driver when not using https://api.<domain>.

3. envd (spec/e2b/infra/packages/envd/spec/)

Transport: Connect v1 (JSON and proto; unary + enveloped server streams + client stream for StreamInput), HTTP/1.1 + h2c.

4. Infrastructure behaviors

4a. Proven by the official SDK suites

e2e/run.py runs upstream’s unmodified packages/python-sdk/tests and packages/js-sdk/tests (pinned to the spec/e2b/ snapshot) against a monolithic vmon serve. Final macOS/HVF run (2026-10-12): Python 1444 passed / 8 failed / 34 skipped, JS 1006 passed / 2 failed / 36 skipped; skips match upstream CI. Remaining failures: egress header rules (Linux-only feature, 3) and signed download URLs under the macOS Python routing workaround (6). The run’s one JS auto-resume regression was fixed afterwards in the ingress; a JS re-run then gave 1007 passed / 1 failed (egress rules) / 36 skipped. Details and the fixes made are listed in e2e/README.md.

5. Deliberate deviations

• Bring-your-own-bucket mounts (vmonS3Mounts, extension). POST /sandboxes and POST /v2/sandboxes accept vmonS3Mounts: [{path, uri, mode?, endpoint?, region?, credentials?: {accessKeyId, secretAccessKey, sessionToken?}, credentialsSecret?}]: uri is s3://bucket[/prefix] or http(s)://endpoint/bucket[/prefix]; mode is ro (default; lazy read-through), overlay (lazy lower + writes kept on the sandbox disk, gone at kill) or writeback (writes persisted to the bucket on close/fsync); credentialsSecret names a team secret whose value is the credentials JSON (re-read at every resume and fork). Paths must be absolute, clean, and must not overlap each other or volumeMounts; at most 8 S3-backed mounts per sandbox. Without credentials requests are anonymous (never the node’s identity). Endpoints resolving to private/loopback addresses are refused unless the node runs with --s3-mounts-allow-private. Credentials are never logged or returned; the store keeps them only encrypted. Unmodified SDKs cannot send extension fields; use raw HTTP and then Sandbox.connect:

  curl -sX POST "$E2B_API_URL/v2/sandboxes" -H "X-API-Key: $E2B_API_KEY" \
    -H 'Content-Type: application/json' -d '{"templateID": "base",
    "vmonS3Mounts": [{"path": "/mnt/data", "uri": "s3://my-bucket/datasets",
      "mode": "ro", "credentialsSecret": "my-bucket-keys"}]}'
  

  Consistency: last writer wins per object, attributes are cached ≤ 1 s (write-back) / 5 s (ro, overlay), no cross-node locking; directory renames are copy + delete (not atomic, ≤ 10 000 keys, else EXDEV); symlinks and hard links are EPERM.

• GPU sandboxes (extension). Upstream E2B has no GPUs. Here a sandbox gets whole NVIDIA GPUs (or vGPU VFs) passed through when metadata["vmon.gpus"] is a decimal 0–8 (works with stock SDKs: Sandbox.create(metadata={"vmon.gpus": "1"})), else when its template’s environment sets VMON_GPUS (Template().set_envs({"VMON_GPUS": "1"}) or Dockerfile ENV VMON_GPUS=1), else it gets none; metadata wins, so "0" opts a GPU template out. Any other value is 400 naming the key. Placement needs a node with that many free assignable GPUs: sandbox_no_compatible_node when no node has that many at all, sandbox_capacity_unavailable while they are all held. GPU sandboxes always cold-boot, and pauses and snapshots keep the filesystem only: pause() (with or without memory), autoPause, and create_snapshot() stop or capture the disk, and the next resume (manual, connect, or autoResume traffic) boots it again with the GPUs re-reserved, losing guest processes and memory. The requested autoPauseMemory/autoResume flags are validated as upstream and kept as asked. GPU sandboxes are not migrated, so draining a node leaves them running there. The cuda default template (CUDA runtime, one GPU) is a vmon addition. See GPUs.

• Sandbox self-keepalive. X-Access-Token: <envdAccessToken> also authenticates GET /sandboxes/{id}, POST /sandboxes/{id}/timeout, and POST /sandboxes/{id}/refreshes, only for the sandbox that owns that token. The token is compared in constant time before a sandbox lookup; wrong tokens, killed sandboxes, and unknown sandboxes return 401. It grants no access to create, list, kill, pause, connect, snapshots, or any other API route. Existing team blocking and maximum-lifetime rules still apply: timeout and refresh clamp expiry to the owning team’s per-execution maximum, exactly as with API-key authentication. This is a vmon extension, not an upstream E2B capability.

• Admin dashboard. /dashboard/ on the public listener and its GET /admin/cluster, GET /admin/teams, GET /admin/sandboxes endpoints (X-Admin-Token) are vmon extensions with no upstream counterpart.

• Draining a node evacuates it. Upstream’s draining only stops new placements. In vmon the raft leader also migrates the node’s running sandboxes (more than 60 s left) and node-local paused sandboxes to compatible ready nodes (same architecture, backend, CPU baseline; capacity, features, node-local volume pins): pause, transfer of the written disk and dirtied memory, resume. The sandbox ID, guest processes, and memory survive; connections open during the move drop. Sandboxes without a compatible node, and GPU sandboxes, stay. POST /admin/sandboxes/{id}/migrate moves one sandbox on demand (vmon extension).

• Sandbox log fields are strings. SandboxLogEntry.fields is map<string,string>, so non-string envd fields (pid, process_result) reach e2b sandbox logs as strings; upstream’s v1 line keeps their JSON types.

• Init system follows upstream provisioning. Debian/Ubuntu/RHEL/Arch templates boot systemd as PID 1 and Alpine boots busybox init + OpenRC, with vmon-agent installed as the envd service at /usr/bin/envd (upstream package set and init setup, vendored under spec/e2b/infra/packages/orchestrator/pkg/template/build/phases/). Images with an unknown distribution, which upstream rejects, boot the agent as PID 1 instead. When the apt install fails on sources using deb.debian.org or security.debian.org, provisioning switches them to archive.debian.org and retries once, so end-of-life Debian releases still provision; upstream fails the build.

• No Firecracker MMDS. Identity (E2B_SANDBOX_ID, E2B_TEMPLATE_ID, access-token hash) is delivered through /init; there is no 169.254.169.254 service.

• Legacy v1 template builds and the docker.<domain> registry are not served; upstream removed them and current CLIs never call them.

• MCP gateway internals are ours. Upstream’s mcp-gateway image is not public. Ours maps every SDK catalog id except circleci (now an OAuth-only remote server in the Docker catalog; its token/url fields have no counterpart) onto the Docker MCP Catalog v3. Catalog servers needing Docker Desktop OAuth, multi-line secret values, and per-server tool prefixes are not supported; on a tool-name clash the first configured server wins. mcp-gateway --config returns after at most 55 s; servers still starting then (runtime image pulls, long installCmds) appear later with notifications/tools/list_changed. Needs a guest kernel that runs Docker (kernel/); the pinned cloud-hypervisor kernel download cannot.

• Rigs (cluster autoscaling objects) answer 501, the response the spec defines for deployments that do not implement them.

• Bare aliases belong to system teams. Upstream’s v1 publish lets any team claim a bare (non-namespaced) alias. Here only a system team’s public templates hold bare aliases (claimed on v1 or v2 publish), so the SDK defaults (base, code-interpreter-v1, desktop, mcp-gateway) resolve for every team and cannot be squatted; ordinary teams are addressed as <slug>/<alias> (ARCHITECTURE.md §7).

• minFreeDiskMb is validated, not applied. Builds accept it (0 to the team’s max_disk_mb) and record it, but every build disk is already a sparse ext4 image of the team’s max_disk_mb. Free space after the steps is that size minus what they wrote; upstream instead grows the filesystem to the requested free space and keeps it smaller otherwise.

• Default templates are multi-arch. templates/ mirrors the upstream definitions but picks architecture-specific packages at build time (JDK 11: OpenJDK x64 / Temurin aarch64; desktop: Google Chrome on amd64, Chromium as google-chrome-stable on arm64; VS Code repo for the native arch).

• vibemon branding on cosmetic surfaces only: the default guest hostname is vibemon (upstream e2b), the desktop wallpaper and the noVNC page title/icons are vibemon’s. Protocol identifiers (E2B_* env vars, /.e2b, /run/e2b*, e2b/* MIME types, events.e2b.local, user, aliases) are unchanged.

6. e2b CLI (@e2b/cli 2.21.2)

Configuration: E2B_API_URL, E2B_DOMAIN, and (without TLS on the sandbox domain) E2B_SANDBOX_URL. Commands authenticate with E2B_API_KEY or with the project key that e2b auth login stores in ~/.e2b/config.json. The CLI opens the login page at ${E2B_DOCS_BASE:-https://$E2B_DOMAIN/docs}/api/cli. The driver serves that page on any Host that does not address a sandbox, so E2B_DOCS_BASE=http://<api>/docs is only needed when the apex domain does not resolve to the driver over HTTPS. The CLI never reads E2B_ACCESS_TOKEN.

All commands below were proven against a single vmon serve on macOS/HVF, driven by scripts (browser simulator; pexpect for the inquirer prompts and for the PTY).

Building and testing

How to build vibemon from source, run its test tiers, and follow the code conventions CI enforces. For contribution licensing see CONTRIBUTING.md; the normative design is ARCHITECTURE.md.

Workspace

Other directories: templates/ (default templates), kernel/ (guest kernel), e2e/ (upstream SDK suites), spec/e2b/ (vendored upstream specs), tests/ (VMM integration tests and the distributed test).

Toolchain

• Rust: pinned in rust-toolchain.toml to nightly-2026-09-14 with rustfmt, clippy, rust-analyzer, and the targets aarch64/x86_64-unknown-linux-gnu, aarch64/x86_64-unknown-linux-musl, aarch64-apple-darwin, and x86_64-pc-windows-msvc. rustup show installs it.
• just: the task runner; just --list shows every recipe.
• macOS native deps: brew install libslirp pkg-config (as in CI).
• Windows: only vmon vmm builds; CI installs libslirp through vcpkg.
• cargo-zigbuild + zig: static guest agent and Linux cross checks. CI installs ziglang==0.13.0 and cargo-zigbuild with pip.
• Docker: just kernel on macOS.
• Python uv: just templates and the e2e harness; the JS e2e suite also needs pnpm and node.
• cargo-deny: just license-check.
• Lima: for the lima-* recipes and seccomp-audit on macOS.

macOS signing

Hypervisor.framework refuses to run a binary that is not signed with the com.apple.security.hypervisor entitlement. just build and just release ad-hoc sign vmon with hvf.entitlements. A plain cargo build, or a binary in a custom CARGO_TARGET_DIR that just did not sign, needs:

codesign --sign - --entitlements hvf.entitlements --force <path/to/vmon>

just bin prints the path just uses (it honors CARGO_TARGET_DIR). cargo test replaces the signed binary with a fresh unsigned copy, so the VM test recipes run tests through demo/hvf-test-runner.sh, a cargo runner that signs the vmon the tests spawn.

Recipes

Host recipes pick the platform implementation automatically: KVM on Linux, HVF on macOS. Set PROFILE=release (or just profile=release …) to build release binaries.

The Lima recipes use the instance kvm (VMON_LIMA_VM) and expect this checkout at the same path inside the guest (LIMA_REPO). Create the instance with:

limactl start --vm-type=vz --set='.nestedVirtualization=true' --name=kvm template:default

Local development loop

just serve                                  # API on 127.0.0.1:3000
vmon admin create-team dev                  # via the server's cluster listener
just templates base

Use the printed key with the E2B SDKs as described in Quickstart.

Test tiers

Without their gate variables the gated tests skip. On macOS, run gated VM tests through demo/hvf-test-runner.sh (set CARGO_TARGET_AARCH64_APPLE_DARWIN_RUNNER and VMON_ENTITLEMENTS, as the recipes do).

CI (.github/workflows/ci.yml) runs on Linux: license check, cargo fmt --check, cargo check and cargo clippy -D warnings for the host and aarch64-unknown-linux-gnu, cargo test (driver store tests included), static agent builds for both musl targets, and cargo audit. On macOS it builds, signs, and compiles the tests; on Windows it checks vmon and lints and unit-tests the vmm crate. A separate workflow runs VMON_E2E=1 on a KVM runner.

Cross checks for Linux-only code

Agent envd and unpack, the egress proxy, and TAP networking compile only for Linux. From macOS, type-check them before sending a change:

cargo zigbuild --target aarch64-unknown-linux-musl -p vmon-agent
cargo zigbuild --target aarch64-unknown-linux-gnu -p vmon-node

Upstream SDK suites

just e2e runs upstream’s unmodified packages/python-sdk/tests and packages/js-sdk/tests, fetched at a pinned E2B commit, against a fresh monolithic vmon serve, the way upstream CI runs them. Each run creates a vmon_<name> database, mints a team key with vmon admin create-team, starts the server with data dir /tmp/vmon-<name>, builds the test templates with the SDK, runs the suites, and tears everything down (--keep retains the data directory and database after shutdown; --reuse reuses retained state). Arguments after -- go to pytest or vitest:

just e2e python -- tests/sync/sandbox_sync
python3 e2e/run.py all --ingress tls      # upstream's TLS topology with a throwaway CA

--ingress plain (default) routes SDK traffic through E2B_API_URL/E2B_SANDBOX_URL; --ingress tls serves *.<name>.localhost on port 443. Use your own --name and --slot when several plain-HTTP stacks share a host; TLS stacks all bind port 443, so only one can run per host. The harness options, the latest pass/fail counts, and the known failures are in e2e/README.md; record new counts there when they change. Per-operation coverage is in E2B compatibility.

Documentation

This site is an mdBook (0.5 or newer) in docs/: pages in docs/src/, the table of contents in docs/src/SUMMARY.md, the theme in docs/theme/custom.css.

mdbook serve docs --port 3100    # live preview on http://localhost:3100
mdbook build docs                # static site in docs/book/ (gitignored)

The Documentation workflow (.github/workflows/docs.yml) builds the book and checks local Markdown links on pull requests that touch docs/, and publishes it to GitHub Pages on pushes to main. Run it by hand with workflow_dispatch to republish.

Code conventions

The full list is in AGENTS.md.

• Formatting: edition 2024, rustfmt.toml with hard tabs (tab width 3), max_width = 100, aligned struct fields, and crate-grouped imports. Always run just format.
• Clippy: workspace lints deny correctness and suspicious and warn on pedantic, nursery, perf, and style; CI runs clippy with -D warnings. Every unsafe block needs a // SAFETY: comment and every #[allow] a reason.
• Dependencies: versions live in the root [workspace.dependencies].
• Async: tokio/axum/hyper in cluster, node, driver, and envd; the vmm crate stays synchronous.
• Errors: E2B-shaped API errors ({code, message, error_code?}, driver/src/error.rs); node API errors {code, message} with stable codes; the VMM uses vmm/src/result.rs.
• Driver store: records are serde structs under db/<table>/… keys in the raft KV (ARCHITECTURE.md §7). Read through Store::read(); write in a Store::tx() loop (guarded reads, staged writes, rerun when commit() returns false). There are no migrations: new record fields need #[serde(default)].
• E2B fidelity: match upstream wire behavior using spec/e2b/; protocol identifiers (E2B_*, /.e2b, /run/e2b, aliases) stay as upstream, user-visible branding is vibemon.
• Platform code: keep OS and arch differences behind vmm/src/{os,hv,arch}, node/src/net, and node/src/egress with #[cfg].
• Design changes: keep ARCHITECTURE.md in sync with behavior.

Distributed, Linux/KVM, and TLS verification

How to rerun the checks for the distributed topology, for the Linux/KVM node path, and for TLS subdomain routing. The design is specified in ARCHITECTURE.md §5 and §9.4.

1. Gated distributed test (tests/cluster.rs)

just cluster-e2e
# equivalent on macOS:
CARGO_TARGET_AARCH64_APPLE_DARWIN_RUNNER=$PWD/demo/hvf-test-runner.sh \
VMON_ENTITLEMENTS=$PWD/hvf.entitlements VMON_CLUSTER_E2E=1 \
  cargo test --test cluster -- --nocapture

Requirements:

• target/test-assets/{Image,vmon-agent}-<arch>, or set VMON_KERNEL and VMON_AGENT.
• Optional: VMON_CLUSTER_NET (nodes’ --net, default user) and VMON_CLUSTER_IMAGE (default alpine:3.20).

The test starts 3 --role driver --bootstrap-expect 3 processes and 2 --role node processes, joined through gossip seeds, on loopback ports. Logs go to /tmp/vce-<pid>/<name>.log. It runs in about 3 minutes, and each [phase] line prints its timings. It asserts each of the following:

1. Raft bootstraps exactly once, every driver lists both nodes in GET /nodes, and all of them report one clusterID.

2. A template builds through d1. A sandbox created through d1 runs envd exec, file upload, and download through the d1, d2, and d3 ingress. Its sbx/<id> KV route reads the same on every driver, both locally and consistently.

3. With the builder draining, the next sandbox lands on the other node through a peer artifact fetch.

4. After SIGKILL of the raft leader:

   • a new leader is elected;
   • the survivors return 0 failed API calls;
   • the new leader’s reaper kills an 8-second-timeout sandbox and records a killed event with reason timeout.

   The restarted driver serves consistent KV, and there is no second bootstrap.

5. After SIGKILL of a node:

   • its sandbox gets the killed event with reason orphaned;
   • placement uses the other node, which needs the artifact-holder record from step 3;
   • when the node restarts, the orphaned VMM, which is still alive, is killed.

6. After SIGTERM of a node, 15 seconds down (gossip Left), and a restart, its sandboxes survive, keep their files, and are adopted.

2. Linux/KVM in Lima (TAP networking + egress proxy)

limactl shell kvm -- bash -lc 'cd <worktree> && . ~/.cargo/env && \
  CARGO_TARGET_DIR=$HOME/target-<you> cargo build -p vmon'
# root is needed for TAP and iptables:
limactl shell kvm -- sudo -E $HOME/target-<you>/debug/vmon serve \
  --data-dir /tmp/<you>/mono --run-dir /tmp/<you>-run \
  --listen 127.0.0.1:3500 --cluster-listen 127.0.0.1:3501 --gossip-listen 127.0.0.1:3502 \
  --net tap --net-pool 172.29.0.0/16 --kernel <Image-aarch64> --agent <static vmon-agent> \
  --cluster-secret <s> --domain cluster.test

1. Create a key with vmon admin --cluster-url http://127.0.0.1:3501 --cluster-secret <s> create-team kvm (inside Lima).

2. Build alp with Template().from_image("alpine:3.20"). This takes about 6½ minutes under nested KVM.

3. Run uv run --with e2b python e2e/cluster/linux.py inside Lima with:

   E2B_API_URL=http://127.0.0.1:3500 E2B_SANDBOX_URL=http://127.0.0.1:3500 \
   E2B_DOMAIN=cluster.test E2B_HTTP_VERSION=1.1
   

   The script exercises create, commands, files, pause, resume, kill, the allowOut domain, a denyOut CIDR, allow_internet_access=false, and volumes shared between two sandboxes.

4. For the distributed variant, start --role driver and --role node processes with distinct ports, data dirs, and run dirs, all with the same --join seeds. These are the same flags that tests/cluster.rs passes.

5. When finished, remove what you created: pkill -f <run-dir>, and check ip -o link | grep tap and iptables -S | grep vmon.

The Python SDK sends the volume content API (Volume.read_file, …) to https://api.<domain> and ignores E2B_API_URL. That API therefore needs the TLS setup in §3.

3. TLS and true subdomain routing (no E2B_SANDBOX_URL)

1. Create a CA and a leaf certificate for *.vmon.test, api.vmon.test, and vmon.test:

   openssl req -x509 -newkey rsa:2048 -nodes -keyout ca.key -out ca.pem -days 30 -subj '/CN=vmon test CA' \
     -addext 'basicConstraints=critical,CA:TRUE' -addext 'keyUsage=critical,keyCertSign,cRLSign'
   openssl req -newkey rsa:2048 -nodes -keyout key.pem -out req.csr -subj '/CN=*.vmon.test'
   printf 'subjectAltName=DNS:*.vmon.test,DNS:api.vmon.test,DNS:vmon.test\nextendedKeyUsage=serverAuth\n' > ext.cnf
   openssl x509 -req -in req.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out leaf.pem -days 30 -extfile ext.cnf
   cat leaf.pem ca.pem > cert.pem
   

2. Start the server: vmon serve … --domain vmon.test --tls-listen 127.0.0.1:3463 --tls-cert cert.pem --tls-key key.pem --public-url https://api.vmon.test.

3. Resolve *.vmon.test. The SDKs’ HTTP stacks have their own resolvers: Python’s pyqwest uses hickory DNS, and Bun uses its own. Without root, use e2e/cluster/resolver_proxy.py instead:

   VMON_TLS_PORT=3463 python3 e2e/cluster/resolver_proxy.py 3464
   

   It is an HTTPS CONNECT proxy that maps *.vmon.test:443 to 127.0.0.1:3463. TLS stays end to end: the SNI, the certificate check, and ALPN negotiation all happen between the SDK and vmon. With root, a dnsmasq address=/vmon.test/127.0.0.1 entry and port 443 do the same job.

4. Python:

   env -u E2B_SANDBOX_URL -u E2B_API_URL -u E2B_HTTP_VERSION E2B_API_KEY=… E2B_DOMAIN=vmon.test \
     uv run --with e2b --with h2 env PYTHONPATH=e2e/cluster/hook VMON_TEST_CA=ca.pem \
     SSL_CERT_FILE=ca.pem HTTPS_PROXY=http://127.0.0.1:3464 python e2e/cluster/tls.py
   

   The hook is needed on macOS because the SDK’s pyqwest transports use the platform verifier, which ignores SSL_CERT_FILE. The hook passes the test CA through tls_ca_cert.

   JS: NODE_EXTRA_CA_CERTS=ca.pem HTTPS_PROXY=http://127.0.0.1:3464 E2B_DOMAIN=vmon.test bun <script>.

   To check that the server offers h2 over ALPN: curl -v --cacert ca.pem --resolve api.vmon.test:3463:127.0.0.1 https://api.vmon.test:3463/health should print ALPN: server accepted h2.