A dedicated host¶
Give openblox a machine of its own, and one command takes it from a fresh OS
install to a hardened sandbox host. It installs Docker, gVisor, openbloxd and
its service, plus an optional mTLS listener for remote callers, a firewall and
automatic security updates. Then it runs a sandbox to prove the host works.
Or read it before you run it:
The source is www/setup.sh.
Its design follows k3s's installer. It runs named phases in a fixed order, and
each phase checks before it changes anything, so running it again is safe. It
also writes an uninstaller that removes exactly what it added.
If you already run Docker and gVisor, or want to put the daemon on a machine that does other work, you don't need this. Install the binary and follow Running in production instead.
Hardware¶
- CPU: amd64 or arm64, at least 4 cores.
- Disk: an SSD. Every sandbox's writable layer lives on it.
-
Memory: the RAM decides how many sandboxes can run at once. A sandbox holds about 1.4× its
memory_mbbefore the kernel's limit lands (#30), and the host keeps 1 GiB for itself:max_sandboxes = floor((RAM − 1 GiB) / (1.4 × memory_mb)), at least 1At the default
memory_mbof 2048:RAM Concurrent sandboxes 8 GiB 2 16 GiB 5 32 GiB 11 64 GiB 22 These are nominal sizes. The kernel reports a little less than the installed RAM, so the script may compute one fewer. It prints its arithmetic.
Choosing an OS¶
- Debian 13, minimal (recommended). It is small, has no snap, and has long support.
- Ubuntu 24.04 LTS is also supported.
Install it with an SSH server and nothing else: no desktop. The script
refuses any other system and says so. On other systems, install Docker and
gVisor yourself and use install.sh.
Run it¶
As root, or as a user with sudo: the script re-runs itself through sudo.
Settings are environment variables, so they work in front of sh setup.sh. Each is
also a flag. Under an explicit sudo, use the flags, because sudo drops
environment variables.
Settings are kept in /var/lib/openblox/settings. Running the script again
with none keeps them, and settings you give again replace them, except client
names, which add to the list.
| Variable | Flag | Default | Meaning |
|---|---|---|---|
OPENBLOX_VERSION |
--version |
latest release | daemon and image version, in lockstep |
OPENBLOX_LISTEN |
--listen |
none (Unix socket only) | host:port for the mTLS listener |
OPENBLOX_CLIENTS |
--client (repeatable) |
sandbox-caller when listening |
client certificate names to issue and allow |
OPENBLOX_ALLOW_FROM |
--allow-from |
any source | CIDR allowed to reach the listener |
OPENBLOX_MEMORY_MB |
--memory-mb |
2048 |
per-sandbox memory |
OPENBLOX_MAX_SANDBOXES |
--max-sandboxes |
sized from RAM | concurrent sandboxes |
The phases, in order:
verify_system: checks the OS, the architecture and cgroup v2, and sizesmax_sandboxesfrom RAM.install_docker: installs Docker Engine from Docker's apt repository. If Docker is already installed, it is left as it is.install_gvisor: installsrunscfrom gVisor's apt repository and registers it with Docker.install_openbloxd: installs the binary throughinstall.sh, which verifies the checksum and, whenghis present, the build attestation. It also creates theopenbloxdsystem user.pin_image: pulls the sandbox image for the same version and pins its digest.write_config: writes/etc/openbloxd/config.yamlwith one profile,code-exec: no network, non-root, capped CPU, memory, disk and processes.issue_certs: runs only with a listener. It issues the server certificate and one client bundle per caller.harden: adds an nftables firewall and turns on automatic security updates. The firewall drops inbound traffic except SSH, on the portssshdreports, and the listener.create_uninstall: writes/usr/local/bin/openblox-uninstall.sh.create_systemd_service, thenservice_enable_and_start: install, enable and start the service.smoke_test: creates a sandbox, requires that its kernel is gVisor and that it cannot reach the network, then removes it.
If a phase fails, the script names it and says what to do next.
Remote callers¶
Your application usually runs on another machine. Give the daemon a listener on a private address and name one client per caller:
curl -fsSL https://openblox.sh/setup.sh -o setup.sh &&
OPENBLOX_LISTEN=10.0.0.5:9443 OPENBLOX_CLIENTS="app-staging app-prod" \
OPENBLOX_ALLOW_FROM=10.0.0.0/24 sh setup.sh
Each caller gets a bundle in /etc/openbloxd/clients/<name>/:
client.crtandclient.key: the caller's identity.ca.crt: the CA that signed the daemon's certificate, which the caller verifies the daemon against.
Copy the directory to that caller over a channel you trust, such as scp.
The key is the one secret in this setup, and it never needs to go anywhere
else. Then connect:
client, err := brokerclient.NewRemote("10.0.0.5:9443", brokerclient.TLSFiles{
CertFile: "/etc/app/openblox/client.crt",
KeyFile: "/etc/app/openblox/client.key",
CAFile: "/etc/app/openblox/ca.crt",
})
The client CA and the server CA are separate. The daemon trusts every certificate the client CA signs, so that CA signs callers and nothing else. Both CA keys stay on the host.
To add a caller, run the script again with just its name, for example
--client app-dev. Only the new bundle is issued. The script never rewrites
your config, so it warns that the name is not yet allowed. Add it to
allowed_client_cns in /etc/openbloxd/config.yaml, then run
systemctl restart openbloxd. To remove a caller, delete its name there and
restart. There is no revocation list.
The listener is bound to the address you give it. How your callers reach that address, whether over a LAN, a VPN or a tailnet, is up to you.
Benchmarking¶
bench.sh measures the daemon on this host, through its own socket, the way
a caller uses it. It is optional, and setup.sh never runs it: setup's last
line prints the stress command, for when you want to know where your hardware
tops out. It needs root, curl and a running openbloxd, and it
removes every sandbox it creates. The source is
www/bench.sh.
The default mode, latency, runs one sandbox at a time. Below is a 2012
Xeon E3-1220 v2 with 4 cores and 4 GB, under gVisor:
min median max
create sandbox 1165 1233 1306 ms
exec: true 43 45 45 ms
exec: python3 startup 73 75 83 ms
exec: job 394 396 402 ms
delete sandbox 812 830 840 ms
on the host, no sandbox:
python3 startup 19 20 21 ms
job 227 232 233 ms
- Create is what the first call in a session pays.
- Exec is what each later call pays on top of its own work.
- Job fills 64 MiB and runs a CPU loop. Beside the same job on the host, it shows what the sandbox costs for real work.
--mode stress answers a different question: how many sandboxes can work at
once before the host is the bottleneck. It steps concurrency up (1, 2, 4, 8,
16 by default), and every sandbox at a level runs the job at the same time:
On the same machine:
sandboxes create_ms job_p50 job_max jobs/s cpu% free_MiB failed
1 1445 393 403 2.51 29 3125 0
2 1995 423 428 4.70 55 2981 0
4 3832 491 534 7.92 97 2867 0
6 5453 706 926 7.80 97 2645 0
8 7154 879 1263 7.79 97 2420 0
12 11228 1389 1952 7.78 98 1963 0
16 14234 1881 2401 7.71 99 1506 0
throughput levels off at 4 concurrent sandboxes (7.92 jobs/s); past that, more
sandboxes add little throughput: the CPU is saturated, so jobs queue for it.
Read the columns like this:
- jobs/s is the throughput. Once it stops growing, more concurrent
sandboxes only make each job slower (job_p50).
- cpu% near 100 means the CPU is the limit.
- free_MiB falling toward the floor means memory is the limit.
- failed above 0 means jobs are being killed, usually for memory.
The test stops by itself at the first failure, or before the host's free
memory drops under --floor-mb (512 MiB by default), so it cannot starve SSH.
Stress mode never exceeds the profile's max_sandboxes. To find the
hardware's limit rather than the config's, raise it in
/etc/openbloxd/config.yaml for the run, then put it back. Edit the file in
place, so it keeps its owner and mode (root:openbloxd, 0640); the daemon
cannot read a copy owned by root alone. Run systemctl restart openbloxd
after each edit.
Options: --rounds, --levels "1 2 4", --job-mb, --cpu, --floor-mb and
--profile. The header of the script describes each one.
Upgrading¶
Run the script again. It keeps your settings, so there is nothing to repeat:
curl -fsSL https://openblox.sh/setup.sh -o setup.sh && sh setup.sh # the latest release
curl -fsSL https://openblox.sh/setup.sh -o setup.sh && OPENBLOX_VERSION=vX.Y.Z sh setup.sh # or a chosen one
It upgrades the daemon and pulls the matching image. It never rewrites
config.yaml, because you may have edited it. Instead it prints a diff
against what it would write now, and leaves that version in
config.yaml.new. Copy the new image: line across by hand, then run
systemctl restart openbloxd.
Uninstalling¶
Setup records everything it creates in /var/lib/openblox/installed, and the
uninstaller removes exactly that list:
- the sandboxes
- the service, the binary and the
openbloxduser /etc/openbloxd, including the CAs and client bundles- the firewall table
- the pinned image
- the apt sources setup added
- the packages setup installed
Anything that was there before setup first ran stays, including a Docker you
had already installed. /var/lib/docker is always kept. Delete it yourself if
you no longer need it.
Known limits¶
- A client certificate is not bound to a profile. Every allowed caller can use every profile on the host. Give callers that need different limits different hosts.
- Kata is not provisioned. The script installs gVisor, the default runtime. To use Kata, see Using Kata instead.