Agent node hardware

An agent node runs two components, on one machine or on separate hosts:

• Agent Runner (agent-runner) listens for agent requests onchain, runs each agent's container in a sandbox and submits a signed result as a member of the request's subcommittee.
• LLM inference server serves the deterministic LLM, Qwen3-30B, over an OpenAI-compatible API. The Agent Runner forwards the LLM requests of agent containers to it.

Agent Runner

Prerequisites

• Linux on x86-64. The published agent-runner binary is a statically linked Linux x86-64 executable. There are no macOS or arm64 builds, and on another platform it fails with exec format error.
• Docker. The runner manages agent containers through the Docker daemon, so it must reach the Docker socket: run it as root or add its user to the docker group.

Hardware

Software

• Docker (required) runs the agent containers.
• iptables isolates the sandbox network and applies its firewall rules.
• gVisor (recommended): its runsc runtime adds another layer of container isolation.

Sizing

• Each agent container uses about 512 MB of RAM at baseline, so memory grows with the number of agents running at once.
• The runner caches the container images it downloads, so disk use grows with the number of different agents it runs.
• The runner sends heartbeats and submits transactions through the Somnia RPC endpoint, so it needs a stable, low-latency connection to it.

LLM inference server

The LLM inference server runs SGLang to serve the deterministic LLM that agents such as LLM Inference call. It needs a high-end GPU.

Hardware

Model

Hosting options

The server only has to expose an OpenAI-compatible endpoint that the Agent Runner can reach, so it can run anywhere with a suitable GPU.

• Google Cloud. Give a G4 machine a 200 GB Hyperdisk Balanced boot disk: pass --boot-disk-type=hyperdisk-balanced to gcloud, since G4 machine types reject the default pd-balanced. G4 also needs NVIDIA_RTX_PRO_6000 GPU quota, which is 0 on a new project, so request an increase before you provision.
• CoreWeave. Request a single-GPU RTX PRO 6000 Blackwell node (96 GB) through CoreWeave Kubernetes Service, in the US-EAST-04A availability zone.
• Bare metal. Any Linux x86-64 machine with an RTX PRO 6000 Blackwell GPU, NVIDIA drivers, Docker and the NVIDIA Container Toolkit.

RunPod

RunPod runs GPU pods with no cloud quota or infrastructure to manage:

1. Create a pod with a GPU of 80 GB VRAM or more, such as an RTX PRO 6000 (96 GB), A100 80 GB or H100 80 GB.

2. Give it a container disk of at least 100 GB. The SGLang image is about 20 GB once pulled, and SGLang caches compiled kernels on first launch, so RunPod's default (often 20 GB) is too small.

3. Attach a volume of at least 100 GB. The model weights, about 60 GB, download to HF_HOME on first boot, and the volume keeps them across restarts.

4. Use the image lmsysorg/sglang:v0.5.8.post1.

5. Set these environment variables:

   PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
   SGLANG_DISABLE_DEEPGEMM=1
   SGLANG_ENABLE_JIT_DEEPGEMM=0
   SGLANG_GEMM_BACKEND=fallback
   HF_HOME=/workspace/.cache/huggingface
   

   Copy

6. As the container start command, use the python3 -m sglang.launch_server command and its flags from Run the LLM container, with your key as the --api-key value. Never leave it empty.

7. Expose port 8000 as a TCP port, not through RunPod's HTTP proxy: the Agent Runner needs a direct connection. Once the pod is running, take its public IP and mapped port from the pod's Connect tab.

8. Point the Agent Runner at it with --llm-upstream-url=http://<public-ip>:<public-port>, and set --llm-api-key to the key you started SGLang with. See Secure the server.

The first boot takes 10 to 20 minutes before the server answers: about 2 to 5 minutes to pull the image, 5 to 10 to download the weights and 1 to 2 to compile kernels. Later boots with a volume take 1 to 2 minutes.

Run the LLM container

Generate an API key for the server and keep it in a variable:

export SGLANG_API_KEY=$(openssl rand -hex 32)

Then start the server:

docker run -d \
  --name sglang \
  --gpus all \
  --shm-size 16g \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/workspace/.cache/huggingface \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e SGLANG_DISABLE_DEEPGEMM=1 \
  -e SGLANG_ENABLE_JIT_DEEPGEMM=0 \
  -e SGLANG_GEMM_BACKEND=fallback \
  -e HF_HOME=/workspace/.cache/huggingface \
  lmsysorg/sglang:v0.5.8.post1 \
  python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-30B-A3B \
    --host 0.0.0.0 \
    --port 8000 \
    --tp 1 \
    --context-length 32000 \
    --attention-backend triton \
    --trust-remote-code \
    --enable-deterministic-inference \
    --enable-custom-logit-processor \
    --tool-call-parser qwen25 \
    --reasoning-parser qwen3 \
    --api-key "${SGLANG_API_KEY:?SGLANG_API_KEY must be set and nonempty}"

The command stops if SGLANG_API_KEY is empty, because SGLang started with an empty key accepts requests without authentication. Give the Agent Runner the same key with --llm-api-key.

Warning

--enable-custom-logit-processor is required. The Agent Runner uses a custom logit processor to keep LLM outputs deterministic and constrained. Without the flag, SGLang rejects those requests and LLM agent requests fail.

The first run downloads the model weights, about 60 GB, into ~/.cache/huggingface, and later starts reuse them. Under root or a systemd service, ~ is that user's home directory, such as /root. To keep the weights at a fixed path such as /opt/huggingface, change both the -v mount and HF_HOME.

Check that the server is ready:

curl -H "Authorization: Bearer $SGLANG_API_KEY" http://localhost:8000/v1/models

Secure the server

With --enable-custom-logit-processor, SGLang loads code sent in a request's custom_logit_processor field, so anyone who can reach its port can run code on the server. Never leave the LLM server open without authentication:

• Start SGLang with --api-key, as in the command above, and give the Agent Runner the same key with --llm-api-key. The runner sends it as a bearer token on every request.
• Allow traffic to port 8000 only from the Agent Runner's address, or keep the two on the same private network.
• On plain HTTP the key and the prompts cross the network unencrypted. If the server is reached over the public internet, put it behind an HTTPS endpoint or a tunnel and use that URL in --llm-upstream-url.

Connect the Agent Runner

The Agent Runner reaches the server through its --llm-upstream-url and --llm-api-key flags. They take effect only with sandbox networking on: a runner started with --disable-sandbox ignores --llm-upstream-url. The URL's port is the port Docker publishes on the host. If you change SGLang's --port, change the container side of -p to match, for example -p 3000:3000 with --port 3000, and set --llm-upstream-url=http://<host>:3000.