# Agent node hardware

> What an agent node needs: the Agent Runner and the LLM inference server, with the hardware, software, model and hosting options for each.

An agent node runs two components, on one machine or on separate hosts:

- **Agent Runner** (`agent-runner`) listens for agent requests onchain, runs each agent's container in a sandbox and submits a signed result as a member of the request's subcommittee.
- **LLM inference server** serves the deterministic LLM, Qwen3-30B, over an OpenAI-compatible API. The Agent Runner forwards the LLM requests of agent containers to it.

## Agent Runner

### Prerequisites

- **Linux on x86-64.** The published `agent-runner` binary is a statically linked Linux x86-64 executable. There are no macOS or arm64 builds, and on another platform it fails with `exec format error`.
- **Docker.** The runner manages agent containers through the Docker daemon, so it must reach the Docker socket: run it as `root` or add its user to the `docker` group.

### Hardware

| Resource | Minimum | Recommended |
| --- | --- | --- |
| CPU | 8 vCPUs (x86-64) | 16 or more vCPUs (x86-64) |
| RAM | 16 GB | 32 GB |
| Disk | 50 GB SSD | 100 GB SSD |
| Network | 100 Mbps | 1 Gbps |
| OS | Linux (x86-64) | Ubuntu 24.04 LTS (x86-64) |

### Software

- **Docker** (required) runs the agent containers.
- **iptables** isolates the sandbox network and applies its firewall rules.
- **gVisor** (recommended): its `runsc` runtime adds another layer of container isolation.

### Sizing

- Each agent container uses about 512 MB of RAM at baseline, so memory grows with the number of agents running at once.
- The runner caches the container images it downloads, so disk use grows with the number of different agents it runs.
- The runner sends heartbeats and submits transactions through the Somnia RPC endpoint, so it needs a stable, low-latency connection to it.

## LLM inference server

The LLM inference server runs [SGLang](https://github.com/sgl-project/sglang) to serve the deterministic LLM that agents such as [LLM Inference](./base-agents/llm-inference) call. It needs a high-end GPU.

### Hardware

| Resource | Specification |
| --- | --- |
| GPU | NVIDIA RTX PRO 6000 Blackwell (96 GB GDDR7) |
| CPU | 48 vCPUs |
| RAM | 180 GB |
| Disk | 200 GB SSD, for the OS, model weights and Docker images |
| Network | 1 Gbps or more |

### Model

| Property | Value |
| --- | --- |
| Model ID | `Qwen/Qwen3-30B-A3B` |
| Architecture | Mixture of Experts: 30B parameters, about 3B active |
| VRAM use | About 60 GB (FP16), within a single 96 GB GPU |
| Context length | 32,000 tokens |
| Serving | SGLang, image `lmsysorg/sglang:v0.5.8.post1` |
| API | OpenAI-compatible (`/v1/chat/completions`) on port 8000 |

### Hosting options

The server only has to expose an OpenAI-compatible endpoint that the Agent Runner can reach, so it can run anywhere with a suitable GPU.

| Provider | Instance | GPU | vCPUs | RAM |
| --- | --- | --- | --- | --- |
| Google Cloud | `g4-standard-48` | 1x RTX PRO 6000 (96 GB) | 48 | 180 GB |
| AWS | `g7e.8xlarge` | 1x RTX PRO 6000 (96 GB) | 32 | 256 GB |
| Microsoft Azure | `Standard_NC128lds_xl_RTXPRO6000BSE_v6` | 1x RTX PRO 6000 (96 GB) | 128 | 256 GB |

- **Google Cloud.** Give a G4 machine a 200 GB Hyperdisk Balanced boot disk: pass `--boot-disk-type=hyperdisk-balanced` to `gcloud`, since G4 machine types reject the default `pd-balanced`. G4 also needs `NVIDIA_RTX_PRO_6000` GPU quota, which is 0 on a new project, so request an increase before you provision.
- **CoreWeave.** Request a single-GPU RTX PRO 6000 Blackwell node (96 GB) through CoreWeave Kubernetes Service, in the `US-EAST-04A` availability zone.
- **Bare metal.** Any Linux x86-64 machine with an RTX PRO 6000 Blackwell GPU, NVIDIA drivers, Docker and the NVIDIA Container Toolkit.

#### RunPod

RunPod runs GPU pods with no cloud quota or infrastructure to manage:

1. Create a pod with a GPU of 80 GB VRAM or more, such as an RTX PRO 6000 (96 GB), A100 80 GB or H100 80 GB.
2. Give it a container disk of at least 100 GB. The SGLang image is about 20 GB once pulled, and SGLang caches compiled kernels on first launch, so RunPod's default (often 20 GB) is too small.
3. Attach a volume of at least 100 GB. The model weights, about 60 GB, download to `HF_HOME` on first boot, and the volume keeps them across restarts.
4. Use the image `lmsysorg/sglang:v0.5.8.post1`.
5. Set these environment variables:

   ```
   PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
   SGLANG_DISABLE_DEEPGEMM=1
   SGLANG_ENABLE_JIT_DEEPGEMM=0
   SGLANG_GEMM_BACKEND=fallback
   HF_HOME=/workspace/.cache/huggingface
   ```

6. As the container start command, use the `python3 -m sglang.launch_server` command and its flags from [Run the LLM container](#run-the-llm-container), with your key as the `--api-key` value. Never leave it empty.
7. Expose port 8000 as a TCP port, not through RunPod's HTTP proxy: the Agent Runner needs a direct connection. Once the pod is running, take its public IP and mapped port from the pod's Connect tab.
8. Point the Agent Runner at it with `--llm-upstream-url=http://<public-ip>:<public-port>`, and set `--llm-api-key` to the key you started SGLang with. See [Secure the server](#secure-the-server).

The first boot takes 10 to 20 minutes before the server answers: about 2 to 5 minutes to pull the image, 5 to 10 to download the weights and 1 to 2 to compile kernels. Later boots with a volume take 1 to 2 minutes.

### Run the LLM container

Generate an API key for the server and keep it in a variable:

```bash
export SGLANG_API_KEY=$(openssl rand -hex 32)
```

Then start the server:

```bash
docker run -d \
  --name sglang \
  --gpus all \
  --shm-size 16g \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/workspace/.cache/huggingface \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e SGLANG_DISABLE_DEEPGEMM=1 \
  -e SGLANG_ENABLE_JIT_DEEPGEMM=0 \
  -e SGLANG_GEMM_BACKEND=fallback \
  -e HF_HOME=/workspace/.cache/huggingface \
  lmsysorg/sglang:v0.5.8.post1 \
  python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-30B-A3B \
    --host 0.0.0.0 \
    --port 8000 \
    --tp 1 \
    --context-length 32000 \
    --attention-backend triton \
    --trust-remote-code \
    --enable-deterministic-inference \
    --enable-custom-logit-processor \
    --tool-call-parser qwen25 \
    --reasoning-parser qwen3 \
    --api-key "${SGLANG_API_KEY:?SGLANG_API_KEY must be set and nonempty}"
```

The command stops if `SGLANG_API_KEY` is empty, because SGLang started with an empty key accepts requests without authentication. Give the Agent Runner the same key with `--llm-api-key`.

> **Warning** `--enable-custom-logit-processor` is required. The Agent Runner uses a custom logit processor to keep LLM outputs deterministic and constrained. Without the flag, SGLang rejects those requests and LLM agent requests fail.

The first run downloads the model weights, about 60 GB, into `~/.cache/huggingface`, and later starts reuse them. Under `root` or a systemd service, `~` is that user's home directory, such as `/root`. To keep the weights at a fixed path such as `/opt/huggingface`, change both the `-v` mount and `HF_HOME`.

Check that the server is ready:

```bash
curl -H "Authorization: Bearer $SGLANG_API_KEY" http://localhost:8000/v1/models
```

### Secure the server

With `--enable-custom-logit-processor`, SGLang loads code sent in a request's `custom_logit_processor` field, so anyone who can reach its port can run code on the server. Never leave the LLM server open without authentication:

- Start SGLang with `--api-key`, as in the command above, and give the Agent Runner the same key with `--llm-api-key`. The runner sends it as a bearer token on every request.
- Allow traffic to port 8000 only from the Agent Runner's address, or keep the two on the same private network.
- On plain HTTP the key and the prompts cross the network unencrypted. If the server is reached over the public internet, put it behind an HTTPS endpoint or a tunnel and use that URL in `--llm-upstream-url`.

### Connect the Agent Runner

The Agent Runner reaches the server through its `--llm-upstream-url` and `--llm-api-key` flags. They take effect only with sandbox networking on: a runner started with `--disable-sandbox` ignores `--llm-upstream-url`. The URL's port is the port Docker publishes on the host. If you change SGLang's `--port`, change the container side of `-p` to match, for example `-p 3000:3000` with `--port 3000`, and set `--llm-upstream-url=http://<host>:3000`.

---

Every page is also Markdown: append `.md` to its URL, or ask for `text/markdown`. Found something wrong, missing or confusing in these docs? Tell us: `POST /api/feedback` on this site with JSON `{"message": "…", "context": "what you were doing and what happened", "page": "…"}`, or a plain-text body. `GET /api/feedback` gives the JSON Schema. The team reads every one.