🏰 Camelot AI Docs

Cluster runners deployment guide

🏰 Camelot AI runs every agent CLI inside an isolated container scheduled by Docker Swarm. This guide covers what an operator needs to set up.

TL;DR

For a hosted CapRover deployment:

  1. Run tecnativa/docker-socket-proxy as a Swarm service on the manager, exposing only SERVICES, TASKS, NETWORKS, NODES, SECRETS.
  2. Label worker nodes with camelot-home=node-X for every X you want to host user runner containers on.
  3. Set 🏰 Camelot AI's env vars (CapRover app config):
    • RUNNER_BACKEND=swarm
    • DOCKER_HOST=tcp://docker-socket-proxy:2375
    • ENCRYPTION_KEY=<32-byte base64>
    • RUNNER_GLOBAL_MAX=20 (or whatever your cluster can handle)
    • RUNNER_PER_USER_MAX=2 (default tier)
    • RUNNER_NETWORKS — optional; defaults to auto (see Runner networking). Set none to keep runners isolated.
  4. Build and push the runner images (.github/workflows/runner-images.yml). Reference them from Agent.runner_image (the Agent CLI's field). A single project can pin a different image via its runner_image_override field (set in the project's edit form) — when non-nil, it wins over the Agent CLI's Agent.runner_image for every task in that project.
  5. For each user, in 🏰 Camelot AI's profile UI:
    • Add their API keys / GitHub PAT as Credentials. SecretSync will populate camelot_user_<id>_<kind> Swarm secrets automatically.
  6. Optionally pin runners to a node label (see Node labels below).

Deployment topologies

🏰 Camelot AI does not require running on a manager. Three supported shapes:

A. Local socket (smallest installs)

manager + camelot

  /var/run/docker.sock ─bind-mount─► camelot

🏰 Camelot AI scheduled onto a manager, the manager's socket mounted in. DOCKER_HOST=unix:///var/run/docker.sock. Simplest, but 🏰 Camelot AI can only ever run on a manager.

B. Remote TCP + TLS (max flexibility)

manager           camelot (anywhere)

 dockerd ──tcp/2376─► REQ

   (mTLS)

Daemon configured with tcp://0.0.0.0:2376 + client cert auth. 🏰 Camelot AI env: DOCKER_HOST=tcp://<manager>:2376 plus DOCKER_CERT_PATH mounted as a Swarm secret. Lets 🏰 Camelot AI live anywhere reachable.

C. Socket proxy as a Swarm service (recommended)

manager (proxy)             camelot (anywhere)

  socket ──► docker-socket-proxy ──tcp/2375─► Camelot

Proxy service constrained to managers; 🏰 Camelot AI reaches it via the in-cluster overlay network. Survives 🏰 Camelot AI getting rescheduled to a worker, restricts the API surface, and Swarm handles proxy failover.

Example proxy service:

docker service create \

  --name docker-socket-proxy \

  --network camelot \

  --constraint node.role==manager \

  --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \

  --env SERVICES=1 --env TASKS=1 --env NODES=1 \

  --env NETWORKS=1 --env SECRETS=1 \

  tecnativa/docker-socket-proxy:latest

🏰 Camelot AI env: DOCKER_HOST=tcp://docker-socket-proxy:2375.

Node labels

Runners can be pinned to a node so a user's profile volume stays local (no NFS required). Label each candidate worker:

docker node update --label-add camelot-home=node-a worker-1

docker node update --label-add camelot-home=node-b worker-2

Then set the same label string as a pin, at whichever level fits (admin-only, in 🏰 Camelot AI's UI). Each pin control shows a dropdown of labels discovered live from GET /nodes when the Docker/Swarm API is reachable, falling back to a free-text field otherwise (e.g. a non-Swarm RUNNER_BACKEND, or no node labelled yet):

  • Project — /projects/:id, applies to that project only.
  • User — /admin/users, applies to all of that user's projects.
  • Instance default — /admin/settings, used when neither of the above is set.

Precedence when more than one applies to a runner: project pin > project owner's user pin > instance default. Changing a pin does not restart already-running containers — it takes effect for runners started or adopted from that point forward.

Pool capacity

Env var Default What it controls
RUNNER_GLOBAL_MAX 20 Cluster-wide concurrent runner count.
RUNNER_PER_USER_MAX 2 Free-tier cap per user. Override per user via RunnerPool.set_user_cap/2.

Sessions never get refused — they queue. Wait time surfaces in the UI; a future paid tier will let users buy higher per-user caps.

Recovering interrupted runs

A run cut short by infrastructure — a deploy replacing the runner container, a Swarm service that vanished — is re-queued, not errored: the every-minute dispatcher runs the task again from its current stage.

Env var Default What it controls
RUNNER_MAX_INTERRUPT_REQUEUES 3 Consecutive automatic re-queues before a task is errored instead. Reset on any forward progress, so only a task that can never run exhausts it.
RUNNER_REDEPLOY_WAIT_MS 60000 How long a force-redeployed service has to produce a runnable replica before its runner is treated as lost. Raise it if your runner images are large and cold pulls are slow.

Re-running a task reuses its service name, and Swarm will not delete a secret that any service still references. Because a secret can only be rotated by deleting and recreating it, 🏰 Camelot AI removes the previous run's service before refreshing the task's secrets — otherwise the new runner would mount the old GitHub App installation token, which expires an hour after it is minted and shows up as remote: Invalid username or token from git clone at container boot. If you delete runner services by hand, delete the matching camelot_task_<task-id>_gh_token secret too.

Runner networking

A task-runner container joins the Swarm bridge network for outbound internet, but Swarm's service-discovery DNS (e.g. a CapRover srv-captain--db) only resolves between containers on the same overlay. A runner that runs the project's test suite against a shared database — DATABASE_URL=ecto://…@srv-captain--db:5432/… — needs to be on that overlay, or it fails with "Name or service not known" / nxdomain.

RUNNER_NETWORKS controls which overlays runners join.

Value What happens
(unset) / auto Default. 🏰 Camelot AI copies the overlay network(s) its own service is on onto each runner.
net-a,net-b Explicit network names/IDs. Combine with auto (auto,net-a) to add to the discovered set.
none Keep runners isolated — bridge only, no overlay.

auto (the default) — a runner reaches exactly what 🏰 Camelot AI reaches, with nothing to hardcode. Discovery reads the app's own task and service via the Docker API (TASKS + SERVICES, already in the socket-proxy allow-list) and memoizes the result. If it can't complete (e.g. plain-Docker / non-Swarm self-hosting, where there is no overlay to discover), runners start with no extra network and a warning is logged — never a hard failure.

Security note: auto places runners on the same overlay as 🏰 Camelot AI, so they can reach the control-plane services on it (including 🏰 Camelot AI's own DB). Set RUNNER_NETWORKS=none — or an explicit, dedicated overlay — if you need runners isolated from the control plane.

Backups & disaster recovery

  • DB is the source of truth. Back up PostgreSQL.
  • Profile volumes are caches. Losing one only costs the next session's bootstrap time (asdf re-install, CLI caches rebuilt). The entrypoint re-materialises credentials from secrets on every spawn.
  • Swarm secrets: rotate via 🏰 Camelot AI UI (or SecretSync.reconcile/2); no manual docker secret calls needed.

Local development

Drop RUNNER_BACKEND=local (the default in dev/test). 🏰 Camelot AI runs the CLI directly on your host via Port.open — identical to the pre-runner behaviour. No Docker required.

To test the containerised path on a dev VM without Swarm: RUNNER_BACKEND=docker. Set up a Docker daemon and bind-mount the socket (or expose it over TCP).