Jarvis on Kubernetes

Plain manifests, no Helm, no operator. They are written to be read as much as applied: every value that is not a Kubernetes default is there because something in Jarvis behaves a particular way, and the comment beside it says which. Read them before you apply them — that is the point of shipping them this way.

You probably do not need this. Docker Compose runs one of everything, takes one command, and is what most instances run. Come here when you want Jarvis to keep serving while a node reboots, and when you already operate a cluster — this directory assumes you have opinions about ingress, storage and a database, because it deliberately does not have them for you.

What has been proven, and what has not. This stack has been brought up on a real cluster — four nodes, three Postgres instances, three Redis with three sentinels, two api pods — and kept serving while its Postgres primary and its Redis primary were killed underneath it. What has NOT happened is a long-running estate serving production traffic this way. If you are early, watch it rather than assume it, and read the three warnings below before you apply anything, because each of them fails silently.

What runs more than once, and what cannot

"Highly available" is not a property a deployment has as a whole, so here it is component by component.

component	replicas	why
`jarvis-web`	any number	stateless nginx over static files
`jarvis-api`	any number	the coordination lives in Postgres and Redis, not in the process
Redis	one logical primary, three processes	Sentinel fails over; it does not spread writes
Postgres	one primary, replicas behind it	and Jarvis depends on there being exactly one
`jarvis-migrate`	one at a time	enforced by a Postgres advisory lock, not by you
`jarvis-mint-secrets`	once, ever	enforced by the API server

The single-writer database is not a limitation Jarvis has failed to overcome — it is what makes the multi-pod api safe. A partial unique index is what stops two pods running the same assistant loop, and an index can only do that because one server decides. Do not point pods at read replicas.

What you need before you apply anything

x86-64 nodes. The api and web images are published for linux/amd64 only, so an arm64 pool leaves the pods in ImagePullBackOff with no matching manifest. The machines you administer have no such limit — the agent ships arm64 builds.
A cluster you can schedule on. kubectl describe node | grep Taints first. A three-node cluster is often three control-plane nodes, and their NoSchedule taint means nothing here schedules at all. These manifests carry no toleration on purpose: adding one is a decision about your cluster.
Postgres 16, highly available, outside these manifests. Use an operator — CloudNativePG or Zalando's — rather than a hand-written StatefulSet. A database you hand-rolled is a database you hand-restore. max_connections must cover (replicas + maxSurge + jobs) × connection_limit + reserve: with 3 api pods at 10 connections each, a migration Job at 5, and headroom, about 150. The stock Postgres image ships 100, and exhausting it presents as an api that starts and then cannot serve.
Redis. 08-redis-sentinel.yaml deploys one, or point REDIS_URL at a managed Redis that fails over behind a single address and skip that file. Persistence is not required: everything Jarvis keeps in Redis is ephemeral by design, which is exactly why the run lock is in Postgres instead.
An Ingress controller and a TLS terminator. Jarvis speaks plain HTTP.

Order

kubectl apply -f 00-namespace.yaml

# The secrets, ONCE. Never regenerate them on an instance that has stored anything.
kubectl apply -f 01-secret.example.yaml   # after editing — the half you write
kubectl apply -f 01-secret-job.yaml       # the half that is random bytes (optional; see the file)
kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m

kubectl apply -f 02-config.yaml
kubectl apply -f 08-redis-sentinel.yaml   # only if you are running your own Redis

# The schema, to completion, before any api pod starts.
kubectl apply -f 03-migrate-job.yaml
kubectl wait --for=condition=complete job/jarvis-migrate -n jarvis --timeout=15m

kubectl apply -f 04-api.yaml
kubectl apply -f 05-web.yaml
kubectl apply -f 06-ingress.yaml
kubectl apply -f 07-disruption.yaml

Every image says REPLACE_ME: pin them by digest, not by tag. :stable is a moving name, so replicas that restart at different times land on different builds and a rollback becomes "hope the tag still points where it did".

docker buildx imagetools inspect git.luxit.be/luxit/jarvis-api:stable

On every upgrade the migration Job runs again before the Deployments roll. It is idempotent and takes a Postgres advisory lock, so running it twice is slow rather than harmful — but it must finish before new pods start, which is what the kubectl wait is for. With Helm or Argo this becomes a pre-upgrade hook or a sync-wave; as plain YAML it is an ordering you keep yourself.

Then open the address and install it. Everything in the main README about the install screen applies unchanged — including that whoever reaches an uninstalled instance first owns it, with no deadline. Point a public hostname at this after you have installed, not before.

The three things most likely to bite

Never mint secrets per pod. On Compose a one-shot generates the vault key and the JWT secrets into a volume every container shares, and its guarantee that it does so exactly once is that the filesystem refuses a second create. As a per-pod initContainer on an emptyDir, each replica mints its own — and the consequence is not a crash. Pod A seals a credential under a key pod B does not have; pod B reports it as corrupt; both log a clean boot. By the time anybody opens a vault entry there are three keys in circulation.

01-secret-job.yaml is that one-shot done correctly here: it creates the Secret once and lets the API server's refusal to create it twice be the exclusion. Its identity can create a Secret and cannot get one, so it cannot read back what it or anyone else wrote. Mint by hand instead if you prefer — the one thing that is not an option is per-pod.

Either way, back the vault key up somewhere that is not this cluster's etcd. Deleting the Secret and re-running the Job mints a new key against data sealed with the old one, silently, with no way back.

TRUST_PROXY_HOPS is one hop here, not two. The Compose files say 2 because a request passes your TLS terminator and then the web container's nginx. Here the Ingress routes /api/ straight at the api Service, so there is exactly one proxy rewriting X-Forwarded-For. Left at 2, req.ip becomes chosen by the caller: anyone can send X-Forwarded-For: 198.51.100.7 and have that written into session records, audit rows and the key the anonymous WebAuthn budget counts on.

Do not add session affinity. It is the reflex the moment websockets are involved and it solves nothing here: the hard case is co-locating an operator's browser with an agent's websocket, which arrives from a different network at a different time and whose id is not even known at the browser's handshake. No ingress annotation can express that. It is handled in the application, over Redis. Both ends pin transports: ["websocket"], so there is no polling to make sticky in the first place.

On exactly three nodes

The common case, and where the arithmetic is tightest.

What lands where. Redis and the sentinels both use required anti-affinity, so three of each means one per node — six pods, two per node, no choice left to the scheduler. Postgres at three instances is the same. That is what you want, and it has a consequence: during a node drain, one Redis pod and one sentinel have nowhere to go and stay Pending until the node returns. That is correct — a quorum of two still holds — but kubectl drain will sit there unless the PodDisruptionBudgets are applied to tell it what is safe.

Three api replicas, not two. 04-api.yaml ships two because that is the smallest number that proves the code is not single-instance. On three nodes, three is strictly better: one per node, and losing any node leaves two. Change replicas: 2 to 3 in it and in 05-web.yaml, and size max_connections for it.

Three nodes tolerates losing one. Not two. That is a property of majority quorums, and the answer to wanting more is a fourth and fifth node, not a different configuration.

What a failover actually costs

Measured, not estimated.

Redis, with its primary killed under a running api:

The api rides it out without restarting. Readiness answered 503 for about fifteen seconds and then 200 again, on the same process — the pod leaves the Service's endpoints while it cannot reach Redis, and returns when it can. Liveness never touches Redis, which is why the pod is not killed and restarted into a cold start.
Commands survive. 99 writes issued across the window, 0 rejected, the last readable from the promoted primary afterwards. Commands queue rather than fail, so a failover is a pause.
Published messages do not. Around 45% of the messages published during the window reached nobody. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change that. In practice: a few seconds in which something happening on one api replica may not reach another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason.

Postgres, primary deleted with --force --grace-period=0:

Three seconds. Both api pods answered 503 {"database":"unreachable"} and were back to 200 on the fourth poll. Both degraded together, which is what pointing every pod at one endpoint means.
Zero container restarts, across two failovers, on pods up throughout — liveness deliberately not touching the database is what keeps three seconds from becoming a cold start of everything.

A failover during the migration Job aborts it with a lost-connection error, which is the correct outcome. Run it again.

What is still worth knowing

AGENT_RELEASE_DIR is per-pod, and must be. It holds the compiled agent binaries. An emptyDir filled by an initContainer is right; a shared RWO volume is not — either every pod is pinned to one node, or the second sits in Multi-Attach error. Leaving it unset is supported: everything works except the agent installer, which answers 503 saying no build is published.
No application volume, and it should stay that way. Avatars, documents and terminal recordings are all in Postgres; exports are rendered in memory.
Autoscale jarvis-web, not jarvis-api. Every api scale-in kills a pod holding agent websockets, live runs and open shells, and CPU is a poor proxy for a load made of long-lived connections. Raise api replicas deliberately.
A rollout still cuts what is in flight. An assistant run is interrupted and resumed with a note; a terminal is not resumable and says so; and the web bundle's content-hashed chunks are per-image, so a browser holding the old shell can 404 on a lazy import during the rollout window.
Validation is not admission. These manifests satisfy the restricted Pod Security Standard that 00-namespace.yaml enforces. If you edit them, note that a schema validator cannot see that policy: a missing seccompProfile or a container without a securityContext does not fail validation, it produces a Deployment reporting FailedCreate and sitting at zero replicas.

Backups

Everything in the main README applies, with one substitution: the vault key is not in a Docker volume, it is in the jarvis-generated Secret.

kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d

Back that up somewhere that is not this cluster, and not alongside your database dump. A database backup that travelled with its own key is a backup that decrypts itself.

12 KiB Raw Blame History Unescape Escape