jarvis-selfhost/kubernetes/README.md

Jarvis on Kubernetes

Plain manifests, no Helm, no operator. They are written to be read as much as applied: every value that is not a Kubernetes default is there because something in Jarvis behaves a particular way, and the comment beside it says which. Read them before you apply them — that is the point of shipping them this way.

You probably do not need this. Docker Compose runs one of everything, takes one command, and is what most instances run. Come here when you want Jarvis to keep serving while a node reboots, and when you already operate a cluster — this directory assumes you have opinions about ingress, storage and a database, because it deliberately does not have them for you.

What has been proven, and what has not. This stack has been brought up on a throwaway four-node cluster — three Redis with three sentinels, two api pods behind the Ingress, synced by Argo CD — and served the console and its realtime connections through it. What has NOT happened is a Postgres or a Redis failover measured under the current api, or a long-running estate serving production traffic this way. If you are early, watch it rather than assume it, and read the three warnings below before you apply anything, because each of them fails silently.

What runs more than once, and what cannot

"Highly available" is not a property a deployment has as a whole, so here it is component by component.

component	replicas	why
jarvis-web	any number	stateless nginx over static files
jarvis-api	any number	the coordination lives in Postgres and Redis, not in the process
Redis	one logical primary, three processes	Sentinel fails over; it does not spread writes
Postgres	one primary, replicas behind it	and Jarvis depends on there being exactly one
jarvis-migrate	one at a time	enforced by a Postgres advisory lock, not by you
jarvis-mint-secrets	once, ever	enforced by the API server

The single-writer database is not a limitation Jarvis has failed to overcome — it is what makes the multi-pod api safe. A partial unique index is what stops two pods running the same assistant loop, and an index can only do that because one server decides. Do not point pods at read replicas.

What you need before you apply anything

• x86-64 nodes. The api and web images are published for linux/amd64 only, so an arm64 pool leaves the pods in ImagePullBackOff with no matching manifest. The machines you administer have no such limit — the agent ships arm64 builds.
• A cluster you can schedule on. kubectl describe node | grep Taints first. A three-node cluster is often three control-plane nodes, and their NoSchedule taint means nothing here schedules at all. These manifests carry no toleration on purpose: adding one is a decision about your cluster.
• Postgres 16, highly available, outside these manifests. Use an operator — CloudNativePG or Zalando's — rather than a hand-written StatefulSet. A database you hand-rolled is a database you hand-restore. max_connections must cover (replicas + maxSurge + jobs) × connection_limit + reserve: with 3 api pods at 10 connections each, a migration Job at 5, and headroom, about 150. The stock Postgres image ships 100, and exhausting it presents as an api that starts and then cannot serve.
• Redis. 08-redis-sentinel.yaml deploys one — three Redis, one of them the primary, and three sentinels — and 02-config.yaml's two REDIS_SENTINEL* lines point the api at it. Or use a managed Redis that fails over behind a single address: skip that file, delete those two lines, and set REDIS_URL to it. Never REDIS_URL alone at the file's redis Service: it is headless, two of the three pods it answers with are read-only replicas, and an api pod that draws one fails every write. Persistence is not required: everything Jarvis keeps in Redis is ephemeral by design, which is exactly why the run lock is in Postgres instead.
• An Ingress controller and a TLS terminator. Jarvis speaks plain HTTP.

Order

kubectl apply -f 00-namespace.yaml

# The secrets, ONCE. Never regenerate them on an instance that has stored anything.
kubectl apply -f 01-secret.example.yaml   # after editing — the half you write
kubectl apply -f 01-secret-job.yaml       # the half that is random bytes (optional; see the file)
kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m

kubectl apply -f 02-config.yaml
kubectl apply -f 08-redis-sentinel.yaml   # unless you use a managed Redis (then edit 02-config.yaml)

# The schema, to completion, before any api pod starts.
kubectl apply -f 03-migrate-job.yaml
kubectl wait --for=condition=complete job/jarvis-migrate -n jarvis --timeout=15m

kubectl apply -f 04-api.yaml
kubectl apply -f 05-web.yaml
kubectl apply -f 06-ingress.yaml
kubectl apply -f 06b-ingress-uploads.yaml   # uploads alone get a ~13 MiB body ceiling
kubectl apply -f 07-disruption.yaml

Every image says REPLACE_ME: pin them by digest, not by tag. :stable is a moving name, so replicas that restart at different times land on different builds and a rollback becomes "hope the tag still points where it did".

docker buildx imagetools inspect git.luxit.be/luxit/jarvis-api:stable

On every upgrade the migration Job runs again before the Deployments roll. It is idempotent and takes a Postgres advisory lock, so running it twice is slow rather than harmful — but it must finish before new pods start, which is what the kubectl wait is for. With Helm or Argo this becomes a pre-upgrade hook or a sync-wave; as plain YAML it is an ordering you keep yourself.

Then open the address and install it. Everything in the main README about the install screen applies unchanged — including that whoever reaches an uninstalled instance first owns it, with no deadline. Point a public hostname at this after you have installed, not before.

GitOps: Argo CD, Flux, or anything that reconciles

kustomization.yaml is the entry point. Point a controller at this directory and it renders every manifest here except one — and argocd-application.example.yaml is a worked Application to copy.

kubectl apply -k .    # the same thing, by hand

The exception is 01-secret.example.yaml, and it is left out on purpose. It is a template full of placeholders. A controller applies what it is pointed at, continuously, without asking — so a Secret full of REPLACE_ME inside the source of truth is a controller that overwrites your real credentials with placeholders on its next reconcile and puts them back every time you fix them. The api cannot reach its database, the vault key is wrong, and nothing about the sync looks unhealthy while it happens. Write jarvis-secrets by hand, or through SealedSecrets, ExternalSecrets or Vault, whose ciphertext is safe to commit. jarvis-generated is minted in-cluster and never exists in git at all.

The order is in the manifests, not in the filenames

A controller reads this directory as a set; 00-, 01- and 02- mean nothing to it. The order that the kubectl apply sequence above keeps by hand is carried by argocd.argoproj.io/sync-wave annotations on the resources themselves, so that it lives beside the thing being ordered:

wave	what	why it cannot be later
-3	Namespace	it carries the Pod Security labels that decide whether any of these pods are admitted
-2	ConfigMaps, the minter's RBAC, Redis	everything below reads the config; the api's readiness probe reaches Redis
-1	jarvis-mint-secrets (hook)	the migration reads DATABASE_URL out of what it writes
0	jarvis-migrate (hook)	an api pod against an unmigrated schema is the failure all of this prevents
1	api and web Deployments, Services	the web pod's /readyz proxies the api's health endpoint, so it cannot be ready before the api is
2	Ingress, PodDisruptionBudgets, the web's HPA	an Ingress applied early is a route to a 503; the HPA, below

Argo does not begin a wave until the previous one reports healthy, which is what the kubectl wait lines are doing by hand. The last wave is applied and not waited on, which is why the web's HorizontalPodAutoscaler is there rather than beside its Deployment: on a first install it has no metric for pods seconds old, Argo reads that as Degraded, and in wave 1 it failed the first sync every time. The application can read Degraded for the minute after a first install, until metrics-server has scraped the new pods; the sync itself has succeeded.

Why the two Jobs are hooks

Without this, the directory works exactly once. A Job's spec.template is immutable, so the second release — the first one with a different image digest — is a patch the API server refuses with field is immutable, and the sync fails there and stays failed. hook: Sync with hook-delete-policy: BeforeHookCreation deletes the previous Job before creating the new one, which is the supported way to say "run it again, with new contents".

The Sync phase rather than PreSync, deliberately: a PreSync hook runs before any ordinary resource is applied, which on a first sync is before the namespace, the ConfigMap and the Secret these Jobs read exist. In the Sync phase they are ordered by the same waves as everything else.

Two consequences worth knowing:

• The previous run's Job object, and its logs, are gone by the time the next one starts. Ship the logs if you need them after the fact.
• A migration waiting on the schema lock is ended by its pod, not its Job. Under Argo, deleting the jarvis-migrate Job leaves it Terminating behind the hook's finalizer, its pod still waiting and the sync waiting for good. Delete the pod instead — it stops in about a second — then end the sync's operation (argocd app terminate-op <application>).
• Every sync runs the migration. That is safe — the phase is idempotent and holds a Postgres advisory lock — and it is why the example Application leaves automated sync OFF. A commit that applies a schema change with nobody watching turns the one thing this directory works hardest to make visible, a migration that REFUSES because it would drop a column, into a notification.

Pin the images in one place

kustomization.yaml carries an images: block, so the three digests are set once rather than in three files. Three, not two: jarvis-agent-dist is an initContainer in 04-api.yaml and is easy to miss, and missing it renders cleanly while the api pod never starts.

A version bump is then those lines and nothing else.

What most estates should actually do

Point the Application at your repository, not at this one, with this directory as a remote base:

# your-repo/jarvis/kustomization.yaml
resources:
  - https://git.luxit.be/Luxit/jarvis-selfhost//kubernetes?ref=v1.2.3
images:
  - name: git.luxit.be/luxit/jarvis-api
    digest: sha256:...
patches:
  - path: replicas.yaml      # your node count, the host of BOTH Ingresses, your storage class

Then the digests you run are in a commit you control, upgrading is a diff you can review and revert, and ?ref= pins the base to a tag rather than a branch that moves under you. Pin it.

There are two Ingresses on the one host, and the host patch must reach both: jarvis, and jarvis-uploads in 06b-ingress-uploads.yaml, TLS block included. Uploads have an Ingress of their own because annotations apply to every path of the object that carries them, and they are the one path that needs bodies over 13 MiB. An overlay that rewrites only the first leaves uploads on the /api/ rule, where small files still work and anything between 4 and 13 MB meets the controller's own 413 — so it looks fine until somebody attaches a large PDF.

To check it once deployed, send an upload of more than 4 MB without signing in, on your own host:

head -c 5000000 /dev/zero > big.bin
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' -F file=@big.bin \
  https://jarvis.example.com/api/uploads/orgs/x/conversations/y/attachments

401 application/json is Jarvis turning away an anonymous request, so the body got through. A 413 with an HTML page is the controller, and means the host patch did not reach jarvis-uploads.

The three things most likely to bite

Never mint secrets per pod. On Compose a one-shot generates the vault key and the JWT secrets into a volume every container shares, and its guarantee that it does so exactly once is that the filesystem refuses a second create. As a per-pod initContainer on an emptyDir, each replica mints its own — and the consequence is not a crash. Pod A seals a credential under a key pod B does not have; pod B reports it as corrupt; both log a clean boot. By the time anybody opens a vault entry there are three keys in circulation.

01-secret-job.yaml is that one-shot done correctly here: it creates the Secret once and lets the API server's refusal to create it twice be the exclusion. Its identity can create a Secret and cannot get one, so it cannot read back what it or anyone else wrote. Mint by hand instead if you prefer — the one thing that is not an option is per-pod.

Either way, back the vault key up somewhere that is not this cluster's etcd. Deleting the Secret and re-running the Job mints a new key against data sealed with the old one, silently, with no way back.

TRUST_PROXY_HOPS is one hop here, not two. The Compose files say 2 because a request passes your TLS terminator and then the web container's nginx. Here the Ingress routes /api/ straight at the api Service, so there is exactly one proxy rewriting X-Forwarded-For. Left at 2, the client address the api records becomes chosen by the caller: anyone can send X-Forwarded-For: 198.51.100.7 and have that written into session records, audit rows and the key the anonymous WebAuthn budget counts on.

Do not add session affinity. It is the reflex the moment websockets are involved and it solves nothing here: the hard case is co-locating an operator's browser with an agent's websocket, which arrives from a different network at a different time and whose id is not even known at the browser's handshake. No ingress annotation can express that. It is handled in the application, over Redis. Both ends speak WebSockets only — the api allows no other transport and the console skips negotiation — so there is no polling to make sticky in the first place.

On exactly three nodes

The common case, and where the arithmetic is tightest.

What lands where. Redis and the sentinels both use required anti-affinity, so three of each means one per node — six pods, two per node, no choice left to the scheduler. Postgres at three instances is the same. That is what you want, and it has a consequence: during a node drain, one Redis pod and one sentinel have nowhere to go and stay Pending until the node returns. That is correct — a quorum of two still holds — but kubectl drain will sit there unless the PodDisruptionBudgets are applied to tell it what is safe.

Three api replicas, not two. 04-api.yaml ships two because that is the smallest number that proves the code is not single-instance. On three nodes, three is strictly better: one per node, and losing any node leaves two. Change replicas: 2 to 3 in it and in 05-web.yaml, and size max_connections for it.

Three nodes tolerates losing one. Not two. That is a property of majority quorums, and the answer to wanting more is a fourth and fifth node, not a different configuration.

What a failover costs

Not measured on the current api: no figure here would be one. What holds by design, and what Redis and Postgres do on their own, is this — how long each window lasts on your cluster is yours to measure.

Redis, its primary lost under a running api:

• The api rides it out without restarting. /api/health/ready answers 503 with "redis": "unreachable" while the api cannot reach Redis, so the pod leaves the Service's endpoints and returns when it can. Liveness never touches Redis, so the pod is not killed and restarted into a cold start.
• Messages published during the window are lost. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change that: for a few seconds, live updates — the chat stream, presence, typing — reach nobody, and something happening on one api replica may not reach another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason.

Postgres, its primary lost:

• Every api pod degrades together: readiness answers 503 with "database": "unreachable" until the promoted primary answers at the address DATABASE_URL names, which is what pointing every pod at one endpoint means.
• No restart: liveness deliberately does not touch the database, which is what keeps a failover from becoming a cold start of everything.

A failover during the migration Job aborts it with a lost-connection error, which is the correct outcome. Run it again.

What is still worth knowing

• AGENT_RELEASE_DIR is per-pod, and must be. It holds the compiled agent binaries. An emptyDir filled by an initContainer is right; a shared RWO volume is not — either every pod is pinned to one node, or the second sits in Multi-Attach error. Leaving it unset is supported: everything works except the agent installer, which answers 503 saying no build is published.
• No application volume, and it should stay that way. Avatars, documents, terminal recordings and attached files are all in Postgres; exports are rendered in memory.
• The Ingresses are written for ingress-nginx. Under another controller their nginx.ingress… annotations do nothing, so set that controller's own equivalents: an hour's read timeout on the two websocket paths, a 4 MB body ceiling on /api/, and 13,376 KiB on /api/uploads/.
• Autoscale jarvis-web, not jarvis-api. Every api scale-in kills a pod holding agent websockets, live runs and open shells, and CPU is a poor proxy for a load made of long-lived connections. Raise api replicas deliberately.
• A rollout still cuts what is in flight. An assistant run is interrupted and resumed with a note; a terminal is not resumable and says so; and the console's files are fingerprinted per image, so during the rollout window a tab holding the old console can miss one of the two parts it loads on demand (the Markdown parser, the QR code generator). It then shows the Markdown as plain text and offers the reload.
• Validation is not admission. These manifests satisfy the restricted Pod Security Standard that 00-namespace.yaml enforces. If you edit them, note that a schema validator cannot see that policy: a missing seccompProfile or a container without a securityContext does not fail validation, it produces a Deployment reporting FailedCreate and sitting at zero replicas.

Backups

Everything in the main README applies, with one substitution: the vault key is not in a Docker volume, it is in a Secret — jarvis-secrets if you set VAULT_MASTER_KEY there yourself, jarvis-generated otherwise. The kustomization mints jarvis-generated either way, so on the manual path it holds a key nothing uses: back up the one the api reads.

kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d
kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_KEY_CREATED_AT}' | base64 -d

Back that up somewhere that is not this cluster, and not alongside your database dump. A database backup that travelled with its own key is a backup that decrypts itself.

The second line is when the key was created, bound to that key by its fingerprint — on a Secret minted from api 0.114.1 on. An older one carries a bare date, which the api ignores here, so a restored copy of it changes nothing and the claim asks for the key. If you ever restore the key by hand into jarvis-secrets, restore that value beside it word for word, or leave it out — never compute a new one. It is what tells the install screen whether an empty database is a first install or a lost one, and an old key dated today makes the first administrator free to whoever reaches the address before your dump is loaded.