Sync the self-hosting stack

kubernetes/00-namespace.yaml

@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: jarvis
+  labels:
+    # Both images run unprivileged and drop every capability, so the strictest profile applies
+    # cleanly. Declared here rather than assumed: under `restricted`, a container that tries to run
+    # as root is refused at admission, which is a far better failure than discovering it later.
+    pod-security.kubernetes.io/enforce: restricted
+    pod-security.kubernetes.io/enforce-version: latest

kubernetes/01-secret-job.yaml

@@ -0,0 +1,119 @@
+# Mint the generated secrets ONCE, from inside the cluster.
+#
+# ─── WHAT THIS REPLACES ───────────────────────────────────────────────────────────────────────
+#
+# On Compose, `init-secrets.cjs` runs as a one-shot writing into a volume every container shares,
+# and its guarantee that a key is generated exactly once is `openSync(path, "wx")` — two writers
+# race, the kernel picks one. There is no shared filesystem across pods, so the exclusion moves to
+# the one thing every pod does share: the API server. Creating a Secret that already exists is a
+# 409, decided by one serialized writer. Same guarantee, same strength.
+#
+# This is the alternative to minting by hand, NOT a replacement for understanding what it mints. If
+# your estate already has a secrets manager — SealedSecrets, ExternalSecrets, Vault — use it and
+# skip this file; write `jarvis-generated` yourself with the four keys listed in the Role below.
+#
+# ─── THE ONE WAY TO GET HURT ──────────────────────────────────────────────────────────────────
+#
+# Deleting `jarvis-generated` and re-running this mints a NEW vault key against a database whose
+# credentials are sealed under the old one. Nothing fails at boot. Every stored credential becomes
+# permanently unreadable, and you find out the first time somebody opens one. There is no undo, and
+# the key is not in the database backup by design.
+#
+#   kubectl apply -f 01-secret.example.yaml   # the half you write: DATABASE_URL, OPENAI_API_KEY…
+#   kubectl apply -f 01-secret-job.yaml       # the half that is random bytes
+#   kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m
+#   # Then back the vault key up, out of this cluster:
+#   kubectl get secret jarvis-generated -n jarvis -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d
+#
+# ──────────────────────────────────────────────────────────────────────────────────────────────
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: jarvis-secret-minter
+  namespace: jarvis
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: jarvis-secret-minter
+  namespace: jarvis
+rules:
+  # `create` AND NOTHING ELSE. No `get`, no `list`, no `update`.
+  #
+  # Deliberate, and worth the paragraph: this identity cannot read any Secret in the namespace,
+  # including the one it just wrote. That is why the script does not look before it writes — the
+  # 409 is the check, and it is a stronger one than a read-then-write could be, because a read and
+  # a write are two operations and something else can happen between them.
+  #
+  # `create` cannot be narrowed by resourceName — Kubernetes evaluates authorization before the
+  # object's name is known — so this permits creating any Secret in this namespace. That is the
+  # floor, and it is why the ServiceAccount belongs to this Job and to nothing else.
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["create"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: jarvis-secret-minter
+  namespace: jarvis
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: jarvis-secret-minter
+subjects:
+  - kind: ServiceAccount
+    name: jarvis-secret-minter
+    namespace: jarvis
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: jarvis-mint-secrets
+  namespace: jarvis
+spec:
+  # Re-running is safe (409 → left untouched), so a retry costs nothing. A high limit would only
+  # mean a broken RBAC took longer to become visible.
+  backoffLimit: 2
+  # The Job object is tidied up an hour later. The Secret it created is a separate object and is
+  # not touched by this.
+  ttlSecondsAfterFinished: 3600
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: jarvis-mint-secrets
+    spec:
+      restartPolicy: Never
+      serviceAccountName: jarvis-secret-minter
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 10001
+        # Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces.
+        # Omitted, the Job is created and its pod is refused — which reads as a Job that never runs.
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: mint
+          # The api image, because the script ships in it and a first deployment should pull one
+          # image rather than three. By digest like every other image here — the same one you put
+          # in 03-migrate-job.yaml and 04-api.yaml.
+          image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME
+          command: ["node", "/app/apps/api/mint-k8s-secret.cjs"]
+          env:
+            # Node verifies the API server's certificate against the cluster CA. Without this the
+            # request would fail — which is the correct failure. Do not reach for
+            # NODE_TLS_REJECT_UNAUTHORIZED: a bearer token good enough to write Secrets is a bearer
+            # token worth stealing, and an unverified TLS session is where that happens.
+            - name: NODE_EXTRA_CA_CERTS
+              value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              memory: 256Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]

kubernetes/01-secret.example.yaml

@@ -0,0 +1,71 @@
+# The secrets an operator writes. Minted ONCE, mounted by every pod.
+#
+# ─── READ THIS BEFORE YOU GENERATE ANYTHING ───────────────────────────────────────────────────
+#
+# There are two Secrets, and the split is not filing: `jarvis-generated` holds the values that are
+# random bytes and `jarvis-secrets` — this file — holds the ones that are facts about your estate.
+# One of those can be regenerated safely and the other cannot, so they do not share an object.
+#
+# The three keys below are here as the MANUAL path. If you would rather have them minted for you,
+# apply `01-secret-job.yaml` instead and leave the `VAULT_*`/`JWT_*` lines out of this file
+# entirely. Do not do both: `04-api.yaml` reads this Secret last on purpose, so a value typed here
+# wins over a generated one, and the two quietly disagreeing is the shape of the accident.
+#
+# `apps/api/init-secrets.cjs` is the Compose generator, and it must NEVER become an initContainer
+# here. Its guarantee that a key is minted exactly once is every writer sharing one filesystem.
+# Per-pod, on an `emptyDir`, each replica generates its own — and the consequence is not a crash.
+# Pod A seals a vault credential under a key pod B does not have; pod B reports that credential as
+# corrupt; both pods log nothing at boot, because from each one's point of view everything is fine.
+# By the time anybody notices, there are several keys in circulation and no way to tell which
+# entries belong to which. `apps/api/mint-k8s-secret.cjs` is the same one-shot done correctly, with
+# the API server's 409 standing in for the filesystem's exclusion.
+#
+# Either way, keep the output. The vault key in particular is the thing every stored credential is
+# encrypted under — a database backup that travelled with its own key would be a backup that
+# decrypts itself, which is why this lives apart from the database.
+#
+#   VAULT_MASTER_KEY   openssl rand -base64 32
+#   JWT_ACCESS_SECRET  openssl rand -base64 48
+#   JWT_REFRESH_SECRET openssl rand -base64 48
+#
+# In anything beyond a trial, do not commit this file with values in it. Use SealedSecrets,
+# ExternalSecrets, or `kubectl create secret generic` from a password manager.
+# ──────────────────────────────────────────────────────────────────────────────────────────────
+apiVersion: v1
+kind: Secret
+metadata:
+  name: jarvis-secrets
+  namespace: jarvis
+type: Opaque
+stringData:
+  # Everything in the vault is sealed under this. Lose it and the credentials are unreadable;
+  # leak it and a database dump is enough to read them all.
+  VAULT_MASTER_KEY: "REPLACE_ME"
+  JWT_ACCESS_SECRET: "REPLACE_ME"
+  JWT_REFRESH_SECRET: "REPLACE_ME"
+
+  # WHEN THE VAULT KEY WAS CREATED, as an ISO 8601 timestamp or epoch seconds.
+  #
+  # Set it when you mint the key, and never move it afterwards. On Compose this is inferred from the
+  # key file's modification time, which is a good answer on a volume nobody rewrites. Here it is a
+  # wrong one: the kubelet writes a projected Secret into every pod at start, so the mtime is the
+  # POD's age and every pod reports a key created seconds ago however old it is.
+  #
+  # What that number decides is narrow and serious. An instance with no accounts can have its first
+  # super-admin claimed by whoever reaches the install screen — unless the vault key is old, which
+  # means the database has gone missing from an instance that already existed (a bad restore, a PVC
+  # pointed at the wrong volume) and the claim needs proof of possession instead. Without this line,
+  # that check reads "brand new instance" on exactly the estate it exists to protect.
+  VAULT_KEY_CREATED_AT: "REPLACE_ME"
+
+  # `connection_limit` is not optional here, and it is the reason this URL is in the Secret rather
+  # than assembled from parts. Prisma's default pool is per-process: N pods times that default will
+  # exhaust `max_connections` long before the application is under any real load. Size it as
+  # (max_connections − reserve) ÷ (replicas + maxSurge + jobs).
+  DATABASE_URL: "postgresql://jarvis:REPLACE_ME@postgres.jarvis.svc.cluster.local:5432/jarvis?schema=public&connection_limit=10&pool_timeout=20"
+
+  # Required to create new organizations, users or agents. Without it an instance keeps running
+  # everything it already has and creates nothing new. See the project README.
+  JARVIS_LICENSE_KEY: ""
+
+  OPENAI_API_KEY: "REPLACE_ME"

kubernetes/02-config.yaml

@@ -0,0 +1,82 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: jarvis-api-config
+  namespace: jarvis
+data:
+  NODE_ENV: "production"
+  API_PORT: "4000"
+
+  # One logical keyspace, shared by every pod. Sentinel or a managed single-primary rather than
+  # Cluster: the Socket.IO adapter and the inter-pod bus both publish across what would otherwise
+  # be different slots.
+  #
+  # A SINGLE REDIS IS A SINGLE POINT OF FAILURE FOR EVERY POD AT ONCE. This line is correct for a
+  # managed Redis that fails over behind one address, and correct-but-fragile for one Deployment of
+  # one Redis. For Sentinel, comment this out and set the two below instead — see
+  # `08-redis-sentinel.yaml`, which deploys exactly that.
+  REDIS_URL: "redis://redis.jarvis.svc.cluster.local:6379"
+
+  # Set BOTH or neither. Sentinel needs the sentinels and the name of the primary they watch; a
+  # name is refused rather than defaulted, because ioredis's own default (`mymaster`) against a
+  # primary called anything else is not an error — it is a pod that never becomes ready, silently.
+  # When these are set, REDIS_URL above is ignored.
+  #
+  #REDIS_SENTINELS: "redis-sentinel-0.redis-sentinel.jarvis.svc.cluster.local:26379,redis-sentinel-1.redis-sentinel.jarvis.svc.cluster.local:26379,redis-sentinel-2.redis-sentinel.jarvis.svc.cluster.local:26379"
+  #REDIS_SENTINEL_NAME: "jarvis"
+
+  # ONE, not the two the compose files use, and getting this wrong is a security fault rather than
+  # a cosmetic one.
+  #
+  # It counts the proxies that REWRITE X-Forwarded-For. On compose a request passes the operator's
+  # TLS terminator and then the web container's nginx: two. Here the Ingress routes /api/ straight
+  # at the API Service, so there is one. Set HIGHER than the truth, the API believes a hop that does
+  # not exist — and since X-Forwarded-For is a request header, anyone can then prepend an address of
+  # their choosing and have it written into session records, audit rows, and the key the anonymous
+  # WebAuthn budget counts on.
+  #
+  # It is a STORED setting: this seeds the first boot, and the install screen — which shows the
+  # chain the API actually received — is where it is confirmed.
+  TRUST_PROXY_HOPS: "1"
+
+  # The seed for the address enrolled agents dial back to. Whatever the install screen stores wins
+  # from then on, so this is a starting value rather than the answer.
+  WEB_ORIGIN: "https://jarvis.example.com"
+
+  # Where the agent binaries are, filled per pod by an initContainer. Leaving it unset is
+  # supported: everything works except the agent installer, which answers 503 saying no build is
+  # published.
+  AGENT_RELEASE_DIR: "/srv/agent-releases"
+
+  AGENT_HEARTBEAT_INTERVAL_SEC: "30"
+  # Must stay comfortably below the pod's terminationGracePeriodSeconds (90). Whichever expires
+  # first decides whether a restart leaves a closed transcript or a half-written one.
+  RUN_SHUTDOWN_GRACE_SEC: "25"
+
+  # No demo organization and no seeded administrator: the install screen is the way in. The seed
+  # refuses an instance that already has accounts anyway, but a Deployment has no business
+  # furnishing anything.
+  SEED_ON_START: "false"
+
+  OPENAI_BASE_URL: "https://api.openai.com/v1"
+  OPENAI_MODEL: "gpt-4o"
+  OPENAI_THINKING_LEVEL: "medium"
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: jarvis-web-config
+  namespace: jarvis
+data:
+  # Where the web container's nginx sends /api/, the socket.io handshake and the agent websocket.
+  #
+  # FULLY QUALIFIED, and that is not decoration: nginx's `resolver` does not apply the search list
+  # from /etc/resolv.conf, so a bare `jarvis-api` would not resolve however correct it looks.
+  JARVIS_API_UPSTREAM: "jarvis-api.jarvis.svc.cluster.local:4000"
+
+  # Who resolves it. The compose default is Docker's embedded DNS at 127.0.0.11, which nothing
+  # listens on inside a pod — every /api/ request answered 502 until this became substitutable.
+  #
+  # Replace with your cluster's DNS ClusterIP:
+  #   kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'
+  JARVIS_DNS_RESOLVER: "REPLACE_WITH_YOUR_CLUSTER_DNS"

kubernetes/03-migrate-job.yaml

@@ -0,0 +1,93 @@
+# The schema, brought up to this image's expectations. Once, before any pod of it starts serving.
+#
+# A JOB, and deliberately neither of the two alternatives:
+#
+#   NOT an initContainer. It would run once per pod — which is exactly the bug this replaces, where
+#   every API container ran `prisma db push` at boot and N replicas raced the same DDL while the
+#   previous generation was still selecting the columns being altered.
+#
+#   NOT a leader election inside the API. A Job is the primitive built for run-once work: it has an
+#   observable status, a bounded number of retries, and a failure that stops the rollout instead of
+#   letting it proceed against a schema that was never applied.
+#
+# With Helm this becomes a `pre-install,pre-upgrade` hook; with Argo, a sync-wave. Applied by hand,
+# the ordering is yours to keep — see the README.
+#
+# It is safe to run twice: every pass is idempotent and the whole phase is wrapped in a Postgres
+# advisory lock, so a second copy waits rather than racing.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: jarvis-migrate
+  namespace: jarvis
+spec:
+  # One writer. The lock makes a second one wait rather than corrupt, but there is no reason to
+  # have one waiting.
+  parallelism: 1
+  completions: 1
+  # Two retries, because the failures worth retrying are transient (a database not finished failing
+  # over). A schema that Prisma REFUSES is not transient and must not be retried into submission —
+  # it is refused for a reason, and the reason is printed.
+  backoffLimit: 2
+  # A migration that has not finished in fifteen minutes is stuck, not slow, and the rollout is
+  # blocked behind it.
+  activeDeadlineSeconds: 900
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: jarvis
+        app.kubernetes.io/component: migrate
+    spec:
+      restartPolicy: Never
+      # Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces. At pod
+      # level so it covers anything added beside this container later. Without it the Job is
+      # accepted and its pod is refused, so the rollout waits on a Job that will never complete.
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: migrate
+          # THE SAME IMAGE, by the same digest, as the API that is about to roll. A migration from a
+          # different build is a schema that matches nothing.
+          image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME
+          command: ["/usr/local/bin/migrate.sh"]
+          envFrom:
+            - configMapRef:
+                name: jarvis-api-config
+            # Generated first, hand-written last, for the reason spelled out in 04-api.yaml: the
+            # last occurrence of a name wins, and a generated value must never override a typed one.
+            - secretRef:
+                name: jarvis-generated
+                optional: true
+            - secretRef:
+                name: jarvis-secrets
+          env:
+            # Its own, smaller pool: this is one process doing one thing, and the connections it
+            # takes are connections the running API cannot have while it works.
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: jarvis-secrets
+                  key: DATABASE_URL
+            # Off, and it should stay off between deliberate acts. `prisma db push` used to run with
+            # `--accept-data-loss` at every boot, which meant an upgrade could destroy a column with
+            # nobody having decided to — survivable on one container, not during a rolling update
+            # where the previous generation is still selecting it.
+            #
+            # Without the flag the phase REFUSES, prints which columns it would have dropped, and
+            # fails the Job — so the rollout stops and nothing is serving a half-migrated schema.
+            # Set it to "true" for the ONE deploy where that drop is intended, then remove it.
+            - name: SCHEMA_ACCEPT_DATA_LOSS
+              value: "false"
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              memory: 1Gi
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 1000
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]

kubernetes/04-api.yaml

@@ -0,0 +1,194 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: jarvis-api
+  namespace: jarvis
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: jarvis
+      app.kubernetes.io/component: api
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      # Never fewer pods than are being served now. A run holds its conversation through a unique
+      # index, so a surge pod cannot double-answer anything — the old constraint that made an
+      # overlap dangerous is gone.
+      maxUnavailable: 0
+      maxSurge: 1
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: jarvis
+        app.kubernetes.io/component: api
+    spec:
+      # 90 seconds, and every part of it is accounted for:
+      #   10s  preStop, so this pod leaves the Service's endpoints BEFORE it starts draining
+      #   25s  RUN_SHUTDOWN_GRACE_SEC — in-flight assistant runs stop and close their transcripts
+      #   ~15s agent sockets closed and their rows written, httpServer.close(), Prisma, Redis
+      #   the rest is margin
+      # The default of 30 would SIGKILL the pod in the middle of the run drain, which is the one
+      # thing the drain exists to prevent.
+      terminationGracePeriodSeconds: 90
+      # Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces. At pod
+      # level rather than per container, because the agent-releases initContainer below needs it
+      # too — and a missing seccompProfile is not a warning, it is the pod being refused outright.
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
+      # SPREAD THE REPLICAS, because nothing else will.
+      #
+      # The scheduler has no reason of its own to put two pods of the same Deployment on different
+      # nodes, and quite often has a reason not to. Two API replicas on one node is a deployment
+      # that reads as highly available in `kubectl get pods` and loses every agent websocket, every
+      # live run and every open shell the moment that one node reboots. The PodDisruptionBudget in
+      # 07-disruption.yaml does not help here: it constrains voluntary evictions, and a node dying
+      # is not one.
+      #
+      # `ScheduleAnyway` rather than `DoNotSchedule`: on a single-node cluster the strict form
+      # leaves the second replica Pending forever, which is a worse first experience than an
+      # unspread pair. On a real cluster it spreads. If yours spans zones, add a second constraint
+      # on `topology.kubernetes.io/zone` — a rack is a failure domain too.
+      topologySpreadConstraints:
+        - maxSkew: 1
+          topologyKey: kubernetes.io/hostname
+          whenUnsatisfiable: ScheduleAnyway
+          labelSelector:
+            matchLabels:
+              app.kubernetes.io/name: jarvis
+              app.kubernetes.io/component: api
+      containers:
+        - name: api
+          # BY DIGEST. `:stable` is a moving name, so replicas that restart at different times land
+          # on different builds — and a rollback becomes "hope the tag still points where it did".
+          image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME
+          ports:
+            - name: http
+              containerPort: 4000
+          lifecycle:
+            preStop:
+              exec:
+                # Endpoint removal and SIGTERM race each other, and losing that race is visible:
+                # a prompt that arrives after Nest sets `draining` is persisted and broadcast as
+                # queued, and this pod will never answer it. Ten seconds is comfortably more than
+                # kube-proxy needs to stop sending new connections here.
+                command: ["/bin/sh", "-c", "sleep 10"]
+          env:
+            - name: JARVIS_SKIP_MIGRATIONS
+              # The schema is the migration Job's business, once, before this rolls. N pods each
+              # running `prisma db push` is N writers racing the same DDL while the previous
+              # generation is still selecting the columns being altered.
+              value: "true"
+          envFrom:
+            - configMapRef:
+                name: jarvis-api-config
+            # The half that is random bytes, minted once by `01-secret-job.yaml`. Kept apart from
+            # the half an operator writes because one of them can be regenerated and the other
+            # cannot: re-running the minter is a no-op by design, which is only safe while nothing
+            # anybody typed lives in the same object.
+            #
+            # `optional: true` so a deployment that mints by hand — or through SealedSecrets,
+            # ExternalSecrets, Vault — does not need this object to exist at all.
+            - secretRef:
+                name: jarvis-generated
+                optional: true
+            # LAST, and the order is the safety property rather than a formatting choice. `envFrom`
+            # resolves in sequence and the last occurrence of a name wins, so anything written by
+            # hand here overrides the generated value — never the other way round. Reversed, an
+            # operator who had already minted a vault key and then applied the Job by reflex would
+            # have every stored credential silently become unreadable.
+            - secretRef:
+                name: jarvis-secrets
+          volumeMounts:
+            - name: agent-releases
+              mountPath: /srv/agent-releases
+              readOnly: true
+          # Liveness NEVER touches the database. A Postgres failover with a database check here
+          # restarts every pod at once, turning a thirty-second blip into a cold start of the whole
+          # deployment at the moment the database can least afford a reconnection stampede.
+          livenessProbe:
+            httpGet:
+              path: /api/health/live
+              port: http
+            periodSeconds: 10
+            failureThreshold: 3
+          # Readiness DOES, and it checks Redis too. A pod cut off from Redis keeps answering HTTP
+          # perfectly while its broadcasts reach nobody — the clients queue rather than fail — so it
+          # looks healthier than a pod that is down and is more dangerous.
+          readinessProbe:
+            httpGet:
+              path: /api/health/ready
+              port: http
+            periodSeconds: 5
+            timeoutSeconds: 3
+            failureThreshold: 3
+          # Boot does a run-recovery sweep and reads settings before it listens. The startup probe
+          # is what stops liveness killing it halfway through on a slow database.
+          startupProbe:
+            httpGet:
+              path: /api/health/live
+              port: http
+            periodSeconds: 5
+            failureThreshold: 30
+          resources:
+            requests:
+              cpu: 500m
+              # Not generous — honest. A terminal recording holds up to 8 MiB of Buffers in heap
+              # for the life of the session, and argon2 takes 64 MiB per share verification.
+              memory: 1Gi
+            limits:
+              memory: 2Gi
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 1000
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+      initContainers:
+        # The compiled agent binaries, into this pod's own emptyDir.
+        #
+        # PER POD ON PURPOSE. The content is immutable and small, so a copy per pod costs nothing;
+        # a shared RWO PVC would either pin every pod to one node or leave the second in
+        # Multi-Attach error. Leaving this out entirely is supported: enrollment still works and the
+        # installer answers 503 saying no build is published.
+        - name: agent-releases
+          image: git.luxit.be/luxit/jarvis-agent-dist@sha256:REPLACE_ME
+          command: ["/bin/sh", "-c", "cp -a /dist/. /srv/agent-releases/"]
+          # AN initContainer IS A CONTAINER, and the `restricted` Pod Security Standard judges it
+          # exactly like the one below. This block was missing, and the result was not a warning
+          # about the initContainer — it was the WHOLE POD refused, so the Deployment sat at zero
+          # replicas reporting FailedCreate while every other line in this file was correct.
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 1000
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          volumeMounts:
+            - name: agent-releases
+              mountPath: /srv/agent-releases
+      volumes:
+        - name: agent-releases
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: jarvis-api
+  namespace: jarvis
+spec:
+  type: ClusterIP
+  # STATED, not left to the default, because the reflex is to reach for it the moment websockets
+  # are involved. It would not help: the hard case is co-locating an operator's browser with an
+  # agent's websocket, which arrives separately and whose id is unknown at the browser's handshake.
+  # That is handled in the application, over Redis. See the README.
+  sessionAffinity: None
+  selector:
+    app.kubernetes.io/name: jarvis
+    app.kubernetes.io/component: api
+  ports:
+    - name: http
+      port: 4000
+      targetPort: http

kubernetes/05-web.yaml

@@ -0,0 +1,152 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: jarvis-web
+  namespace: jarvis
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: jarvis
+      app.kubernetes.io/component: web
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 0
+      maxSurge: 1
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: jarvis
+        app.kubernetes.io/component: web
+    spec:
+      # Nothing here holds a long-lived connection of its own — the websockets go to the API — so
+      # this is only about letting in-flight responses finish.
+      terminationGracePeriodSeconds: 30
+      # Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces.
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
+      # Same reasoning as 04-api.yaml: the scheduler will happily put both replicas on one node,
+      # and a pair that shares a node is a pair that shares a failure. Cheaper to satisfy here —
+      # nothing about serving static files pins a pod anywhere.
+      topologySpreadConstraints:
+        - maxSkew: 1
+          topologyKey: kubernetes.io/hostname
+          whenUnsatisfiable: ScheduleAnyway
+          labelSelector:
+            matchLabels:
+              app.kubernetes.io/name: jarvis
+              app.kubernetes.io/component: web
+      containers:
+        - name: web
+          # By digest, like the API. It matters more here than it looks: the bundle's JavaScript
+          # chunks are content-hashed PER BUILD and answer 404 rather than falling through to
+          # index.html, so during a rollout a browser holding the previous shell can fail a lazy
+          # import. Two pods on two different builds widen that window; two pods on one digest do
+          # not have it at all.
+          image: git.luxit.be/luxit/jarvis-web@sha256:REPLACE_ME
+          ports:
+            - name: http
+              containerPort: 8080
+          lifecycle:
+            preStop:
+              exec:
+                command: ["/bin/sh", "-c", "sleep 5"]
+          envFrom:
+            - configMapRef:
+                name: jarvis-web-config
+          # Is this nginx answering? Nothing more. Restarting the web container because the API is
+          # down would take out the one thing still able to show an error page.
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: http
+            periodSeconds: 10
+            failureThreshold: 3
+          # Can THIS pod reach the API? It proxies /api/health/live, not /ready, on purpose: the
+          # question is whether this pod's route works — a broken DNS name, a NetworkPolicy — and
+          # not whether the API's dependencies are healthy. Pointed at /ready, one Postgres blip
+          # would fail every web pod at once and take the console offline for a fault that has
+          # nothing to do with serving a bundle.
+          readinessProbe:
+            httpGet:
+              path: /readyz
+              port: http
+            periodSeconds: 5
+            timeoutSeconds: 3
+            failureThreshold: 3
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              memory: 256Mi
+          securityContext:
+            # uid 101 is the `nginx` user the image already ships and the Dockerfile switches to.
+            runAsNonRoot: true
+            runAsUser: 101
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+            readOnlyRootFilesystem: true
+          volumeMounts:
+            # With a read-only root filesystem, the three paths this container writes at runtime
+            # have to be given to it. All three are ephemeral by nature — the substituted config is
+            # regenerated at every start from the template and the ConfigMap.
+            - name: nginx-conf
+              mountPath: /etc/nginx/conf.d
+            - name: nginx-cache
+              mountPath: /var/cache/nginx
+            - name: tmp
+              # The pid file lives here rather than in /var/run, which the nginx user does not own.
+              mountPath: /tmp
+      volumes:
+        - name: nginx-conf
+          emptyDir: {}
+        - name: nginx-cache
+          emptyDir: {}
+        - name: tmp
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: jarvis-web
+  namespace: jarvis
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: jarvis
+    app.kubernetes.io/component: web
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http
+---
+# Web scales on CPU; the API deliberately does not.
+#
+# Every API scale-in kills a pod holding agent websockets, live assistant runs and open shells —
+# the fleet reconnects, runs are interrupted and resumed, terminals die — and CPU is a poor proxy
+# for a load made almost entirely of long-lived connections. Raise `jarvis-api` deliberately
+# instead. If it ever does get an HPA, give it a scale-down stabilization window of ten minutes or
+# more.
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: jarvis-web
+  namespace: jarvis
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: jarvis-web
+  minReplicas: 2
+  maxReplicas: 5
+  metrics:
+    - type: Resource
+      resource:
+        name: cpu
+        target:
+          type: Utilization
+          averageUtilization: 70

kubernetes/06-ingress.yaml

@@ -0,0 +1,81 @@
+# One hostname, two backends.
+#
+# The API is routed STRAIGHT AT ITS SERVICE rather than through the web container's nginx, which is
+# what that nginx does on Compose. Two reasons: it removes a hop from every API request and every
+# websocket frame, and it makes the web pod a pure static server again — no dependency on resolving
+# the API, nothing to keep in sync between an ingress and a config file.
+#
+# The consequence has to be applied, not just noted: there is now exactly ONE proxy rewriting
+# X-Forwarded-For, so `TRUST_PROXY_HOPS` is 1. It is 2 in the compose files, and leaving it there
+# would make `req.ip` a value the caller chooses. See 02-config.yaml.
+#
+# Set `externalTrafficPolicy: Local` on the ingress controller's own Service as well, or the node's
+# SNAT will make every request in the world appear to come from a handful of addresses — which
+# defeats every per-address limit in the application at once.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: jarvis
+  namespace: jarvis
+  annotations:
+    # An agent is idle between operations and the server pings every 30s, so this only has to
+    # outlast a quiet period comfortably. Too short and a healthy fleet reconnects all day.
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
+    # Matches BODY_LIMIT in apps/api/src/main.ts. A document may hold 400,000 characters, which is
+    # over a megabyte once accented text is UTF-8 encoded, and a diagram export posts back the SVG
+    # the browser rendered. Below this they are rejected as a bare 413 with no message.
+    nginx.ingress.kubernetes.io/proxy-body-size: "4m"
+    #
+    # NO affinity annotation, and none should be added. The reflex is understandable — there are
+    # websockets here — but stickiness cannot solve the problem it would be hired for: the hard case
+    # is co-locating an operator's browser with an AGENT's websocket, which arrives from a different
+    # network at a different time and whose id is not even known at the browser's handshake. No
+    # ingress annotation can express that; it is handled in the application, over Redis.
+    #
+    # And there is no polling to make sticky: both the client and the server pin
+    # transports: ["websocket"], so one logical socket is one TCP connection.
+spec:
+  ingressClassName: nginx
+  tls:
+    - hosts: ["jarvis.example.com"]
+      secretName: jarvis-tls
+  rules:
+    - host: jarvis.example.com
+      http:
+        paths:
+          # Longest prefix first for readers; the controller sorts by specificity itself.
+          #
+          # The agent websocket has its own rule even though it sits under /api/, mirroring the
+          # split in apps/web/nginx.conf: it is the one path where an idle connection legitimately
+          # stays open for an hour between frames.
+          - path: /api/agents/ws
+            pathType: Prefix
+            backend:
+              service:
+                name: jarvis-api
+                port:
+                  name: http
+          - path: /socket.io/
+            pathType: Prefix
+            backend:
+              service:
+                name: jarvis-api
+                port:
+                  name: http
+          - path: /api/
+            pathType: Prefix
+            backend:
+              service:
+                name: jarvis-api
+                port:
+                  name: http
+          # Everything else is the SPA, whose nginx answers index.html for any path it does not
+          # have a file for.
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: jarvis-web
+                port:
+                  name: http

kubernetes/07-disruption.yaml

@@ -0,0 +1,28 @@
+# What a node drain may take away.
+#
+# `minAvailable: 1` on both, so a cluster upgrade cannot evict the last pod of either. With the
+# API's 90-second grace period a drain will take a minute or two per pod — that is the run drain
+# doing its job, not a stall.
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: jarvis-api
+  namespace: jarvis
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: jarvis
+      app.kubernetes.io/component: api
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: jarvis-web
+  namespace: jarvis
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: jarvis
+      app.kubernetes.io/component: web

kubernetes/08-redis-sentinel.yaml

@@ -0,0 +1,341 @@
+# A Redis that survives losing a node.
+#
+# ─── WHAT THIS IS, AND WHAT IT IS NOT ─────────────────────────────────────────────────────────
+#
+# OPTIONAL. Apply it only if you are running your own Redis. A managed Redis that publishes one
+# address and fails over behind it needs none of this — point `REDIS_URL` at it and skip the file.
+#
+# It is written to be correct and it is small enough to read in full, but a datastore you operate
+# yourself is a datastore you maintain: version upgrades, capacity, the failover drill. If you want
+# somebody else to hold that, use a managed Redis or an operator. What this directory owes you is
+# the shape the application needs, and this is it.
+#
+# The application half is proven rather than assumed: an ioredis client built exactly the way
+# `apps/api/src/redis/redis-connection.ts` builds one was run against a live three-sentinel
+# topology while its primary was killed. It followed the promotion to a different server, and 99
+# writes issued across the window were rejected zero times. What it does NOT do is preserve pub/sub
+# during the switch — see "What a failover actually costs" at the bottom.
+#
+#   kubectl apply -f 08-redis-sentinel.yaml
+#   # then in 02-config.yaml: comment out REDIS_URL, uncomment the two REDIS_SENTINEL* lines
+#
+# ──────────────────────────────────────────────────────────────────────────────────────────────
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: redis-scripts
+  namespace: jarvis
+data:
+  # WHO IS THE PRIMARY — asked, never assumed.
+  #
+  # The naive version of this file pins `replicaof redis-0` into every replica. It works exactly
+  # until the first failover, after which redis-0 comes back, is told it is the primary, and the
+  # cluster has two — a split brain that presents as writes vanishing rather than as an error.
+  #
+  # So a starting node asks the sentinels who the primary is now, and only falls back to redis-0
+  # when no sentinel answers, which is true on a first apply and at no other time.
+  discover.sh: |
+    #!/bin/sh
+    # Prints the current primary's hostname. Empty output is impossible: the fallback is redis-0.
+    for i in 0 1 2; do
+      peer="redis-sentinel-${i}.redis-sentinel.${NAMESPACE}.svc.cluster.local"
+      found=$(redis-cli -h "$peer" -p 26379 sentinel get-master-addr-by-name jarvis 2>/dev/null | head -1)
+      if [ -n "$found" ]; then
+        echo "$found"
+        exit 0
+      fi
+    done
+    echo "redis-0.redis.${NAMESPACE}.svc.cluster.local"
+
+  redis-start.sh: |
+    #!/bin/sh
+    set -eu
+    master=$(sh /scripts/discover.sh)
+    me="${HOSTNAME}.redis.${NAMESPACE}.svc.cluster.local"
+    echo "[redis] I am ${me}; the primary is ${master}"
+    if [ "$master" = "$me" ]; then
+      exec redis-server /etc/redis/redis.conf
+    fi
+    exec redis-server /etc/redis/redis.conf --replicaof "$master" 6379
+
+  sentinel-start.sh: |
+    #!/bin/sh
+    set -eu
+    master=$(sh /scripts/discover.sh)
+    echo "[sentinel] monitoring ${master}"
+    # Rebuilt every start rather than persisted. A sentinel REWRITES its own config file as it
+    # learns, so the file cannot be a read-only ConfigMap mount; and its learned state is
+    # recoverable from its peers in seconds, so persisting it buys nothing and can resurrect a
+    # stale view of a topology that has since moved.
+    cp /etc/sentinel-template/sentinel.conf /data/sentinel.conf
+    {
+      echo "sentinel monitor jarvis ${master} 6379 2"
+      echo "sentinel down-after-milliseconds jarvis 5000"
+      echo "sentinel failover-timeout jarvis 60000"
+      echo "sentinel parallel-syncs jarvis 1"
+    } >> /data/sentinel.conf
+    exec redis-sentinel /data/sentinel.conf
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: redis-config
+  namespace: jarvis
+data:
+  redis.conf: |
+    port 6379
+    # NO PERSISTENCE, and it is a decision rather than an omission. Everything Jarvis keeps in Redis
+    # is ephemeral by design and rebuilds itself: agent ownership keys carry a TTL the owning pod
+    # refreshes, cluster leases expire, and the realtime adapter's state is the live connections
+    # themselves. The one thing that must NOT be lost — which pod owns a run — is a row in Postgres
+    # precisely so that losing Redis cannot lose it.
+    save ""
+    appendonly no
+    # Replicas answer reads while they catch up rather than erroring. Nothing here reads from a
+    # replica today, but a client that briefly lands on one during a promotion should see slightly
+    # old data rather than a failure.
+    replica-serve-stale-data yes
+    # A replica must never accept a write, however it was reached.
+    replica-read-only yes
+
+  sentinel.conf: |
+    port 26379
+    # HOSTNAMES, NOT POD IPs, and this is the line most likely to be dropped as noise.
+    #
+    # By default a sentinel hands clients the primary's IP. A pod's IP does not survive its
+    # restart, so after a failover-then-restart the sentinels answer with an address nothing is
+    # listening on — and a client pointed at it does not fail, it retries forever against a dead
+    # address while the sentinels insist that is where the primary lives. Observed, not theorised.
+    sentinel resolve-hostnames yes
+    sentinel announce-hostnames yes
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis
+  namespace: jarvis
+spec:
+  # Headless: the point is the per-pod DNS names (redis-0.redis…), which are what the sentinels
+  # hand out and therefore what the API ends up connecting to.
+  clusterIP: None
+  publishNotReadyAddresses: true
+  selector:
+    app.kubernetes.io/name: redis
+  ports:
+    - name: redis
+      port: 6379
+      targetPort: 6379
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis-sentinel
+  namespace: jarvis
+spec:
+  clusterIP: None
+  publishNotReadyAddresses: true
+  selector:
+    app.kubernetes.io/name: redis-sentinel
+  ports:
+    - name: sentinel
+      port: 26379
+      targetPort: 26379
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: redis
+  namespace: jarvis
+spec:
+  serviceName: redis
+  replicas: 3
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: redis
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: redis
+    spec:
+      terminationGracePeriodSeconds: 30
+      affinity:
+        podAntiAffinity:
+          # REQUIRED, not preferred. Three replicas on one node is a topology that looks highly
+          # available in `kubectl get pods` and is not one.
+          requiredDuringSchedulingIgnoredDuringExecution:
+            - labelSelector:
+                matchLabels:
+                  app.kubernetes.io/name: redis
+              topologyKey: kubernetes.io/hostname
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 999
+        fsGroup: 999
+        # REQUIRED by the `restricted` Pod Security Standard, which 00-namespace.yaml enforces.
+        # Without it the pods are not warned about, they are REFUSED — the StatefulSet is created,
+        # reports FailedCreate, and stays at zero replicas. Found by applying it, not by
+        # kubeconform: schema validation cannot see an admission policy.
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: redis
+          image: redis:7-alpine
+          command: ["sh", "/scripts/redis-start.sh"]
+          env:
+            - name: NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          ports:
+            - name: redis
+              containerPort: 6379
+          readinessProbe:
+            exec:
+              command: ["redis-cli", "ping"]
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          livenessProbe:
+            exec:
+              command: ["redis-cli", "ping"]
+            initialDelaySeconds: 15
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 512Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          volumeMounts:
+            - name: scripts
+              mountPath: /scripts
+            - name: config
+              mountPath: /etc/redis
+            - name: data
+              mountPath: /data
+      volumes:
+        - name: scripts
+          configMap:
+            name: redis-scripts
+        - name: config
+          configMap:
+            name: redis-config
+        - name: data
+          # No PVC on purpose — see `save ""` above. A volumeClaimTemplate here would persist a
+          # dataset that is meant to be rebuildable, and give a restarted node a stale one.
+          emptyDir: {}
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: redis-sentinel
+  namespace: jarvis
+spec:
+  serviceName: redis-sentinel
+  replicas: 3
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: redis-sentinel
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: redis-sentinel
+    spec:
+      terminationGracePeriodSeconds: 30
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            - labelSelector:
+                matchLabels:
+                  app.kubernetes.io/name: redis-sentinel
+              topologyKey: kubernetes.io/hostname
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 999
+        fsGroup: 999
+        # REQUIRED by the `restricted` Pod Security Standard, which 00-namespace.yaml enforces.
+        # Without it the pods are not warned about, they are REFUSED — the StatefulSet is created,
+        # reports FailedCreate, and stays at zero replicas. Found by applying it, not by
+        # kubeconform: schema validation cannot see an admission policy.
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: sentinel
+          image: redis:7-alpine
+          command: ["sh", "/scripts/sentinel-start.sh"]
+          env:
+            - name: NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          ports:
+            - name: sentinel
+              containerPort: 26379
+          readinessProbe:
+            exec:
+              command: ["redis-cli", "-p", "26379", "ping"]
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              memory: 128Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            # The root filesystem is read-only and the sentinel still works, because the one thing
+            # it must write — its own config, which it rewrites as it learns the topology — lives on
+            # the writable emptyDir at /data. That is why the template is COPIED there at start
+            # rather than run from the ConfigMap mount, which is read-only whatever this says.
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          volumeMounts:
+            - name: scripts
+              mountPath: /scripts
+            - name: sentinel-template
+              mountPath: /etc/sentinel-template
+            - name: data
+              mountPath: /data
+      volumes:
+        - name: scripts
+          configMap:
+            name: redis-scripts
+        - name: sentinel-template
+          configMap:
+            name: redis-config
+            items:
+              - key: sentinel.conf
+                path: sentinel.conf
+        - name: data
+          emptyDir: {}
+---
+# Two of three sentinels must be reachable for the quorum to hold. Without this, a node drain can
+# take two at once — and a Redis with no quorum does not fail over, which is the entire point of
+# having three.
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: redis-sentinel
+  namespace: jarvis
+spec:
+  minAvailable: 2
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: redis-sentinel
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: redis
+  namespace: jarvis
+spec:
+  minAvailable: 2
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: redis

kubernetes/README.md

@@ -0,0 +1,206 @@
+# Jarvis on Kubernetes
+
+Plain manifests, no Helm, no operator. They are written to be read as much as applied: every value
+that is not a Kubernetes default is there because something in Jarvis behaves a particular way, and
+the comment beside it says which. Read them before you apply them — that is the point of shipping
+them this way.
+
+**You probably do not need this.** [Docker Compose](../README.md#install) runs one of everything,
+takes one command, and is what most instances run. Come here when you want Jarvis to keep serving
+while a node reboots, and when you already operate a cluster — this directory assumes you have
+opinions about ingress, storage and a database, because it deliberately does not have them for you.
+
+> **What has been proven, and what has not.** This stack has been brought up on a real cluster —
+> four nodes, three Postgres instances, three Redis with three sentinels, two api pods — and kept
+> serving while its Postgres primary and its Redis primary were killed underneath it. What has NOT
+> happened is a long-running estate serving production traffic this way. If you are early, watch it
+> rather than assume it, and read the three warnings below before you apply anything, because each
+> of them fails silently.
+
+## What runs more than once, and what cannot
+
+"Highly available" is not a property a deployment has as a whole, so here it is component by
+component.
+
+| component | replicas | why |
+| --- | --- | --- |
+| `jarvis-web` | any number | stateless nginx over static files |
+| `jarvis-api` | any number | the coordination lives in Postgres and Redis, not in the process |
+| Redis | one **logical** primary, three processes | Sentinel fails over; it does not spread writes |
+| Postgres | one **primary**, replicas behind it | and Jarvis depends on there being exactly one |
+| `jarvis-migrate` | one at a time | enforced by a Postgres advisory lock, not by you |
+| `jarvis-mint-secrets` | once, ever | enforced by the API server |
+
+The single-writer database is not a limitation Jarvis has failed to overcome — it is what makes the
+multi-pod api safe. A partial unique index is what stops two pods running the same assistant loop,
+and an index can only do that because one server decides. Do not point pods at read replicas.
+
+## What you need before you apply anything
+
+- **x86-64 nodes.** The api and web images are published for `linux/amd64` only, so an arm64 pool
+  leaves the pods in `ImagePullBackOff` with no matching manifest. The machines you *administer*
+  have no such limit — the agent ships arm64 builds.
+- **A cluster you can schedule on.** `kubectl describe node | grep Taints` first. A three-node
+  cluster is often three control-plane nodes, and their `NoSchedule` taint means nothing here
+  schedules at all. These manifests carry no toleration on purpose: adding one is a decision about
+  your cluster.
+- **Postgres 16, highly available, outside these manifests.** Use an operator — CloudNativePG or
+  Zalando's — rather than a hand-written StatefulSet. A database you hand-rolled is a database you
+  hand-restore. `max_connections` must cover
+  `(replicas + maxSurge + jobs) × connection_limit + reserve`: with 3 api pods at 10 connections
+  each, a migration Job at 5, and headroom, about **150**. The stock Postgres image ships 100, and
+  exhausting it presents as an api that starts and then cannot serve.
+- **Redis.** [`08-redis-sentinel.yaml`](08-redis-sentinel.yaml) deploys one, or point `REDIS_URL` at
+  a managed Redis that fails over behind a single address and skip that file. Persistence is not
+  required: everything Jarvis keeps in Redis is ephemeral by design, which is exactly why the run
+  lock is in Postgres instead.
+- **An Ingress controller and a TLS terminator.** Jarvis speaks plain HTTP.
+
+## Order
+
+```sh
+kubectl apply -f 00-namespace.yaml
+
+# The secrets, ONCE. Never regenerate them on an instance that has stored anything.
+kubectl apply -f 01-secret.example.yaml   # after editing — the half you write
+kubectl apply -f 01-secret-job.yaml       # the half that is random bytes (optional; see the file)
+kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m
+
+kubectl apply -f 02-config.yaml
+kubectl apply -f 08-redis-sentinel.yaml   # only if you are running your own Redis
+
+# The schema, to completion, before any api pod starts.
+kubectl apply -f 03-migrate-job.yaml
+kubectl wait --for=condition=complete job/jarvis-migrate -n jarvis --timeout=15m
+
+kubectl apply -f 04-api.yaml
+kubectl apply -f 05-web.yaml
+kubectl apply -f 06-ingress.yaml
+kubectl apply -f 07-disruption.yaml
+```
+
+Every image says `REPLACE_ME`: pin them by digest, not by tag. `:stable` is a moving name, so
+replicas that restart at different times land on different builds and a rollback becomes "hope the
+tag still points where it did".
+
+```sh
+docker buildx imagetools inspect git.luxit.be/luxit/jarvis-api:stable
+```
+
+On every upgrade the migration Job runs again before the Deployments roll. It is idempotent and
+takes a Postgres advisory lock, so running it twice is slow rather than harmful — but it must
+finish before new pods start, which is what the `kubectl wait` is for. With Helm or Argo this
+becomes a `pre-upgrade` hook or a sync-wave; as plain YAML it is an ordering you keep yourself.
+
+**Then open the address and install it.** Everything in [the main README](../README.md#install)
+about the install screen applies unchanged — including that **whoever reaches an uninstalled
+instance first owns it**, with no deadline. Point a public hostname at this after you have
+installed, not before.
+
+## The three things most likely to bite
+
+**Never mint secrets per pod.** On Compose a one-shot generates the vault key and the JWT secrets
+into a volume every container shares, and its guarantee that it does so exactly once is that the
+filesystem refuses a second create. As a per-pod initContainer on an `emptyDir`, each replica mints
+its own — and the consequence is not a crash. Pod A seals a credential under a key pod B does not
+have; pod B reports it as corrupt; both log a clean boot. By the time anybody opens a vault entry
+there are three keys in circulation.
+
+[`01-secret-job.yaml`](01-secret-job.yaml) is that one-shot done correctly here: it creates the
+Secret once and lets the API server's refusal to create it twice be the exclusion. Its identity can
+`create` a Secret and cannot `get` one, so it cannot read back what it or anyone else wrote. Mint by
+hand instead if you prefer — the one thing that is not an option is per-pod.
+
+Either way, **back the vault key up somewhere that is not this cluster's etcd.** Deleting the Secret
+and re-running the Job mints a new key against data sealed with the old one, silently, with no way
+back.
+
+**`TRUST_PROXY_HOPS` is one hop here, not two.** The Compose files say 2 because a request passes
+your TLS terminator and then the web container's nginx. Here the Ingress routes `/api/` straight at
+the api Service, so there is exactly one proxy rewriting `X-Forwarded-For`. Left at 2, `req.ip`
+becomes **chosen by the caller**: anyone can send `X-Forwarded-For: 198.51.100.7` and have that
+written into session records, audit rows and the key the anonymous WebAuthn budget counts on.
+
+**Do not add session affinity.** It is the reflex the moment websockets are involved and it solves
+nothing here: the hard case is co-locating an operator's browser with an *agent's* websocket, which
+arrives from a different network at a different time and whose id is not even known at the browser's
+handshake. No ingress annotation can express that. It is handled in the application, over Redis.
+Both ends pin `transports: ["websocket"]`, so there is no polling to make sticky in the first place.
+
+## On exactly three nodes
+
+The common case, and where the arithmetic is tightest.
+
+**What lands where.** Redis and the sentinels both use *required* anti-affinity, so three of each
+means one per node — six pods, two per node, no choice left to the scheduler. Postgres at three
+instances is the same. That is what you want, and it has a consequence: **during a node drain, one
+Redis pod and one sentinel have nowhere to go and stay `Pending` until the node returns.** That is
+correct — a quorum of two still holds — but `kubectl drain` will sit there unless the
+PodDisruptionBudgets are applied to tell it what is safe.
+
+**Three api replicas, not two.** [`04-api.yaml`](04-api.yaml) ships two because that is the smallest
+number that proves the code is not single-instance. On three nodes, three is strictly better: one
+per node, and losing any node leaves two. Change `replicas: 2` to `3` in it and in
+[`05-web.yaml`](05-web.yaml), and size `max_connections` for it.
+
+**Three nodes tolerates losing one.** Not two. That is a property of majority quorums, and the
+answer to wanting more is a fourth and fifth node, not a different configuration.
+
+## What a failover actually costs
+
+Measured, not estimated.
+
+**Redis**, with its primary killed under a running api:
+
+- **The api rides it out without restarting.** Readiness answered 503 for about fifteen seconds and
+  then 200 again, on the same process — the pod leaves the Service's endpoints while it cannot reach
+  Redis, and returns when it can. Liveness never touches Redis, which is why the pod is not killed
+  and restarted into a cold start.
+- **Commands survive.** 99 writes issued across the window, 0 rejected, the last readable from the
+  promoted primary afterwards. Commands queue rather than fail, so a failover is a pause.
+- **Published messages do not.** Around 45% of the messages published during the window reached
+  nobody. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change
+  that. In practice: a few seconds in which something happening on one api replica may not reach
+  another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason.
+
+**Postgres**, primary deleted with `--force --grace-period=0`:
+
+- **Three seconds.** Both api pods answered `503 {"database":"unreachable"}` and were back to 200 on
+  the fourth poll. Both degraded together, which is what pointing every pod at one endpoint means.
+- **Zero container restarts**, across two failovers, on pods up throughout — liveness deliberately
+  not touching the database is what keeps three seconds from becoming a cold start of everything.
+
+A failover *during* the migration Job aborts it with a lost-connection error, which is the correct
+outcome. Run it again.
+
+## What is still worth knowing
+
+- **`AGENT_RELEASE_DIR` is per-pod, and must be.** It holds the compiled agent binaries. An
+  `emptyDir` filled by an initContainer is right; a shared RWO volume is not — either every pod is
+  pinned to one node, or the second sits in `Multi-Attach error`. Leaving it unset is supported:
+  everything works except the agent installer, which answers 503 saying no build is published.
+- **No application volume, and it should stay that way.** Avatars, documents and terminal recordings
+  are all in Postgres; exports are rendered in memory.
+- **Autoscale `jarvis-web`, not `jarvis-api`.** Every api scale-in kills a pod holding agent
+  websockets, live runs and open shells, and CPU is a poor proxy for a load made of long-lived
+  connections. Raise api replicas deliberately.
+- **A rollout still cuts what is in flight.** An assistant run is interrupted and resumed with a
+  note; a terminal is not resumable and says so; and the web bundle's content-hashed chunks are
+  per-image, so a browser holding the old shell can 404 on a lazy import during the rollout window.
+- **Validation is not admission.** These manifests satisfy the `restricted` Pod Security Standard
+  that [`00-namespace.yaml`](00-namespace.yaml) enforces. If you edit them, note that a schema
+  validator cannot see that policy: a missing `seccompProfile` or a container without a
+  `securityContext` does not fail validation, it produces a Deployment reporting `FailedCreate` and
+  sitting at zero replicas.
+
+## Backups
+
+Everything in [the main README](../README.md#backups-and-restoring-one) applies, with one
+substitution: the vault key is not in a Docker volume, it is in the `jarvis-generated` Secret.
+
+```sh
+kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d
+```
+
+Back that up somewhere that is not this cluster, and not alongside your database dump. A database
+backup that travelled with its own key is a backup that decrypts itself.