From 02ab18aff1c776e2fdb95b0639b43ffd3558835c Mon Sep 17 00:00:00 2001 From: Luxit Date: Thu, 27 Aug 2026 17:52:23 +0200 Subject: [PATCH] Sync the self-hosting stack --- README.md | 30 +++ kubernetes/00-namespace.yaml | 10 + kubernetes/01-secret-job.yaml | 119 +++++++++++ kubernetes/01-secret.example.yaml | 71 +++++++ kubernetes/02-config.yaml | 82 +++++++ kubernetes/03-migrate-job.yaml | 93 ++++++++ kubernetes/04-api.yaml | 194 +++++++++++++++++ kubernetes/05-web.yaml | 152 +++++++++++++ kubernetes/06-ingress.yaml | 81 +++++++ kubernetes/07-disruption.yaml | 28 +++ kubernetes/08-redis-sentinel.yaml | 341 ++++++++++++++++++++++++++++++ kubernetes/README.md | 206 ++++++++++++++++++ 12 files changed, 1407 insertions(+) create mode 100644 kubernetes/00-namespace.yaml create mode 100644 kubernetes/01-secret-job.yaml create mode 100644 kubernetes/01-secret.example.yaml create mode 100644 kubernetes/02-config.yaml create mode 100644 kubernetes/03-migrate-job.yaml create mode 100644 kubernetes/04-api.yaml create mode 100644 kubernetes/05-web.yaml create mode 100644 kubernetes/06-ingress.yaml create mode 100644 kubernetes/07-disruption.yaml create mode 100644 kubernetes/08-redis-sentinel.yaml create mode 100644 kubernetes/README.md diff --git a/README.md b/README.md index 2a0a064..f50a416 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,26 @@ before repeating anything that writes. This is why the api asks for a stop grace your instance, signs each session with a key whose private half never leaves it, and reports its inventory on every heartbeat. +## Two ways to run it + +**Docker Compose — one of everything.** One api container, one web container, one Postgres, one +Redis, on one host. Two commands and about a minute. This is what most instances run and what the +rest of this README describes; if you are not sure which you want, it is this one. A reboot of that +host is downtime, and for the overwhelming majority of deployments that is a fine trade. + +**Kubernetes — several of everything that can be several.** Several api pods and several web pods +behind one address, a Postgres with a primary and replicas, a Redis that fails over. A node can +reboot and the instance keeps serving. It costs you a cluster to operate and a database to run +properly, and it asks you to make decisions Compose makes for you. + +Everything below is the Compose path. For the other one, the manifests and their own guide are in +[`kubernetes/`](kubernetes/README.md) — start there rather than here, then come back for the +install screen, the reverse proxy requirements and the backup section, which are the same either +way. + +Moving between them later is a database dump and the vault key; nothing about your data is specific +to how the containers are scheduled. + ## What you need - Docker with Compose v2, on **x86-64 Linux**. The api and web images are published for @@ -276,6 +296,11 @@ header a caller can forge. Change it later under **Settings → Platform**. docker compose pull && docker compose up -d ``` +> **On Kubernetes** the shape is different, because the schema is a Job rather than something the +> api does on its way up: the migration runs to completion first, then the Deployments roll. See +> [`kubernetes/README.md`](kubernetes/README.md#order). Everything below about channels applies +> unchanged, except that you pin images by digest rather than tracking a channel tag. + That is the whole upgrade: the api, the web, and — with the agent overlay on — the agent release all track the **`stable`** channel by default. Postgres and Redis are not on a Jarvis channel; they follow their own upstream tags, `postgres:16-alpine` and `redis:7-alpine`. Schema changes apply @@ -557,6 +582,11 @@ socket fan-out, all of which rebuild themselves; `agent_releases` is refilled by They are separate volumes precisely so that they can be, and must be, backed up separately. A dump that travelled with the key that opens it is a dump that opens itself. +> **On Kubernetes** the same two things matter and only their location changes: your Postgres +> operator owns the dump, and the vault key is in the `jarvis-generated` Secret rather than a volume. +> See [`kubernetes/README.md`](kubernetes/README.md#backups). The restore procedure below is +> otherwise identical — it is a database and a key, wherever they are kept. + ```sh docker compose exec -T postgres pg_dump -U jarvis jarvis | gzip > jarvis.sql.gz ``` diff --git a/kubernetes/00-namespace.yaml b/kubernetes/00-namespace.yaml new file mode 100644 index 0000000..6b98f2a --- /dev/null +++ b/kubernetes/00-namespace.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: jarvis + labels: + # Both images run unprivileged and drop every capability, so the strictest profile applies + # cleanly. Declared here rather than assumed: under `restricted`, a container that tries to run + # as root is refused at admission, which is a far better failure than discovering it later. + pod-security.kubernetes.io/enforce: restricted + pod-security.kubernetes.io/enforce-version: latest diff --git a/kubernetes/01-secret-job.yaml b/kubernetes/01-secret-job.yaml new file mode 100644 index 0000000..5ac315f --- /dev/null +++ b/kubernetes/01-secret-job.yaml @@ -0,0 +1,119 @@ +# Mint the generated secrets ONCE, from inside the cluster. +# +# ─── WHAT THIS REPLACES ─────────────────────────────────────────────────────────────────────── +# +# On Compose, `init-secrets.cjs` runs as a one-shot writing into a volume every container shares, +# and its guarantee that a key is generated exactly once is `openSync(path, "wx")` — two writers +# race, the kernel picks one. There is no shared filesystem across pods, so the exclusion moves to +# the one thing every pod does share: the API server. Creating a Secret that already exists is a +# 409, decided by one serialized writer. Same guarantee, same strength. +# +# This is the alternative to minting by hand, NOT a replacement for understanding what it mints. If +# your estate already has a secrets manager — SealedSecrets, ExternalSecrets, Vault — use it and +# skip this file; write `jarvis-generated` yourself with the four keys listed in the Role below. +# +# ─── THE ONE WAY TO GET HURT ────────────────────────────────────────────────────────────────── +# +# Deleting `jarvis-generated` and re-running this mints a NEW vault key against a database whose +# credentials are sealed under the old one. Nothing fails at boot. Every stored credential becomes +# permanently unreadable, and you find out the first time somebody opens one. There is no undo, and +# the key is not in the database backup by design. +# +# kubectl apply -f 01-secret.example.yaml # the half you write: DATABASE_URL, OPENAI_API_KEY… +# kubectl apply -f 01-secret-job.yaml # the half that is random bytes +# kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m +# # Then back the vault key up, out of this cluster: +# kubectl get secret jarvis-generated -n jarvis -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d +# +# ────────────────────────────────────────────────────────────────────────────────────────────── +apiVersion: v1 +kind: ServiceAccount +metadata: + name: jarvis-secret-minter + namespace: jarvis +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: jarvis-secret-minter + namespace: jarvis +rules: + # `create` AND NOTHING ELSE. No `get`, no `list`, no `update`. + # + # Deliberate, and worth the paragraph: this identity cannot read any Secret in the namespace, + # including the one it just wrote. That is why the script does not look before it writes — the + # 409 is the check, and it is a stronger one than a read-then-write could be, because a read and + # a write are two operations and something else can happen between them. + # + # `create` cannot be narrowed by resourceName — Kubernetes evaluates authorization before the + # object's name is known — so this permits creating any Secret in this namespace. That is the + # floor, and it is why the ServiceAccount belongs to this Job and to nothing else. + - apiGroups: [""] + resources: ["secrets"] + verbs: ["create"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: jarvis-secret-minter + namespace: jarvis +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: jarvis-secret-minter +subjects: + - kind: ServiceAccount + name: jarvis-secret-minter + namespace: jarvis +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: jarvis-mint-secrets + namespace: jarvis +spec: + # Re-running is safe (409 → left untouched), so a retry costs nothing. A high limit would only + # mean a broken RBAC took longer to become visible. + backoffLimit: 2 + # The Job object is tidied up an hour later. The Secret it created is a separate object and is + # not touched by this. + ttlSecondsAfterFinished: 3600 + template: + metadata: + labels: + app.kubernetes.io/name: jarvis-mint-secrets + spec: + restartPolicy: Never + serviceAccountName: jarvis-secret-minter + securityContext: + runAsNonRoot: true + runAsUser: 10001 + # Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces. + # Omitted, the Job is created and its pod is refused — which reads as a Job that never runs. + seccompProfile: + type: RuntimeDefault + containers: + - name: mint + # The api image, because the script ships in it and a first deployment should pull one + # image rather than three. By digest like every other image here — the same one you put + # in 03-migrate-job.yaml and 04-api.yaml. + image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME + command: ["node", "/app/apps/api/mint-k8s-secret.cjs"] + env: + # Node verifies the API server's certificate against the cluster CA. Without this the + # request would fail — which is the correct failure. Do not reach for + # NODE_TLS_REJECT_UNAUTHORIZED: a bearer token good enough to write Secrets is a bearer + # token worth stealing, and an unverified TLS session is where that happens. + - name: NODE_EXTRA_CA_CERTS + value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + memory: 256Mi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] diff --git a/kubernetes/01-secret.example.yaml b/kubernetes/01-secret.example.yaml new file mode 100644 index 0000000..a708727 --- /dev/null +++ b/kubernetes/01-secret.example.yaml @@ -0,0 +1,71 @@ +# The secrets an operator writes. Minted ONCE, mounted by every pod. +# +# ─── READ THIS BEFORE YOU GENERATE ANYTHING ─────────────────────────────────────────────────── +# +# There are two Secrets, and the split is not filing: `jarvis-generated` holds the values that are +# random bytes and `jarvis-secrets` — this file — holds the ones that are facts about your estate. +# One of those can be regenerated safely and the other cannot, so they do not share an object. +# +# The three keys below are here as the MANUAL path. If you would rather have them minted for you, +# apply `01-secret-job.yaml` instead and leave the `VAULT_*`/`JWT_*` lines out of this file +# entirely. Do not do both: `04-api.yaml` reads this Secret last on purpose, so a value typed here +# wins over a generated one, and the two quietly disagreeing is the shape of the accident. +# +# `apps/api/init-secrets.cjs` is the Compose generator, and it must NEVER become an initContainer +# here. Its guarantee that a key is minted exactly once is every writer sharing one filesystem. +# Per-pod, on an `emptyDir`, each replica generates its own — and the consequence is not a crash. +# Pod A seals a vault credential under a key pod B does not have; pod B reports that credential as +# corrupt; both pods log nothing at boot, because from each one's point of view everything is fine. +# By the time anybody notices, there are several keys in circulation and no way to tell which +# entries belong to which. `apps/api/mint-k8s-secret.cjs` is the same one-shot done correctly, with +# the API server's 409 standing in for the filesystem's exclusion. +# +# Either way, keep the output. The vault key in particular is the thing every stored credential is +# encrypted under — a database backup that travelled with its own key would be a backup that +# decrypts itself, which is why this lives apart from the database. +# +# VAULT_MASTER_KEY openssl rand -base64 32 +# JWT_ACCESS_SECRET openssl rand -base64 48 +# JWT_REFRESH_SECRET openssl rand -base64 48 +# +# In anything beyond a trial, do not commit this file with values in it. Use SealedSecrets, +# ExternalSecrets, or `kubectl create secret generic` from a password manager. +# ────────────────────────────────────────────────────────────────────────────────────────────── +apiVersion: v1 +kind: Secret +metadata: + name: jarvis-secrets + namespace: jarvis +type: Opaque +stringData: + # Everything in the vault is sealed under this. Lose it and the credentials are unreadable; + # leak it and a database dump is enough to read them all. + VAULT_MASTER_KEY: "REPLACE_ME" + JWT_ACCESS_SECRET: "REPLACE_ME" + JWT_REFRESH_SECRET: "REPLACE_ME" + + # WHEN THE VAULT KEY WAS CREATED, as an ISO 8601 timestamp or epoch seconds. + # + # Set it when you mint the key, and never move it afterwards. On Compose this is inferred from the + # key file's modification time, which is a good answer on a volume nobody rewrites. Here it is a + # wrong one: the kubelet writes a projected Secret into every pod at start, so the mtime is the + # POD's age and every pod reports a key created seconds ago however old it is. + # + # What that number decides is narrow and serious. An instance with no accounts can have its first + # super-admin claimed by whoever reaches the install screen — unless the vault key is old, which + # means the database has gone missing from an instance that already existed (a bad restore, a PVC + # pointed at the wrong volume) and the claim needs proof of possession instead. Without this line, + # that check reads "brand new instance" on exactly the estate it exists to protect. + VAULT_KEY_CREATED_AT: "REPLACE_ME" + + # `connection_limit` is not optional here, and it is the reason this URL is in the Secret rather + # than assembled from parts. Prisma's default pool is per-process: N pods times that default will + # exhaust `max_connections` long before the application is under any real load. Size it as + # (max_connections − reserve) ÷ (replicas + maxSurge + jobs). + DATABASE_URL: "postgresql://jarvis:REPLACE_ME@postgres.jarvis.svc.cluster.local:5432/jarvis?schema=public&connection_limit=10&pool_timeout=20" + + # Required to create new organizations, users or agents. Without it an instance keeps running + # everything it already has and creates nothing new. See the project README. + JARVIS_LICENSE_KEY: "" + + OPENAI_API_KEY: "REPLACE_ME" diff --git a/kubernetes/02-config.yaml b/kubernetes/02-config.yaml new file mode 100644 index 0000000..1df3e12 --- /dev/null +++ b/kubernetes/02-config.yaml @@ -0,0 +1,82 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: jarvis-api-config + namespace: jarvis +data: + NODE_ENV: "production" + API_PORT: "4000" + + # One logical keyspace, shared by every pod. Sentinel or a managed single-primary rather than + # Cluster: the Socket.IO adapter and the inter-pod bus both publish across what would otherwise + # be different slots. + # + # A SINGLE REDIS IS A SINGLE POINT OF FAILURE FOR EVERY POD AT ONCE. This line is correct for a + # managed Redis that fails over behind one address, and correct-but-fragile for one Deployment of + # one Redis. For Sentinel, comment this out and set the two below instead — see + # `08-redis-sentinel.yaml`, which deploys exactly that. + REDIS_URL: "redis://redis.jarvis.svc.cluster.local:6379" + + # Set BOTH or neither. Sentinel needs the sentinels and the name of the primary they watch; a + # name is refused rather than defaulted, because ioredis's own default (`mymaster`) against a + # primary called anything else is not an error — it is a pod that never becomes ready, silently. + # When these are set, REDIS_URL above is ignored. + # + #REDIS_SENTINELS: "redis-sentinel-0.redis-sentinel.jarvis.svc.cluster.local:26379,redis-sentinel-1.redis-sentinel.jarvis.svc.cluster.local:26379,redis-sentinel-2.redis-sentinel.jarvis.svc.cluster.local:26379" + #REDIS_SENTINEL_NAME: "jarvis" + + # ONE, not the two the compose files use, and getting this wrong is a security fault rather than + # a cosmetic one. + # + # It counts the proxies that REWRITE X-Forwarded-For. On compose a request passes the operator's + # TLS terminator and then the web container's nginx: two. Here the Ingress routes /api/ straight + # at the API Service, so there is one. Set HIGHER than the truth, the API believes a hop that does + # not exist — and since X-Forwarded-For is a request header, anyone can then prepend an address of + # their choosing and have it written into session records, audit rows, and the key the anonymous + # WebAuthn budget counts on. + # + # It is a STORED setting: this seeds the first boot, and the install screen — which shows the + # chain the API actually received — is where it is confirmed. + TRUST_PROXY_HOPS: "1" + + # The seed for the address enrolled agents dial back to. Whatever the install screen stores wins + # from then on, so this is a starting value rather than the answer. + WEB_ORIGIN: "https://jarvis.example.com" + + # Where the agent binaries are, filled per pod by an initContainer. Leaving it unset is + # supported: everything works except the agent installer, which answers 503 saying no build is + # published. + AGENT_RELEASE_DIR: "/srv/agent-releases" + + AGENT_HEARTBEAT_INTERVAL_SEC: "30" + # Must stay comfortably below the pod's terminationGracePeriodSeconds (90). Whichever expires + # first decides whether a restart leaves a closed transcript or a half-written one. + RUN_SHUTDOWN_GRACE_SEC: "25" + + # No demo organization and no seeded administrator: the install screen is the way in. The seed + # refuses an instance that already has accounts anyway, but a Deployment has no business + # furnishing anything. + SEED_ON_START: "false" + + OPENAI_BASE_URL: "https://api.openai.com/v1" + OPENAI_MODEL: "gpt-4o" + OPENAI_THINKING_LEVEL: "medium" +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: jarvis-web-config + namespace: jarvis +data: + # Where the web container's nginx sends /api/, the socket.io handshake and the agent websocket. + # + # FULLY QUALIFIED, and that is not decoration: nginx's `resolver` does not apply the search list + # from /etc/resolv.conf, so a bare `jarvis-api` would not resolve however correct it looks. + JARVIS_API_UPSTREAM: "jarvis-api.jarvis.svc.cluster.local:4000" + + # Who resolves it. The compose default is Docker's embedded DNS at 127.0.0.11, which nothing + # listens on inside a pod — every /api/ request answered 502 until this became substitutable. + # + # Replace with your cluster's DNS ClusterIP: + # kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}' + JARVIS_DNS_RESOLVER: "REPLACE_WITH_YOUR_CLUSTER_DNS" diff --git a/kubernetes/03-migrate-job.yaml b/kubernetes/03-migrate-job.yaml new file mode 100644 index 0000000..8463852 --- /dev/null +++ b/kubernetes/03-migrate-job.yaml @@ -0,0 +1,93 @@ +# The schema, brought up to this image's expectations. Once, before any pod of it starts serving. +# +# A JOB, and deliberately neither of the two alternatives: +# +# NOT an initContainer. It would run once per pod — which is exactly the bug this replaces, where +# every API container ran `prisma db push` at boot and N replicas raced the same DDL while the +# previous generation was still selecting the columns being altered. +# +# NOT a leader election inside the API. A Job is the primitive built for run-once work: it has an +# observable status, a bounded number of retries, and a failure that stops the rollout instead of +# letting it proceed against a schema that was never applied. +# +# With Helm this becomes a `pre-install,pre-upgrade` hook; with Argo, a sync-wave. Applied by hand, +# the ordering is yours to keep — see the README. +# +# It is safe to run twice: every pass is idempotent and the whole phase is wrapped in a Postgres +# advisory lock, so a second copy waits rather than racing. +apiVersion: batch/v1 +kind: Job +metadata: + name: jarvis-migrate + namespace: jarvis +spec: + # One writer. The lock makes a second one wait rather than corrupt, but there is no reason to + # have one waiting. + parallelism: 1 + completions: 1 + # Two retries, because the failures worth retrying are transient (a database not finished failing + # over). A schema that Prisma REFUSES is not transient and must not be retried into submission — + # it is refused for a reason, and the reason is printed. + backoffLimit: 2 + # A migration that has not finished in fifteen minutes is stuck, not slow, and the rollout is + # blocked behind it. + activeDeadlineSeconds: 900 + template: + metadata: + labels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: migrate + spec: + restartPolicy: Never + # Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces. At pod + # level so it covers anything added beside this container later. Without it the Job is + # accepted and its pod is refused, so the rollout waits on a Job that will never complete. + securityContext: + seccompProfile: + type: RuntimeDefault + containers: + - name: migrate + # THE SAME IMAGE, by the same digest, as the API that is about to roll. A migration from a + # different build is a schema that matches nothing. + image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME + command: ["/usr/local/bin/migrate.sh"] + envFrom: + - configMapRef: + name: jarvis-api-config + # Generated first, hand-written last, for the reason spelled out in 04-api.yaml: the + # last occurrence of a name wins, and a generated value must never override a typed one. + - secretRef: + name: jarvis-generated + optional: true + - secretRef: + name: jarvis-secrets + env: + # Its own, smaller pool: this is one process doing one thing, and the connections it + # takes are connections the running API cannot have while it works. + - name: DATABASE_URL + valueFrom: + secretKeyRef: + name: jarvis-secrets + key: DATABASE_URL + # Off, and it should stay off between deliberate acts. `prisma db push` used to run with + # `--accept-data-loss` at every boot, which meant an upgrade could destroy a column with + # nobody having decided to — survivable on one container, not during a rolling update + # where the previous generation is still selecting it. + # + # Without the flag the phase REFUSES, prints which columns it would have dropped, and + # fails the Job — so the rollout stops and nothing is serving a half-migrated schema. + # Set it to "true" for the ONE deploy where that drop is intended, then remove it. + - name: SCHEMA_ACCEPT_DATA_LOSS + value: "false" + resources: + requests: + cpu: 200m + memory: 512Mi + limits: + memory: 1Gi + securityContext: + runAsNonRoot: true + runAsUser: 1000 + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] diff --git a/kubernetes/04-api.yaml b/kubernetes/04-api.yaml new file mode 100644 index 0000000..b3f7d3c --- /dev/null +++ b/kubernetes/04-api.yaml @@ -0,0 +1,194 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: jarvis-api + namespace: jarvis +spec: + replicas: 2 + selector: + matchLabels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: api + strategy: + type: RollingUpdate + rollingUpdate: + # Never fewer pods than are being served now. A run holds its conversation through a unique + # index, so a surge pod cannot double-answer anything — the old constraint that made an + # overlap dangerous is gone. + maxUnavailable: 0 + maxSurge: 1 + template: + metadata: + labels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: api + spec: + # 90 seconds, and every part of it is accounted for: + # 10s preStop, so this pod leaves the Service's endpoints BEFORE it starts draining + # 25s RUN_SHUTDOWN_GRACE_SEC — in-flight assistant runs stop and close their transcripts + # ~15s agent sockets closed and their rows written, httpServer.close(), Prisma, Redis + # the rest is margin + # The default of 30 would SIGKILL the pod in the middle of the run drain, which is the one + # thing the drain exists to prevent. + terminationGracePeriodSeconds: 90 + # Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces. At pod + # level rather than per container, because the agent-releases initContainer below needs it + # too — and a missing seccompProfile is not a warning, it is the pod being refused outright. + securityContext: + seccompProfile: + type: RuntimeDefault + # SPREAD THE REPLICAS, because nothing else will. + # + # The scheduler has no reason of its own to put two pods of the same Deployment on different + # nodes, and quite often has a reason not to. Two API replicas on one node is a deployment + # that reads as highly available in `kubectl get pods` and loses every agent websocket, every + # live run and every open shell the moment that one node reboots. The PodDisruptionBudget in + # 07-disruption.yaml does not help here: it constrains voluntary evictions, and a node dying + # is not one. + # + # `ScheduleAnyway` rather than `DoNotSchedule`: on a single-node cluster the strict form + # leaves the second replica Pending forever, which is a worse first experience than an + # unspread pair. On a real cluster it spreads. If yours spans zones, add a second constraint + # on `topology.kubernetes.io/zone` — a rack is a failure domain too. + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: api + containers: + - name: api + # BY DIGEST. `:stable` is a moving name, so replicas that restart at different times land + # on different builds — and a rollback becomes "hope the tag still points where it did". + image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME + ports: + - name: http + containerPort: 4000 + lifecycle: + preStop: + exec: + # Endpoint removal and SIGTERM race each other, and losing that race is visible: + # a prompt that arrives after Nest sets `draining` is persisted and broadcast as + # queued, and this pod will never answer it. Ten seconds is comfortably more than + # kube-proxy needs to stop sending new connections here. + command: ["/bin/sh", "-c", "sleep 10"] + env: + - name: JARVIS_SKIP_MIGRATIONS + # The schema is the migration Job's business, once, before this rolls. N pods each + # running `prisma db push` is N writers racing the same DDL while the previous + # generation is still selecting the columns being altered. + value: "true" + envFrom: + - configMapRef: + name: jarvis-api-config + # The half that is random bytes, minted once by `01-secret-job.yaml`. Kept apart from + # the half an operator writes because one of them can be regenerated and the other + # cannot: re-running the minter is a no-op by design, which is only safe while nothing + # anybody typed lives in the same object. + # + # `optional: true` so a deployment that mints by hand — or through SealedSecrets, + # ExternalSecrets, Vault — does not need this object to exist at all. + - secretRef: + name: jarvis-generated + optional: true + # LAST, and the order is the safety property rather than a formatting choice. `envFrom` + # resolves in sequence and the last occurrence of a name wins, so anything written by + # hand here overrides the generated value — never the other way round. Reversed, an + # operator who had already minted a vault key and then applied the Job by reflex would + # have every stored credential silently become unreadable. + - secretRef: + name: jarvis-secrets + volumeMounts: + - name: agent-releases + mountPath: /srv/agent-releases + readOnly: true + # Liveness NEVER touches the database. A Postgres failover with a database check here + # restarts every pod at once, turning a thirty-second blip into a cold start of the whole + # deployment at the moment the database can least afford a reconnection stampede. + livenessProbe: + httpGet: + path: /api/health/live + port: http + periodSeconds: 10 + failureThreshold: 3 + # Readiness DOES, and it checks Redis too. A pod cut off from Redis keeps answering HTTP + # perfectly while its broadcasts reach nobody — the clients queue rather than fail — so it + # looks healthier than a pod that is down and is more dangerous. + readinessProbe: + httpGet: + path: /api/health/ready + port: http + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + # Boot does a run-recovery sweep and reads settings before it listens. The startup probe + # is what stops liveness killing it halfway through on a slow database. + startupProbe: + httpGet: + path: /api/health/live + port: http + periodSeconds: 5 + failureThreshold: 30 + resources: + requests: + cpu: 500m + # Not generous — honest. A terminal recording holds up to 8 MiB of Buffers in heap + # for the life of the session, and argon2 takes 64 MiB per share verification. + memory: 1Gi + limits: + memory: 2Gi + securityContext: + runAsNonRoot: true + runAsUser: 1000 + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + initContainers: + # The compiled agent binaries, into this pod's own emptyDir. + # + # PER POD ON PURPOSE. The content is immutable and small, so a copy per pod costs nothing; + # a shared RWO PVC would either pin every pod to one node or leave the second in + # Multi-Attach error. Leaving this out entirely is supported: enrollment still works and the + # installer answers 503 saying no build is published. + - name: agent-releases + image: git.luxit.be/luxit/jarvis-agent-dist@sha256:REPLACE_ME + command: ["/bin/sh", "-c", "cp -a /dist/. /srv/agent-releases/"] + # AN initContainer IS A CONTAINER, and the `restricted` Pod Security Standard judges it + # exactly like the one below. This block was missing, and the result was not a warning + # about the initContainer — it was the WHOLE POD refused, so the Deployment sat at zero + # replicas reporting FailedCreate while every other line in this file was correct. + securityContext: + runAsNonRoot: true + runAsUser: 1000 + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + volumeMounts: + - name: agent-releases + mountPath: /srv/agent-releases + volumes: + - name: agent-releases + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: jarvis-api + namespace: jarvis +spec: + type: ClusterIP + # STATED, not left to the default, because the reflex is to reach for it the moment websockets + # are involved. It would not help: the hard case is co-locating an operator's browser with an + # agent's websocket, which arrives separately and whose id is unknown at the browser's handshake. + # That is handled in the application, over Redis. See the README. + sessionAffinity: None + selector: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: api + ports: + - name: http + port: 4000 + targetPort: http diff --git a/kubernetes/05-web.yaml b/kubernetes/05-web.yaml new file mode 100644 index 0000000..f9f0e2f --- /dev/null +++ b/kubernetes/05-web.yaml @@ -0,0 +1,152 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: jarvis-web + namespace: jarvis +spec: + replicas: 2 + selector: + matchLabels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: web + strategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 0 + maxSurge: 1 + template: + metadata: + labels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: web + spec: + # Nothing here holds a long-lived connection of its own — the websockets go to the API — so + # this is only about letting in-flight responses finish. + terminationGracePeriodSeconds: 30 + # Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces. + securityContext: + seccompProfile: + type: RuntimeDefault + # Same reasoning as 04-api.yaml: the scheduler will happily put both replicas on one node, + # and a pair that shares a node is a pair that shares a failure. Cheaper to satisfy here — + # nothing about serving static files pins a pod anywhere. + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: web + containers: + - name: web + # By digest, like the API. It matters more here than it looks: the bundle's JavaScript + # chunks are content-hashed PER BUILD and answer 404 rather than falling through to + # index.html, so during a rollout a browser holding the previous shell can fail a lazy + # import. Two pods on two different builds widen that window; two pods on one digest do + # not have it at all. + image: git.luxit.be/luxit/jarvis-web@sha256:REPLACE_ME + ports: + - name: http + containerPort: 8080 + lifecycle: + preStop: + exec: + command: ["/bin/sh", "-c", "sleep 5"] + envFrom: + - configMapRef: + name: jarvis-web-config + # Is this nginx answering? Nothing more. Restarting the web container because the API is + # down would take out the one thing still able to show an error page. + livenessProbe: + httpGet: + path: /healthz + port: http + periodSeconds: 10 + failureThreshold: 3 + # Can THIS pod reach the API? It proxies /api/health/live, not /ready, on purpose: the + # question is whether this pod's route works — a broken DNS name, a NetworkPolicy — and + # not whether the API's dependencies are healthy. Pointed at /ready, one Postgres blip + # would fail every web pod at once and take the console offline for a fault that has + # nothing to do with serving a bundle. + readinessProbe: + httpGet: + path: /readyz + port: http + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + memory: 256Mi + securityContext: + # uid 101 is the `nginx` user the image already ships and the Dockerfile switches to. + runAsNonRoot: true + runAsUser: 101 + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + readOnlyRootFilesystem: true + volumeMounts: + # With a read-only root filesystem, the three paths this container writes at runtime + # have to be given to it. All three are ephemeral by nature — the substituted config is + # regenerated at every start from the template and the ConfigMap. + - name: nginx-conf + mountPath: /etc/nginx/conf.d + - name: nginx-cache + mountPath: /var/cache/nginx + - name: tmp + # The pid file lives here rather than in /var/run, which the nginx user does not own. + mountPath: /tmp + volumes: + - name: nginx-conf + emptyDir: {} + - name: nginx-cache + emptyDir: {} + - name: tmp + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: jarvis-web + namespace: jarvis +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: web + ports: + - name: http + port: 8080 + targetPort: http +--- +# Web scales on CPU; the API deliberately does not. +# +# Every API scale-in kills a pod holding agent websockets, live assistant runs and open shells — +# the fleet reconnects, runs are interrupted and resumed, terminals die — and CPU is a poor proxy +# for a load made almost entirely of long-lived connections. Raise `jarvis-api` deliberately +# instead. If it ever does get an HPA, give it a scale-down stabilization window of ten minutes or +# more. +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: jarvis-web + namespace: jarvis +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: jarvis-web + minReplicas: 2 + maxReplicas: 5 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 diff --git a/kubernetes/06-ingress.yaml b/kubernetes/06-ingress.yaml new file mode 100644 index 0000000..db64e77 --- /dev/null +++ b/kubernetes/06-ingress.yaml @@ -0,0 +1,81 @@ +# One hostname, two backends. +# +# The API is routed STRAIGHT AT ITS SERVICE rather than through the web container's nginx, which is +# what that nginx does on Compose. Two reasons: it removes a hop from every API request and every +# websocket frame, and it makes the web pod a pure static server again — no dependency on resolving +# the API, nothing to keep in sync between an ingress and a config file. +# +# The consequence has to be applied, not just noted: there is now exactly ONE proxy rewriting +# X-Forwarded-For, so `TRUST_PROXY_HOPS` is 1. It is 2 in the compose files, and leaving it there +# would make `req.ip` a value the caller chooses. See 02-config.yaml. +# +# Set `externalTrafficPolicy: Local` on the ingress controller's own Service as well, or the node's +# SNAT will make every request in the world appear to come from a handful of addresses — which +# defeats every per-address limit in the application at once. +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: jarvis + namespace: jarvis + annotations: + # An agent is idle between operations and the server pings every 30s, so this only has to + # outlast a quiet period comfortably. Too short and a healthy fleet reconnects all day. + nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "3600" + # Matches BODY_LIMIT in apps/api/src/main.ts. A document may hold 400,000 characters, which is + # over a megabyte once accented text is UTF-8 encoded, and a diagram export posts back the SVG + # the browser rendered. Below this they are rejected as a bare 413 with no message. + nginx.ingress.kubernetes.io/proxy-body-size: "4m" + # + # NO affinity annotation, and none should be added. The reflex is understandable — there are + # websockets here — but stickiness cannot solve the problem it would be hired for: the hard case + # is co-locating an operator's browser with an AGENT's websocket, which arrives from a different + # network at a different time and whose id is not even known at the browser's handshake. No + # ingress annotation can express that; it is handled in the application, over Redis. + # + # And there is no polling to make sticky: both the client and the server pin + # transports: ["websocket"], so one logical socket is one TCP connection. +spec: + ingressClassName: nginx + tls: + - hosts: ["jarvis.example.com"] + secretName: jarvis-tls + rules: + - host: jarvis.example.com + http: + paths: + # Longest prefix first for readers; the controller sorts by specificity itself. + # + # The agent websocket has its own rule even though it sits under /api/, mirroring the + # split in apps/web/nginx.conf: it is the one path where an idle connection legitimately + # stays open for an hour between frames. + - path: /api/agents/ws + pathType: Prefix + backend: + service: + name: jarvis-api + port: + name: http + - path: /socket.io/ + pathType: Prefix + backend: + service: + name: jarvis-api + port: + name: http + - path: /api/ + pathType: Prefix + backend: + service: + name: jarvis-api + port: + name: http + # Everything else is the SPA, whose nginx answers index.html for any path it does not + # have a file for. + - path: / + pathType: Prefix + backend: + service: + name: jarvis-web + port: + name: http diff --git a/kubernetes/07-disruption.yaml b/kubernetes/07-disruption.yaml new file mode 100644 index 0000000..06e2839 --- /dev/null +++ b/kubernetes/07-disruption.yaml @@ -0,0 +1,28 @@ +# What a node drain may take away. +# +# `minAvailable: 1` on both, so a cluster upgrade cannot evict the last pod of either. With the +# API's 90-second grace period a drain will take a minute or two per pod — that is the run drain +# doing its job, not a stall. +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: jarvis-api + namespace: jarvis +spec: + minAvailable: 1 + selector: + matchLabels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: api +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: jarvis-web + namespace: jarvis +spec: + minAvailable: 1 + selector: + matchLabels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: web diff --git a/kubernetes/08-redis-sentinel.yaml b/kubernetes/08-redis-sentinel.yaml new file mode 100644 index 0000000..24050c5 --- /dev/null +++ b/kubernetes/08-redis-sentinel.yaml @@ -0,0 +1,341 @@ +# A Redis that survives losing a node. +# +# ─── WHAT THIS IS, AND WHAT IT IS NOT ───────────────────────────────────────────────────────── +# +# OPTIONAL. Apply it only if you are running your own Redis. A managed Redis that publishes one +# address and fails over behind it needs none of this — point `REDIS_URL` at it and skip the file. +# +# It is written to be correct and it is small enough to read in full, but a datastore you operate +# yourself is a datastore you maintain: version upgrades, capacity, the failover drill. If you want +# somebody else to hold that, use a managed Redis or an operator. What this directory owes you is +# the shape the application needs, and this is it. +# +# The application half is proven rather than assumed: an ioredis client built exactly the way +# `apps/api/src/redis/redis-connection.ts` builds one was run against a live three-sentinel +# topology while its primary was killed. It followed the promotion to a different server, and 99 +# writes issued across the window were rejected zero times. What it does NOT do is preserve pub/sub +# during the switch — see "What a failover actually costs" at the bottom. +# +# kubectl apply -f 08-redis-sentinel.yaml +# # then in 02-config.yaml: comment out REDIS_URL, uncomment the two REDIS_SENTINEL* lines +# +# ────────────────────────────────────────────────────────────────────────────────────────────── +apiVersion: v1 +kind: ConfigMap +metadata: + name: redis-scripts + namespace: jarvis +data: + # WHO IS THE PRIMARY — asked, never assumed. + # + # The naive version of this file pins `replicaof redis-0` into every replica. It works exactly + # until the first failover, after which redis-0 comes back, is told it is the primary, and the + # cluster has two — a split brain that presents as writes vanishing rather than as an error. + # + # So a starting node asks the sentinels who the primary is now, and only falls back to redis-0 + # when no sentinel answers, which is true on a first apply and at no other time. + discover.sh: | + #!/bin/sh + # Prints the current primary's hostname. Empty output is impossible: the fallback is redis-0. + for i in 0 1 2; do + peer="redis-sentinel-${i}.redis-sentinel.${NAMESPACE}.svc.cluster.local" + found=$(redis-cli -h "$peer" -p 26379 sentinel get-master-addr-by-name jarvis 2>/dev/null | head -1) + if [ -n "$found" ]; then + echo "$found" + exit 0 + fi + done + echo "redis-0.redis.${NAMESPACE}.svc.cluster.local" + + redis-start.sh: | + #!/bin/sh + set -eu + master=$(sh /scripts/discover.sh) + me="${HOSTNAME}.redis.${NAMESPACE}.svc.cluster.local" + echo "[redis] I am ${me}; the primary is ${master}" + if [ "$master" = "$me" ]; then + exec redis-server /etc/redis/redis.conf + fi + exec redis-server /etc/redis/redis.conf --replicaof "$master" 6379 + + sentinel-start.sh: | + #!/bin/sh + set -eu + master=$(sh /scripts/discover.sh) + echo "[sentinel] monitoring ${master}" + # Rebuilt every start rather than persisted. A sentinel REWRITES its own config file as it + # learns, so the file cannot be a read-only ConfigMap mount; and its learned state is + # recoverable from its peers in seconds, so persisting it buys nothing and can resurrect a + # stale view of a topology that has since moved. + cp /etc/sentinel-template/sentinel.conf /data/sentinel.conf + { + echo "sentinel monitor jarvis ${master} 6379 2" + echo "sentinel down-after-milliseconds jarvis 5000" + echo "sentinel failover-timeout jarvis 60000" + echo "sentinel parallel-syncs jarvis 1" + } >> /data/sentinel.conf + exec redis-sentinel /data/sentinel.conf +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: redis-config + namespace: jarvis +data: + redis.conf: | + port 6379 + # NO PERSISTENCE, and it is a decision rather than an omission. Everything Jarvis keeps in Redis + # is ephemeral by design and rebuilds itself: agent ownership keys carry a TTL the owning pod + # refreshes, cluster leases expire, and the realtime adapter's state is the live connections + # themselves. The one thing that must NOT be lost — which pod owns a run — is a row in Postgres + # precisely so that losing Redis cannot lose it. + save "" + appendonly no + # Replicas answer reads while they catch up rather than erroring. Nothing here reads from a + # replica today, but a client that briefly lands on one during a promotion should see slightly + # old data rather than a failure. + replica-serve-stale-data yes + # A replica must never accept a write, however it was reached. + replica-read-only yes + + sentinel.conf: | + port 26379 + # HOSTNAMES, NOT POD IPs, and this is the line most likely to be dropped as noise. + # + # By default a sentinel hands clients the primary's IP. A pod's IP does not survive its + # restart, so after a failover-then-restart the sentinels answer with an address nothing is + # listening on — and a client pointed at it does not fail, it retries forever against a dead + # address while the sentinels insist that is where the primary lives. Observed, not theorised. + sentinel resolve-hostnames yes + sentinel announce-hostnames yes +--- +apiVersion: v1 +kind: Service +metadata: + name: redis + namespace: jarvis +spec: + # Headless: the point is the per-pod DNS names (redis-0.redis…), which are what the sentinels + # hand out and therefore what the API ends up connecting to. + clusterIP: None + publishNotReadyAddresses: true + selector: + app.kubernetes.io/name: redis + ports: + - name: redis + port: 6379 + targetPort: 6379 +--- +apiVersion: v1 +kind: Service +metadata: + name: redis-sentinel + namespace: jarvis +spec: + clusterIP: None + publishNotReadyAddresses: true + selector: + app.kubernetes.io/name: redis-sentinel + ports: + - name: sentinel + port: 26379 + targetPort: 26379 +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: redis + namespace: jarvis +spec: + serviceName: redis + replicas: 3 + selector: + matchLabels: + app.kubernetes.io/name: redis + template: + metadata: + labels: + app.kubernetes.io/name: redis + spec: + terminationGracePeriodSeconds: 30 + affinity: + podAntiAffinity: + # REQUIRED, not preferred. Three replicas on one node is a topology that looks highly + # available in `kubectl get pods` and is not one. + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: redis + topologyKey: kubernetes.io/hostname + securityContext: + runAsNonRoot: true + runAsUser: 999 + fsGroup: 999 + # REQUIRED by the `restricted` Pod Security Standard, which 00-namespace.yaml enforces. + # Without it the pods are not warned about, they are REFUSED — the StatefulSet is created, + # reports FailedCreate, and stays at zero replicas. Found by applying it, not by + # kubeconform: schema validation cannot see an admission policy. + seccompProfile: + type: RuntimeDefault + containers: + - name: redis + image: redis:7-alpine + command: ["sh", "/scripts/redis-start.sh"] + env: + - name: NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + ports: + - name: redis + containerPort: 6379 + readinessProbe: + exec: + command: ["redis-cli", "ping"] + initialDelaySeconds: 5 + periodSeconds: 5 + livenessProbe: + exec: + command: ["redis-cli", "ping"] + initialDelaySeconds: 15 + periodSeconds: 10 + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + memory: 512Mi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + volumeMounts: + - name: scripts + mountPath: /scripts + - name: config + mountPath: /etc/redis + - name: data + mountPath: /data + volumes: + - name: scripts + configMap: + name: redis-scripts + - name: config + configMap: + name: redis-config + - name: data + # No PVC on purpose — see `save ""` above. A volumeClaimTemplate here would persist a + # dataset that is meant to be rebuildable, and give a restarted node a stale one. + emptyDir: {} +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: redis-sentinel + namespace: jarvis +spec: + serviceName: redis-sentinel + replicas: 3 + selector: + matchLabels: + app.kubernetes.io/name: redis-sentinel + template: + metadata: + labels: + app.kubernetes.io/name: redis-sentinel + spec: + terminationGracePeriodSeconds: 30 + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: redis-sentinel + topologyKey: kubernetes.io/hostname + securityContext: + runAsNonRoot: true + runAsUser: 999 + fsGroup: 999 + # REQUIRED by the `restricted` Pod Security Standard, which 00-namespace.yaml enforces. + # Without it the pods are not warned about, they are REFUSED — the StatefulSet is created, + # reports FailedCreate, and stays at zero replicas. Found by applying it, not by + # kubeconform: schema validation cannot see an admission policy. + seccompProfile: + type: RuntimeDefault + containers: + - name: sentinel + image: redis:7-alpine + command: ["sh", "/scripts/sentinel-start.sh"] + env: + - name: NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + ports: + - name: sentinel + containerPort: 26379 + readinessProbe: + exec: + command: ["redis-cli", "-p", "26379", "ping"] + initialDelaySeconds: 5 + periodSeconds: 5 + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + memory: 128Mi + securityContext: + allowPrivilegeEscalation: false + # The root filesystem is read-only and the sentinel still works, because the one thing + # it must write — its own config, which it rewrites as it learns the topology — lives on + # the writable emptyDir at /data. That is why the template is COPIED there at start + # rather than run from the ConfigMap mount, which is read-only whatever this says. + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + volumeMounts: + - name: scripts + mountPath: /scripts + - name: sentinel-template + mountPath: /etc/sentinel-template + - name: data + mountPath: /data + volumes: + - name: scripts + configMap: + name: redis-scripts + - name: sentinel-template + configMap: + name: redis-config + items: + - key: sentinel.conf + path: sentinel.conf + - name: data + emptyDir: {} +--- +# Two of three sentinels must be reachable for the quorum to hold. Without this, a node drain can +# take two at once — and a Redis with no quorum does not fail over, which is the entire point of +# having three. +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: redis-sentinel + namespace: jarvis +spec: + minAvailable: 2 + selector: + matchLabels: + app.kubernetes.io/name: redis-sentinel +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: redis + namespace: jarvis +spec: + minAvailable: 2 + selector: + matchLabels: + app.kubernetes.io/name: redis diff --git a/kubernetes/README.md b/kubernetes/README.md new file mode 100644 index 0000000..5cecb76 --- /dev/null +++ b/kubernetes/README.md @@ -0,0 +1,206 @@ +# Jarvis on Kubernetes + +Plain manifests, no Helm, no operator. They are written to be read as much as applied: every value +that is not a Kubernetes default is there because something in Jarvis behaves a particular way, and +the comment beside it says which. Read them before you apply them — that is the point of shipping +them this way. + +**You probably do not need this.** [Docker Compose](../README.md#install) runs one of everything, +takes one command, and is what most instances run. Come here when you want Jarvis to keep serving +while a node reboots, and when you already operate a cluster — this directory assumes you have +opinions about ingress, storage and a database, because it deliberately does not have them for you. + +> **What has been proven, and what has not.** This stack has been brought up on a real cluster — +> four nodes, three Postgres instances, three Redis with three sentinels, two api pods — and kept +> serving while its Postgres primary and its Redis primary were killed underneath it. What has NOT +> happened is a long-running estate serving production traffic this way. If you are early, watch it +> rather than assume it, and read the three warnings below before you apply anything, because each +> of them fails silently. + +## What runs more than once, and what cannot + +"Highly available" is not a property a deployment has as a whole, so here it is component by +component. + +| component | replicas | why | +| --- | --- | --- | +| `jarvis-web` | any number | stateless nginx over static files | +| `jarvis-api` | any number | the coordination lives in Postgres and Redis, not in the process | +| Redis | one **logical** primary, three processes | Sentinel fails over; it does not spread writes | +| Postgres | one **primary**, replicas behind it | and Jarvis depends on there being exactly one | +| `jarvis-migrate` | one at a time | enforced by a Postgres advisory lock, not by you | +| `jarvis-mint-secrets` | once, ever | enforced by the API server | + +The single-writer database is not a limitation Jarvis has failed to overcome — it is what makes the +multi-pod api safe. A partial unique index is what stops two pods running the same assistant loop, +and an index can only do that because one server decides. Do not point pods at read replicas. + +## What you need before you apply anything + +- **x86-64 nodes.** The api and web images are published for `linux/amd64` only, so an arm64 pool + leaves the pods in `ImagePullBackOff` with no matching manifest. The machines you *administer* + have no such limit — the agent ships arm64 builds. +- **A cluster you can schedule on.** `kubectl describe node | grep Taints` first. A three-node + cluster is often three control-plane nodes, and their `NoSchedule` taint means nothing here + schedules at all. These manifests carry no toleration on purpose: adding one is a decision about + your cluster. +- **Postgres 16, highly available, outside these manifests.** Use an operator — CloudNativePG or + Zalando's — rather than a hand-written StatefulSet. A database you hand-rolled is a database you + hand-restore. `max_connections` must cover + `(replicas + maxSurge + jobs) × connection_limit + reserve`: with 3 api pods at 10 connections + each, a migration Job at 5, and headroom, about **150**. The stock Postgres image ships 100, and + exhausting it presents as an api that starts and then cannot serve. +- **Redis.** [`08-redis-sentinel.yaml`](08-redis-sentinel.yaml) deploys one, or point `REDIS_URL` at + a managed Redis that fails over behind a single address and skip that file. Persistence is not + required: everything Jarvis keeps in Redis is ephemeral by design, which is exactly why the run + lock is in Postgres instead. +- **An Ingress controller and a TLS terminator.** Jarvis speaks plain HTTP. + +## Order + +```sh +kubectl apply -f 00-namespace.yaml + +# The secrets, ONCE. Never regenerate them on an instance that has stored anything. +kubectl apply -f 01-secret.example.yaml # after editing — the half you write +kubectl apply -f 01-secret-job.yaml # the half that is random bytes (optional; see the file) +kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m + +kubectl apply -f 02-config.yaml +kubectl apply -f 08-redis-sentinel.yaml # only if you are running your own Redis + +# The schema, to completion, before any api pod starts. +kubectl apply -f 03-migrate-job.yaml +kubectl wait --for=condition=complete job/jarvis-migrate -n jarvis --timeout=15m + +kubectl apply -f 04-api.yaml +kubectl apply -f 05-web.yaml +kubectl apply -f 06-ingress.yaml +kubectl apply -f 07-disruption.yaml +``` + +Every image says `REPLACE_ME`: pin them by digest, not by tag. `:stable` is a moving name, so +replicas that restart at different times land on different builds and a rollback becomes "hope the +tag still points where it did". + +```sh +docker buildx imagetools inspect git.luxit.be/luxit/jarvis-api:stable +``` + +On every upgrade the migration Job runs again before the Deployments roll. It is idempotent and +takes a Postgres advisory lock, so running it twice is slow rather than harmful — but it must +finish before new pods start, which is what the `kubectl wait` is for. With Helm or Argo this +becomes a `pre-upgrade` hook or a sync-wave; as plain YAML it is an ordering you keep yourself. + +**Then open the address and install it.** Everything in [the main README](../README.md#install) +about the install screen applies unchanged — including that **whoever reaches an uninstalled +instance first owns it**, with no deadline. Point a public hostname at this after you have +installed, not before. + +## The three things most likely to bite + +**Never mint secrets per pod.** On Compose a one-shot generates the vault key and the JWT secrets +into a volume every container shares, and its guarantee that it does so exactly once is that the +filesystem refuses a second create. As a per-pod initContainer on an `emptyDir`, each replica mints +its own — and the consequence is not a crash. Pod A seals a credential under a key pod B does not +have; pod B reports it as corrupt; both log a clean boot. By the time anybody opens a vault entry +there are three keys in circulation. + +[`01-secret-job.yaml`](01-secret-job.yaml) is that one-shot done correctly here: it creates the +Secret once and lets the API server's refusal to create it twice be the exclusion. Its identity can +`create` a Secret and cannot `get` one, so it cannot read back what it or anyone else wrote. Mint by +hand instead if you prefer — the one thing that is not an option is per-pod. + +Either way, **back the vault key up somewhere that is not this cluster's etcd.** Deleting the Secret +and re-running the Job mints a new key against data sealed with the old one, silently, with no way +back. + +**`TRUST_PROXY_HOPS` is one hop here, not two.** The Compose files say 2 because a request passes +your TLS terminator and then the web container's nginx. Here the Ingress routes `/api/` straight at +the api Service, so there is exactly one proxy rewriting `X-Forwarded-For`. Left at 2, `req.ip` +becomes **chosen by the caller**: anyone can send `X-Forwarded-For: 198.51.100.7` and have that +written into session records, audit rows and the key the anonymous WebAuthn budget counts on. + +**Do not add session affinity.** It is the reflex the moment websockets are involved and it solves +nothing here: the hard case is co-locating an operator's browser with an *agent's* websocket, which +arrives from a different network at a different time and whose id is not even known at the browser's +handshake. No ingress annotation can express that. It is handled in the application, over Redis. +Both ends pin `transports: ["websocket"]`, so there is no polling to make sticky in the first place. + +## On exactly three nodes + +The common case, and where the arithmetic is tightest. + +**What lands where.** Redis and the sentinels both use *required* anti-affinity, so three of each +means one per node — six pods, two per node, no choice left to the scheduler. Postgres at three +instances is the same. That is what you want, and it has a consequence: **during a node drain, one +Redis pod and one sentinel have nowhere to go and stay `Pending` until the node returns.** That is +correct — a quorum of two still holds — but `kubectl drain` will sit there unless the +PodDisruptionBudgets are applied to tell it what is safe. + +**Three api replicas, not two.** [`04-api.yaml`](04-api.yaml) ships two because that is the smallest +number that proves the code is not single-instance. On three nodes, three is strictly better: one +per node, and losing any node leaves two. Change `replicas: 2` to `3` in it and in +[`05-web.yaml`](05-web.yaml), and size `max_connections` for it. + +**Three nodes tolerates losing one.** Not two. That is a property of majority quorums, and the +answer to wanting more is a fourth and fifth node, not a different configuration. + +## What a failover actually costs + +Measured, not estimated. + +**Redis**, with its primary killed under a running api: + +- **The api rides it out without restarting.** Readiness answered 503 for about fifteen seconds and + then 200 again, on the same process — the pod leaves the Service's endpoints while it cannot reach + Redis, and returns when it can. Liveness never touches Redis, which is why the pod is not killed + and restarted into a cold start. +- **Commands survive.** 99 writes issued across the window, 0 rejected, the last readable from the + promoted primary afterwards. Commands queue rather than fail, so a failover is a pause. +- **Published messages do not.** Around 45% of the messages published during the window reached + nobody. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change + that. In practice: a few seconds in which something happening on one api replica may not reach + another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason. + +**Postgres**, primary deleted with `--force --grace-period=0`: + +- **Three seconds.** Both api pods answered `503 {"database":"unreachable"}` and were back to 200 on + the fourth poll. Both degraded together, which is what pointing every pod at one endpoint means. +- **Zero container restarts**, across two failovers, on pods up throughout — liveness deliberately + not touching the database is what keeps three seconds from becoming a cold start of everything. + +A failover *during* the migration Job aborts it with a lost-connection error, which is the correct +outcome. Run it again. + +## What is still worth knowing + +- **`AGENT_RELEASE_DIR` is per-pod, and must be.** It holds the compiled agent binaries. An + `emptyDir` filled by an initContainer is right; a shared RWO volume is not — either every pod is + pinned to one node, or the second sits in `Multi-Attach error`. Leaving it unset is supported: + everything works except the agent installer, which answers 503 saying no build is published. +- **No application volume, and it should stay that way.** Avatars, documents and terminal recordings + are all in Postgres; exports are rendered in memory. +- **Autoscale `jarvis-web`, not `jarvis-api`.** Every api scale-in kills a pod holding agent + websockets, live runs and open shells, and CPU is a poor proxy for a load made of long-lived + connections. Raise api replicas deliberately. +- **A rollout still cuts what is in flight.** An assistant run is interrupted and resumed with a + note; a terminal is not resumable and says so; and the web bundle's content-hashed chunks are + per-image, so a browser holding the old shell can 404 on a lazy import during the rollout window. +- **Validation is not admission.** These manifests satisfy the `restricted` Pod Security Standard + that [`00-namespace.yaml`](00-namespace.yaml) enforces. If you edit them, note that a schema + validator cannot see that policy: a missing `seccompProfile` or a container without a + `securityContext` does not fail validation, it produces a Deployment reporting `FailedCreate` and + sitting at zero replicas. + +## Backups + +Everything in [the main README](../README.md#backups-and-restoring-one) applies, with one +substitution: the vault key is not in a Docker volume, it is in the `jarvis-generated` Secret. + +```sh +kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d +``` + +Back that up somewhere that is not this cluster, and not alongside your database dump. A database +backup that travelled with its own key is a backup that decrypts itself.