jarvis-selfhost/kubernetes/README.md

# Jarvis on Kubernetes

Plain manifests, no Helm, no operator. They are written to be read as much as applied: every value
that is not a Kubernetes default is there because something in Jarvis behaves a particular way, and
the comment beside it says which. Read them before you apply them — that is the point of shipping
them this way.

**You probably do not need this.** [Docker Compose](../README.md#install) runs one of everything,
takes one command, and is what most instances run. Come here when you want Jarvis to keep serving
while a node reboots, and when you already operate a cluster — this directory assumes you have
opinions about ingress, storage and a database, because it deliberately does not have them for you.

> **What has been proven, and what has not.** This stack has been brought up on a real cluster —
> four nodes, three Postgres instances, three Redis with three sentinels, two api pods — and kept
> serving while its Postgres primary and its Redis primary were killed underneath it. What has NOT
> happened is a long-running estate serving production traffic this way. If you are early, watch it
> rather than assume it, and read the three warnings below before you apply anything, because each
> of them fails silently.

## What runs more than once, and what cannot

"Highly available" is not a property a deployment has as a whole, so here it is component by
component.

| component | replicas | why |
| --- | --- | --- |
| `jarvis-web` | any number | stateless nginx over static files |
| `jarvis-api` | any number | the coordination lives in Postgres and Redis, not in the process |
| Redis | one **logical** primary, three processes | Sentinel fails over; it does not spread writes |
| Postgres | one **primary**, replicas behind it | and Jarvis depends on there being exactly one |
| `jarvis-migrate` | one at a time | enforced by a Postgres advisory lock, not by you |
| `jarvis-mint-secrets` | once, ever | enforced by the API server |

The single-writer database is not a limitation Jarvis has failed to overcome — it is what makes the
multi-pod api safe. A partial unique index is what stops two pods running the same assistant loop,
and an index can only do that because one server decides. Do not point pods at read replicas.

## What you need before you apply anything

- **x86-64 nodes.** The api and web images are published for `linux/amd64` only, so an arm64 pool
  leaves the pods in `ImagePullBackOff` with no matching manifest. The machines you *administer*
  have no such limit — the agent ships arm64 builds.
- **A cluster you can schedule on.** `kubectl describe node | grep Taints` first. A three-node
  cluster is often three control-plane nodes, and their `NoSchedule` taint means nothing here
  schedules at all. These manifests carry no toleration on purpose: adding one is a decision about
  your cluster.
- **Postgres 16, highly available, outside these manifests.** Use an operator — CloudNativePG or
  Zalando's — rather than a hand-written StatefulSet. A database you hand-rolled is a database you
  hand-restore. `max_connections` must cover
  `(replicas + maxSurge + jobs) × connection_limit + reserve`: with 3 api pods at 10 connections
  each, a migration Job at 5, and headroom, about **150**. The stock Postgres image ships 100, and
  exhausting it presents as an api that starts and then cannot serve.
- **Redis.** [`08-redis-sentinel.yaml`](08-redis-sentinel.yaml) deploys one — three Redis, one of
  them the primary, and three sentinels — and `02-config.yaml`'s two `REDIS_SENTINEL*` lines point
  the api at it. Or use a managed Redis that fails over behind a single address: skip that file,
  delete those two lines, and set `REDIS_URL` to it. Never `REDIS_URL` alone at the file's `redis`
  Service: it is headless, two of the three pods it answers with are read-only replicas, and an api
  pod that draws one fails every write. Persistence is not required: everything Jarvis keeps in
  Redis is ephemeral by design, which is exactly why the run lock is in Postgres instead.
- **An Ingress controller and a TLS terminator.** Jarvis speaks plain HTTP.

## Order

```sh
kubectl apply -f 00-namespace.yaml

# The secrets, ONCE. Never regenerate them on an instance that has stored anything.
kubectl apply -f 01-secret.example.yaml   # after editing — the half you write
kubectl apply -f 01-secret-job.yaml       # the half that is random bytes (optional; see the file)
kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m

kubectl apply -f 02-config.yaml
kubectl apply -f 08-redis-sentinel.yaml   # unless you use a managed Redis (then edit 02-config.yaml)

# The schema, to completion, before any api pod starts.
kubectl apply -f 03-migrate-job.yaml
kubectl wait --for=condition=complete job/jarvis-migrate -n jarvis --timeout=15m

kubectl apply -f 04-api.yaml
kubectl apply -f 05-web.yaml
kubectl apply -f 06-ingress.yaml
kubectl apply -f 06b-ingress-uploads.yaml   # uploads alone get a ~13 MiB body ceiling
kubectl apply -f 07-disruption.yaml
```

Every image says `REPLACE_ME`: pin them by digest, not by tag. `:stable` is a moving name, so
replicas that restart at different times land on different builds and a rollback becomes "hope the
tag still points where it did".

```sh
docker buildx imagetools inspect git.luxit.be/luxit/jarvis-api:stable
```

On every upgrade the migration Job runs again before the Deployments roll. It is idempotent and
takes a Postgres advisory lock, so running it twice is slow rather than harmful — but it must
finish before new pods start, which is what the `kubectl wait` is for. With Helm or Argo this
becomes a `pre-upgrade` hook or a sync-wave; as plain YAML it is an ordering you keep yourself.

**Then open the address and install it.** Everything in [the main README](../README.md#install)
about the install screen applies unchanged — including that **whoever reaches an uninstalled
instance first owns it**, with no deadline. Point a public hostname at this after you have
installed, not before.

## GitOps: Argo CD, Flux, or anything that reconciles

`kustomization.yaml` is the entry point. Point a controller at this directory and it renders every
manifest here except one — and `argocd-application.example.yaml` is a worked Application to copy.

```sh
kubectl apply -k .    # the same thing, by hand
```

**The exception is `01-secret.example.yaml`, and it is left out on purpose.** It is a template full
of placeholders. A controller applies what it is pointed at, continuously, without asking — so a
Secret full of `REPLACE_ME` inside the source of truth is a controller that overwrites your real
credentials with placeholders on its next reconcile and puts them back every time you fix them. The
api cannot reach its database, the vault key is wrong, and nothing about the sync looks unhealthy
while it happens. Write `jarvis-secrets` by hand, or through SealedSecrets, ExternalSecrets or
Vault, whose ciphertext is safe to commit. `jarvis-generated` is minted in-cluster and never exists
in git at all.

### The order is in the manifests, not in the filenames

A controller reads this directory as a set; `00-`, `01-` and `02-` mean nothing to it. The order
that the `kubectl apply` sequence above keeps by hand is carried by
`argocd.argoproj.io/sync-wave` annotations on the resources themselves, so that it lives beside the
thing being ordered:

| wave | what | why it cannot be later |
| --- | --- | --- |
| `-3` | Namespace | it carries the Pod Security labels that decide whether any of these pods are admitted |
| `-2` | ConfigMaps, the minter's RBAC, Redis | everything below reads the config; the api's readiness probe reaches Redis |
| `-1` | `jarvis-mint-secrets` **(hook)** | the migration reads `DATABASE_URL` out of what it writes |
| `0` | `jarvis-migrate` **(hook)** | an api pod against an unmigrated schema is the failure all of this prevents |
| `1` | api and web Deployments, Services | the web pod's `/readyz` proxies the api's health endpoint, so it cannot be ready before the api is |
| `2` | Ingress, PodDisruptionBudgets, the web's HPA | an Ingress applied early is a route to a 503; the HPA, below |

Argo does not begin a wave until the previous one reports healthy, which is what the `kubectl wait`
lines are doing by hand. The last wave is applied and not waited on, which is why the web's
HorizontalPodAutoscaler is there rather than beside its Deployment: on a first install it has no
metric for pods seconds old, Argo reads that as Degraded, and in wave 1 it failed the first sync
every time. The application can read Degraded for the minute after a first install, until
metrics-server has scraped the new pods; the sync itself has succeeded.

### Why the two Jobs are hooks

Without this, the directory works exactly once. **A Job's `spec.template` is immutable**, so the
second release — the first one with a different image digest — is a patch the API server refuses
with `field is immutable`, and the sync fails there and stays failed. `hook: Sync` with
`hook-delete-policy: BeforeHookCreation` deletes the previous Job before creating the new one, which
is the supported way to say "run it again, with new contents".

The Sync phase rather than PreSync, deliberately: a PreSync hook runs before any ordinary resource
is applied, which on a first sync is before the namespace, the ConfigMap and the Secret these Jobs
read exist. In the Sync phase they are ordered by the same waves as everything else.

Two consequences worth knowing:

- **The previous run's Job object, and its logs, are gone** by the time the next one starts. Ship
  the logs if you need them after the fact.
- **A migration waiting on the schema lock is ended by its pod, not its Job.** Under Argo,
  deleting the `jarvis-migrate` Job leaves it `Terminating` behind the hook's finalizer, its pod still
  waiting and the sync waiting for good. Delete the pod instead — it stops in about a second — then
  end the sync's operation (`argocd app terminate-op <application>`).
- **Every sync runs the migration.** That is safe — the phase is idempotent and holds a Postgres
  advisory lock — and it is why the example Application leaves `automated` sync OFF. A commit that
  applies a schema change with nobody watching turns the one thing this directory works hardest to
  make visible, a migration that REFUSES because it would drop a column, into a notification.

### Pin the images in one place

`kustomization.yaml` carries an `images:` block, so the three digests are set once rather than in
three files. Three, not two: `jarvis-agent-dist` is an initContainer in `04-api.yaml` and is easy to
miss, and missing it renders cleanly while the api pod never starts.

A version bump is then those lines and nothing else.

### What most estates should actually do

Point the Application at **your** repository, not at this one, with this directory as a remote base:

```yaml
# your-repo/jarvis/kustomization.yaml
resources:
  - https://git.luxit.be/Luxit/jarvis-selfhost//kubernetes?ref=v1.2.3
images:
  - name: git.luxit.be/luxit/jarvis-api
    digest: sha256:...
patches:
  - path: replicas.yaml      # your node count, the host of BOTH Ingresses, your storage class
```

Then the digests you run are in a commit you control, upgrading is a diff you can review and revert,
and `?ref=` pins the base to a tag rather than a branch that moves under you. Pin it.

**There are two Ingresses on the one host, and the host patch must reach both**: `jarvis`, and
`jarvis-uploads` in `06b-ingress-uploads.yaml`, TLS block included. Uploads have an Ingress of their
own because annotations apply to every path of the object that carries them, and they are the one
path that needs bodies over 13 MiB. An overlay that rewrites only the first leaves uploads on the
`/api/` rule, where small files still work and anything between 4 and 13 MB meets the controller's
own 413 — so it looks fine until somebody attaches a large PDF.

To check it once deployed, send an upload of more than 4 MB without signing in, on your own host:

```sh
head -c 5000000 /dev/zero > big.bin
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' -F file=@big.bin \
  https://jarvis.example.com/api/uploads/orgs/x/conversations/y/attachments
```

`401 application/json` is Jarvis turning away an anonymous request, so the body got through. A
`413` with an HTML page is the controller, and means the host patch did not reach `jarvis-uploads`.

## The three things most likely to bite

**Never mint secrets per pod.** On Compose a one-shot generates the vault key and the JWT secrets
into a volume every container shares, and its guarantee that it does so exactly once is that the
filesystem refuses a second create. As a per-pod initContainer on an `emptyDir`, each replica mints
its own — and the consequence is not a crash. Pod A seals a credential under a key pod B does not
have; pod B reports it as corrupt; both log a clean boot. By the time anybody opens a vault entry
there are three keys in circulation.

[`01-secret-job.yaml`](01-secret-job.yaml) is that one-shot done correctly here: it creates the
Secret once and lets the API server's refusal to create it twice be the exclusion. Its identity can
`create` a Secret and cannot `get` one, so it cannot read back what it or anyone else wrote. Mint by
hand instead if you prefer — the one thing that is not an option is per-pod.

Either way, **back the vault key up somewhere that is not this cluster's etcd.** Deleting the Secret
and re-running the Job mints a new key against data sealed with the old one, silently, with no way
back.

**`TRUST_PROXY_HOPS` is one hop here, not two.** The Compose files say 2 because a request passes
your TLS terminator and then the web container's nginx. Here the Ingress routes `/api/` straight at
the api Service, so there is exactly one proxy rewriting `X-Forwarded-For`. Left at 2, the client
address the api records becomes **chosen by the caller**: anyone can send `X-Forwarded-For: 198.51.100.7` and have that
written into session records, audit rows and the key the anonymous WebAuthn budget counts on.

**Do not add session affinity.** It is the reflex the moment websockets are involved and it solves
nothing here: the hard case is co-locating an operator's browser with an *agent's* websocket, which
arrives from a different network at a different time and whose id is not even known at the browser's
handshake. No ingress annotation can express that. It is handled in the application, over Redis.
Both ends speak WebSockets only — the api allows no other transport and the console skips
negotiation — so there is no polling to make sticky in the first place.

## On exactly three nodes

The common case, and where the arithmetic is tightest.

**What lands where.** Redis and the sentinels both use *required* anti-affinity, so three of each
means one per node — six pods, two per node, no choice left to the scheduler. Postgres at three
instances is the same. That is what you want, and it has a consequence: **during a node drain, one
Redis pod and one sentinel have nowhere to go and stay `Pending` until the node returns.** That is
correct — a quorum of two still holds — but `kubectl drain` will sit there unless the
PodDisruptionBudgets are applied to tell it what is safe.

**Three api replicas, not two.** [`04-api.yaml`](04-api.yaml) ships two because that is the smallest
number that proves the code is not single-instance. On three nodes, three is strictly better: one
per node, and losing any node leaves two. Change `replicas: 2` to `3` in it and in
[`05-web.yaml`](05-web.yaml), and size `max_connections` for it.

**Three nodes tolerates losing one.** Not two. That is a property of majority quorums, and the
answer to wanting more is a fourth and fifth node, not a different configuration.

## What a failover actually costs

Measured, not estimated — on the api releases before 1.0.0, which were written in TypeScript. The
readiness and liveness design below is unchanged in 1.0.0; the figures have not been measured again
since.

**Redis**, with its primary killed under a running api:

- **The api rides it out without restarting.** Readiness answered 503 for about fifteen seconds and
  then 200 again, on the same process — the pod leaves the Service's endpoints while it cannot reach
  Redis, and returns when it can. Liveness never touches Redis, which is why the pod is not killed
  and restarted into a cold start.
- **Commands survive.** 99 writes issued across the window, 0 rejected, the last readable from the
  promoted primary afterwards. Commands queue rather than fail, so a failover is a pause.
- **Published messages do not.** Around 45% of the messages published during the window reached
  nobody. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change
  that. In practice: a few seconds in which something happening on one api replica may not reach
  another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason.

**Postgres**, primary deleted with `--force --grace-period=0`:

- **Three seconds.** Both api pods answered `503 {"database":"unreachable"}` and were back to 200 on
  the fourth poll. Both degraded together, which is what pointing every pod at one endpoint means.
- **Zero container restarts**, across two failovers, on pods up throughout — liveness deliberately
  not touching the database is what keeps three seconds from becoming a cold start of everything.

A failover *during* the migration Job aborts it with a lost-connection error, which is the correct
outcome. Run it again.

## What is still worth knowing

- **`AGENT_RELEASE_DIR` is per-pod, and must be.** It holds the compiled agent binaries. An
  `emptyDir` filled by an initContainer is right; a shared RWO volume is not — either every pod is
  pinned to one node, or the second sits in `Multi-Attach error`. Leaving it unset is supported:
  everything works except the agent installer, which answers 503 saying no build is published.
- **No application volume, and it should stay that way.** Avatars, documents, terminal recordings
  and attached files are all in Postgres; exports are rendered in memory.
- **The Ingresses are written for ingress-nginx.** Under another controller their `nginx.ingress…`
  annotations do nothing, so set that controller's own equivalents: an hour's read timeout on the
  two websocket paths, a 4 MB body ceiling on `/api/`, and 13,376 KiB on `/api/uploads/`.
- **Autoscale `jarvis-web`, not `jarvis-api`.** Every api scale-in kills a pod holding agent
  websockets, live runs and open shells, and CPU is a poor proxy for a load made of long-lived
  connections. Raise api replicas deliberately.
- **A rollout still cuts what is in flight.** An assistant run is interrupted and resumed with a
  note; a terminal is not resumable and says so; and the console's files are fingerprinted per
  image, so during the rollout window a tab holding the old console can miss one of the two parts
  it loads on demand (the Markdown parser, the QR code generator). It then shows the Markdown as
  plain text and offers the reload.
- **Validation is not admission.** These manifests satisfy the `restricted` Pod Security Standard
  that [`00-namespace.yaml`](00-namespace.yaml) enforces. If you edit them, note that a schema
  validator cannot see that policy: a missing `seccompProfile` or a container without a
  `securityContext` does not fail validation, it produces a Deployment reporting `FailedCreate` and
  sitting at zero replicas.

## Backups

Everything in [the main README](../README.md#backups-and-restoring-one) applies, with one
substitution: the vault key is not in a Docker volume, it is in a Secret — `jarvis-secrets` if you
set `VAULT_MASTER_KEY` there yourself, `jarvis-generated` otherwise. The kustomization mints
`jarvis-generated` either way, so on the manual path it holds a key nothing uses: back up the one
the api reads.

```sh
kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d
kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_KEY_CREATED_AT}' | base64 -d
```

Back that up somewhere that is not this cluster, and not alongside your database dump. A database
backup that travelled with its own key is a backup that decrypts itself.

The second line is when the key was created, bound to that key by its fingerprint — on a Secret
minted from api 0.114.1 on. An older one carries a bare date, which the api ignores here, so a
restored copy of it changes nothing and the claim asks for the key. If you ever restore the key by
hand into `jarvis-secrets`, restore that value beside it word for word, or leave it out — never
compute a new one. It is what tells the install screen whether an empty database is a
first install or a lost one, and an old key dated today makes the first administrator free to
whoever reaches the address before your dump is loaded.