207 lines
12 KiB
Markdown
207 lines
12 KiB
Markdown
# Jarvis on Kubernetes
|
||
|
||
Plain manifests, no Helm, no operator. They are written to be read as much as applied: every value
|
||
that is not a Kubernetes default is there because something in Jarvis behaves a particular way, and
|
||
the comment beside it says which. Read them before you apply them — that is the point of shipping
|
||
them this way.
|
||
|
||
**You probably do not need this.** [Docker Compose](../README.md#install) runs one of everything,
|
||
takes one command, and is what most instances run. Come here when you want Jarvis to keep serving
|
||
while a node reboots, and when you already operate a cluster — this directory assumes you have
|
||
opinions about ingress, storage and a database, because it deliberately does not have them for you.
|
||
|
||
> **What has been proven, and what has not.** This stack has been brought up on a real cluster —
|
||
> four nodes, three Postgres instances, three Redis with three sentinels, two api pods — and kept
|
||
> serving while its Postgres primary and its Redis primary were killed underneath it. What has NOT
|
||
> happened is a long-running estate serving production traffic this way. If you are early, watch it
|
||
> rather than assume it, and read the three warnings below before you apply anything, because each
|
||
> of them fails silently.
|
||
|
||
## What runs more than once, and what cannot
|
||
|
||
"Highly available" is not a property a deployment has as a whole, so here it is component by
|
||
component.
|
||
|
||
| component | replicas | why |
|
||
| --- | --- | --- |
|
||
| `jarvis-web` | any number | stateless nginx over static files |
|
||
| `jarvis-api` | any number | the coordination lives in Postgres and Redis, not in the process |
|
||
| Redis | one **logical** primary, three processes | Sentinel fails over; it does not spread writes |
|
||
| Postgres | one **primary**, replicas behind it | and Jarvis depends on there being exactly one |
|
||
| `jarvis-migrate` | one at a time | enforced by a Postgres advisory lock, not by you |
|
||
| `jarvis-mint-secrets` | once, ever | enforced by the API server |
|
||
|
||
The single-writer database is not a limitation Jarvis has failed to overcome — it is what makes the
|
||
multi-pod api safe. A partial unique index is what stops two pods running the same assistant loop,
|
||
and an index can only do that because one server decides. Do not point pods at read replicas.
|
||
|
||
## What you need before you apply anything
|
||
|
||
- **x86-64 nodes.** The api and web images are published for `linux/amd64` only, so an arm64 pool
|
||
leaves the pods in `ImagePullBackOff` with no matching manifest. The machines you *administer*
|
||
have no such limit — the agent ships arm64 builds.
|
||
- **A cluster you can schedule on.** `kubectl describe node | grep Taints` first. A three-node
|
||
cluster is often three control-plane nodes, and their `NoSchedule` taint means nothing here
|
||
schedules at all. These manifests carry no toleration on purpose: adding one is a decision about
|
||
your cluster.
|
||
- **Postgres 16, highly available, outside these manifests.** Use an operator — CloudNativePG or
|
||
Zalando's — rather than a hand-written StatefulSet. A database you hand-rolled is a database you
|
||
hand-restore. `max_connections` must cover
|
||
`(replicas + maxSurge + jobs) × connection_limit + reserve`: with 3 api pods at 10 connections
|
||
each, a migration Job at 5, and headroom, about **150**. The stock Postgres image ships 100, and
|
||
exhausting it presents as an api that starts and then cannot serve.
|
||
- **Redis.** [`08-redis-sentinel.yaml`](08-redis-sentinel.yaml) deploys one, or point `REDIS_URL` at
|
||
a managed Redis that fails over behind a single address and skip that file. Persistence is not
|
||
required: everything Jarvis keeps in Redis is ephemeral by design, which is exactly why the run
|
||
lock is in Postgres instead.
|
||
- **An Ingress controller and a TLS terminator.** Jarvis speaks plain HTTP.
|
||
|
||
## Order
|
||
|
||
```sh
|
||
kubectl apply -f 00-namespace.yaml
|
||
|
||
# The secrets, ONCE. Never regenerate them on an instance that has stored anything.
|
||
kubectl apply -f 01-secret.example.yaml # after editing — the half you write
|
||
kubectl apply -f 01-secret-job.yaml # the half that is random bytes (optional; see the file)
|
||
kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m
|
||
|
||
kubectl apply -f 02-config.yaml
|
||
kubectl apply -f 08-redis-sentinel.yaml # only if you are running your own Redis
|
||
|
||
# The schema, to completion, before any api pod starts.
|
||
kubectl apply -f 03-migrate-job.yaml
|
||
kubectl wait --for=condition=complete job/jarvis-migrate -n jarvis --timeout=15m
|
||
|
||
kubectl apply -f 04-api.yaml
|
||
kubectl apply -f 05-web.yaml
|
||
kubectl apply -f 06-ingress.yaml
|
||
kubectl apply -f 07-disruption.yaml
|
||
```
|
||
|
||
Every image says `REPLACE_ME`: pin them by digest, not by tag. `:stable` is a moving name, so
|
||
replicas that restart at different times land on different builds and a rollback becomes "hope the
|
||
tag still points where it did".
|
||
|
||
```sh
|
||
docker buildx imagetools inspect git.luxit.be/luxit/jarvis-api:stable
|
||
```
|
||
|
||
On every upgrade the migration Job runs again before the Deployments roll. It is idempotent and
|
||
takes a Postgres advisory lock, so running it twice is slow rather than harmful — but it must
|
||
finish before new pods start, which is what the `kubectl wait` is for. With Helm or Argo this
|
||
becomes a `pre-upgrade` hook or a sync-wave; as plain YAML it is an ordering you keep yourself.
|
||
|
||
**Then open the address and install it.** Everything in [the main README](../README.md#install)
|
||
about the install screen applies unchanged — including that **whoever reaches an uninstalled
|
||
instance first owns it**, with no deadline. Point a public hostname at this after you have
|
||
installed, not before.
|
||
|
||
## The three things most likely to bite
|
||
|
||
**Never mint secrets per pod.** On Compose a one-shot generates the vault key and the JWT secrets
|
||
into a volume every container shares, and its guarantee that it does so exactly once is that the
|
||
filesystem refuses a second create. As a per-pod initContainer on an `emptyDir`, each replica mints
|
||
its own — and the consequence is not a crash. Pod A seals a credential under a key pod B does not
|
||
have; pod B reports it as corrupt; both log a clean boot. By the time anybody opens a vault entry
|
||
there are three keys in circulation.
|
||
|
||
[`01-secret-job.yaml`](01-secret-job.yaml) is that one-shot done correctly here: it creates the
|
||
Secret once and lets the API server's refusal to create it twice be the exclusion. Its identity can
|
||
`create` a Secret and cannot `get` one, so it cannot read back what it or anyone else wrote. Mint by
|
||
hand instead if you prefer — the one thing that is not an option is per-pod.
|
||
|
||
Either way, **back the vault key up somewhere that is not this cluster's etcd.** Deleting the Secret
|
||
and re-running the Job mints a new key against data sealed with the old one, silently, with no way
|
||
back.
|
||
|
||
**`TRUST_PROXY_HOPS` is one hop here, not two.** The Compose files say 2 because a request passes
|
||
your TLS terminator and then the web container's nginx. Here the Ingress routes `/api/` straight at
|
||
the api Service, so there is exactly one proxy rewriting `X-Forwarded-For`. Left at 2, `req.ip`
|
||
becomes **chosen by the caller**: anyone can send `X-Forwarded-For: 198.51.100.7` and have that
|
||
written into session records, audit rows and the key the anonymous WebAuthn budget counts on.
|
||
|
||
**Do not add session affinity.** It is the reflex the moment websockets are involved and it solves
|
||
nothing here: the hard case is co-locating an operator's browser with an *agent's* websocket, which
|
||
arrives from a different network at a different time and whose id is not even known at the browser's
|
||
handshake. No ingress annotation can express that. It is handled in the application, over Redis.
|
||
Both ends pin `transports: ["websocket"]`, so there is no polling to make sticky in the first place.
|
||
|
||
## On exactly three nodes
|
||
|
||
The common case, and where the arithmetic is tightest.
|
||
|
||
**What lands where.** Redis and the sentinels both use *required* anti-affinity, so three of each
|
||
means one per node — six pods, two per node, no choice left to the scheduler. Postgres at three
|
||
instances is the same. That is what you want, and it has a consequence: **during a node drain, one
|
||
Redis pod and one sentinel have nowhere to go and stay `Pending` until the node returns.** That is
|
||
correct — a quorum of two still holds — but `kubectl drain` will sit there unless the
|
||
PodDisruptionBudgets are applied to tell it what is safe.
|
||
|
||
**Three api replicas, not two.** [`04-api.yaml`](04-api.yaml) ships two because that is the smallest
|
||
number that proves the code is not single-instance. On three nodes, three is strictly better: one
|
||
per node, and losing any node leaves two. Change `replicas: 2` to `3` in it and in
|
||
[`05-web.yaml`](05-web.yaml), and size `max_connections` for it.
|
||
|
||
**Three nodes tolerates losing one.** Not two. That is a property of majority quorums, and the
|
||
answer to wanting more is a fourth and fifth node, not a different configuration.
|
||
|
||
## What a failover actually costs
|
||
|
||
Measured, not estimated.
|
||
|
||
**Redis**, with its primary killed under a running api:
|
||
|
||
- **The api rides it out without restarting.** Readiness answered 503 for about fifteen seconds and
|
||
then 200 again, on the same process — the pod leaves the Service's endpoints while it cannot reach
|
||
Redis, and returns when it can. Liveness never touches Redis, which is why the pod is not killed
|
||
and restarted into a cold start.
|
||
- **Commands survive.** 99 writes issued across the window, 0 rejected, the last readable from the
|
||
promoted primary afterwards. Commands queue rather than fail, so a failover is a pause.
|
||
- **Published messages do not.** Around 45% of the messages published during the window reached
|
||
nobody. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change
|
||
that. In practice: a few seconds in which something happening on one api replica may not reach
|
||
another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason.
|
||
|
||
**Postgres**, primary deleted with `--force --grace-period=0`:
|
||
|
||
- **Three seconds.** Both api pods answered `503 {"database":"unreachable"}` and were back to 200 on
|
||
the fourth poll. Both degraded together, which is what pointing every pod at one endpoint means.
|
||
- **Zero container restarts**, across two failovers, on pods up throughout — liveness deliberately
|
||
not touching the database is what keeps three seconds from becoming a cold start of everything.
|
||
|
||
A failover *during* the migration Job aborts it with a lost-connection error, which is the correct
|
||
outcome. Run it again.
|
||
|
||
## What is still worth knowing
|
||
|
||
- **`AGENT_RELEASE_DIR` is per-pod, and must be.** It holds the compiled agent binaries. An
|
||
`emptyDir` filled by an initContainer is right; a shared RWO volume is not — either every pod is
|
||
pinned to one node, or the second sits in `Multi-Attach error`. Leaving it unset is supported:
|
||
everything works except the agent installer, which answers 503 saying no build is published.
|
||
- **No application volume, and it should stay that way.** Avatars, documents and terminal recordings
|
||
are all in Postgres; exports are rendered in memory.
|
||
- **Autoscale `jarvis-web`, not `jarvis-api`.** Every api scale-in kills a pod holding agent
|
||
websockets, live runs and open shells, and CPU is a poor proxy for a load made of long-lived
|
||
connections. Raise api replicas deliberately.
|
||
- **A rollout still cuts what is in flight.** An assistant run is interrupted and resumed with a
|
||
note; a terminal is not resumable and says so; and the web bundle's content-hashed chunks are
|
||
per-image, so a browser holding the old shell can 404 on a lazy import during the rollout window.
|
||
- **Validation is not admission.** These manifests satisfy the `restricted` Pod Security Standard
|
||
that [`00-namespace.yaml`](00-namespace.yaml) enforces. If you edit them, note that a schema
|
||
validator cannot see that policy: a missing `seccompProfile` or a container without a
|
||
`securityContext` does not fail validation, it produces a Deployment reporting `FailedCreate` and
|
||
sitting at zero replicas.
|
||
|
||
## Backups
|
||
|
||
Everything in [the main README](../README.md#backups-and-restoring-one) applies, with one
|
||
substitution: the vault key is not in a Docker volume, it is in the `jarvis-generated` Secret.
|
||
|
||
```sh
|
||
kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d
|
||
```
|
||
|
||
Back that up somewhere that is not this cluster, and not alongside your database dump. A database
|
||
backup that travelled with its own key is a backup that decrypts itself.
|