303 lines
18 KiB
Markdown
303 lines
18 KiB
Markdown
# Jarvis on Kubernetes
|
||
|
||
Plain manifests, no Helm, no operator. They are written to be read as much as applied: every value
|
||
that is not a Kubernetes default is there because something in Jarvis behaves a particular way, and
|
||
the comment beside it says which. Read them before you apply them — that is the point of shipping
|
||
them this way.
|
||
|
||
**You probably do not need this.** [Docker Compose](../README.md#install) runs one of everything,
|
||
takes one command, and is what most instances run. Come here when you want Jarvis to keep serving
|
||
while a node reboots, and when you already operate a cluster — this directory assumes you have
|
||
opinions about ingress, storage and a database, because it deliberately does not have them for you.
|
||
|
||
> **What has been proven, and what has not.** This stack has been brought up on a real cluster —
|
||
> four nodes, three Postgres instances, three Redis with three sentinels, two api pods — and kept
|
||
> serving while its Postgres primary and its Redis primary were killed underneath it. What has NOT
|
||
> happened is a long-running estate serving production traffic this way. If you are early, watch it
|
||
> rather than assume it, and read the three warnings below before you apply anything, because each
|
||
> of them fails silently.
|
||
|
||
## What runs more than once, and what cannot
|
||
|
||
"Highly available" is not a property a deployment has as a whole, so here it is component by
|
||
component.
|
||
|
||
| component | replicas | why |
|
||
| --- | --- | --- |
|
||
| `jarvis-web` | any number | stateless nginx over static files |
|
||
| `jarvis-api` | any number | the coordination lives in Postgres and Redis, not in the process |
|
||
| Redis | one **logical** primary, three processes | Sentinel fails over; it does not spread writes |
|
||
| Postgres | one **primary**, replicas behind it | and Jarvis depends on there being exactly one |
|
||
| `jarvis-migrate` | one at a time | enforced by a Postgres advisory lock, not by you |
|
||
| `jarvis-mint-secrets` | once, ever | enforced by the API server |
|
||
|
||
The single-writer database is not a limitation Jarvis has failed to overcome — it is what makes the
|
||
multi-pod api safe. A partial unique index is what stops two pods running the same assistant loop,
|
||
and an index can only do that because one server decides. Do not point pods at read replicas.
|
||
|
||
## What you need before you apply anything
|
||
|
||
- **x86-64 nodes.** The api and web images are published for `linux/amd64` only, so an arm64 pool
|
||
leaves the pods in `ImagePullBackOff` with no matching manifest. The machines you *administer*
|
||
have no such limit — the agent ships arm64 builds.
|
||
- **A cluster you can schedule on.** `kubectl describe node | grep Taints` first. A three-node
|
||
cluster is often three control-plane nodes, and their `NoSchedule` taint means nothing here
|
||
schedules at all. These manifests carry no toleration on purpose: adding one is a decision about
|
||
your cluster.
|
||
- **Postgres 16, highly available, outside these manifests.** Use an operator — CloudNativePG or
|
||
Zalando's — rather than a hand-written StatefulSet. A database you hand-rolled is a database you
|
||
hand-restore. `max_connections` must cover
|
||
`(replicas + maxSurge + jobs) × connection_limit + reserve`: with 3 api pods at 10 connections
|
||
each, a migration Job at 5, and headroom, about **150**. The stock Postgres image ships 100, and
|
||
exhausting it presents as an api that starts and then cannot serve.
|
||
- **Redis.** [`08-redis-sentinel.yaml`](08-redis-sentinel.yaml) deploys one, or point `REDIS_URL` at
|
||
a managed Redis that fails over behind a single address and skip that file. Persistence is not
|
||
required: everything Jarvis keeps in Redis is ephemeral by design, which is exactly why the run
|
||
lock is in Postgres instead.
|
||
- **An Ingress controller and a TLS terminator.** Jarvis speaks plain HTTP.
|
||
|
||
## Order
|
||
|
||
```sh
|
||
kubectl apply -f 00-namespace.yaml
|
||
|
||
# The secrets, ONCE. Never regenerate them on an instance that has stored anything.
|
||
kubectl apply -f 01-secret.example.yaml # after editing — the half you write
|
||
kubectl apply -f 01-secret-job.yaml # the half that is random bytes (optional; see the file)
|
||
kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m
|
||
|
||
kubectl apply -f 02-config.yaml
|
||
kubectl apply -f 08-redis-sentinel.yaml # only if you are running your own Redis
|
||
|
||
# The schema, to completion, before any api pod starts.
|
||
kubectl apply -f 03-migrate-job.yaml
|
||
kubectl wait --for=condition=complete job/jarvis-migrate -n jarvis --timeout=15m
|
||
|
||
kubectl apply -f 04-api.yaml
|
||
kubectl apply -f 05-web.yaml
|
||
kubectl apply -f 06-ingress.yaml
|
||
kubectl apply -f 07-disruption.yaml
|
||
```
|
||
|
||
Every image says `REPLACE_ME`: pin them by digest, not by tag. `:stable` is a moving name, so
|
||
replicas that restart at different times land on different builds and a rollback becomes "hope the
|
||
tag still points where it did".
|
||
|
||
```sh
|
||
docker buildx imagetools inspect git.luxit.be/luxit/jarvis-api:stable
|
||
```
|
||
|
||
On every upgrade the migration Job runs again before the Deployments roll. It is idempotent and
|
||
takes a Postgres advisory lock, so running it twice is slow rather than harmful — but it must
|
||
finish before new pods start, which is what the `kubectl wait` is for. With Helm or Argo this
|
||
becomes a `pre-upgrade` hook or a sync-wave; as plain YAML it is an ordering you keep yourself.
|
||
|
||
**Then open the address and install it.** Everything in [the main README](../README.md#install)
|
||
about the install screen applies unchanged — including that **whoever reaches an uninstalled
|
||
instance first owns it**, with no deadline. Point a public hostname at this after you have
|
||
installed, not before.
|
||
|
||
## GitOps: Argo CD, Flux, or anything that reconciles
|
||
|
||
`kustomization.yaml` is the entry point. Point a controller at this directory and it renders every
|
||
manifest here except one — and `argocd-application.example.yaml` is a worked Application to copy.
|
||
|
||
```sh
|
||
kubectl apply -k . # the same thing, by hand
|
||
```
|
||
|
||
**The exception is `01-secret.example.yaml`, and it is left out on purpose.** It is a template full
|
||
of placeholders. A controller applies what it is pointed at, continuously, without asking — so a
|
||
Secret full of `REPLACE_ME` inside the source of truth is a controller that overwrites your real
|
||
credentials with placeholders on its next reconcile and puts them back every time you fix them. The
|
||
api cannot reach its database, the vault key is wrong, and nothing about the sync looks unhealthy
|
||
while it happens. Write `jarvis-secrets` by hand, or through SealedSecrets, ExternalSecrets or
|
||
Vault, whose ciphertext is safe to commit. `jarvis-generated` is minted in-cluster and never exists
|
||
in git at all.
|
||
|
||
### The order is in the manifests, not in the filenames
|
||
|
||
A controller reads this directory as a set; `00-`, `01-` and `02-` mean nothing to it. The order
|
||
that the `kubectl apply` sequence above keeps by hand is carried by
|
||
`argocd.argoproj.io/sync-wave` annotations on the resources themselves, so that it lives beside the
|
||
thing being ordered:
|
||
|
||
| wave | what | why it cannot be later |
|
||
| --- | --- | --- |
|
||
| `-3` | Namespace | it carries the Pod Security labels that decide whether any of these pods are admitted |
|
||
| `-2` | ConfigMaps, the minter's RBAC, Redis | everything below reads the config; the api's readiness probe reaches Redis |
|
||
| `-1` | `jarvis-mint-secrets` **(hook)** | the migration reads `DATABASE_URL` out of what it writes |
|
||
| `0` | `jarvis-migrate` **(hook)** | an api pod against an unmigrated schema is the failure all of this prevents |
|
||
| `1` | api and web Deployments, Services | the web pod's `/readyz` proxies the api's health endpoint, so it cannot be ready before the api is |
|
||
| `2` | Ingress, PodDisruptionBudgets | an Ingress applied early is a route to a 503 |
|
||
|
||
Argo does not begin a wave until the previous one reports healthy, which is what the `kubectl wait`
|
||
lines are doing by hand.
|
||
|
||
### Why the two Jobs are hooks
|
||
|
||
Without this, the directory works exactly once. **A Job's `spec.template` is immutable**, so the
|
||
second release — the first one with a different image digest — is a patch the API server refuses
|
||
with `field is immutable`, and the sync fails there and stays failed. `hook: Sync` with
|
||
`hook-delete-policy: BeforeHookCreation` deletes the previous Job before creating the new one, which
|
||
is the supported way to say "run it again, with new contents".
|
||
|
||
The Sync phase rather than PreSync, deliberately: a PreSync hook runs before any ordinary resource
|
||
is applied, which on a first sync is before the namespace, the ConfigMap and the Secret these Jobs
|
||
read exist. In the Sync phase they are ordered by the same waves as everything else.
|
||
|
||
Two consequences worth knowing:
|
||
|
||
- **The previous run's Job object, and its logs, are gone** by the time the next one starts. Ship
|
||
the logs if you need them after the fact.
|
||
- **Every sync runs the migration.** That is safe — the phase is idempotent and holds a Postgres
|
||
advisory lock — and it is why the example Application leaves `automated` sync OFF. A commit that
|
||
applies a schema change with nobody watching turns the one thing this directory works hardest to
|
||
make visible, a migration that REFUSES because it would drop a column, into a notification.
|
||
|
||
### Pin the images in one place
|
||
|
||
`kustomization.yaml` carries an `images:` block, so the three digests are set once rather than in
|
||
three files. Three, not two: `jarvis-agent-dist` is an initContainer in `04-api.yaml` and is easy to
|
||
miss, and missing it renders cleanly while the api pod never starts.
|
||
|
||
A version bump is then those lines and nothing else.
|
||
|
||
### What most estates should actually do
|
||
|
||
Point the Application at **your** repository, not at this one, with this directory as a remote base:
|
||
|
||
```yaml
|
||
# your-repo/jarvis/kustomization.yaml
|
||
resources:
|
||
- https://git.luxit.be/Luxit/jarvis-selfhost//kubernetes?ref=v1.2.3
|
||
images:
|
||
- name: git.luxit.be/luxit/jarvis-api
|
||
digest: sha256:...
|
||
patches:
|
||
- path: replicas.yaml # your node count, your ingress host, your storage class
|
||
```
|
||
|
||
Then the digests you run are in a commit you control, upgrading is a diff you can review and revert,
|
||
and `?ref=` pins the base to a tag rather than a branch that moves under you. Pin it.
|
||
|
||
## The three things most likely to bite
|
||
|
||
**Never mint secrets per pod.** On Compose a one-shot generates the vault key and the JWT secrets
|
||
into a volume every container shares, and its guarantee that it does so exactly once is that the
|
||
filesystem refuses a second create. As a per-pod initContainer on an `emptyDir`, each replica mints
|
||
its own — and the consequence is not a crash. Pod A seals a credential under a key pod B does not
|
||
have; pod B reports it as corrupt; both log a clean boot. By the time anybody opens a vault entry
|
||
there are three keys in circulation.
|
||
|
||
[`01-secret-job.yaml`](01-secret-job.yaml) is that one-shot done correctly here: it creates the
|
||
Secret once and lets the API server's refusal to create it twice be the exclusion. Its identity can
|
||
`create` a Secret and cannot `get` one, so it cannot read back what it or anyone else wrote. Mint by
|
||
hand instead if you prefer — the one thing that is not an option is per-pod.
|
||
|
||
Either way, **back the vault key up somewhere that is not this cluster's etcd.** Deleting the Secret
|
||
and re-running the Job mints a new key against data sealed with the old one, silently, with no way
|
||
back.
|
||
|
||
**`TRUST_PROXY_HOPS` is one hop here, not two.** The Compose files say 2 because a request passes
|
||
your TLS terminator and then the web container's nginx. Here the Ingress routes `/api/` straight at
|
||
the api Service, so there is exactly one proxy rewriting `X-Forwarded-For`. Left at 2, `req.ip`
|
||
becomes **chosen by the caller**: anyone can send `X-Forwarded-For: 198.51.100.7` and have that
|
||
written into session records, audit rows and the key the anonymous WebAuthn budget counts on.
|
||
|
||
**Do not add session affinity.** It is the reflex the moment websockets are involved and it solves
|
||
nothing here: the hard case is co-locating an operator's browser with an *agent's* websocket, which
|
||
arrives from a different network at a different time and whose id is not even known at the browser's
|
||
handshake. No ingress annotation can express that. It is handled in the application, over Redis.
|
||
Both ends pin `transports: ["websocket"]`, so there is no polling to make sticky in the first place.
|
||
|
||
## On exactly three nodes
|
||
|
||
The common case, and where the arithmetic is tightest.
|
||
|
||
**What lands where.** Redis and the sentinels both use *required* anti-affinity, so three of each
|
||
means one per node — six pods, two per node, no choice left to the scheduler. Postgres at three
|
||
instances is the same. That is what you want, and it has a consequence: **during a node drain, one
|
||
Redis pod and one sentinel have nowhere to go and stay `Pending` until the node returns.** That is
|
||
correct — a quorum of two still holds — but `kubectl drain` will sit there unless the
|
||
PodDisruptionBudgets are applied to tell it what is safe.
|
||
|
||
**Three api replicas, not two.** [`04-api.yaml`](04-api.yaml) ships two because that is the smallest
|
||
number that proves the code is not single-instance. On three nodes, three is strictly better: one
|
||
per node, and losing any node leaves two. Change `replicas: 2` to `3` in it and in
|
||
[`05-web.yaml`](05-web.yaml), and size `max_connections` for it.
|
||
|
||
**Three nodes tolerates losing one.** Not two. That is a property of majority quorums, and the
|
||
answer to wanting more is a fourth and fifth node, not a different configuration.
|
||
|
||
## What a failover actually costs
|
||
|
||
Measured, not estimated.
|
||
|
||
**Redis**, with its primary killed under a running api:
|
||
|
||
- **The api rides it out without restarting.** Readiness answered 503 for about fifteen seconds and
|
||
then 200 again, on the same process — the pod leaves the Service's endpoints while it cannot reach
|
||
Redis, and returns when it can. Liveness never touches Redis, which is why the pod is not killed
|
||
and restarted into a cold start.
|
||
- **Commands survive.** 99 writes issued across the window, 0 rejected, the last readable from the
|
||
promoted primary afterwards. Commands queue rather than fail, so a failover is a pause.
|
||
- **Published messages do not.** Around 45% of the messages published during the window reached
|
||
nobody. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change
|
||
that. In practice: a few seconds in which something happening on one api replica may not reach
|
||
another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason.
|
||
|
||
**Postgres**, primary deleted with `--force --grace-period=0`:
|
||
|
||
- **Three seconds.** Both api pods answered `503 {"database":"unreachable"}` and were back to 200 on
|
||
the fourth poll. Both degraded together, which is what pointing every pod at one endpoint means.
|
||
- **Zero container restarts**, across two failovers, on pods up throughout — liveness deliberately
|
||
not touching the database is what keeps three seconds from becoming a cold start of everything.
|
||
|
||
A failover *during* the migration Job aborts it with a lost-connection error, which is the correct
|
||
outcome. Run it again.
|
||
|
||
## What is still worth knowing
|
||
|
||
- **`AGENT_RELEASE_DIR` is per-pod, and must be.** It holds the compiled agent binaries. An
|
||
`emptyDir` filled by an initContainer is right; a shared RWO volume is not — either every pod is
|
||
pinned to one node, or the second sits in `Multi-Attach error`. Leaving it unset is supported:
|
||
everything works except the agent installer, which answers 503 saying no build is published.
|
||
- **No application volume, and it should stay that way.** Avatars, documents and terminal recordings
|
||
are all in Postgres; exports are rendered in memory.
|
||
- **Autoscale `jarvis-web`, not `jarvis-api`.** Every api scale-in kills a pod holding agent
|
||
websockets, live runs and open shells, and CPU is a poor proxy for a load made of long-lived
|
||
connections. Raise api replicas deliberately.
|
||
- **A rollout still cuts what is in flight.** An assistant run is interrupted and resumed with a
|
||
note; a terminal is not resumable and says so; and the web bundle's content-hashed chunks are
|
||
per-image, so a browser holding the old shell can 404 on a lazy import during the rollout window.
|
||
- **Validation is not admission.** These manifests satisfy the `restricted` Pod Security Standard
|
||
that [`00-namespace.yaml`](00-namespace.yaml) enforces. If you edit them, note that a schema
|
||
validator cannot see that policy: a missing `seccompProfile` or a container without a
|
||
`securityContext` does not fail validation, it produces a Deployment reporting `FailedCreate` and
|
||
sitting at zero replicas.
|
||
|
||
## Backups
|
||
|
||
Everything in [the main README](../README.md#backups-and-restoring-one) applies, with one
|
||
substitution: the vault key is not in a Docker volume, it is in a Secret — `jarvis-secrets` if you
|
||
set `VAULT_MASTER_KEY` there yourself, `jarvis-generated` otherwise. The kustomization mints
|
||
`jarvis-generated` either way, so on the manual path it holds a key nothing uses: back up the one
|
||
the api reads.
|
||
|
||
```sh
|
||
kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d
|
||
kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_KEY_CREATED_AT}' | base64 -d
|
||
```
|
||
|
||
Back that up somewhere that is not this cluster, and not alongside your database dump. A database
|
||
backup that travelled with its own key is a backup that decrypts itself.
|
||
|
||
The second line is when the key was created, bound to that key by its fingerprint — on a Secret
|
||
minted from api 0.114.1 on. An older one carries a bare date, which the api ignores here, so a
|
||
restored copy of it changes nothing and the claim asks for the key. If you ever restore the key by
|
||
hand into `jarvis-secrets`, restore that value beside it word for word, or leave it out — never
|
||
compute a new one. It is what tells the install screen whether an empty database is a
|
||
first install or a lost one, and an old key dated today makes the first administrator free to
|
||
whoever reaches the address before your dump is loaded.
|