Sync the self-hosting stack
This commit is contained in:
@@ -162,6 +162,26 @@ before repeating anything that writes. This is why the api asks for a stop grace
|
||||
your instance, signs each session with a key whose private half never leaves it, and reports its
|
||||
inventory on every heartbeat.
|
||||
|
||||
## Two ways to run it
|
||||
|
||||
**Docker Compose — one of everything.** One api container, one web container, one Postgres, one
|
||||
Redis, on one host. Two commands and about a minute. This is what most instances run and what the
|
||||
rest of this README describes; if you are not sure which you want, it is this one. A reboot of that
|
||||
host is downtime, and for the overwhelming majority of deployments that is a fine trade.
|
||||
|
||||
**Kubernetes — several of everything that can be several.** Several api pods and several web pods
|
||||
behind one address, a Postgres with a primary and replicas, a Redis that fails over. A node can
|
||||
reboot and the instance keeps serving. It costs you a cluster to operate and a database to run
|
||||
properly, and it asks you to make decisions Compose makes for you.
|
||||
|
||||
Everything below is the Compose path. For the other one, the manifests and their own guide are in
|
||||
[`kubernetes/`](kubernetes/README.md) — start there rather than here, then come back for the
|
||||
install screen, the reverse proxy requirements and the backup section, which are the same either
|
||||
way.
|
||||
|
||||
Moving between them later is a database dump and the vault key; nothing about your data is specific
|
||||
to how the containers are scheduled.
|
||||
|
||||
## What you need
|
||||
|
||||
- Docker with Compose v2, on **x86-64 Linux**. The api and web images are published for
|
||||
@@ -276,6 +296,11 @@ header a caller can forge. Change it later under **Settings → Platform**.
|
||||
docker compose pull && docker compose up -d
|
||||
```
|
||||
|
||||
> **On Kubernetes** the shape is different, because the schema is a Job rather than something the
|
||||
> api does on its way up: the migration runs to completion first, then the Deployments roll. See
|
||||
> [`kubernetes/README.md`](kubernetes/README.md#order). Everything below about channels applies
|
||||
> unchanged, except that you pin images by digest rather than tracking a channel tag.
|
||||
|
||||
That is the whole upgrade: the api, the web, and — with the agent overlay on — the agent release all
|
||||
track the **`stable`** channel by default. Postgres and Redis are not on a Jarvis channel; they
|
||||
follow their own upstream tags, `postgres:16-alpine` and `redis:7-alpine`. Schema changes apply
|
||||
@@ -557,6 +582,11 @@ socket fan-out, all of which rebuild themselves; `agent_releases` is refilled by
|
||||
They are separate volumes precisely so that they can be, and must be, backed up separately. A dump
|
||||
that travelled with the key that opens it is a dump that opens itself.
|
||||
|
||||
> **On Kubernetes** the same two things matter and only their location changes: your Postgres
|
||||
> operator owns the dump, and the vault key is in the `jarvis-generated` Secret rather than a volume.
|
||||
> See [`kubernetes/README.md`](kubernetes/README.md#backups). The restore procedure below is
|
||||
> otherwise identical — it is a database and a key, wherever they are kept.
|
||||
|
||||
```sh
|
||||
docker compose exec -T postgres pg_dump -U jarvis jarvis | gzip > jarvis.sql.gz
|
||||
```
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: jarvis
|
||||
labels:
|
||||
# Both images run unprivileged and drop every capability, so the strictest profile applies
|
||||
# cleanly. Declared here rather than assumed: under `restricted`, a container that tries to run
|
||||
# as root is refused at admission, which is a far better failure than discovering it later.
|
||||
pod-security.kubernetes.io/enforce: restricted
|
||||
pod-security.kubernetes.io/enforce-version: latest
|
||||
@@ -0,0 +1,119 @@
|
||||
# Mint the generated secrets ONCE, from inside the cluster.
|
||||
#
|
||||
# ─── WHAT THIS REPLACES ───────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# On Compose, `init-secrets.cjs` runs as a one-shot writing into a volume every container shares,
|
||||
# and its guarantee that a key is generated exactly once is `openSync(path, "wx")` — two writers
|
||||
# race, the kernel picks one. There is no shared filesystem across pods, so the exclusion moves to
|
||||
# the one thing every pod does share: the API server. Creating a Secret that already exists is a
|
||||
# 409, decided by one serialized writer. Same guarantee, same strength.
|
||||
#
|
||||
# This is the alternative to minting by hand, NOT a replacement for understanding what it mints. If
|
||||
# your estate already has a secrets manager — SealedSecrets, ExternalSecrets, Vault — use it and
|
||||
# skip this file; write `jarvis-generated` yourself with the four keys listed in the Role below.
|
||||
#
|
||||
# ─── THE ONE WAY TO GET HURT ──────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# Deleting `jarvis-generated` and re-running this mints a NEW vault key against a database whose
|
||||
# credentials are sealed under the old one. Nothing fails at boot. Every stored credential becomes
|
||||
# permanently unreadable, and you find out the first time somebody opens one. There is no undo, and
|
||||
# the key is not in the database backup by design.
|
||||
#
|
||||
# kubectl apply -f 01-secret.example.yaml # the half you write: DATABASE_URL, OPENAI_API_KEY…
|
||||
# kubectl apply -f 01-secret-job.yaml # the half that is random bytes
|
||||
# kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m
|
||||
# # Then back the vault key up, out of this cluster:
|
||||
# kubectl get secret jarvis-generated -n jarvis -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d
|
||||
#
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: jarvis-secret-minter
|
||||
namespace: jarvis
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: jarvis-secret-minter
|
||||
namespace: jarvis
|
||||
rules:
|
||||
# `create` AND NOTHING ELSE. No `get`, no `list`, no `update`.
|
||||
#
|
||||
# Deliberate, and worth the paragraph: this identity cannot read any Secret in the namespace,
|
||||
# including the one it just wrote. That is why the script does not look before it writes — the
|
||||
# 409 is the check, and it is a stronger one than a read-then-write could be, because a read and
|
||||
# a write are two operations and something else can happen between them.
|
||||
#
|
||||
# `create` cannot be narrowed by resourceName — Kubernetes evaluates authorization before the
|
||||
# object's name is known — so this permits creating any Secret in this namespace. That is the
|
||||
# floor, and it is why the ServiceAccount belongs to this Job and to nothing else.
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["create"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: jarvis-secret-minter
|
||||
namespace: jarvis
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: jarvis-secret-minter
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: jarvis-secret-minter
|
||||
namespace: jarvis
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: jarvis-mint-secrets
|
||||
namespace: jarvis
|
||||
spec:
|
||||
# Re-running is safe (409 → left untouched), so a retry costs nothing. A high limit would only
|
||||
# mean a broken RBAC took longer to become visible.
|
||||
backoffLimit: 2
|
||||
# The Job object is tidied up an hour later. The Secret it created is a separate object and is
|
||||
# not touched by this.
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: jarvis-mint-secrets
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
serviceAccountName: jarvis-secret-minter
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 10001
|
||||
# Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces.
|
||||
# Omitted, the Job is created and its pod is refused — which reads as a Job that never runs.
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: mint
|
||||
# The api image, because the script ships in it and a first deployment should pull one
|
||||
# image rather than three. By digest like every other image here — the same one you put
|
||||
# in 03-migrate-job.yaml and 04-api.yaml.
|
||||
image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME
|
||||
command: ["node", "/app/apps/api/mint-k8s-secret.cjs"]
|
||||
env:
|
||||
# Node verifies the API server's certificate against the cluster CA. Without this the
|
||||
# request would fail — which is the correct failure. Do not reach for
|
||||
# NODE_TLS_REJECT_UNAUTHORIZED: a bearer token good enough to write Secrets is a bearer
|
||||
# token worth stealing, and an unverified TLS session is where that happens.
|
||||
- name: NODE_EXTRA_CA_CERTS
|
||||
value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 256Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
@@ -0,0 +1,71 @@
|
||||
# The secrets an operator writes. Minted ONCE, mounted by every pod.
|
||||
#
|
||||
# ─── READ THIS BEFORE YOU GENERATE ANYTHING ───────────────────────────────────────────────────
|
||||
#
|
||||
# There are two Secrets, and the split is not filing: `jarvis-generated` holds the values that are
|
||||
# random bytes and `jarvis-secrets` — this file — holds the ones that are facts about your estate.
|
||||
# One of those can be regenerated safely and the other cannot, so they do not share an object.
|
||||
#
|
||||
# The three keys below are here as the MANUAL path. If you would rather have them minted for you,
|
||||
# apply `01-secret-job.yaml` instead and leave the `VAULT_*`/`JWT_*` lines out of this file
|
||||
# entirely. Do not do both: `04-api.yaml` reads this Secret last on purpose, so a value typed here
|
||||
# wins over a generated one, and the two quietly disagreeing is the shape of the accident.
|
||||
#
|
||||
# `apps/api/init-secrets.cjs` is the Compose generator, and it must NEVER become an initContainer
|
||||
# here. Its guarantee that a key is minted exactly once is every writer sharing one filesystem.
|
||||
# Per-pod, on an `emptyDir`, each replica generates its own — and the consequence is not a crash.
|
||||
# Pod A seals a vault credential under a key pod B does not have; pod B reports that credential as
|
||||
# corrupt; both pods log nothing at boot, because from each one's point of view everything is fine.
|
||||
# By the time anybody notices, there are several keys in circulation and no way to tell which
|
||||
# entries belong to which. `apps/api/mint-k8s-secret.cjs` is the same one-shot done correctly, with
|
||||
# the API server's 409 standing in for the filesystem's exclusion.
|
||||
#
|
||||
# Either way, keep the output. The vault key in particular is the thing every stored credential is
|
||||
# encrypted under — a database backup that travelled with its own key would be a backup that
|
||||
# decrypts itself, which is why this lives apart from the database.
|
||||
#
|
||||
# VAULT_MASTER_KEY openssl rand -base64 32
|
||||
# JWT_ACCESS_SECRET openssl rand -base64 48
|
||||
# JWT_REFRESH_SECRET openssl rand -base64 48
|
||||
#
|
||||
# In anything beyond a trial, do not commit this file with values in it. Use SealedSecrets,
|
||||
# ExternalSecrets, or `kubectl create secret generic` from a password manager.
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: jarvis-secrets
|
||||
namespace: jarvis
|
||||
type: Opaque
|
||||
stringData:
|
||||
# Everything in the vault is sealed under this. Lose it and the credentials are unreadable;
|
||||
# leak it and a database dump is enough to read them all.
|
||||
VAULT_MASTER_KEY: "REPLACE_ME"
|
||||
JWT_ACCESS_SECRET: "REPLACE_ME"
|
||||
JWT_REFRESH_SECRET: "REPLACE_ME"
|
||||
|
||||
# WHEN THE VAULT KEY WAS CREATED, as an ISO 8601 timestamp or epoch seconds.
|
||||
#
|
||||
# Set it when you mint the key, and never move it afterwards. On Compose this is inferred from the
|
||||
# key file's modification time, which is a good answer on a volume nobody rewrites. Here it is a
|
||||
# wrong one: the kubelet writes a projected Secret into every pod at start, so the mtime is the
|
||||
# POD's age and every pod reports a key created seconds ago however old it is.
|
||||
#
|
||||
# What that number decides is narrow and serious. An instance with no accounts can have its first
|
||||
# super-admin claimed by whoever reaches the install screen — unless the vault key is old, which
|
||||
# means the database has gone missing from an instance that already existed (a bad restore, a PVC
|
||||
# pointed at the wrong volume) and the claim needs proof of possession instead. Without this line,
|
||||
# that check reads "brand new instance" on exactly the estate it exists to protect.
|
||||
VAULT_KEY_CREATED_AT: "REPLACE_ME"
|
||||
|
||||
# `connection_limit` is not optional here, and it is the reason this URL is in the Secret rather
|
||||
# than assembled from parts. Prisma's default pool is per-process: N pods times that default will
|
||||
# exhaust `max_connections` long before the application is under any real load. Size it as
|
||||
# (max_connections − reserve) ÷ (replicas + maxSurge + jobs).
|
||||
DATABASE_URL: "postgresql://jarvis:REPLACE_ME@postgres.jarvis.svc.cluster.local:5432/jarvis?schema=public&connection_limit=10&pool_timeout=20"
|
||||
|
||||
# Required to create new organizations, users or agents. Without it an instance keeps running
|
||||
# everything it already has and creates nothing new. See the project README.
|
||||
JARVIS_LICENSE_KEY: ""
|
||||
|
||||
OPENAI_API_KEY: "REPLACE_ME"
|
||||
@@ -0,0 +1,82 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: jarvis-api-config
|
||||
namespace: jarvis
|
||||
data:
|
||||
NODE_ENV: "production"
|
||||
API_PORT: "4000"
|
||||
|
||||
# One logical keyspace, shared by every pod. Sentinel or a managed single-primary rather than
|
||||
# Cluster: the Socket.IO adapter and the inter-pod bus both publish across what would otherwise
|
||||
# be different slots.
|
||||
#
|
||||
# A SINGLE REDIS IS A SINGLE POINT OF FAILURE FOR EVERY POD AT ONCE. This line is correct for a
|
||||
# managed Redis that fails over behind one address, and correct-but-fragile for one Deployment of
|
||||
# one Redis. For Sentinel, comment this out and set the two below instead — see
|
||||
# `08-redis-sentinel.yaml`, which deploys exactly that.
|
||||
REDIS_URL: "redis://redis.jarvis.svc.cluster.local:6379"
|
||||
|
||||
# Set BOTH or neither. Sentinel needs the sentinels and the name of the primary they watch; a
|
||||
# name is refused rather than defaulted, because ioredis's own default (`mymaster`) against a
|
||||
# primary called anything else is not an error — it is a pod that never becomes ready, silently.
|
||||
# When these are set, REDIS_URL above is ignored.
|
||||
#
|
||||
#REDIS_SENTINELS: "redis-sentinel-0.redis-sentinel.jarvis.svc.cluster.local:26379,redis-sentinel-1.redis-sentinel.jarvis.svc.cluster.local:26379,redis-sentinel-2.redis-sentinel.jarvis.svc.cluster.local:26379"
|
||||
#REDIS_SENTINEL_NAME: "jarvis"
|
||||
|
||||
# ONE, not the two the compose files use, and getting this wrong is a security fault rather than
|
||||
# a cosmetic one.
|
||||
#
|
||||
# It counts the proxies that REWRITE X-Forwarded-For. On compose a request passes the operator's
|
||||
# TLS terminator and then the web container's nginx: two. Here the Ingress routes /api/ straight
|
||||
# at the API Service, so there is one. Set HIGHER than the truth, the API believes a hop that does
|
||||
# not exist — and since X-Forwarded-For is a request header, anyone can then prepend an address of
|
||||
# their choosing and have it written into session records, audit rows, and the key the anonymous
|
||||
# WebAuthn budget counts on.
|
||||
#
|
||||
# It is a STORED setting: this seeds the first boot, and the install screen — which shows the
|
||||
# chain the API actually received — is where it is confirmed.
|
||||
TRUST_PROXY_HOPS: "1"
|
||||
|
||||
# The seed for the address enrolled agents dial back to. Whatever the install screen stores wins
|
||||
# from then on, so this is a starting value rather than the answer.
|
||||
WEB_ORIGIN: "https://jarvis.example.com"
|
||||
|
||||
# Where the agent binaries are, filled per pod by an initContainer. Leaving it unset is
|
||||
# supported: everything works except the agent installer, which answers 503 saying no build is
|
||||
# published.
|
||||
AGENT_RELEASE_DIR: "/srv/agent-releases"
|
||||
|
||||
AGENT_HEARTBEAT_INTERVAL_SEC: "30"
|
||||
# Must stay comfortably below the pod's terminationGracePeriodSeconds (90). Whichever expires
|
||||
# first decides whether a restart leaves a closed transcript or a half-written one.
|
||||
RUN_SHUTDOWN_GRACE_SEC: "25"
|
||||
|
||||
# No demo organization and no seeded administrator: the install screen is the way in. The seed
|
||||
# refuses an instance that already has accounts anyway, but a Deployment has no business
|
||||
# furnishing anything.
|
||||
SEED_ON_START: "false"
|
||||
|
||||
OPENAI_BASE_URL: "https://api.openai.com/v1"
|
||||
OPENAI_MODEL: "gpt-4o"
|
||||
OPENAI_THINKING_LEVEL: "medium"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: jarvis-web-config
|
||||
namespace: jarvis
|
||||
data:
|
||||
# Where the web container's nginx sends /api/, the socket.io handshake and the agent websocket.
|
||||
#
|
||||
# FULLY QUALIFIED, and that is not decoration: nginx's `resolver` does not apply the search list
|
||||
# from /etc/resolv.conf, so a bare `jarvis-api` would not resolve however correct it looks.
|
||||
JARVIS_API_UPSTREAM: "jarvis-api.jarvis.svc.cluster.local:4000"
|
||||
|
||||
# Who resolves it. The compose default is Docker's embedded DNS at 127.0.0.11, which nothing
|
||||
# listens on inside a pod — every /api/ request answered 502 until this became substitutable.
|
||||
#
|
||||
# Replace with your cluster's DNS ClusterIP:
|
||||
# kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'
|
||||
JARVIS_DNS_RESOLVER: "REPLACE_WITH_YOUR_CLUSTER_DNS"
|
||||
@@ -0,0 +1,93 @@
|
||||
# The schema, brought up to this image's expectations. Once, before any pod of it starts serving.
|
||||
#
|
||||
# A JOB, and deliberately neither of the two alternatives:
|
||||
#
|
||||
# NOT an initContainer. It would run once per pod — which is exactly the bug this replaces, where
|
||||
# every API container ran `prisma db push` at boot and N replicas raced the same DDL while the
|
||||
# previous generation was still selecting the columns being altered.
|
||||
#
|
||||
# NOT a leader election inside the API. A Job is the primitive built for run-once work: it has an
|
||||
# observable status, a bounded number of retries, and a failure that stops the rollout instead of
|
||||
# letting it proceed against a schema that was never applied.
|
||||
#
|
||||
# With Helm this becomes a `pre-install,pre-upgrade` hook; with Argo, a sync-wave. Applied by hand,
|
||||
# the ordering is yours to keep — see the README.
|
||||
#
|
||||
# It is safe to run twice: every pass is idempotent and the whole phase is wrapped in a Postgres
|
||||
# advisory lock, so a second copy waits rather than racing.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: jarvis-migrate
|
||||
namespace: jarvis
|
||||
spec:
|
||||
# One writer. The lock makes a second one wait rather than corrupt, but there is no reason to
|
||||
# have one waiting.
|
||||
parallelism: 1
|
||||
completions: 1
|
||||
# Two retries, because the failures worth retrying are transient (a database not finished failing
|
||||
# over). A schema that Prisma REFUSES is not transient and must not be retried into submission —
|
||||
# it is refused for a reason, and the reason is printed.
|
||||
backoffLimit: 2
|
||||
# A migration that has not finished in fifteen minutes is stuck, not slow, and the rollout is
|
||||
# blocked behind it.
|
||||
activeDeadlineSeconds: 900
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: migrate
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
# Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces. At pod
|
||||
# level so it covers anything added beside this container later. Without it the Job is
|
||||
# accepted and its pod is refused, so the rollout waits on a Job that will never complete.
|
||||
securityContext:
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: migrate
|
||||
# THE SAME IMAGE, by the same digest, as the API that is about to roll. A migration from a
|
||||
# different build is a schema that matches nothing.
|
||||
image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME
|
||||
command: ["/usr/local/bin/migrate.sh"]
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: jarvis-api-config
|
||||
# Generated first, hand-written last, for the reason spelled out in 04-api.yaml: the
|
||||
# last occurrence of a name wins, and a generated value must never override a typed one.
|
||||
- secretRef:
|
||||
name: jarvis-generated
|
||||
optional: true
|
||||
- secretRef:
|
||||
name: jarvis-secrets
|
||||
env:
|
||||
# Its own, smaller pool: this is one process doing one thing, and the connections it
|
||||
# takes are connections the running API cannot have while it works.
|
||||
- name: DATABASE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: jarvis-secrets
|
||||
key: DATABASE_URL
|
||||
# Off, and it should stay off between deliberate acts. `prisma db push` used to run with
|
||||
# `--accept-data-loss` at every boot, which meant an upgrade could destroy a column with
|
||||
# nobody having decided to — survivable on one container, not during a rolling update
|
||||
# where the previous generation is still selecting it.
|
||||
#
|
||||
# Without the flag the phase REFUSES, prints which columns it would have dropped, and
|
||||
# fails the Job — so the rollout stops and nothing is serving a half-migrated schema.
|
||||
# Set it to "true" for the ONE deploy where that drop is intended, then remove it.
|
||||
- name: SCHEMA_ACCEPT_DATA_LOSS
|
||||
value: "false"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
memory: 1Gi
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
@@ -0,0 +1,194 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: jarvis-api
|
||||
namespace: jarvis
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: api
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
# Never fewer pods than are being served now. A run holds its conversation through a unique
|
||||
# index, so a surge pod cannot double-answer anything — the old constraint that made an
|
||||
# overlap dangerous is gone.
|
||||
maxUnavailable: 0
|
||||
maxSurge: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: api
|
||||
spec:
|
||||
# 90 seconds, and every part of it is accounted for:
|
||||
# 10s preStop, so this pod leaves the Service's endpoints BEFORE it starts draining
|
||||
# 25s RUN_SHUTDOWN_GRACE_SEC — in-flight assistant runs stop and close their transcripts
|
||||
# ~15s agent sockets closed and their rows written, httpServer.close(), Prisma, Redis
|
||||
# the rest is margin
|
||||
# The default of 30 would SIGKILL the pod in the middle of the run drain, which is the one
|
||||
# thing the drain exists to prevent.
|
||||
terminationGracePeriodSeconds: 90
|
||||
# Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces. At pod
|
||||
# level rather than per container, because the agent-releases initContainer below needs it
|
||||
# too — and a missing seccompProfile is not a warning, it is the pod being refused outright.
|
||||
securityContext:
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
# SPREAD THE REPLICAS, because nothing else will.
|
||||
#
|
||||
# The scheduler has no reason of its own to put two pods of the same Deployment on different
|
||||
# nodes, and quite often has a reason not to. Two API replicas on one node is a deployment
|
||||
# that reads as highly available in `kubectl get pods` and loses every agent websocket, every
|
||||
# live run and every open shell the moment that one node reboots. The PodDisruptionBudget in
|
||||
# 07-disruption.yaml does not help here: it constrains voluntary evictions, and a node dying
|
||||
# is not one.
|
||||
#
|
||||
# `ScheduleAnyway` rather than `DoNotSchedule`: on a single-node cluster the strict form
|
||||
# leaves the second replica Pending forever, which is a worse first experience than an
|
||||
# unspread pair. On a real cluster it spreads. If yours spans zones, add a second constraint
|
||||
# on `topology.kubernetes.io/zone` — a rack is a failure domain too.
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: api
|
||||
containers:
|
||||
- name: api
|
||||
# BY DIGEST. `:stable` is a moving name, so replicas that restart at different times land
|
||||
# on different builds — and a rollback becomes "hope the tag still points where it did".
|
||||
image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4000
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
# Endpoint removal and SIGTERM race each other, and losing that race is visible:
|
||||
# a prompt that arrives after Nest sets `draining` is persisted and broadcast as
|
||||
# queued, and this pod will never answer it. Ten seconds is comfortably more than
|
||||
# kube-proxy needs to stop sending new connections here.
|
||||
command: ["/bin/sh", "-c", "sleep 10"]
|
||||
env:
|
||||
- name: JARVIS_SKIP_MIGRATIONS
|
||||
# The schema is the migration Job's business, once, before this rolls. N pods each
|
||||
# running `prisma db push` is N writers racing the same DDL while the previous
|
||||
# generation is still selecting the columns being altered.
|
||||
value: "true"
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: jarvis-api-config
|
||||
# The half that is random bytes, minted once by `01-secret-job.yaml`. Kept apart from
|
||||
# the half an operator writes because one of them can be regenerated and the other
|
||||
# cannot: re-running the minter is a no-op by design, which is only safe while nothing
|
||||
# anybody typed lives in the same object.
|
||||
#
|
||||
# `optional: true` so a deployment that mints by hand — or through SealedSecrets,
|
||||
# ExternalSecrets, Vault — does not need this object to exist at all.
|
||||
- secretRef:
|
||||
name: jarvis-generated
|
||||
optional: true
|
||||
# LAST, and the order is the safety property rather than a formatting choice. `envFrom`
|
||||
# resolves in sequence and the last occurrence of a name wins, so anything written by
|
||||
# hand here overrides the generated value — never the other way round. Reversed, an
|
||||
# operator who had already minted a vault key and then applied the Job by reflex would
|
||||
# have every stored credential silently become unreadable.
|
||||
- secretRef:
|
||||
name: jarvis-secrets
|
||||
volumeMounts:
|
||||
- name: agent-releases
|
||||
mountPath: /srv/agent-releases
|
||||
readOnly: true
|
||||
# Liveness NEVER touches the database. A Postgres failover with a database check here
|
||||
# restarts every pod at once, turning a thirty-second blip into a cold start of the whole
|
||||
# deployment at the moment the database can least afford a reconnection stampede.
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /api/health/live
|
||||
port: http
|
||||
periodSeconds: 10
|
||||
failureThreshold: 3
|
||||
# Readiness DOES, and it checks Redis too. A pod cut off from Redis keeps answering HTTP
|
||||
# perfectly while its broadcasts reach nobody — the clients queue rather than fail — so it
|
||||
# looks healthier than a pod that is down and is more dangerous.
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /api/health/ready
|
||||
port: http
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 3
|
||||
# Boot does a run-recovery sweep and reads settings before it listens. The startup probe
|
||||
# is what stops liveness killing it halfway through on a slow database.
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /api/health/live
|
||||
port: http
|
||||
periodSeconds: 5
|
||||
failureThreshold: 30
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
# Not generous — honest. A terminal recording holds up to 8 MiB of Buffers in heap
|
||||
# for the life of the session, and argon2 takes 64 MiB per share verification.
|
||||
memory: 1Gi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
initContainers:
|
||||
# The compiled agent binaries, into this pod's own emptyDir.
|
||||
#
|
||||
# PER POD ON PURPOSE. The content is immutable and small, so a copy per pod costs nothing;
|
||||
# a shared RWO PVC would either pin every pod to one node or leave the second in
|
||||
# Multi-Attach error. Leaving this out entirely is supported: enrollment still works and the
|
||||
# installer answers 503 saying no build is published.
|
||||
- name: agent-releases
|
||||
image: git.luxit.be/luxit/jarvis-agent-dist@sha256:REPLACE_ME
|
||||
command: ["/bin/sh", "-c", "cp -a /dist/. /srv/agent-releases/"]
|
||||
# AN initContainer IS A CONTAINER, and the `restricted` Pod Security Standard judges it
|
||||
# exactly like the one below. This block was missing, and the result was not a warning
|
||||
# about the initContainer — it was the WHOLE POD refused, so the Deployment sat at zero
|
||||
# replicas reporting FailedCreate while every other line in this file was correct.
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
volumeMounts:
|
||||
- name: agent-releases
|
||||
mountPath: /srv/agent-releases
|
||||
volumes:
|
||||
- name: agent-releases
|
||||
emptyDir: {}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: jarvis-api
|
||||
namespace: jarvis
|
||||
spec:
|
||||
type: ClusterIP
|
||||
# STATED, not left to the default, because the reflex is to reach for it the moment websockets
|
||||
# are involved. It would not help: the hard case is co-locating an operator's browser with an
|
||||
# agent's websocket, which arrives separately and whose id is unknown at the browser's handshake.
|
||||
# That is handled in the application, over Redis. See the README.
|
||||
sessionAffinity: None
|
||||
selector:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: api
|
||||
ports:
|
||||
- name: http
|
||||
port: 4000
|
||||
targetPort: http
|
||||
@@ -0,0 +1,152 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: jarvis-web
|
||||
namespace: jarvis
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: web
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxUnavailable: 0
|
||||
maxSurge: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: web
|
||||
spec:
|
||||
# Nothing here holds a long-lived connection of its own — the websockets go to the API — so
|
||||
# this is only about letting in-flight responses finish.
|
||||
terminationGracePeriodSeconds: 30
|
||||
# Required by the `restricted` Pod Security Standard that 00-namespace.yaml enforces.
|
||||
securityContext:
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
# Same reasoning as 04-api.yaml: the scheduler will happily put both replicas on one node,
|
||||
# and a pair that shares a node is a pair that shares a failure. Cheaper to satisfy here —
|
||||
# nothing about serving static files pins a pod anywhere.
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: web
|
||||
containers:
|
||||
- name: web
|
||||
# By digest, like the API. It matters more here than it looks: the bundle's JavaScript
|
||||
# chunks are content-hashed PER BUILD and answer 404 rather than falling through to
|
||||
# index.html, so during a rollout a browser holding the previous shell can fail a lazy
|
||||
# import. Two pods on two different builds widen that window; two pods on one digest do
|
||||
# not have it at all.
|
||||
image: git.luxit.be/luxit/jarvis-web@sha256:REPLACE_ME
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8080
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ["/bin/sh", "-c", "sleep 5"]
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: jarvis-web-config
|
||||
# Is this nginx answering? Nothing more. Restarting the web container because the API is
|
||||
# down would take out the one thing still able to show an error page.
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
periodSeconds: 10
|
||||
failureThreshold: 3
|
||||
# Can THIS pod reach the API? It proxies /api/health/live, not /ready, on purpose: the
|
||||
# question is whether this pod's route works — a broken DNS name, a NetworkPolicy — and
|
||||
# not whether the API's dependencies are healthy. Pointed at /ready, one Postgres blip
|
||||
# would fail every web pod at once and take the console offline for a fault that has
|
||||
# nothing to do with serving a bundle.
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /readyz
|
||||
port: http
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 3
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 256Mi
|
||||
securityContext:
|
||||
# uid 101 is the `nginx` user the image already ships and the Dockerfile switches to.
|
||||
runAsNonRoot: true
|
||||
runAsUser: 101
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
readOnlyRootFilesystem: true
|
||||
volumeMounts:
|
||||
# With a read-only root filesystem, the three paths this container writes at runtime
|
||||
# have to be given to it. All three are ephemeral by nature — the substituted config is
|
||||
# regenerated at every start from the template and the ConfigMap.
|
||||
- name: nginx-conf
|
||||
mountPath: /etc/nginx/conf.d
|
||||
- name: nginx-cache
|
||||
mountPath: /var/cache/nginx
|
||||
- name: tmp
|
||||
# The pid file lives here rather than in /var/run, which the nginx user does not own.
|
||||
mountPath: /tmp
|
||||
volumes:
|
||||
- name: nginx-conf
|
||||
emptyDir: {}
|
||||
- name: nginx-cache
|
||||
emptyDir: {}
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: jarvis-web
|
||||
namespace: jarvis
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: web
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
targetPort: http
|
||||
---
|
||||
# Web scales on CPU; the API deliberately does not.
|
||||
#
|
||||
# Every API scale-in kills a pod holding agent websockets, live assistant runs and open shells —
|
||||
# the fleet reconnects, runs are interrupted and resumed, terminals die — and CPU is a poor proxy
|
||||
# for a load made almost entirely of long-lived connections. Raise `jarvis-api` deliberately
|
||||
# instead. If it ever does get an HPA, give it a scale-down stabilization window of ten minutes or
|
||||
# more.
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: jarvis-web
|
||||
namespace: jarvis
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: jarvis-web
|
||||
minReplicas: 2
|
||||
maxReplicas: 5
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70
|
||||
@@ -0,0 +1,81 @@
|
||||
# One hostname, two backends.
|
||||
#
|
||||
# The API is routed STRAIGHT AT ITS SERVICE rather than through the web container's nginx, which is
|
||||
# what that nginx does on Compose. Two reasons: it removes a hop from every API request and every
|
||||
# websocket frame, and it makes the web pod a pure static server again — no dependency on resolving
|
||||
# the API, nothing to keep in sync between an ingress and a config file.
|
||||
#
|
||||
# The consequence has to be applied, not just noted: there is now exactly ONE proxy rewriting
|
||||
# X-Forwarded-For, so `TRUST_PROXY_HOPS` is 1. It is 2 in the compose files, and leaving it there
|
||||
# would make `req.ip` a value the caller chooses. See 02-config.yaml.
|
||||
#
|
||||
# Set `externalTrafficPolicy: Local` on the ingress controller's own Service as well, or the node's
|
||||
# SNAT will make every request in the world appear to come from a handful of addresses — which
|
||||
# defeats every per-address limit in the application at once.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: jarvis
|
||||
namespace: jarvis
|
||||
annotations:
|
||||
# An agent is idle between operations and the server pings every 30s, so this only has to
|
||||
# outlast a quiet period comfortably. Too short and a healthy fleet reconnects all day.
|
||||
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
|
||||
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
|
||||
# Matches BODY_LIMIT in apps/api/src/main.ts. A document may hold 400,000 characters, which is
|
||||
# over a megabyte once accented text is UTF-8 encoded, and a diagram export posts back the SVG
|
||||
# the browser rendered. Below this they are rejected as a bare 413 with no message.
|
||||
nginx.ingress.kubernetes.io/proxy-body-size: "4m"
|
||||
#
|
||||
# NO affinity annotation, and none should be added. The reflex is understandable — there are
|
||||
# websockets here — but stickiness cannot solve the problem it would be hired for: the hard case
|
||||
# is co-locating an operator's browser with an AGENT's websocket, which arrives from a different
|
||||
# network at a different time and whose id is not even known at the browser's handshake. No
|
||||
# ingress annotation can express that; it is handled in the application, over Redis.
|
||||
#
|
||||
# And there is no polling to make sticky: both the client and the server pin
|
||||
# transports: ["websocket"], so one logical socket is one TCP connection.
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
tls:
|
||||
- hosts: ["jarvis.example.com"]
|
||||
secretName: jarvis-tls
|
||||
rules:
|
||||
- host: jarvis.example.com
|
||||
http:
|
||||
paths:
|
||||
# Longest prefix first for readers; the controller sorts by specificity itself.
|
||||
#
|
||||
# The agent websocket has its own rule even though it sits under /api/, mirroring the
|
||||
# split in apps/web/nginx.conf: it is the one path where an idle connection legitimately
|
||||
# stays open for an hour between frames.
|
||||
- path: /api/agents/ws
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: jarvis-api
|
||||
port:
|
||||
name: http
|
||||
- path: /socket.io/
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: jarvis-api
|
||||
port:
|
||||
name: http
|
||||
- path: /api/
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: jarvis-api
|
||||
port:
|
||||
name: http
|
||||
# Everything else is the SPA, whose nginx answers index.html for any path it does not
|
||||
# have a file for.
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: jarvis-web
|
||||
port:
|
||||
name: http
|
||||
@@ -0,0 +1,28 @@
|
||||
# What a node drain may take away.
|
||||
#
|
||||
# `minAvailable: 1` on both, so a cluster upgrade cannot evict the last pod of either. With the
|
||||
# API's 90-second grace period a drain will take a minute or two per pod — that is the run drain
|
||||
# doing its job, not a stall.
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: jarvis-api
|
||||
namespace: jarvis
|
||||
spec:
|
||||
minAvailable: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: api
|
||||
---
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: jarvis-web
|
||||
namespace: jarvis
|
||||
spec:
|
||||
minAvailable: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: web
|
||||
@@ -0,0 +1,341 @@
|
||||
# A Redis that survives losing a node.
|
||||
#
|
||||
# ─── WHAT THIS IS, AND WHAT IT IS NOT ─────────────────────────────────────────────────────────
|
||||
#
|
||||
# OPTIONAL. Apply it only if you are running your own Redis. A managed Redis that publishes one
|
||||
# address and fails over behind it needs none of this — point `REDIS_URL` at it and skip the file.
|
||||
#
|
||||
# It is written to be correct and it is small enough to read in full, but a datastore you operate
|
||||
# yourself is a datastore you maintain: version upgrades, capacity, the failover drill. If you want
|
||||
# somebody else to hold that, use a managed Redis or an operator. What this directory owes you is
|
||||
# the shape the application needs, and this is it.
|
||||
#
|
||||
# The application half is proven rather than assumed: an ioredis client built exactly the way
|
||||
# `apps/api/src/redis/redis-connection.ts` builds one was run against a live three-sentinel
|
||||
# topology while its primary was killed. It followed the promotion to a different server, and 99
|
||||
# writes issued across the window were rejected zero times. What it does NOT do is preserve pub/sub
|
||||
# during the switch — see "What a failover actually costs" at the bottom.
|
||||
#
|
||||
# kubectl apply -f 08-redis-sentinel.yaml
|
||||
# # then in 02-config.yaml: comment out REDIS_URL, uncomment the two REDIS_SENTINEL* lines
|
||||
#
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: redis-scripts
|
||||
namespace: jarvis
|
||||
data:
|
||||
# WHO IS THE PRIMARY — asked, never assumed.
|
||||
#
|
||||
# The naive version of this file pins `replicaof redis-0` into every replica. It works exactly
|
||||
# until the first failover, after which redis-0 comes back, is told it is the primary, and the
|
||||
# cluster has two — a split brain that presents as writes vanishing rather than as an error.
|
||||
#
|
||||
# So a starting node asks the sentinels who the primary is now, and only falls back to redis-0
|
||||
# when no sentinel answers, which is true on a first apply and at no other time.
|
||||
discover.sh: |
|
||||
#!/bin/sh
|
||||
# Prints the current primary's hostname. Empty output is impossible: the fallback is redis-0.
|
||||
for i in 0 1 2; do
|
||||
peer="redis-sentinel-${i}.redis-sentinel.${NAMESPACE}.svc.cluster.local"
|
||||
found=$(redis-cli -h "$peer" -p 26379 sentinel get-master-addr-by-name jarvis 2>/dev/null | head -1)
|
||||
if [ -n "$found" ]; then
|
||||
echo "$found"
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
echo "redis-0.redis.${NAMESPACE}.svc.cluster.local"
|
||||
|
||||
redis-start.sh: |
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
master=$(sh /scripts/discover.sh)
|
||||
me="${HOSTNAME}.redis.${NAMESPACE}.svc.cluster.local"
|
||||
echo "[redis] I am ${me}; the primary is ${master}"
|
||||
if [ "$master" = "$me" ]; then
|
||||
exec redis-server /etc/redis/redis.conf
|
||||
fi
|
||||
exec redis-server /etc/redis/redis.conf --replicaof "$master" 6379
|
||||
|
||||
sentinel-start.sh: |
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
master=$(sh /scripts/discover.sh)
|
||||
echo "[sentinel] monitoring ${master}"
|
||||
# Rebuilt every start rather than persisted. A sentinel REWRITES its own config file as it
|
||||
# learns, so the file cannot be a read-only ConfigMap mount; and its learned state is
|
||||
# recoverable from its peers in seconds, so persisting it buys nothing and can resurrect a
|
||||
# stale view of a topology that has since moved.
|
||||
cp /etc/sentinel-template/sentinel.conf /data/sentinel.conf
|
||||
{
|
||||
echo "sentinel monitor jarvis ${master} 6379 2"
|
||||
echo "sentinel down-after-milliseconds jarvis 5000"
|
||||
echo "sentinel failover-timeout jarvis 60000"
|
||||
echo "sentinel parallel-syncs jarvis 1"
|
||||
} >> /data/sentinel.conf
|
||||
exec redis-sentinel /data/sentinel.conf
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: redis-config
|
||||
namespace: jarvis
|
||||
data:
|
||||
redis.conf: |
|
||||
port 6379
|
||||
# NO PERSISTENCE, and it is a decision rather than an omission. Everything Jarvis keeps in Redis
|
||||
# is ephemeral by design and rebuilds itself: agent ownership keys carry a TTL the owning pod
|
||||
# refreshes, cluster leases expire, and the realtime adapter's state is the live connections
|
||||
# themselves. The one thing that must NOT be lost — which pod owns a run — is a row in Postgres
|
||||
# precisely so that losing Redis cannot lose it.
|
||||
save ""
|
||||
appendonly no
|
||||
# Replicas answer reads while they catch up rather than erroring. Nothing here reads from a
|
||||
# replica today, but a client that briefly lands on one during a promotion should see slightly
|
||||
# old data rather than a failure.
|
||||
replica-serve-stale-data yes
|
||||
# A replica must never accept a write, however it was reached.
|
||||
replica-read-only yes
|
||||
|
||||
sentinel.conf: |
|
||||
port 26379
|
||||
# HOSTNAMES, NOT POD IPs, and this is the line most likely to be dropped as noise.
|
||||
#
|
||||
# By default a sentinel hands clients the primary's IP. A pod's IP does not survive its
|
||||
# restart, so after a failover-then-restart the sentinels answer with an address nothing is
|
||||
# listening on — and a client pointed at it does not fail, it retries forever against a dead
|
||||
# address while the sentinels insist that is where the primary lives. Observed, not theorised.
|
||||
sentinel resolve-hostnames yes
|
||||
sentinel announce-hostnames yes
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: jarvis
|
||||
spec:
|
||||
# Headless: the point is the per-pod DNS names (redis-0.redis…), which are what the sentinels
|
||||
# hand out and therefore what the API ends up connecting to.
|
||||
clusterIP: None
|
||||
publishNotReadyAddresses: true
|
||||
selector:
|
||||
app.kubernetes.io/name: redis
|
||||
ports:
|
||||
- name: redis
|
||||
port: 6379
|
||||
targetPort: 6379
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: redis-sentinel
|
||||
namespace: jarvis
|
||||
spec:
|
||||
clusterIP: None
|
||||
publishNotReadyAddresses: true
|
||||
selector:
|
||||
app.kubernetes.io/name: redis-sentinel
|
||||
ports:
|
||||
- name: sentinel
|
||||
port: 26379
|
||||
targetPort: 26379
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: jarvis
|
||||
spec:
|
||||
serviceName: redis
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: redis
|
||||
spec:
|
||||
terminationGracePeriodSeconds: 30
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
# REQUIRED, not preferred. Three replicas on one node is a topology that looks highly
|
||||
# available in `kubectl get pods` and is not one.
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
- labelSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
topologyKey: kubernetes.io/hostname
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 999
|
||||
fsGroup: 999
|
||||
# REQUIRED by the `restricted` Pod Security Standard, which 00-namespace.yaml enforces.
|
||||
# Without it the pods are not warned about, they are REFUSED — the StatefulSet is created,
|
||||
# reports FailedCreate, and stays at zero replicas. Found by applying it, not by
|
||||
# kubeconform: schema validation cannot see an admission policy.
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: redis
|
||||
image: redis:7-alpine
|
||||
command: ["sh", "/scripts/redis-start.sh"]
|
||||
env:
|
||||
- name: NAMESPACE
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.namespace
|
||||
ports:
|
||||
- name: redis
|
||||
containerPort: 6379
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["redis-cli", "ping"]
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["redis-cli", "ping"]
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
volumeMounts:
|
||||
- name: scripts
|
||||
mountPath: /scripts
|
||||
- name: config
|
||||
mountPath: /etc/redis
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: scripts
|
||||
configMap:
|
||||
name: redis-scripts
|
||||
- name: config
|
||||
configMap:
|
||||
name: redis-config
|
||||
- name: data
|
||||
# No PVC on purpose — see `save ""` above. A volumeClaimTemplate here would persist a
|
||||
# dataset that is meant to be rebuildable, and give a restarted node a stale one.
|
||||
emptyDir: {}
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: redis-sentinel
|
||||
namespace: jarvis
|
||||
spec:
|
||||
serviceName: redis-sentinel
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis-sentinel
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: redis-sentinel
|
||||
spec:
|
||||
terminationGracePeriodSeconds: 30
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
- labelSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis-sentinel
|
||||
topologyKey: kubernetes.io/hostname
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 999
|
||||
fsGroup: 999
|
||||
# REQUIRED by the `restricted` Pod Security Standard, which 00-namespace.yaml enforces.
|
||||
# Without it the pods are not warned about, they are REFUSED — the StatefulSet is created,
|
||||
# reports FailedCreate, and stays at zero replicas. Found by applying it, not by
|
||||
# kubeconform: schema validation cannot see an admission policy.
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: sentinel
|
||||
image: redis:7-alpine
|
||||
command: ["sh", "/scripts/sentinel-start.sh"]
|
||||
env:
|
||||
- name: NAMESPACE
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.namespace
|
||||
ports:
|
||||
- name: sentinel
|
||||
containerPort: 26379
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["redis-cli", "-p", "26379", "ping"]
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
# The root filesystem is read-only and the sentinel still works, because the one thing
|
||||
# it must write — its own config, which it rewrites as it learns the topology — lives on
|
||||
# the writable emptyDir at /data. That is why the template is COPIED there at start
|
||||
# rather than run from the ConfigMap mount, which is read-only whatever this says.
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
volumeMounts:
|
||||
- name: scripts
|
||||
mountPath: /scripts
|
||||
- name: sentinel-template
|
||||
mountPath: /etc/sentinel-template
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: scripts
|
||||
configMap:
|
||||
name: redis-scripts
|
||||
- name: sentinel-template
|
||||
configMap:
|
||||
name: redis-config
|
||||
items:
|
||||
- key: sentinel.conf
|
||||
path: sentinel.conf
|
||||
- name: data
|
||||
emptyDir: {}
|
||||
---
|
||||
# Two of three sentinels must be reachable for the quorum to hold. Without this, a node drain can
|
||||
# take two at once — and a Redis with no quorum does not fail over, which is the entire point of
|
||||
# having three.
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: redis-sentinel
|
||||
namespace: jarvis
|
||||
spec:
|
||||
minAvailable: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis-sentinel
|
||||
---
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: jarvis
|
||||
spec:
|
||||
minAvailable: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
@@ -0,0 +1,206 @@
|
||||
# Jarvis on Kubernetes
|
||||
|
||||
Plain manifests, no Helm, no operator. They are written to be read as much as applied: every value
|
||||
that is not a Kubernetes default is there because something in Jarvis behaves a particular way, and
|
||||
the comment beside it says which. Read them before you apply them — that is the point of shipping
|
||||
them this way.
|
||||
|
||||
**You probably do not need this.** [Docker Compose](../README.md#install) runs one of everything,
|
||||
takes one command, and is what most instances run. Come here when you want Jarvis to keep serving
|
||||
while a node reboots, and when you already operate a cluster — this directory assumes you have
|
||||
opinions about ingress, storage and a database, because it deliberately does not have them for you.
|
||||
|
||||
> **What has been proven, and what has not.** This stack has been brought up on a real cluster —
|
||||
> four nodes, three Postgres instances, three Redis with three sentinels, two api pods — and kept
|
||||
> serving while its Postgres primary and its Redis primary were killed underneath it. What has NOT
|
||||
> happened is a long-running estate serving production traffic this way. If you are early, watch it
|
||||
> rather than assume it, and read the three warnings below before you apply anything, because each
|
||||
> of them fails silently.
|
||||
|
||||
## What runs more than once, and what cannot
|
||||
|
||||
"Highly available" is not a property a deployment has as a whole, so here it is component by
|
||||
component.
|
||||
|
||||
| component | replicas | why |
|
||||
| --- | --- | --- |
|
||||
| `jarvis-web` | any number | stateless nginx over static files |
|
||||
| `jarvis-api` | any number | the coordination lives in Postgres and Redis, not in the process |
|
||||
| Redis | one **logical** primary, three processes | Sentinel fails over; it does not spread writes |
|
||||
| Postgres | one **primary**, replicas behind it | and Jarvis depends on there being exactly one |
|
||||
| `jarvis-migrate` | one at a time | enforced by a Postgres advisory lock, not by you |
|
||||
| `jarvis-mint-secrets` | once, ever | enforced by the API server |
|
||||
|
||||
The single-writer database is not a limitation Jarvis has failed to overcome — it is what makes the
|
||||
multi-pod api safe. A partial unique index is what stops two pods running the same assistant loop,
|
||||
and an index can only do that because one server decides. Do not point pods at read replicas.
|
||||
|
||||
## What you need before you apply anything
|
||||
|
||||
- **x86-64 nodes.** The api and web images are published for `linux/amd64` only, so an arm64 pool
|
||||
leaves the pods in `ImagePullBackOff` with no matching manifest. The machines you *administer*
|
||||
have no such limit — the agent ships arm64 builds.
|
||||
- **A cluster you can schedule on.** `kubectl describe node | grep Taints` first. A three-node
|
||||
cluster is often three control-plane nodes, and their `NoSchedule` taint means nothing here
|
||||
schedules at all. These manifests carry no toleration on purpose: adding one is a decision about
|
||||
your cluster.
|
||||
- **Postgres 16, highly available, outside these manifests.** Use an operator — CloudNativePG or
|
||||
Zalando's — rather than a hand-written StatefulSet. A database you hand-rolled is a database you
|
||||
hand-restore. `max_connections` must cover
|
||||
`(replicas + maxSurge + jobs) × connection_limit + reserve`: with 3 api pods at 10 connections
|
||||
each, a migration Job at 5, and headroom, about **150**. The stock Postgres image ships 100, and
|
||||
exhausting it presents as an api that starts and then cannot serve.
|
||||
- **Redis.** [`08-redis-sentinel.yaml`](08-redis-sentinel.yaml) deploys one, or point `REDIS_URL` at
|
||||
a managed Redis that fails over behind a single address and skip that file. Persistence is not
|
||||
required: everything Jarvis keeps in Redis is ephemeral by design, which is exactly why the run
|
||||
lock is in Postgres instead.
|
||||
- **An Ingress controller and a TLS terminator.** Jarvis speaks plain HTTP.
|
||||
|
||||
## Order
|
||||
|
||||
```sh
|
||||
kubectl apply -f 00-namespace.yaml
|
||||
|
||||
# The secrets, ONCE. Never regenerate them on an instance that has stored anything.
|
||||
kubectl apply -f 01-secret.example.yaml # after editing — the half you write
|
||||
kubectl apply -f 01-secret-job.yaml # the half that is random bytes (optional; see the file)
|
||||
kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m
|
||||
|
||||
kubectl apply -f 02-config.yaml
|
||||
kubectl apply -f 08-redis-sentinel.yaml # only if you are running your own Redis
|
||||
|
||||
# The schema, to completion, before any api pod starts.
|
||||
kubectl apply -f 03-migrate-job.yaml
|
||||
kubectl wait --for=condition=complete job/jarvis-migrate -n jarvis --timeout=15m
|
||||
|
||||
kubectl apply -f 04-api.yaml
|
||||
kubectl apply -f 05-web.yaml
|
||||
kubectl apply -f 06-ingress.yaml
|
||||
kubectl apply -f 07-disruption.yaml
|
||||
```
|
||||
|
||||
Every image says `REPLACE_ME`: pin them by digest, not by tag. `:stable` is a moving name, so
|
||||
replicas that restart at different times land on different builds and a rollback becomes "hope the
|
||||
tag still points where it did".
|
||||
|
||||
```sh
|
||||
docker buildx imagetools inspect git.luxit.be/luxit/jarvis-api:stable
|
||||
```
|
||||
|
||||
On every upgrade the migration Job runs again before the Deployments roll. It is idempotent and
|
||||
takes a Postgres advisory lock, so running it twice is slow rather than harmful — but it must
|
||||
finish before new pods start, which is what the `kubectl wait` is for. With Helm or Argo this
|
||||
becomes a `pre-upgrade` hook or a sync-wave; as plain YAML it is an ordering you keep yourself.
|
||||
|
||||
**Then open the address and install it.** Everything in [the main README](../README.md#install)
|
||||
about the install screen applies unchanged — including that **whoever reaches an uninstalled
|
||||
instance first owns it**, with no deadline. Point a public hostname at this after you have
|
||||
installed, not before.
|
||||
|
||||
## The three things most likely to bite
|
||||
|
||||
**Never mint secrets per pod.** On Compose a one-shot generates the vault key and the JWT secrets
|
||||
into a volume every container shares, and its guarantee that it does so exactly once is that the
|
||||
filesystem refuses a second create. As a per-pod initContainer on an `emptyDir`, each replica mints
|
||||
its own — and the consequence is not a crash. Pod A seals a credential under a key pod B does not
|
||||
have; pod B reports it as corrupt; both log a clean boot. By the time anybody opens a vault entry
|
||||
there are three keys in circulation.
|
||||
|
||||
[`01-secret-job.yaml`](01-secret-job.yaml) is that one-shot done correctly here: it creates the
|
||||
Secret once and lets the API server's refusal to create it twice be the exclusion. Its identity can
|
||||
`create` a Secret and cannot `get` one, so it cannot read back what it or anyone else wrote. Mint by
|
||||
hand instead if you prefer — the one thing that is not an option is per-pod.
|
||||
|
||||
Either way, **back the vault key up somewhere that is not this cluster's etcd.** Deleting the Secret
|
||||
and re-running the Job mints a new key against data sealed with the old one, silently, with no way
|
||||
back.
|
||||
|
||||
**`TRUST_PROXY_HOPS` is one hop here, not two.** The Compose files say 2 because a request passes
|
||||
your TLS terminator and then the web container's nginx. Here the Ingress routes `/api/` straight at
|
||||
the api Service, so there is exactly one proxy rewriting `X-Forwarded-For`. Left at 2, `req.ip`
|
||||
becomes **chosen by the caller**: anyone can send `X-Forwarded-For: 198.51.100.7` and have that
|
||||
written into session records, audit rows and the key the anonymous WebAuthn budget counts on.
|
||||
|
||||
**Do not add session affinity.** It is the reflex the moment websockets are involved and it solves
|
||||
nothing here: the hard case is co-locating an operator's browser with an *agent's* websocket, which
|
||||
arrives from a different network at a different time and whose id is not even known at the browser's
|
||||
handshake. No ingress annotation can express that. It is handled in the application, over Redis.
|
||||
Both ends pin `transports: ["websocket"]`, so there is no polling to make sticky in the first place.
|
||||
|
||||
## On exactly three nodes
|
||||
|
||||
The common case, and where the arithmetic is tightest.
|
||||
|
||||
**What lands where.** Redis and the sentinels both use *required* anti-affinity, so three of each
|
||||
means one per node — six pods, two per node, no choice left to the scheduler. Postgres at three
|
||||
instances is the same. That is what you want, and it has a consequence: **during a node drain, one
|
||||
Redis pod and one sentinel have nowhere to go and stay `Pending` until the node returns.** That is
|
||||
correct — a quorum of two still holds — but `kubectl drain` will sit there unless the
|
||||
PodDisruptionBudgets are applied to tell it what is safe.
|
||||
|
||||
**Three api replicas, not two.** [`04-api.yaml`](04-api.yaml) ships two because that is the smallest
|
||||
number that proves the code is not single-instance. On three nodes, three is strictly better: one
|
||||
per node, and losing any node leaves two. Change `replicas: 2` to `3` in it and in
|
||||
[`05-web.yaml`](05-web.yaml), and size `max_connections` for it.
|
||||
|
||||
**Three nodes tolerates losing one.** Not two. That is a property of majority quorums, and the
|
||||
answer to wanting more is a fourth and fifth node, not a different configuration.
|
||||
|
||||
## What a failover actually costs
|
||||
|
||||
Measured, not estimated.
|
||||
|
||||
**Redis**, with its primary killed under a running api:
|
||||
|
||||
- **The api rides it out without restarting.** Readiness answered 503 for about fifteen seconds and
|
||||
then 200 again, on the same process — the pod leaves the Service's endpoints while it cannot reach
|
||||
Redis, and returns when it can. Liveness never touches Redis, which is why the pod is not killed
|
||||
and restarted into a cold start.
|
||||
- **Commands survive.** 99 writes issued across the window, 0 rejected, the last readable from the
|
||||
promoted primary afterwards. Commands queue rather than fail, so a failover is a pause.
|
||||
- **Published messages do not.** Around 45% of the messages published during the window reached
|
||||
nobody. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change
|
||||
that. In practice: a few seconds in which something happening on one api replica may not reach
|
||||
another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason.
|
||||
|
||||
**Postgres**, primary deleted with `--force --grace-period=0`:
|
||||
|
||||
- **Three seconds.** Both api pods answered `503 {"database":"unreachable"}` and were back to 200 on
|
||||
the fourth poll. Both degraded together, which is what pointing every pod at one endpoint means.
|
||||
- **Zero container restarts**, across two failovers, on pods up throughout — liveness deliberately
|
||||
not touching the database is what keeps three seconds from becoming a cold start of everything.
|
||||
|
||||
A failover *during* the migration Job aborts it with a lost-connection error, which is the correct
|
||||
outcome. Run it again.
|
||||
|
||||
## What is still worth knowing
|
||||
|
||||
- **`AGENT_RELEASE_DIR` is per-pod, and must be.** It holds the compiled agent binaries. An
|
||||
`emptyDir` filled by an initContainer is right; a shared RWO volume is not — either every pod is
|
||||
pinned to one node, or the second sits in `Multi-Attach error`. Leaving it unset is supported:
|
||||
everything works except the agent installer, which answers 503 saying no build is published.
|
||||
- **No application volume, and it should stay that way.** Avatars, documents and terminal recordings
|
||||
are all in Postgres; exports are rendered in memory.
|
||||
- **Autoscale `jarvis-web`, not `jarvis-api`.** Every api scale-in kills a pod holding agent
|
||||
websockets, live runs and open shells, and CPU is a poor proxy for a load made of long-lived
|
||||
connections. Raise api replicas deliberately.
|
||||
- **A rollout still cuts what is in flight.** An assistant run is interrupted and resumed with a
|
||||
note; a terminal is not resumable and says so; and the web bundle's content-hashed chunks are
|
||||
per-image, so a browser holding the old shell can 404 on a lazy import during the rollout window.
|
||||
- **Validation is not admission.** These manifests satisfy the `restricted` Pod Security Standard
|
||||
that [`00-namespace.yaml`](00-namespace.yaml) enforces. If you edit them, note that a schema
|
||||
validator cannot see that policy: a missing `seccompProfile` or a container without a
|
||||
`securityContext` does not fail validation, it produces a Deployment reporting `FailedCreate` and
|
||||
sitting at zero replicas.
|
||||
|
||||
## Backups
|
||||
|
||||
Everything in [the main README](../README.md#backups-and-restoring-one) applies, with one
|
||||
substitution: the vault key is not in a Docker volume, it is in the `jarvis-generated` Secret.
|
||||
|
||||
```sh
|
||||
kubectl -n jarvis get secret jarvis-generated -o jsonpath='{.data.VAULT_MASTER_KEY}' | base64 -d
|
||||
```
|
||||
|
||||
Back that up somewhere that is not this cluster, and not alongside your database dump. A database
|
||||
backup that travelled with its own key is a backup that decrypts itself.
|
||||
Reference in New Issue
Block a user