Sync the self-hosting stack

kubernetes/02-config.yaml

@@ -18,17 +18,24 @@ data:
   #
   # A SINGLE REDIS IS A SINGLE POINT OF FAILURE FOR EVERY POD AT ONCE. This line is correct for a
   # managed Redis that fails over behind one address, and correct-but-fragile for one Deployment of
-  # one Redis. For Sentinel, comment this out and set the two below instead — see
-  # `08-redis-sentinel.yaml`, which deploys exactly that.
+  # one Redis. It is IGNORED while the two Sentinel lines below are set — and they are, because
+  # `kustomization.yaml` deploys `08-redis-sentinel.yaml`. With a managed Redis instead: remove that
+  # file from `kustomization.yaml`, delete the two lines below, and point this one at it.
+  #
+  # NEVER THIS LINE ALONE WITH 08 DEPLOYED. Its `redis` Service is headless: the name answers with
+  # all three Redis pods, and two of them are read-only replicas. Each api connection lands on one
+  # of the three, so a pod that drew a replica fails every write it makes — `READONLY You can't
+  # write against a read only replica` — while its readiness probe, which only reads, stays green.
+  # That was this file's default while the two lines below were commented out.
   REDIS_URL: "redis://redis.jarvis.svc.cluster.local:6379"
 
   # Set BOTH or neither. Sentinel needs the sentinels and the name of the primary they watch; a
   # name is refused rather than defaulted, because ioredis's own default (`mymaster`) against a
   # primary called anything else is not an error — it is a pod that never becomes ready, silently.
-  # When these are set, REDIS_URL above is ignored.
-  #
-  #REDIS_SENTINELS: "redis-sentinel-0.redis-sentinel.jarvis.svc.cluster.local:26379,redis-sentinel-1.redis-sentinel.jarvis.svc.cluster.local:26379,redis-sentinel-2.redis-sentinel.jarvis.svc.cluster.local:26379"
-  #REDIS_SENTINEL_NAME: "jarvis"
+  # When these are set, REDIS_URL above is ignored. They name what `08-redis-sentinel.yaml` deploys:
+  # its three sentinels, and `jarvis`, the primary they monitor.
+  REDIS_SENTINELS: "redis-sentinel-0.redis-sentinel.jarvis.svc.cluster.local:26379,redis-sentinel-1.redis-sentinel.jarvis.svc.cluster.local:26379,redis-sentinel-2.redis-sentinel.jarvis.svc.cluster.local:26379"
+  REDIS_SENTINEL_NAME: "jarvis"
 
   # ONE, not the two the compose files use, and getting this wrong is a security fault rather than
   # a cosmetic one.

kubernetes/05-web.yaml

@@ -147,9 +147,16 @@ spec:
 apiVersion: autoscaling/v2
 kind: HorizontalPodAutoscaler
 metadata:
-  # With the Deployment it belongs to.
+  # A WAVE AFTER the Deployment it scales, in the last one, and that is the whole point of the line.
+  #
+  # Argo waits for every resource of a wave to report healthy before it starts the next, and reads
+  # an HPA as Degraded while the HPA reads no metric for its pods. On a FIRST install those pods are
+  # seconds old and metrics-server has not scraped them yet ("the HPA was unable to compute the
+  # replica count … no metrics returned"), so in wave 1 the first sync failed on it every time, and
+  # only an automated policy's retry got past it. The last wave is applied and not waited on: the
+  # sync succeeds, and the HPA turns healthy by itself within a minute. Nothing reads it before.
   annotations:
-    argocd.argoproj.io/sync-wave: "1"
+    argocd.argoproj.io/sync-wave: "2"
   name: jarvis-web
   namespace: jarvis
 spec:

kubernetes/08-redis-sentinel.yaml

@@ -17,7 +17,11 @@
 # during the switch — see "What a failover actually costs" at the bottom.
 #
 #   kubectl apply -f 08-redis-sentinel.yaml
-#   # then in 02-config.yaml: comment out REDIS_URL, uncomment the two REDIS_SENTINEL* lines
+#
+# 02-config.yaml points the api at it already: its two REDIS_SENTINEL* lines name these sentinels
+# and the primary they monitor. Skip this file for a managed Redis, and delete those two lines with
+# it — while they are set, REDIS_URL is ignored. Never point REDIS_URL at the `redis` Service below
+# instead: it is headless, and two of the three pods it answers with are read-only replicas.
 #
 # ──────────────────────────────────────────────────────────────────────────────────────────────
 apiVersion: v1

kubernetes/README.md

@@ -50,10 +50,13 @@ and an index can only do that because one server decides. Do not point pods at r
   `(replicas + maxSurge + jobs) × connection_limit + reserve`: with 3 api pods at 10 connections
   each, a migration Job at 5, and headroom, about **150**. The stock Postgres image ships 100, and
   exhausting it presents as an api that starts and then cannot serve.
-- **Redis.** [`08-redis-sentinel.yaml`](08-redis-sentinel.yaml) deploys one, or point `REDIS_URL` at
-  a managed Redis that fails over behind a single address and skip that file. Persistence is not
-  required: everything Jarvis keeps in Redis is ephemeral by design, which is exactly why the run
-  lock is in Postgres instead.
+- **Redis.** [`08-redis-sentinel.yaml`](08-redis-sentinel.yaml) deploys one — three Redis, one of
+  them the primary, and three sentinels — and `02-config.yaml`'s two `REDIS_SENTINEL*` lines point
+  the api at it. Or use a managed Redis that fails over behind a single address: skip that file,
+  delete those two lines, and set `REDIS_URL` to it. Never `REDIS_URL` alone at the file's `redis`
+  Service: it is headless, two of the three pods it answers with are read-only replicas, and an api
+  pod that draws one fails every write. Persistence is not required: everything Jarvis keeps in
+  Redis is ephemeral by design, which is exactly why the run lock is in Postgres instead.
 - **An Ingress controller and a TLS terminator.** Jarvis speaks plain HTTP.
 
 ## Order
@@ -67,7 +70,7 @@ kubectl apply -f 01-secret-job.yaml       # the half that is random bytes (optio
 kubectl wait --for=condition=complete job/jarvis-mint-secrets -n jarvis --timeout=2m
 
 kubectl apply -f 02-config.yaml
-kubectl apply -f 08-redis-sentinel.yaml   # only if you are running your own Redis
+kubectl apply -f 08-redis-sentinel.yaml   # unless you use a managed Redis (then edit 02-config.yaml)
 
 # The schema, to completion, before any api pod starts.
 kubectl apply -f 03-migrate-job.yaml
@@ -130,10 +133,14 @@ thing being ordered:
 | `-1` | `jarvis-mint-secrets` **(hook)** | the migration reads `DATABASE_URL` out of what it writes |
 | `0` | `jarvis-migrate` **(hook)** | an api pod against an unmigrated schema is the failure all of this prevents |
 | `1` | api and web Deployments, Services | the web pod's `/readyz` proxies the api's health endpoint, so it cannot be ready before the api is |
-| `2` | Ingress, PodDisruptionBudgets | an Ingress applied early is a route to a 503 |
+| `2` | Ingress, PodDisruptionBudgets, the web's HPA | an Ingress applied early is a route to a 503; the HPA, below |
 
 Argo does not begin a wave until the previous one reports healthy, which is what the `kubectl wait`
-lines are doing by hand.
+lines are doing by hand. The last wave is applied and not waited on, which is why the web's
+HorizontalPodAutoscaler is there rather than beside its Deployment: on a first install it has no
+metric for pods seconds old, Argo reads that as Degraded, and in wave 1 it failed the first sync
+every time. The application can read Degraded for the minute after a first install, until
+metrics-server has scraped the new pods; the sync itself has succeeded.
 
 ### Why the two Jobs are hooks
 

kubernetes/kustomization.yaml

@@ -42,8 +42,9 @@ resources:
   # host along with the main one.
   - 06b-ingress-uploads.yaml
   - 07-disruption.yaml
-  # Remove this line if you point REDIS_URL at a managed Redis. It deploys three Redis processes and
-  # three sentinels; see the README on what that costs on three nodes.
+  # Three Redis processes and three sentinels; see the README on what that costs on three nodes.
+  # 02-config.yaml's two REDIS_SENTINEL* lines point the api at them. For a managed Redis instead,
+  # remove this line AND those two lines, and set REDIS_URL there to your Redis.
   - 08-redis-sentinel.yaml
   # The headless browser and the NetworkPolicy that isolates it. COMMENTED OUT: web access is a
   # decision rather than a default, and an assistant that can read the internet is a new class of