Sync the self-hosting stack

kubernetes/README.md

@@ -10,12 +10,12 @@ takes one command, and is what most instances run. Come here when you want Jarvi
 while a node reboots, and when you already operate a cluster — this directory assumes you have
 opinions about ingress, storage and a database, because it deliberately does not have them for you.
 
-> **What has been proven, and what has not.** This stack has been brought up on a real cluster —
-> four nodes, three Postgres instances, three Redis with three sentinels, two api pods — and kept
-> serving while its Postgres primary and its Redis primary were killed underneath it. What has NOT
-> happened is a long-running estate serving production traffic this way. If you are early, watch it
-> rather than assume it, and read the three warnings below before you apply anything, because each
-> of them fails silently.
+> **What has been proven, and what has not.** This stack has been brought up on a throwaway
+> four-node cluster — three Redis with three sentinels, two api pods behind the Ingress, synced by
+> Argo CD — and served the console and its realtime connections through it. What has NOT happened
+> is a Postgres or a Redis failover measured under the current api, or a long-running estate serving
+> production traffic this way. If you are early, watch it rather than assume it, and read the three
+> warnings below before you apply anything, because each of them fails silently.
 
 ## What runs more than once, and what cannot
 
@@ -261,31 +261,30 @@ per node, and losing any node leaves two. Change `replicas: 2` to `3` in it and
 **Three nodes tolerates losing one.** Not two. That is a property of majority quorums, and the
 answer to wanting more is a fourth and fifth node, not a different configuration.
 
-## What a failover actually costs
+## What a failover costs
 
-Measured, not estimated — on the api releases before 1.0.0, which were written in TypeScript. The
-readiness and liveness design below is unchanged in 1.0.0; the figures have not been measured again
-since.
+**Not measured on the current api**: no figure here would be one. What holds by design, and what
+Redis and Postgres do on their own, is this — how long each window lasts on your cluster is yours to
+measure.
 
-**Redis**, with its primary killed under a running api:
+**Redis**, its primary lost under a running api:
 
-- **The api rides it out without restarting.** Readiness answered 503 for about fifteen seconds and
-  then 200 again, on the same process — the pod leaves the Service's endpoints while it cannot reach
-  Redis, and returns when it can. Liveness never touches Redis, which is why the pod is not killed
-  and restarted into a cold start.
-- **Commands survive.** 99 writes issued across the window, 0 rejected, the last readable from the
-  promoted primary afterwards. Commands queue rather than fail, so a failover is a pause.
-- **Published messages do not.** Around 45% of the messages published during the window reached
-  nobody. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change
-  that. In practice: a few seconds in which something happening on one api replica may not reach
+- **The api rides it out without restarting.** `/api/health/ready` answers 503 with `"redis":
+  "unreachable"` while the api cannot reach Redis, so the pod leaves the Service's endpoints and
+  returns when it can. Liveness never touches Redis, so the pod is not killed and restarted into a
+  cold start.
+- **Messages published during the window are lost.** Redis pub/sub has no buffer and no redelivery,
+  and nothing on the client side can change that: for a few seconds, live updates — the chat
+  stream, presence, typing — reach nobody, and something happening on one api replica may not reach
   another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason.
 
-**Postgres**, primary deleted with `--force --grace-period=0`:
+**Postgres**, its primary lost:
 
-- **Three seconds.** Both api pods answered `503 {"database":"unreachable"}` and were back to 200 on
-  the fourth poll. Both degraded together, which is what pointing every pod at one endpoint means.
-- **Zero container restarts**, across two failovers, on pods up throughout — liveness deliberately
-  not touching the database is what keeps three seconds from becoming a cold start of everything.
+- **Every api pod degrades together**: readiness answers `503` with `"database": "unreachable"` until
+  the promoted primary answers at the address `DATABASE_URL` names, which is what pointing every pod
+  at one endpoint means.
+- **No restart**: liveness deliberately does not touch the database, which is what keeps a failover
+  from becoming a cold start of everything.
 
 A failover *during* the migration Job aborts it with a lost-connection error, which is the correct
 outcome. Run it again.