Sync the self-hosting stack

kubernetes/01-secret-job.yaml

@@ -129,8 +129,8 @@ spec:
           # Exec form: `jarvis` is a bash script that execs the API, and a `/bin/sh -c` (dash) in between
           # would drop the image's dotted logging variables. It verifies the API server's certificate
           # against the service account's own ca.crt, and trusts nothing else for it — so no variable
-          # names the CA (Node's NODE_EXTRA_CA_CERTS went with the TypeScript image). A bearer token good
-          # enough to write Secrets is a bearer token worth stealing: never turn that check off.
+          # names the CA and there is none to set. A bearer token good enough to write Secrets is a
+          # bearer token worth stealing: never turn that check off.
           command: ["jarvis", "mint-k8s-secret"]
           resources:
             requests:

kubernetes/03-migrate-job.yaml

@@ -2,9 +2,8 @@
 #
 # A JOB, and deliberately neither of the two alternatives:
 #
-#   NOT an initContainer. It would run once per pod — which is exactly the bug this replaces, where
-#   every API container ran `prisma db push` at boot and N replicas raced the same DDL while the
-#   previous generation was still selecting the columns being altered.
+#   NOT an initContainer. It would run once per pod: N replicas would race the same DDL while the
+#   previous generation is still selecting the columns being altered.
 #
 #   NOT a leader election inside the API. A Job is the primitive built for run-once work: it has an
 #   observable status, a bounded number of retries, and a failure that stops the rollout instead of
@@ -89,10 +88,9 @@ spec:
                 secretKeyRef:
                   name: jarvis-secrets
                   key: DATABASE_URL
-            # Off, and it should stay off between deliberate acts. `prisma db push` used to run with
-            # `--accept-data-loss` at every boot, which meant an upgrade could destroy a column with
-            # nobody having decided to — survivable on one container, not during a rolling update
-            # where the previous generation is still selecting it.
+            # Off, and it should stay off between deliberate acts. Left on, every upgrade could
+            # destroy a column with nobody having decided to — survivable on one container, not
+            # during a rolling update where the previous generation is still selecting it.
             #
             # Without it a pending migration that drops a table or a column is REFUSED, naming them,
             # and fails the Job — so the rollout stops and nothing is serving a half-migrated schema.

kubernetes/04-api.yaml

@@ -14,21 +14,13 @@ spec:
     matchLabels:
       app.kubernetes.io/name: jarvis
       app.kubernetes.io/component: api
-  # RECREATED, FOR api 1.0.0 ONLY — the release that crosses from the TypeScript images. A rolling
-  # update would keep a TypeScript pod and a .NET pod serving side by side behind one Service, and
-  # their realtime protocols differ under /socket.io/ (Socket.IO against SignalR): every old pod stops
-  # before the first new one starts, a gap of about thirty seconds, once. The next release puts the
-  # rolling update back:
-  #
-  #   type: RollingUpdate
-  #   rollingUpdate:
-  #     maxUnavailable: 0   # never fewer pods than are being served now
-  #     maxSurge: 1
-  #
-  # With `kubectl apply` or Argo's default client-side apply this switches both ways. Under
-  # server-side apply, a field manager that owns `rollingUpdate` makes the API server refuse it.
   strategy:
-    type: Recreate
+    type: RollingUpdate
+    rollingUpdate:
+      # Never fewer pods than are being served now. A run holds its conversation through a unique
+      # index, so a surge pod cannot double-answer anything.
+      maxUnavailable: 0
+      maxSurge: 1
   template:
     metadata:
       labels:
@@ -156,14 +148,11 @@ spec:
               # `Files/PdfTextWorker.cs`) that is killed past its own limits, so a hostile file costs
               # that process and not this one.
               #
-              # MEASURED on this image, in a cgroup like this pod's (lot 11 P, proof 2j): every upload
-              # slot of two pods busy with a 10 MB, 500-page PDF — the most pages the worker reads —
-              # peaked at 619 MiB and 740 MiB of `memory.current`, the API and its workers together,
-              # from 446 and 496 MiB before. The TypeScript image sized this limit: Prisma's JSON
-              # protocol copied an upload's bytes several times over (~1.1 GiB for three at once)
-              # and a PDF worker's buffers could reach ~500 MB, ~2.75 GiB with every holder full.
-              # None of that is in this process; the 3Gi limit stands until it is lowered on purpose
-              # (docs/dotnet/PLAN.md, "Left to Antoine after lot 11"). The request is what the
+              # MEASURED on this image, in a cgroup like this pod's: every upload slot of two pods
+              # busy with a 10 MB, 500-page PDF — the most pages the worker reads — peaked at 619 MiB
+              # and 740 MiB of `memory.current`, the API and its workers together, from 446 and
+              # 496 MiB before. The 3Gi limit was sized generously and sits well above that
+              # 0.6–0.75 GiB peak; it stands until it is lowered on purpose. The request is what the
               # scheduler sets aside for an ordinary day, not for the worst minute.
               memory: 1Gi
             limits:

kubernetes/08-redis-sentinel.yaml

@@ -10,19 +10,15 @@
 # somebody else to hold that, use a managed Redis or an operator. What this directory owes you is
 # the shape the application needs, and this is it.
 #
-# The application half is proven rather than assumed, though not all of it on the api that ships
-# now. On the TypeScript api, an ioredis client built exactly the way its `redis-connection.ts`
-# built one was run against a live three-sentinel topology while its primary was killed. It
-# followed the promotion to a different server, and 99 writes issued across the window were
-# rejected zero times. What it does NOT do is preserve pub/sub during the switch — see the failover
-# section of the README beside this file.
-#
-# The .NET api reaches Redis through StackExchange.Redis, asking these sentinels for the primary
-# (src/Jarvis.Api/Infrastructure/RedisConnection.cs). Proven for it, on a throwaway cluster synced
-# from these manifests with 02-config.yaml's two REDIS_SENTINEL* lines: every pod wrote to the
-# primary — its leases taken, the console's realtime socket open, no READONLY refusal — where
-# REDIS_URL alone, on the headless Service below, landed writes on a read-only replica. Its
-# failover has NOT been measured: that it follows a promotion as ioredis did is not known.
+# The application half is proven as far as it is claimed, and no further. The api reaches Redis
+# through StackExchange.Redis, asking these sentinels for the primary
+# (src/Jarvis.Api/Infrastructure/RedisConnection.cs). Proven on a throwaway cluster synced from
+# these manifests with 02-config.yaml's two REDIS_SENTINEL* lines: every pod wrote to the primary —
+# its leases taken, the console's realtime socket open, no READONLY refusal — where REDIS_URL
+# alone, on the headless Service below, landed writes on a read-only replica. Its failover has NOT
+# been measured: how fast it follows a promotion to a different server, and what a write issued
+# across the window meets, is not known. Pub/sub does not survive the switch whatever the client —
+# see the failover section of the README beside this file.
 #
 #   kubectl apply -f 08-redis-sentinel.yaml
 #

kubernetes/README.md

@@ -10,12 +10,12 @@ takes one command, and is what most instances run. Come here when you want Jarvi
 while a node reboots, and when you already operate a cluster — this directory assumes you have
 opinions about ingress, storage and a database, because it deliberately does not have them for you.
 
-> **What has been proven, and what has not.** This stack has been brought up on a real cluster —
-> four nodes, three Postgres instances, three Redis with three sentinels, two api pods — and kept
-> serving while its Postgres primary and its Redis primary were killed underneath it. What has NOT
-> happened is a long-running estate serving production traffic this way. If you are early, watch it
-> rather than assume it, and read the three warnings below before you apply anything, because each
-> of them fails silently.
+> **What has been proven, and what has not.** This stack has been brought up on a throwaway
+> four-node cluster — three Redis with three sentinels, two api pods behind the Ingress, synced by
+> Argo CD — and served the console and its realtime connections through it. What has NOT happened
+> is a Postgres or a Redis failover measured under the current api, or a long-running estate serving
+> production traffic this way. If you are early, watch it rather than assume it, and read the three
+> warnings below before you apply anything, because each of them fails silently.
 
 ## What runs more than once, and what cannot
 
@@ -261,31 +261,30 @@ per node, and losing any node leaves two. Change `replicas: 2` to `3` in it and
 **Three nodes tolerates losing one.** Not two. That is a property of majority quorums, and the
 answer to wanting more is a fourth and fifth node, not a different configuration.
 
-## What a failover actually costs
+## What a failover costs
 
-Measured, not estimated — on the api releases before 1.0.0, which were written in TypeScript. The
-readiness and liveness design below is unchanged in 1.0.0; the figures have not been measured again
-since.
+**Not measured on the current api**: no figure here would be one. What holds by design, and what
+Redis and Postgres do on their own, is this — how long each window lasts on your cluster is yours to
+measure.
 
-**Redis**, with its primary killed under a running api:
+**Redis**, its primary lost under a running api:
 
-- **The api rides it out without restarting.** Readiness answered 503 for about fifteen seconds and
-  then 200 again, on the same process — the pod leaves the Service's endpoints while it cannot reach
-  Redis, and returns when it can. Liveness never touches Redis, which is why the pod is not killed
-  and restarted into a cold start.
-- **Commands survive.** 99 writes issued across the window, 0 rejected, the last readable from the
-  promoted primary afterwards. Commands queue rather than fail, so a failover is a pause.
-- **Published messages do not.** Around 45% of the messages published during the window reached
-  nobody. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change
-  that. In practice: a few seconds in which something happening on one api replica may not reach
+- **The api rides it out without restarting.** `/api/health/ready` answers 503 with `"redis":
+  "unreachable"` while the api cannot reach Redis, so the pod leaves the Service's endpoints and
+  returns when it can. Liveness never touches Redis, so the pod is not killed and restarted into a
+  cold start.
+- **Messages published during the window are lost.** Redis pub/sub has no buffer and no redelivery,
+  and nothing on the client side can change that: for a few seconds, live updates — the chat
+  stream, presence, typing — reach nobody, and something happening on one api replica may not reach
   another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason.
 
-**Postgres**, primary deleted with `--force --grace-period=0`:
+**Postgres**, its primary lost:
 
-- **Three seconds.** Both api pods answered `503 {"database":"unreachable"}` and were back to 200 on
-  the fourth poll. Both degraded together, which is what pointing every pod at one endpoint means.
-- **Zero container restarts**, across two failovers, on pods up throughout — liveness deliberately
-  not touching the database is what keeps three seconds from becoming a cold start of everything.
+- **Every api pod degrades together**: readiness answers `503` with `"database": "unreachable"` until
+  the promoted primary answers at the address `DATABASE_URL` names, which is what pointing every pod
+  at one endpoint means.
+- **No restart**: liveness deliberately does not touch the database, which is what keeps a failover
+  from becoming a cold start of everything.
 
 A failover *during* the migration Job aborts it with a lost-connection error, which is the correct
 outcome. Run it again.