Sync the self-hosting stack
This commit is contained in:
@@ -2,9 +2,9 @@
|
||||
#
|
||||
# ─── WHAT THIS REPLACES ───────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# On Compose, `init-secrets.cjs` runs as a one-shot writing into a volume every container shares,
|
||||
# and its guarantee that a key is generated exactly once is `openSync(path, "wx")` — two writers
|
||||
# race, the kernel picks one. There is no shared filesystem across pods, so the exclusion moves to
|
||||
# On Compose, `jarvis init-secrets` runs as a one-shot writing into a volume every container shares,
|
||||
# and its guarantee that a key is generated exactly once is creating a file that must not exist yet
|
||||
# (`O_EXCL`) — two writers race, the kernel picks one. There is no shared filesystem across pods, so the exclusion moves to
|
||||
# the one thing every pod does share: the API server. Creating a Secret that already exists is a
|
||||
# 409, decided by one serialized writer. Same guarantee, same strength.
|
||||
#
|
||||
@@ -50,7 +50,7 @@ rules:
|
||||
# `create` AND NOTHING ELSE. No `get`, no `list`, no `update`.
|
||||
#
|
||||
# Deliberate, and worth the paragraph: this identity cannot read any Secret in the namespace,
|
||||
# including the one it just wrote. That is why the script does not look before it writes — the
|
||||
# including the one it just wrote. That is why the command does not look before it writes — the
|
||||
# 409 is the check, and it is a stronger one than a read-then-write could be, because a read and
|
||||
# a write are two operations and something else can happen between them.
|
||||
#
|
||||
@@ -122,18 +122,16 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: mint
|
||||
# The api image, because the script ships in it and a first deployment should pull one
|
||||
# The api image, because the command ships in it and a first deployment should pull one
|
||||
# image rather than three. By digest like every other image here — the same one you put
|
||||
# in 03-migrate-job.yaml and 04-api.yaml.
|
||||
image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME
|
||||
command: ["node", "/app/apps/api/mint-k8s-secret.cjs"]
|
||||
env:
|
||||
# Node verifies the API server's certificate against the cluster CA. Without this the
|
||||
# request would fail — which is the correct failure. Do not reach for
|
||||
# NODE_TLS_REJECT_UNAUTHORIZED: a bearer token good enough to write Secrets is a bearer
|
||||
# token worth stealing, and an unverified TLS session is where that happens.
|
||||
- name: NODE_EXTRA_CA_CERTS
|
||||
value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
# Exec form: `jarvis` is a bash script that execs the API, and a `/bin/sh -c` (dash) in between
|
||||
# would drop the image's dotted logging variables. It verifies the API server's certificate
|
||||
# against the service account's own ca.crt, and trusts nothing else for it — so no variable
|
||||
# names the CA (Node's NODE_EXTRA_CA_CERTS went with the TypeScript image). A bearer token good
|
||||
# enough to write Secrets is a bearer token worth stealing: never turn that check off.
|
||||
command: ["jarvis", "mint-k8s-secret"]
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
|
||||
@@ -11,13 +11,13 @@
|
||||
# entirely. Do not do both: `04-api.yaml` reads this Secret last on purpose, so a value typed here
|
||||
# wins over a generated one, and the two quietly disagreeing is the shape of the accident.
|
||||
#
|
||||
# `apps/api/init-secrets.cjs` is the Compose generator, and it must NEVER become an initContainer
|
||||
# `jarvis init-secrets` is the Compose generator, and it must NEVER become an initContainer
|
||||
# here. Its guarantee that a key is minted exactly once is every writer sharing one filesystem.
|
||||
# Per-pod, on an `emptyDir`, each replica generates its own — and the consequence is not a crash.
|
||||
# Pod A seals a vault credential under a key pod B does not have; pod B reports that credential as
|
||||
# corrupt; both pods log nothing at boot, because from each one's point of view everything is fine.
|
||||
# By the time anybody notices, there are several keys in circulation and no way to tell which
|
||||
# entries belong to which. `apps/api/mint-k8s-secret.cjs` is the same one-shot done correctly, with
|
||||
# entries belong to which. `jarvis mint-k8s-secret` is the same one-shot done correctly, with
|
||||
# the API server's 409 standing in for the filesystem's exclusion.
|
||||
#
|
||||
# Either way, keep the output. The vault key in particular is the thing every stored credential is
|
||||
@@ -71,7 +71,7 @@ stringData:
|
||||
VAULT_KEY_CREATED_AT: "REPLACE_ME"
|
||||
|
||||
# `connection_limit` is not optional here, and it is the reason this URL is in the Secret rather
|
||||
# than assembled from parts. Prisma's default pool is per-process: N pods times that default will
|
||||
# than assembled from parts. The api's pool is per-process: N pods times its default will
|
||||
# exhaust `max_connections` long before the application is under any real load. Size it as
|
||||
# (max_connections − reserve) ÷ (replicas + maxSurge + jobs).
|
||||
DATABASE_URL: "postgresql://jarvis:REPLACE_ME@postgres.jarvis.svc.cluster.local:5432/jarvis?schema=public&connection_limit=10&pool_timeout=20"
|
||||
|
||||
@@ -9,6 +9,7 @@ metadata:
|
||||
name: jarvis-api-config
|
||||
namespace: jarvis
|
||||
data:
|
||||
# Read by the api under this name: the refresh cookie is `Secure` only in production.
|
||||
NODE_ENV: "production"
|
||||
API_PORT: "4000"
|
||||
|
||||
@@ -30,7 +31,7 @@ data:
|
||||
REDIS_URL: "redis://redis.jarvis.svc.cluster.local:6379"
|
||||
|
||||
# Set BOTH or neither. Sentinel needs the sentinels and the name of the primary they watch; a
|
||||
# name is refused rather than defaulted, because ioredis's own default (`mymaster`) against a
|
||||
# name is refused rather than defaulted, because a client's usual default (`mymaster`) against a
|
||||
# primary called anything else is not an error — it is a pod that never becomes ready, silently.
|
||||
# When these are set, REDIS_URL above is ignored. They name what `08-redis-sentinel.yaml` deploys:
|
||||
# its three sentinels, and `jarvis`, the primary they monitor.
|
||||
|
||||
@@ -46,7 +46,7 @@ spec:
|
||||
parallelism: 1
|
||||
completions: 1
|
||||
# Two retries, because the failures worth retrying are transient (a database not finished failing
|
||||
# over). A schema that Prisma REFUSES is not transient and must not be retried into submission —
|
||||
# over). A schema `jarvis migrate` REFUSES is not transient and must not be retried into submission —
|
||||
# it is refused for a reason, and the reason is printed.
|
||||
backoffLimit: 2
|
||||
# A migration that has not finished in fifteen minutes is stuck, not slow, and the rollout is
|
||||
@@ -70,7 +70,7 @@ spec:
|
||||
# THE SAME IMAGE, by the same digest, as the API that is about to roll. A migration from a
|
||||
# different build is a schema that matches nothing.
|
||||
image: git.luxit.be/luxit/jarvis-api@sha256:REPLACE_ME
|
||||
command: ["/usr/local/bin/migrate.sh"]
|
||||
command: ["jarvis", "migrate"]
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: jarvis-api-config
|
||||
@@ -94,8 +94,8 @@ spec:
|
||||
# nobody having decided to — survivable on one container, not during a rolling update
|
||||
# where the previous generation is still selecting it.
|
||||
#
|
||||
# Without the flag the phase REFUSES, prints which columns it would have dropped, and
|
||||
# fails the Job — so the rollout stops and nothing is serving a half-migrated schema.
|
||||
# Without it a pending migration that drops a table or a column is REFUSED, naming them,
|
||||
# and fails the Job — so the rollout stops and nothing is serving a half-migrated schema.
|
||||
# Set it to "true" for the ONE deploy where that drop is intended, then remove it.
|
||||
- name: SCHEMA_ACCEPT_DATA_LOSS
|
||||
value: "false"
|
||||
|
||||
+34
-50
@@ -14,14 +14,21 @@ spec:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: api
|
||||
# RECREATED, FOR api 1.0.0 ONLY — the release that crosses from the TypeScript images. A rolling
|
||||
# update would keep a TypeScript pod and a .NET pod serving side by side behind one Service, and
|
||||
# their realtime protocols differ under /socket.io/ (Socket.IO against SignalR): every old pod stops
|
||||
# before the first new one starts, a gap of about thirty seconds, once. The next release puts the
|
||||
# rolling update back:
|
||||
#
|
||||
# type: RollingUpdate
|
||||
# rollingUpdate:
|
||||
# maxUnavailable: 0 # never fewer pods than are being served now
|
||||
# maxSurge: 1
|
||||
#
|
||||
# With `kubectl apply` or Argo's default client-side apply this switches both ways. Under
|
||||
# server-side apply, a field manager that owns `rollingUpdate` makes the API server refuse it.
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
# Never fewer pods than are being served now. A run holds its conversation through a unique
|
||||
# index, so a surge pod cannot double-answer anything — the old constraint that made an
|
||||
# overlap dangerous is gone.
|
||||
maxUnavailable: 0
|
||||
maxSurge: 1
|
||||
type: Recreate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
@@ -31,7 +38,7 @@ spec:
|
||||
# 90 seconds, and every part of it is accounted for:
|
||||
# 10s preStop, so this pod leaves the Service's endpoints BEFORE it starts draining
|
||||
# 25s RUN_SHUTDOWN_GRACE_SEC — in-flight assistant runs stop and close their transcripts
|
||||
# ~15s agent sockets closed and their rows written, httpServer.close(), Prisma, Redis
|
||||
# ~15s agent sockets closed and their rows written, Kestrel, the database pool, Redis
|
||||
# the rest is margin
|
||||
# The default of 30 would SIGKILL the pod in the middle of the run drain, which is the one
|
||||
# thing the drain exists to prevent.
|
||||
@@ -75,14 +82,14 @@ spec:
|
||||
preStop:
|
||||
exec:
|
||||
# Endpoint removal and SIGTERM race each other, and losing that race is visible:
|
||||
# a prompt that arrives after Nest sets `draining` is persisted and broadcast as
|
||||
# a prompt that arrives after the api starts draining is persisted and broadcast as
|
||||
# queued, and this pod will never answer it. Ten seconds is comfortably more than
|
||||
# kube-proxy needs to stop sending new connections here.
|
||||
command: ["/bin/sh", "-c", "sleep 10"]
|
||||
env:
|
||||
- name: JARVIS_SKIP_MIGRATIONS
|
||||
# The schema is the migration Job's business, once, before this rolls. N pods each
|
||||
# running `prisma db push` is N writers racing the same DDL while the previous
|
||||
# migrating is N writers racing the same DDL while the previous
|
||||
# generation is still selecting the columns being altered.
|
||||
value: "true"
|
||||
envFrom:
|
||||
@@ -139,48 +146,25 @@ spec:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
# Not generous — honest. A terminal recording holds up to 8 MiB of Buffers in heap
|
||||
# for the life of the session, and argon2 takes 64 MiB per share verification.
|
||||
# Not generous — honest. A terminal recording holds up to 8 MiB in memory for the life
|
||||
# of the session, and argon2 takes 64 MiB per share verification.
|
||||
#
|
||||
# Attached and uploaded files add four holders, every one capped, so that their sum is
|
||||
# a number rather than a hope. Their figures are MEASURED on this image, in a cgroup
|
||||
# like this pod's, not estimated — the estimates were five to seven times too low.
|
||||
# Attached and uploaded files are the holders that grow, and every one is capped:
|
||||
# uploads being checked and stored, 3 at a time per pod (`Files/UploadSlots.cs`); files
|
||||
# read back, their bytes or a window of their text; images prepared for the model; and
|
||||
# a PDF's text, read in a worker PROCESS of its own (`Files/PdfText.cs`,
|
||||
# `Files/PdfTextWorker.cs`) that is killed past its own limits, so a hostile file costs
|
||||
# that process and not this one.
|
||||
#
|
||||
# Uploads being checked and stored: 3 at a time (`files/upload-slots.ts`). Prisma is
|
||||
# what costs: its JSON protocol turns the bytes into base64, then into a request
|
||||
# string, and its engine copies that again. A 10 MiB file adds ~200 MiB of RSS while
|
||||
# it is written; a workspace image with the 3 MiB copy of it prepared for the model,
|
||||
# ~310 MiB, and ~330 to 360 MiB each with three at once, three bursts in a row. So
|
||||
# ~1.1 GiB. Six at a time, as this once allowed, took a 2 GiB container to 1.8-1.95
|
||||
# GiB with nothing else in it, and three such bursts were OOM-killed.
|
||||
#
|
||||
# Files read back — their bytes over HTTP, a window of their text for the viewer or
|
||||
# for the assistant's `read_attachment`: 8 at a time, up to ~40 MiB each (a 10 MiB log
|
||||
# is decoded whole, 25 to 39 MiB measured, until its window is cut). So 320 MiB.
|
||||
#
|
||||
# Images for the model: 64 MiB of `data:` URLs, reserved before a byte is read, so the
|
||||
# same 64 MiB also bounds what is being read for them at any instant — at most ~48 MiB
|
||||
# raw, which Prisma's transfer copies a few times more for the milliseconds a request
|
||||
# takes to build. So ~200 MiB at the worst instant.
|
||||
#
|
||||
# A PDF worker is the one without a flat cap. `resourceLimits` bounds its V8 heap
|
||||
# (192 MB, plus 32 MB of young generation) and NOT its ArrayBuffers, where a
|
||||
# decompressed stream lives. Those are read from outside every 50 ms and the worker is
|
||||
# killed past 128 MB, which a buffer doubling between two reads can overshoot to ~256
|
||||
# MB. So one hostile file can hold up to ~500 MB before it dies, two ~1 GB. Measured:
|
||||
# two decompression bombs at once added ~310 MB to the process, both refused in under
|
||||
# two seconds.
|
||||
#
|
||||
# Every holder full at once is therefore ~2.5 GiB (1.1 GiB of uploads, 320 MiB of
|
||||
# files read back, ~200 MiB of images, ~1 GB of PDF workers), on top of the API itself
|
||||
# (~200 MiB once booted): ~2.75 GiB. That cannot fit 2Gi at any upload concurrency
|
||||
# worth having — at 2Gi, one upload at a time beside two hostile PDFs is all there is
|
||||
# room for — so the limit is 3Gi, with ~250 MiB left for the terminal recordings and
|
||||
# share verifications above. It runs well past the room above this request, and that
|
||||
# is deliberate: the request is what the scheduler sets aside for an ordinary day, not
|
||||
# for the worst minute. If more margin is ever wanted, the levers are the PDF worker's
|
||||
# own caps (`files/pdf-text.ts`) and keeping an upload's bytes out of Prisma's JSON
|
||||
# protocol — not fewer upload slots.
|
||||
# MEASURED on this image, in a cgroup like this pod's (lot 11 P, proof 2j): every upload
|
||||
# slot of two pods busy with a 10 MB, 500-page PDF — the most pages the worker reads —
|
||||
# peaked at 619 MiB and 740 MiB of `memory.current`, the API and its workers together,
|
||||
# from 446 and 496 MiB before. The TypeScript image sized this limit: Prisma's JSON
|
||||
# protocol copied an upload's bytes several times over (~1.1 GiB for three at once)
|
||||
# and a PDF worker's buffers could reach ~500 MB, ~2.75 GiB with every holder full.
|
||||
# None of that is in this process; the 3Gi limit stands until it is lowered on purpose
|
||||
# (docs/dotnet/PLAN.md, "Left to Antoine after lot 11"). The request is what the
|
||||
# scheduler sets aside for an ordinary day, not for the worst minute.
|
||||
memory: 1Gi
|
||||
limits:
|
||||
memory: 3Gi
|
||||
|
||||
@@ -29,9 +29,11 @@ metadata:
|
||||
# outlast a quiet period comfortably. Too short and a healthy fleet reconnects all day.
|
||||
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
|
||||
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
|
||||
# Matches BODY_LIMIT in apps/api/src/main.ts. A document may hold 400,000 characters, which is
|
||||
# over a megabyte once accented text is UTF-8 encoded, and a diagram export posts back the SVG
|
||||
# the browser rendered. Below this they are rejected as a bare 413 with no message.
|
||||
# Matches the API's own ceiling on a JSON body, `BodyLimit.MaxBytes` in
|
||||
# src/Jarvis.Api/Common/BodyLimit.cs: 4 MiB, and nginx's `m` is MiB. A document may hold 400,000
|
||||
# characters, which is over a megabyte once accented text is UTF-8 encoded, and a diagram export
|
||||
# posts back the SVG the browser rendered. Below this they are rejected as a bare 413 with no
|
||||
# message.
|
||||
#
|
||||
# File uploads need more and are NOT raised here: they post under /api/uploads/, which has its
|
||||
# own Ingress in 06b-ingress-uploads.yaml, because ingress-nginx applies these annotations to
|
||||
@@ -58,8 +60,8 @@ spec:
|
||||
# Longest prefix first for readers; the controller sorts by specificity itself.
|
||||
#
|
||||
# The agent websocket has its own rule even though it sits under /api/, mirroring the
|
||||
# split in apps/web/nginx.conf: it is the one path where an idle connection legitimately
|
||||
# stays open for an hour between frames.
|
||||
# split in src/Jarvis.Web/nginx.conf: it is the one path where an idle connection
|
||||
# legitimately stays open for an hour between frames.
|
||||
- path: /api/agents/ws
|
||||
pathType: Prefix
|
||||
backend:
|
||||
|
||||
@@ -22,11 +22,12 @@ metadata:
|
||||
argocd.argoproj.io/sync-wave: "2"
|
||||
# 13376k is 13 MiB and 64 KiB: exactly the most the API itself reads on this path — a workspace
|
||||
# image of up to 10 MiB, the copy of it prepared for the model (at most 3 MiB), and 64 KiB for
|
||||
# the multipart framing and the name (`BODY_MAX_BYTES.documents` + `MULTIPART_OVERHEAD`, in
|
||||
# apps/api/src/files/upload-gate.interceptor.ts; both refuse strictly above). Not less: a body
|
||||
# the API would take would meet this ceiling first, and the controller's answer is an HTML page
|
||||
# the client cannot translate. Not more: the API refuses anything larger unread. A composer
|
||||
# attachment over 10 MiB still gets through here, and meets the API's own JSON 413.
|
||||
# the multipart framing and the name (`UploadGate.BodyMaxBytes(UploadRoute.Documents)` +
|
||||
# `UploadGate.MultipartOverhead`, in src/Jarvis.Api/Files/UploadEndpoint.cs; both refuse
|
||||
# strictly above). Not less: a body the API would take would meet this ceiling first, and the
|
||||
# controller's answer is an HTML page the client cannot translate. Not more: the API refuses
|
||||
# anything larger unread. A composer attachment over 10 MiB still gets through here, and meets
|
||||
# the API's own JSON 413.
|
||||
nginx.ingress.kubernetes.io/proxy-body-size: "13376k"
|
||||
# Everything the API does with a file happens before it answers: a wait for an upload slot, a
|
||||
# PDF's text extraction (capped at 20s), a 13 MB row write. Two minutes is headroom over that,
|
||||
|
||||
@@ -10,11 +10,19 @@
|
||||
# somebody else to hold that, use a managed Redis or an operator. What this directory owes you is
|
||||
# the shape the application needs, and this is it.
|
||||
#
|
||||
# The application half is proven rather than assumed: an ioredis client built exactly the way
|
||||
# `apps/api/src/redis/redis-connection.ts` builds one was run against a live three-sentinel
|
||||
# topology while its primary was killed. It followed the promotion to a different server, and 99
|
||||
# writes issued across the window were rejected zero times. What it does NOT do is preserve pub/sub
|
||||
# during the switch — see "What a failover actually costs" at the bottom.
|
||||
# The application half is proven rather than assumed, though not all of it on the api that ships
|
||||
# now. On the TypeScript api, an ioredis client built exactly the way its `redis-connection.ts`
|
||||
# built one was run against a live three-sentinel topology while its primary was killed. It
|
||||
# followed the promotion to a different server, and 99 writes issued across the window were
|
||||
# rejected zero times. What it does NOT do is preserve pub/sub during the switch — see the failover
|
||||
# section of the README beside this file.
|
||||
#
|
||||
# The .NET api reaches Redis through StackExchange.Redis, asking these sentinels for the primary
|
||||
# (src/Jarvis.Api/Infrastructure/RedisConnection.cs). Proven for it, on a throwaway cluster synced
|
||||
# from these manifests with 02-config.yaml's two REDIS_SENTINEL* lines: every pod wrote to the
|
||||
# primary — its leases taken, the console's realtime socket open, no READONLY refusal — where
|
||||
# REDIS_URL alone, on the headless Service below, landed writes on a read-only replica. Its
|
||||
# failover has NOT been measured: that it follows a promotion as ioredis did is not known.
|
||||
#
|
||||
# kubectl apply -f 08-redis-sentinel.yaml
|
||||
#
|
||||
|
||||
@@ -8,10 +8,11 @@
|
||||
# loud. PowerShell is in the api image and this is not, and the difference is exactly that: the
|
||||
# Exchange module talks to Microsoft, and a browser talks to whoever the model was told to look at.
|
||||
#
|
||||
# THE NETWORK POLICY BELOW IS THE REAL CONTROL. `browser.client.ts` refuses private addresses before
|
||||
# it asks, which is the half that can EXPLAIN itself to the model — but it cannot survive a redirect
|
||||
# and it cannot survive DNS rebinding, because by then the browser is already connecting. The policy
|
||||
# can, because it is enforced on the packet.
|
||||
# THE NETWORK POLICY BELOW IS THE REAL CONTROL. The api's `BrowserClient`
|
||||
# (src/Jarvis.Api/Connectors/BrowserClient.cs) refuses private addresses before it asks, which is
|
||||
# the half that can EXPLAIN itself to the model — but it cannot survive a redirect and it cannot
|
||||
# survive DNS rebinding, because by then the browser is already connecting. The policy can, because
|
||||
# it is enforced on the packet.
|
||||
#
|
||||
# DELETE THIS FILE (and its line in kustomization.yaml, and BROWSER_URL from 02-config.yaml) if you
|
||||
# would rather Jarvis had no web access. Nothing else stops working: the two web tools report
|
||||
@@ -121,8 +122,8 @@ spec:
|
||||
#
|
||||
# NOTE THIS NEEDS A CNI THAT ENFORCES NetworkPolicy. Calico, Cilium and most managed clusters do;
|
||||
# plain kube-proxy with flannel does NOT, and on such a cluster this file is documentation rather
|
||||
# than a control. Check before relying on it — `browser.client.ts` still refuses private addresses
|
||||
# on the near side either way, but that half cannot survive a redirect.
|
||||
# than a control. Check before relying on it — the api's `BrowserClient` still refuses private
|
||||
# addresses on the near side either way, but that half cannot survive a redirect.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
|
||||
+15
-6
@@ -158,6 +158,10 @@ Two consequences worth knowing:
|
||||
|
||||
- **The previous run's Job object, and its logs, are gone** by the time the next one starts. Ship
|
||||
the logs if you need them after the fact.
|
||||
- **A migration waiting on the schema lock is ended by its pod, not its Job.** Under Argo,
|
||||
deleting the `jarvis-migrate` Job leaves it `Terminating` behind the hook's finalizer, its pod still
|
||||
waiting and the sync waiting for good. Delete the pod instead — it stops in about a second — then
|
||||
end the sync's operation (`argocd app terminate-op <application>`).
|
||||
- **Every sync runs the migration.** That is safe — the phase is idempotent and holds a Postgres
|
||||
advisory lock — and it is why the example Application leaves `automated` sync OFF. A commit that
|
||||
applies a schema change with nobody watching turns the one thing this directory works hardest to
|
||||
@@ -227,15 +231,16 @@ back.
|
||||
|
||||
**`TRUST_PROXY_HOPS` is one hop here, not two.** The Compose files say 2 because a request passes
|
||||
your TLS terminator and then the web container's nginx. Here the Ingress routes `/api/` straight at
|
||||
the api Service, so there is exactly one proxy rewriting `X-Forwarded-For`. Left at 2, `req.ip`
|
||||
becomes **chosen by the caller**: anyone can send `X-Forwarded-For: 198.51.100.7` and have that
|
||||
the api Service, so there is exactly one proxy rewriting `X-Forwarded-For`. Left at 2, the client
|
||||
address the api records becomes **chosen by the caller**: anyone can send `X-Forwarded-For: 198.51.100.7` and have that
|
||||
written into session records, audit rows and the key the anonymous WebAuthn budget counts on.
|
||||
|
||||
**Do not add session affinity.** It is the reflex the moment websockets are involved and it solves
|
||||
nothing here: the hard case is co-locating an operator's browser with an *agent's* websocket, which
|
||||
arrives from a different network at a different time and whose id is not even known at the browser's
|
||||
handshake. No ingress annotation can express that. It is handled in the application, over Redis.
|
||||
Both ends pin `transports: ["websocket"]`, so there is no polling to make sticky in the first place.
|
||||
Both ends speak WebSockets only — the api allows no other transport and the console skips
|
||||
negotiation — so there is no polling to make sticky in the first place.
|
||||
|
||||
## On exactly three nodes
|
||||
|
||||
@@ -258,7 +263,9 @@ answer to wanting more is a fourth and fifth node, not a different configuration
|
||||
|
||||
## What a failover actually costs
|
||||
|
||||
Measured, not estimated.
|
||||
Measured, not estimated — on the api releases before 1.0.0, which were written in TypeScript. The
|
||||
readiness and liveness design below is unchanged in 1.0.0; the figures have not been measured again
|
||||
since.
|
||||
|
||||
**Redis**, with its primary killed under a running api:
|
||||
|
||||
@@ -298,8 +305,10 @@ outcome. Run it again.
|
||||
websockets, live runs and open shells, and CPU is a poor proxy for a load made of long-lived
|
||||
connections. Raise api replicas deliberately.
|
||||
- **A rollout still cuts what is in flight.** An assistant run is interrupted and resumed with a
|
||||
note; a terminal is not resumable and says so; and the web bundle's content-hashed chunks are
|
||||
per-image, so a browser holding the old shell can 404 on a lazy import during the rollout window.
|
||||
note; a terminal is not resumable and says so; and the console's files are fingerprinted per
|
||||
image, so during the rollout window a tab holding the old console can miss one of the two parts
|
||||
it loads on demand (the Markdown parser, the QR code generator). It then shows the Markdown as
|
||||
plain text and offers the reload.
|
||||
- **Validation is not admission.** These manifests satisfy the `restricted` Pod Security Standard
|
||||
that [`00-namespace.yaml`](00-namespace.yaml) enforces. If you edit them, note that a schema
|
||||
validator cannot see that policy: a missing `seccompProfile` or a container without a
|
||||
|
||||
Reference in New Issue
Block a user