Sync the self-hosting stack
This commit is contained in:
+41
-1
@@ -141,9 +141,49 @@ spec:
|
||||
cpu: 500m
|
||||
# Not generous — honest. A terminal recording holds up to 8 MiB of Buffers in heap
|
||||
# for the life of the session, and argon2 takes 64 MiB per share verification.
|
||||
#
|
||||
# Attached and uploaded files add four holders, every one capped, so that their sum is
|
||||
# a number rather than a hope. Their figures are MEASURED on this image, in a cgroup
|
||||
# like this pod's, not estimated — the estimates were five to seven times too low.
|
||||
#
|
||||
# Uploads being checked and stored: 3 at a time (`files/upload-slots.ts`). Prisma is
|
||||
# what costs: its JSON protocol turns the bytes into base64, then into a request
|
||||
# string, and its engine copies that again. A 10 MiB file adds ~200 MiB of RSS while
|
||||
# it is written; a workspace image with the 3 MiB copy of it prepared for the model,
|
||||
# ~310 MiB, and ~330 to 360 MiB each with three at once, three bursts in a row. So
|
||||
# ~1.1 GiB. Six at a time, as this once allowed, took a 2 GiB container to 1.8-1.95
|
||||
# GiB with nothing else in it, and three such bursts were OOM-killed.
|
||||
#
|
||||
# Files read back — their bytes over HTTP, a window of their text for the viewer or
|
||||
# for the assistant's `read_attachment`: 8 at a time, up to ~40 MiB each (a 10 MiB log
|
||||
# is decoded whole, 25 to 39 MiB measured, until its window is cut). So 320 MiB.
|
||||
#
|
||||
# Images for the model: 64 MiB of `data:` URLs, reserved before a byte is read, so the
|
||||
# same 64 MiB also bounds what is being read for them at any instant — at most ~48 MiB
|
||||
# raw, which Prisma's transfer copies a few times more for the milliseconds a request
|
||||
# takes to build. So ~200 MiB at the worst instant.
|
||||
#
|
||||
# A PDF worker is the one without a flat cap. `resourceLimits` bounds its V8 heap
|
||||
# (192 MB, plus 32 MB of young generation) and NOT its ArrayBuffers, where a
|
||||
# decompressed stream lives. Those are read from outside every 50 ms and the worker is
|
||||
# killed past 128 MB, which a buffer doubling between two reads can overshoot to ~256
|
||||
# MB. So one hostile file can hold up to ~500 MB before it dies, two ~1 GB. Measured:
|
||||
# two decompression bombs at once added ~310 MB to the process, both refused in under
|
||||
# two seconds.
|
||||
#
|
||||
# Every holder full at once is therefore ~2.5 GiB (1.1 GiB of uploads, 320 MiB of
|
||||
# files read back, ~200 MiB of images, ~1 GB of PDF workers), on top of the API itself
|
||||
# (~200 MiB once booted): ~2.75 GiB. That cannot fit 2Gi at any upload concurrency
|
||||
# worth having — at 2Gi, one upload at a time beside two hostile PDFs is all there is
|
||||
# room for — so the limit is 3Gi, with ~250 MiB left for the terminal recordings and
|
||||
# share verifications above. It runs well past the room above this request, and that
|
||||
# is deliberate: the request is what the scheduler sets aside for an ordinary day, not
|
||||
# for the worst minute. If more margin is ever wanted, the levers are the PDF worker's
|
||||
# own caps (`files/pdf-text.ts`) and keeping an upload's bytes out of Prisma's JSON
|
||||
# protocol — not fewer upload slots.
|
||||
memory: 1Gi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
memory: 3Gi
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
|
||||
@@ -15,13 +15,16 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
# Last, so that the first traffic admitted from outside meets Services with endpoints behind
|
||||
# them. An Ingress applied early is a route to a 503.
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-wave: "2"
|
||||
name: jarvis
|
||||
namespace: jarvis
|
||||
# ONE annotations map. This file used to carry two `annotations:` keys under metadata — the
|
||||
# sync-wave above the name, the proxy settings below it — and tools disagree about a duplicated
|
||||
# key: kustomize merged the two, while `kubectl apply -f` kept only the second and dropped the
|
||||
# sync-wave without a warning. What a cluster received depended on which tool rendered the file.
|
||||
annotations:
|
||||
# Last, so that the first traffic admitted from outside meets Services with endpoints behind
|
||||
# them. An Ingress applied early is a route to a 503.
|
||||
argocd.argoproj.io/sync-wave: "2"
|
||||
# An agent is idle between operations and the server pings every 30s, so this only has to
|
||||
# outlast a quiet period comfortably. Too short and a healthy fleet reconnects all day.
|
||||
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
|
||||
@@ -29,6 +32,10 @@ metadata:
|
||||
# Matches BODY_LIMIT in apps/api/src/main.ts. A document may hold 400,000 characters, which is
|
||||
# over a megabyte once accented text is UTF-8 encoded, and a diagram export posts back the SVG
|
||||
# the browser rendered. Below this they are rejected as a bare 413 with no message.
|
||||
#
|
||||
# File uploads need more and are NOT raised here: they post under /api/uploads/, which has its
|
||||
# own Ingress in 06b-ingress-uploads.yaml, because ingress-nginx applies these annotations to
|
||||
# every path of the object that carries them — 13376k here would be 13376k on every API route.
|
||||
nginx.ingress.kubernetes.io/proxy-body-size: "4m"
|
||||
#
|
||||
# NO affinity annotation, and none should be added. The reflex is understandable — there are
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# File uploads: a second Ingress on the same host, for one path.
|
||||
#
|
||||
# A SEPARATE OBJECT, NOT A FOURTH PATH IN 06-ingress.yaml. ingress-nginx applies an Ingress's
|
||||
# annotations to every path that object declares, so the only way to give /api/uploads/ a larger
|
||||
# body ceiling without handing the same ceiling to every API route is an object of its own. The
|
||||
# controller merges both Ingresses into one server for the host and picks the longest matching
|
||||
# prefix per request, so /api/uploads/ lands here and everything else under /api/ keeps the 4m of
|
||||
# the main Ingress.
|
||||
#
|
||||
# PATCH ITS HOST LIKE THE MAIN ONE. An overlay that rewrites `jarvis.example.com` in 06-ingress.yaml
|
||||
# must rewrite it here too, TLS block included. Left unpatched, this rule serves a host nobody
|
||||
# visits and uploads quietly fall back to the /api/ rule — where every upload between 4 and 13 MB
|
||||
# meets the controller's HTML 413 instead of Jarvis's own answer, and small files keep working, so
|
||||
# nothing looks broken until somebody attaches a large one.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: jarvis-uploads
|
||||
namespace: jarvis
|
||||
annotations:
|
||||
# With the main Ingress, for the same reason: last, once the Services have endpoints.
|
||||
argocd.argoproj.io/sync-wave: "2"
|
||||
# 13376k is 13 MiB and 64 KiB: exactly the most the API itself reads on this path — a workspace
|
||||
# image of up to 10 MiB, the copy of it prepared for the model (at most 3 MiB), and 64 KiB for
|
||||
# the multipart framing and the name (`BODY_MAX_BYTES.documents` + `MULTIPART_OVERHEAD`, in
|
||||
# apps/api/src/files/upload-gate.interceptor.ts; both refuse strictly above). Not less: a body
|
||||
# the API would take would meet this ceiling first, and the controller's answer is an HTML page
|
||||
# the client cannot translate. Not more: the API refuses anything larger unread. A composer
|
||||
# attachment over 10 MiB still gets through here, and meets the API's own JSON 413.
|
||||
nginx.ingress.kubernetes.io/proxy-body-size: "13376k"
|
||||
# Everything the API does with a file happens before it answers: a wait for an upload slot, a
|
||||
# PDF's text extraction (capped at 20s), a 13 MB row write. Two minutes is headroom over that,
|
||||
# and far below the hour the main Ingress keeps open for idle websockets, which have no business
|
||||
# on this path.
|
||||
nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
|
||||
nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
|
||||
# Request buffering is left at the controller's default, ON: the controller reads the whole body
|
||||
# before it opens a connection to the api, so a slow client costs the controller's disk rather
|
||||
# than one of the api pod's few upload slots and the memory behind it.
|
||||
#
|
||||
# And no `custom-http-errors` override. A controller that sets it replaces the body of the
|
||||
# statuses it lists, and every refusal the api makes on this path is a 4xx — busy included,
|
||||
# which is a 429, never a 503. The one 4xx such lists usually name, 404, is a status whose body
|
||||
# the client never reads.
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
tls:
|
||||
- hosts: ["jarvis.example.com"]
|
||||
secretName: jarvis-tls
|
||||
rules:
|
||||
- host: jarvis.example.com
|
||||
http:
|
||||
paths:
|
||||
- path: /api/uploads/
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: jarvis-api
|
||||
port:
|
||||
name: http
|
||||
+25
-3
@@ -76,6 +76,7 @@ kubectl wait --for=condition=complete job/jarvis-migrate -n jarvis --timeout=15m
|
||||
kubectl apply -f 04-api.yaml
|
||||
kubectl apply -f 05-web.yaml
|
||||
kubectl apply -f 06-ingress.yaml
|
||||
kubectl apply -f 06b-ingress-uploads.yaml # uploads alone get a ~13 MiB body ceiling
|
||||
kubectl apply -f 07-disruption.yaml
|
||||
```
|
||||
|
||||
@@ -175,12 +176,30 @@ images:
|
||||
- name: git.luxit.be/luxit/jarvis-api
|
||||
digest: sha256:...
|
||||
patches:
|
||||
- path: replicas.yaml # your node count, your ingress host, your storage class
|
||||
- path: replicas.yaml # your node count, the host of BOTH Ingresses, your storage class
|
||||
```
|
||||
|
||||
Then the digests you run are in a commit you control, upgrading is a diff you can review and revert,
|
||||
and `?ref=` pins the base to a tag rather than a branch that moves under you. Pin it.
|
||||
|
||||
**There are two Ingresses on the one host, and the host patch must reach both**: `jarvis`, and
|
||||
`jarvis-uploads` in `06b-ingress-uploads.yaml`, TLS block included. Uploads have an Ingress of their
|
||||
own because annotations apply to every path of the object that carries them, and they are the one
|
||||
path that needs bodies over 13 MiB. An overlay that rewrites only the first leaves uploads on the
|
||||
`/api/` rule, where small files still work and anything between 4 and 13 MB meets the controller's
|
||||
own 413 — so it looks fine until somebody attaches a large PDF.
|
||||
|
||||
To check it once deployed, send an upload of more than 4 MB without signing in, on your own host:
|
||||
|
||||
```sh
|
||||
head -c 5000000 /dev/zero > big.bin
|
||||
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' -F file=@big.bin \
|
||||
https://jarvis.example.com/api/uploads/orgs/x/conversations/y/attachments
|
||||
```
|
||||
|
||||
`401 application/json` is Jarvis turning away an anonymous request, so the body got through. A
|
||||
`413` with an HTML page is the controller, and means the host patch did not reach `jarvis-uploads`.
|
||||
|
||||
## The three things most likely to bite
|
||||
|
||||
**Never mint secrets per pod.** On Compose a one-shot generates the vault key and the JWT secrets
|
||||
@@ -263,8 +282,11 @@ outcome. Run it again.
|
||||
`emptyDir` filled by an initContainer is right; a shared RWO volume is not — either every pod is
|
||||
pinned to one node, or the second sits in `Multi-Attach error`. Leaving it unset is supported:
|
||||
everything works except the agent installer, which answers 503 saying no build is published.
|
||||
- **No application volume, and it should stay that way.** Avatars, documents and terminal recordings
|
||||
are all in Postgres; exports are rendered in memory.
|
||||
- **No application volume, and it should stay that way.** Avatars, documents, terminal recordings
|
||||
and attached files are all in Postgres; exports are rendered in memory.
|
||||
- **The Ingresses are written for ingress-nginx.** Under another controller their `nginx.ingress…`
|
||||
annotations do nothing, so set that controller's own equivalents: an hour's read timeout on the
|
||||
two websocket paths, a 4 MB body ceiling on `/api/`, and 13,376 KiB on `/api/uploads/`.
|
||||
- **Autoscale `jarvis-web`, not `jarvis-api`.** Every api scale-in kills a pod holding agent
|
||||
websockets, live runs and open shells, and CPU is a poor proxy for a load made of long-lived
|
||||
connections. Raise api replicas deliberately.
|
||||
|
||||
@@ -25,7 +25,8 @@
|
||||
# - name: git.luxit.be/luxit/jarvis-web
|
||||
# digest: sha256:<the one you are running>
|
||||
# patches:
|
||||
# - path: replicas.yaml # your node count, your ingress host, your storage class
|
||||
# - path: replicas.yaml # your node count, the host of BOTH Ingresses, your storage class
|
||||
# # (jarvis and jarvis-uploads, TLS blocks included)
|
||||
#
|
||||
# Then the Application points at YOUR repository, your digests are in a commit you control, and
|
||||
# upgrading is a one-line diff you can review and revert. `?ref=` pins the base to a tag or a commit
|
||||
|
||||
@@ -38,6 +38,9 @@ resources:
|
||||
- 04-api.yaml
|
||||
- 05-web.yaml
|
||||
- 06-ingress.yaml
|
||||
# Uploads get their own Ingress so their larger body ceiling reaches no other path. Patch its
|
||||
# host along with the main one.
|
||||
- 06b-ingress-uploads.yaml
|
||||
- 07-disruption.yaml
|
||||
# Remove this line if you point REDIS_URL at a managed Redis. It deploys three Redis processes and
|
||||
# three sentinels; see the README on what that costs on three nodes.
|
||||
|
||||
Reference in New Issue
Block a user