Sync the self-hosting stack
This commit is contained in:
@@ -9,6 +9,32 @@ one of them changed. An entry says which artefact and which version, so `web 0.1
|
||||
Every version published to a channel appears here. Nothing is written from memory afterwards: a build
|
||||
cannot be published without its entry.
|
||||
|
||||
## 2026-10-22 — A rebuilt database; 2.0 starts from an empty one `api 2.0.0` `web 2.0.0`
|
||||
|
||||
**2.0 does not upgrade a 1.x instance.** Its database schema was rebuilt — table and column names,
|
||||
timestamps with their time zone, enums stored as text — and there is no migration from the 1.x
|
||||
schema. Pointed at a 1.x database, the api changes nothing, says it was not created by this version
|
||||
of Jarvis, and restarts into the same refusal. Read "Upgrading from 1.x" in the README before you
|
||||
pull: an instance moves to 2.0 by being installed again on an empty database, its machines enrolled
|
||||
again.
|
||||
|
||||
**What changes for you**
|
||||
|
||||
- **The api image answers `jarvis` only.** The `node`, `/usr/local/bin/entrypoint.sh` and
|
||||
`/usr/local/bin/migrate.sh` commands that kept the 0.x compose files and manifests working are
|
||||
gone. Take the current `docker-compose.yml` and manifests.
|
||||
- **On Kubernetes, `04-api.yaml` rolls the api again** — the Recreate was for 1.0.0 only.
|
||||
- **The README's SQL recipes use the new names** (`"Users"."PlatformRole" = 'SuperAdmin'`). A recipe
|
||||
kept from an older README fails on 2.0.
|
||||
- **Fixed**: the remote-desktop tab is named after the machine when the screen is opened from inside
|
||||
the console, not only on a reload; the admin searches match `%` and `_` literally; removing a
|
||||
profile or a platform role another administrator removed at the same moment answers "not found"
|
||||
and files no audit entry; a numeric HTML entity beyond Unicode in a Microsoft 365 message no longer
|
||||
fails the read.
|
||||
|
||||
Update the API and web images together. No agent update is required, but every machine is enrolled
|
||||
again on the new instance.
|
||||
|
||||
## 2026-10-21 — Jarvis runs on .NET `api 1.0.0` `web 1.0.0`
|
||||
|
||||
**The api and the console are rewritten on .NET** — the api on ASP.NET Core, the console in Blazor
|
||||
|
||||
@@ -305,9 +305,9 @@ parameter stops the api with `DATABASE_URL: unsupported parameter "<name>"` rath
|
||||
ignored.
|
||||
|
||||
`sslmode=require` encrypts without checking the server's certificate, as it always has here;
|
||||
`sslaccept=strict` checks it and the host name. `verify-ca` and `verify-full` check it too — before
|
||||
api 1.0.0 they were read as `prefer` — so a URL that carried one may now refuse a certificate the
|
||||
container cannot verify: mount the CA and name it with `sslrootcert`. The migration holds a session
|
||||
`sslaccept=strict` checks it and the host name. `verify-ca` and `verify-full` check it too, so a URL
|
||||
that carries one refuses a certificate the container cannot verify: mount the CA and name it with
|
||||
`sslrootcert`. The migration holds a session
|
||||
lock, so it needs a direct connection to Postgres, not PgBouncer in transaction mode; `pgbouncer=true`
|
||||
in the URL is warned about.
|
||||
|
||||
@@ -388,41 +388,28 @@ prerequisite rather than a precaution.
|
||||
docker compose exec -T postgres pg_dump -U jarvis jarvis | gzip > jarvis-$(date +%F).sql.gz
|
||||
```
|
||||
|
||||
### Upgrading from the TypeScript releases (api 0.x)
|
||||
### Upgrading from 1.x
|
||||
|
||||
api 1.0.0 and web 1.0.0 are Jarvis rewritten on .NET. The images keep their names and their channels,
|
||||
so the upgrade is the usual one, with four things to know first.
|
||||
**2.0 starts from an empty database; it does not upgrade a 1.x one.** Its schema was rebuilt, and
|
||||
there is no migration from the 1.x schema. Pointed at a 1.x database, the api changes nothing,
|
||||
logs a `[migrate]` line containing `it was not created by this version of Jarvis`, and restarts
|
||||
into the same refusal.
|
||||
|
||||
- **Be on api 0.151.0 first** — the last 0.x release — and let it start once. A database it has not
|
||||
migrated is refused by 1.0.0 without a change: the api logs `[migrate] This database was built by
|
||||
the TypeScript API, but not by its last release`, the release to install and what is missing, and
|
||||
restarts into the same refusal. Pin `JARVIS_IMAGE_API=git.luxit.be/luxit/jarvis-api:0.151.0` in
|
||||
`.env`, `docker compose up -d`, wait until it serves, remove the pin and `docker compose up -d`
|
||||
again.
|
||||
- **Move the api and the web together** if you pin: `JARVIS_IMAGE_API` and `JARVIS_IMAGE_WEB` both.
|
||||
A 1.x api behind a 0.x console, or the reverse, works without live updates.
|
||||
- **Take the new `docker-compose.yml`.** Two lines changed: `init` runs `jarvis init-secrets` and the
|
||||
api's healthcheck `jarvis healthcheck`. Your current file keeps working — the 1.x api image still
|
||||
answers the three `node` commands the 0.x files give it, and refuses any other — until a later
|
||||
release removes them, with `/usr/local/bin/migrate.sh` and `/usr/local/bin/entrypoint.sh`; its
|
||||
CHANGELOG entry will say so.
|
||||
- **Nothing to add to `.env`.** Every variable keeps its name and its meaning, `NODE_ENV` included.
|
||||
To move an instance to 2.0, start it on a new, empty database and install it again:
|
||||
|
||||
The first 1.x start records where the schema stands (it creates `__EFMigrationsHistory` and changes
|
||||
nothing else) and takes seconds. Open tabs lose live updates until they are reloaded; agents
|
||||
reconnect by themselves.
|
||||
1. Note what you will re-enter: organizations, users, connections, the vault's secrets. The 1.x dump
|
||||
cannot be imported into 2.0.
|
||||
2. `docker compose down`, then remove the database volume (`docker volume rm jarvis_postgres_data`)
|
||||
or point `DATABASE_URL` at a new database. Keep `.env` and the `jarvis_secrets` volume if you like —
|
||||
a new vault key is just as good, since the vault starts empty.
|
||||
3. Take the new `docker-compose.yml`, `docker compose pull && docker compose up -d`, and go through
|
||||
the installation wizard.
|
||||
4. Enroll your machines again: an agent's credentials lived in the old database. Run the install
|
||||
command the console gives you on each machine: it enrolls the machine with the new instance,
|
||||
keeping the agent's key.
|
||||
|
||||
**On Kubernetes**, pin both digests in one change, with the new manifests: `01-secret-job.yaml` runs
|
||||
`jarvis mint-k8s-secret` (and no longer sets `NODE_EXTRA_CA_CERTS`), `03-migrate-job.yaml` runs
|
||||
`jarvis migrate`, and `04-api.yaml` recreates the api for this release — every old pod stops before
|
||||
a new one starts, a gap of about thirty seconds, once — and rolls it again from the next. Overlays of
|
||||
your own that call `/usr/local/bin/migrate.sh` keep working until that release.
|
||||
|
||||
**The way back**, until the first release after 1.0.0 whose entry says it changes the schema: pin
|
||||
both previous images and either set `SCHEMA_ACCEPT_DATA_LOSS=true` for one start — the 0.x api then
|
||||
drops `__EFMigrationsHistory` and nothing else; its log names what it drops, and anything else in
|
||||
that list means: stop, and restore — or set `JARVIS_SKIP_MIGRATIONS=true` until you upgrade again.
|
||||
After that release, the dump is the way back.
|
||||
**On Kubernetes**, the same: delete the database (or its PVCs), apply the new manifests, and let the
|
||||
migration Job create the schema.
|
||||
|
||||
### Channels
|
||||
|
||||
@@ -696,7 +683,7 @@ wrong on the screen that asks for it.
|
||||
| Everything works but one kind of thing cannot be created | A limit is full — the community edition's, or your licence's. The message names which. See below. |
|
||||
| An agent installer answers 503 | No agent build is published — the overlay is off. That is a supported state. |
|
||||
| The remote terminal works and remote desktop never connects | The proxy forwards `/api/agents/ws` but not the paths under it. The media socket is `/api/agents/ws/desktop` and needs the upgrade too. |
|
||||
| api restarts in a loop, logs `This database was built by the TypeScript API, but not by its last release` | The upgrade skipped a release. Install the one it names, let it start, then this one — see "Upgrading from the TypeScript releases". |
|
||||
| api restarts in a loop, logs `it was not created by this version of Jarvis` | A 1.x database: 2.0 starts from an empty one — see "Upgrading from 1.x". |
|
||||
| api restarts in a loop, logs `DATABASE_URL: unsupported parameter` | A parameter the api does not read. The list it does read is under "Your own Postgres". |
|
||||
| The console stays blank behind your proxy; the browser's console names a Content Security Policy | Your proxy's policy refuses the console's scripts — see "Your reverse proxy has three requirements". |
|
||||
| A remote desktop session opens and ends immediately | Almost always the notice. It ends by design if the banner cannot be kept on top, if the person refuses, or if nobody answers the request within the minute. The end reason on the session says which. |
|
||||
@@ -718,7 +705,7 @@ minutes, and it turns every case below into somebody else clicking a button.
|
||||
|
||||
```sh
|
||||
docker compose exec -T postgres psql -U jarvis -d jarvis \
|
||||
-c "UPDATE \"User\" SET \"platformRole\" = 'SUPER_ADMIN' WHERE email = 'colleague@example.com';"
|
||||
-c "UPDATE \"Users\" SET \"PlatformRole\" = 'SuperAdmin' WHERE \"Email\" = 'colleague@example.com';"
|
||||
```
|
||||
|
||||
They then reset the locked-out password under **Admin Center → Users**.
|
||||
@@ -738,8 +725,8 @@ Then write it, and make sure the account is active and privileged:
|
||||
|
||||
```sh
|
||||
docker compose exec -T postgres psql -U jarvis -d jarvis -c \
|
||||
"UPDATE \"User\" SET \"passwordHash\" = '<the \$argon2id\$… string>', \"isActive\" = true,
|
||||
\"platformRole\" = 'SUPER_ADMIN' WHERE email = 'you@example.com';"
|
||||
"UPDATE \"Users\" SET \"PasswordHash\" = '<the \$argon2id\$… string>', \"IsActive\" = true,
|
||||
\"PlatformRole\" = 'SuperAdmin' WHERE \"Email\" = 'you@example.com';"
|
||||
```
|
||||
|
||||
Quote the hash in single quotes — it contains `$` characters your shell would otherwise eat.
|
||||
@@ -752,9 +739,9 @@ demanding one:
|
||||
|
||||
```sh
|
||||
docker compose exec -T postgres psql -U jarvis -d jarvis <<'SQL'
|
||||
DELETE FROM "WebAuthnCredential" WHERE "userId" = (SELECT id FROM "User" WHERE email = 'you@example.com');
|
||||
DELETE FROM "UserTotpCredential" WHERE "userId" = (SELECT id FROM "User" WHERE email = 'you@example.com');
|
||||
UPDATE "User" SET "mfaEnabled" = false, "mfaEnabledAt" = NULL WHERE email = 'you@example.com';
|
||||
DELETE FROM "WebAuthnCredentials" WHERE "UserId" = (SELECT "Id" FROM "Users" WHERE "Email" = 'you@example.com');
|
||||
DELETE FROM "UserTotpCredentials" WHERE "UserId" = (SELECT "Id" FROM "Users" WHERE "Email" = 'you@example.com');
|
||||
UPDATE "Users" SET "MfaEnabled" = false, "MfaEnabledAt" = NULL WHERE "Email" = 'you@example.com';
|
||||
SQL
|
||||
```
|
||||
|
||||
|
||||
@@ -129,8 +129,8 @@ spec:
|
||||
# Exec form: `jarvis` is a bash script that execs the API, and a `/bin/sh -c` (dash) in between
|
||||
# would drop the image's dotted logging variables. It verifies the API server's certificate
|
||||
# against the service account's own ca.crt, and trusts nothing else for it — so no variable
|
||||
# names the CA (Node's NODE_EXTRA_CA_CERTS went with the TypeScript image). A bearer token good
|
||||
# enough to write Secrets is a bearer token worth stealing: never turn that check off.
|
||||
# names the CA and there is none to set. A bearer token good enough to write Secrets is a
|
||||
# bearer token worth stealing: never turn that check off.
|
||||
command: ["jarvis", "mint-k8s-secret"]
|
||||
resources:
|
||||
requests:
|
||||
|
||||
@@ -2,9 +2,8 @@
|
||||
#
|
||||
# A JOB, and deliberately neither of the two alternatives:
|
||||
#
|
||||
# NOT an initContainer. It would run once per pod — which is exactly the bug this replaces, where
|
||||
# every API container ran `prisma db push` at boot and N replicas raced the same DDL while the
|
||||
# previous generation was still selecting the columns being altered.
|
||||
# NOT an initContainer. It would run once per pod: N replicas would race the same DDL while the
|
||||
# previous generation is still selecting the columns being altered.
|
||||
#
|
||||
# NOT a leader election inside the API. A Job is the primitive built for run-once work: it has an
|
||||
# observable status, a bounded number of retries, and a failure that stops the rollout instead of
|
||||
@@ -89,10 +88,9 @@ spec:
|
||||
secretKeyRef:
|
||||
name: jarvis-secrets
|
||||
key: DATABASE_URL
|
||||
# Off, and it should stay off between deliberate acts. `prisma db push` used to run with
|
||||
# `--accept-data-loss` at every boot, which meant an upgrade could destroy a column with
|
||||
# nobody having decided to — survivable on one container, not during a rolling update
|
||||
# where the previous generation is still selecting it.
|
||||
# Off, and it should stay off between deliberate acts. Left on, every upgrade could
|
||||
# destroy a column with nobody having decided to — survivable on one container, not
|
||||
# during a rolling update where the previous generation is still selecting it.
|
||||
#
|
||||
# Without it a pending migration that drops a table or a column is REFUSED, naming them,
|
||||
# and fails the Job — so the rollout stops and nothing is serving a half-migrated schema.
|
||||
|
||||
+11
-22
@@ -14,21 +14,13 @@ spec:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jarvis
|
||||
app.kubernetes.io/component: api
|
||||
# RECREATED, FOR api 1.0.0 ONLY — the release that crosses from the TypeScript images. A rolling
|
||||
# update would keep a TypeScript pod and a .NET pod serving side by side behind one Service, and
|
||||
# their realtime protocols differ under /socket.io/ (Socket.IO against SignalR): every old pod stops
|
||||
# before the first new one starts, a gap of about thirty seconds, once. The next release puts the
|
||||
# rolling update back:
|
||||
#
|
||||
# type: RollingUpdate
|
||||
# rollingUpdate:
|
||||
# maxUnavailable: 0 # never fewer pods than are being served now
|
||||
# maxSurge: 1
|
||||
#
|
||||
# With `kubectl apply` or Argo's default client-side apply this switches both ways. Under
|
||||
# server-side apply, a field manager that owns `rollingUpdate` makes the API server refuse it.
|
||||
strategy:
|
||||
type: Recreate
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
# Never fewer pods than are being served now. A run holds its conversation through a unique
|
||||
# index, so a surge pod cannot double-answer anything.
|
||||
maxUnavailable: 0
|
||||
maxSurge: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
@@ -156,14 +148,11 @@ spec:
|
||||
# `Files/PdfTextWorker.cs`) that is killed past its own limits, so a hostile file costs
|
||||
# that process and not this one.
|
||||
#
|
||||
# MEASURED on this image, in a cgroup like this pod's (lot 11 P, proof 2j): every upload
|
||||
# slot of two pods busy with a 10 MB, 500-page PDF — the most pages the worker reads —
|
||||
# peaked at 619 MiB and 740 MiB of `memory.current`, the API and its workers together,
|
||||
# from 446 and 496 MiB before. The TypeScript image sized this limit: Prisma's JSON
|
||||
# protocol copied an upload's bytes several times over (~1.1 GiB for three at once)
|
||||
# and a PDF worker's buffers could reach ~500 MB, ~2.75 GiB with every holder full.
|
||||
# None of that is in this process; the 3Gi limit stands until it is lowered on purpose
|
||||
# (docs/dotnet/PLAN.md, "Left to Antoine after lot 11"). The request is what the
|
||||
# MEASURED on this image, in a cgroup like this pod's: every upload slot of two pods
|
||||
# busy with a 10 MB, 500-page PDF — the most pages the worker reads — peaked at 619 MiB
|
||||
# and 740 MiB of `memory.current`, the API and its workers together, from 446 and
|
||||
# 496 MiB before. The 3Gi limit was sized generously and sits well above that
|
||||
# 0.6–0.75 GiB peak; it stands until it is lowered on purpose. The request is what the
|
||||
# scheduler sets aside for an ordinary day, not for the worst minute.
|
||||
memory: 1Gi
|
||||
limits:
|
||||
|
||||
@@ -10,19 +10,15 @@
|
||||
# somebody else to hold that, use a managed Redis or an operator. What this directory owes you is
|
||||
# the shape the application needs, and this is it.
|
||||
#
|
||||
# The application half is proven rather than assumed, though not all of it on the api that ships
|
||||
# now. On the TypeScript api, an ioredis client built exactly the way its `redis-connection.ts`
|
||||
# built one was run against a live three-sentinel topology while its primary was killed. It
|
||||
# followed the promotion to a different server, and 99 writes issued across the window were
|
||||
# rejected zero times. What it does NOT do is preserve pub/sub during the switch — see the failover
|
||||
# section of the README beside this file.
|
||||
#
|
||||
# The .NET api reaches Redis through StackExchange.Redis, asking these sentinels for the primary
|
||||
# (src/Jarvis.Api/Infrastructure/RedisConnection.cs). Proven for it, on a throwaway cluster synced
|
||||
# from these manifests with 02-config.yaml's two REDIS_SENTINEL* lines: every pod wrote to the
|
||||
# primary — its leases taken, the console's realtime socket open, no READONLY refusal — where
|
||||
# REDIS_URL alone, on the headless Service below, landed writes on a read-only replica. Its
|
||||
# failover has NOT been measured: that it follows a promotion as ioredis did is not known.
|
||||
# The application half is proven as far as it is claimed, and no further. The api reaches Redis
|
||||
# through StackExchange.Redis, asking these sentinels for the primary
|
||||
# (src/Jarvis.Api/Infrastructure/RedisConnection.cs). Proven on a throwaway cluster synced from
|
||||
# these manifests with 02-config.yaml's two REDIS_SENTINEL* lines: every pod wrote to the primary —
|
||||
# its leases taken, the console's realtime socket open, no READONLY refusal — where REDIS_URL
|
||||
# alone, on the headless Service below, landed writes on a read-only replica. Its failover has NOT
|
||||
# been measured: how fast it follows a promotion to a different server, and what a write issued
|
||||
# across the window meets, is not known. Pub/sub does not survive the switch whatever the client —
|
||||
# see the failover section of the README beside this file.
|
||||
#
|
||||
# kubectl apply -f 08-redis-sentinel.yaml
|
||||
#
|
||||
|
||||
+24
-25
@@ -10,12 +10,12 @@ takes one command, and is what most instances run. Come here when you want Jarvi
|
||||
while a node reboots, and when you already operate a cluster — this directory assumes you have
|
||||
opinions about ingress, storage and a database, because it deliberately does not have them for you.
|
||||
|
||||
> **What has been proven, and what has not.** This stack has been brought up on a real cluster —
|
||||
> four nodes, three Postgres instances, three Redis with three sentinels, two api pods — and kept
|
||||
> serving while its Postgres primary and its Redis primary were killed underneath it. What has NOT
|
||||
> happened is a long-running estate serving production traffic this way. If you are early, watch it
|
||||
> rather than assume it, and read the three warnings below before you apply anything, because each
|
||||
> of them fails silently.
|
||||
> **What has been proven, and what has not.** This stack has been brought up on a throwaway
|
||||
> four-node cluster — three Redis with three sentinels, two api pods behind the Ingress, synced by
|
||||
> Argo CD — and served the console and its realtime connections through it. What has NOT happened
|
||||
> is a Postgres or a Redis failover measured under the current api, or a long-running estate serving
|
||||
> production traffic this way. If you are early, watch it rather than assume it, and read the three
|
||||
> warnings below before you apply anything, because each of them fails silently.
|
||||
|
||||
## What runs more than once, and what cannot
|
||||
|
||||
@@ -261,31 +261,30 @@ per node, and losing any node leaves two. Change `replicas: 2` to `3` in it and
|
||||
**Three nodes tolerates losing one.** Not two. That is a property of majority quorums, and the
|
||||
answer to wanting more is a fourth and fifth node, not a different configuration.
|
||||
|
||||
## What a failover actually costs
|
||||
## What a failover costs
|
||||
|
||||
Measured, not estimated — on the api releases before 1.0.0, which were written in TypeScript. The
|
||||
readiness and liveness design below is unchanged in 1.0.0; the figures have not been measured again
|
||||
since.
|
||||
**Not measured on the current api**: no figure here would be one. What holds by design, and what
|
||||
Redis and Postgres do on their own, is this — how long each window lasts on your cluster is yours to
|
||||
measure.
|
||||
|
||||
**Redis**, with its primary killed under a running api:
|
||||
**Redis**, its primary lost under a running api:
|
||||
|
||||
- **The api rides it out without restarting.** Readiness answered 503 for about fifteen seconds and
|
||||
then 200 again, on the same process — the pod leaves the Service's endpoints while it cannot reach
|
||||
Redis, and returns when it can. Liveness never touches Redis, which is why the pod is not killed
|
||||
and restarted into a cold start.
|
||||
- **Commands survive.** 99 writes issued across the window, 0 rejected, the last readable from the
|
||||
promoted primary afterwards. Commands queue rather than fail, so a failover is a pause.
|
||||
- **Published messages do not.** Around 45% of the messages published during the window reached
|
||||
nobody. Redis pub/sub has no buffer and no redelivery, and nothing on the client side can change
|
||||
that. In practice: a few seconds in which something happening on one api replica may not reach
|
||||
- **The api rides it out without restarting.** `/api/health/ready` answers 503 with `"redis":
|
||||
"unreachable"` while the api cannot reach Redis, so the pod leaves the Service's endpoints and
|
||||
returns when it can. Liveness never touches Redis, so the pod is not killed and restarted into a
|
||||
cold start.
|
||||
- **Messages published during the window are lost.** Redis pub/sub has no buffer and no redelivery,
|
||||
and nothing on the client side can change that: for a few seconds, live updates — the chat
|
||||
stream, presence, typing — reach nobody, and something happening on one api replica may not reach
|
||||
another. Nothing is corrupted by it — what must survive is in Postgres for exactly this reason.
|
||||
|
||||
**Postgres**, primary deleted with `--force --grace-period=0`:
|
||||
**Postgres**, its primary lost:
|
||||
|
||||
- **Three seconds.** Both api pods answered `503 {"database":"unreachable"}` and were back to 200 on
|
||||
the fourth poll. Both degraded together, which is what pointing every pod at one endpoint means.
|
||||
- **Zero container restarts**, across two failovers, on pods up throughout — liveness deliberately
|
||||
not touching the database is what keeps three seconds from becoming a cold start of everything.
|
||||
- **Every api pod degrades together**: readiness answers `503` with `"database": "unreachable"` until
|
||||
the promoted primary answers at the address `DATABASE_URL` names, which is what pointing every pod
|
||||
at one endpoint means.
|
||||
- **No restart**: liveness deliberately does not touch the database, which is what keeps a failover
|
||||
from becoming a cold start of everything.
|
||||
|
||||
A failover *during* the migration Job aborts it with a lost-connection error, which is the correct
|
||||
outcome. Run it again.
|
||||
|
||||
Reference in New Issue
Block a user