From fdd35f2d2774f4d76527175ad75cf307a9d75919 Mon Sep 17 00:00:00 2001 From: Luxit Date: Mon, 31 Aug 2026 21:43:38 +0200 Subject: [PATCH] Sync the self-hosting stack --- CHANGELOG.md | 109 ++++++++++++++++++++++ docker-compose.yml | 63 +++++++++++++ kubernetes/02-config.yaml | 4 + kubernetes/09-browser.yaml | 171 ++++++++++++++++++++++++++++++++++ kubernetes/kustomization.yaml | 5 + 5 files changed, 352 insertions(+) create mode 100644 kubernetes/09-browser.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 01ac1b4..3024c2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,115 @@ one of them changed. An entry says which artefact and which version, so `web 0.1 Every version published to a channel appears here. Nothing is written from memory afterwards: a build cannot be published without its entry. +## 2026-08-31 — Jarvis can wake itself up, knows what day it is, and can read the web `api 0.102.0` `web 0.125.0` `browser 0.1.0` + +**The schema step stops once on the way in — see the upgrade note below.** It is the same guard as +`api 0.101.0`, tripped by a different change. + +### Taking this build: the schema step stops once again + +Same guard as `api 0.101.0`, tripped by a different change — and for the same reason it is harmless +here. Reshaping scheduled tasks around the conversation drops the columns the old organization-wide +version needed, and the guard refuses a drop without being told to: + +``` +⚠️ There might be data loss when applying the changes: + • The `Schedule` table will lose columns: kind, autonomy, toolName, assetId, … +Error: Use the --accept-data-loss flag … +``` + +**Nothing of yours is in those columns.** They existed only in `api 0.101.0`, which was published +this afternoon, and only a wake-up created through the removed Settings screen could have filled +them. Check before you take our word for it: + +```sql +SELECT count(*) FROM "Schedule"; +``` + +If that is 0 — which it is on every instance that upgraded straight from 0.100.0 or earlier — the +migration cannot lose anything. If it is not 0, those rows are wake-ups you created today and they +will be dropped; note down what they were before you continue, and recreate them by asking Jarvis +inside the conversation you want them in. + +Then let it through **once**, and put the guard back: + +- **compose** — set `SCHEMA_ACCEPT_DATA_LOSS=true` in `.env`, `docker compose up -d`, wait for the + api, then remove the line. Leaving it set would let a genuinely destructive change through + unannounced on some future release. +- **Kubernetes** — set the same variable on the `jarvis-migrate` Job, run it to completion, then take + it off. The api pods keep serving the old build while the Job fails, so there is no outage. + +### Scheduled tasks are now part of a conversation + +`api 0.101.0` shipped these a few hours ago as an organization-wide screen, and that was the wrong +shape. **The "Scheduled work" section and the `schedule:view` / `schedule:manage` permissions are +gone**; if you had started using them, nothing is silently broken — but read on, because what +replaced them is not in the same place. + +The thing people actually want this for needs the conversation. Two cases, and both: + +- **Waiting.** A rebuild is running, a DNS change is propagating, a maintenance window opens at + 03:00. Jarvis leaves itself a note and picks the work up later with everything it had already + established still in front of it. +- **A standing check.** "Look at that file every day at noon." Every occurrence lands in the same + thread, so the seventh day can say "unchanged since Tuesday" instead of reporting into a vacuum. + +The old version opened a _fresh_ conversation for each occurrence — which meant "carry on where you +left off" was impossible, because the woken assistant arrived knowing nothing. Now a wake-up belongs +to one conversation, wakes back into it, and appears under a new **Tasks** tab in that +conversation's workspace, beside the documents and the approvals. You pause, run and cancel from +there; you create one by asking ("check this again tomorrow morning"), because the instruction is +written with the conversation around it. + +**It runs as whoever scheduled it, checked when it fires rather than when it was written.** If +somebody's access to a machine is revoked, the work they left behind stops reaching it the same day +and says so on its row. Three failures in a row switch a wake-up off, so a broken one becomes a +thing you find rather than a notification you learn to ignore. + +Wake-ups inherit the conversation's autonomy and its asset scope; they cannot raise either. + +### Jarvis knows what the date is + +It did not before, and it did not know that it did not. Asked what day it is, a language model +answers from its training data — confidently, and usually months out. Everything an administrator +asks is dated: "the logs from the last week", "has this certificate expired", "wake me tomorrow at +noon". Each of those was a subtraction from a number nothing was giving it. + +The current time is now stated at the start of every turn, and there is a `current_time` tool for +reading the clock during a long run and for converting into a customer's time zone. Nothing to +configure. If you have ever been given a date that was quietly wrong, this is why. + +### Jarvis can read the web — if you turn it on + +Two new abilities: search the web, and open a page and read it. A real browser, so pages that build +themselves with JavaScript work — vendor advisories, status pages, release notes, consoles. + +It also fixes the same class of problem as the clock: a model's knowledge has a cut-off. Versions, +end-of-life dates, advisories and prices are exactly what it will otherwise answer from memory and +get wrong, so it is now told to look things up instead. + +**It is off by default, and turning it on is a deliberate act.** An assistant that can read the +internet is a new kind of input into a system that administers your customers. To enable it: + +- **compose** — `COMPOSE_PROFILES=browser docker compose up -d`, and set `BROWSER_URL=http://browser:3000`. +- **Kubernetes** — uncomment `09-browser.yaml` in `kustomization.yaml` and `BROWSER_URL` in + `02-config.yaml`. + +With it off, the two tools report themselves unavailable and nothing else changes. + +**Chromium runs in its own container, never in the api.** The api holds your vault key and talks to +your database; a browser's whole job is to run code written by strangers. Keeping them apart is what +stops a web page being used to read your database, your cache or your cloud provider's metadata +endpoint. The browser container holds no secret, no volume and no database credentials, and the +Kubernetes manifest ships a NetworkPolicy that denies it every private address range. **That policy +needs a CNI that enforces NetworkPolicy** — Calico, Cilium and most managed clusters do; plain +flannel does not, in which case the in-process address check is the only control and it cannot +survive a redirect. + +**What comes back is treated as untrusted.** Page text is delimited and labelled as data rather than +instruction, because a page can be written specifically to be read by an assistant with access to +your systems. The tools only read: no clicking, no forms, no signing in. + ## 2026-08-31 — Jarvis works while you are asleep, and can finally administer Exchange `api 0.101.0` `web 0.124.0` Two capabilities, and both are about the half of managed services that does not happen in a diff --git a/docker-compose.yml b/docker-compose.yml index 79eb0ee..3e80ac1 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -215,10 +215,20 @@ services: # answer you want from an upgrade that was not expecting to lose anything. SCHEMA_ACCEPT_DATA_LOSS: ${SCHEMA_ACCEPT_DATA_LOSS:-false} + # The browser the assistant may use, if you run one. Unset is a supported state: the two web + # tools report themselves unavailable and everything else works. It is a separate service ON + # PURPOSE — a browser runs code written by strangers, and this container holds the vault key. + # Empty by default, matching the profile above: with no browser running, pointing the api at + # one would only produce timeouts. Set it when you enable the profile. + BROWSER_URL: ${BROWSER_URL:-} + # Where the agent binaries live, if you have them. Leaving this unset is a supported state: # everything except the agent installer works, and the installer answers 503 saying no build is # published. See the README — a self-hosted instance has no way to produce these. AGENT_RELEASE_DIR: ${AGENT_RELEASE_DIR:-} + networks: + - default + - browsernet healthcheck: test: - CMD @@ -231,6 +241,49 @@ services: # First boot syncs the schema and runs the data backfills before it listens. start_period: 40s + # The headless browser, and everything about this service is about keeping it away from the rest. + # + # WHY IT IS NOT PART OF THE API. A browser's whole job is to fetch code written by strangers and + # run it. The api container holds VAULT_MASTER_KEY and a live connection to Postgres and Redis, so + # a browser sharing its network would let any page the assistant is pointed at reach them — and + # reach a cloud metadata endpoint, which is how instance credentials get read out loud. + # + # THE ISOLATION IS THE NETWORK, not the good intentions. This service is on `browsernet` and + # NOTHING ELSE. postgres and redis are on `default`, which it is not a member of, so it cannot + # resolve or reach them at all. The api is on both, because it is the only thing that calls this. + # It also carries no secret, no volume and no database URL: a renderer that escapes lands in a + # container that holds nothing worth having. + # + # Delete this service and unset BROWSER_URL if you would rather Jarvis had no web access. Nothing + # else stops working; the two web tools report themselves unavailable. + browser: + image: ${JARVIS_IMAGE_BROWSER:-git.luxit.be/luxit/jarvis-browser:stable} + # OPT-IN. A profiled service does not start unless you ask for it: + # COMPOSE_PROFILES=browser docker compose up -d + # Web access is a decision, not a default — an assistant that can read the internet is a new + # class of input into a system that administers your customers, so you turn it on deliberately. + profiles: + - browser + restart: unless-stopped + networks: + - browsernet + # A renderer is a memory hog, and a compromised one is a fork bomb. Both are bounded here rather + # than left to take the host down with them. + mem_limit: 1g + pids_limit: 512 + security_opt: + - no-new-privileges:true + healthcheck: + test: + - CMD + - node + - -e + - "fetch('http://localhost:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))" + interval: 30s + timeout: 5s + retries: 3 + start_period: 20s + web: image: ${JARVIS_IMAGE_WEB:-git.luxit.be/luxit/jarvis-web:stable} restart: unless-stopped @@ -247,3 +300,13 @@ volumes: # The keys, deliberately apart from postgres_data. Back it up, and not to the same place: a # database dump is worthless to a thief without this, and worthless to YOU without it either. jarvis_secrets: + +# Two networks, and the split is a security control rather than tidiness. +# +# default is what every service was already on implicitly — postgres, redis, api, web. +# browsernet exists so the browser can be reached by the api and reach the internet, WITHOUT +# being able to resolve or connect to the database and the cache. A service that names no +# network is on default, so nothing changes for the ones that were already there. +networks: + default: + browsernet: diff --git a/kubernetes/02-config.yaml b/kubernetes/02-config.yaml index 28c49c2..842495e 100644 --- a/kubernetes/02-config.yaml +++ b/kubernetes/02-config.yaml @@ -58,6 +58,10 @@ data: # first decides whether a restart leaves a closed transcript or a half-written one. RUN_SHUTDOWN_GRACE_SEC: "25" + # The browser service — a separate pod on purpose, see 09-browser.yaml. COMMENTED OUT to match + # that file: with no browser deployed, setting this only turns "unavailable" into a timeout. + # BROWSER_URL: "http://jarvis-browser:3000" + # No demo organization and no seeded administrator: the install screen is the way in. The seed # refuses an instance that already has accounts anyway, but a Deployment has no business # furnishing anything. diff --git a/kubernetes/09-browser.yaml b/kubernetes/09-browser.yaml new file mode 100644 index 0000000..100c933 --- /dev/null +++ b/kubernetes/09-browser.yaml @@ -0,0 +1,171 @@ +# The headless browser Jarvis is allowed to use, and the policy that keeps it away from everything +# else. +# +# WHY IT IS ITS OWN POD. A browser's entire purpose is to fetch code written by strangers and run it. +# The api pod holds VAULT_MASTER_KEY, the database URL and a live connection to Postgres and Redis; +# putting Chromium in it would let any page the assistant is pointed at reach all three — and reach +# the node's metadata endpoint on 169.254.169.254, which is how instance credentials get read out +# loud. PowerShell is in the api image and this is not, and the difference is exactly that: the +# Exchange module talks to Microsoft, and a browser talks to whoever the model was told to look at. +# +# THE NETWORK POLICY BELOW IS THE REAL CONTROL. `browser.client.ts` refuses private addresses before +# it asks, which is the half that can EXPLAIN itself to the model — but it cannot survive a redirect +# and it cannot survive DNS rebinding, because by then the browser is already connecting. The policy +# can, because it is enforced on the packet. +# +# DELETE THIS FILE (and its line in kustomization.yaml, and BROWSER_URL from 02-config.yaml) if you +# would rather Jarvis had no web access. Nothing else stops working: the two web tools report +# themselves unavailable and every other tool is unaffected. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: jarvis-browser + labels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: browser +spec: + # One is enough and more is not better: each replica is a Chromium, and a render is short. Scale + # this only if the assistant is genuinely browsing concurrently across many conversations. + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: browser + template: + metadata: + labels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: browser + spec: + # No service account token. This pod has no business talking to the Kubernetes API, and a + # mounted token is the first thing anything escaping a renderer would look for. + automountServiceAccountToken: false + containers: + - name: browser + image: git.luxit.be/luxit/jarvis-browser:stable + ports: + - containerPort: 3000 + name: http + # No envFrom, no secretRef, no volume. Deliberately: this container should hold nothing + # worth stealing, so that escaping the renderer is not the same as reaching anything. + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + # Chromium will use whatever it is given on a heavy page. The limit is what stops one + # bad page evicting the api from the same node. + memory: 1Gi + securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true + # `pwuser` in the Playwright image. + runAsUser: 1000 + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault + # NOT read-only: Chromium writes its profile, its cache and its shared memory. The + # writable surface is an emptyDir that dies with the pod rather than the image. + volumeMounts: + - name: tmp + mountPath: /tmp + - name: home + mountPath: /home/pwuser + livenessProbe: + httpGet: + path: /health + port: http + initialDelaySeconds: 20 + periodSeconds: 30 + readinessProbe: + httpGet: + path: /health + port: http + initialDelaySeconds: 5 + periodSeconds: 10 + volumes: + - name: tmp + emptyDir: + # /dev/shm is small by default and Chromium is the classic victim of it; the server + # passes --disable-dev-shm-usage and writes here instead. + sizeLimit: 512Mi + - name: home + emptyDir: + sizeLimit: 512Mi +--- +apiVersion: v1 +kind: Service +metadata: + name: jarvis-browser +spec: + selector: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: browser + ports: + - port: 3000 + targetPort: http + name: http +--- +# WHAT THE BROWSER MAY TALK TO, and it is the file's reason for existing. +# +# Ingress: the api pods, and nothing else. Nothing in this namespace should be calling a browser. +# +# Egress: DNS, and then the internet MINUS every private range. The exception list is the whole +# point — 10/8, 172.16/12 and 192.168/16 are where the cluster and the customer's estate live, and +# 169.254.0.0/16 is the metadata endpoint. Without these `except` clauses a page could ask the +# browser to fetch http://169.254.169.254/ and read the node's cloud credentials back as page text. +# +# NOTE THIS NEEDS A CNI THAT ENFORCES NetworkPolicy. Calico, Cilium and most managed clusters do; +# plain kube-proxy with flannel does NOT, and on such a cluster this file is documentation rather +# than a control. Check before relying on it — `browser.client.ts` still refuses private addresses +# on the near side either way, but that half cannot survive a redirect. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: jarvis-browser +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: browser + policyTypes: + - Ingress + - Egress + ingress: + - from: + - podSelector: + matchLabels: + app.kubernetes.io/name: jarvis + app.kubernetes.io/component: api + ports: + - protocol: TCP + port: 3000 + egress: + # DNS, to the cluster resolver. Without this nothing resolves and every page fails. + - to: + - namespaceSelector: {} + podSelector: + matchLabels: + k8s-app: kube-dns + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + # The public internet, and nothing that is not. + - to: + - ipBlock: + cidr: 0.0.0.0/0 + except: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + - 169.254.0.0/16 + - 127.0.0.0/8 + - 100.64.0.0/10 + ports: + - protocol: TCP + port: 80 + - protocol: TCP + port: 443 diff --git a/kubernetes/kustomization.yaml b/kubernetes/kustomization.yaml index b6be1d2..ceeadaa 100644 --- a/kubernetes/kustomization.yaml +++ b/kubernetes/kustomization.yaml @@ -42,6 +42,11 @@ resources: # Remove this line if you point REDIS_URL at a managed Redis. It deploys three Redis processes and # three sentinels; see the README on what that costs on three nodes. - 08-redis-sentinel.yaml + # The headless browser and the NetworkPolicy that isolates it. COMMENTED OUT: web access is a + # decision rather than a default, and an assistant that can read the internet is a new class of + # input into a system that administers customers. Uncomment this AND set BROWSER_URL in + # 02-config.yaml to enable it; leaving both alone changes nothing. + # - 09-browser.yaml # ─── THE THREE IMAGES ─────────────────────────────────────────────────────────────────────────── #