Kubernetes

The manifests in deploy/k8s/ run a complete Nix deployment on a k3s cluster: the application, its database, broker, object store, the Zitadel identity provider and the Sites services. They are rendered with envsubst and applied by deploy/k8s/deploy.sh.

The Kubernetes path is kept for an explicit switch. Every script in deploy/k8s/ and deploy/docker/build-and-push.sh refuses to run unless you set:

export NIX_DEPLOY_TARGET=kubernetes

What you need#

• A k3s cluster, version 1.32 or later, with its bundled Traefik v3 ingress controller. The manifests use the traefik ingress class and Traefik's ipAllowList middleware.

• Traefik's Service set to externalTrafficPolicy: Local, so it sees the real client address. On k3s, set this with a HelmChartConfig for traefik in kube-system. Without it, the Cloudflare-only allowlist answers 403 to everyone.

• The local-path storage class that k3s provides, and a local-path-retain class for the volumes that must outlive their claims (the object store and backups). The repository does not define the second one; create it as a copy of local-path with reclaimPolicy: Retain.

• A domain proxied through Cloudflare, and a Cloudflare Origin CA certificate and key for it.

• kubectl, envsubst, openssl and a checkout of the exact commit you will run.

• Images for that commit. CI publishes multi-architecture images for every commit on main, so you can point REGISTRY at them. deploy/docker/build-and-push.sh builds and pushes your own instead.

Layout#

Secrets#

Run deploy/k8s/create-secrets.sh once per cluster before the first deploy. It generates strong random passwords and keys, derives the connection strings, and creates these Secrets in the nix namespace:

It refuses to run again once nix-db exists, so it can never silently rotate passwords the database does not know. To change one external service later, run it with one of --rabbitmq-only, --object-store-only, --calendar-only, --sites-only, --sites-cloudflare-only, --push-only, --tls-only or --zitadel-only. The sites server's key must be a different, read-only key; the script refuses to reuse the write key.

Values reach kubectl through your shell. Once the deployment is more than a first bring-up, consider sealed secrets or an external secrets operator.

Deploying#

export NIX_DEPLOY_TARGET=kubernetes
REGISTRY=ghcr.io/<owner>/nix TAG=<full-commit-sha> \
OIDC_CLIENT_ID=<client-id> DOMAIN=<your-domain> \
deploy/k8s/deploy.sh

DOMAIN is the app's address and also the sites domain: the app answers at <your-domain>, Zitadel at auth.<your-domain> and sites at <slug>.<your-domain>. OIDC_ISSUER defaults to https://auth.<your-domain>; set it only for an external provider.

The script applies a release in an order that stops a bad step before it reaches users:

1. checks that every required Secret and key exists;

2. removes retired workloads, keeping their volumes;

3. PostgreSQL;

4. the object store, then the storage job;

5. RabbitMQ's configuration and broker;

6. the seed job for database roles;

7. Zitadel's database job and Zitadel;

8. Core and document migrations;

9. template presets;

10. Caddy's configuration;

11. Core, collaboration, the worker, the web app and the Sites services, waiting for each rollout;

12. the Cloudflare-only middleware and the Ingresses;

13. the worker rollout, whose readiness depends on the public origin;

14. the backup CronJob;

15. the managed template library, if enabled.

First install#

A first install has nothing to sign in with yet, so it takes two passes:

1. Set NIX_TEMPLATE_BOOT_ENABLED=false. The managed template library needs a workspace, a Zitadel project and a service account key that do not exist yet; with this set, that step is skipped and everything else is applied.

2. Deploy once with a placeholder OIDC_CLIENT_ID. Zitadel starts with the first administrator from the nix-zitadel Secret, who must change their password at first sign-in. Self-registration is off.

3. Sign in to Zitadel at https://auth.<your-domain>, create the application for Nix, and deploy again with its real client ID.

4. When you want the managed templates, set NIX_TEMPLATE_BOOT_ENABLED=true with the TEMPLATE_BOOT_WORKSPACE_ID, TEMPLATE_BOOT_OIDC_AUDIENCE, TEMPLATE_BOOT_PVC and TEMPLATE_BOOT_SERVICE_KEY_SECRET it requires.

Other settings:

Ingress#

There are three Ingresses, all behind the Cloudflare-only middleware and all using the origin certificate:

1. The app: <your-domain> goes to Caddy in the web container, which does all the path routing; auth.<your-domain> goes to Zitadel.

2. The sites wildcard: *.<your-domain> goes to the sites server. It is a separate Ingress at the lowest router priority, so auth and the app always win.

3. Custom site hostnames: a rule with no host at the lowest priority catches any other name Cloudflare proxies to the origin and sends it to the sites server, which answers only for hostnames that have been verified.

The middleware admits only Cloudflare's published address ranges, so nobody can reach the origin directly and forge the client address header that Caddy trusts. Refresh cloudflare-only.yaml when Cloudflare changes its ranges. Cert-manager is not used.

Network policies#

Fitting a small node#

Requests are deliberately small and limits are capped:

The combined worker exists to fit a single small node. Split it back into one Deployment per role when you have the memory. Zitadel's start-up probe allows several minutes on a slow node.

Checking a deploy#

DOMAIN=<your-domain> SITE_HOST=<slug>.<your-domain> deploy/k8s/verify.sh

verify.sh runs its checks in order, each failing in its own way: pods, Core's health and status, log scans, collaboration, RabbitMQ (vhost, required and retired accounts, exact permissions), the workers, the object store (an unsigned request must be refused), Zitadel's issuer, the sites server, the public app routes, that /internal/authorize answers 404 publicly, that a Content Security Policy is present, that the apex is not served by Sites and, with SITE_HOST, that a site answers with its own policy and sets no cookie.

It cannot check what needs a browser. Do these by hand: sign in end to end, edit the same note in two browsers, export a note as PDF and reload a deep link.