Operations guide

Deploy the platform in a predictable order.

Terraform owns infrastructure and PostgreSQL. Helmfile owns the OpenIssuer application releases. Keeping that boundary clear makes deployment, recovery, and troubleshooting substantially safer.

Terraform

Cluster services, Gateway API dependencies, DNS, certificates, and CloudNativePG.

Sealed Secrets

Registry, service credentials, tenant seed configuration, and other sensitive values.

Helmfile

Application releases and their repository-specific Helm values in the configured namespace.

Namespace convention

The reference deployment currently uses main, but OpenIssuer does not require that namespace. Set the namespace consistently in Terraform, Helmfile, Helm values, secrets, and Gateway API resources. The commands below use an environment variable so the value can be changed.

export NAMESPACE=main

1. Provision platform resources

Run the deployment Makefile from the Terraform repository root:

make platform-init
make platform-apply
make platform-output
make generate-utils-vars
make utils-init
make utils-apply

The Makefile loads secret Terraform variables from macOS Keychain. Do not commit DigitalOcean tokens, PostgreSQL passwords, or generated tfvars.

2. Verify PostgreSQL

OpenIssuer uses one CloudNativePG cluster per application database. The reference configuration provisions 5 GiB volumes in the configured namespace.

kubectl get clusters.postgresql.cnpg.io -n "$NAMESPACE"
kubectl get pvc -n "$NAMESPACE"
kubectl get pods -n "$NAMESPACE"

Continue only when every required database cluster reports a healthy primary and its PVC is bound.

3. Apply application secrets

Create or refresh the registry and service secrets through the repository targets:

make github-registry-secret
make account-rest-service-secret
make email-rest-service-secret
make do-s3-secret

Tenant seed values are converted into SPRING_APPLICATION_JSONand sealed before they are stored in the cluster. For the current zsh-based local setup, run:

NAMESPACE="$NAMESPACE" ./scripts/apply-authorization-seed-from-zshrc.sh
kubectl get sealedsecret authorization-seed -n "$NAMESPACE"
kubectl get secret authorization-seed -n "$NAMESPACE"

4. Deploy applications with Helmfile

From the deployment repository root, synchronize all managed releases:

helmfile sync

Deploy or update one application by its Helmfile release name:

helmfile -l name=authorization-server sync
helmfile -l name=attempt-rest-service sync

Helmfile release names match Kubernetes workload names. Internal WebClient calls resolve Kubernetes Service names supplied by each chart'sfullnameOverride.

5. Handle reusable image tags

A tag such as latest can point to a new image without changing the Deployment manifest. Helmfile therefore may have nothing to update. Force new pods after the image has been published:

kubectl rollout restart deployment/authorization-server -n "$NAMESPACE"
kubectl rollout status deployment/authorization-server -n "$NAMESPACE"

RESTARTS counts container restarts inside a pod. A successful rollout normally creates a new pod with a recent age and a restart count of zero.

6. Verify a release

  1. Confirm CloudNativePG clusters report a healthy primary before starting applications.
  2. Confirm required Kubernetes and Sealed Secrets exist in the configured namespace.
  3. Run Helmfile from the deployment repository so sibling values paths resolve correctly.
  4. Wait for deployment rollouts and inspect pod age, readiness, and logs.
  5. Smoke-test tenant login, OAuth callback, admin access, and passkey management.
helm list -n "$NAMESPACE"
kubectl get deployments,pods,httproutes -n "$NAMESPACE"
kubectl logs -n "$NAMESPACE" deployment/authorization-server --tail=100

Database reset and recovery

Deleting CloudNativePG clusters or PVCs permanently removes application data. Scale dependent applications down first, confirm backups or accept the data loss, remove only the intended database resources, and reapply Terraform from the utils directory. Start applications only after every recreated cluster and PVC is healthy.

Do not use broad deletion commands until kubectl get output has been reviewed. Terraform must be run from platform orutils; running it from the repository root has no configuration.

Rollback and diagnose

Image did not change
Confirm the registry build completed, then force a rollout for reusable tags.
InvalidImageName
Inspect the release values and confirm both image repository and tag are present.
Helm operation in progress
Inspect Helm history and release status before retrying or rolling back.
Route returns 404 or misses CSS
Inspect the HTTPRoute path prefixes and verify the request reaches the intended Service.
Application starts before its schema exists
Verify database readiness and review Flyway output before restarting the service.
helm history authorization-server -n "$NAMESPACE"
helm rollback authorization-server REVISION -n "$NAMESPACE"
kubectl describe pod POD_NAME -n "$NAMESPACE"