Deploying OpenRun on Kubernetes
On Kubernetes, OpenRun runs as a Deployment and starts each containerized app as its own Deployment and Service. Images are built in-cluster with Kaniko and pushed to a shared registry; app metadata lives in Postgres. This page covers:
- Installation with the Helm chart, and registry config for the node container runtime
- Install using Terraform on AWS (EKS, ECR, RDS)
- Configuration: enabling Kubernetes mode and namespaces
- Resources and autoscaling per app
- Binding providers on multi-replica installs
- Architecture: how resources are created, rolled out and cleaned up
Installation
OpenRun can be installed on a Kubernetes cluster using the Helm chart.
helm repo add openrun https://openrundev.github.io/openrun-helm-charts/
# Install a registry for use with OpenRun (for testing)
helm install openrun1 openrun/openrun \
--namespace openrun --create-namespace --set registry.enabled=true
# Install with an external registry (recommended)
helm install openrun1 openrun/openrun \
--namespace openrun --create-namespace --set config.registry.url=<registry_url> --set config.registry.insecure=falseRunning the Helm chart creates:
- A service which runs the OpenRun API server
- Optionally, a Postgres database for metadata. An external Postgres database can be used instead.
- Optionally, a container registry is installed. An external registry can be used instead.
| Component | Requirement |
|---|---|
| Metadata | Postgres. SQLite based metadata is not supported when using Kubernetes. The Postgres metadata needs to be managed properly, with backups and scheduled upgrades. An externally managed Postgres installation is recommended for production Kubernetes OpenRun installations. |
| Registry | Required. The registry can be installed through the Helm chart or an external registry can be used. |
An installation with external Postgres and Registry is shown below.
Registry Config
OpenRun on Kubernetes requires a registry to which Kaniko built images can be pushed and from which pods can pull. Image pulls require a registry trusted by the node container runtime. A private CA can be configured on each node; for example, K3s supports ca_file in its private registry configuration. An alternative for local testing is an explicitly configured HTTP registry. Details depend on the Kubernetes installation being used.
Install the registry:2 images as a service. If registry is started at registry.svc.cluster.local:5000, edit /etc/rancher/k3s/registries.yaml to add:
mirrors:
"registry.svc.cluster.local:5000":
endpoint:
- "http://registry.svc.cluster.local:5000"The registry IP might have to added to /etc/hosts depending on how it is accessed.
The Helm chart sets insecure = true by default. Change that to insecure = false if using a HTTPS registry.
Install using Terraform
The Terraform config creates a production-ready OpenRun installation on AWS, with EKS cluster, ECR registry and RDS Postgres for metadata.
Check out and configure
- Check out the terraform config
git clone git@github.com:openrundev/openrun.git - Switch to the config directory
cd openrun/deploy/terraform - Create a copy of sample config
cp tfvars.sample terraform.tfvars - Update
terraform.tfvars, set the values as appropriate.
Prepare the tools
- Ensure that AWS CLI is installed and credentials are configured.
- Ensure that Helm CLI is installed and the OpenRun repo is updated:
helm repo add openrun https://openrundev.github.io/openrun-helm-charts/; helm repo update - Terraform state is by default saved on local disk. If required, update to use S3.
Apply
- Run
terraform init. Add-backend-configif using remote state. - Run
terraform plan --var-file terraform.tfvars, verify the plan. - Run
terraform apply --var-file terraform.tfvars, or apply the plan created earlier.
After installation
- Save the password for the admin user using
terraform output openrun_admin_password. - Add the DNS entries as output under
openrun_dns_records. Theroot_aandwildcard_aDNS entries enable installing apps at the domain level. - Wait for the DNS entries to propagate before attempting to access the url (to allow TLS cert creation to work).
- Run the CLI inside an OpenRun server pod with
kubectl exec -n openrun -it <openrun-pod> -- openrun sync schedule --approve --promote <git-config-path>to set up sync. All subsequent operations are done by checking in config updates to the app config in Git.
To destroy the resource created,
- Ensure that kubectl config is set, like
aws eks update-kubeconfig --region us-west-2 --name openrun-eks --profile openrun. Update the region, the name format is<prefix>-eks, defaultopenrun-eks. - Run
terraform destroy --var-file terraform.tfvars. Run it a second time in case there is a timeout. All resource should be deleted.
Configuration
[system]
container_command = "kubernetes"is the main config which enables Kubernetes mode. By default, Kaniko based builds are used. Delegated builds can be configured instead, see delegated builds. See registry for registry config.
| Namespace | Default | Used for |
|---|---|---|
| Main | openrun | OpenRun service and Kaniko jobs. When the server runs as a pod, its own namespace is used; otherwise kubernetes.namespace in openrun.toml sets it. |
| Apps | <main_ns>-apps (openrun-apps) | Applications. Automatically created by the Helm chart; it has to exist before apps are started. |
[kubernetes]
namespace = "openrun" # main namespace, used when the server is not running as a pod
use_node_port = false # expose app Services as NodePort instead of ClusterIP, for a k3s style
# single node cluster where the OpenRun server runs outside the clusterTo clear all apps (including any volume data), run kubectl delete namespace openrun-apps; kubectl create namespace openrun-apps.
Resources and Autoscaling
Set resource requests and replica counts with container_opts in an app declaration:
app("/api", "github.com/myorg/api", spec="container",
container_opts={
"cpus": "0.5",
"memory": "512m",
"kubernetes.min_replicas": "2",
"kubernetes.max_replicas": "5",
},
app_config={"container.idle_shutdown_secs": 0})| Option | Default | Effect |
|---|---|---|
cpus | The app container’s CPU request, without a CPU limit | |
memory | Both the memory request and limit of the app container | |
kubernetes.min_replicas | 1 | Initial replica count, for stateless apps |
kubernetes.max_replicas | When greater than 1, OpenRun creates a HorizontalPodAutoscaler with a minimum of at least one replica | |
app_config.kubernetes.scaling_threshold_cpu | 80 | Target CPU utilization (percent) of the HPA, relative to the CPU request |
Kubernetes-specific options can use the kubernetes. prefix so they are ignored by Docker and Podman.
- HPA metric: with app sidecars, the metric measures the main app container’s CPU. Memory limits do not enable memory-based autoscaling.
- Metrics requirement: configure a CPU request and ensure the cluster provides resource metrics, typically through Metrics Server. See the Kubernetes HPA documentation for metrics requirements.
- Idle shutdown: OpenRun’s idle shutdown still applies independently of replica minimums. The example disables it with
container.idle_shutdown_secs = 0to keep the app running after startup. Keep the default idle timeout if the app should scale to zero without traffic. - Lazy creation: resources are created on first access or deployment verification.
- Persistent volumes: apps with persistent volumes, including SQLite bindings, always run with one replica and the
Recreatestrategy. OpenRun ignores higher replica counts and does not create an HPA for these apps.
Binding Providers
Out-of-process binding providers (sqlserver, oracle, mongodb, snowflake, clickhouse, databricks) work on Kubernetes without any special setup:
- Source of truth: installed providers are registered in the metadata database (Postgres, required for Kubernetes installs).
- Per replica: each server replica materializes the provider executables into a node-local cache directory at startup, before serving traffic, and re-downloads them (with checksum verification) whenever a pod starts on a fresh node.
- Propagation: installs and uninstalls propagate to all replicas through Postgres notifications, so
openrun provider installworks on multi-replica deployments exactly as on a single server. - License: the providers are not Apache 2.0 licensed. They are under the PolyForm Free Trial License, see binding providers for the license terms.
Declarative install
For declarative, config-managed deployments, declare the providers in the Helm values instead of using the CLI:
bindings:
install:
mongodb: v0.1.0
sqlserver: v0.1.0
# Optional mirror for clusters with restricted egress; {provider}, {version},
# {os} and {arch} are replaced.
# releaseUrlTemplate: "https://mirror.internal/{provider}/{version}/openrun-binding-{provider}-{os}-{arch}"or with --set bindings.install.mongodb=v0.1.0 on the Helm command line. For Terraform-managed EKS installs, set the openrun_binding_providers variable in terraform.tfvars:
openrun_binding_providers = {
mongodb = "v0.1.0"
sqlserver = "v0.1.0"
}The chart renders these into the [bindings.install] section of the generated server config:
[bindings.install]
mongodb = "v0.1.0"
sqlserver = "v0.1.0"- Startup install: every replica installs the declared providers at startup, downloading them from
bindings.release_url_template(the openrundev/bindings GitHub releases by default; point it at an internal mirror when the cluster restricts egress). - Digest pinning: append
@sha256:<hex>to a version to pin the binary digest, GitOps-style; the install fails before anything is written or executed if the download does not match. - Mixed-architecture clusters: list one digest per platform separated by commas (
v0.1.0@sha256:<amd64-hex>,<arm64-hex>); each replica’s download must match one of them. - CLI interaction: providers declared this way cannot be modified with the provider CLI; version upgrades are config changes followed by a rollout. Removing an entry does not uninstall the provider: it stays installed and becomes manageable with the provider CLI (
openrun provider uninstall). Providers installed imperatively withopenrun provider installcan be used alongside config-declared ones.
Recovery and provider process details
- A replica that misses an install notification (for example after a transient database disconnect) recovers on demand: a binding operation against a service type that is installed but not yet registered triggers a reconcile of that provider before the operation proceeds.
- The provider processes run inside the OpenRun server pod, so a provider binary must run in the server container image; the published providers are static pure-Go binaries and need nothing beyond the stock image.
- Provider startup materialization completes before the server starts serving, so binding operations never race provider registration during a rollout.
OCI image install
Providers are also published as minimal OCI images (ghcr.io/openrundev/openrun-binding-<provider>, a FROM scratch image holding the provider binary). For clusters where binaries must come through image provenance policies, or where the server pods have no egress at all, declare the providers as images instead of versions:
bindings:
images:
mongodb: ghcr.io/openrundev/openrun-binding-mongodb:v0.1.0
sqlserver: ghcr.io/openrundev/openrun-binding-sqlserver@sha256:8f4e...- How it works: the Helm chart renders one init container per provider, which copies the binary from its image into a shared volume (
exportsubcommand of the provider binary). The server registers the pre-placed providers at startup, with no downloads and no database registration. - Integrity: comes from the image digests that placed the binaries (reference images by digest where policy requires it). The server pins the discovered binary’s checksum so every provider launch verifies the file has not changed.
- Image pulls: images are pulled by the container runtime, so
imagePullSecretsand registry mirrors apply as for any other image.
With either declarative path, bindings.disableInstall: true additionally rejects the imperative openrun provider install/uninstall CLI on the deployment, making the chart values the only way providers are added.
Architecture
OpenRun is installed as a Kubernetes Deployment, with a Service for routing API calls. For each containerized app installed on OpenRun, a ClusterIP Service is created for app traffic. API calls to the OpenRun API Server are routed to the app-specific Service using its cluster IP.
- Lazy creation: all Kubernetes resources are created lazily, on the first API call to the app (or when reload/apply is done with the verify option). If an app is running version 1, and a code/config change updates it to version 2, the deployment update happens on the next API call to the app. That API call is blocked while the container image is rebuilt if required and the Kubernetes resources are updated using Server Side Apply API calls.
- Rollout wait: OpenRun waits until Kubernetes reports the expected new version rollout is complete before processing further API calls. It watches Deployment rollout status so readiness and Kubernetes-declared rollout failures are detected without waiting for the next polling interval, while still honoring the
container.deploy_health_attemptsdeployment wait budget.
Stateless apps
- OpenRun deploys each new version as a separate Deployment and promotes it by switching the app Service selector after the new Deployment is ready. This keeps the previous version serving traffic until the replacement is healthy.
- After promotion, OpenRun waits briefly for the Service’s EndpointSlices to route only to ready pods from the new version, then removes inactive Deployments for the app and their owned HPA, Secret, and ConfigMap objects.
- The EndpointSlice wait is best-effort; if the cluster API or OpenRun’s Kubernetes RBAC permissions do not allow listing EndpointSlices, OpenRun skips that wait instead of delaying the deployment.
Apps with persistent volumes
Apps with OpenRun-managed persistent volumes use a stable Deployment name with a single replica and a Recreate update strategy, because a ReadWriteOnce PVC cannot be mounted by two running pods at the same time. There is no second version to switch traffic to, so a new version is rolled out in place: the previous pod is terminated and the new one started, a brief window in which the app is not served.
Rollout during an update operation (reload, apply, sync)
- The live Deployment is not touched before the operation’s metadata transaction commits: the new version’s objects are validated with a server-side dry run beforehand, and the rollout runs when the operation commits.
- If another app in the same operation fails, the persistent-volume app stays on its current version, with its Deployment untouched.
- If the rollout itself fails at commit, the Deployment (and its owned ConfigMaps and Secrets) is restored to the previous version and the operation reports the failure. The app metadata is already on the new version, so the next request retries the rollout, and
openrun version revertswitches the app back if the new version cannot run. - PVCs are preserved when an app moves from a persistent-volume configuration to a stateless configuration. If the app later adds the same volume again, the existing PVC is reused and its previous data is still present.
Image-spec apps
- OpenRun resolves the supplied image reference directly. Public images such as
nginxare pulled from their normal registry instead of being rewritten to the OpenRun build registry. - When the registry returns a digest, OpenRun pins the pod image to that digest. If a later refresh has a transient registry failure, OpenRun keeps using the last known digest; missing images, invalid references, and registry authorization errors fail the deployment instead of falling back to a floating tag.
Secrets and ConfigMaps
- Secrets and ConfigMaps generated for mounted app config are named from the workload name and a short hash suffix so that Kubernetes object names stay within the 63-character DNS label limit.
- The app’s environment (resolved
{{secret}}parameters and binding credentials included) is stored in a Secret named<workload>-envwhich the container loads withenvFrom, never as literal values in the Deployment or Job spec, so reading the pod template does not reveal it. - The
<workload>-envSecret carries the workload’s ownership labels and is snapshotted, restored and removed with the version’s other Secrets; a job run’s Secret is owned by its Job and removed with it.