Production Security Hardening
This guide is the production security baseline for KubeOsprey. It applies to Helm installations, the standalone Kubernetes manifest, and Cluster Agent deployments.
KubeOsprey is a Kubernetes control plane. It can create privileged Kubernetes clients, terminal pods, exec sessions, proxy requests, and AI-assisted resource changes. Treat a KubeOsprey installation as a cluster administration boundary.
Production baseline
Before exposing KubeOsprey to users, verify all of the following:
- Use HTTPS for KubeOsprey, OAuth/OIDC callbacks, Cluster Agent connections, and WebSocket traffic.
- Set
KUBE_OSPREY_ENCRYPT_KEYto a high-entropy value and store it in an external Secret. Never use the public example value from development documentation. - Set
JWT_SECRETexplicitly when you need stable sessions across restarts or multiple replicas. - Set
ANONYMOUS_USER_ENABLED=falseand verify that anonymous access is disabled. - Use a narrow
TRUSTED_PROXIESlist. Do not trust arbitrary client-supplied forwarding headers. SetTRUSTED_PROXIES=noneto make KubeOsprey ignore all client-supplied forwarding headers. - Use an external PostgreSQL or MySQL database, or provision a persistent volume for SQLite.
- Pin KubeOsprey, terminal, and Cluster Agent images to a release tag or digest. Do not use
latestin production. - Review every
cluster-admin, wildcard RBAC rule, privileged container, host namespace, and host path before deployment. - Enable only the terminal and Cluster Agent capabilities that your operators need.
- Back up the database and the encryption key together. A database backup without its encryption key cannot restore encrypted fields.
Secrets and encryption
SecretString fields in the database include credentials such as kubeconfig data, OAuth and LDAP secrets, Helm repository passwords, API keys, MFA secrets, and Cluster Agent private keys. They are encrypted with KUBE_OSPREY_ENCRYPT_KEY.
User passwords are stored as one-way password hashes; they are not recovered with KUBE_OSPREY_ENCRYPT_KEY.
Use an existing Kubernetes Secret when possible:
secret:
create: false
existingSecret: kube-osprey-secretsKUBE_OSPREY_ENCRYPT_KEY is validated at startup. KubeOsprey refuses to start when the key is empty, shorter than 32 characters, or still set to the built-in development default. The only escape hatch is KUBE_OSPREY_ALLOW_INSECURE_DEFAULTS=true, which is intended for local development only.
The Secret must contain the keys required by the selected deployment mode. Keep the encryption key stable while restoring an existing database. Rotating it requires a planned migration that can decrypt with the old key and re-encrypt with the new key.
Do not put real keys in values.yaml, Git, shell history, issue trackers, or generated manifests.
Kubernetes permissions
KubeOsprey has application-level RBAC, while Kubernetes separately evaluates the ServiceAccount or kubeconfig used by the KubeOsprey server and Cluster Agent. The effective permission is the intersection of both systems, but an overly privileged Kubernetes identity can still make a mistake or compromised session more damaging.
Review these areas separately:
- KubeOsprey server ServiceAccount and ClusterRole;
- Cluster Agent ServiceAccount and ClusterRoleBinding;
- kubectl terminal ServiceAccount and ClusterRoleBinding;
- Node Terminal host access, privileged mode, host namespaces, and host paths;
imagePullSecretsand the namespaces where terminal pods are created.
Avoid wildcard rules such as apiGroups: ["*"], resources: ["*"], and verbs: ["*"] for production unless the installation is intentionally an administrator-only break-glass system.
Cluster Agent manifests support permission profiles:
readOnly: read-only on all resources.operator: read/write on common workload resources (pods including exec/attach/portforward, services, configmaps, PVCs, deployments, statefulsets, daemonsets, replicasets, jobs, cronjobs, ingresses, network policies, PDBs, HPAs) plus read-only on namespaces, nodes, persistent volumes, storage classes, events, CRDs, metrics, and Secrets. Secret and RBAC mutation is excluded.admin: bindscluster-admin(legacy high-trust behavior).custom: binds an existing ClusterRole of your choice.
The default profile is admin for backward compatibility; prefer operator or readOnly for production. Note that Secrets are readable under operator because the KubeOsprey console and Helm release browsing read Secret-backed data; use custom if that is unacceptable.
ClusterRoleBinding roleRef is immutable in Kubernetes. When switching an existing Cluster Agent to a different profile, delete the existing Cluster Agent ClusterRoleBinding before applying the new manifest (named kube-osprey-cluster-agent; installs upgraded from a pre-rename release use kite-cluster-agent).
Terminal and exec capabilities
Pod exec, kubectl terminal, and node terminal are not ordinary read-only features:
- Pod exec can run commands inside workloads.
- kubectl terminal can operate on the cluster using its configured ServiceAccount.
- Node terminal can expose host-level capabilities.
Use explicit administrator permissions, short sessions, automatic cleanup, and audit logging. Keep kubectl and node terminal disabled unless they are required. Test that disabled capabilities do not create ServiceAccounts, bindings, or agent pods.
Prometheus and raw PromQL
The PromQL Explorer is a cluster-level observability tool. It should be granted only to users who are allowed to query the configured Prometheus data source. Kubernetes namespace RBAC does not automatically filter Prometheus results.
If KubeOsprey is used as a multi-tenant platform, enforce tenant isolation in the Prometheus layer or replace unrestricted PromQL with approved query templates. Do not attempt to provide isolation by blindly appending a namespace selector to arbitrary PromQL.
Database and persistence
The default SQLite setup is suitable for evaluation only. Without a persistent volume, replacing the Pod loses the database. For production:
- Prefer PostgreSQL or MySQL for multiple replicas and operational backups.
- If using SQLite, provision a persistent volume and use a deployment strategy compatible with a single-writer volume.
- Test restore procedures, not only backup creation.
- Back up the database, encryption key, and deployment configuration as one recovery set.
Audit policy
Every state-changing console request is written to the operation audit log. The policy is configured in Settings → Audit:
- Log retention — how many days operation logs are kept (1–3650, default 30). Older entries are purged automatically.
- Skipped modules — changes in the selected modules are not written to the audit log. Keep the list as small as possible; skipping a module means its changes leave no audit trail.
If the audit policy cannot be read, auditing fails open: mutations keep being recorded with the default retention.
Cluster Agent checklist
- Use an HTTPS server URL with a certificate trusted by the Agent.
- Treat the one-time Agent token and generated YAML as credentials.
- Rotate or revoke the Agent token after suspected exposure.
- Verify the Agent image and digest before applying the manifest.
- Inspect the ServiceAccount, ClusterRoleBinding, Secret, and Deployment before applying.
- When changing the permission profile, delete the existing ClusterRoleBinding first (
roleRefcannot be updated in place; namedkube-osprey-cluster-agent, orkite-cluster-agenton installs upgraded from a pre-rename release). - Ensure the reverse proxy supports WebSocket upgrade and long-lived connections.
- For multiple KubeOsprey replicas, validate session routing and shared state before production use.
Verification commands
Render and lint the Chart:
helm lint charts/kube-osprey
helm template kube-osprey charts/kube-osprey --namespace kube-osprey-system
bash scripts/validate-chart.shReview effective Kubernetes permissions in a test cluster:
kubectl auth can-i --list \
--as=system:serviceaccount:<namespace>:<service-account>
kubectl get clusterrolebinding <binding-name> -o yaml
kubectl get serviceaccount <service-account> -n <namespace> -o yamlRun application checks:
go test ./...
pnpm --dir ui exec tsc --noEmit -p tsconfig.app.jsonCheck readiness separately from process liveness. A process can be running while its database, encryption key, or cluster configuration is unusable.
Current limitations and roadmap
The following items require separate design work rather than a blind permission reduction:
- Cluster Agent currently uses its registered Kubernetes credentials when forwarding requests; it does not impersonate the KubeOsprey user at the Kubernetes API.
- kubectl terminal sessions now support configurable permission modes (
admin/operator/custom) with per-session roles; node terminal still requires break-glass controls and a more narrowly scoped identity. - Encryption-key rotation needs a migration path for existing
SecretStringvalues. - Raw PromQL is not namespace-rewritten by KubeOsprey. Multi-tenant Prometheus isolation must be provided by the metrics architecture or a restricted query model.
- Passkey ceremonies, Cluster Agent sessions, and other in-memory state need shared storage before a multi-replica deployment is considered fully supported.