Server operations

Exec plugins and cluster access#

The server runs the commands of your kubeconfigs' exec credential plugins (aws eks get-token, gke-gcloud-auth-plugin, kubelogin…) on the server, as the server's user. The container image is minimal and contains none of them: build your own image on top of it, or mount the binaries and put them on PATH.

FROM harbor.tyrola.dev/kubyl/server:latest
COPY --chmod=755 aws-iam-authenticator /usr/local/bin/

Plugins that need a browser or a login you do once (az login, gcloud auth login) must have their state in a directory the server's user can read: mount it and set the plugin's home variable. OIDC logins with the device-code flow or the browser flow can also be done from the phone (Clusters, Sign in); the server keeps the refresh token.

Network: Tailscale, WireGuard#

The server holds every cluster's credentials, so it should be reachable only from your own devices.

• Put the server and your phone on a Tailscale tailnet or a WireGuard VPN and use the VPN address or name as KUBYL_SERVER_PUBLIC_URL, for example https://kubyl.tail1234.ts.net:8443. Bind the server to that interface with --listen 100.x.y.z:8443 when it runs on the host, or publish the container port only on it (-p 100.x.y.z:8443:8443).
• Do not forward the port from the internet. The pairing codes and the device limit are no substitute for a private network.
• Do not terminate TLS in front of the server (a reverse proxy, tailscale serve): the app pins the server's own certificate key, and a proxy presents another one. If you must, use a proxy that passes TLS through. If you give the server your own certificate (--tls-cert, --tls-key), a renewal that keeps the private key does not unpair devices; a new key does, so re-pair.
• Helm chart: the default is a ClusterIP Service, reachable inside the cluster only (use your VPN's Kubernetes operator). service.type: NodePort or LoadBalancer, ingress and gatewayAPI.tlsRoute are off by default and make the server reachable from outside at your own risk. They must pass TLS through (ingress-nginx ssl-passthrough, a Gateway API TLSRoute). tls.certManager (or tls.existingSecret) gives the server a real CA certificate with a key that survives renewals; restart the pod after a renewal.
• If you put a proxy in front of the web UI anyway: requests that carry forwarding headers (X-Forwarded-For, Forwarded, X-Real-IP…) are never treated as “on this machine”, so they need the admin token.

Web UI#

Open https://<server>:8443/ in a browser. Accept the certificate warning once, or compare its public key hash with the one on the Status page. It has four pages: Pair (QR code and link, new code), Devices (list, revoke), License (state, activate, apply a lease file, renew now, remove) and Status (version, clusters and their connection state).

• A browser on the server's own machine (the address is localhost or an IP) gets in without a token.
• Everywhere else you sign in with the admin token. It is made at the first start, shown once (in the console's first screen, or on stdout without a terminal) and only its hash is kept. In the console T makes a new one: the old one stops working, open sessions stay.
• The session is an HTTP-only SameSite=Strict cookie, and every change needs a CSRF token. Failed sign-ins are limited per address.

Console keys#

Tab and Shift Tab or 1 to 4 switch tabs, ? lists the keys, Q quits (it asks first).

Windows smaller than 60x20 show “make the window larger”. Without a terminal, kill -USR1 <pid> prints a new pairing code to stdout.

Push notifications#

Kubyl Mobile can notify you about firing Alertmanager alerts, failed Argo CD syncs and nodes that go NotReady. The phone registers its push token with your server; the server decides what is worth a notification and sends it through a relay at https://kubyl.dev/api/push (change it with push.relay_url; it must be https). The relay holds the Apple and Google credentials and forwards the message; it keeps no tokens and no payloads.

• Each notification is encrypted for one device with a key the phone made and keeps in its Keychain or Keystore. Apple, Google and the relay see only a generic text (“2 alerts firing”) and opaque ids. Cluster names, alert text and object names are inside the encrypted part.
• The relay checks your install's signature and that its license is active, so push stops when the server is Unlicensed or Locked.
• Set what you want per device and per cluster in the app: alerts firing or resolved, sync failed, node not ready, minimum severity and clusters.
• The server sends at most 10 pushes per device per 10 minutes (the rest fold into one message) and never repeats the same event within 15 minutes.
• The server watches every connected cluster that some device wants to hear about. It connects to them for that, even when no phone is open.

What the mobile client can and cannot do#

Can: browse every cluster and resource type with live tables, open objects as JSON or YAML, read logs (follow, previous, since), open a shell in a pod or on a node, see metrics and Prometheus charts, alerts and silences, Argo CD applications (sync, rollback, refresh), Helm releases (read-only), OLM operators and upgrade approvals, what each cluster runs and could be updated to, describe and related objects, pod files, port-forward and web views through a one-time address. It can also make changes: delete, scale, rollout restart and undo, cordon and drain, CronJob run and suspend, and YAML apply with a dry-run diff.

Every change is two steps: prepare shows what would happen, confirm applies it. The server enforces your rules: a cluster marked read-only refuses writes, and on a production cluster confirming needs the cluster's name typed in. Set both per cluster in settings.json with kubernetes.contexts.<cluster id>.read_only and .production.

Cannot: see Secret values (they are masked in objects, YAML and manifests, and Secrets cannot be applied or edited from the phone), write Helm releases, or use the desktop app through the server (the kube-API proxy for Kubyl on the desktop comes after the mobile release).

Updates#

Pull a newer image tag and restart the container; the config volume carries everything over. Each image ships THIRD-PARTY-NOTICES.html in /usr/share/doc/kubyl-server/, with the open-source licenses of everything linked into the binary.

Troubleshooting#

• The phone says the certificate doesn't match. The server's key changed (config directory lost, or a new --tls-key): pair again.
• `license_required` in the app. The server is Unlicensed or Locked: open the License tab.
• `device_limit` when pairing. Your plan allows 3 paired devices: revoke one on the Devices tab.
• The server won't start: “can't decrypt identity.enc”. The key doesn't match the store: restore the right KUBYL_SERVER_KEY or credentials.key.
• The web UI asks for a token on the server's own machine. You reached it through a proxy or a host name that isn't localhost or an IP: use the admin token or open https://localhost:8443/.