Closing the Gap Between Customers and Their On-Prem Deployment

The first post in a series on what we've learned shipping and supporting Scensei OPTIMA on-premises. On-premises deployments come with a support dynamic that cloud-hosted products rarely face: when something goes wrong, you often cannot access it yourself, if you’re not physically present at the customers’ premises.

What the reality often looks like

Some problems cannot be moved to the cloud. The organizations we work with build systems that model the physical world in high fidelity — radar coverage under different atmospheric conditions, detection thresholds across terrain configurations, the performance envelope of sensors tracking fast-moving targets. The data those simulations produce is sensitive by nature, and the infrastructure running them is isolated by design. Air-gapped, sovereign, and fully under the customer's control.

Managed platforms like Nuvibit have made Kubernetes genuinely approachable for teams that want the operational benefits without the overhead — but they assume an internet-connected environment and infrastructure you can delegate. Our customers cannot delegate it. They own the hardware, they own the network perimeter, and they need the same quality of deployment and support experience on infrastructure that never touches a public endpoint.

That is the constraint we have built around. And it changes what a support call looks like.

The person on the other end is not a Kubernetes specialist. They are a system administrator, a developer, or sometimes just the person who was asked to handle the deployment. You have to guide them through diagnosing a problem in an environment you cannot see.

One early case made that gap concrete. A customer's simulation pod was failing silently. The culprit was a misconfigured PVC mount point — the volume was attached, the pod was running, but the path the application expected and the path the filesystem actually exposed were different by one directory level. To confirm it, we needed someone on their side to open a shell inside the pod and run ls. That is it. But getting there with raw kubectl meant walking a non-specialist through finding the exact pod name across a wall of output, constructing the exec command from scratch, and keeping the session alive long enough to get the information we needed. A five-minute diagnosis took the better part of an hour.

A tool that meets customers where they are

KubePilot is a single binary we ship alongside OPTIMA. Run it, use the arrow keys to pick a component from a list, and choose what you want to do: stream its logs, forward a port to it, or open a terminal inside it.

The value is not the feature set — kubectl can do all of the same things. The value is that a customer can follow a support engineer's instructions on a screen share without the call derailing because they misread a pod name or got the port-forwarding command wrong. The value is that in case of an incident, the customer can already provide actionable context to the report before any of your engineers even log in. And finally, the value is that support calls can stay what they're supposed to be — actual support calls and not ad hoc Kubernetes tutorials.

Why this matters for air-gapped deployments specifically

Air-gapped deployments are a core part of what OPTIMA supports, and they sharpen this problem considerably. In an air-gapped environment, the support engineer by design cannot access the system remotely. Diagnosis is a collaborative, guided process — and every point of friction costs real time.

Port-forwarding is a good example. In an air-gapped cluster with no external ingress, forwarding a port from a pod to a customer's local machine is often the only way to access the application directly during a support session. It lets both parties look at the same screen, test the same endpoint, and reproduce the same issue.

The same logic applies to log streaming and shell access. The faster a customer can hand us a log tail or confirm what is mounted where, the faster we close the incident.

The broader point

KubePilot is a small tool. It does one job, does it well and exits. The typical operations described above require several kubectl steps that are easy to get wrong. With KubePilot it is one command and a keypress.

Customers choose on-prem for reasons that matter to them — data sovereignty, compliance requirements, network isolation. They should not have to pay for that choice with a worse support experience. Developing tools like KubePilot is one of the many steps we take to ensure our customers have a seamless support experience in on-prem deployment.

KubePilot is open-source and available on GitHub.