Operating AI/ML Workloads on Kubernetes: A Headlamp Plugin for Kubeflow

Kubernetes has quietly become the default platform for AI and machine learning.

Whether you run notebook servers for data scientists, schedule distributed training jobs, tune hyperparameters, or orchestrate multi-step ML pipelines, those workloads increasingly land on a Kubernetes cluster.

Kubeflow is one of the most popular ways to assemble that stack, and it does so the Kubernetes-native way: every capability is exposed as a Custom Resource Definition (CRD). That design is a gift to cluster operators, because it means ML workloads can be observed and managed with the same primitives as everything else in the cluster.

But in practice the specialized ML dashboards that ship with these platforms hide the Kubernetes layer underneath. When a notebook is stuck or a training run fails, the operator is often left dropping back to kubectl to find out what actually happened at the Pod level.

This post introduces the Headlamp Kubeflow plugin, which closes that gap by surfacing Kubeflow's custom resources directly inside a general-purpose Kubernetes UI. It is a worked example of a pattern any CRD-heavy platform can follow: meet operators where they already work, and show them the cluster-level truth.

Headlamp itself is an extensible Kubernetes web UI maintained under Kubernetes SIG UI and licensed under Apache 2.0. It runs as a desktop app or in-cluster, and its plugin system lets anyone add first-class views for custom resources. Why operators need a different view Purpose-built ML dashboards help data scientists submit experiments, pipelines, and notebooks.

Cluster operators and site reliability engineers (SREs) troubleshoot the Kubernetes resources underneath, and they ask different questions: Why is a notebook stuck? Is it ImagePullBackOff, OOMKilled, or a Pod waiting on a PersistentVolumeClaim? Which Run resources failed recently across namespaces?

Which parameter set does a Katib Experiment report as optimal? Do TrainJob resources reference the expected TrainingRuntime resources? Which batch workloads are running, and what state does Kubernetes report?

The Headlamp Kubeflow plugin helps answer these questions by reading directly from the Kubernetes API server. It shows Pod conditions, Kubernetes failure reasons, and resources across namespaces without requiring an intermediary ML service or database. What the plugin covers Kubeflow is modular, and teams often install only the components they need.

The plugin discovers the Kubeflow API groups on a cluster and displays only the corresponding sections.

The plugin supports the following component families and API resources: Component Purpose API resources Notebooks Provides development environments such as Jupyter, VS Code, and RStudio Notebook, Profile, PodDefault Pipelines Defines and tracks pipelines, versions, experiments, runs, and schedules Pipeline, PipelineVersion, Run, RecurringRun, Experiment Katib Automates hyperparameter tuning and neural architecture search Experiment, Trial, Suggestion Training Runs distributed training workloads such as PyTorch and TensorFlow jobs TrainJob, TrainingRuntime, ClusterTrainingRuntime Spark Runs large-scale data processing with Apache Spark SparkApplication, ScheduledSparkApplication What you can see Inspect notebook Pods The Notebook detail view shows Pod conditions and their reason and message fields.

It also shows CPU, memory, and GPU requests and limits; volume mounts and their backing types, such as PersistentVolumeClaim, ConfigMap, Secret, or emptyDir; environment variables that reference Secret or ConfigMap objects; sidecar containers; and node tolerations. This view consolidates information that would otherwise require several kubectl describe commands.

Inspect hyperparameter tuning The Katib views show the tuning algorithm, search space, every Trial with its live status, and the current best Trial with its metric values and parameter assignments. They also show the early-stopping configuration and the number of Trial resources that stopped early, so you can follow the search without leaving the cluster UI.

Inspect pipeline state without the backend database The Pipelines views read Kubernetes API resources directly and do not query the Kubeflow Pipelines API service or backend database. You can inspect stored pipeline state even when that service is unavailable. The Pipeline detail view compares the latest and previous PipelineVersion specifications in a side-by-side YAML diff.

Run views show state and duration, RecurringRun views show human-readable schedules, and the artifacts view aggregates pipelineRoot values from recent Run resources. Map ML resources The plugin registers a Headlamp map source that renders Notebook, Profile, PodDefault, Experiment, Pipeline, SparkApplication, and TrainJob resources as graph nodes.

It draws edges between supported resources based on .metadata.ownerReferences. Headlamp also shows inline summaries for these resource types when you hover over them. Try it The Kubeflow plugin README explains installation and local-cluster setup, including a lightweight CRD-only path for evaluation.

Because the plugin discovers installed API groups, you can use it with an existing modular Kubeflow installation or create an evaluation cluster with only the CRDs and sample resources. Apply the pattern to other platforms Kubeflow illustrates a broader pattern. Platforms often model domain-specific workflows with custom resources.

Their dashboards focus on those workflows, while Kubernetes operators also need the state of the underlying API resources and Pods. A CRD-driven plugin in a general Kubernetes UI can expose that state without making operators switch between unrelated tools. The plugin uses the Apache 2.0 license and is developed under Kubernetes SIG UI.

To report a problem or contribute an improvement, use the Headlamp plugins repository's issue tracker or pull requests.