Kubernetes has given platform teams a consistent way to deploy, scale, and operate containerized applications. Now, many of those same teams are being asked to support AI.

The transition is already underway. According to the CNCF 2025 Annual Cloud Native Survey, 66% of organizations hosting generative AI models use Kubernetes for some or all of their inference workloads. Yet only 7% of organizations deploy AI models daily. That gap highlights an important distinction: running AI on Kubernetes and having a Kubernetes platform ready to operate AI continuously are not the same thing.

The challenge is also showing up within platform teams. The 2025 State of AI in Platform Engineering research found that 35% of platform teams still don’t orchestrate AI workloads, pointing to a gap between AI adoption and the operational platforms needed to support it at scale.

AI doesn’t require platform teams to abandon cloud native practices. Kubernetes, GitOps, observability, automation, and self-service remain valuable foundations. But AI introduces new requirements around compute, scheduling, model delivery, and operations.

So, what needs to change?

Extend the resource model beyond CPU and memory

AI is often discussed as a GPU workload, but production AI pipelines are heterogeneous.

Data preparation, preprocessing, retrieval, orchestration, and application logic may run on CPUs, while training or inference uses GPUs or other accelerators. A single workload may depend on several resource types.

This changes scheduling. Platform teams need to consider accelerator type and availability alongside CPU, memory, topology, and workload characteristics.

Kubernetes is evolving to support this. Dynamic Resource Allocation (DRA), for example, provides a more flexible, declarative way for workloads to request specialized hardware.

The goal isn’t simply to make GPUs available. It is to make a heterogeneous compute part of a consistent Kubernetes resource model.

Extend CI/CD to the model lifecycle

Cloud native teams have made application delivery repeatable through CI/CD and GitOps. AI adds another critical artifact: the model.

Instead of managing only:

Code → build → test → deploy

teams may now need to manage:

Code + model + configuration → evaluate → deploy → observe → update

Models can be large, depend on specific runtimes or hardware, and require evaluation before deployment. Platform teams, therefore, need a clear way to answer the following questions: Which application, model, and configuration are running, and can we reproduce the deployment?

The principle is familiar: changes should remain versioned, repeatable, and auditable. The delivery pipeline simply needs to account for more than application code.

Observe the workload, not just the cluster

Traditional infrastructure metrics remain important, but CPU, memory, and request latency don’t tell the whole story for AI.

Depending on the workload, teams may also need visibility into accelerator utilization and memory usage, scheduling and queue times, inference latency, model loading time, throughput, and endpoint health.

The important point isn’t to create a separate monitoring stack for AI. It is to extend existing cloud-native observability so that infrastructure-, application-, and AI-specific telemetry can be correlated across the workload.

That makes it easier to answer a question that GPU utilization alone cannot: Where is the workload actually waiting?

Give AI developers a golden path

AI developers shouldn’t need to become Kubernetes infrastructure experts to deploy a model.

Platform teams can provide standardized self-service paths that encode common infrastructure and operational decisions:

Model → resources → deployment → endpoint → observability → policy

Developers specify what the workload needs; the platform provides a repeatable implementation.

This is the same platform engineering principle that helped simplify cloud native application delivery. The difference is that the golden path now needs to understand models and accelerators alongside containers, CPU, and memory.

Make AI another production workload

AI introduces new resource types and lifecycle requirements, but many of the underlying operational challenges are familiar.

The cloud native community already has strong patterns for orchestration, declarative infrastructure, automated delivery, observability, policy, and developer self-service. The opportunity is to extend those practices rather than create a parallel operational model for AI.

A Kubernetes platform becomes AI-ready when teams can deploy AI workloads consistently, observe them end-to-end, allocate the right resources, and provide developers with a consistent way to move from experimentation to production.

The question is no longer whether Kubernetes can run AI. It is whether operating AI on Kubernetes can become as routine as operating any other production workload.

Related articles