An AI factory is not just a model or a cluster. It is a pool of GPUs that many teams draw from at once: one team fine-tuning, another serving inference, a third running evaluations, all on the same accelerators. NVIDIA frames it as “infrastructure for the full AI lifecycle, from data preparation through training, fine-tuning, and high-volume inference”. In an enterprise that means one fleet, many teams, and different quotas, policies, and trust boundaries layered on top. The hard question is no longer how to train a model. It is how to give every team safe, isolated access to the same expensive hardware without anyone stepping on anyone else.

Two years ago every platform team was building a developer platform. Kubernetes already had mature primitives for containers, RBAC, autoscaling, and policy. What it did not have was a clean answer for accelerators, or for keeping tenants apart on the same nodes. That is the gap an AI factory has to close, and the cloud native ecosystem now supplies most of the parts to close it.

The bottleneck is utilization, not model serving

Accelerators are the dominant capital expense in the building, and the metric that decides whether that spend pays off is utilization, not a peak tokens-per-second number from a single run. The market grades GPU clouds the same way. SemiAnalysis’s ClusterMAX scores providers on security, networking, storage, reliability, and support rather than raw throughput, and its security criteria reward hard per-tenant isolation, down to per-tenant Kubernetes clusters and DPU-based isolation, while flagging weak boundaries like putting many tenants on one cluster. The wrapper around the GPUs is what gets judged.

Two things keep utilization low. First, the resource model: in the traditional device-plugin model a pod asks for nvidia.com/gpu: 1 and pins a whole accelerator even at ten percent use. Dynamic Resource Allocation (DRA), GA in Kubernetes 1.34, lets the scheduler treat accelerators as rich devices with attributes, memory, and topology, though it does not by itself carve a GPU into fractions; density comes from the device layer underneath. Second, the isolation model: to keep teams apart, platforms default to a dedicated cluster or a dedicated set of GPUs per team, which is the safe choice when trust is strict and wastes most of the hardware.

The same pattern recurs in the field: operators managing tenants with a bare metal provisioner and manual workarounds, or handing each customer a dedicated block of GPUs and turning away demand they cannot isolate cleanly. The fix is not a new model server. It is a stack that allocates accelerators so capacity is neither stranded nor unsafe, and isolates tenants so packing them together holds up.

The stack, layer by layer

An AI factory is an assembly problem. Most layers are Kubernetes native or CNCF projects, with a few OSS tools such as NVIDIA’s MIG, vCluster and Dynamo. The diagram below shows the shape, and the table lists the job each layer does.

Layer	Job	Building blocks
Hardware lifecycle	Provision and validate bare metal	Metal3 / Ironic, Tinkerbell, Redfish, NetBox
Cluster lifecycle	Create and version clusters, GitOps	Cluster API, Argo CD or Flux, NVIDIA AICR
Node inventory	Label GPUs, NICs, MIG, topology	Node Feature Discovery, GPU & Network Operator (NVIDIA, AMD)
Tenant isolation	Keep teams apart on the same hardware	Tenant clusters (vCluster), sandboxed runtimes
GPU allocation	Allocate and schedule accelerators	DRA, MIG, HAMi, time-slicing; KAI Scheduler (Topology Aware Scheduling), Volcano, Kueue
Inference and serving	Run models behind an API	vLLM, KServe, llm-d
Batch and HPC	Run SLURM workloads	Slinky (SLURM on Kubernetes)
VMs	Run virtual machines for tenants	KubeVirt
Gateway and autoscaling	Route and scale endpoints	Gateway API, Envoy, LiteLLM, KEDA, HPA
Networking	Move data between GPUs, isolate tenants	Cilium, Multus, SR-IOV, RDMA based networking solution
Storage and data	Persist datasets, checkpoints, models	CSI, Rook / Ceph, parallel filesystem CSI, object storage
Observability	See utilization, logs, traces	Prometheus, OpenTelemetry, DCGM exporter
Identity and policy	Authn, authz, quotas, guardrails	Keycloak (OIDC), ResourceQuota / Kueue, Kyverno or OPA
Secrets and security	Secrets, runtime, supply chain	OpenBao + External Secrets, Falco, Trivy
Reliability and remediation	Detect and recover from node failures	DCGM health checks, Node Problem Detector, drain / cordon
Self-service and billing	Provision and charge tenants	API / OpenTofu / GitOps, OpenCost, DCGM GPU-seconds

From bare metal to validated capacity

Everything starts at the rack.

Take a typical modern AI supercomputing platform as an example. Before any GPU can run a workload, something has to turn raw servers into a usable pool. That is the provisioning layer, often a proprietary hardware manager that ships with the system.

It works in steps. First it discovers each node, taking inventory: which GPUs and how many, whether memory is healthy (ECC state, meaning error-correction is on and not logging faults), and the identities of the network cards (InfiniBand GUIDs and NIC MACs, the permanent hardware IDs used to wire up and boot the node). Next it network boots the node and installs an OS image with the GPU driver and the CUDA and NCCL libraries baked in, so it can compute the moment it comes up. It then applies BIOS settings that match the node’s goal: baseline, performance, or confidential-compute.

Before a node joins the pool, it is tested. This burn-in runs the node under load to catch early failures, and an NCCL test confirms the GPUs actually talk to each other at full bandwidth. The result is written to a source of truth like NetBox, which also tracks IP address assignments (IPAM). Retiring a node runs the flow in reverse: wipe the disks, reset the remote-management login (eg the BMC), and return the clean node to the pool.

That proprietary manager is the vendor’s all-in-one take on this layer, tightly coupled to its own systems. The other path is to assemble the same loop from open building blocks: Metal3 driving Ironic, or a solution like vMetal, giving you the same discover, image, validate, and reclaim cycle on your own terms instead of adopting the vendor’s stack wholesale. That is the build-versus-assemble choice, and it recurs at every layer above. I come back to it at the end.

GPU allocation: the layer that makes the economics work

Figure 1. Whole-GPU allocation versus a partitioned GPU.

DRA gives a richer device-claim model, but fractional density comes from the device implementation. HAMi, a CNCF Incubating project, enforces per-pod memory and compute limits in software so several pods run on one card with guardrails between them, and it spans multiple accelerator vendors

Operators heading toward confidential computing do not place untrusted tenants on the same physical GPU; they give each tenant a whole GPU and reserve partitioning for workloads inside a single trust domain. MIG does isolate memory and faults in hardware, but its use as a boundary between hostile tenants is contested, so the conservative default is whole-GPU per tenant. The layer has two jobs: whole-GPU allocation for tenant isolation, and partitioning for density within a tenant. Scheduling is separate: KAI Scheduler and Volcano handle gang and topology-aware placement, and Kueue handles queueing, admission, and quota.

The workload layers: serving, Slurm, and VMs

Above allocation sit the things teams actually run. For inference, vLLM is a common engine and KServe, a CNCF incubating project, wraps it with autoscaling and standard endpoints, while NVIDIA Dynamo and llm-d push disaggregated inference for larger deployments. In front, Gateway API handles routing and LiteLLM adds an OpenAI-compatible gateway so dozens of specialized models speak one API.

Training customers usually live in Slurm, and the pattern has converged on running it on Kubernetes through SchedMD’s Slinky, which represents the Slurm daemons as CRDs and integrates with the GPU Operator and DRA for topology-aware scheduling, with pyxis and enroot, GPUDirect RDMA at full NCCL bandwidth, and prolog and epilog health checks. And some tenants want plain virtual machines rather than pods; KubeVirt runs VMs as Kubernetes workloads, so one platform hands out both containers and VMs from the same pooled fleet under the same RBAC and quotas.

Networking, storage, and observability

Training and disaggregated inference are bandwidth-bound, so the network is part of the design. Cilium handles the primary CNI and network policy; for the fast path, Multus and SR-IOV expose the NIC directly and RDMA over RoCEv2 or InfiniBand carries inter-node GPU traffic, with the isolation layer kept off that data path.

A real cloud also gives tenants the cloud-edge services they expect: elastic IPs, NAT, and L4 load balancing from a gateway in front of the fabric. Storage needs per-tenant persistence, usually CSI with Rook and Ceph or a parallel filesystem, governed by per-tenant StorageClasses and quotas.

For observability, OpenTelemetry is the neutral collection layer that keeps backends swappable, with Prometheus for metrics and VictoriaLogs for logs; the DCGM exporter publishes GPU telemetry that becomes per-tenant only with labels and a cost pipeline, and OpenCost turns GPU-seconds into chargeback.

Reliability and security

At fleet scale GPUs fail constantly: ECC errors, cards that fall off the bus, NVLink and thermal faults. The operator’s job is to catch these before a tenant does, which makes health a first-class layer rather than a dashboard afterthought.

Active and passive checks on DCGM watch for degradation, Node Problem Detector turns hardware signals into node conditions, and a remediation loop cordons and drains a suspect node before new work lands on it. This is one of the categories the rating systems weigh most, because reliability, not peak throughput, is what a customer feels first. Identity and policy round it out: Keycloak over OIDC, OpenBao with the External Secrets Operator, Kyverno or OPA for guardrails, and Falco and Trivy for runtime and supply chain, with audit logs exported to the observability stack and traffic encrypted in transit.

Isolating tenants

Every layer above assumes one thing: that you can safely run more than one team on the same hardware. That is the tenant-isolation problem, and it has two halves worth keeping separate.

The first is the control plane. The tenant-cluster pattern gives each team a virtual control plane: a full Kubernetes API server with its own CRDs, admission webhooks, versions, and RBAC, running as a workload on a single underlying cluster, with no view into another tenant. Several CNCF and open source projects implement this pattern, like vCluster. Because each tenant cluster is conformant Kubernetes, plain kubectl, Helm, and Argo CD with no proprietary extensions, the model gives tenants a clean exit path rather than lock-in.

Figure 2. Tenant clusters on one underlying cluster, drawing from a pooled GPU fleet.

In practice operators run two tiers. High-trust or enterprise tenants get a dedicated cluster, sometimes dedicated hardware, where the boundary is physical; smaller or cost-sensitive tenants get a tenant cluster on pooled capacity. The same control plane drives both. Reliability follows from the same design: because a tenant control plane runs as pods, Kubernetes reschedules it on failure, and the open question is blast radius, so operators cap how many tenants ride one underlying cluster.

The second half is the data plane, which a tenant cluster does not solve on its own. You still need network isolation, storage isolation, quotas, Pod Security, and a runtime boundary. Network isolation usually comes from the fabric rather than from Kubernetes: a control plane carves per-tenant VPCs with VXLAN and EVPN on the Ethernet side and partition keys on InfiniBand. Increasingly that enforcement is pushed into hardware, where DPUs (Data Processing Units) such as NVIDIA BlueField or AMD Pensando move isolation and encryption off the host CPU, which is also how operators reach a confidential computing posture.

For the runtime boundary on shared nodes, the options range from dedicated nodes to sandboxed runtimes such as vNode. The bar for a real cloud is hardware-enforced isolation, not namespaces and good intentions.

What makes it a cloud, not just infrastructure

The line between a pile of GPUs and a cloud is that a customer can provision it themselves and get a bill that makes sense. Both are cloud native problems. Self-service means API-first with no UI-only paths: a tenant creates and deletes clusters through an API, a Terraform provider, or GitOps, with resources expressed as declarative CRDs reconciled by Flux or Argo CD, and access scoped by RBAC through OIDC.

The bill comes from the metering layer: DCGM-driven GPU-seconds and OpenCost allocation, exported per tenant. None of this is glamorous, and it is usually the widest gap between a lab and a product. It is also, more than raw performance, what customers experience day to day.

From demo to production

Put the stack together and the demo is simple: two teams, two tenant clusters, two model endpoints, one physical GPU partitioned by MIG or software limits, each with its own RBAC, network policy, metrics, and cost line, neither aware of the other. This has been shown live on stage at KubeCon + CloudNativeCon with a single modern GPU serving two models at once.

Two things turn it into production. The first is conformance: tooling like NVIDIA’s AI Cluster Runtime validates cluster configurations against the hardware you actually have and emits reproducible Helm or GitOps artifacts, and the Kubernetes AI Conformance program, introduced in the 1.35 release, pushes the same idea at the platform level.

The second is scale: the design has to hold at hundreds of GPU nodes and several data centers, not the handful you prove it on, which is the real reason the foundation is GitOps, declarative tenants, and a single source of truth. There is a strategic choice here too, because the hardware vendor is moving into this layer with an integrated suite, NVIDIA’s DSX OS, so an operator decides layer by layer whether to adopt it, assemble the equivalent from cloud native projects, or compose the two.

The Takeaway

An AI factory is not another AI platform or model serving product. It is an operating model for running GPU infrastructure at scale on Kubernetes. Just as Kubernetes became the operating system for cloud native applications, it is becoming the foundation for AI infrastructure, making GPUs schedulable resources, providing isolated environments for tenants, and enabling on demand compute. The challenge is not deploying technologies like MIG, DRA, HAMi, or vLLM, but combining them into a platform that balances utilization, isolation, and cost while allowing multiple teams to safely share expensive GPU infrastructure without compromising performance or security.

Software is only half of it. The hardware layer is just as hard, often harder. Topology decides performance: which GPUs share an NVLink or NVSwitch domain, how each node attaches to a rail-optimized InfiniBand or RoCE fabric, whether the GPU, NIC, and CPU sit on the same NUMA node, and whether GPUDirect RDMA has a clean path. Schedule work without accounting for any of it and collective operations stall on the slowest hop, no matter how healthy the platform looks on paper. The stack has to be topology-aware, not just resource-aware.

The hard part is not naming the tools. It is making density, isolation, and chargeback work together, with hardware-enforced boundaries where the trust model demands them, without hiding the GPU data path behind an abstraction.

Related articles