MIG, time-slicing, HAMi or DRA: which GPU workloads go where
Kubernetes hands out GPUs the way it hands out nothing else: whole. A pod that asks for nvidia.com/gpu: 1 gets an entire H100, 80 GB and all, even if it runs a notebook that wakes up twice a day. Cast AI's 2026 optimization report measured average GPU utilization of 5% across roughly 23,000 Kubernetes clusters. At $6.88 per H100-hour on AWS on-demand, one idle card is about $5,000 a month.
The fix is sharing. There are four ways to do it on Kubernetes today, plus NVIDIA's scheduler on top, and they are not interchangeable. Picking the wrong one either wastes the savings or breaks a production service. Here is how I decide.
The options
MIG (Multi-Instance GPU). The hardware splits one card into isolated instances, each with its own memory, compute slices and memory bandwidth. On an H100 80 GB the profiles are fixed: 1g.10gb (up to 7 per card), 1g.20gb (4), 2g.20gb (3), 3g.40gb (2), 4g.40gb (1) and 7g.80gb (1). A crash or a memory leak in one instance cannot touch another. Two costs: changing the layout needs a GPU reset, which the GPU Operator's MIG Manager does by draining and usually rebooting the node; and NCCL and GPU-to-GPU P2P do not work across MIG instances, so multi-GPU training needs whole cards. Available on A100, H100, H200 and Blackwell; not on A10G, L4 or L40S.
Time-slicing. The device plugin advertises each card several times (replicas: 4 turns 8 GPUs into 32 schedulable units) and the GPU switches between processes. It works on any NVIDIA GPU and needs no reboot. NVIDIA's documentation is explicit that there is no memory or fault isolation between replicas: one process can take all the memory, and an out-of-memory error in one container can affect the others. Each pod still believes it has the whole card.
HAMi. A CNCF project (incubating since July 2026) that adds a scheduler extender and a device plugin, plus a small library inside the container (HAMi-core) that enforces a memory limit and a compute share per pod. Pods ask for nvidia.com/gpumem: 10240 and nvidia.com/gpucores: 30 instead of a whole card. Sizes are arbitrary rather than fixed profiles, and nothing reboots. Isolation is enforced in software, so it is stronger than time-slicing and weaker than MIG.
DRA (Dynamic Resource Allocation). The new Kubernetes API for devices, enabled by default since Kubernetes 1.34. Instead of counting nvidia.com/gpu, a pod references a ResourceClaim that selects devices by attributes with CEL, for example "a MIG instance with profile 1g.10gb" or "two MIG instances on the same physical GPU." NVIDIA's DRA driver now lives under Kubernetes SIGs and supports MIG, time-slicing and MPS. Creating MIG partitions on demand through DRA is still an alpha feature gate in late 2026. DRA is where this is heading; for most teams in 2026 it is a migration to plan, not the first step.
Run:ai / KAI Scheduler. NVIDIA open-sourced the full Run:ai platform on 13 October 2026; its scheduler core, KAI, was already Apache-2.0 and a CNCF sandbox project. It adds queues, fair-share quotas per team and GPU fractioning on top of Kubernetes. The trade-off is that the control plane is NVIDIA-specific, so it is the natural choice for an all-NVIDIA shop and the wrong one for mixed hardware.
The decision, workload by workload
| Workload | Typical profile | Use |
|---|---|---|
| Notebooks and dev pods | Hours idle, short bursts, 3–15 GB | HAMi slices or time-slicing, plus an idle culler |
| Small inference services (embeddings, classifiers, OCR) | Steady low load, 5–20 GB, latency matters | MIG 1g.10gb / 2g.20gb on H100/A100; HAMi on A10G/L4 |
| Multi-tenant inference with SLOs | Needs predictable latency per tenant | MIG |
| Large LLM serving | 40–80 GB per GPU, high utilization | Whole GPUs, tensor parallel; do not share |
| Distributed training | NCCL across GPUs | Whole GPUs, never MIG |
| Batch jobs and sweeps | Bursty, finite | Whole GPUs with a queue (Kueue, Volcano) and a TTL so they actually end |
| CI and smoke tests | Seconds of GPU | Time-slicing |
Two rules hold across the table:
- Size by peak memory, not by average utilization. A service at 4% utilization that peaks at 60 GB still needs most of a card. Measure
DCGM_FI_DEV_FB_USEDover a week, add 20% headroom, and pick the slice from that. - Mixing is normal. One node pool of MIG-partitioned H100s for small inference, one pool of whole H100s for training and big models, and a HAMi or time-sliced pool of cheaper cards for notebooks is a common end state.
The part that is not technical
Sharing saves money only if people accept it. What makes it stick:
- Quotas at admission. Reject a pod that asks for a whole H100 in a namespace whose quota says 20 GB, with an error message that tells the user what to ask for instead. Kyverno or a ResourceQuota on
nvidia.com/gpumemdoes this. - Idle culling with a resume path. Notebooks that stop after an hour of inactivity and come back in a minute get accepted. Notebooks that lose work do not.
- A dashboard per team. When a team can see its own GPU-hours and utilization, requests shrink without anyone policing them.
On an ML platform I run, moving from one whole H100 per user to HAMi memory slices with admission-time quota checks took an 8-GPU node from 8 possible concurrent workloads to a ceiling of 72, and let more teams onto the same hardware.
Find out where you stand
Before choosing any of this, measure. I published a read-only scanner that reads your nodes, pods and dcgm-exporter metrics and produces a report in your browser: which GPUs are idle, which workloads fit a slice, and how many nodes you could turn off. There is no upload step.
Sources: Cast AI, 2026 State of Kubernetes Optimization Report; AWS EKS user guide, "Use multi-instance GPUs (MIG) with NVIDIA GPUs"; NVIDIA GPU Operator documentation, "Time-Slicing GPUs in Kubernetes"; CNCF blog, "HAMi becomes a CNCF incubating project" (July 2026); DRA Driver for NVIDIA GPUs documentation; AWS EC2 on-demand pricing (p5.48xlarge, October 2026).
Measure before you choose
The free scanner reads your nodes, pods and dcgm-exporter metrics and shows which GPUs are idle, which workloads fit a slice and how many nodes you could turn off. Read-only, and the report runs in your browser.