Why 65.9% of AI Teams Struggle with GPU Capacity Planning
Ask an AI engineering lead what gives them the most trouble these days and the model itself rarely comes up. The real problem sits one layer below, in the infrastructure, and it comes down to a single question: how much GPU capacity should we actually secure? A March 2026 Akamai survey of 200 AI practitioners put a number on that feeling. 65.9% of AI-native teams named GPU capacity planning the hardest challenge they face when scaling, ranking it above every other operational issue in the survey, including latency and cost. [1] This is not a niche complaint. 75.5% of respondents were engineers, architects, or DevOps practitioners, and 76% were decision-makers who set budgets and direction, so practitioners and leaders named the same problem. The same pattern shows up elsewhere. In a 2026 VentureBeat survey of 107 companies, 83% reported GPU utilization at or below 50%, and only 44% said they were properly tracking AI compute costs. [7] The problem these teams ran into is not a shortage of GPUs. It is the absence of a reliable way to know how many they need.
Why Capacity Planning Breaks Down
Traditional infrastructure capacity planning rests on one assumption: that there is a predictable relationship between user count and load. LLM workloads break that assumption from the start. When a typical web service grows from 10,000 users to 1 million, the pressure lands mostly on the database and the API layer, and that can be modeled in advance to a reasonable degree. When an AI service scales up, GPU memory, token throughput, batching efficiency, and cold starts are all affected at once, and none of them move in step with user count. Token demand is hard to forecast because it does not rise gradually. It spikes. The table below shows how that pattern repeats across very different workloads.
Workload type | Token growth pattern | What happened |
Marketing campaign | Order-of-magnitude jump overnight | A single campaign drove traffic to the landing page and produced an unpredictable spike |
Financial chatbot | 12x surge within 48 hours | Cloud autoscalers could not fully absorb the surge, and the real bottleneck was KV cache exhaustion. Tuning batching parameters and shortening context length improved P95 latency by 60% without adding a single GPU [2] |
Agentic workloads | Tool calls, retries, and growing context compound token consumption |
Reservation-based static capacity planning was never designed for workloads that move like this. What makes these cases worth sitting with is that the team had already provisioned extra nodes, and that still was not the fix. Extra capacity applied at the wrong point produces the same latency wall as having too little.
Too Much Is a Problem, and So Is Too Little
Overestimate demand and you keep paying for GPUs nobody uses. Teams that size for the worst-case traffic surge end up carrying idle hardware for months when demand grows more slowly than expected. One Fortune 500 company secured 300% more capacity than it needed, and the excess sat unused for two years. Underestimate demand and the bill can be far worse, because scrambling to secure capacity after the fact costs much more than carrying a reasonable buffer from the start. In one well-known case, a company underestimated demand by 400% and spent $800 million on emergency procurement alone to close the gap. [5] The two cases fail in opposite directions, but the root cause is identical: both tried to predict and serve a constantly shifting workload with a single fixed number. This is why “just buy more GPUs” is not the answer. Even among organizations that have already made substantial infrastructure investments, half still miss their latency targets during peak traffic. [1] Wasted spend on idle capacity and degraded performance at peak are not two separate problems. They are the same structural mismatch between shifting demand and fixed infrastructure, showing up once as cost and once as performance.
What Overestimating Actually Costs
“Idle GPU” stays abstract until you put a number on it, and the real figures are worse than most teams expect. Cast AI's 2026 State of Kubernetes Optimization Report analyzed telemetry from roughly 23,000 production clusters running on AWS, Azure, and GCP. Average GPU utilization across those clusters was 5%. Put simply, companies are provisioning about 20 times the GPU capacity they actually use. [8] [9]
Utilization varied by cloud provider, but it was low everywhere. Average GPU utilization by managed Kubernetes service was 2% on AKS, 5% on EKS, and 6% on GKE. [10] This is less a problem with any one cloud vendor than with how GPU resources are allocated and operated in general.
Translated into cost, it looks like this. At AWS p5 on-demand pricing, keeping one unused H100 allocated costs roughly $8,850 per month. [10] If a team of 20 engineers each spins up an H100 for development and never shuts it down, about $177,000 a month can disappear without a single warning. Idle GPU cost is risky precisely because it is easy to miss. Nothing is broken and nothing throws an error, so as long as the resource stays allocated the charges keep accruing even with no workload running. The contrast with CPUs makes it clearer. An idle CPU core typically costs cents per hour, while an idle GPU runs to several dollars per hour. [8] Leaving some CPU headroom may barely register in the overall bill, but over-provisioning GPUs the same way turns into a line item the CFO will ask about.
One structural reason this persists is that spot instance usage remains extremely low. Through most of 2025, fewer than 2% of GPU workloads ran on spot instances, even though interruption-tolerant workloads could have cut costs by 60 to 91% at spot prices. [8] GPU supply constraints did make spot GPUs genuinely hard to obtain for stretches of that period. But the operating habit of treating reserved, always-on capacity as the “safe” choice also locked in low utilization. Securing GPUs ahead of time regardless of how demand moves is what produces an average utilization of 5%.
What Mature Teams Are Actually Changing
So is the answer simply to forecast demand more accurately? Mature teams are changing the question instead. Rather than trying to land on a single correct number for future demand, they build infrastructure that responds flexibly when actual demand turns out different from the plan. The point is not better forecast accuracy. It is a greater ability to absorb uncertainty. Three patterns show up consistently in how these teams operate.
1. Elastic capacity instead of reserved capacity
A fixed reservation is a bet on one specific usage forecast. When that forecast is wrong (and as the survey data above suggests, it usually is 🥲) you pay for it in idle cost or in downtime. Infrastructure that scales GPU allocation up and down automatically with real-time demand removes the premise that you have to get the number right in the first place.
2. Cost visibility at the token level, not the GPU-hour level
As we explored in an earlier post, managing AI cost has become an essential operational function. But looking only at billed GPU hours makes it hard to tell which model, request, or feature drove the spend. Cost has to be traceable down to the token before you can pinpoint what is actually growing. Teams with that visibility can find and address the cause at the moment a cost spike happens, which is a different speed of response from discovering the problem when the invoice arrives three weeks later.
3. Distributing infrastructure instead of centralizing it
As covered in an earlier post, AI infrastructure is moving away from concentrating all computation in one large data center. Inference workloads in particular run continuously across multiple regions, carry loads that are hard to predict, and are sensitive to data location and latency. What matters more now is the ability to choose compute that is available at the time and place you need it. A wrong demand forecast does not get fixed by holding more GPUs.
A Practical Sequence for Getting Capacity Planning Under Control
If the three patterns above show the direction, the next step is applying them in practice. What follows is the sequence that shows up consistently among infrastructure teams that manage this problem well. It is less a checklist you must follow than a baseline to work through before committing more budget and headcount.
Classify workloads before you size capacity. Training, fine-tuning, and inference have fundamentally different compute, memory, and networking requirements. Size all three against a single standard and you will inevitably end up with surplus for some workloads and shortfall for others. Separating workloads by profile is the starting point for any capacity plan. [11]
Measure a real usage baseline before committing to long-term reservations. Committing to reserved capacity without enough production data makes it easy to mistake early, unstable demand for normal usage. Traffic for agent and inference services in particular swings widely during the first few weeks after launch. Observe at least 30 days of real traffic, separate sustained load from temporary spikes, and then decide on reserved capacity. [12]
Serve baseline demand and burst demand separately. In production, the effective pattern is a small dedicated pool that keeps models warm and handles baseline traffic, paired with elastic capacity that expands only when demand surges. Try to serve both from one fixed allocation and GPUs sit idle most of the time while latency climbs at peak. You end up choosing between idle cost and degraded performance. [12]
Put cost attribution in place before you add people. You can only optimize correctly once you can see which team, model, and feature is generating GPU spend. In a recent enterprise infrastructure survey, 74% of teams said they were not satisfied with the scheduling and resource allocation tools they use today. Adding engineers to capacity planning without visibility only increases the number of people making judgment calls on incomplete information. [13]
Revisit workload classification and capacity decisions on a regular cadence. Workload profiles and traffic patterns keep changing. Capacity that was right six months ago is not guaranteed to be right now. Review workload classification and baseline usage at least quarterly, and feed the results back into your spot instance mix, autoscaling configuration, and reserved capacity. [8]
None of these five steps involves buying new GPUs or new management tooling. What they require first is an operating practice that measures actual usage continuously and revises decisions accordingly, rather than setting capacity once and moving on. Capacity planning has to be a repeated decision process, not a one-time provisioning task. The 65.9% figure above says most teams have not made that shift yet.
Closing
GPU capacity planning ranks as the biggest scaling challenge for AI-native teams not because those teams are bad at forecasting. It is because the thing being forecast, token demand, moves fast enough and irregularly enough that traditional methods cannot pin it down. The answer is therefore not a more sophisticated spreadsheet or more GPUs. What is needed is infrastructure that stays stable when the forecast is wrong: capacity that scales up and down automatically with real usage, cost visibility that connects spend to tokens rather than GPU hours, and an architecture that can absorb a sudden 10x increase in demand without an emergency response.
Air Cloud connects distributed GPU resources into a single infrastructure layer and is designed to scale to what your workload actually needs. If you need infrastructure that adapts to shifting demand instead of locking in fixed GPU capacity up front, run your workload on Air Cloud and compare cost and performance for yourself.
.
.
.
