GPU clusters: how to allocate capacity without priority inflation

The answer is to turn capacity into a measurable budget and make the queue compensate for accumulated usage instead of trusting labels such as high or urgent. On October 9, 2026, Ai2 explained that it replaced its priority-based scheduler with three components: hierarchically defined GPU-time budgets, fair-share calculated over a rolling window, and a contract that protects a minimum execution period before allowing preemption. Leadership decides in advance what share of capacity funds each program; the scheduler then favors groups that have received less than their allocation and treats cycles with no active demand as unallocated capacity. A team can temporarily yield GPUs without losing its future entitlement, while a workload that holds resources without progress spends the launching team's budget. The key is not one formula: it is the separation of strategy, arbitration, and execution so every decision leaves auditable evidence.

The news is three Lima calendar days old when this article is published. Ai2 reports results from its own environment—thousands of GPUs, roughly 150 researchers, and demand two to three times capacity; they are not a universal benchmark. The transferable contribution is the mechanism and how it is evaluated. A startup with eight GPUs and a platform with hundreds need different scales, but both can distinguish promised capacity, consumed capacity, preemptible work, and useful progress.

A free priority tends to inflate. If marking a job urgent has no cost, every team learns that the only defense against the queue is to use the highest label. The scheduler loses signal, and negotiation returns to chat, tickets, or the most influential person. A budget changes the incentive: every protected execution consumes time from a visible allocation. Asking for more stops being a configuration trick and becomes an explicit conversation about impact, deadline, and opportunity cost.

Fair-share translates that administrative decision into queue order. In the system Ai2 describes, every leaf in the tree—program, project, or researcher—has a share, and the scheduler compares actual usage with allocation over a seven-day window. Under-served groups rise; over-served groups yield. Slurm Fair Tree applies a related idea by ranking associations according to shares and effective usage at each level. The Dominant Resource Fairness paper adds another warning: when CPU, memory, network, and GPU are different bottlenecks, fairness must consider the dominant resource rather than counting jobs as if they all cost the same.

The execution contract solves the time problem. A job declares the minimum period it needs to make useful progress and whether it can resume. It is protected during that window; afterward it may be preempted and requeued. Without reliable checkpoints, preemption turns fairness into wasted work. Without a limit, one long training run freezes allocation. The contract must therefore connect protected time, recoverable state, and an observable definition of progress.

Do not copy another cluster's parameters. First design a contract your organization can explain and verify. These five decisions form one joint gate: if one is missing, the scheduler can optimize occupancy while making actual delivery worse.

Imagine a SaaS company with eight GPUs shared by three initiatives. Product needs to evaluate a classification model weekly; engineering trains a retrieval model for several hours; research explores an uncertain architecture. The old scheme offers high, medium, and low priorities. After two weeks, all three teams use high: evaluation fears missing its window, engineering protects an expensive run, and research needs iteration. The queue no longer expresses strategy.

A budgeted design assigns 40% of time to product delivery, 35% to platform, and 25% to exploration, reviewed monthly. Debug jobs request one GPU for fifteen minutes and are resumable by definition. Evaluations reserve short windows tied to a release. Training runs declare checkpoints every thirty minutes and a maximum protected period of two hours. If product does not use its share on Tuesday, research consumes the cycles as unallocated time; when product returns on Wednesday, its fair-share debt moves it up without evicting a job before its protected window.

This example is hypothetical and does not represent a Wasyra customer or result. Its purpose is to show that the initial percentage is not the hard part. Implementation must verify whether checkpoints restore optimizer, data, seeds, and metrics; whether a deadline-bound evaluation deserves a separate class; and whether CPU, storage, or network will prevent GPU use even when the scheduler releases capacity.

Start with replay and simulation. Reproduce at least four scenarios: normal demand, a burst of small jobs, a large training run that requires many nodes, and maintenance that forces hosts to drain. Compare the current and proposed policies using the same arrivals. Measure p50 and p90 wait time by class, time to first progress, work lost on preemption, share delivery when demand exists, starvation, and fragmented capacity. A global average can hide that large jobs never find a contiguous block.

In production, separate availability, occupancy, impact, and utilization. In a thirty-day test, Ai2 reported delivering 98% of owed time, maintaining 98% occupancy before and after the change, and serving 18% of delivered GPU time as unallocated capacity. It also observed that p90 debug-job wait fell from two hours to thirty seconds and that repairs requiring human intervention dropped by 74%. These are provider measurements from its cluster and workload mix; they show which dimensions to observe, not a result others can promise.

Define gates before rollout: no program with sustained demand may remain below its share beyond the agreed window; preemption may not lose more work than the approved limit; deadline-bound jobs must meet their SLO; and every exception must expire. Publish a dashboard explaining share, usage, debt, and the next execution opportunity. Explainability reduces folklore and lets teams debate the right budget instead of blaming the queue.

Fair-share enforces a relationship between allocation and usage; it does not know whether an experiment is scientifically sound, whether a customer is waiting, or whether a project should continue. Those decisions remain human and belong in budget review. A poorly designed tree can institutionalize old priorities with a mathematical appearance. Require an owner, review date, impact hypothesis, and a path for reallocating capacity.

Preemption also redistributes cost. Frequent checkpoints consume storage and bandwidth; infrequent checkpoints increase lost work. Interactive sessions preserve volatile state that may not be restorable. Ai2 acknowledges this limitation and plans to move some of those sessions to CPU infrastructure with restoration. Classify workloads by actual recovery capability before treating them as interchangeable.

Finally, watch fragmentation, affinity, and heterogeneity. Eight free GPUs across four hosts are not equivalent to eight contiguous GPUs for distributed training; a B300 and an H100 are not identical units either. If the budget uses only GPU-hours, add rules for topology, memory, energy, region, and cost. The allocation model must represent the dominant bottleneck that actually constrains the work.

A shared cluster needs two planes. The portfolio plane decides which outcomes deserve capacity and how much risk to accept; the execution plane translates that decision into queue order, time protection, preemption, and recovery. Combining both in one priority label invites inflation and constant negotiation. Separating them makes debt measurable, decisions explainable, and budgets adjustable without rewriting the scheduler.

Before buying more GPUs, replay real demand and answer the five decisions: unit, hierarchy, window, contract, and exception. If the queue cannot explain who received less than promised, how much progress interruption would lose, and what capacity remained idle because of fragmentation, you still lack evidence to distinguish physical scarcity from a defective policy. Wasyra can help design and implement custom AI systems, but the first deliverable should be this verifiable operating contract, not a promise of perfect utilization.