Meta Prometheus AI Supercluster: 1 Million GPUs
Meta is scaling Prometheus toward one million Nvidia GPUs. See how its architecture, reliability fixes and power demands shape large-scale AI training.
Meta's Prometheus Supercluster Is Built to Scale AI Training
A big AI training run can be derailed by a single machine failing. Scale that problem to tens of thousands of GPUs, and "keeping things running" stops being a background concern - it becomes the main engineering job. Meta says its Prometheus supercluster in New Albany, Ohio, already runs hundreds of thousands of Nvidia GPUs and is meant to grow to around one million.
That goal is impressive, but the interesting part isn't the GPU count. Prometheus brings together multiple generations of Nvidia systems, different deployment models across buildings, long-distance networking, reserved hardware for failures, and software meant to handle sharp electricity swings. Meta says the first phase was used to train its Muse family of models, including Muse Wonder.
So the real question isn't only how many GPUs Meta can install. It's how much useful training time the cluster can deliver, how reliably the machines can coordinate, and what it costs - in power and operations - to keep the whole setup working. This article looks at Prometheus's architecture, the reliability work behind it, and the trade-offs that come with scaling AI infrastructure.
What Meta's Prometheus supercluster is
Prometheus is Meta's large-scale AI computing cluster in New Albany, Ohio. Meta calls it a gigawatt-scale effort and says its completed first phase runs hundreds of thousands of Nvidia GPUs across multiple buildings. The company's stated end goal is roughly one million distributed GPUs that all work in a coordinated way during training.
That wording matters. Prometheus isn't a single massive computer in one room. It's a distributed system: compute is spread across buildings, and networking ties those sites together so large training jobs can draw from the cluster as a whole. "In lockstep" refers to how training workloads coordinate across many devices. It shouldn't be read as "every GPU is identical and nothing ever varies," because real systems always have differences and some level of variability in operation.
Meta says it used the first phase to train its Muse family of models, including Muse Wonder. That means Prometheus is already doing work - not just standing by for future capacity. The initial cluster has been used for model development, and later expansion is intended to increase the amount of compute available for subsequent training.
From web traffic to AI infrastructure
Meta says the New Albany site used to be a conventional data center running web traffic two years ago. Turning it into an AI supercluster reflects a shift in what large tech companies expect from their infrastructure. Online services and AI training put stress on systems in different ways.
Training jobs keep accelerators, memory, networking, and power delivery under sustained load. They also often depend on many machines finishing together. That changes the impact of faults: a problem that might affect a single service request in a typical web setup can disrupt an entire coordinated training run if the failed hardware is part of the dependency chain.
The scale-up is also an investment decision. Big clusters require upfront capital, power delivery, network capacity, and teams who can operate and maintain the systems. The value depends on whether the compute actually gets used to produce useful model and product results that justify the cost. A headline GPU number doesn't tell you that by itself.
What the million-GPU target tells us - and what it does not
Meta's target signals the size of its planned AI infrastructure investment. It still doesn't answer key operational questions: how many GPUs will be available at any given time, how workloads will be scheduled across the cluster, or how fast any specific model will train. Those outcomes depend on the full design, including software, workload scheduling, and reliability across the stack.
Meta's operational reporting makes a similar point: installed capacity isn't the same as useful capacity. The company reserves GPUs and racks to cover failures, and it reports productive-time rates during training. In practice, some hardware has to be held back as backup rather than running jobs continuously. It's a trade-off - fewer GPUs are assigned at once, but the system is better positioned to keep important training moving when something breaks.
How the hardware and network fit together
Meta says Prometheus uses two Nvidia platforms today: Catalina, built around the GB200 NVL72, and Clemente, built around the GB300 NVL72. Both use racks of 72 GPUs meant to operate like a single accelerator system. A third platform, San Miguel, is planned to add Nvidia Vera Rubin NVL72 racks.
Using multiple generations is pretty much unavoidable when you build over time. New hardware arrives in waves, and you need to bring capacity online without waiting for every future component to be ready. The trade-off is operational complexity. Different platforms can bring different performance traits, firmware behavior, failure modes, and maintenance needs - so the software and operations teams have to manage a mixed environment.
Racks are only one layer of the design
The GPUs sit in a mix of traditional data centers, AI-optimized buildings, and rapid-deployment structures. Meta says each building's network can connect up to 20,736 GPUs. A separate aggregation network links buildings over distances from roughly 2 kilometers to more than 100 kilometers. Meta also uses capacity leased from partners.
It's a layered setup. GPUs communicate locally inside a rack; the building network ties groups of racks together; and the aggregation network connects buildings. Each layer has different constraints. Local GPU communication needs high bandwidth and low latency. Cross-building links cover longer distances and add more routes and network behavior to manage. Across all of it, the goal is to keep data moving so the network doesn't erase the benefit of the accelerator capacity.
Leased capacity adds another layer of coordination. It can expand access to compute without requiring Meta to own and build every part of the infrastructure. But if partners or multiple sites are involved, scheduling and responsibility still have to line up with the training workloads - capacity, connectivity, and operational procedures all matter.
Why "distributed" matters for model training
Distributed training spreads work across many accelerators. That can make it possible to train at a scale a single machine - or a smaller cluster - can't reach. But adding GPUs doesn't automatically translate into a linear gain in useful training throughput. The system still has to keep accelerators fed with data, coordinate their operation, and absorb slowdowns or failures.
That's why the network is part of the compute system, not just a supporting detail. If communication becomes a bottleneck, adding accelerators can cost money without delivering the expected training-speed gains. And if a fault interrupts a coordinated job, the GPUs' value drops until the underlying issue is resolved or the job can recover.
In the end, cluster performance comes from the system as a whole - accelerators, memory, network behavior, software, power delivery, and operations. Meta's description of Prometheus gives a sense of that engineering challenge, even while leaving many specifics out, such as workload scheduling strategies and model-by-model performance.
Why reliability is a scaling problem
Meta says interruptions during Prometheus's first phase peaked in mid-2025 at nearly 1,000 events per day for every 8,000 GPUs. After a year of engineering work, the company says it cut that rate by 99%. The lesson here is straightforward: at this scale, you can't just assume hardware won't fail. You have to detect failures, bring their frequency down, and design the system so it can handle the failures that still happen.
Meta also points to signal quality across more than 600,000 NVLink connections as an early major issue. As it scaled up its GB300 racks, memory faults became a bigger concern. These aren't the same kind of failure, which is why reliability work can't be reduced to "fix one thing and be done." Improving link quality doesn't automatically prevent memory errors, firmware problems, or failures elsewhere in the stack.
Productive time is more useful than a raw GPU count
During Muse Wonder training, Meta says Prometheus kept more than 85% of its pre-training time productive and more than 90% of its reinforcement-learning time productive. It also reports an average of about 47,000 GPU hours between interruptions, rising to more than 54,000 after firmware fixes. Meta's target is about 100,000 GPU hours between interruptions.
Those metrics track different pieces of health. Productive-time rates reflect how much of the scheduled workload time was spent doing useful work. GPU hours between interruptions captures how often the system stalls. Neither metric is a complete measure of a model's final quality, but both are more directly tied to operations than installed capacity alone.
They also suggest reliability improves in steps, not in a single breakthrough. Meta says it has made substantial progress, but its target remains out of reach. For companies building AI infrastructure, that gap matters: a cluster can be large and valuable and still require ongoing work to keep availability high and interruptions low.
Redundancy has a direct capacity cost
Meta says each 72-GPU rack runs a 64-GPU slice and keeps eight GPUs as spares. The company also holds about 2% of racks in reserve. With that setup, a training run that uses 32,768 GPUs requires 37,656 GPUs total - about 13% more than the active count.
That reserve isn't automatically wasted capacity. It's there to keep the system resilient. In runs that rely on thousands of accelerators, even a small number of component failures can cause a disproportionately large disruption. Spares let operators work around failures without immediately stopping a run or redesigning the job.
The business trade-off is simple: redundancy increases the infrastructure you need to deliver a given amount of active compute. Remove the reserve and you may see more interruptions and lower productive time. The "right" amount depends on how expensive downtime is, how well workloads can recover, and what it costs to hold spare capacity. For clusters used in long, coordinated training runs, treating reliability as optional would be a gamble.
Power, cost and the wider business implications
A supercluster's compute capacity depends on more than GPUs and networking. It also depends on whether the facility can deliver power reliably, including during rapid changes in workload. Meta says Prometheus's power draw can swing by 30% to 40% when tens of thousands of GPUs compute and communicate in step. Starting or pausing a large job can shift demand by tens of megawatts in seconds.
That variability brings a different infrastructure problem. Power systems are often planned around expected demand patterns, but training workloads can change their draw quickly. Operators have to manage those shifts at multiple levels - rack, facility, and grid. If power control is slow or inflexible, the cluster can hit limits in the energy system even when the GPUs are available.
Managing rapid changes in demand
Meta says it uses software called Dynamo Dimmer to cap GPU power gradually. Batteries at the rack, facility, and grid level support that approach. The point is to avoid treating power like a fixed input: software plus storage help manage demand swings while keeping the system within operational limits.
This is also an example of software shaping hardware behavior. The goal isn't just to make each GPU draw less power all the time. It's to manage the cluster's overall demand so large jobs can start, pause, or change activity without creating an unmanageable step change for the surrounding infrastructure.
There are trade-offs. Power caps can reduce how much compute is available at a given moment, and batteries add cost and operational complexity. The key question is whether the controls improve reliability and usable capacity across the whole system. For clusters that see fast, large demand swings, power management has to be part of the design from the beginning.
Energy commitments and local impact
Meta says it pays the full cost of the energy it uses in New Albany and matches that use with new clean and renewable energy. The report also notes that Meta left RE100 in July, as its gas build-out for AI outpaced renewables. Those points sit side by side: a company can make energy-matching commitments while also relying on gas generation in the near term.
This tension shows up in the business case for AI infrastructure. Power availability, cost, and emissions are related, but they aren't interchangeable. Matching energy use with new renewables doesn't eliminate questions about local generation, timing, grid capacity, or how quickly new supply can actually come online. These details affect both the communities that host data centers and the companies building and operating them.
Meta says its next supercluster, Hyperion, in Richland Parish, Louisiana, is being designed using lessons from Prometheus. The company has also pledged $1 billion to data center communities. Those commitments are part of the broader infrastructure story, but the real impact will depend on execution and on how benefits and costs land locally.
For other companies, the takeaway isn't that every AI project needs a million GPUs. The takeaway is that infrastructure planning has to cover power delivery, resilience, networking, and community considerations alongside accelerator purchases. Buying hardware is the visible part. Getting the system to stay productive and supportable over time is the harder part.
The takeaway: capacity only matters when it stays productive
Prometheus illustrates what scaling AI infrastructure really means. Meta is combining multiple generations of Nvidia systems, connecting GPUs across buildings, keeping spare capacity to handle failures, and managing power swings that can reach tens of megawatts. Its move from frequent interruptions to a much lower rate is a meaningful improvement, but Meta's own numbers show reliability work doesn't end.
The practical metric isn't the biggest GPU count a company announces. It's how much useful training the infrastructure can actually deliver - at what cost, with what resilience, and under what energy constraints. If you're evaluating AI infrastructure investments, it's worth starting with those operational questions rather than treating accelerator count as a stand-in for capability.
For the original engineering details, read The Next Web's report on Meta's Prometheus supercluster.