- Model Training: Meta’s Prometheus AI supercomputer has completed its first phase and trained Muse AI models, with hundreds of thousands of GPUs connected.
- Fault Recovery: Each 72-GPU group keeps eight processors spare; isolated failures can be handled without pausing the entire training job.
- Parallel Training: Meta says Prometheus can run multiple training jobs, each using tens of thousands of GPUs, at once.
- Expansion: Meta is expanding toward roughly one million GPUs, with NVIDIA’s Vera Rubin hardware still upcoming at Prometheus.
Meta’s Prometheus AI supercomputer has completed its first phase and trained the company’s Muse models. Its October 19, 2026 engineering disclosure explains how spare processors and recovery software help Meta’s model developers keep large training jobs moving when hardware fails.
The system connects hundreds of thousands of graphics processing units, or GPUs, across buildings in New Albany, Ohio. Meta is expanding the fleet toward roughly one million GPUs, with no completion date specified. The site served conventional web traffic two years before the disclosure; it now hosts the tightly coordinated computation needed to train AI models.
That coordination makes failures costly. Many processors work on the same training job, so a fault on one machine can force others to wait or repeat work. Prometheus’s networking and fault-tolerance design addresses both sides of the problem: keeping data moving between processors and limiting how much training stops when equipment fails.
Spare GPUs Limit the Cost of a Failure
Prometheus uses NVIDIA’s NVL72 architecture, which connects 72 GPUs through fast NVLink connections so they can cooperate closely. Meta’s previous H100 generation grouped eight GPUs this way. Expanding the group to 72 increases the amount of tightly connected hardware that a fault can affect.
Meta allocates 64 GPUs in each group to active training and keeps the other eight as local spares. A replacement can take over without waiting for a physical repair. The company also holds roughly 2% additional racks in reserve for rarer failures that take an entire rack out of service.
Training is divided among data-parallel replicas, copies of the model participating in the same job. If a fault is confined to one replica, Meta’s parallelism-aware fault tolerance takes that copy offline. The remaining replicas repeat the affected step and continue training, while the failed portion restarts on spare hardware and rejoins them.
A wider failure requires the whole job to restart from a checkpoint, a saved copy of the model’s training state. Frequent saves reduce the amount of work that has to be repeated. Meta saves that state asynchronously and stores changes between checkpoints to limit the time consumed by the saving process itself.
Fixing Weak Links and Memory Faults
Meta says interruptions peaked at nearly 1,000 events per 8,000 GPUs per day during Prometheus’s initial phase in mid-2025. Its definition covers faults that force a job to pause and begin recovery. The company reports a 99% reduction from that peak after a year of engineering, enabling multiple jobs using tens of thousands of GPUs to run in parallel.
Early GB200 GPU deployments were particularly vulnerable to weak NVLink signals. Across more than 600,000 connections, even marginal degradation could trigger recovery cycles. Meta added continuous monitoring, automated fault flags and diagnostic screening to identify weak connections before production use. Repaired systems are validated before returning to the fleet.
Memory faults became another major source of interruptions as Meta expanded its higher-memory GB300 systems. Meta and NVIDIA are deploying changes to memory firmware, error correction and predictive detection used to retire deteriorating devices before they disrupt training. Their responsibilities are complementary: Meta handles scheduling and infrastructure, while NVIDIA works on firmware, signal diagnostics and memory handling.
Faster Links Across Buildings
Prometheus extends an approach Meta used in its earlier training systems. In September 2025, the company described a 129,000-H100 cluster assembled across five data centers. The newer system combines a larger Blackwell-family GPU fleet with fast local connections and a network designed for training across greater distances.
Ordinary service traffic and direct GPU communication use separate networks. Within a building, the training network provides low-latency, high-bandwidth connections for the most communication-sensitive work inside a model replica. Traffic between replicas can tolerate more delay and crosses the network linking buildings, which spans distances from 2 kilometers to more than 100 kilometers. Meta worked with its model-training teams to adapt communication patterns to those distances so GPUs spend less time waiting for data.
The same concern shaped its choice of hardware. Meta’s GB200 configuration, called Catalina, splits a 72-GPU group across two racks. Two 400G network adapters give each GPU 800G of external bandwidth, or 800 gigabits per second. Meta chose that arrangement because standard external bandwidth would otherwise have remained at 400G while computing capacity and communication inside the group increased. The company expected that imbalance to reduce performance.
Its GB300 platform, Clemente, supplies 800G per GPU natively and fits all 72 GPUs into one rack. Memory per GPU rises from 192 GB on Catalina to 288 GB on Clemente. The changes give Meta more memory and a denser physical deployment while retaining the fast links needed to connect the groups into a larger training system.
Keeping Power and Cooling in Step
The processors’ coordinated activity also puts pressure on electrical infrastructure. When thousands of GPUs switch between computation and communication together, Meta reports load fluctuations of 30–40%. Starting or pausing a major job can change demand by tens of megawatts in seconds.
Meta’s Dynamo Dimmer power-management system gradually adjusts GPU power limits as the cluster approaches its electrical ceiling. Rack batteries, capacitors and facility-level battery storage help absorb rapid changes. The control system is designed to keep training running as power demand approaches the available supply.
Meta’s original Prometheus plan combined its own campus with leased capacity. Cooling consequently varies across the fleet. Older buildings and rapid-deployment structures use modular air-assisted liquid cooling; a purpose-built AI building uses facility liquid cooling. Meta’s newest AI-optimized buildings circulate coolant directly to the chips in closed loops, allowing more GPUs to fit into each rack.
Rubin Is Next at Prometheus
Meta plans to expand Prometheus with additional GB300 systems and Vera Rubin NVL72 hardware through its upcoming San Miguel platform. Rubin is intended to join the existing network.
The same generation is already running customer workloads elsewhere. AI cloud provider CoreWeave has put Vera Rubin NVL72 into production with Cognition, the developer of the Devin AI coding system. Cognition uses CoreWeave to train models and run inference, the process of using a trained model to produce answers.
