The gigawatt-scale installation in Ohio served as the training ground for Meta's Muse models. In a notable move for the industry, Meta released detailed failure data, revealing how the facility reduced interruptions from a daily peak of 1,000 events to a 99% reduction.

Spanning multiple buildings across New Albany, Ohio, Meta's Prometheus supercluster currently operates hundreds of thousands of Nvidia GPUs with plans to expand to roughly 1 million units. The company shared these technical specifications in an engineering blog post published on Monday.

The initial phase of this gigawatt-scale operation has reached completion. Meta deployed the infrastructure to train its Muse family of models, which includes the Muse Wonder variants. The Ohio site underwent transformation from a traditional data centre handling web services just two years prior.

At full scale, Prometheus will comprise roughly 1 million distributed GPUs, all functioning in lockstep

Meta

Three generations of Nvidia racks

The Prometheus installation currently operates across two Nvidia platforms, each built on racks containing 72 GPUs functioning as unified accelerators. Catalina leverages the GB200 NVL72 configuration, while Clemente employs the more recent GB300 NVL72 design. A forthcoming third platform, San Miguel, will introduce Nvidia's Vera Rubin NVL72 racks to the mix.

The GPU infrastructure occupies conventional data centres, AI-focused facilities and modular deployment structures, all connected by extensive fibre networks spanning multiple kilometres. Within each physical location, up to 20,736 GPUs connect through local networking. A separate interconnection layer bridges the buildings across distances ranging from 2 kilometres to over 100 kilometres. Meta supplements its owned capacity with leased resources from external partners.

From 1,000 interruptions a day

Operating at this magnitude means that a single machine malfunction can derail an entire training operation. During mid-2025, in the facility's initial phase, the system experienced nearly 1,000 interruption events daily per 8,000 GPUs, according to Meta's disclosure. Twelve months of engineering effort has achieved a 99% reduction in this metric.

The most significant early challenge involved signal integrity across more than 600,000 NVLink connections linking the GPUs together. As Meta expanded the GB300 racks, memory errors emerged as an increasingly critical concern.

Throughout Muse Wonder's training phase, the cluster maintained productivity levels exceeding 85% of pre-training time and surpassing 90% during reinforcement learning tasks. The system averaged approximately 47,000 GPU hours between interruption events, a figure that has since climbed above 54,000 following firmware improvements. Meta targets approximately 100,000 GPU hours as its reliability objective.

Achieving reliability requires hardware redundancy. Each 72-GPU rack allocates 64 GPUs to active computation while reserving eight units as spares, and Meta maintains approximately 2% of all racks in standby reserve. A training job utilizing 32,768 GPUs therefore demands 37,656 total units, representing roughly 13% additional capacity.

Power swings of tens of megawatts

When tens of thousands of GPUs operate and exchange data synchronously, the cluster's electrical consumption fluctuates between 30% and 40%. Initiating or halting a large-scale job can create power swings of tens of megawatts within seconds. Meta implements gradual GPU power reduction through proprietary software termed Dynamo Dimmer, supported by battery systems positioned at the rack, facility and electrical grid levels.

Meta states it covers the complete energy expenses for the New Albany facility and offsets consumption through investments in new clean and renewable energy sources. In July, the company withdrew from the RE100 commitment as its gas infrastructure expansion for AI capabilities outpaced renewable energy development.

The company's subsequent supercluster project, Hyperion in Richland Parish, Louisiana, incorporates design principles derived from Prometheus experience, Meta indicated. Additionally, Meta has committed $1 billion toward supporting data centre communities.

Source: The Next Web