Cloud service giants have amassed thousands of GPUs to meet the burgeoning demand for AI operations. Despite these significant investments, new insights from TechInsights (via The Register) suggest a stark under-utilization of these powerful processors. Findings from 2023 indicate that 878,000 accelerators, deployed to handle complex AI tasks, contributed to only seven million GPU-hours of work. This level of activity corresponds to an estimated $5.8 billion in revenue, a figure that falls short of projections, signaling a lack of full capacity utilization.
The Economics of GPU Utilization in the Cloud
Cloud providers, such as AWS with its UltraScale clusters featuring Nvidia H100 GPUs, have set up extensive infrastructures allowing customers to rent GPU instances. Analyst Owen Rogers points out that at full capacity, these clusters could bring in $6.5 billion annually for Amazon alone. This disparity between potential and actual revenue raises concerns about how effectively these accelerators are being used. While internal workloads for cloud providers may account for some of this discrepancy, profitability still hinges on external revenue generation.
Beneath this utilization gap is the inherent nature of cloud services: they are valued for their ability to ramp up with minimal notice, offering cutting-edge technology on-demand. GPUs exemplify this, offering unparalleled power for tasks such as training large language models — yet their high costs and specialized nature can lead to sporadic use. As demand for AI increases, exemplified by Nvidia's H100 PCIe cards reaching $40,000 on secondary markets like eBay, cloud providers have had to maintain capacity for peak demands, resulting in a surplus of resources during off-peak periods.
Potential Solutions and Market Dynamics
Efforts to improve utilization rates have been underway. AWS and Google Cloud, for example, have introduced services aimed at optimizing schedules for cost and availability. Additionally, higher levels of service abstraction, such as Amazon's SageMaker, reduce complexity, allowing customers to run AI/ML workloads without concern for the specifics of accelerator optimization.
Despite under-utilization, cloud providers are not alone in the GPU market. Companies like CoreWeave offer competitive rates and specialize in accommodating substantial GPU workloads over short periods. However, market dynamics may shift, with major cloud providers leveraging their scale and purchasing power to dominate the market.
The extensive hype surrounding AI's capabilities in the cloud is unlikely to eclipse the ongoing necessity for a diverse range of services, including CPUs, vast storage, and ample memory. The integration and interplay of these resources will shape the future of cloud computing, beyond just AI workloads.
