For a large part of the past decade, ‘cloud first’ was less of a strategic choice and more of a reaction for enterprise technology leaders. However, AI is now changing that response. With enterprises transitioning generative AI out of pilots and into production, the issues around data sovereignty, costs and the requirement for performance are prompting CIOs to ask an obvious question again: where should each workload really run? For some, the answer lies in re-establishing control of the captive data center.
The AI Price Shock
When an organization scales an AI app into production, the economics change. In the pilot stage, a public-cloud LLM service with token-based pricing may not seem to be expensive, since the workload is small and only a few people are testing it.
However, as usage increases from 30 to 30,000 users, the economics get a lot more complex. Usage becomes less predictable, and some of the costs that seemed reasonable when the application was being tested may become (or soon become) a large, unanticipated budget item. While the application may be very popular and provide a lot of value, it can still be a shock to the budget.
This is where CIOs need to work with their enterprise architects and ask a fundamental question: does this use case require the latest, most powerful proprietary model, or could an open-source model running in the organization’s own data center meet the business requirement? There is a strong probability that the latter suffices.
Cloud Costs and Workload Maturity
Cloud economics recommends pay-as-you-go to perfectly variable, bursting workloads. Yet large, mature enterprise workloads are paying through the nose for elastic capacity that they are not consuming. Add onto this rapidly mounting costs for the data egress-elastically moving data out of a cloud ecosystem-infrastructure-to support data analytics and AI training pipelines.
It’s not an unfamiliar pattern: a piece of infrastructure that has remained flat and predictable for decades is still priced as if it will spike up tomorrow.
Data Sovereignty
Beyond cost, using private data centers for local deployment can give IT leaders more control over where their data lives, security and performance. While this may make private data centers applicable to enterprise AI not en- masse but as one component of a CIO’s overall deployment mix.
For banks, insurers, health care providers and others who are already accountable to regulators in terms of where their data resides and who can see it, the issue of control is a matter for the board, not just a technical fix. Training or tuning a model on internal sensitive data raises it even more.
The Correct Strategy
For enterprise technology decision-makers, the right approach would be to select the appropriate model for the job. For the right workloads, a self-hosted model might give more control and predictable costing, in part by not relying on a usage-based service. However, any comparison of the two should factor in infrastructure, operational and scaling costs, not just the cost per token.
This means that the right approach is a hybrid one. Speed, agility and access to emerging features can be derived from the public cloud, while private infrastructure is likely to be more appropriate for specific workloads, those with a particular cost profile and those with certain data demands. CIOs need to assess where an AI workload fits.
The New Age Datacenter
While there is a business case for enterprises to build a data center, it certainly doesn’t entail returning to the traditional server rooms. For the data center CIOs are considering today, this new data center must be: modular and high-density, designed to house GPU-ready racks with the latest cooling, co-located, and automated with the self-service portals expected from the cloud. It must also be connected into the public cloud so workloads can be shifted on demand.
Of course, it has risks too. CIOs need to plan and purchase hardware in anticipation of demand, power and cooling are real bottlenecks, and finding the right skilled resources to manage such infrastructures can be tricky. An under-utilized GPU farm is as expensive as an out of control token bill. The rule is simple: account for all costs, match each workload to the right environment, and reassess as demand scales.
That said, this can not be a substitute for the cloud but for those businesses which had begun to perceive the cloud as the sole solution, the captive has now earned a renewed right to the table.
